Novel Slot Detection With an Incremental Setting
Authors: Chen Liang
Hongliang Li
Changhao Guan
Qingbin Liu
Jian Liu
Jinan Xu
Zhe Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: dialog system; novel slot detection; incremental learning

Abstract: Current dialogue systems face diverse user requests and rapid change domains, making quickly adapt to scenarios with previous unseen slot types become a major challenge.
Recently, researchers have introduced novel slot detection (NSD) to discover potential new types. However, dialogue system with NSD does not bring practical improvements due to the system still cannot handle novel slots in subsequent interactions.
In this paper, we define incremental novel slot detection (INSD), which separates the dialogue system to deal with novel types as two major phrases: 1) model discovers unknown slots, 2) training model to possess the capability to handle new classes.
We provide an effective model to extract novel slots with set prediction strategy and propose a query-enhanced approach to overcome catastrophic forgetting during the process of INSD.
We construct two INSD datasets to evaluate our method and experimental results show that our approach exhibits superior performance.

Novel Slot Detection with an Incremental Setting
Chen Liang'; Hongliang Li Changhao Guan 1x Qingbin Liu²,
Jian Liu', Jinan Xu' Zhe Zhao
Beijing Key Lab of Traffic Data Analysis and Mining,
Beijing Jiaotong University, Beijing, China Platform and Content Group, Tencent, China Tencent AI Lab
(21120367, 23120367, 23120344, jianliu, jaxu (bjutu cn Iqingbinliv, nlpzhezhao @tencent. com
Abstract
User: find the schedule for tooth will out at sunrise slot
Dialog system
Current dialogue systems face diverse user requests and rapid change domains, making quickly adapt scenarios with previous seen slot types become a major challenge. Recently, researchers have introduced novel slot detection (NSD) to discover potential new types. However, dialogue system with NSD does not bring practical improvements due to the system still cannot handle novel slots in subsequent interactions. this paper, we de- fine incremental novel slot detection (INSD), which separates the dialogue system to deal with novel types as two major phrases: I) model discovers unknown slots, 2) training model to possess the capability to handle new classes. We provide an effective model to ex- tract novel slots with set prediction strategy and propose a query-enhanced approach to overcome catastrophic forgetting during the process of INSD. We construct two INSD datasets to evaluate our method and experi- mental results show that our approach exhibits superior performance. We release the data and the code https://githii com/ cs cs-1iame
without Novel Slot Detection (NSD)
schedule object type (v)
tooth object_ttyppppeee (x)
Response: tooth will fail off at any time
with Novel Slot Detection (NSD)
schedule - object type (V)
looth will out novel
Response: tooth will out is novel slot,
current system can not handle
with Incremental Novel Slat Detection (INSD)
first
schedule - object type footh will out novel
Response:
out novel slot,
current system not handle
later
schedule - object (v) tooth... out movie_namaa name (v))
Response: at 7.30 am.
An example of how dialogue systems respond to the user request containing an unknown slot 'tooth will out'. Without NSD, system misclassifies slot type (the right type 'movie name') and delivers an incorrect reply. With NSD, system returns an inability to handle the request every time. With INSD, system can provide satisfying response in subsequent interactions.
Figure 1:
1 Introduction
However, prior research in NSD only focuses on identifying new slots (Wu et al., 2021, 2022), which is too idealized and does not bring practical improvements to the dialogue system. Consider the user's request contained an unknown type of slot 'tooth will out' Figure 1, the dialogue system with novel slot detection just provides a reply that is not wrong but less than satisfactory, and will de- 2019; He et al., 2020; Liu et al., 2020b), which is known novel slot detection (NSD) task (Wu al., 2021, 2022).
Slot filling (SF), crucial component of task- oriented dialogue systems, aims identify specific spans in user utterances (Zhang and Wang, 2016; Shah et al., 2019; Qin al., 2021: Liang et al., 2021a; a; Hudeček et al., 2021; Yan et al., 2022). As dialogue systems are widely used in daily life, tra- ditional slot filling needs to be quickly adapted to scenarios with previous unseen types (Shah et al.,
liver an inability to handle it all the time. Actually, dialogue systems need specialized procedure to deal with discovered new slot or entity types.
In this paper, we focus on more practical set- ting: after the model recognizes new types, the dialogue system will view them as known and can return satisfying replies to user requests concerning
Work was done under the guidance of Jian when Liang Beijing Jiaotong University, Li and Guan pro- vided some experimental results. Conresponning author.

---

these types in subsequent interactions, as shown the bottom Figure 1. We start with a collec- tion of annotated examples with some pre-defined types. Our goal is to identify examples with novel types (i.e., types not contained in the pre-defined set), and then, the new types are seen as known, and we should train model to have the ability to identify entities of these types, therefore resulting in incremental learning (Cao et al., 2020; Monaikul et al., 2021; Yu al., 2021).
strategies overcome catastrophic forgetting during incremental learning.
We propose a novel slot detection (NSD) model with set prediction, which demon- strates broader applicability.
Results show that our method delivers the best performance on INSD and NSD tasks.
Related Work
We separate the dialogue system to deal with novel types as two major phrases: 1) model discov- ers slots with unknown classes, 2) training model to possess the capability to handle new classes. We call this unified process incremental novel slot de- tection (INSD). For identifying new slots, we em- ploy the set prediction method to generate a great number of triples (start, end, type), and subdivide the non-pre con-preeefined entity triple into the non-entity triple and the novel-entity triple to extract novel spans. Given that the NSD task lacks a clear ob- jective for fitting due to unseen information of new classes at the training stage, we further provide two strategies to make the model better suit this task, which can improve robustness and encourage model to generate diverse outputs. For the INSD task, we construct two datasets based on slot filling dataset (Coucke et al., 2018; Hemphill et al., 1990). It is generally challenging to mitigate the catas- trophic forgetting (McCloskey and Cohen, 1989) during the process of INSD, inspired by previous works (Cao et al., 2020; Monaikul et al., 2021; Liu et al., 2022: Xia al., 2022), we propose query- induced strategy to realize explicitly knowledge learning from stored data and implicitly knowledge transferring from old model together.
Novel Slot Detection. Prior works have exten- sively studied slot filling (SF) task (Zhang and Wang, 2016; Bapna et al., 2017; Rastogi al., 2017; Shah et al., 2019; Rastogi et al., 2020; Liang et al., 202 Yan et al., 2022) to recognize specific entities and fill into semantic slots for user utter- ances. Recently, Wu et al. (2021) define novel slot detection (NSD) to extract out-of-domain or poten- tial new entities and propose two-step pipeline to detect them, which has attracted much atten- tion. Subsequent works focus on the end-to-end paradigm (Liang et al., 2019) to directly obtain novel entities without intermediate steps. Wu et al. (2023) design a bi-criteria active learning framework and Wu et al. (2022) introduce semi- supervised learning scheme to perform iterative clustering. In contrast to the previous method, we propose a generative method based set predic- tion to produce new entity triples.
The set prediction is an end-to-end method that enables model to directly generate the final ex- pected set, which was proposed in DETR (Carion et al., 2020) to solve object detection task. Re- scarchers have introduced it to natural language processing (NLP). Tan et al. (2021) propose sequence-to-set network handle the complexity of nested relationships in named entity recognition task. Dongjie and Huang (2022) obtain multimodal inputs representation by DETR's s decoder.
We evaluate the proposed method on two com- mon benchmarks and two datasets we constructed. From the results, our approach exhibits promising results INSD and NSD (5 5). We also perform ablation studies assess our method of discover- ing new classes and solving catastrophic forgetting (5 6. Finally, we conduct further experiments to explore the NSD model's 's ability to identify in- domain and novel slots (3 6.2).
Incremental Learning. Incremental learning (Wang et al., 2019; Madotto et al., 2020; Kim et al., 2023) has been introduced to simulate the setting of type/class-increasinn in real-world sce- narios, which is mainly researched in information extraction (IE) (Wei et al., 2022; Zhang and Chen, 2023). Different from assuming data arrived with definite entity types at each stage (Cao et al., 2020; Monaikul et al., 2021), we define a new task incre- mental novel slot detection that type is unknown and needs to discover in the new data stream.
To summarize, the contributions of our work are three-fold:
We define incremental novel slot detection (INSD) aligned with real-world scenarios to improve the dialogue system's 's ability to deal with new slot types, and we provide effective

---

Catastrophic forgetting (McCloskey and Cohen, 1989) is a long-standing problem in incremental learning. Current approaches to overcome this is- sue can be roughly classified into two groups. The first is knowledge distillation (KD) (Li and Hoiem, 2017; Monaikul et al., 2021; Yu et al., 2021; Zheng et al.. 2022; Kang et al., 2022), which transfers the knowledge from the previous model into the current model. The second is data retrospection (Cao al., 2020; Liu et al., 2022; Xia al., 2022), which samples old data and stores them in a mem- ory buffer with a fixed capacity. In this paper, we propose a query-induced strategy to enhance KD and data retrospection.
at the top of Figure 2.
3.2 Novel Detection with Set Predicted Network
Next, we describe the proposed NSD model, which is depicted at the bottom-left of Figure 2. We first present the details of the model structure, which consists of two main components: query represen- tation and learning and bipartite matching. Finally, we show how to extract novel entities.
Query Representation. Our model locates novel entities by recognizing the start positions and end positions based on group of queries. Consider- ing the information of the novel entity is entirely invisible NSD task, we employ a random ini- tialization strategy for the representation matrix of queries and optimize them by the model itself during training, which is distinct from the joint encoding approach in the machine reading com- prehension (MRC) model (Liu et al., 2020a). Let QN E IRdx and Qe E Rdd stand for the embed- ding matrix of queries Q = {q1.... q to predict start position and end position respectively, where I is the number of queries and d is their dimension.
3 Approach
3.1 Problem Definition
Novel slot detection (NSD) is a fresh task in slot filling (SF). For given sentence S, a NSD model aims at extracting potential new entities or slots in S (Wu et al., 2021). The slot in the NSD dataset /) consists of two type sets: pre-defined type set Tр and novel type set TN, Sentences in the training set only contain Tp, whereas sentences in the test set include both Tp and Tn.
Learning and Bipartite Matching. For the given sentence S, we adopt BERT-large (Devlin et al., 2018) encode it and get representation H. By fusing Qs, Q. and H, the model can generate triples (s.e, 1). which is composed of the start posi- tion s, end position e and slot type t. Specifically, we first calculate probability distributions of each token being start index and end index, as follows:
In realistic scenarios, once the model has discov- ered new slot classes, these types should be treated as known. Therefore, we define a new task incre- mental novel slot detection (INSD) aligned with real-world scenarios. Following previous work in incremental learning (Cao et al., 2020; Monaikul et al., 2021), we adopt the assumption that the entity class arrived at different time points. But different from them, novel types will be identified at each stage and be seen as known in the next stage in our setting.
P. = softmar(Q.. H)
(1)
Pe softmar(Qe HI)
(2)
For INSD task, we construct the dataset D D&} with & stage, where Dj is an-
{Di, (2,,,,.... D2...., notated and others is unlabeled. Each Dj in D .2..... where
Then integrate the above positional probabil- ity information into H to get span representation, which can be regarded as an attention mechanism:
contains a slot type set Tpx types in TpI are pre-arranged in the initial stage and (1 > 1) is composed of new types detected stage).
H Hayan = Σ(Cp! H(0) + (P()). H()) (3)
by model M (M;1 is training at In other words, the slot classes that are predicted novel in stage will be viewed as in-domain types in i-th stage. At step i, we first apply the model Mc-1 to annotate D; with slot types Tp which are new discovered classes step Then, we train M; 1 on labeled D, and get M,. M; also has the competence to identify novel classes contained in next-stage data. The process is shown
where the superscript (6) means the i-th value in matrix. Finally, we predict the slot class of H spann which comprises pre-defined slot types and label 'O' (indicates that span does not belong to in- domain slot types). We compute the probability distribution of each class as follows:
P. = T. H Hspon + bb AbCRR. lx((+11)
(4)

---

unloboled
Data1
Stage
Data2
Stage
Data3
Stage 3
countyy strte faciity
diecover new
Fack
aridaee
discover
sort
input
+
R10
adi no sing
ON Data
Trsining Dutu
query
BERT
query
Data
Set Prection
discarded dedd dala selected data
prolstype representation
Retrospection
generate
INSD
Old Model
non-entity (-1. [SEP]]
country)
I input
3 prob
17, 7, album;
(14, 15, '01
(11,12, track)
novel
(17, 17, 01
Current Model
pre-defined
non-pre-defined
Knowladge Distiliation
Figure 2: The overview of our method. At each stage, NSD model discover novel slot types and annotate next-stage data with these types. Our proposed NSD model is illustrated at the bottom left and the approach of overcoming catastrophic forgetting is shown bottom right.
where T and b are parameters in the linear layer, and is the number of pre-defined entity type.
of the model to in-domain types and promote the model to generate diverse triples. Random values added to the query representation Qs and Qe at the training stage.
Given that sentence may contain entities, use hungarian algorithm (Kuhn, 1955) optimize the matching of triples between the predicted entity y = (s.e, 1) and golden entity á
more
3.3 Query Enhanced Incremental Learning
(8, ê, f).
Based constructed NSD model, now intro- duce approach mitigate the problem of catas- trophic forgetting during the process of INSD, sep- arated by data retrospection and knowledge distil- lation with query representation enhancement. We summarize the procedure in Algorithm 1.
For training model, we adopt the following loss:
Cly, 0) We P w;P:(((0)
(5)
where Wss We and wy are the weights for different tokens and entity types.
Query Enhanced Data Retrospection Data ret- rospection method manages a constrained memory buffer B with a capacity of c to store small num- ber of old examples. The capacity size is constant due data storage costs. When new data comes, will update the memory buffer release allo- cated space for accommodating new types. There- fore, selecting uppropriate examples from old data is crucial for data retrospection. Prototype repre- sentation has been successfully utilized in samples filtering (Cao et al., 2020; Zalmout and Li, 2022), which is center vector integrating all features from given dataset. The caleulation is follows:
We modify the model's loss to predict novel entities. Specifically, the value for the last token '(SEP)' and the
Novel Type Prediction.
in w/ and value in w, for the label 'O" are both set to smaller value than others. After training the model on the corrected loss, there are two forms triplet for non- pre-defined entities: (Sxeps Exep: t): the span is the last token "[SEP]' with any label, ii) (s, e, O): the label 'O' with any span. By assigning appro- priate values for Wh, We and wy, we can treat the second form (s, e, O) as novel entity triple.
We further present two strategies to make the model better suit the NSD: i) contrastive learn- ing: we pull the entity embeddings of the same type together in mini-batch (Zhang et al., 2021) improve model robustness. ii) noise-induced adversarial learning: we propose adversarial train- ing with noise injection to alleviate the overfitting
H N
H
(6)
where N the number of data, H the embed- ding vector of the first token '[CLS]' in a sentence and is seen as the sentence representation.

---

We believe there are distinctions between slot classes, attributed to differences in data volume and learning difficulty. Consequently, we propose a new prototype learning with a query-induced data split strategy, rather than uniformly partitioning the space according to slot types. We first leverage query to cluster data: the sentences in which novel entities were successfully identified by query qi grouped together subset. Then we compute the prototype representation of the subset and calculate the distance between the center and each sentence subsets. The sentence with a small distance will be sampled and stored in the buffer B.
Algorithm 1 Addresses Catastrophic Forgetting
Input: training set {D1, D..., D&} with stage, memory buffer B with capacity c, NSD model M
for i= 1,2..... do
2: get training data D; U
3: train model M; and distill from M;-1
4: split D; based on query and select c/t data release c/è space and store selected data
end for
Dataset
Train
Dev.
Test
Slot-Type
In our method, we treat data at each stage equally, the memory B is divided evenly according to the number of stages. At stage i, discard c/i data in B and utilize that memory buffer store the examples sampled in D;.
Snips ATIS
13,084
700
700
39
4,478
500
893
44
Table I: Statistics of Snips and ATIS.
Query Enhancement Integration We integrate the aforementioned two query enhancement strate- gies to realize explicitly knowledge learning from old data and implicitly knowledge transferring from previous model together. We summarize the proce- dure Algorithm 1.
Query Enhanced Knowledge Distillation Knowledge distillation addresses catastrophic forgetting by transferring the knowledge already acquired in the previous model. At i-th stage, M4-1 acts as a teacher model that possesses the ability to identify old slot classes, and M; serves as a student model. By distilling the knowledge from M,1 to M,, M, can largely overcome the catastrophic forgetting for old types.
Experimental setups
Experimental Setting. We conduct experiments on two slot filling datasets: Snips (Coucke et al.. 2018) and ATIS (Hemphill et al., 1990). Snips is annotated with 13,084 sentences and has 39 slot types. ATIS provides 4,478 training examples with 44 slot types. The statistics of the datasets are shown in Table 1.
The distillation module of previous methods gen- erally contains encoded sentence representation and predicted probability distribution (the proba- bility of the start position, end position, and type classification in our model). The loss is calculated as follows:
For NSD task, we follow the data setup in Wu et al. (2021). We employ the remove processing strategy that randomly selects several classes as unknown and the sentences containing unknown slots are deleted from the training set. For metrics, apply the span-level precision (P), recall (R), and fl-scores (FI).
Leent = MS
(7)
Sp = (L=
(8)
where KL represents the KL divergence loss and MSE denotes the mean squared error function. F] 1 and [:]; are the value computed by model Mi-1 and M,, respectively. Considering that the query plays crucial role in the extraction of span and corresponding type, we also distill the query representation:
For INSD task, we construct two multi-stage datasets based on Snips and ATIS. We divide the official training set into k subsets ( (D1, D2, DR) and assign slot types {Tpl, Tp2.... ......,yy Tpk for each stage. We process the D; as follows: 1) directly remove the sentences including types in Typ (>i), 2) label the slot values belonging to Tn (-<i) with OO' and the corresponding text tokens are replaced with 'MASK', 3) D; (i >1) is unannotated and needs NSD model to discover and label novel slots.
Equery = MSE(()))-1,
(9)
Finally, we train the model with the following loss:
Lxd = Lsent + Cp + Exuery
(10) We exploit four metrics to assess model perfor-

---

in iirroet for known types (Smips)
matro-fl for Inownn types 15mps]
micmol for all hylpee [Srips]
marroet for all types (5mpp)
60
50
50
50
40
40
40
50
30
30-
20
-0n
3.5
20
10
10-
types
tyoes
types
typen
60
60
40
T5BC
PLNC
20
20
15
Duse
Figure 3: The results of the INSD model at each stage the test set of Snips (first row) and ATIS (second row) with four evaluation metrics. 'max' refers to the upper bound, which trains model with all available data. 'base" means the lower bound without incremental learning strategy. For i-th stage, 'known types' is the types in stage 1-i.
5%
10 %
15%
Dataset
Model
IND
NSD
IND
NSD
IND
NSD
GDA-min (Wu al., 2021)
93.14
29.73
85.28
14.28
90.07
31.96
Snips
GDA-differemt (Wu al., 2021)
93.10
31.67
86.22
15.06
90.18
32.19
Our method
93.57
39.49
91.52
35.19
91.30
40.72
GDA-min (Wu et al., 2021)
90.14
10.17
93.57
23.18
93.92
50.92
ATIS
GDA-different (Wu et al., 2021)
90.68
10.27
94.01
22.98
93.88
43.78
Our method
94.84
39.20
93.04
38.53
93.37
61.71
Table 2: Model performance (fl-score) two benchmarks. 'IND' and 'NSD' denote pre-defined and novel entity types respectively. '.6'' the proportion potential new classes.
mance: micro-averaged fl and macro-averaged fl for current known types, and micro-averaged fl and macro-averaged fl for all types.
strategies GDA-min and GDA-different.
Overall Results
Implementations. We employ BERT (Devlin et al., 2018) as contextualized encoder to imple- ment our model. For constructed multi-stage INSD dataset, the k is set to 5 in Snips and 3 in ATIS. The model is trained for 30 epochs with learning rate of le-5 and batch size of The query number is selected (20, 40, 60, 80) and the weight to generate novel slot span is chosen from (1e-1, 1e-3, le-5). We set the capacity of memory as 100.
In this section, we present the main results of incre- mental novel slot detection (INSD) framework and novel slot detection (NSD) model.
Model Performance of INSD We visualize the results of our proposed approach and baselines dur- ing the incremental learning process in Figure 3. It can be observed that our method demonstrates the best performance compared with baseline ap- proaches for all metrics and datasets, achieving 22.73% macro-fl and 41.75% micro-fl Snips and 26.37% macro-fl and 72.53% micro-fl on ATIS respectively. Such results indicate that our method shows the best ability to overcome catas- trophic forgetting for the INSD task. From the fig- ure, we also see that the data retrospection method
Baselines. For INSD task, compare ap- proach to the knowledge distillation model KL (Monaikul al., 2021) and the data retrospection model KCN (Cao al., 2020). For NSD task, we compare the proposed method to the pipeline model GDA (Wu et al., 2021) with two distance

---

(KCN) brings much higher fl improvements than the knowledge distillation approach (KL), which suggests that the explicit data-storing strategy is more effective. Our proposed approach introduces queries to enhance them and can further improve model performance.
Stage
Stage
Setting
P
R
F1
P
R
F1
Full model
58.30
51.53
54.71
54,41
33.87
41.75
Base model
26.74
21.46
23.81
11.38
5.83
7.71
w KL
42.04
31.99
36.33
37.18
8.16
13.38
w KI(Q)
40.67
33.69
36.85
45.77
8.51
14.36
w DutaStore
54.12
44.51
48.85
44.96
27.34
34.01
w (ataStorc())
56.57
52.64
54.53
51.34
33.29
40.39
Model Performance of NSD Table presents the performance of our method as well as base- lines on two datasets. For extracting novel slots (NSD), we can see that our model outperforms pre- vious methods by a large margin for all datasets and scenarios. For example, GDA-different by gaining improvements 7,82%, 20.13%, and 8.52% in terms of Fl FI-scores differ- ent settings on Snips. For identifying in-domain slot (IND), our model also obtains comparable re- sults compared with baselines, but it exhibits in- ferior performance in some settings such as 15% sampling ATIS. By comprehensively analyzing, we can find that our proposed model demonstrates better capability in discovering novel types of slots than known slots.
Table 3: Ablation study of INDS model on Snips dataset. We report the micro- averaged P. R. and Fl-score for current known types at the 3-th and 5-th stage. 'Base' means the model without incremental learning strategy, "(Q)" refers to method with query enhancement.
method beats
Dataset Setting
5%
10% 15%
Aavg
Base model
34.25 31.16 33.55
w CL
35.24 29.23 35.79
0.41
Snips
w NL
36.63 33.16 39.41
3.4
Full model
39.49 35.19 40.72
5.5
Base model
37.85 35.93 58.20
w CL 38. 11 35.19 59.28
0.2
ATIS
w NL
38.38 35.14 59.82
0.41
Full model 39.20 38.53 61.71
2.5
6 Discussion
Table 4:
Ablation study of our NDS model two datasets. 'Base model' stands for the base set predic- tion model without 'CL' (contrastive learning) and 'NL' moise-induuee adversarial learning).
We conduct further experiments to investigate the proposed INSD framework and NSD model, sepa- rated by ablation study and the learning ability of the NSD model.
6.1 Ablation Experiment
tion, unlike typical NLP tasks, NSD task lacks a clear objective for fitting and trained with slot filling (SF) task, which makes essential to intro- duce noise during training stage. By employing both CL and NL, the model demonstrates a much higher improvement in F1-score, surpassing the sum of the gains obtained by each method individu- ally. A reasonable explanation is that our proposed CL can enhance model robustness by encouraging entity embeddings of the same type to be close, which mitigates the negative impact of noise when training model.
Ablation Study of INSD We perform ablation experiments to investigate the proposed INSD method. The results of 3-th and 5-th stage are depicted in table 3. As can be seen, query enhancement approach can lead better perfor- mance, attaining 1% and 6% increases in terms of Fl separately in knowledge distillation and data storage. Results demonstrate that query represen- tation is important in novel slot extraction and in- dicate that first clustering the data via queries and then calculating the prototype representation can select beneficial samples.
The Impact of Weight Parameters and Query Number We study the effect of weight and query number on model performance. For simplicity, we set fixed entity type weight w, for the label 'O' and assign the same value for the last position weight w, and we. Figure 4 depicts the results. We can conclude that: 1) Our model exhibits high sensitiv- ity to the weight and query number. 2) As the count of novel entities increases (5%-) 100% 10%0+100011 +15)) the
Ablation Study of NSD We conduct an ablation study to explore the impact of contractive learning (CL) and noise-induced adversarial learning (NL) on our NSD model. The results are presented in Table As expected, CL and NL can both yield greater Fl scores than the base model. Overall, the NL method is more effective because noise can contribute to generating diverse outputs. In addi-

---

35
32
35
Snips
Snips 10%
-30
Snips 15%
-30
30
1e-1
1e-2
1e-1
Weight
1e.2
Weight
1e.3
Weight
1e:2
25
26
1e:3
1e-4
1e:3
20
20
1e.4
1e-5
10.4
20
30
40
50
60
70
10
20
30
40
50
60
30
40
50
60
70
80
Query Number
Query Number
Query Number
58
ATIS 5%
35.0
ATIS 10%
ATIS 15%
57
1e-2
32.5
1e-4
30
1e-4
Weight
1e-5
Weight
1e-5
30.0
55
1e.4
1e-6
25
1e-6
1e:5
1e-7
1e-7
54
Query Number
Query Number
Query Number
Figure 4: The impact of the count of query and the value of weight.
model requires greater number of queries and reduced weight values, which both encourage our model to generate larger quantity of novel entity triplets (s, e, f). Specifically, the increasing number of queries allows the model to learn various query representations for different novel slot types, while decreasing the weight values result in higher count of (s, e, O) triplets (the label 'O' with any span) and lower count of (Ssep: Cxep; t) triplets (the span is the last token "[SEP]' with any label).
Model
SimL
SimM
DisimL
DisimM
GDA-min GDA-diff
5.70
16.41
15.38
20.50
5.71
16.38
15.41
20.63
Our Method
20.52
21.58
16.76
24.11
scores of different novel slot class setup. 'Sim' and 'Disim' represent whether the novel types are similar to pre-defined types. 'L' (little) and 'M' (many) stand for the number of novel types.
Table
6.2 Learning Ability of NSD Model
The Ability Discover Novel Entity We want to explore the NSD model's inclination in identi- fying novel slot classes, are newly discovered and pre-defined types similar or dissimilar? To answer this, we first randomly select a subset of classes in-domain and then choose novel classes from the remains based on label semantics similarity and entity representation similarity. We conduct periments two settings 'Sim' and 'Disim'. The results are shown in Table 5. We can observe that the previous model GDA tends to discover novel types that differ from pre-defined types, whereas our model delivers superior performance in all set- tings. Results reveal the broader applicability of our model.
Figure 5: Visualization of the proportion of new and pre-defined slots identified by each query.
where the queries that only recognize the non-entity span were not drawn and we only present the re- sult in 5% and 15% scenarios on Snips. From the figure, we can see that the count of queries for non- entity greatly outnumbers the queries for the entity, such as 7 entity queries and 73 non-entity queries in the 15% sampling setting, primarily due the higher number of non-entity spans. Interestingly, it can be observed that each entity query possesses the capability to recognize both potential new and pre-defined entities in our model.
Learning Ability of Queries We investigate the ability of queries identify various types of spans, containing non-entity, novel entity, and in nn-oomain entity. The proportion of new and pre-defined enti- ties extracted by each query is plotted Figure 5,

---

7 Conclusion
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbreehh, Francesco Calta- girone, Thibaut Lavril, al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by-desige voice interfaces. arXiv preprint arXiv: 1805. 10790.
In this paper, we introduce a new task incremental novel slot detection (INSD) and establish bench- mark with two datasets and four metrics. Moreover, we propose a query-indueed strategy to alleviate the problem of catastrophic forgetting during in- cremental learning and provide an effective model to identify novel slots. Our proposed approach achieves state-of-the- performance in all datasets and scenarios.
Jacob Devlin, Ming-Wac Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv:/810.04805.
Zhang Dongjie and Longtao Huang. 2022. Multimodal knowledge learning for named entity disambiguation. In EMNLP.
8 Limitations
Although the proposed method exhibits superior performance in overcoming catastrophic forgetting and identifying novel slots, the extracted entity quantity decreases as the stages advance, and sev- eral classes cannot even be recognized. However, for incremental novel slot detection task, it is cru- cial to discover as many unseen classes possi- ble. Another limitation is that our model to detect new slots shows high sensitivity to the weight and query number, which means that researchers need to spend time to carefully tune the parameters for different datasets.
Keqing He, Jinchao Zhang, Yuanmeng Yan. Weiran Xu, Cheng Niu, and Jie Zhou. 2020. Contrastive zero-shot learning for cross-domain slot filling with adversarial attack. Proceedings of the 28th Inter- national Conference on Computational Linguistics.
Charles T Hemphill, John Godfrey, and George R Doddington. 1990. The atis spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Vojtëch Hudečck, Ondřcjj Dušck, and Zhou Yu. 2021. Discovering dialogue slots with weak supervision. In ACI.
Minsoo Kang, Jacyoo Park, and Bohyung Han, 2022. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVT conference on computer vision and pattern recognition.
9 Acknowledgements
This work was supported by the Fundamen- tal Research Funds for the Central Universities 2023JBMC058. It was also supported by the National Natural Science Foundation of China (No.62106016 61976015, 61976016, 61876198 and 61370130), and the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zix- uan Ke, and Bing Liu. 2023. Open-world continual learning: Unifying novelty detection and continual learning, arXiv preprint arXiv:2304. 10038.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly.
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence.
References
Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017. Towards zero-shot frame se- mantic parsing for domain scaling. arXiv preprint arXiv.1707.02363.
Yunlong Liang. Fandong Meng, Jinchao Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2021a. A dependency syntactic knowledge augmented interactive architec- ture for end-to-end aspect-basee sentiment analysis. Neurocompating.
Pengfei Cao, Yubo Chen, Jun Zhao, and Taifeng Wang. 2020. Incremental event detection via knowledge consolidation networks. In EMNLP.
Yunlong Liang, Fandong Meng, Jinchao Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 202 An iterative multi- knowledge transfer network for aspect-based sentiment analysis. In EMNLP.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey 7agoruyko. 2020. End-to-end object detection with transformers. In Computer Vision-ECCO 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Pari I 16.
Yunlong Liang, Fandong Meng. Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019. A novel aspect-guided deep transition model for aspect based sentiment analysis. In EMNLP.

---

Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020a. Event extraction as machine reading comprehension. In EMNLP.
Yanan Wu, Zhiyuan Zeng, Keqing He, Hong Xu, Yuan- meng Yan, Huixing Jiang, and Weiran Xu. 2021. Novel slot detection: A benchmark for discovering unknown slot types in the trie-oriented dialogue sys- tem. arXiv preprint arXiv:2105. 14313.
Minqian Liu, Shiyu Chang, and Lifu Huang. 2022. Incremental prompting: Episodic memory prompt for lifelong event detection. arXiv preprint
Yuxia Wu, Tianhao Dai, Zhedong Zheng, and Lizi Liao. 2023. Actively discovering new slots for task-oriented conversation. arXiv preprint arXiv:2305.040%
arXiv:2204.07275.
Zihan Liu, Genta Indra Winata, Peng Xu, and Pas- cale Fung. 2020b. Coach: A coarse-to-he to-fine ap- proach for cross-domain slot filling, arXiv preprint WrXiv2200..11227
Yuxia Wu, Lizi Liao, Xuemming Qian, and Tat-Seng Chua. 2022. Semi-supervised new slot discovery with incremental clustering. In EMNLP.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Se- ungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. 2020. Continual learning in task-oriented dialogue systems. arXiv preprint Xxxx:2012./5500
Xia, Quan Wang, Yajuan Lyu, Yong Zhu, Wenhao Wu, Sujian Li, and Dai Dai. 2022. Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples. In ACL.
Michael McCloskey and Neal J Cohen. 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. In Psychology of learn- ing and motivation.
Yang Yan, Junda Ye, Zhongbao Zhang, and Liwen Wang. 2022. Aisfg: Abundant information slot filling generator. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4180-4187.
Natawut Monaikul, Giuseppe Castellucci, Simone Fil- ice, and Oleg Rokhlenko. 2021. Continual learning for named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence.
Pengfei Yu, Heng Ji, and Prem Natarajan. 2021. Life- long event detection with knowledge transfer. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanx-
iang Che, and Ting Liu. 2021. Gl-gin: Fast and accurate non- non-autoregressive model for joint multi- ple intent detection and slot filling. arXiv preprint
Nasser Zalmout and Xian Li. 2022. Prototype- representations for training data filtering in weakly- supervised information extraction.
arXiv:2106.01925.
Abhinav Rastogi, Dilek Hakkami-Tin Türr, and Larry Heck. 2017. Scalable multi-domain dialogue state tracking. In 2017 IEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE.
Hanlei Zhang, Hua Xu, Ting-Ea En Lin, and Rui Lyu. 2021. Discovering new intents with deep aligned clustering. In AAAL.
Xiaodong Zhang and Houfeng Wang. 2016. A joint model of intent determination and slot filling for spo- ken language understanding. In ISCAI, volume 16.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema- schema-guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence
Yunan Zhang and Qingcai Chen. 2023. neural span- based continual named entity recognition model. arXiv preprint arXiv: 12200.
Darsh Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani- Tur. 2019. Robust zero-shot cross-domaie slot filling with example values. arXiv preprint arXiv:1906.06870
Junhao Zheng, Zhanxian Liang, Haibin Chen, and Qianli Ma. 2022. Distilling causal effect from miscel- lancous other-class for continual named entity recog- nition. arXiv preprint arXiv:2210.03980..
Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. 2021. A sequence-to-set net- work for nested named entity recognition. arXiv preprint arXhv:2105.0800)
Weikang Wang, Jiajun Zhang, Qian Li, Mei-Mei Hwang, Chengqing Zong, and Zhifei Li. 2019, Incremental learning from scratch for task-orie oriented dialogue sys- tems. arXiv preprint arxiv: (00.:9900.49999 04991.
Kaiwen Wei, Zequn Zhang, Li Jin, Zhi Guo, Shuchao Li, Weihong Wang, and Jianwei Lv. 2022. Heft: history-enhaneee feature transfer framework for incremental event detection, KBS.


Reviews:
[{"review_id": "JqdAQyVCcm", "reviewer": "Reviewer_XgD7", "summary": "", "questions": "1. How catastrophic forgetting  influence on novel slots detection,  any supported , quantitative or qualitative experiments and analyses？", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful and detailed reviews.\n\nQ1: How catastrophic forgetting influence on novel slots detection?\n\nA1: The influence of catastrophic forgetting is in INSD (incremental novel slots detection),   not in NSD (novel slots detection). For INSD, at stage i, we can only access data $D_i$, but the data in previous stages (1 to i-1) is not available. Catastrophic forgetting will impact the extraction of slot types belonging to the previous stages.  The results are shown in Figure 3 (Section 5) and Table 3 (Section 6.1). \nIn the Figure 3, ’base’ means the method without an incremental learning strategy, which trains the model only with the data at that stage, while ’max’ refers to the upper bound for solving catastrophic forgetting, which trains the model with all available data (data in stage 1 to i).\n\nQ2: Experiments for contrastive learning and adversarial learning are missing.\n\nA2: The related experiments are described in Section 6.1 (Ablation Study of NSD) and the results are shown in Table 4. \n\nQ3: The method is little bit complex and hard to follow.\n\nA3:  Thank you for this concern. We have included example codes in the Supplementary Materials and will make all codes available once the paper is accepted.\n\nQ4: About the study on large language models.\n\nA4: LLM possesses the capability to recognize all entities known to humans, while the NSD task focuses on identifying unseen slot types based on the understanding of known slot types. Therefore, using LLM to solve NSD may not be reasonable, unless it has not encountered some slot types during training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduce a new incremental new slot identification method to overcome catastrophic forgetting problem. \n\n2. The paper adopts quite a lot of techniques to improve the tricky issue of new slot detection and forgetting issue, like query-induced strategy, knowledge distillation,  data retrospection, prototype representation,  query  representation  learning and bipartite matching, contrastive learning, noise-induced adversarial learning,  etc. \n", "reasons_to_reject": "1. The problem that how catastrophic forgetting exerts a strong influence on novel slots detection keeps unclear. \n\n2. The paper does not study on large language models, which may be the current SOTA  models  for novel slots detection and  their effective usage in dialogue context.\n\n3. The method is little bit complex and hard to follow, e.g., how the method implement  the final effect in Figure?\nThe proposed  may be not easy to implement in real-world scenarios. \n\n4. some experiments are missing , e.g.,  contrastive learning and  adversarial learning.\n\n5. The comparing baselines  is only few while the proposed method is claimed to be SOTA model.", "questions_to_authors": "1. How catastrophic forgetting  influence on novel slots detection,  any supported , quantitative or qualitative experiments and analyses？", "paper_topic_and_main_contributions": "This paper introduces a new slot detection approach for dialogue system and proposes a two-stage method , i.e., first finding the new slot types and then training the model to deal with these new slot classes, to incrementally to alleviate catastrophic forgetting issue and improve the model performance for novel slot discovery.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "028: ?-> specific citation"}, "extra_text": {}}, {"review_id": "C0GIlGInO6", "reviewer": "Reviewer_J6xJ", "summary": "", "questions": "1. Have you tried to use self-labeling strategy like clustering to replace human-labeling?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful and detailed reviews.\n\nQ: Have you tried to use self-labeling strategy like clustering to replace human-labeling?\n\nA: Thank you for your suggestion, we will consider self-labeling strategies in our future work. In this paper, the provided datasets are based on the processing of existing datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem is interesting and valuable.\n2. The experiments and analysis are solid and convincing.", "reasons_to_reject": "Lacking more baselines. Maybe add several slot filling or incremental learning baselines.", "questions_to_authors": "1. Have you tried to use self-labeling strategy like clustering to replace human-labeling?", "paper_topic_and_main_contributions": "This paper introduces a new task incremental novel slot detection (INSD) and proposes a query-induced strategy to alleviate the problem of catastrophic forgetting during incremental learning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0yejs8KQBH", "reviewer": "Reviewer_Eu35", "summary": "", "questions": "* What's the meaning of '$j$' in Equation (4)? What is T? What does $H_{span}$ refer to?\n* What is $P_s(s|sˆ)$ in Equation (5)?\n* Why does pulling the entity embeddings of the same type together in a mini-batch help?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the insightful and detailed reviews.\n\nQ1: How a novel class bind to a concrete slot (e.g., movie_name) in databases?\n\nA1: The model is capable of identifying whether slots belongs to novel classes, while the corresponding concrete types require human in the loop or the addition of extra classifiers. Because we are assuming that the novel types are completely unseen, we have not supplied a set of new types for alignment in our method.\n\n\nQ2: How to avoid annotation errors？\n\nA2: Sorry for not detailed description for this. In our setting, we will filter the false annotations based on the golden annotations to improve the quality of labeled data.\n\n\nQ3: For real application, how to name $Tp_{i}$ at different state?\n\nA3: This depends on the data from real-world scenarios. Assuming the model is currently in stage i, it will receive a substantial amount of user data $D$ that includes novel types. The slot types in $Tp_{i+1}$ are those new types in $D$ that the model identified.\n\n\nQ4: About the meaning of Equation (4) $P_{t} = T \\cdot H_{span} + b \\in \\mathbb{R}^{1 \\times (j+1)}$?\n\nA4: Eq.4 is employed to classify the type of the identified slot, where $H_{span} \\in \\mathbb{R}^{1 \\times dim} $ is the vector representation of the slot span, $T$ and $b$ stand for the weight and bias in the linear layer respectively, $j$ is the number of pre-defined slot types and the addition of ‘+1’ is to consider label ‘O’. After calculation, we obtain the probability distribution $ P_{t} \\in \\mathbb{R}^{1 \\times (j+1)}$.\n\nQ5: What is $P_{s}(s|\\hat{s})$ in Equation (5)?\n\nA5: This is done to compute the cross-entropy loss between the predicted values $s$ and the golden values $\\hat{s}$ concerning the start positions.\n\n\nQ6: Why does pulling the entity embeddings of the same type together in a mini-batch help?\n\nA6: For NSD task, it can establish a clearer boundary between pre-defined slot types and label ‘O’, thereby preventing the pre-defined types from affecting the process of finding novel slots within spans labeled as 'O'.\n\n\nAdditionally, sorry for the typos and we have revised them in the final version. Thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method can improve performance of novel slot detection (NSD) significantly.", "reasons_to_reject": "* The authors claim that their method can handle new classes with a satisfying response in subsequent interactions. However, I didn't find how a novel class bind to a concrete slot (e.g., movie_name) in databases. Is it human in the loop?\n* [Line 178-179] The assumption that the entity class arrived at different time points may not be right for a real application with many users.\n* The authors apply the model $M_{i−1}$ to annotate $D_i$ with slot types $T_{p_i}$, while it may contain annotation errors. How to avoid that?\n* For real application, how to name $T_{p_i}$ at different state?\n* I think the authors may overstate the contributions of this work. I would say the proposed method is a new novel slot detection method incorporating with incremental learning. It does not show how to handle new classes to provide a satisfying response.", "questions_to_authors": "* What's the meaning of '$j$' in Equation (4)? What is T? What does $H_{span}$ refer to?\n* What is $P_s(s|sˆ)$ in Equation (5)?\n* Why does pulling the entity embeddings of the same type together in a mini-batch help?", "paper_topic_and_main_contributions": "This paper aims to solve the problem of novel slot detection, which can discover potential new types in dialogue systems. The authors define incremental novel slot detection (INSD), which separates the dialogue system to deal with novel types as two major phrases: 1) model discovers unknown slots, 2) training model to possess the capability to handle new classes. They construct two INSD datasets to evaluate the proposed method and experimental results show that their approach exhibits superior performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* 'i-1' -> '$i-1$'\n* 'i-th' -> '$i$-th'\n"}, "extra_text": {}}, {"review_id": "MQwLC7aBqC", "reviewer": "Reviewer_Mmoy", "summary": "", "questions": "1. Table 2. For the Snips dataset, why the NSD results in a decrease from 5% to 10% and an increase from 10% to 15%?\n\n2. Line 374. Step 2, Why replace the text token with MASK, and slot values belonging (T_p<i) are labeled with O? Should it be T_p>i?\n\n3. Line 190 After a new slot value is detected at the current stage, the novel slots will be viewed as in-domain types. Does that mean only the set of the new slot names becomes available? Or all the labeled data for those new slots are available?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful and detailed reviews.\n\nQ1: For the Snips dataset, why the NSD results in a decrease from 5% to 10% and an increase from 10% to 15% in Table 2?\n\nA1: That is because it contains novel types that are relatively difficult to identify in 10% setting, such as ‘compartment’. In all three scenarios, the selection of novel types is random.\n\nQ2: Line 374. Step 2. Why replace the text token with MASK, and slot values belonging ($T_p<i$) are labeled with O? Should it be $T_p$>I? \n\nA2: Slot values that belong to ($T_p$<i) are labeled with O, not $T_p$>i. In our setting, we assume that the data $D_i$ at the i-th stage only includes the slot types of the current stage. For sentences containing slot types from stage j (where j>i), as shown in line 371, we directly remove these sentences from $D_i$. For sentences containing types from stages j (where j<i), we replace the corresponding text tokens with MASK and slot values with 'O' in $D_i$, which ensures that $D_i$ does not contain the slot types belonging to stages 1 to i-1.\n\nQ3: Line 190. After a new slot value is detected at the current stage, the novel slots will be viewed as in-domain types. Does that mean only the set of the new slot names becomes available? Or all the labeled data for those new slots are available?\n\nA3:  The labeled data for those new slots are available. As described in line 192, we will employ a NSD model to annotate these novel slot types within the data.\n\nAdditionally, we appreciate the provided missing references and presentation adjustments. We have revised them in the final version. Thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Propose a research problem that is more relevant to the realistic setting, i.e., incremental slot detection (both out-of-distribution and in-distribution), which requires continually detecting new slot values and deploying the new slot prediction models. This could be an interesting research topic for online dialog system development.\n\n2. For slot detection, the paper utilizes learnable query vectors for feature matching and sequence prediction and use Hungarian algorithm to optimize the labeled triples. the same query can also be used to retrieve some representative data from the training set for incremental learning to avoid the catastrophic forgetting problem. Knowledge distillation is also used for fast model adaptation. This framework could potentially be useful and efficient in real dialog systems.", "reasons_to_reject": "1. Lacking strong baselines. The incremental learning for dialog systems is a classical topic, many different approaches, like architecture-based, memory-based, and data-retrieval-based are proposed in previous work[1], and also compared with strong multi-task learning baseline.  Since this paper focuses on incremental learning, it needs more comparison with different baselines.\n\n2. Lacking more complex benchmarks to verify the effectiveness of the method. Current benchmarks are small and toylike. More realistic dialog datasets like MultiWOZ, SGD, and dataset on DialogStudio should be considered.  In addition, ChatGPT or GPT-4 results on this benchmark should be included for more comparison. This may not be required, but it's crucial to demonstrate how hard and challenging the new task is.\n\n[1] Madotto A, Lin Z, Zhou Z, et al. Continual learning in task-oriented dialogue systems[J]. arXiv preprint arXiv:2012.15504, 2020.\n", "questions_to_authors": "1. Table 2. For the Snips dataset, why the NSD results in a decrease from 5% to 10% and an increase from 10% to 15%?\n\n2. Line 374. Step 2, Why replace the text token with MASK, and slot values belonging (T_p<i) are labeled with O? Should it be T_p>i?\n\n3. Line 190 After a new slot value is detected at the current stage, the novel slots will be viewed as in-domain types. Does that mean only the set of the new slot names becomes available? Or all the labeled data for those new slots are available?\n", "paper_topic_and_main_contributions": "This paper proposes a task called incremental novel slot detection that continually detects new slots, labels new slots, and retrains the slot detection model for the next-round detection. The authors adapt the SNIPS and ATIS, two classical slot-filing datasets to fulfill this setting. They proposed to combine contrastive learning and noise-induced adversarial learning for novel type prediction and propose a Query Enhanced Knowledge Distillation to alleviate the catastrophe-forgetting problem during incremental learning.  Experimental results show the efficacy of the proposed framework when compared with a novel-type prediction baseline and two continual learning baselines. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Open-vocabulary slot-filling:\n\n1. Rastogi A, Zang X, Sunkara S, et al. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(05): 8689-8696.\n\n2. Rastogi A, Hakkani-Tür D, Heck L. Scalable multi-domain dialogue state tracking[C]//2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017: 561-568.\n\n3. Bapna A, Tur G, Hakkani-Tur D, et al. Towards zero-shot frame semantic parsing for domain scaling[J]. arXiv preprint arXiv:1707.02363, 2017.\n\n4. Dai Y, Zhang Y, Liu H, et al. Elastic CRFs for Open-Ontology Slot Filling[J]. Applied Sciences, 2021, 11(22): 10675.\n\n\nIncremental dialog learning\n\n1. Wang W, Zhang J, Li Q, et al. Incremental learning from scratch for task-oriented dialogue systems[J]. arXiv preprint arXiv:1906.04991, 2019.\n\n2. Madotto A, Lin Z, Zhou Z, et al. Continual learning in task-oriented dialogue systems[J]. arXiv preprint arXiv:2012.15504, 2020.\n\n3. Kim G, Xiao C, Konishi T, et al. Open-World Continual Learning: Unifying Novelty Detection and Continual Learning[J]. arXiv preprint arXiv:2304.10038, 2023.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "1.  The contribution of the paper is not very focused. The new task, or the bipartite matching for NSD, or the query-enhanced data retrospection?  Better to focus on one aspect and add extensive experiments or introduce new realistic datasets. \n\n2. Line 28 citation ?\n\n3. Line 34 research -> researcher\n\n4. Line 256  what is w_b ? is it w_s?\n"}}]

Author rebuttal:

Meta-review: The paper focuses on a newly proposed task, incremental novel slot detection, and introduces a method combining multiple techniques to tackle this task and mitigate the catastrophic forgetting problem. All reviewers agree on the relevance and applicability of the new task to real-world dialogue system development, and the potential of the framework proposed.

Despite recognizing the merits of the paper in targeting an area of interest, the reviewers raise concerns about the paper due to the shortcomings in experiments, supporting methodology and baselining against already established models/methods.
Decision: Accept-Findings