Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation
Authors: Xuxin Cheng
Zhihong Zhu
Wanshi Xu
Yaowei Li
Hongxiang Li
Yuexian Zou
Venue: EMNLP 2023 Findings (2023)
Keywords: Multiple Intent Detection and Slot Filling
Knowledge Distillation
Non-Autoregressive

Abstract: Recent non-autoregressive Spoken Language Understanding (SLU) models attracts increasing attention owing to the high inference speed. However, most of them still (1) suffer from the multi-modality problem since the prior knowledge about the reference is relatively poor during inference; (2) fail to achieve a satisfactory inference speed limited by their complex frameworks. To tackle these problems, in this paper, we propose a $\textbf{T}$argeted $\textbf{K}$nowledge $\textbf{D}$istillation $\textbf{F}$ramework (TKDF), which applies knowledge distillation to improve the performance. Specifically, we first train an SLU model as a teacher model, which has higher accuracy while slower inference speed. Then we introduce an evaluator and utilize the curriculum learning strategy to select proper targets for the student model. Experiment results on two public multi-intent SLU datasets demonstrate that our method can realize a flexible trade-off between inference speed and accuracy, achieving comparable performance to the state-of-the-art models while speeding up by over 4.5 times.

Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation
Xuxin Cheng, Zhihong Zhu, Wanshi Xu, Yaowei Li, Hongxiang Li, Yuexian Zou School of ECE, Peking University, China
{chengxx, zhihongzhu, xwanshi, ywl, lihongxiang j @ @stu.pku.edu.cn zouyx @ @pku.edu.cn
Abstract
Tokens
Possibility
Reference
intent:
atis ciiy, aiis, sirport
atis, llggtt. aiis ground Itoloc.city name
Recent non-autoregressive Spoken Language Understanding (SLU) models attracts increas- ing attention owing to the high inference speed. However, most of them still (1) suffer from the multi-modality problem since the prior knowl- edge about the reference relatively poor dur- ing inference; (2) fail to achieve a satisfactory inference speed limited by their complex frame- works. To tackle these problems, in this paper, we propose Targeted Knowledge Distillation Framework (TKDF), which applies knowledge distillation to improve the performance. Specif- ically, we first train an SLU model as a teacher model, which has higher accuracy while slower inference speed. Then we introduce an evalua- tor and utilize the curriculum learning strategy select proper targets for the student model. Experiment results on two public multi-intent SLU datasets demonstrate that our method can realize flexible trade-off between inference speed and accuracy, achieving comparable per- formance to the of the art models while speeding up by over 4.5 times.
city
slot:
B. B-toloc.city name
intent:
alis meal, alis aireralt I-ranspoot
atis light, aiss ground
service
stot:
intent: atis airline, atis airport atis atis_fight, atis ground fire slot:
ungeles
Hairport
I-cily
Table 1: Three examples of the multi-modality prob- lem in con-autoregressive multi-intent SLU. We use the ground-truth intent label of the utterance as the intent of each token in the utterance.
slot filling. Recently, several graph-based models have shown promising results in jointly handling multiple intent detection and slot filling. Qin et al. (2020) proposes AGIF, which utilizes graph atten- tion networks (GAT) (Velickovic et al., 2018) 8) to predict fine-graine grained multi-intents by integrating in- tent information into the autoregressive decoding process of slot filling. Qin et al. (2021b) proposes the first non-autoregressive SLU model, achieving both speedup and the improved accuracy in multi- intent situations. Xing and Tsang (2022a) proposes the Co-guiding Net, which uses a two-stuge frame- work achieving the mutual guidance between slot and intent. And Xing and Tsang (2022b) proposes Rela-Net, which incorporates a heterogeneous la- bel graph to represent the statistical dependencies and hierarchies in rich relations, along with the current heterogencous label matching network capture and leverage beneficial label correlations in an end-to-end manner. Song et al. (2022) proposes to leverage the statistical co-occurrence frequency between intents and slots as prior knowledge effec- tively in order to enhance the joint multiple intent detection and slot filling through constructing an intent-slot co-occurrenee graph based on the entire training corpus globally discover the correlation between intents and slots. Although existing non- autoregressive multi-intent SLUU models have made promising progress (Qin et al., 202 Xing and Tsang, 2022a,b; Cheng et al., 2023c), we find that
1 Introduction
Spoken language understanding (SLU) plays a piv- otal and indispensable role in task-oriented spoken dialog systems (Young et al., 2013). It aims to un- derstand the queries of users, which includes two subtasks: intent detection and slot filling (Tur and De Mori, 2011). Specifically, intent detection task aims to predict the intent of the given utterance and slot filling task aims to extract the additional infor- mation or constraints expressed in the utterance. In real-world scenarios, it is common for an ut- terance to contain multiple intentions. response to this challenge, Xu and Sarikaya (2013) and Kim et al. (2017) starts addressing the multi-intent SLU task. Gangadharaiah and Narayanaswamy (2019) makes the first attempt to use multi-task frame- work to jointly model multiple intent detection and Corresponding author.

---

most of them still suffer from two problems:
terms of prediction accuracy for the low-frequency words (Ding et al., 2021; Du al., 2021). In SLU task, owing to the limited categories of intents and slots, the decrease in terms of prediction accuracy of low-frequency words has a more serious impact on the overall performance.
(1) Multi-modality Problem. As other non- autoregressive methods (Gu et al., 2018; Ma et al., 2019; Bao et al., 2021), non-autoregressive SLU methods also suffer from the multi-modality prob- lem. As shown in Table 1, there might be multiple possible correct slots of a token and this problem also appears in intent detection with the widespread use of the token-level intent detection decoder. De- spite (GAT) (Velickovic et al., 2018) is adopted to model the interaction between intents and slots, ex- isting models still have little prior knowledge about the reference during the inference progress, leading to some errors that do not generally occur in the au- toregressive models, which limits the performance of the non -on-autoreeresssvve models.
For the second problem, knowledge distillation also seems to be an effective solution. Knowledge distillation has been widely used to improve reason- ing speed without overly sacrificing performance in various tasks (Heo et al., 2019; Liu et al., 2020; Jiao et al., 2020), As a result, we decide to advance along this technical path and apply knowledge dis- tillation to choose the teacher model with the higher accuracy and select a student model with the higher inference speed. By this means, the distilled SLU model would obtain a higher inference speed while almost maintaining the accuracy.
(2) Poor trade-off between inference speed and accuracy. Although Co-guiding Net (Xing and Tsang, 2022a) and Rela-Net (Xing and Tsang, 2022b) achieve new state-of-the-art performance. they are both limited by their complex model archi- tectures and thus cannot infer as fast as the past non- autoregressive SLU models, such as GL-GIN (Qin et al., b021b) and LR-Traanformmrr (Cheng et al.. 2021a). For non-autoregressive models, inference speed is crucial evaluation metric. As a result, it is crucial to implement flexible trade-off between inference speed and accuracy.
In order to address these above two problems at the same time, we try to improve the past methods instead of directly applying traditional knowledge distillation, thereby mitigating the multi-modality problem and achieving the trade-off between infer- ence speed and accuracy. Motivated by the recent success of advanced knowledge distillation in other tasks (Huang and Wang. 2017: Liu et al., 2023; Gou et al., 2023a,b), we propose a Targeted Knowledge Distillation Framework (TKDF). Specifically, we first train multi-intent SLU model as the teacher SLU model with higher accuracy while slower in- ference speed. We regard its output as the distilled data. Then, we train new SLU model with a sim- pler network on the distilled data as the evaluator and use a curriculum learning strategy to obtain the selected data by replacing the original distilled data with original data dynamically during the training process. Finally we train an SLU model with the same architecture as the evaluator on the selected data as the student SLU model. We also propose metric to score the outputs, and the evaluator model assesses each utterance in the original training set through scoring the output intents and output slots.
For the first problem, very common solution is utilize knowledge distillation to first training an autoregressive model in the original training corpus and then applying the greedy outputs of the teacher model as the targets to train the non-autoregressive student model (Hinton et al., 2015; Gu et al., 2018; Stahlberg, 2020; Gou et al., 2021). However, there is still a lack of rescarch on applying knowledge distillation address the multi multi-modality problem in SLU, A very intuitive idea is to directly employ knowledge distillation to non-autoregressive SLU models, where the student SLU model could learn from the teacher SLU model. Unfortunately, our preliminary experimental results demonstrate that this method fails to solve multi-modality problem effectively, whose details could be seen in Sec.6.3.
The utterances with higher scores are chosen as the targets, which typically contain more slight modal- ity changes compared to the distilled data but show better prediction quality. We believe the selected utterances could be more safely exposed to the stu- dent model during training, which is beneficial for the overall performance.
We argue that the reason for these divergent re- sults is that SLU is classification task, while neu- ral machine translation a generative task (Liu and Lane, 2016; Zhu et al., 2023a). Considering that has been verified that directly applying knowledge distillation to the non- an-autorregresive machine trans- lation models will lead to the significant decrease in
We conduct all the experiments on two public benchmark multi-intent SLU datasets, MixATIS

---

and MixSNIPS (Hemphill et al., 1990; Coucke al., 2018: Qin et al., 2020) and over three SLU architectures, Co-guiding Net (Xing and Tsang, 2022a), Rela-Net (Xing and Tsang, 2022b) and SSRAN (Cheng et al., 2022). Experiment results demonstrate that our method can realize a flexible trade-off between inference speed and accuracy, and further analysis also verifies the advantages of our framework. In summary, the core contributions of this work could be concluded as follows:
for slot filling. Co-guiding Net (Xing and Tsang, 2022a) introduces two-stage framework facil- itate targeted enhancements through mutual guid- ance between two tasks. Rela-Net (Xing and Tsang, 2022b) leverages the heterogeneous label graph to further enhance the performance, incorporating sta- tistical dependencies derived from co-occurrence patterns and the hierarchies in slot labels, as well as capturing rich relations among the label nodes. Compared to previous works, the advantuge of our framework is that we mitigate the negative im- pact of the multi-mddality problem and achieve the trade-off between inference speed and accuracy.
We propose a non-oonoroe -oo-eusoreerresive multi-intent SLU framework TKDF, which applics the tar- geted knowledge distillation method.
The proposed method can achieve comparable performance to the state-of-the-art SLU mod- els while achieving faster inference speed.
2.2 Knowledge Distillation
Knowledge distillation (Hinton et al., 2015) is re- garded as an effective method to mitigate the multi- modality problem in non-autoregressive models. In neural machine translation, sequence-level knowl- edge distillation (Kim and Rush, 2016) is widely used to substitute the original translations with the output generated by the pretrained teacher model, Zhou et al. (2020a) further explores knowledge dis- tillation and proposes a range of techniques to fine- tune the complexity of a dataset in order to align it with the capacity of the non- non-autoregressive neural machine translation model, which aims to optimize the performance through effectively matching the intricacy of the dataset with the capabilities of the model. Shao et al. (2022) proposes diverse distilla- tion with reference selection, which generates the new dataset with multiple high-quality references for each source sentence and selects the most fitting reference to train the non-autoregreesive models, underscoring the importance of incorporating the diverse references in knowledge distillation.
Further analysis shows that distilling only 4% of the original data could help the student SLU model surpass its counterpart trained on the original data by a large margin.
2 Related Work
2.1 Intent Detection and Slot Filling
As two primary subtasks of SLU, intent detection and slot filling have sparked the research interests of an increasing number of researchers (Surdeanu, 2013; Mesnil et al., 2014; Hakkani- Tür et al., 2016; Zhang and Wang, 2016; Zhang et al., 2017; E et al.,
2019; Liu et al., 2019; Qin et al., 2019, 2021a; Xie et al., 2023; Huang et al., 2023; Cheng al., 2023a,d; Zhu et al., 2023c). As the strong interde- pendence between these two tasks is verified, an increasing number of joint models are achieving excellent performance (Chen and Yu, 2019; Zhang et al., 2019a; Qin et al., 2020; Bhathiya and Thaya- sivam, 2020; Qin et al., 2021b; Hui et al., 2021; Xing and Tsang, 2022a: Abro et al.. 2022; Xing and Tsang, 2022b; Weld et al., 2022; Cheng et al., 2023b,e; Wu et al., 2023).
Knowledge distillation has also proven its effec- tiveness in SLU task. Chen et al. (2022) proposes a self-distillation approach SDJN for improving joint modeling, allowing the model to leverage an auxil- iary loop and exploits the interrelated connection between intent and slot information in depth. In our work, we utilize knowledge distillation to mitigate the multi-modality problem.
Recently, as the multi-intent SLU problem grad- ually obtains more and more attention, some SLU models based on graph attention are proposed grad- ually. AGIF proposed by Qin et al. (2020) is built upon graph attention model, enabling the direct connections between the slot nodes of each token and all predicted intent nodes. This deliberate es- tablishment of the correlations between slots and in- tents improves the grasp of their relationship, facil- itating a deeper understanding. GL.-GIN (Qin et al., 2021b) constructs a non- -on-autoreeressvvee graph in- teraction network conducting the parallel decoding
2.3 Curriculum Learning
Curriculum learning (Bengio et al., 2009) has been applied in numerous tasks, showcasing its versatil- ity and efficacy (Braun et al., 2017: Graves et al., 2017: Matiisen et al., 2019). In the field of natu- ral language understanding, Xu et al. (2020) uses curriculum learning and defines several easy exam-

---

Distilled
Selected
Data
Data
Targeted Distillation Module
Teacher
Non-autoregressive
Dynamic
auna-utoregreaeive
Score
Model
Evaluator
Threshold
Student Model
Curriculum
Learning
Original Data
Predicted
Results
The architecture of our proposed framework, which consists of three components: (I) original data is fed into the teacher model to obtain the distilled data; (2) an evaluator constructs the selected data via curriculum learning according to the calculated score and the dynamic threshold; (3) non-autoregressive student model is trained on the selected data and produces predicted results.
Figure 1:
ples as those well solved by exact models. In the field of computer vision, Zhang et al. (2019b) pro- poses a novel collaborative self-paced curriculum learning regime, which leverages both the instance level prior-knowledge and the image level prior- knowledge in the unified, collaborative, and robust learning framework. Recently, Huang et al. (2020) embeds the idea of curriculum learning to the loss function to achieve the novel training strategy for deep face recognition, which mainly addresses easy samples during the early training stage and hard ones during the later stage.
where N, denotes the number of the single intent labels, Ng denotes the number of slot labels, g.i.))
denotes the reference intent, Y/ (A-)) denotes asso- ciated predicted intent, jf1..) denotes the reference slot, and y ( denotes its associated predicted slot. Following Qin et al. (2020, 2021b); Zhu et al. (2023b), the final joint objective is formulated as:
L= + BLs
(4)
where and B are hyper-parameters.
3 Background
4 Method
In general, intent detection and slot filling are two subtasks in SLU. Given an input utterance a (I1, ......... En), where is the length of æ, mul- tiple intent detection is multi- classification task and the reference intent sequence is denoted as g' = () g(2.1),.. ÿ ()m.)), where m is the number of intents in æ. Slot filling a sequence labeling task and the reference slot sequence is de- noted as S = (g(1.5), g(2.5), (ins).............. gln..)). And use Ò to denote the the union of ÿ' and gs. Owing to the correlation between slots and in- tents. joint models are utilized to jointly optimize these two subtasks. The multi-intent detection ob- jective and the slot filling objective are as follows:
As shown in Figure 1, our TKDF could be divided into the following parts. First, we feed the original data into the teacher model with higher accuracy while slower inference speed obtain the distilled data. Then, utilize an evaluator module struct the selected data, replacing the distilled data with the original data dynamically. Finally, we train the student model on the selected data.
The presence of multi-modality problem poses challenge for the -otooooooreeesi SLU models, because it hinders accurate slot filling for certain tokens, consequently leading to misguided intent detection. Furthermore, the wide adoption of token- level intent detection decoders also exacerbates the negative impact of the multi-modality problem on intent detection performance.
CF(i. v) = ÿ log (v) + (1 () log (1 v)
Li = N,
In contrast, distilled data output by the teacher (2) SLU model could simplify the training process of non-autoregressive models by reducing the target complexity. However, relying exclusively on dis- (3) tilled data not the optimal approach, as can be vulnerable to the errors made by the teacher model.
CF
(.)) -3. (
1=1
Ni
Ls
Σ>
(5.8)
log
(,5)))

---

An intuitive approach is leverage the strengths of both the original data and the distilled data by combining them. Motivated by Wang et (2021); Liu et al. (2023), we propose a non-autoregressivee evaluator to determine whether the student model should be trained on the original data or the distilled data. The evaluator is first trained on the distilled data. Subsequently, we feed the original utterances into the evaluator, which assesses each prediction made on the original data. In cases where the non- autoregressive evaluator fails to produce outputs that closely align with the reference, we substitute the original labels with the corresponding distilled versions. Specifically, for the utterance æ, we first get the output including the predicted intents and slots utilizing the non-autoregressive evaluator:
from SNIPS dataset (Coucke et al., 2018). Com- pared to the single-domain MixATIS dataset, the MiXSNIPS dataset is more complex due to its di- verse intents and larger vocabulary. The statistics of datasets used are shown in Table 2.
Dataset
MiXATIS
MixSNIPS
Vocabulary Size Intent Categories Slot Categories
722
11241
17
6
116
71
Training Set Size
13162
39776
Validation Set Size
756
2198
Test Set Size
828
2199
Table 2: Dataset statistics.
For all the experiments, we select the SLU model which works the best on the dev set and then evalu- ate it on the tes/ set. For slot filling, we measure the performance using the Fl score, which provides an evaluation of the ability to correctly identify and fill slots in the input utterances. For intent detec- tion, we utilize the accuracy score, which measures the ability accurately predict the intent behind each utterance. Furthermore, we evaluate the over- all accuracy of the utterance-level semantic frame parsing, which accounts for the correctness of all predicted metrics within an utterance.
y = /iaacher( (x)
Then we use the overall accuracy the metric score evaluate the original label ý. Predictions with the higher scores are considered more friendly for the the non-autoregressive student model. The score could be formulated as:
score(æx, 9) Overall(y, 9)
(6)
Inspired by recent success of curriculum learn- ing (Graves et al., 2017; Doan et al., 2019; Zhou et al., 2020b; Luo et al., 2020), we apply curriculum learning to improve the performance. As the train- ing progresses, the ratio of original data gradually diminishes. We establish threshold T, wherein if the score surpasses this threshold, the distilled data will be replaced by the corresponding original data. The specific threshold Tg is determined based on the current training step k:
5.2 Baselines
We compare our method with the following strong multi-intent SLU baselines:
Attention BiRNN (Liu and Lane, 2016); an attention- neural network model for joint intent detection and slot filling.
Slot-Gated (Goo et al., 2018): joint SLU model with slot gate that focuses on learn- ing the relationship between slot and intent to obtain better semantic frame result.
(K. k K 2
Th =
Bi-Model (Wang et al., 201 8): Bi-model to consider the impact between slot and intent.
where (.) denotes the sigmoid function and K denotes the total number of updates.
SF-ID (E et al., 2019): a joint model to estab- lish the connections for slot and intent.
5 Experiments
Stack- Propppprppagtioonn (Qin et al., 2019): a stack- propagation framework to incorporate intent detection to guide slot filling.
5.1 Datasets and Metrics
We evaluate our model using two public benchmark multi-intent SLU " including the cleaned versions of MiXATIS and MixSNIPS. MiXATIS dataset is derived from the ATIS dataset (Hemphill et al., 1990), while MiXSNIPS dataset is derived
AGIF (Qin et al., 2020): an LSTM-based adap- tive framework to achieve the multi-intent in- formation integration.
LR- Transformer (Cheng et al., 2021a): a joint SLU model based on the Transformer with the
hhttp::/ http://ggtttttttee httpill.coooooaaagoeooe wwwp/oopprr///GTTT

---

MiXATIS Slot (FI) Incent(AAc)
MixSNIPS Slot(F])
Models
Speedup
Avec)
(vec)ll(Acc)
Intent(AAc)
Atlention BiRNN (Liu and Lane, 2016)
39.1
86.4
74,6
59.5
89.4
95.4
1.0x
Slot-Gated (Goo et al., 2018)
35.5
87.7
55.4
87.9
94.6
0.9
Bi-Model (Wang ct al., 2018)
34.4
83.9
70.3
63.4
90.7
95.6
1.1x
SF-ID (E al., 2019)
34,9
87.4
66.2
59.9
90.6
95.0
1.2x
Bucck-rropggttons (Qin al., 2019)
40.1
87.8
72.9
94.2
96.0
1.4x
AGII' (Qin et al., 2020)
40.8
86.7
74.4
74.2
94.2
95.1
1.1x
LR-Transformer (Cheng al., 2021b.a)
43.3
88.0
76.1
74.9
94.4
96.6
10.8
GL-GIN (Qin
bb2
43.0
88.2
76.3
73.7
94.0
95.7
11.2x
SDIN (Chen et al., 2022)
44.6
88.2
77.1
75.7
94.4
96.5
0.8x
Co-guiding Net (Xing and Tsang, 2022a)
51,3
89.8
79.1
77.5
95.1
97.7
2.4x
GISCo (Song et al., 2022)
48.2
88.5
75.0
75.9
95.0
95.5
0.8x
DARER (Xing and Tsang, 2023)
49.0
76.3
94.9
96.7
2.5
ReLa-Net (Xing and Tsang, 2022b)
52.2
90.1
78.5
76.1
94.7
97.6
2.8
SSRAN (Cheng al., 2022)
48.9
89.4
77.9
77.5
95.8
98.4
4.2x
Co-guiding w/ Standard KD
48.41
88.51
76.6t
74.8
94.3t
96.21
10.8
Co-guiding w/ TKDF (ours)
50.81
89.61
78.81
77.31
94.61
97.4
10.8
Rela-Net w/ Standard KD
48.81
88.91
76.3
74.41
94.11
95.9
14.3x
Rela-Net w/ TKDF (ours)
51.21
89.81
78.4
75.9
94,21
97.0*
14.3
SSRAN Standard KD
46.8t
88.21
76.81
74.51
94.81
96.67
19.3x
SSRAN w/ TKDF (ours)
48.5t
89,21
77.6
77.2
95.41
98.17
19.3x
Table 3:
Main results. denotes model significantly outperforms baselines with p < 0.01 under t-test. "w/ Standard KD" means that we use the same standard knowledge distillation method as Gu al. (2018).
layered refined mechanism.
mizer (Kingma and Ba, 2015) with B1 0.9, B2 0.98 to optimize parameters in our model. The learning rate warms up 5e and then decays with a inverse square-root schedule. The total num- ber of updates is set to 4000, and the value label smoothing is set to 0.1. For other hyperparam- cters, we follow the values provided in the corre- sponding paper. All the experiments are conducted single Nvidia V100 GPU.
GL-GIN (Qin et al., 2021b): a LSTM-based
global-locally graph interaction framework.
SDJN (Chen et al., 2022): a joint SLU model using self knowledge distillation.
Co-guiding Net (Xing and Tsang, 2022a): a two-stage model achieving the mutual guid- ance between slot filling and intent detection.
GISCo (Song et al., 2022): an SLU model constructing co-occurrence graph based on the entire corpus to make use of the co-occurrence frequency between slot and intent.
5,4 Main Results
Table demonstrates the experiment results, from which we find that TKDF could achieve the compa- rable performance to three strong baselines, includ- ing Co-guiding Net, Rela-Net and SSRAN while increasing the inference speed by large margin. This is due the fact that our TKDF mitigates the negative impact of multi-modality problem by introducing an evaluator to obtain the selected data by replacing the original distilled data with original data dynamically in the training process.
DARER (Xing and Tsang, 2023): an SLU model utilizing the relational Transformer to achieve fine-grained temporal modeling.
ReLa-Net (Xing and Tsang, 2022b): an SLU
model exploiting the label typologics and re- lations among the labels.
SSRAN (Cheng et al., 2022): an SLU model with a scope recognizer and a result network.
Model Analysis
5.3 Training Settings
6.1 Effect of Dynamic Threshold
We select three models the teacher models, in- cluding Co-guiding (Xing and Tsang, 2022a), Rela- Net (Xing and Tsang, 2022b) and SSRAN (Cheng et al., 2022). And we select GL-GIN (Qin et al., 202 lb) as the evaluator model. We use Adam opti-
To evaluate the effectiveness of the dynamic thresh- old strategy, we conduct an experiment comparing its performance with the use of a fixed threshold. Co-guiding Net is employed as the teacher model,

---

51
Standard KD
Targeted KD
50
40-
48
Overall(Aco
Overal(AAc)
47
20
45
44
10
Fixed Threshold
43
Dynamic Threshold
0.0
0.2
0.4
0.6
0.8
1.0
<10
(10,20)
(20,30]
(30,40)
(40.50)
>50
Ratio KD data
Length
Figure 2: Overall accuracy Co-guiding Net + TKDF the MixATIS dataset with the fixed threshold and the proposed dynamical threshold.
Figure 3: Overall accuracy of Co-guiding Net TKDF examples of different lengths the MixATIS dataset.
lengths. Notably, we observe that longer utterances benefit more from targeted knowledge distillation. Intuitively, longer utterances more prone errors throughout the distillation process. Conse- quently, leveraging real data aids the student model in avoiding or rectifying these errors, thereby en- hancing its ability to effectively model the longer utterances. We also observe a slight performance decrease for utterances with fewer than 10 tokens. It is owing to the fact that shorter utterances typi- cally exhibit higher average scores, resulting in pro- longed exposure the non-autoregressive student model. However, the extended exposure orig- inal data may inadvertently introduce confounding factors during model training, as the model be- comes influenced by the inherent multi-modality problem present in the original data.
while GL-GIN serves as the student model. In the experiment, we set a fixed threshold value, denoted as T, to ensure that the training data remained con- stant throughout the training process.
As shown in Figure 2, the model demonstrates significant improvements and surpasses its counter- part by 14.6% in terms of overall accuracy, utilizing only 4% of refined data. This result further val- idates the effectiveness of our targeted distillation method. By selectively filtering original data, the training data complexity could be reduced, lead- ing the enhanced performance. It is also worth noting that the percentage of distilled data in- creases, the rate of performance growth begins to gradually diminish. The superiority of the dynamic threshold strategy over all fixed threshold settings not only shows the advantages of curriculum learn- ing strategy but also enhances the flexibility of our proposed overall model architecture.
6.3 Case Study
To further demonstrate the superiority of our model relative to the previous works on the multiple intent detection and slot-filling problems, we provide a case study as illustrated in Figure 4 which includes the prediction results of utterance sequence der different distillation strategies. From the case, can find that different distillation strategies can improve the original predicted incorrect utterances to various degrees. The prediction results for both slots and intents without distillation have some er-
6.2 Effect of Targeted Knowledge Distillation
To verify the effectiveness of targeted distillation strategy, compare it against the standard distilla- tion. Our experiments are conducted utterances of varying lengths, applying Co-guiding Net as the teacher model and GL--GIN as the student model. Figure 3 showcases the experimental results, specif- ically the overall scores for utterances of different

---

Models
Utterance:
tell
me
about the
ground
transportation from
nashville
airport
Slot:
B-airport_ name
I-airport_ name
Ref.
Intent:
atis_ airfare day name ground service
Slot:
B- B-fromloc.airport name
I-fromloc.airport name
w/o KD
Intent:
atis flight atls ail_ground service atis day name
Slot:
B-alrport_ name
I-airport_namm name
Standard
Intent:
atis_ aii__rrfare atis_ground service atis day_dame name
Slot:
B-airport_ name
l-airport name
TKDF
Intent:
atis airfare _tis_day_me nay_ adls_ground service
Figure 4: A study of our framework under different distillation strategies. Intents and slots in red are those that are predicted incorrectly, and intents in blue are those that are missed in prediction.
due the multi-modality problem. The dis- tilled data encompasses some valuable information derived from high-accuracy teachers, and employ- ing a standard distillation strategy could partially alleviate aforementioned errors. However, the tar- geted distillation strategy yields highly favorable outcomes by accurately predicting intentions and slots. We attribute this success to its capability to not only amalgamate information from the teachers but also retain the original and crucial information.
not sufficient enough and limits the performance some extent. In the future, we plan explore more techniques to further improve the performance.
Acknowledgements
We thank all anonymous reviewers for their con- structive comments. This paper was partially sup- ported by Shenzhen Science & Technology Re- search Program (No: GXWD20201231 202008141 151010001)))) and NSFC (No: 62176008).
7 Conclusion
In this paper, we propose TKDF, a simple yet effec- tive method to solve the multi-modality problem of non-autoreerressvve SLU. We introduce an eval- uator and apply curriculum learning strategy to select proper targets for the student model. Experi- ments and analysis demonstrate the effectiveness TKDE, which could achieve flexible trade-off between inference speed and accuracy.
References
Waheed Ahmed Abro, Guilin Qi. Muhammad Aamir, and Zafar Ali. 2022. Joint intent detection and slot filling using weighted finite state transducer and bert. Applied Intelligence.
Bao, Shujian Huang. Tong Xiao, Dongqi Wang. Xinyu Dai. and Jiajun Chen. 2021. Non- autoregressive translation by learning target categori- cal codes. In Proc. of NAACL.
Ethics Statement
Yoshua Bengio, Jérôme Louradour, Ronan Collohert, and Jason Weston. 2009. Curriculum learning. Proc. of ICML.
We conduct all the experiments using two public datasets. These datasets do contain any person- ally identifiable information, offensive content, or data that can be used to identify individual people.
Hemanthage S Bhathiya and Uthayasanker Thayasivam. 2020. Meta learning for few-shot joint intent detec- tion and slot-filling. In Proc. of ICML.
Limitations
Although our TKDF achieves a flexible trade-off between inference speed and accuracy, does not change the inherent structure. We suppose that the understanding module existing SLU models is
Stefan Braun, Daniel Neil, and Shih Chii Liu. 2017. A curriculum learning method for improved noise ro- bustness in automatic speech recognition. In 2017 25th European Signal Processing Conference (EU- SIPCO).

---

Lisong Chen, Peilin Zhou, and Yuexian Zou. 2022. Joint multiple intent detection and slot filling via self-distillation. In Proc. of ICASSP.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order- agnostic cross entropy for -on-autoreeressvve ma- chine translation. In Proc. of TCML..
Sixuan Chen and Shuai Yu. 2019. WAIS: word attention for joint intent detection and slot filling. In Proc. of AAAI
Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. A novel bi-directional interrelated model for joint intent detection and slot filling. In Proc. of ACL.
Lizhi Cheng, Weijia Jia, and Wenmian Yang. 202 An effective non-autoregressive model for spoken language understanding, In Proc. of CIKM.
Rashmi
Gangadharaiah and Balakrishnan Narayanaswamy, 2019. Joint multiple intent detection and slot labeling for goal-oriented dialog. In Proc. of NAACL.
Lizhi Cheng, Wenmian Yang, and Weijia Jia. 2021b.
result based portable framework for spoken language understanding. Proc. of ICME.
Chih- Wen Goo, Guang Gao. Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng- Veng-Wei Hsu, and Yun-Nung Chen. 2018. Slot llo-gated modeling for joint slot filling and intent prediction. In Proc. of NAACL.
Lizhi Cheng, Wenmian Yang, and Weijia Jia. 2022. scope sensitive and result attentive model for multi-intent spoken language understanding. ArXiv preprint.
Jianping Gou, Liyuan Sun, Baosheng Yu, Shaohua Wan, and Dacheng Tao. 2023a. Hierarchical multi- attention transfer for knowledge distillation. ACM Transactions on Multimedia Computing, Communi- cations and Applications.
Xuxin Cheng, Bowen Cao, Qichen Ye, Zhihong Zhu, Hongxiang Li, and Yuexian Zou. 2023a. MI-Imel: Mutual learning and large-margin contrastive learn- ing for improving asr robustness in spoken language understanding. In Proc. of ACL Findings.
Xuxin Cheng, Wanshi Xu, Ziyu Yao, Zhihong Zhu, Yaowei Li, Hongxiang Li, and Yuexian Zou. 2023b. FC-MTLF: A Fine- and Coarse-graince
Jianping Gou, Xiangshuo Xiong, Baosheng Yu, Lan Du, Yibing Zhan, and Dacheng Tao. 2023b. Multi-target knowledge distillation via student self-reflection. ternational Journal of Computer Vision.
Multi-Tase Learning Framework for Cross-Lingual Spoken Lan- guage Understanding. In Proc. of Interspeech.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A sur- vey. International Journal of Computer Vision.
Xuxin Cheng, Wanshi Xu, Zhihong Zhu, Hongxiang Li, and Yuexian Zou, 2023c. Towards spoken lan- guage understanding via multi-level multi- guati- contrastive learning. In Proc. of CIKM.
Alex Graves, Mare G. Bellemare, Jacob Menick. Rémi Munos, and Koray Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In Proc. of ICMI..
Xuxin Cheng, Ziyu Yao, Zhihong Zhu, Yaowei Li, Hongxiang Li, and Yuexian Zou. 2023d. C A- SLU: Cross and Contrastive Attention for Improving ASR Robustness in Spoken Language Understanding. In Proc. of Interspeech.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. aoo-auttoreressivve neural machine translation. In Proc. ICLR.
Xuxin Cheng, Zhihong Zhu, Bowen Cao, Qichen Ye, and Yuexian Zou. 2023e. Mrr: Modifying the reference via reinforcement learning for non- autoregressive joint multiple intent detection and slot filling. In Proc. of EMNLP Findings.
Dilek Hakkani- Tür, Gökhan Tür, Asli Celikyilmaz, Yun- Nung Chen, Jianfeng Gao, Li Deng, and Ye- Yi Wang. joint semantic frame parsing using bi-directional RNN-LSTM. In Proc. of INTER- SPEECH.
2016.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by-destt be-den voice interfaces. ArXiv preprint.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990.
Liang Ding, Longyue Wang, Xuebo Liu, Derek F.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. 2019. compre- hensive overhaul of feature distillation. In Proc. of ICCV.
Wong, Dacheng Tao, and Zhaopeng Tu. 2021. Un- derstanding and improving lexical choice in non- autoregressive translation. In Proc. ICLR.
Thang Doan, João Monteiro, Isabela Albuquerque, Bog- dan Mazoure, Audrey Durand, Joelle Pineau, and R. Devon Hjelm. 2019. On-line adaptative curricu- lum learning for gans. In Proc. of AAAI.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge neural network. ArXiv preprint.

---

Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. 2020. Curricularface: Adaptive curriculum learning loss for deep face recognition. In Proc. of CVPR.
Tambet Matiisnn, Avital Oliver. Taco Cohen, and John Schulman. 2019. Teacher-student curriculum learn- ing. IEFE transactions on neural networks and learn- ing systems.
Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng. Dilek Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, et al. 2014. Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Zehao Huang and Naiyan Wang. 2017. Like what you like: Knowledge distill via neuron selectivity transfer. ArXiv preprint.
Zhiqi Huang, Dongsheng Chen, Zhihong Zhu, and Xuxin Cheng. 2023. Mclf: A culti-grained con- trastive learning framework for asr-robust spoken language understanding. In Proc. of EMNLP.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. A stack-propen succkppropggttion framework with token-level intent detection for spoken language understanding. In Proc. of EMNLP.
Yanfei Hui, Jianzong Wang, Ning Cheng. Fengying Yu, Tianbo Wu, and Jing Xiao. 2021. Joint intent detection and slot filling based on continual learning model. In Proc. of ICASSP.
Libo Qin. Tailu Liu, Wanxiang Che, Bingbing Kang. Sendong Zhao, and Ting Liu. 202Ia. A co-interactivv transformer for joint slot filling and intent detection. In Proc. of ICASSP.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Proc. of EMNLP Findings.
Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanx- iang Che, and Ting Liu. 2021b. GL-GIN: Fast and accurate -on-autoregressvve model for joint multiple intent detection and slot filling. In Proc. of ACI..
Byeongchang Kim, Seonghan Ryu, and Gary Geunbae Lee. 2017. Two-stage multi-intent detection for spo- ken language understanding, Multimedia Tools and Applications.
Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020. AGIF: An adaptive graph-interactive framework for joint multiple intent detection and slot filling. In Proc. of EMNLP Findings.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation. In Proc. of EMNLP.
Chenze Shao, Xuanfu Wu, and Yang Feng. 2022. One reference is not enough: Diverse distillation with reference selection for von-autoregressive translation. In Proc. of NAACI.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
Bing Liu and Ian R. Lane. 2016. Attention-base -baeed recur- rent neural network models for joint intent detection and slot filling. In Proc. of INTERSPEECH.
Mengxiao Song, Bowen Yu, Li Quangang, Wang Yubin, Tingwen Liu, and Hongbo Xu. 2022. Enhancing joint multiple intent detection and slot filling with global intent-slot co-occurrence. In Proc. of EMNLP.
Min Liu, Yu Bao, Chengqi Zhao, and Shujian Huang. 2023. Selective knowledge distillation for non- autoregressive neural machine translation. In Proc. of AAAL
Felix Stahlberg. 2020. Neural machine translation: A review. Journal of Artificial Intelligence Research.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: a self- distilling BERT with adaptive inference time. In Proc. of ACL.
Mihai Surdeanu. 2013. Overview of the tac2013 knowl- edge base population evaluation: English slot filling and temporal slot filling. TAC.
Gokhan Tur and Renato De Mori. 2011. Spoken lan- guage understanding: Syxtems for extracting seman- tic information from speech. John Wiley & Sons.
Yijin Liu, Fandong Meng, Jinchao Zhang, Jie Zhou, Yufeng Chen, and Jinan Xu. 2019, CM-net: A novel collaborative memory network for spoken language understanding. Proc. of EMNLP.
Petar Velickovie, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In Proc. of ICLR.
Sha Luo, Hamidreza Kasaei, and Lambert Schomaker. 2020. Accelerating reinforcement learning for reach- ing using continuous curriculum learning. In Proc. of IJCNN.
Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. 2021. Selective knowledge distillation for neural machine translation. In Proc. of ACL.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neu- big, and Eduard Hovy. 2019. FlowSeq: Non- autoregressive conditional sequence generation with generative flow, In Proc. of EMNLP.
Yu Wang, Yilin Shen, and Hongxia Jin. 2018. A bi- model based RNN semantic frame parsing model for intent detection and slot filling. In Proc. of NAACL.

---

Henry Weld, Xiaoqi Huang, Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022. A survey of joint intent detection and slot filling models in natural language understanding. ACM Computing Surveys.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020a. Understanding knowledge distillation in non- autoregressive machine translation. In Proc. of ICLR.
Yikai Zhou, Baosong Yang. Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020b. Uncertainty cur- riculum learning for neural machine translation. In Proc. of ACI.
Yirui Wu, Hao Li. Lilai Zhang. Chen Dong, Qian Huang, and Shaohua Wan. 2023. Joint intent detection model for task-oriented comancco dialogue system using asynchronous training. ACM Transactions on Asian and Low-Resoure, Language Information Pro- cessing.
Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, Dongsheng Chen, and Yuexian Zou. 2023a. Enhancing code- switching for cross-lingual slu: A unified view of semantic and grammatical coherence. In Proc. of EMNLP.
Yifeng Xie, Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, and Dongsheng Chen. 2023. Syntax matters: To- wards spoken language understanding via syntax- aware attention. In Proc. of EMNLP.
Zhihong Zhu, Xuxin Cheng. Zhiqi Huang. Dongsheng Chen, and Yuexian Zou. 2023b. Towards unified spoken language understanding decoding via label- aware compact linguistics representations. Proc. of ACL Findings.
Bowen Xing and Ivor Tsang. 2022a. Co-guiding net: Achieving mutual guidances between multiple intent detection and slot filling via heterogeneous semantics- label graphs. In Proc. of EMNLP.
Zhihong Zhu, Weiyuan Xu, Xuxin Cheng, Tengtao Song, and Yuexian Zou. 2023c. A dynamic graph in- teractive framework with label-semantic injection for spoken language understanding. In Proc. of ICASSP.
Bowen Xing and Ivor Tsang, 2022b. Group better than individual: Exploiting label topologies and label relations for joint multiple intent detection and slot filling. Proc. of EMNLP.
Bowen Xing and Ivor W. Tsang- 2023. Relational temporal graph reasoning for dual-task dialogue lan- guage understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understand- ing. In Proc. of ACI.
Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In 2013 ieee workshop on automatic speech recognition and understanding.
Steve Young, Milica Gašić, Blaise Thomson, and Ja- son D Williams. 2013. Pomdp-based statistical spo- ken dialog systems: review. Proceedings of the IEEE
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019a. Joint slot filling intent detec- tion via capsule neural networks. In Proc. of ACL.
Dingwen Zhang, Junwei Han, Long Zhao, and Deyu Meng. 2019b. Leveraging prior-knowledge for weakly supervised object detection under a collab- orative self-paced curriculum learning framework. International Journal of Computer Vision.
Xiaodong Zhang and Houfeng Wang. 2016. A joint model of intent determination and slot filling for spo- language understanding. Proc. of IJCAL
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proc. of EMNLP.


Reviews:
[{"review_id": "yDYtgL3hTD", "reviewer": "Reviewer_bSiD", "summary": "", "questions": "1. In the result, the authors claimed that TKDF \"accelerates the inference speed by 4.5 times, 5.1 times and 4.6 times Co-guiding, Rela-Net and SSRAN, respectively. Since inherent structure of the model is not changed, does the speed improvement come from parallel computing?\n2. In Section 5.2, does the MixSNIPS show similar phenomena that only 4% of the original data can help the student model outperform 14.6% overall accuracy and \"80% distilled data slightly outperforms the model trained on fully distilled data\" as on MixATIS. MixATIS is a simpler task which contains a lot of repeated expressions. MixSNIPS is a more challenging benchmark while MixATIS also need further analysis.\n3. Why dynamic threshold in Figure 3 is a constant number?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. \n\nQ1: Metrics are needed for 4 situations in Table 2 to show its performance.\n\nA1: Thanks for your advice. Due to the limited space, we only report the results for the entire dataset. We will add the results for each of the 4 situations in Table 2 to further show the performance.\n\nQ2: Analyses are missing on MixSNIPS, which is a more challenging benchmark and has better generality.\n\nA2: Thanks for your advice. Due to the limited space, we follow previous work [1] to only conduct analyses on MixATIS. We will take your advice and add the analyses on MixSNIPS in the final version.\n\nQ3: Since inherent structure of the model is not changed, does the speed improvement come from parallel computing?\n\nA3: Thanks for your question. Yes, you are right. The speed improvement comes from parallel computing.\n\nQ4: Does the MixSNIPS show similar phenomena that only 4% of the original data can help the student model outperform 14.6% overall accuracy and \"80% distilled data slightly outperforms the model trained on fully distilled data\" as on MixATIS?\n\nA4: Thanks for your question. Only 3% of the original data can help the student model outperform 4.2% overall accuracy and 75% distilled data slightly outperforms the model trained on fully distilled data on MixSNIPS. Note that since the overall accuracy of GL-GIIN is 73.7 on MixSNIPS, which is relatively high, the improvement in overall accuracy on MixSNIPS is not as significant as with MixATIS. We will add these results on MixSNIPS in the final version.\n\nQ5: Why dynamic threshold in Figure 3 is a constant number?\n\nA5: Thanks for your question. The dynamic threshold in Figure 3 means that the threshold is calculated by using the Equation 8 on line 354, so the corresponding overall accuracy is a constant number.\n \n[1] Xing, Bowen, and Ivor Tsang. \"Group is better than individual: Exploiting Label Topologies and Label Relations for Joint Multiple Intent Detection and Slot Filling.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the paper proposes a novel framework using targeted knowledge distillation and curriculum learning that achieves trade-offs of inference speed and accuracy.\n2. Experimental results shows it outperforms standard knowledge distillation and improves inference speed by a large margin.", "reasons_to_reject": "1. The motivation of multi-modality problem is not well addressed in empirical experiments. Metrics are needed for 4 situations in Table 2 to show its performance.\n2. It requires more evidence to show the necessity of using teacher-student model. Analyses are missing on MixSNIPS, which is a more challenging benchmark and has better generality. ", "questions_to_authors": "1. In the result, the authors claimed that TKDF \"accelerates the inference speed by 4.5 times, 5.1 times and 4.6 times Co-guiding, Rela-Net and SSRAN, respectively. Since inherent structure of the model is not changed, does the speed improvement come from parallel computing?\n2. In Section 5.2, does the MixSNIPS show similar phenomena that only 4% of the original data can help the student model outperform 14.6% overall accuracy and \"80% distilled data slightly outperforms the model trained on fully distilled data\" as on MixATIS. MixATIS is a simpler task which contains a lot of repeated expressions. MixSNIPS is a more challenging benchmark while MixATIS also need further analysis.\n3. Why dynamic threshold in Figure 3 is a constant number?", "paper_topic_and_main_contributions": "The paper proposes TKDF, a model for non-autoregressive multi-intent Spoken Language Understanding using targeted knowledge distillation. The proposed model accelerates reference speed while achieving results similar to the state-of-the-art models. Further analysis reveals that by only 4% of the original data can help the student model outperform baseline student model trained on the entire original data by a significant margin.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g6IStimCId", "reviewer": "Reviewer_y4ok", "summary": "", "questions": "Check Reasons To Reject", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. \n\nQ1: The performance improvement does not appear significant to me. \n\nA1: As stated in the title of the article, the goal of this paper is to accelerate multiple intent detection and slot filling. Although the inference speed of non-autogressive models is usually faster than that of autogressive models, it's also common that non-autogressive models perform worse than autogressive models. We aim to improve the inference speed while causing as little performance degradation as possible. The models with our proposed TKDF outperforms the models with Standard KD and achieves a $\\textbf{much faster}$ inference speed with the original models. Therefore, the performance improvement is significant.\n\nQ2: The whole distinction seems very subjective.\n\nA2: Thanks for your question. We provide 4 situations where the distilled data is different from the original ones to help readers understand the motivation more clearly. Actually, you can also make other distinctions, which will not affect the results.\n\nQ3: The discussion in section 5.3, with just one error example, is not very convincing.\n\nA3: Thanks for your advice. Due to the limited space, we only provide one error example. We will add more analysis in the final version.\n\nQ4:  More detailed ablation study is needed here.\n\nA4: Thanks for your advice.  We will add more detailed ablation studies in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Significant speedup compared to autoregressive Spoken Language Understanding models while achieving comparable performance.\n2. Paper is mostly well written and does a good job to motivate their approach.", "reasons_to_reject": "1. The performance improvement does not appear significant to me. I'm unsure which baselines are used for the p-tests in Table 3. According to the text in 4.4, the baselines should have been Co-guiding Net, 407 Rela-Net, and SSRAN. However, in some cases, such as Co-guiding Net vs. Co-guiding Net w/ TKDF on MixSNIPS, the proposed approach doesn't seem to outperform these baselines. Am I misunderstanding something here?\n\n2. Section 3.1 was a bit hard to read, and I found myself confused about the difference between Slight/Serious Modality Change and Slight/Serious Mistake. The whole distinction seems very subjective to me.\n\n3. One major weakness, in my opinion, is the lack of a strong connection between their motivation, i.e., Multi-modality, and the results. The discussion in section 5.3, with just one error example, is not very convincing.\n\n4. The novelty of their approach seems rather incremental to me, especially considering I'm not entirely convinced if their approach truly addresses the multi-modality aspect. I feel more detailed ablation study is needed here.", "questions_to_authors": "Check Reasons To Reject", "paper_topic_and_main_contributions": "The paper introduces a Targeted Knowledge Distillation Framework (TKDF) designed to enhance the performance of non-autoregressive Spoken Language Understanding (SLU) models. To achieve this, the authors first train a multi-intent autoregressive SLU model as the teacher model. Then, they proceed to train a non-autoregressive SLU model on the distilled data, which acts as an evaluator to determine when to replace the distilled data with the original while training another non-autoregressive student SLU model.\n\nThe authors observed that their approach resulted in a speedup over autoregressive SLU models while also narrowing the performance gap between autoregressive and non-autoregressive SLU models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tMwMMqgQlG", "reviewer": "Reviewer_3Dd1", "summary": "", "questions": "1. Could you explain what is the multi-modality problem of multiple slots illustrated from Table 1? Is there just one slot possibility for each example?\n2. Could you explain and define what original data, distilled data, and selected data are and how you use the benchmark datasets for each of these?\n3. From L119-L121, do you mean you take the predicted labels of the teacher model and their inputs to train an evaluator? Like pseudo-labeling?\n4. Is the distilled data totally separate from the original data or the same data but with the predicted outputs from the teacher model? If that is the case, is it coming from validation or test set to train the evaluator? Then what is the accuracy for the evaluator? What percentage of original data is left to train the final student model?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe that you may misunderstand the paper. We should emphasize that both the teacher model and the evaluator have been trained on the benchmark dataset. We hope you will clear up the misunderstandings based on our response and consider improving your score. For the typos, we will fix them.\n\nQ1: What is the multi-modality problem of multiple slots illustrated from Table 1?\n\nA1: Thanks for your question! Multi-modality problem is a common problem in non-autoregressive models. For a word, there may be many corresponding slots. Since non-autoregressive models do not depend on the previous output, the prediction may be other corresponding slots, which leads to errors. In Table 1, I-transport_type and O are both corresponding slots of service in different utterances. However, the model incorrectly predicts O as I-transport_type.\n\nQ2: What is original data, distilled data, and selected data?\n\nA2: As we mentioned in L119-125. The original data is the data from the original benchmark dataset, the distilled data is the output of the teacher model with higher accuracy while slower inference speed, and the selected data is obtained by Eq. 7 in L347. You could read the Algorithm 1 in Page 5 to understand it more clearly.\n\nQ3: Do you mean you take the predicted labels of the teacher model and their inputs to train an evaluator?\n\nA3: As we mentioned in L390-391. We use a trained GL-GIN on the benchmark dataset as the evaluator. We will further emphasize this point in the final version.\n\nQ4: Is the distilled data totally separate from the original data or the same data but with the predicted outputs from the teacher model? What percentage of original data is left to train the final student model?\n\nA4: The distilled data is the output of the teacher model. Note that both the teacher model and the evaluator have been trained on the benchmark dataset. All the original data is used to replace the distilled data dynamically during the training stage, according to Equation 7 in L347.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a curriculum learning method to select high-quality data for averting multi-modality problem and achieving more competitive performance.\n2. Many quantitative and qualitative experiments are conducted to verify the performance of the model and achieve comparable performance to the state-of-the-art models.", "reasons_to_reject": "1. It’s hard to follow the framework details where additional rewriting and explanation would be suggested. It took me several times to understand the key idea of several sentences like what constitutes distilled data or selected data. \n2. The data usage for each stage of training and testing is vaguely explained which makes it hard to catch up the idea for data filtering process. And some terms are rarely defined and explained. ", "questions_to_authors": "1. Could you explain what is the multi-modality problem of multiple slots illustrated from Table 1? Is there just one slot possibility for each example?\n2. Could you explain and define what original data, distilled data, and selected data are and how you use the benchmark datasets for each of these?\n3. From L119-L121, do you mean you take the predicted labels of the teacher model and their inputs to train an evaluator? Like pseudo-labeling?\n4. Is the distilled data totally separate from the original data or the same data but with the predicted outputs from the teacher model? If that is the case, is it coming from validation or test set to train the evaluator? Then what is the accuracy for the evaluator? What percentage of original data is left to train the final student model?", "paper_topic_and_main_contributions": "The paper proposes a knowledge distillation method to alleviate the multi-modality and accuracy problem in non-autoregressive multi-intent SLU methods. They show comparable performance to the state-of-the-art models while using only few original data to train a student model.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L60 design → desgins\nL118 inference inference → inference\nL154 Can an → can\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the multi-modality Spoken Language Understanding (SLU) problem. The authors aim to enhance the performance of a small-sized, fast-inference-speed student model (which is non-autoregressive) by distilling knowledge from a larger, slower-inference-speed teacher model (which is autoregressive). A novel "Targeted Knowledge Distillation Framework" (TKDF) is introduced. This framework incorporates an evaluator and employs a curriculum learning approach to choose appropriate targets for the student model. Experiments reveal that the distilled student model, though smaller in size, matches the performance of the larger model while being 4.5 times faster in inference.

**Soundness Scores**: (3, 3, 2). 
The paper's soundness is good. It includes results from two benchmarks. In the rebuttal phase, the authors introduced another benchmark result. TKDF exhibits a performance advantage over standard knowledge distillation, even though their inference speeds are comparable. However, to bolster the paper's claims, the following enhancements are suggested: 
1. A comprehensive analysis of the model across all test benchmarks.
2. Inclusion of detailed experimental outcomes in the appendix.

**Excitement Scores**: (3, 3, 3). 
From an excitement standpoint, the paper doesn't particularly stand out. The rationale for employing TKDF to tackle the SLU issue isn't convincingly articulated in the manuscript. As a result, the proposed TKDF appears to be a modest incremental improvement over existing methods.
Decision: Accept-Findings