Dual Class Knowledge Propagation Network for Multi-label Few-shot Intent Detection
Year: 2023

Abstract:
Multi-label intent detection aims to assign multiple labels to utterances and attracts increasing attention as a practical task in task-oriented dialogue systems. As dialogue domains change rapidly and new intents emerge fast, the lack of annotated data motivates multi-label few-shot intent detection. However, previous studies are confused by the identical representation of the utterance with multiple labels and overlook the intrinsic intra-class and inter-class interactions. To address these two limitations, we propose a novel dual class knowledge propagation network in this paper. In order to learn well-separated representations for utterances with multiple intents, we first introduce a labelsemantic augmentation module incorporating class name information. For better consideration of the inherent intra-class and inter-class relations, an instance-level and a class-level graph neural network are constructed, which not only propagate label information but also propagate feature structure. And we use a simple yet effective method to predict the intent count of each utterance. Extensive experimental results on two multi-label intent datasets have demonstrated that our proposed method outperforms strong baselines by a large margin.

heading: 1 Introduction
text: Multi-label intent detection aims to assign multiple labels to a single utterance rather than only one label since the user utterance often carries multiple different intents. As a practical task in real-world scenarios, multi-label intent detection attracts more attention with the rapid development of conversational AI (Louvan and Magnini, 2020). Previous works have achieved promising performance in multi-label intent detection, but they rely on large amounts of labeled training samples (Xu
∗ Corresponding author.
and Sarikaya, 2013; Qin et al., 2020a,b, 2021). However, due to fast-emerging intents and rapidly changing domains, the lack of adequate labeled data for each intent is inevitable, thus motivating the multi-label few-shot intent detection (Hou et al., 2021). Few-shot learning, a promising way to tackle data scarcity challenges, can recognize new classes from only a few labeled samples by exploiting the prior knowledge from source domain.
Recent works for multi-label intent detection mainly focus on threshold-based strategy which consists of relevance scoring and threshold estimation (Gangadharaiah and Narayanaswamy, 2019; Hou et al., 2021) to improve performance. Nevertheless, they suffer from the following two limitations. First, for multi-label utterances, most of them neglect the semantic connections between labels and input utterances and therefore learn the identical utterance representations for different labels, which confuses the similarity scoring. As Figure 1 shows, the embeddings for request_date and request_time are the same, i.e., the embedding of utterance x1. Furthermore, in few-shot scenarios, it’s critical to obtain proper intra-class and inter-class relations from only a few samples, while previous works overlook the high-order interaction of samples and just usually use a specific metric to compute the relevance score directly, e.g., cosine similarity.
8605
In this paper, we propose a novel framework for multi-label few-shot intent detection, as illustrated in Figure 2. It obtains well-separated class representations by fully exploiting the label semantic information and propagates class knowledge and feature structure by graph neural network to reduce intra-class variance and model proper inter-class relations. To deal with the confused embeddings for multi-label utterances, we introduce label-semantic augmentation (LA) to obtain well-separated representations without any complicated model architecture. Inspired by the effectiveness of label name information (Luo et al., 2021), LA appends each class label name after utterances to construct multiple label-specific utterances, where each label-specific utterance is corresponding to only one label. In this way, LA not only eliminates ambiguity but also generates more labeled training samples.
To better capture more complete and more adaptive interactions among samples, we propose a novel dual class knowledge propagation network, containing an instance-level graph and a class-level graph. It not only propagates label information from labeled support samples to unlabeled queries but also propagates feature structure in their latent space. The instance-level graph smoothing the representations among highly related utterances reduces the intrinsic gap between augmented support samples and queries. The class-level graph provides label correlation and generates completed pseudo class-level representations for queries with the guidance of the instance-level graph. Also, we design a calibration strategy to transfer label dependency into instance representations. Then the
two representations are fused to classify queries. Finally, to select accurately multiple intents for an utterance, we employ an adaptive and powerful method to predict the number of intents.
The contributions of this paper are as follows: (1) We propose a novel dual class knowledge propagation network (DCKPN). DCKPN incorporates label semantic connections to guide feature propagation and label information propagation. (2) We introduce a novel label-semantic augmentation method for multi-label samples to obtain more discriminative representations. And we employ a simple inference module to predict the number of intents in utterances adaptively. (3) To verify the effectiveness of our proposed model, we conduct a series of experiments on two datasets. The empirical study shows that our model can achieve state-of-the-art performance in comparison with other strong baselines.

heading: 2 Related Work
text: Multi-label Classification The existing multilabel classification (MLC) methods mainly focus on learning enhanced representations (Liu et al., 2017) and modeling label dependency (Yang et al., 2018; Tsai and Lee, 2020). Recently, some works (You et al., 2019; Xiao et al., 2019) have been proposed to learn a label-specific representation by employing attention mechanism or utilize the label statistical co-occurrences to explore the semantic interactions (Ma et al., 2021), which shows the importance of exploring semantic interactions for MLC. However, these methods require large amounts of labeled samples for training, thus it’s
hard to be applied to fast-emerging new domains in few-shot scenarios.
Few-shot Learning Few-shot learning aims to recognize novel classes from a handful of annotated samples by leveraging the previous knowledge from source domains (Fei-Fei et al., 2006). Recently, meta-learning has become a successful paradigm for solving the few-shot problem, which can be divided into three categories: model-based methods (Munkhdalai and Yu, 2017), optimizationbased methods (Finn et al., 2017; Yoon et al., 2018) and metric-based methods (Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018). However, these methods focus on single-label classification and only a few works investigate multi-label fewshot classification. Previous studies focus on image domain (Alfassy et al., 2019), audio domain (Cheng et al., 2019) and sentiment analysis (Hu et al., 2021). CTLR (Hou et al., 2021) is the first work to address multi-label few-shot intent detection tasks which proposes a meta-calibrated threshold mechanism and learns anchored label representation. However, it relies on a specific metric to estimate the relevance score, losing the inherent relations among samples and classes.
Propagation in Graph Model Graph neural networks are effective in modeling the relationship among different nodes and have shown powerful performance in many real applications, e.g., nodes classification (Kim et al., 2019) and link prediction (Zhang and Chen, 2018). In few-shot scenarios, several approaches have been explored. Specifically, DPGN (Yang et al., 2020) constructs both the distribution-level and instance-level graph. Liu et al. (2019) propose a transductive propagation network (TPN) by utilizing label propagation (Zhou et al., 2003), a classical and effective method to transfer knowledge from neighbors of each node. Rodríguez et al. (2020) yield a smoother embedding space by employing embedding propagation. Inspired by this, our method aims to propagate information both in intra-class and inter-class, called dual class knowledge propagation network (DCKPN).

heading: 3 Proposed Method

heading: 3.1 Problem Definition
text: Few-shot learning aims to train a model that can perform well in cases where only a few samples are given. We follow the episodic paradigm, an
effective solution for few-shot learning, which is commonly employed in various studies (Snell et al., 2017; Yang et al., 2021). In general, the model is first trained on a series of tasks from source domains with numerous samples ( Dtrain) episode by episode until convergence, then directly adopted to another set of unseen novel tasks (Dtest), where a few labeled samples are available. There is no overlap between the training classes and testing classes. Specifically, each episode (task) has a support set S = {(xi,yi)}N×Ki=1 , which usually includes N classes with K labeled samples for each class, i.e., the N -way K-shot setting, and a query set Q = {(xj ,yj)}qj=1, where q is the number of query samples.
Multi-label few-shot intent detection enables each utterance to be assigned with a set of intent labels simultaneously. To be more specific, for each example (x,y), x is the utterance and the label vector is denoted as y = {y1, y2, ..., y|C|} ∈ R|C|, where C is the set of intent labels in the episode and yi ∈ {0, 1}. In this paper, we focus on multi-label few-shot intent detection, i.e., selecting multiple associated intent labels for each utterance in few-shot scenarios.

heading: 3.2 Label-semantic Augmentation
text: For multi-label few-shot intent detection tasks, one of the key points is to learn more discriminative class representations which could capture accurate class-relevant information while eliminating ambiguity via a few examples. To achieve this goal, we propose to use label name to augment the support set. Different from Luo et al. (2021), which have explored integrating label information into sentences, we propose to append each class label name after each sentence with multiple possible labels. As Figure 3 shows, for each utterance in support set,
we append its labels one by one after it, i.e., "[CLS] utterance [SEP] label name [SEP]", thus generating different label-specific utterances corresponding to single different class labels. For each sample in query set, due to unknown labels, we take the common format "[CLS] utterance [SEP]" as input text.
There are two benefits of using the label name to augment utterances. First, we make use of different appended label name to distinguish the same utterance with multiple labels, thus obtaining more discriminative representations for each class. As illustrated in Figure 3, we can obtain different prototypes for class 1 and class 2, eliminating the problem that the multi-label utterances share the same representation essentially without any complex architecture or parameters. Second, this enables us to utilize the label name information as well as increase the number of support samples to eliminate the ambiguity caused by the scarcity of utterances. In Section 4.4, we analyze the experimental results to verify the improvements of augmentation.

heading: 3.3 Class Knowledge Propagation Network
text: Although one can directly calculate class prototypes via label-semantic augmented support set to classify the queries, doing so will result in the loss of intra-class and inter-class information, which is critical for multi-label classification tasks (Zhang et al., 2018). To mitigate these issues, we propose a dual class knowledge propagation network (DCKPN) to propagate class knowledge from support samples to queries. As illustrated in Figure 2, we first utilize the support and query samples to build the instance-level graph GI = (V I ,W I) and utilize label name information to build the classlevel graph GC = (V C ,WC). Notably, in the following section, the support samples are labelsemantic augmented samples described in Section 3.2, which means that each augmented support sample in instance-level graph has its corresponding single label in the class-level graph. Then, we modify the instance-level embeddings based on pairwise node relations. Guided by the classlevel graph, we also calibrate relations of supportsupport nodes to integrate label semantic relevance explicitly. Next, we update class-level representations based on instance relations to generate pseudo query class-level representations. Finally, the two representations are merged to guide the inference of the queries.

heading: 3.3.1 Graph Construction
text: For each sample (xi,yi,ai) ∈ S in augmented support set, we employ a pre-trained language model as the backbone to extract its instance-level feature vIi = fϕ(xi) ∈ Rd and its class-level feature vCi = fϕ(ai) ∈ Rd, where ϕ is the parameters of the backbone model and ai is the label name of utterance xi. Due to unknown query labels, for each query sample xj ∈ Q in query set, we regard vIj = fϕ(xj) ∈ Rd as its instance-level feature and 0 ∈ Rd as its class-level representation. Then, the adjacency matrix is defined as follows:
Wij = exp(− d(vi,vj)
σ2 ),Wii = 0, (1)
where σ2 = V ar(d(vi,vj)) and d(vi,vj) = ∥vi − vj∥22. Thus, we obtain the instance-level graph GI = (V I ,W I), where V I ∈ R(|S|+|Q|)×d, and W I ∈ R(|S|+|Q|)×(|S|+|Q|). While for classlevel graph GC = (V C ,WC), V C ∈ R(|S|+|Q|)×d consists of the feature embeddings of the label names in the support set V Cs ∈ R|S|×d as well as a matrix of zeros 0 ∈ R|Q|×d for queries, and WC ∈ R(|S|+|Q|)×(|S|+|Q|) is as follows:
WC = ( WCss 0 0 0 ) . (2)

heading: 3.3.2 Instance-level Propagation
text: We now perform message passing on the instancelevel graph to conduct higher-order interactions between augmented support embeddings and query embeddings. Specifically, we first normalize the weight matrix W I as,
LI = D−1/2W ID−1/2, Dii = ∑
j
W Iij . (3)
Then, using the propagation solution described in (Zhou et al., 2003), we calculate the propagation matrix and rectify the instance-level embeddings as,
Ṽ I = (I − αLI)−1V I , (4)
where I is the identity matrix and α ∈ R is a smoothing factor.
To further capture the inter-class relationships and reduce the intra-class variance, we utilize the guidance of relations of support-support pairs in class-level graph to calibrate the support-support pairs in instance-level graph, thus further encouraging the relation learning of instances with
semantically related classes. For example, request_temperature share more semantic information with request_low_temperature than with appreciate. We implement the semantic loss by calibrating the pair relation in support set:
Lsem = 1
|WCss|
|S|∑
i=1
|S|∑
j=1
|wCij − wIij |. (5)

heading: 3.3.3 Class-level Propagation
text: As illustrated in Figure 2, the instance information flows back into the class-level graph to generate the completed semantic knowledge representations for both support set and query set. Concretly, we calculate the Laplacian of the updated instance embeddings L̃I = D̃−1/2W̃ ID̃−1/2, where D̃ii = ∑ j W̃ I ij , W̃ I ij = exp(−d(ṽIi , ṽIj )/σ̃2), σ̃2 = V ar(d(ṽIi , ṽ I j )), and ṽ I i is obtained from Eq. (4). Next, the complete class-level embeddings are obtained as:
Ṽ C = (I − αL̃I)−1V C , (6)
where V C is the original label name embedding matrix in class-level graph. Through the guidance of connectivity strength of instance pairs, we finally generate pseudo class representations for queries without labels. Thus, by explicitly establishing the bidirectional interaction between instance-level graph and class-level graph, we conduct feature propagation and derive completed representations to model intra-class and inter-class relationships.

heading: 3.4 Objective
text: Prototypical Classifier. Due to the two-level features being complementary, we concatenate the embeddings to obtain better feature representations. For each sample xi, the embedding is defined as follows:
ṽi = ṽ I i ||ṽCi , (7)
where || denotes the concatenation and ṽIi , ṽCi are sample xi’s instance-level and class-level embeddings respectively. The prototype p̃c of intent class c ∈ C can be calculated as p̃c = 1|Sc| ∑ xi∈Sc ṽi, where Sc denotes the set of support samples with intent class c and C is the class set. Given a query data (xi,yi) ∈ Q, we compute the conditional probability p(y = c|xi,S) based on negative squared Euclidean distance:
p(y = c|xi,S) = exp(−d(ṽi, p̃c))∑
c′∈C exp(−d(ṽi, p̃c′)) . (8)
Finally, we perform cross-entropy loss on all query samples. Note that in the multi-label setting, as an utterance may have multiple labels, we need to consider |C| labels for each query sentence. The classification loss is calculated as:
Lce = − 1 |Q| ∑
xi∈Q
|C|∑
j=1
yji logp(y = j|xi,S), (9)
where yi = {y1i , ..., y |C| i } is the label and y j i ∈ {0, 1}. Label Number Prediction. For multi-label fewshot intent detection, one of the challenges is to determine the number of intents in the utterance. We leverage a simple and effective method to decide the number of labels. Given an utterance x, we can obtain its fused feature ṽ, and then we use a multi-layer perceptron to predict the number of intents in x:
n = softmax(MLP(ṽ)), (10)
where n ∈ RN is the indicator for the number of intents and N is the maximum count of possible intents. In meta-training stage, we calculate the cross entropy loss of intent count as follows:
Lnum = − 1 |Q| ∑
xi∈Q
N∑
j=1
tji log(n j i ), (11)
where ti = {t1i , ..., tNi } is the ground-truth intent count vector of xi and t j i ∈ {0, 1}.
By combining Eqs. (5), (9) and (11), the overall loss of our proposed framework is:
L = Lce + βLsem + γLnum, (12)
where β and γ are adjustable weight parameters.

heading: 4 Experiments

heading: 4.1 Datasets and Setups

heading: 4.1.1 Datasets
text: We follow (Hou et al., 2021) to conduct experiments on two multi-label intent detection datasets: TourSG and StanfordLU. TourSG is a dataset of touristic information in Singapore, which includes 25751 samples annotated with multiple labels from 6 different domains: It (Itinerary), Ac (Accommodation), At (Attraction), Fo (Food), Tr (Transportation), and Sh (Shopping). StanfordLU is a Stanford dialogue dataset (Eric et al., 2017) and contains
8038 utterances re-annotated by Hou et al. (2021) from 3 domains: Sc (Schedule), Na (Navigate) and We (Weather). Different from the standard Nway K-shot setting, an utterance may have multiple possible intents in multi-label few-shot intent detection and we refer you to (Hou et al., 2021) for more episode construction details. Table 3 shows the detailed dataset statistics.

heading: 4.1.2 Experiment Setups
text: Evaluation Metrics We follow (Hou et al., 2021) to use micro F1 scores to evaluate the performance. All reported results are the average results of 5 different runs.
Parameter Settings We conduct experiments on 1-shot and 5-shot settings. For each episode from different domains, the support set has Ns classes (Ns-way), where Ns is the label numbers in domain s. And the query set size is 16 for TourSG and 32 for StanfordLU. In terms of feature extraction, we use the pre-trained Electra-small (Clark et al., 2020) and Bert-base-uncased (Devlin et al., 2019) model. For the loss function, we set β as 0.1 and γ as 0.01. The smooth factor α is 0.1. We use AdamW (Loshchilov and Hutter, 2019) optimizer with the initial learning rate 2e − 4 for Electra, 1e− 4 for Bert, and the dropout rate is 0.1. All the hyperparameters are selected based on the performance of the development domain.

heading: 4.2 Baselines
text: We compare our proposed method with the following strong baselines.
TransferM is a standard fine-tune based transfer learning method, which is composed of a pretrained language model as encoder and a multilayer perceptron as classifier. It is trained on source
domains and fine-tuned by using support set from the target domain.
Multi-label Prototypical Network (MPN) is a variant of the vanilla prototypical network (Snell et al., 2017). It utilizes the negative Euclidean distance between queries and prototypes and selects a fixed threshold tuned on development domain to determine multiple labels.
Multi-label Matching Network (MMN) intends to train the model as MPN while leveraging Matching Network (Vinyals et al., 2016) to obtain the classification results by calculating the cosine similarity as distance measure.
CTLR (Hou et al., 2021) leverages meta calibrated threshold and anchored label representation to obtain dynamic threshold and separated label representations.

heading: 4.3 Main Results
text: Tables 1, 2, and 4 report the experimental results for 1-shot and 5-shot multi-label intent detection tasks on TourSG and StanfordLU. We use both Electra-small (+E) and Bert-base (+B) as feature encoders. The baseline results are taken from (Hou et al., 2021) and the top 1 results are highlighted in bold.
Average Improvements From the results, we can observe that DCKPN performs much better than other baselines, which demonstrates the superiority of our method. Specifically, in terms of average F1 scores of the TourSG dataset from Tables 1 and 2, DCKPN improves upon the most competitive baseline CTLR by 3.86% and 4.37% on 1/5-shot settings respectively when using Electra and by 1.66% and 1.60% when using Bert. In terms of average F1 scores of StanfordLU from
Table 4, DCKPN improves upon CTLR by 14.07% and 11.18% on 1/5-shot settings for Electra and by 11.22% and 14.39% for Bert. The reason is that DCKPN accurately integrates label name as auxiliary information to learn more discriminative class prototypes and constructs the dual graph to obtain better feature embeddings and propagate class knowledge.
Parameter Efficiency Bert-base (110M params) may be too computationally intensive in real industry deployments, thus we additionally test model performance using Electra-small (14M params) that is nearly 10 times lighter. Interestingly, from Tables 1, 2 and 4, we can obtain that in terms of average F1 scores, our proposed method using Electra-small outperforms the strongest baseline CTLR using Bert-base by 0.44%, 5.70% and 4.61% in the 5-shot of TourSG and 1/5-shot of StanfordLU settings respectively and achieves competitive results in 1-shot of TourSG setting. As a result, larger models achieve better overall performance and our model reduces computational load while preserving competitive performance, which reflects the efficiency of our method.
Label Relationship When comparing to different baselines, our improvement on StanfordLU is much higher than those on TourSG. This large margin may come from the discrepancy in the label set of the two datasets. We find that the labels in StanfordLU are similar and fine-grained, where the label correlation is more informative while labels in TourSG are a little separated and the label correlation is less informative. To verify this, we show the relevance scores among labels using Eq. (1). We choose Sc domain and Sh domain from two
datasets respectively and visualize the relevance scores of the classes in Figure 5 in Appendix A. As can be seen, the classes in StanfordLU are more related than those in TourSG.

heading: 4.4 Ablation Studies
text: We conduct several ablation studies using Electra embeddings to examine the relative contributions of different components of our model in Table 5.
Label-semantic Augmentation For our model without label-semantic augmentation (LA), we directly use the original utterances as input of the encoder and concatenate their multiple labels as the corresponding class-level knowledge. From Table 5, massive F1 score decreases have been seen, particularly in 1-shot setting. On one hand, the model without LA is often confused by co-occurring intents in 1-shot setting, e.g., label 1 and 2 share an identical prototype in Figure 3, which can be separated by LA. On the other hand, utterances are ambiguous as described in Section 3.2, while LA is able to eliminate ambiguity and obtain more separated prototypes.
Impact of Dual Graph We independently remove class-level propagation network (CPN) and dual propagation network (DPN) including CPN
and IPN. As presented in Table 5, DCKPN w/o CPN performs worse than DCKPN, especially for StanfordLU, which demonstrates the effectiveness of CPN for modeling label correlations in multilabel few-shot intent detection tasks. Moreover, when we remove DPN, the F1 scores drop hugely. The reason is that there exists an intrinsic gap between augmented support samples and original query samples while DPN reduces the feature gap and propagates label information.

heading: 4.5 Visualization
text: To better observe how the embeddings change with label-semantic augmentation and dual propagation network, we sample 50 episodes from Schedule domain in StanfordLU and use t-SNE (Van der Maaten and Hinton, 2008) to visualize the support embeddings obtained from CTRL (Hou et al., 2021), DCKPN w/o DPN and DCKPN. Note that each support embedding in CTRL is simply the average of utterance and label name embedding, that in DCKPN w/o DPN is the embedding of labelsemantic augmented utterance without propagation, and that in DCKPN is obtained from our method. From Figure 4, it’s easy to find that the distribution generated by CTRL has a lot of overlaps. The label enhancement in DCKPN w/o DPN can help to separate the embeddings to some extent. The dual propagation network can further generate more compact clusters and useful correlations.

heading: 5 Conclusion
text: In this paper, we propose a novel framework DCKPN for multi-label few-shot intent detection to capture the semantic interactions. We first exploit label-semantic augmentation to extract discriminative representations for user utterances with multiple intents. Then, we introduce an instance-level
and a class-level propagation network to accomplish feature propagation and derive complete class representations. After that, we use the fused representations and an adaptive module to predict the possible multiple intents. Extensive experiments verify the effectiveness of our method and identify the cause of performance improvements our method brings. For future work, we will explore joint learning integrated with slot filling in dialogue systems.

heading: Acknowledgements
text: This work is supported by the Science and Technology Project of State Grid Corporation of China (No. 5700-202341285A-1-1-ZN).
Limitations
In this paper, we explore label-semantic augmentation (LA) for multi-label few-shot intent detection via appending label name after utterances, which is similar to instruction learning or prompt learning. However, we don’t further study the relationship between LA and instruction learning due to space limitations. We believe that instruction learning integrated with labels will inspire further investigation for multi-label few-shot intent detection.

heading: A Label Similarity
text: Figure 5(a) and 5(b) describes the label similarity of schedule (Sc) in StnafordLU and shopping (Sh) in TourSG using Bert-base embedding, which is calculated by Eq. (1). We can observe that (1) the classes in schedule are more similar and the classes in shopping are more distant. (2) the similarity of label description reflects co-occurrence between labels to some extent from the pragmatics view, e.g., in Sc, the similarity of request date and request time is high, and in real world scenarios, human usually asks the two details together, as well as request date and request location, which is crucial in multi-label tasks with low resources.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4.1.2
3 B1. Did you cite the creators of artifacts you used? 4.1.2
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1.1 and Table 3.
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.1.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.1 and 4.3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.1.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we will explore joint learning integrated with slot filling in dialogue systems