Venue: ACL 2023

# Dual Class Knowledge Propagation Network for Multi-label Few-shot Intent Detection

Feng Zhang\({}^{1}\) Wei Chen\({}^{1,2,3}\) Fei Ding\({}^{4,5}\) Tengjiao Wang\({}^{1,2,3}\)

\({}^{1}\)Key Lab of High Confidence Software Technologies (MOE), School of Computer Science, Peking University, Beijing, China

\({}^{2}\)Research Center for Computational Social Science, Peking University

\({}^{3}\)Institute of Computational Social Science, Peking University (Qingdao)

\({}^{4}\)School of Intelligence Science and Technology, Peking University

\({}^{5}\)Institute for Artificial Intelligence, Peking University

{zhangfeng,dingfeij@stu.pku.edu.cn, {pekingchenwei,tjwang}@pku.edu.cn

###### Abstract

Multi-label intent detection aims to assign multiple labels to utterances and attracts increasing attention as a practical task in task-oriented dialogue systems. As dialogue domains change rapidly and new intents emerge fast, the lack of annotated data motivates multi-label few-shot intent detection. However, previous studies are confused by the identical representation of the utterance with multiple labels and overlook the intrinsic intra-class and inter-class interactions. To address these two limitations, we propose a novel dual class knowledge propagation network in this paper. In order to learn well-separated representations for utterances with multiple intents, we first introduce a label-semantic augmentation module incorporating class name information. For better consideration of the inherent intra-class and inter-class relations, an instance-level and a class-level graph neural network are constructed, which not only propagate label information but also propagate feature structure. And we use a simple yet effective method to predict the intent count of each utterance. Extensive experimental results on two multi-label intent datasets have demonstrated that our proposed method outperforms strong baselines by a large margin.

+
Footnote †: _Volume 1: Long Papers_, pages 8605–8618


In this paper, we propose a novel framework for multi-label few-shot intent detection, as illustrated in Figure 2. It obtains well-separated class representations by fully exploiting the label semantic information and propagates class knowledge and feature structure by graph neural network to reduce intra-class variance and model proper inter-class relations. To deal with the confused embeddings for multi-label utterances, we introduce label-semantic augmentation (LA) to obtain well-separated representations without any complicated model architecture. Inspired by the effectiveness of label name information Luo et al. (2021), LA appends each class label name after utterances to construct multiple label-specific utterances, where each label-specific utterance is corresponding to only one label. In this way, LA not only eliminates ambiguity but also generates more labeled training samples.

To better capture more complete and more adaptive interactions among samples, we propose a novel dual class knowledge propagation network, containing an instance-level graph and a class-level graph. It not only propagates label information from labeled support samples to unlabeled queries but also propagates feature structure in their latent space. The instance-level graph smoothing the representations among highly related utterances reduces the intrinsic gap between augmented support samples and queries. The class-level graph provides label correlation and generates completed pseudo class-level representations for queries with the guidance of the instance-level graph. Also, we design a calibration strategy to transfer label dependency into instance representations. Then the two representations are fused to classify queries. Finally, to select accurately multiple intents for an utterance, we employ an adaptive and powerful method to predict the number of intents.

The contributions of this paper are as follows: (1) We propose a novel dual class knowledge propagation network (DCKPN). DCKPN incorporates label semantic connections to guide feature propagation and label information propagation. (2) We introduce a novel label-semantic augmentation method for multi-label samples to obtain more discriminative representations. And we employ a simple inference module to predict the number of intents in utterances adaptively. (3) To verify the effectiveness of our proposed model, we conduct a series of experiments on two datasets. The empirical study shows that our model can achieve state-of-the-art performance in comparison with other strong baselines.

## 2 Related Work

Multi-label ClassificationThe existing multi-label classification (MLC) methods mainly focus on learning enhanced representations Liu et al. (2017) and modeling label dependency Yang et al. (2018); Tsai and Lee (2020). Recently, some works You et al. (2019); Xiao et al. (2019) have been proposed to learn a label-specific representation by employing attention mechanism or utilize the label statistical co-occurrences to explore the semantic interactions Ma et al. (2021), which shows the importance of exploring semantic interactions for MLC. However, these methods require large amounts of labeled samples for training, thus it's

Figure 2: The architecture of the proposed framework DCKPN.

 

hard to be applied to fast-emerging new domains in few-shot scenarios.

Few-shot LearningFew-shot learning aims to recognize novel classes from a handful of annotated samples by leveraging the previous knowledge from source domains Fei-Fei et al. (2006). Recently, meta-learning has become a successful paradigm for solving the few-shot problem, which can be divided into three categories: model-based methods Munkhdalai and Yu (2017), optimization-based methods Finn et al. (2017); Yoon et al. (2018) and metric-based methods Vinyals et al. (2016); Snell et al. (2017); Sung et al. (2018). However, these methods focus on single-label classification and only a few works investigate multi-label few-shot classification. Previous studies focus on image domain Alfassy et al. (2019), audio domain Cheng et al. (2019) and sentiment analysis Hu et al. (2021). CTLR Hou et al. (2021) is the first work to address multi-label few-shot intent detection tasks which proposes a meta-calibrated threshold mechanism and learns anchored label representation. However, it relies on a specific metric to estimate the relevance score, losing the inherent relations among samples and classes.

Propagation in Graph ModelGraph neural networks are effective in modeling the relationship among different nodes and have shown powerful performance in many real applications, e.g., nodes classification Kim et al. (2019) and link prediction Zhang and Chen (2018). In few-shot scenarios, several approaches have been explored. Specifically, DPGN Yang et al. (2020) constructs both the distribution-level and instance-level graph. Liu et al. (2019) propose a transductive propagation network (TPN) by utilizing label propagation Zhou et al. (2003), a classical and effective method to transfer knowledge from neighbors of each node. Rodriguez et al. (2020) yield a smoother embedding space by employing embedding propagation. Inspired by this, our method aims to propagate information both in intra-class and inter-class, called dual class knowledge propagation network (DCKPN).

## 3 Proposed Method

### Problem Definition

Few-shot learning aims to train a model that can perform well in cases where only a few samples are given. We follow the episodic paradigm, an effective solution for few-shot learning, which is commonly employed in various studies Snell et al. (2017); Yang et al. (2021). In general, the model is first trained on a series of tasks from source domains with numerous samples ( \({\cal D}_{train}\)) episode by episode until convergence, then directly adopted to another set of unseen novel tasks (\({\cal D}_{test}\)), where a few labeled samples are available. There is no overlap between the training classes and testing classes. Specifically, each episode (task) has a support set \({\cal S}=\{(\mathbf{x}_{i},\mathbf{y}_{i})\}^{N\times K}_{i=1}\), which usually includes \(N\) classes with \(K\) labeled samples for each class, i.e., the \(N\)-way \(K\)-shot setting, and a query set \({\cal Q}=\{(\mathbf{x}_{j},\mathbf{y}_{j})\}^{q}_{j=1}\), where \(q\) is the number of query samples.

Multi-label few-shot intent detection enables each utterance to be assigned with a set of intent labels simultaneously. To be more specific, for each example \((\mathbf{x},\mathbf{y})\), \(\mathbf{x}\) is the utterance and the label vector is denoted as \(\mathbf{y}=\{y^{1},y^{2},...,y^{|{\cal C}|}\}\in\mathbb{R}^{|{\cal C}|}\), where \({\cal C}\) is the set of intent labels in the episode and \(y^{i}\in\{0,1\}\). In this paper, we focus on multi-label few-shot intent detection, i.e., selecting multiple associated intent labels for each utterance in few-shot scenarios.

### Label-semantic Augmentation

For multi-label few-shot intent detection tasks, one of the key points is to learn more discriminative class representations which could capture accurate class-relevant information while eliminating ambiguity via a few examples. To achieve this goal, we propose to use label name to augment the support set. Different from Luo et al. (2021), which have explored integrating label information into sentences, we propose to append each class label name after each sentence with multiple possible labels. As Figure 3 shows, for each utterance in support set,

Figure 3: Illustration of a simple 3-way 1-shot task.

 

we append its labels one by one after it, i.e., "[CLS] utterance [SEP] label name [SEP]", thus generating different label-specific utterances corresponding to single different class labels. For each sample in query set, due to unknown labels, we take the common format "[CLS] utterance [SEP]" as input text.

There are two benefits of using the label name to augment utterances. First, we make use of different appended label name to distinguish the same utterance with multiple labels, thus obtaining more discriminative representations for each class. As illustrated in Figure 3, we can obtain different prototypes for class 1 and class 2, eliminating the problem that the multi-label utterances share the same representation essentially without any complex architecture or parameters. Second, this enables us to utilize the label name information as well as increase the number of support samples to eliminate the ambiguity caused by the scarcity of utterances. In Section 4.4, we analyze the experimental results to verify the improvements of augmentation.

### Class Knowledge Propagation Network

Although one can directly calculate class prototypes via label-semantic augmented support set to classify the queries, doing so will result in the loss of intra-class and inter-class information, which is critical for multi-label classification tasks Zhang et al. (2018). To mitigate these issues, we propose a dual class knowledge propagation network (DCKPN) to propagate class knowledge from support samples to queries. As illustrated in Figure 2, we first utilize the support and query samples to build the instance-level graph \({\cal G}^{I}=(V^{I},W^{I})\) and utilize label name information to build the class-level graph \({\cal G}^{C}=(V^{C},W^{C})\). Notably, in the following section, the support samples are label-semantic augmented samples described in Section 3.2, which means that each augmented support sample in instance-level graph has its corresponding single label in the class-level graph. Then, we modify the instance-level embeddings based on pairwise node relations. Guided by the class-level graph, we also calibrate relations of support-support nodes to integrate label semantic relevance explicitly. Next, we update class-level representations based on instance relations to generate pseudo query class-level representations. Finally, the two representations are merged to guide the inference of the queries.

#### 3.3.1 Graph Construction

For each sample \((\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{a}_{i})\in\mathcal{S}\) in augmented support set, we employ a pre-trained language model as the backbone to extract its instance-level feature \(\mathbf{v}_{i}^{I}=f_{\phi}(\mathbf{x}_{i})\in\mathbb{R}^{d}\) and its class-level feature \(\mathbf{v}_{i}^{C}=f_{\phi}(\mathbf{a}_{i})\in\mathbb{R}^{d}\), where \(\phi\) is the parameters of the backbone model and \(\mathbf{a}_{i}\) is the label name of utterance \(\mathbf{x}_{i}\). Due to unknown query labels, for each query sample \(\mathbf{x}_{j}\in\mathcal{Q}\) in query set, we regard \(\mathbf{v}_{j}^{I}=f_{\phi}(\mathbf{x}_{j})\in\mathbb{R}^{d}\) as its instance-level feature and \(\mathbf{0}\in\mathbb{R}^{d}\) as its class-level representation. Then, the adjacency matrix is defined as follows:

\[W_{ij}=\exp(-\frac{d(\mathbf{v}_{i},\mathbf{v}_{j})}{\sigma^{2}}),W_{ii}=0, \tag{1}\]

where \(\sigma^{2}=Var(d(\mathbf{v}_{i},\mathbf{v}_{j}))\) and \(d(\mathbf{v}_{i},\mathbf{v}_{j})=\|\mathbf{v}_{i}-\mathbf{v}_{j}\|_{2}^{2}\). Thus, we obtain the instance-level graph \({\cal G}^{I}=(V^{I},W^{I})\), where \(V^{I}\in\mathbb{R}^{(|\mathcal{S}|+|\mathcal{Q}|)\times d}\), and \(W^{I}\in\mathbb{R}^{(|\mathcal{S}|+|\mathcal{Q}|)\times(|\mathcal{S}|+| \mathcal{Q}|)}\). While for class-level graph \({\cal G}^{C}=(V^{C},W^{C})\), \(V^{C}\in\mathbb{R}^{(|\mathcal{S}|+|\mathcal{Q}|)\times d}\) consists of the feature embeddings of the label names in the support set \(V^{C}_{s}\in\mathbb{R}^{|\mathcal{S}|\times d}\) as well as a matrix of zeros \(\mathbf{0}\in\mathbb{R}^{|\mathcal{Q}|\times d}\) for queries, and \(W^{C}\in\mathbb{R}^{(|\mathcal{S}|+|\mathcal{Q}|)\times(|\mathcal{S}|+| \mathcal{Q}|)}\) is as follows:

\[W^{C}=\left(\begin{array}{cc}W^{C}_{ss}&\mathbf{0}\\ \mathbf{0}&\mathbf{0}\end{array}\right). \tag{2}\]

#### 3.3.2 Instance-level Propagation

We now perform message passing on the instance-level graph to conduct higher-order interactions between augmented support embeddings and query embeddings. Specifically, we first normalize the weight matrix \(W^{I}\) as,

\[L^{I}=D^{-1/2}W^{I}D^{-1/2},D_{ii}=\sum_{j}W^{I}_{ij}. \tag{3}\]

Then, using the propagation solution described in Zhou et al. (2003), we calculate the propagation matrix and rectify the instance-level embeddings as,

\[\widetilde{V}^{I}=(I-\alpha L^{I})^{-1}V^{I}, \tag{4}\]

where \(I\) is the identity matrix and \(\alpha\in\mathbb{R}\) is a smoothing factor.

To further capture the inter-class relationships and reduce the intra-class variance, we utilize the guidance of relations of support-support pairs in class-level graph to calibrate the support-support pairs in instance-level graph, thus further encouraging the relation learning of instances with 

semantically related classes. For example, _request_temperature_ share more semantic information with _request_low_temperature_ than with _appreciate_. We implement the semantic loss by calibrating the pair relation in support set:

\[{\mathcal{L}}_{sem}=\frac{1}{|W^{C}_{ss}|}\sum_{i=1}^{|\mathcal{S}|}\sum_{j=1}^ {|\mathcal{S}|}|w^{C}_{ij}-w^{I}_{ij}|. \tag{5}\]

#### 3.3.3 Class-level Propagation

As illustrated in Figure 2, the instance information flows back into the class-level graph to generate the completed semantic knowledge representations for both support set and query set. Concretly, we calculate the Laplacian of the updated instance embeddings \(\widetilde{L}^{I}=\widetilde{D}^{-1/2}\widetilde{W}^{I}\widetilde{D}^{-1/2}\), where \(\widetilde{D}_{ii}=\sum_{j}\widetilde{W}^{I}_{ij}\), \(\widetilde{W}^{I}_{ij}=\exp(-d(\widetilde{v}^{I}_{i},\widetilde{v}^{I}_{j})/ \widetilde{\sigma}^{2})\), \(\widetilde{\sigma}^{2}=Var(d(\widetilde{v}^{I}_{i},\widetilde{v}^{I}_{j}))\), and \(\widetilde{v}^{I}_{i}\) is obtained from Eq. (4). Next, the complete class-level embeddings are obtained as:

\[\widetilde{V}^{C}=(I-\alpha\widetilde{L}^{I})^{-1}V^{C}, \tag{6}\]

where \(V^{C}\) is the original label name embedding matrix in class-level graph. Through the guidance of connectivity strength of instance pairs, we finally generate pseudo class representations for queries without labels. Thus, by explicitly establishing the bidirectional interaction between instance-level graph and class-level graph, we conduct feature propagation and derive completed representations to model intra-class and inter-class relationships.

### Objective

**Prototypical Classifier.** Due to the two-level features being complementary, we concatenate the embeddings to obtain better feature representations. For each sample \(\mathbf{x}_{i}\), the embedding is defined as follows:

\[\widetilde{\mathbf{v}}_{i}=\widetilde{\mathbf{v}}^{I}_{i}||\widetilde{\mathbf{v}}^{C}_{i}, \tag{7}\]

where \(||\) denotes the concatenation and \(\widetilde{\mathbf{v}}^{I}_{i},\widetilde{\mathbf{v}}^{C}_{i}\) are sample \(\mathbf{x}_{i}\)'s instance-level and class-level embeddings respectively. The prototype \(\widetilde{\mathbf{p}}_{c}\) of intent class \(c\in\mathcal{C}\) can be calculated as \(\widetilde{\mathbf{p}}_{c}=\frac{1}{|\mathcal{S}_{c}|}\sum_{\mathbf{x}_{i}\in\mathcal{ S}_{c}}\widetilde{\mathbf{v}}_{i}\), where \(\mathcal{S}_{c}\) denotes the set of support samples with intent class \(c\) and \(\mathcal{C}\) is the class set. Given a query data \((\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{Q}\), we compute the conditional probability \(p(y=c|\mathbf{x}_{i},\mathcal{S})\) based on negative squared Euclidean distance:

\[p(y=c|\mathbf{x}_{i},\mathcal{S})=\frac{\exp(-d(\widetilde{\mathbf{v}}_{i},\widetilde {\mathbf{p}}_{c}))}{\sum_{c^{\prime}\in\mathcal{C}}\exp(-d(\widetilde{\mathbf{v}}_{i}, \widetilde{\mathbf{p}}_{c^{\prime}}))}. \tag{8}\]

Finally, we perform cross-entropy loss on all query samples. Note that in the multi-label setting, as an utterance may have multiple labels, we need to consider \(|\mathcal{C}|\) labels for each query sentence. The classification loss is calculated as:

\[{\mathcal{L}}_{ce}=-\frac{1}{|\mathcal{Q}|}\sum_{\mathbf{x}_{i}\in\mathcal{Q}}\sum _{j=1}^{|\mathcal{C}|}y^{j}_{i}{\rm log}p(y=j|\mathbf{x}_{i},\mathcal{S}), \tag{9}\]

where \(\mathbf{y}_{i}=\{y^{1}_{i},...,y^{|\mathcal{C}|}_{i}\}\) is the label and \(y^{j}_{i}\in\{0,1\}\).

**Label Number Prediction.** For multi-label few-shot intent detection, one of the challenges is to determine the number of intents in the utterance. We leverage a simple and effective method to decide the number of labels. Given an utterance \(\mathbf{x}\), we can obtain its fused feature \(\widetilde{\mathbf{v}}\), and then we use a multi-layer perceptron to predict the number of intents in \(\mathbf{x}\):

\[\mathbf{n}={\rm softmax}({\rm MLP}(\widetilde{\mathbf{v}})), \tag{10}\]

where \(\mathbf{n}\in\mathbb{R}^{N}\) is the indicator for the number of intents and \(N\) is the maximum count of possible intents. In meta-training stage, we calculate the cross entropy loss of intent count as follows:

\[{\mathcal{L}}_{num}=-\frac{1}{|\mathcal{Q}|}\sum_{\mathbf{x}_{i}\in\mathcal{Q}} \sum_{j=1}^{N}t^{j}_{i}\log(n^{j}_{i}), \tag{11}\]

where \(\mathbf{t}_{i}=\{t^{1}_{i},...,t^{N}_{i}\}\) is the ground-truth intent count vector of \(\mathbf{x}_{i}\) and \(t^{j}_{i}\in\{0,1\}\).

By combining Eqs. (5), (9) and (11), the overall loss of our proposed framework is:

\[{\mathcal{L}}={\mathcal{L}}_{ce}+\beta{\mathcal{L}}_{sem}+\gamma{\mathcal{L}}_ {num}, \tag{12}\]

where \(\beta\) and \(\gamma\) are adjustable weight parameters.

## 4 Experiments

### Datasets and Setups

#### 4.1.1 Datasets

We follow Hou et al. (2021) to conduct experiments on two multi-label intent detection datasets: TourSG and StanfordLU. TourSG is a dataset of touristic information in Singapore, which includes 25751 samples annotated with multiple labels from 6 different domains: It (Itinerary), Ac (Accommodation), At (Attraction), Fo (Food), Tr (Transportation), and Sh (Shopping). StanfordLU is a Stanford dialogue dataset Eric et al. (2017) and contains 

8038 utterances re-annotated by Hou et al. (2021) from 3 domains: Sc Schedule, Na Navigate and We Weather. Different from the standard N-way K-shot setting, an utterance may have multiple possible intents in multi-label few-shot intent detection and we refer you to Hou et al. (2021) for more episode construction details. Table 3 shows the detailed dataset statistics.

#### 4.1.2 Experiment Setups

Evaluation MetricsWe follow Hou et al. (2021) to use micro F1 scores to evaluate the performance. All reported results are the average results of 5 different runs.

Parameter SettingsWe conduct experiments on 1-shot and 5-shot settings. For each episode from different domains, the support set has \(N_{s}\) classes (\(N_{s}\)-way), where \(N_{s}\) is the label numbers in domain \(s\). And the query set size is 16 for TourSG and 32 for StanfordLU. In terms of feature extraction, we use the pre-trained Electra-small Clark et al. (2020) and Bert-base-uncased Devlin et al. (2019) model. For the loss function, we set \(\beta\) as \(0.1\) and \(\gamma\) as \(0.01\). The smooth factor \(\alpha\) is \(0.1\). We use AdamW Loshchilov and Hutter (2019) optimizer with the initial learning rate \(2e-4\) for Electra, \(1e-4\) for Bert, and the dropout rate is \(0.1\). All the hyperparameters are selected based on the performance of the development domain.

### Baselines

We compare our proposed method with the following strong baselines.

**TransferM** is a standard fine-tune based transfer learning method, which is composed of a pretrained language model as encoder and a multilayer perceptron as classifier. It is trained on source

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline  & **Model** & **It** & **Ac** & **At** & **Fo** & **Tr** & **Sh** & **Ave.** \\ \hline \multirow{7}{*}{+E} & TransferM & 14.34\({}_{\pm 0.82}\) & 14.75\({}_{\pm 0.91}\) & 16.13\({}_{\pm 1.35}\) & 11.79\({}_{\pm 1.54}\) & 13.64\({}_{\pm 0.33}\) & 14.32\({}_{\pm 1.17}\) & 14.16\({}_{\pm 1.02}\) \\  & MMN & 9.98\({}_{\pm 1.80}\) & 7.81\({}_{\pm 0.70}\) & 8.37\({}_{\pm 0.86}\) & 7.81\({}_{\pm 0.35}\) & 10.65\({}_{\pm 1.62}\) & 11.56\({}_{\pm 0.79}\) & 9.36\({}_{\pm 1.02}\) \\  & MPN & 12.24\({}_{\pm 0.92}\) & 10.38\({}_{\pm 1.21}\) & 10.00\({}_{\pm 0.54}\) & 10.47\({}_{\pm 0.42}\) & 13.61\({}_{\pm 0.92}\) & 11.41\({}_{\pm 0.31}\) & 11.35\({}_{\mp 0.72}\) \\  & CTLR & 39.98\({}_{\pm 0.56}\) & 51.55\({}_{\pm 1.53}\) & 55.16\({}_{\pm 2.43}\) & 52.16\({}_{\pm 0.98}\) & 55.36\({}_{\pm 0.96}\) & 52.20\({}_{\pm 1.03}\) & 51.07\({}_{\pm 1.24}\) \\  & DCKPN & **44.21\({}_{\pm 0.64}\)** & **55.91\({}_{\pm 0.35}\)** & **59.74\({}_{\pm 0.89}\)** & **56.55\({}_{\pm 0.83}\)** & **57.48\({}_{\pm 1.04}\)** & **55.71\({}_{\pm 0.82}\)** & **54.93\({}_{\pm 0.76}\)** \\ \hline \multirow{7}{*}{+B} & TransferM & 16.78\({}_{\pm 0.05}\) & 18.62\({}_{\pm 0.59}\) & 14.92\({}_{\pm 2.22}\) & 16.40\({}_{\pm 2.58}\) & 15.68\({}_{\pm 0.32}\) & 14.50\({}_{\pm 2.18}\) & 16.15\({}_{\pm 1.32}\) \\  & MMN & 10.89\({}_{\pm 3.35}\) & 7.72\({}_{\pm 1.44}\) & 8.92\({}_{\pm 1.45}\) & 9.32\({}_{\pm 1.40}\) & 13.75\({}_{\pm 0.70}\) & 10.87\({}_{\pm 4.31}\) & 10.24\({}_{\pm 2.11}\) \\ \cline{1-1}  & MPN & 13.77\({}_{\pm 0.38}\) & 12.38\({}_{\pm 0.32}\) & 13.46\({}_{\pm 0.14}\) & 10.23\({}_{\pm 0.30}\) & 16.19\({}_{\pm 0.19}\) & 15.79\({}_{\pm 0.38}\) & 13.64\({}_{\pm 0.28}\) \\ \cline{1-1}  & CTLR & 44.58\({}_{\pm 0.71}\) & 57.11\({}_{\pm 1.22}\) & 60.34\({}_{\pm 0.92}\) & 56.49\({}_{\pm 0.67}\) & 60.18\({}_{\pm 0.85}\) & 55.60\({}_{\pm 0.66}\) & 55.72\({}_{\pm 1.03}\) \\ \cline{1-1}  & DCKPN & **48.19\({}_{\pm 1.01}\)** & **58.32\({}_{\pm 0.68}\)** & **60.93\({}_{\pm 0.42}\)** & **58.22\({}_{\pm 1.36}\)** & **61.05\({}_{\pm 0.62}\)** & **57.62\({}_{\pm 0.64}\)** & **57.38\({}_{\pm 0.79}\)** \\ \hline \hline \end{tabular}
\end{table}
Table 1: The 1-shot average F1 scores of multi-label intent detection on TourSG dataset.

\begin{table}
\begin{tabular}{c l|c c c c c c} \hline \hline  & **Model** & **It** & **Ac** & **At** & **Fo** & **Tr** & **Sh** & **Ave.** \\ \hline \multirow{7}{*}{+E} & TransferM & 14.72\({}_{\pm 0.53}\) & 19.20\({}_{\pm 1.59}\) & 16.18\({}_{\pm 1.03}\) & 18.86\({}_{\pm 1.04}\) & 17.17\({}_{\pm 1.19}\) & 17.51\({}_{\pm 1.63}\) & 17.27\({}_{\pm 1.17}\) \\  & MMN & 14.11\({}_{\pm 0.83}\) & 10.58\({}_{\pm 1.35}\) & 17.80\({}_{\pm 1.12}\) & 12.74\({}_{\pm 0.87}\) & 18.01\({}_{\pm 0.90}\) & 16.76\({}_{\pm 0.92}\) & 15.00\({}_{\pm 1.00}\) \\  & OPN & 15.18\({}_{\pm 0.63}\) & 15.56\({}_{\pm 0.54}\) & 17.60\({}_{\pm 1.15}\) & 15.01\({}_{\pm 0.19}\) & 17.99\({}_{\pm 0.36}\) & 17.17\({}_{\pm 1.09}\) & 16.42\({}_{\pm 0.66}\) \\  & CTRL & 44.21\({}_{\pm 0.71}\) & 51.37\({}_{\pm 1.22}\) & 55.76\({}_{\pm 0.92}\) & 54.50\({}_{\pm 0.58}\) & 55.37\({}_{\pm 0.95}\) & 54.55\({}_{\pm 0.86}\) & 52.63\({}_{\pm 0.87}\) \\  & DCKPN & **47.76\({}_{\pm 0.73}\)** & **55.83\({}_{\pm 1.08}\)** & **59.48\({}_{\pm 0.32}\)** & **60.06\({}_{\pm 1.23}\)** & **60.23\({}_{\pm 0.63}\)** & **58.66\({}_{\pm 0.52}\)** & **57.00\({}_{\pm 0.75}\)** \\ \hline \multirow{7}{*}{+B} & TransferM & 17.98\({}_{\pm 1.80}\) & 16.51\({}_{\pm 1.95}\) & 

domains and fine-tuned by using support set from the target domain.

**Multi-label Prototypical Network (MPN)** is a variant of the vanilla prototypical network Snell et al. (2017). It utilizes the negative Euclidean distance between queries and prototypes and selects a fixed threshold tuned on development domain to determine multiple labels.

**Multi-label Matching Network (MMN)** intends to train the model as MPN while leveraging Matching Network Vinyals et al. (2016) to obtain the classification results by calculating the cosine similarity as distance measure.

**CTLR**Hou et al. (2021) leverages meta calibrated threshold and anchored label representation to obtain dynamic threshold and separated label representations.

### Main Results

Tables 1, 2, and 4 report the experimental results for 1-shot and 5-shot multi-label intent detection tasks on TourSG and StanfordLU. We use both Electra-small (+E) and Bert-base (+B) as feature encoders. The baseline results are taken from Hou et al. (2021) and the top 1 results are highlighted in bold.

Average ImprovementsFrom the results, we can observe that DCKPN performs much better than other baselines, which demonstrates the superiority of our method. Specifically, in terms of average F1 scores of the TourSG dataset from Tables 1 and 2, DCKPN improves upon the most competitive baseline CTLR by 3.86% and 4.37% on 1/5-shot settings respectively when using Electrora and by 1.66% and 1.60% when using Bert. In terms of average F1 scores of StanfordLU from Table 4, DCKPN improves upon CTRR by 14.07% and 11.18% on 1/5-shot settings for Electra and by 11.22% and 14.39% for Bert. The reason is that DCKPN accurately integrates label name as auxiliary information to learn more discriminative class prototypes and constructs the dual graph to obtain better feature embeddings and propagate class knowledge.

Parameter EfficiencyBert-base (110M params) may be too computationally intensive in real industry deployments, thus we additionally test model performance using Electra-small (14M params) that is nearly 10 times lighter. Interestingly, from Tables 1, 2 and 4, we can obtain that in terms of average F1 scores, our proposed method using Electra-small outperforms the strongest baseline CTLR using Bert-base by 0.44%, 5.70% and 4.61% in the 5-shot of TourSG and 1/5-shot of StanfordLU settings respectively and achieves competitive results in 1-shot of TourSG setting. As a result, larger models achieve better overall performance and our model reduces computational load while preserving competitive performance, which reflects the efficiency of our method.

Label RelationshipWhen comparing to different baselines, our improvement on StanfordLU is much higher than those on TourSG. This large margin may come from the discrepancy in the label set of the two datasets. We find that the labels in StanfordLU are similar and fine-grained, where the label correlation is more informative while labels in TourSG are a little separated and the label correlation is less informative. To verify this, we show the relevance scores among labels using Eq. (1). We choose \({\bf Sc}\) domain and \({\bf Sh}\) domain from two

\begin{table}
\begin{tabular}{c l c c c c c c c c} \hline \hline  & \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**1-shot**} & \multicolumn{4}{c}{**5-shot**} \\ \cline{3-10}  & & \multicolumn{1}{c}{**Sc**} & \multicolumn{1}{c}{**Na**} & \multicolumn{1}{c}{**We**} & \multicolumn{1}{c}{**Ave.**} & \multicolumn{1}{c}{**Sc**} & \multicolumn{1}{c}{**Na**} & \multicolumn{1}{c}{**We**} & \multicolumn{1}{c}{**Ave.**} \\ \hline \multirow{4}{*}{+E} & TransferM & 16.96\({}_{\pm 0.73}\) & 22.99\({}_{\pm 0.51}\) & 21.01\({}_{\pm 0.57}\) & 20.32\({}_{\pm 0.60}\) & 16.99\({}_{\pm 0.94}\) & 23.79\({}_{\pm 0.27}\) & 23.92\({}_{\pm 1.78}\) & 21.57\({}_{\pm 1.00}\) \\  & MMN & 31.22\({}_{\pm 4.96}\) & 24.41\({}_{\pm 3.28}\) & 48.01\({}_{\pm 1.10}\) & 34.55\({}_{\pm 3.11}\) & 41.91\({}_{\pm 4.49}\) & 37.94\({}_{\pm 1.38}\) & 60.67\({}_{\pm 1.23}\) & 46.84\({}_{\pm 2.37}\) \\  & MPN & 32.44\({}_{\pm 3.75}\) & 17.83\({}_{\pm 3.83}\) & 38.86\({}_{\pm 4.18}\) & 29.71\({}_{\pm 3.92}\) & 35.92\({}_{\pm 2.79}\) & 27.65\({}_{\pm 5.45}\) & 58.07\({}_{\pm 1.88}\) & 40.55\({}_{\pm 3.08}\) \\  & CTLR & 40.61\({}_{\pm 1.05}\) & 40.76\({}_{\pm 0.89}\) & 46.16\({}_{\pm 0.96}\) & 42.51\({}_{\pm 0.97}\) & 51.83\({}_{\pm 1.31}\) & 46.44\({}_{\pm 1.60}\) & 54.17\({}_{\pm 1.70}\) & 50.82\({}_{\pm 1.54}\) \\  & DCKPN & **52.08\({}_{\pm 1.36}\)** & **51.37\({}_{\pm 0.82}\)** & **66.29\({}_{\pm 0.72}\)** & **56.58\({}_{\pm 0.97}\)** & **55.04\({}_{\pm 1.21}\)** & **55.64\({}_{\pm 0.91}\)** & **75.32\({}_{\pm 1.24}\)** & **62.00\({}_{\pm 1.12}\)** \\ \hline \multirow{4}{*}{+B} & TransferM & 18.00\({}_{\pm 0.62}\) & 24.65\({}_{\pm 0.79}\) & 22.26\({}_{\pm 0.64}\) & 21.64\({}_{\pm 0.68}\) & 16.62\({}_{\pm 0.18}\) & 23.69\({}_{\pm 0.46}\) & 26.64\({}_{\pm 0.24}\) & 22.31\({}_{\pm 0.89}\) \\  & MMN & 39.18\({}_{\pm 0.52}\) & 35.35\({}_{\pm 1.72}\) & 45.87\({}_{\pm 2.81}\) & 40.13\({}_{\pm 1.68}\) & 43.65\({}_{\pm 6.24}\) & 51.94\({}_{\pm 1.03}\) & 46.65\({}_{\pm 0.48}\) & 47.41\({}_{\pm 2.58}\) \\  & MPN & 39.34\({}_{\pm 1.38}\) & 36.09\({}_{\pm 0.77}\) & 45.86\({}_{\pm 2.50}\) & 40.43\({}_{\pm 1.55}\) & 41.45\({}_{\pm 2.83}\) & 50.51\({}_{\pm 2.94}\) & 54.96\({}_{\pm 9.76}\) & 48.97\({}_{\pm 5.18}\) \\  & CTLR & 42.55\({}_{\pm 0.40}\) & 56.95\({}_{\pm 0.77}\) & 53.14\({}_{\pm 1.89}\) & 50.88\({}_{\pm 1.02}\) & 52.17\({}_{\pm 1.29}\) & 60.36\({}_{\pm 1.55}\) & 59.63\({}_{\pm 2.23}\) & 57.39\({}_{\pm 1.69}\) \\  & DCKPN & **53.81\({}_{\pm 0.72}\)** & **58.48\({}_{\pm 0.31}\)** & **74.02\({}_{\pm 0.74}\)** & **62.10\({}_{\pm 0.59}\)** & **57.81\({}_{\pm 0.62}\)** & **63.71\({}_{\pm 0.35}\)** & **93.83\({}_{\pm 0.36}\)** & **71.78\({}_{\pm 0.44}\)** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The 1-shot and 5-shot average F1 scores of multi-label intent detection on StanfordLU dataset.

 

datasets respectively and visualize the relevance scores of the classes in Figure 5 in Appendix A. As can be seen, the classes in StanfordLU are more related than those in TourSG.

### Ablation Studies

We conduct several ablation studies using Electra embeddings to examine the relative contributions of different components of our model in Table 5.

Label-semantic AugmentationFor our model without label-semantic augmentation (LA), we directly use the original utterances as input of the encoder and concatenate their multiple labels as the corresponding class-level knowledge. From Table 5, massive F1 score decreases have been seen, particularly in 1-shot setting. On one hand, the model without LA is often confused by co-occurring intents in 1-shot setting, e.g., label 1 and 2 share an identical prototype in Figure 3, which can be separated by LA. On the other hand, utterances are ambiguous as described in Section 3.2, while LA is able to eliminate ambiguity and obtain more separated prototypes.

Impact of Dual GraphWe independently remove class-level propagation network (CPN) and dual propagation network (DPN) including CPN and IPN. As presented in Table 5, DCKPN w/o CPN performs worse than DCKPN, especially for StanfordLU, which demonstrates the effectiveness of CPN for modeling label correlations in multi-label few-shot intent detection tasks. Moreover, when we remove DPN, the F1 scores drop hugely. The reason is that there exists an intrinsic gap between augmented support samples and original query samples while DPN reduces the feature gap and propagates label information.

### Visualization

To better observe how the embeddings change with label-semantic augmentation and dual propagation network, we sample 50 episodes from Schedule domain in StanfordLU and use t-SNE Van der Maaten and Hinton (2008) to visualize the support embeddings obtained from CTRL Hou et al. (2021), DCKPN w/o DPN and DCKPN. Note that each support embedding in CTRL is simply the average of utterance and label name embedding, that in DCKPN w/o DPN is the embedding of label-semantic augmented utterance without propagation, and that in DCKPN is obtained from our method. From Figure 4, it's easy to find that the distribution generated by CTRL has a lot of overlaps. The label enhancement in DCKPN w/o DPN can help to separate the embeddings to some extent. The dual propagation network can further generate more compact clusters and useful correlations.

## 5 Conclusion

In this paper, we propose a novel framework DCKPN for multi-label few-shot intent detection to capture the semantic interactions. We first exploit label-semantic augmentation to extract discriminative representations for user utterances with multiple intents. Then, we introduce an instance-level

Figure 4: Visualization of rectified support embeddings of Schedule in StanfordLU obtained from CTRL, DCKPN w/o DPN, and DCKPN respectively. Data points with the same color contain the same intent.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Setting**} & \multicolumn{2}{c}{**TourSG**} & \multicolumn{2}{c}{**StanfordLU**} \\ \cline{2-6}  & 1-shot & 5-shot & 1-shot & 5-shot \\ \hline DCKPN & 54.93 & 57.00 & 56.58 & 62.00 \\ - LA & 40.82 & 51.96 & 47.11 & 60.74 \\ - OPN & 53.61 & 54.86 & 51.77 & 59.62 \\ - DPN & 51.59 & 53.64 & 48.47 & 57.12 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablations on TourSG and StanfordLU. The average F1 scores of all domains are reported.

 

and a class-level propagation network to accomplish feature propagation and derive complete class representations. After that, we use the fused representations and an adaptive module to predict the possible multiple intents. Extensive experiments verify the effectiveness of our method and identify the cause of performance improvements our method brings. For future work, we will explore joint learning integrated with slot filling in dialogue systems.

## Acknowledgements

This work is supported by the Science and Technology Project of State Grid Corporation of China (No. 5700-202341285A-1-ZN).

## Limitations

In this paper, we explore label-semantic augmentation (LA) for multi-label few-shot intent detection via appending label name after utterances, which is similar to instruction learning or prompt learning. However, we don't further study the relationship between LA and instruction learning due to space limitations. We believe that instruction learning integrated with labels will inspire further investigation for multi-label few-shot intent detection.

## References

* A. Alfassy, L. Karlinsky, A. Aides, J. Shtok, S. Harary, R. Ferris, R. Giryes, and A. M. Bronstein (2019)Laso: Label-set operations networks for multi-label few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 6548-6557. Cited by: SS1.
* K. Cheng, S. Chou, and Y. Yang (2019)Multi-label few-shot learning for sound event recognition. In 21st IEEE International Workshop on Multimedia Signal Processing, MMSP, Vol. , pp. 1-5. External Links: Document Cited by: SS1.
* K. Clark, M. Luong, Q. V. Le, and C. D. Manning (2020)ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR, Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186. Cited by: SS1.
* M. Eric, L. Krishnan, F. Charette, and C. D. Manning (2017)Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 37-49. Cited by: SS1.
* L. Fei-Fei, R. Fergus, and P. Perona (2006)One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell.2594-611. Cited by: SS1.
* C. Finn, P. Abbeel, and S. Levine (2017)Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, pp. 1126-1135. Cited by: SS1.
* R. Gangadharaiah and B. Narayanany (2019)Joint multiple intent detection and slot labeling for goal-oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pp. 564-569. Cited by: SS1.
* Y. Hou, Y. Lai, Y. Wu, W. Che, and T. Liu (2021)Few-shot learning for multi-label intent detection. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI, pp. 13036-13044. Cited by: SS1.
* M. Hu, S. Zhao, H. Guo, C. Xue, H. Gao, T. Gao, R. Cheng, and Z. Su (2021)Multi-label few-shot learning for aspect category detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP, pp. 6330-6340. Cited by: SS1.
* J. Kim, T. Kim, S. Kim, and C. D. Yoo (2019)Edge-labeling graph neural network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 11-20. Cited by: SS1.
* J. Liu, W. Chang, Y. Wu, and Y. Yang (2017)Deep learning for extreme multi-label text classification. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 115-124. Cited by: SS1.
* Y. Liu, J. Lee, M. Park, S. Kim, E. Yang, S. Ju Hwang, and Y. Yang (2019)Learning to propagate labels: transductive propagation network for few-shot learning. In 7th International Conference on Learning Representations, ICLR, Cited by: SS1.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR, Cited by: SS1.
* S. Louvan and B. Magnini (2020)Recent neural methods on slot filling and intent classification for task-oriented dialogue systems: a survey. In 

Proceedings of the 28th International Conference on Computational Linguistics, COLING_, pages 480-496.
* Luo et al. (2021) Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, and Wei Zhang. 2021. Don't miss the labels: Label-semantic augmented meta-learner for few-shot text classification. In _Findings of the Association for Computational Linguistics: ACL/IJCNLP_, pages 2773-2782.
* Ma et al. (2021) Qianwen Ma, Chunyuan Yuan, Wei Zhou, and Songlin Hu. 2021. Label-specific dual graph neural network for multi-label text classification. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLI/JCNLP_, pages 3855-3864.
* Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In _Proceedings of the 34th International Conference on Machine Learning, ICML_, pages 2554-2563.
* Qin et al. (2021) Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanxiang Che, and Ting Liu. 2021. GL-GIN: fast and accurate non-autoregressive model for joint multiple intent detection and slot filling. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACLI/IJCNLP_, pages 178-188.
* Qin et al. (2020a) Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020a. TD-GIN: token-level dynamic graph-interactive network for joint multiple intent detection and slot filling. _CoRR_.
* Qin et al. (2020b) Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020b. Towards fine-grained transfer: An adaptive graph-interactive framework for joint multiple intent detection and slot filling. In _Findings of the Association for Computational Linguistics: EMNLP_, pages 1807-1816.
* Rodriguez et al. (2020) Pau Rodriguez, Issam H. Laradji, Alexandre Drouin, and Alexandre Lacoste. 2020. Embedding propagation: Smoother manifold for few-shot classification. In _ECCV_, pages 121-138.
* Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems_, pages 4077-4087.
* Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR_, pages 1199-1208.
* Tsai and Lee (2020) Che-Ping Tsai and Hung-yi Lee. 2020. Order-free learning alleviating exposure bias in multi-label classification. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence_, pages 6038-6045.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(11).
* Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems_, pages 3630-3638.
* Xiao et al. (2019) Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-specific document representation for multi-label text classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP_, pages 466-475.
* Xu and Sarikaya (2013) Puyang Xu and Ruhi Sarikaya. 2013. Exploiting shared information for multi-intent natural language sentence classification. In _INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association_, pages 3785-3789.
* Yang et al. (2020) Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. 2020. DPGN: distribution propagation graph network for few-shot learning. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_, pages 13387-13396.
* Yang et al. (2018) Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. SGM: sequence generation model for multi-label classification. In _Proceedings of the 27th International Conference on Computational Linguistics, COLING_, pages 3915-3926.
* Yang et al. (2021) Shuuo Yang, Lu Liu, and Min Xu. 2021. Free lunch for few-shot learning: Distribution calibration. In _9th International Conference on Learning Representations, ICLR_.
* Yoon et al. (2018) Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. 2018. Bayesian model-agnostic meta-learning. In _Annual Conference on Neural Information Processing Systems, NeurIPS_, pages 7343-7353.
* You et al. (2019) Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, NeurIPS_, pages 5812-5822.
* Zhang and Chen (2018) Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurIPS_, pages 5171-5181.

 

Wenjie Zhang, Junchi Yan, Xiangfeng Wang, and Hongyuan Zha. 2018. Deep extreme multi-label learning. In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, ICMR_, pages 100-107.
* Zhou et al. (2003) Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Scholkopf. 2003. Learning with local and global consistency. In _Advances in Neural Information Processing Systems 16_, pages 321-328.

## Appendix A Label Similarity

Figure 5(a) and 5(b) describes the label similarity of _schedule_ (**Sc**) in StanfordLU and _shopping_ (**Sh**) in TourSG using Bert-base embedding, which is calculated by Eq. (1). We can observe that (1) the classes in _schedule_ are more similar and the classes in _shopping_ are more distant. (2) the similarity of label description reflects co-occurrence between labels to some extent from the pragmatics view, e.g., in **Sc**, the similarity of _request date_ and _request time_ is high, and in real world scenarios, human usually asks the two details together, as well as _request date_ and _request location_, which is crucial in multi-label tasks with low resources




* [14] M. C. B. K. and J. A. S. G. Wilson, _The Theory of Computing_. Cambridge University Press, New York, 2003.
* [15] D. R. T. Jones, _A Course on the Theory of Computing_. Oxford University Press, Oxford, 1987.
* [16] E. W. H. Lieb, _Introduction to the Theory of Computing_. Springer-Verlag, Berlin, 1987.
* [17] F. L. Lieb, _On the theory of computing_. Oxford University Press, Oxford, 1987.

[MISSING_PAGE_POST]

 



**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.



C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _4.1.2_
* C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _4.1 and 4.3_
* C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _4.1.2_
* D **Did you use human annotators (e.g., crowdworkers) or research with human participants?** _Left blank._
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _No response._
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _No response._
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _No response._
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _No response._
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _No response._