PCMID: Multi-Intent Detection through Supervised Prototypical Contrastive Learning
Authors: Yurun song
Junchen Zhao
Spencer B. Koehler
Amir Abdullah
Ian Harris
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue System
Multi-Intent Detection

Abstract: Intent detection is a major task in Natural Language Understanding (NLU) and is the component of dialogue systems for interpreting users’ intentions based on their utterances. Many works have explored detecting intents by assuming that each utterance represents only a single intent. Such systems have achieved very good results; however, intent detection is a far more challenging task in typical real-world scenarios, where each user utterance can be highly complex and express multiple intents. Therefore, in this paper, we propose PCMID, a novel Multi-Intent Detection framework enabled by Prototypical Contrastive Learning under a supervised setting. The PCMID model can learn multiple semantic representations of a given user utterance under the context of different intent labels in an optimized semantic space. Our experiments show that PCMID achieves the current state-of-the-art performance on both multiple public benchmark datasets and a private real-world dataset for the multi-intent detection task.

PCMID: Multi-Intent Detection through Supervised Prototypical
Contrastive Learning
Yurun Song*
Junchen Zhao
Ian G. Harris
UC Irvine yuruns@uci. edu
UC Irvine
UC Irvine
junchez3@uci. edu
harris@ics. uci edu
Spencer B. Koehler
Amir Abdullah
Aktify. Inc koehlersb747@gmail. com
Aktify. Inc ami ralrilli weraal19880 1185899ail.commm
Abstract
2019, Qin al., 2020a). These models currently achieve very promising results.
Intent detection is major task in Natural Lan- guage Understanding (NLU) and is the core component of dialogue systems for interpret- ing users' intentions based on their utterances. Many works have explored detecting intents by assuming that each utterance represents only single intent. Such systems have achieved very good results; however, intent detection is far more challenging task in typical real-world sce- narios, where each user utterance can be highly complex and express multiple intents. There- fore, in this paper, we propose PCMID, novel Multi-Intent Detection framework enabled by Prototypical Contrastive Learning under pervised setting. The PCMID model can learn multiple semantic representations of a given user utterance under the context of different intent labels in an optimized semantic space. Our experiments show that PCMID achieves the current state-of -f-tt--r-----ttttt performance on both multiple public benchmark datasets and a pri- vate real-world dataset for the multi-intent de- tection task.
Despite the promising results single-intent detection, the assumption that user utterances each represent only a single intent does not always hold in real-world scenarios where an utterance can rep- resent multiple intents (Li et al., 2018b, Rastogi et al., 2019). For example, consider a natural ex- pansion of the previous sample utterance: "Show me the rype of aircraft that Delta uses and what time zone is Denver in?". In this case, the classifier should predict multiple intent labels including Air- craft Type and City instead of just one. In scenarios such as these, failing to consider multiple intent predictions will result in incomplete Natural Lan- guage Understanding and limit the performance of a dialogue system.
In order to deal with this issue, Gangadharaiah and Narayanaswamy (2019) explore the joint multi- intent detection and slot filling task using a multi- task framework with a slot-gated mechanism (Goo et al., 2018). This work incorporates multi-intent information within single context vector but does not integrate fine-graid fine-grained intent information for token-level slots, resulting in ambiguous relation- ships between filled slots and intents. Following this work, Qin al. (2020b) introduce an intent- slot graph interaction layer to model correlations between slots and intents, but the solution does not scale to cases where conjunctions are implicitly defined in utterances.
1 Introduction
Natural Language Understanding (NLU) is a key component of dialogue systems, particularly for the task of intent detection (Tur, 2011) to interpret users' utterances throughout conversations. Intent detection can essentially be regarded as semantic classification problem. Given an utterance like "Show me the type of aircraft that Delta uses," " intent detection can be implemented as a classifier that predicts an intent label for the utterance such as "Aircraft Type." This type of intent detection is often called single-intent detection which has been amply explored in previous works. Many of these works adapt joint models to also capture the relationship between intent detection and slot filling (Goo et al., 201 8, Li et al., 201 8a, al.,
Most recently, Wu et al. (2021) propose a label- aware BERT attention network, which achieves the state of the art performance on multi-inten detec- tion. This work first applies two BERT encoders, one to capture the semantics of utterances without explicit conjunctions and the other for the words in the intent labels. Then, label-aware layer is constructed which iteratively projects intent label representations to utterance representations in or- der to compute a projection weight that represents
Equally contributed.

---

the similarity between utterance and intent labels for the purpose of multi-intent detection.
doza and Zamora, 2009) and graph based methods (Hu al., 2009) for intent detection tasks. With the prevalence of deep neural network methods, later works achieve intent detection via RNNs (Ravuri and Stolcke, 2015, Sreelakshmi et al., 2018), CNNs (Gupta et al., 2019; Hashemi et al.. 2016), Capsule Networks (Xia et al., 2018) and Transformer-basee models (Vaswani al., 2017; Castellucci et al., 2019; Chen et al., 2019; Zhang al., 2019; Mehri and Eric, 2021).
Although this intuitive approach works well for detecting multiple intents, it fails to take the high complexity of the intent label embedding space into account and does not consider the semantics between individual utterance tokens and the intent label words. This results in ambiguity in matching intent labels utterances such that the related in- tent information for each utterance token cannot be captured. Furthermore, because two BERT en- coders are applicd, model size and computational efficiency become issues for both the training and inference phases.
Although intent detection works that focus on the single intent use-case have achieved great per- not
formance on this task, user utterances stricted to conveying one intent at time in practice. Consequently, multi-intent detection is needed for advancing this research field. The work by (Xu and Sarikaya, 2013) applies a log-linear model atop n-gram features for multi-label detection. Rychal- ska et al. (2018) propose a hierarchical model for multi-intent detection. Gangadharaiah and Narayanaswamy (2019) propose joint multi-intent detection and slot filling using multi-tasking framework and slot-gated mechanism but cannot resolve ambiguous relationships between slots and intents. Qin et al. (2020b) propose an adaptive intent-slot graph interaction layer which mitigates the issue of ambiguuus relationships from the pre- vious work but requires intents to be detectable by the presence of conjunctions in utterances. Wu et al. (2021) propose label-aware BERT atten- tion network which able to detect implicit intents without explicit conjunctions.
Therefore, in this paper, we propose our frame- work PCMID": Multi-Intent Detection through Su- pervised Prototypical Contrastive Learning, with the following main steps and associated contri- butions:
We first obtain the utterance and label embedding by constructing an utterance- label encoder using a single BERT encoder. This achieves smaller overall model size with higher computational efficiency.
Next, we obtain the utterance-label attention by passing the utterance and bel embedding to a multi-head attention block that builds the relationship between each ut- terance token and its respective multi-intent labels, allowing each utterance token to cap- ture information relevant to its intent labels.
la-
Finally, we apply multi-label Supervised Pro- totypical Contrastive Learning to learn multi- ple semantic representations of a given user utterance under the context of different intent labels in an optimized semantic embedding space of intent labels and utterance-label at- tention.
2.2 Contrastive Learning (CL)
Contrastive Learning (CL) is specific form of self supervised learning, that focuses on using "posi- tive" and "negative" samples to improve the latent embedding space of models for better classification results. The core idea of CL is finding strong manifold representation via pulling "positive" pairs closer together, and repelling negative samples un- der dot product similarity metric. This idea was first applied in the field of Computer Vision and found great success in multiple tasks such as Data Augmentation (Gao et al., 2021a, la Meng al., 2021, Yan et al., 2021) and Instance Discrimination (Wu et al., 2018, Oord et al., 2018, Ye et al., 2019, He et al., 2019).
We conduet experiments multiple public benchmark datasets and private real-world dataset and show that our approach achieves the current state-of-the-art performance on the multi-intent de- tection task.
2
Related Works
2.1 Intent Detection
Early works apply traditional machine learning ap- proaches such as Support Vector Machines (Men-
Although previous works in CL have achieved great performance, many them do not consider the semantic structure data, a fact which usually

---

leads to samples being treated as either positive or negative pairs long they are from different instances without considering their semantic simi- larity. This results samples with shared semantic similarity being pushed apart in the semantic space and prevents the model from improving. To ad- dress this issue, Li et al. (2020) propose Prototyp- ical Contrastive Learning (PCL), which accounts for semantic structure. Khosla al. (2020) pro- pose Supervised Contrastive Learning which ex- tends the self-supervised CL to fully-superrvsed CL and more effectively utilizes the label informa- tion CL. Dao et al. (2021) propose multi-labe labell Supervised CL that allows model to learn multiple representations of an image under the context of different given labels.
Loas bec
Loss,
Projectoon
PCL.
Utterance Label Embeddings
MAB
PPNN
Multi-hae allention
Value
Query
Utterance Embuldinge
Ezabeddings
Pooling
In this paper, we apply the principles from these prior works to the multi-intent detection task by applying multi-label supervised CL, and PCL. Fi- nally, we show our approach's 's effectiveness in our experiments.
Utterance Label
X Fied
Encoder
BERT
Uttersmness
3 Approach
X1, X2, X3 Xn
L1,
this section, we describe detail our proposed model architecture, PCMID, as illustrated in Fig- ure 1. The architecture consists of an utterance- label encoder module that uses single BERT en- coder, multi-head attention block module, and multi-label supervised contrastive learning module.
Figure I: Model Architecture PCMID is mainly com- posed by three components, which are Utterance-Labe Encoder, Multi-head Attention Block and Prototypical Contrastive learning.
M labels in total. Formally, the process of obtain- ing the utterance embedding, Ex, and intent label embedding, Ey, is defined as,
3.1 Utterance-Label encoder
The first component in our model architecture is an utterance-late encoder, as shown at the bot- tom of Figure 1, based on BERT and denoted as BERT below. This encoder is used to obtain both teken-level embedding of the utterance and sentence-level embedding of the intent label. Given the utterance inputs X (21, .2...., .... pn)) we obtain token-level embeddings for each utter- ance by simply feeding each x; into the BERT encoder. Given the intent label inputs Y (4,, lm) we also compute the intent label sentence-level embeddings by first feeding each 1, to the BERT,, encoder with pooling mechanism to get the sentence-level hidden representation of each intent label and then stacking them together. Constructing the intent label embedding the sen- tence level enables the model learn the hidden representation of multiple intent labels given sin- gle utterance such that each I; E X associated with multiple intent labels selected from the set of
Ex BERT
Fy Stack (hys : ha bha::....pp hym.)
(1)
where hyg = pooling (BERT
In Equation 1, Ex E RR the token-level em- bedding of utterance X with maximum sequence length of L and embedding size of H. Ey e M the intent label sentence-level embed- ding with the label input size of M. hy, RR. I is the sentence-level hidden representation of each label 1,. where j E (1,...m) constructed using pooling methods including default BERT pooling, mean pooling and self attentive pooling Wu et al. (2021) to convert the token-level intent label hid- den representations to sentence-level intent label hidden representation.
3.2 Multi-head Attention Block
The next component of PCMID is the Multi-head Attention Block (MAB) inspired by the work

---

of Dao al. (2021) shown in the middle of Fig- ure The Multi-head Attention Block is adapted from the BERT encoder without considering the positional embedding.
is the batch size of the utterance samples, M is the number of intent labels, and H is the embedding size.
Multi-head Attention. Multi-head attention is an exiension of attention which is first introduced by Vaswani et al. (2017). Compared to single attention, multi-head attention projects Query, Key, Value to h different head vectors separately. Then, the attention applied the projections individually and results linear transformation of the combination of all attention outputs. By applying multi-head attention, the model can attend to information from different hidden representation subspaces at different positions. In our work, the Query is the intent label embedding Ey, the Key and Value are the utterance embedding Ex. We express multi-head attention as follows:
3.3 Multi-label Supervised Contrastive Learning
The final component of our work is the Multi-labed Supervised Contrastive Learning module, which is done through Contrastive Learning (CL) and Pro- totypical Contrastive Learning (PCL), as shown at the top of Figure and as described in the following subsections.
Supervised CL. Contrastive Learning (CL) has been increasingly popular and has proven to be effective learning mechanism. It shows extraordi- nary performance for optimizing sentence seman- tic hidden representations in an embedding space in a single-label setting and in self-supervised way (Gao et al., 2021b; Chen et al., 2022: Ke et al., 2021). CL helps improve the performance of models, but, as self-supervised technique, CL does not leverage the label information and the model's learning capability is limited thereby. Con- sequently, Khosla et al. (2020) propose Supervised CL (SCL) to utilize the label information by select- ing positive and negative samples to achieve a sig- nificant improvement in model performance, Such improvements are also realized in similar works, like (Liu et al., 2021; Hu et al., 2022; Gunel et al., 2020). Motivated by these works, we adapt SCL to the multi-intent detection task.
MHA Ex, Ex)) =
where O₁ Att(Ey W4
Ex W xb.
Ex W
(2)
where the Wo, W Wh W are the Uh nq. parameters which are learnable and Fy-WW, Ex Ex mn Ex are the projected vectors based our Query, Key and Values.
Multi-head Attention Block. Based on multi-head attention, we define the Multi-head Attention Block (MAB) which is adapted from the encoder block of BERT, without considering the positional em- bedding and dropout. In our work, the MAB can be viewed as 12-layer BERT model with an ex- tra layer top of the BERT model, taking Ey the Query and Ex as the Key and Value. By designing the MAB in this way, each intent label queries the utterance Keys and determines whether the utterance is close to the intent label semanti- cally. The MAB also obtains an utterance-label embedding, denoted as G, which represents the se- mantic association between each utterance's 's hidden representation in both its embedding and its intent label embedding. The utterance-labed embedding G through MAB is formally defined such that:
Naively extending SCL to multi-intent detection can be complicated by a label mismatch issue. For multi-labe SCL, if we have multiple positive sam- ples with an unequal number of negative samples, SCL is apt to find erroncous manifold represen- tations, which in turn would decrease the model performance.
In order to solve this problem, we implement SCL in multi-label setting by applying our con- structed utterance-label embedding G. In this em- bedding. since each utterance's 's hidden representa- tion is correlated with each intent label's hidden representation in their respective embeddings, the number of positive samples and negative samples will be the same and the label mismatch issue is avoided.
G = MAB(Ey, Fy, Ex; Ex)
LayerNorm Ey FFNN(E
(3)
Given the utterance-label embedding G with N utterance instances in each mini-batch, we rep- resent each utterance's hidden representation with respect to intent label hidden representation
B LayerNorm(Ey- MHA(EH Ex, Ex))
Then, we have the output G E NxMH where N

---

j such that (h(i,)) E Gii, E (J....Ny:j e
We combine the multi-label classification loss, the Binary Cross-Entropy loss (BCE), and the Su- pervised PCL loss together during training, for our work's s (PCMID's ') loss as:
{I.... ..}} and we represent the ground truth of each utterance sample regarding each intent label j such that {Jky E {0. 1)li E ...,3....,N): E (1,2,3,... M)}- Hence, each multi-intent utter- ance i can be transformed to the single-intent rep- resentation h() with respect to single ground truth label; so, the multi-intent problem can be solved as single-intent classification, For SCI., regard hij as an anchor with the goal of pulling the hi with the same intent label j closer. Accordingly, we define the set containing all positive samples of hy as j) = {hzj E 5) = Bky = 1} and the set containing all negative samples as 5i, = G \ P(i,)). Then, we define the SCL
EPOMID = {CC + +
(6)
where the parameter is used to control the trade- off between the two losses.
4 Experiment
Datasets. We conduct experiments on three pop- ular public datasets including MiXATIS², MixS- NIPS3 and Facebook Semantic Parsing System (FSPS)", which contain 18, 7, and 25 unique in- tent labels, and 14749, 44173, and 44783 utterance samples, respectively (Qin et al., 2020b, Coucke et al., 2018, Gupta et al., 2018). We also use a real-world private commercial dataset in the credit repair domain, referred to as CREDITI6' which contains 16 unique intent labels and 4384 utterance samples. This dataset reflects the real-world chal- lenges regarding the multi-intent detection task.
loss as:
log exp(hsphe/T) caplhcphh/7)) (GE5(1))
Exd
Σ
P(iJl
N NrEP(is)
(4) where My E j)i,j) is the positive sample, h E ((i,j) is the negative sample and is its tempera- ture.
Supervised Prototypical Contrastive Learning Li et al. (2020) propose self supervised Prototyp- ical Contrastive Learning (PCL), which adapts clustering mechanism to CL in image tasks. Works including (Medina al., 2020; Yue al., 2021) use PCL for the pre-training of few-shot model and achieve great performance. Wang et al. (2021) apply PCL for improving the label semantic em- bedding space for the zero-shot slot-filling task. Unlike self-supervised PCL, which finds the distri- bution of prototypes via clustering, we adapt PCL supervised method which marks the labels as prototypes and optimizes the representation of both the intent labels and the utterance samples in the utterance-labll embedding space G.
We pre-process the MiXATIS, MixSNIPS and FSPS datasets following the data processing pipeline from the previous work (Wu et al., 2021; Qin et al., 2020b), Also, we pre-process the CREDITI6 dataset by lower-casing and then moving punctuation characters and emojis from the utterances and intent labels. Next, we remove lineline characters and repeated white space char- acters from each utterance sample. We split the CREDITI6 dataset into train, validation and test sets using the ratio of [0.8, 0.1, 0.1J, respectively. The detailed dataset characteristics are shown in Table 1.
The definition of the positive set of samples and negative set of samples differs between PCL and CL. In PCL, we set the prototype to the intent la- bel embedding Ey E M Then, we define (i.,)) as the hard positive set of samples which satisfies that (i, Eylug- 1} and jii,j) is hard negative of samples which satisfies jh E Eylsi A 1}. We define the Super- vised Prototypical Contrastive Learning (SPCL)
MiXATIS
MiXSNIPS
FSPS
CREDIT16
Num. Label
18
7
25
16
Num. Utt
14749
44173
44783
4384
Train
13162
397768
31279
3570
Validution
828
2199
9042
439
Test
759
2198
4462
439
Table 1: Dataset Characteristics
loss as:
httn/data/MixATTs_cleee
thttps://gittub....
maih/data/MtissMMF__cleaa
Σ log capths hi/TT capthesphe/T)
Lapcl
main/data/SNIPS
R(EQ(1
(5)
main/data/Aktify

---

MixATIS
MixSNIPS
FSPS
CREDITI6
Models
Acc
F1
Acc
F1
Ace
F1
Acc
FI
Stack Prop
0.719
0.790
0,9446
0.976
0.723
0.911
Joint MID-SI SF
0.731
0.806
0.951
0,980
0.780
0.877
AGIF
0.758
0,980
0.749
0.914
LABAN
0.765.0.3%
0.90520.43
0.955a0.3%
0.98040.2%
0.91040.1%
0.937&0.18
0.44720.9%
0.716:0.78
PCM pooler
0.773=0.5%%
0.912+0.2%
0.960+0.2%
0,983+0.1 1%
0.913+0,
0.941+0.11 1%
0.471+1.2%
8.759=0.8%
PCMID mean
0.792=0.69%
0.911=0.7%
0.95940.75 7%
0.98240.6%
0.910s0.1 1%
0.940+0: 1%
0.482=1
0.747=0.7%
PCMID pooler
0.837":40.7%
0.940"=0.79 %
0.96010.2%
0.983=0.1%
0.916a0.1%
0.94210.11 1%
0.478220.6%
0.745=0.6%
Pire mean
0.819=1.22
0.93320.8%
0.960H0.3%
0.982+0.1%
0,917*=0.1%
0,495f=1.0% 0.944*=0.1%
0.759=0.4%
S-PCMID pooler
0.8 0.816=0.9%%
0.917=0.5%
0.983+0.2%
0,912+0,255
0.940+0. 1%
0.472 44%
0.742=0.8%
S-PCMID mean
0.809e1
0.919#0.7%
0.958+0.5%
0.98240.15 1%
0.913+0.2%
0.941 1
0.400=0.7%
0.749=0.5%
Table 2: Multi-intent detection results four datasets. We report accuracy (Acc) for all intent exact matches and F1 scores based on individual intent calculations. indicates the significant improvement of p-value < 0.05 compared the previous models. base indicates BERT-base, and large indicates BERT-large.
Baseline Experiments. We conduct experiments comparing our approach to previous works including:
our learning rate scheduler, in which it adjusts the learning every 4 training steps with multiplicative factor of learning rate decay equal to 0.9. We set the training epochs, initial learning rate, and batch size hyperparameters to 50, 1e 5, and 128, respectively in our experiments on the MiXATIS, MixSNIPS, and FSPS datasets. However, our CREDITI6 experiment, we failed to converge within the 50 training epochs, so we increased the number of training epochs to 100 for the CREDITI6 dataset.
Stack-Prop (Qin et al., 2019), which proposes shared self-attentive encoder and two decoders for joint intent and slot filling tasks separately. and Narayanaswamy, 2019), which uses Bi-LSTMs with a slot-gated mechanism for slot filling and intent detection tasks.
Joint-MID-SF
(Gangadharaiah
AGIF (Qin et al., 2020b), which adapts graph BI-LSTM interactive architectures for both slot filling and intent detection tasks simultancously. LABAN (Wu et al., 2021), which encodes both utterances and intent labels through two different encoders, along with an adaptive label-aware attentive layer for the multi intent detection task specifically.
We set the threshold for multi-label classifica- tion 0.5 and evaluate the results based on the Fi-score and Accuracy metrics. We run the train- ing experiments 10 times per datasct and obtain the trained model parameters based on the best vali- dation accuracy epoch for computing the testing results. We report results of the final model by averaging 5 testing results per dataset.
Experimental Setup. our work, PCMID, we conduct experiments based on the pre-trained BERT-Base-Uncased language model. We apply a pre-trained sentence embedding, the BERT-Base- NLI-STSB-Mean T from sentence trans- former (Reimers and Gurevych, 2019), to further improve model's semantic understanding ca- pability in our later experiments. Additionally, we have also conducted experiments using pre-trained BERT-Large-Uncas model our PCIMID frame- work. We use BERT default pooling (PCMID- pooler), mean pooling (PCMID-mean) and self- attentive pooling (S-PCMID-pooles in our exper- iments. Our experiments are conducted using an Nvidia Titan RTX with 24GB GPU memory.
Results
Multi-Intent Results. Our main experimental re- sults for the multi-intent detection task are shown in Table 2. One of the previous works which we compare to is LABAN (Wu et al., 2021), however we could not reproduce the results presented in their publication. Upon inspection of their github repository, we found that their published results their validation set as part of their test set. We generated the LABAN results shown in Table 2 by using the code provided in their github repository, but we separated the test set from the validation set.
We find that all PCMID models achieve close results on the MiXSNIPS datasets. S-PCMID base pooler, which is PCMID with BERT default pool- ing using pre-trained sentence embeddings, is slightly better than other models. This fact sug- gests that the model can capture better contex-
During our training we use the AdamW opti- mizer as our parameter optimizer, apply StepLR R as
'https: / ht/hg/hggggggaceeeeeeee co/sentence -eteeeenneeecccceeeeeeeeeeeee- wwwpencceerrrssforoennoeseo frrmsssormess/ bert-base malasenniiae nwi--nnissttte sii--sttb-maan tokens

---

Samples
Labels
Predictions
give me the fares from miami cleveland next sunday and which airline is
atis airfare
atis_ atis_airfare,
atis airline
atis airline
atis atis_city,
2 list la and how much is limousine service in la guardia
atis ground fare
atis ground_farr already_ _elient money_ coney_loo_eete sonee_loo__xpensivee
understand that already have safety alert center and it's expensive for what
already _liiett money_too_ tloo_eopessiieee sio_calling. don't "on't_owe_anythine
3
1 i don't t have debt with you 2 ignore your calls
4
emails and messages because i'm no longer interested
Table 3: Real samples extracted from the MixATIS and CREDITI6 Datasets via the PCMIDiose mean model.
ATIS
SNIPS datasets and compare the experiment re- SNIPS
Models
Acc
Acc
sults of our work with previous works. As can PCMID mean ap- proach using BERT-large pre-trained word em- bedding achieves the highest performance on the single-intent detection task for the accuracy evalua- tion metrics. We observe that although the other re- sults from our approach, such as PCMIDbose pooler and S-PCMID pooler, are not achieving accu- racy results good the PCMIDIarge mean, they still perform relatively well on both datasets and have minimal performance gap compared to pre- vious works.
Stack-Prop
0.969
0.980
be seen in Table
Joint MID-SF
0.954
0.972
AGIF
0.971
0.981
LABAN
0.976 +0.2%
0.977 = 0.2%
PCMID bouse pooler
0.968 +0.6%
0.979 + 0.4%
PCMID base mean
0.979 +0.2%
0.982 + 0.2%
PCMID large pooler
0.973 #0.4%
0.980 0.2%
PCMI Bange mean
0.980 +0.2%
0.982 = 0.1%
S-PCMID pooler
0.978 +0.3%
0.980 + 0.4%
S-PCMID bese mcan
0.977 +0.2%
0.981 + 0.3%
Table Single-intent detection results on datasets, We report the accuracy (Acc) for the verbatims labelled correctly.
6 Analysis
PCL on Multi-Intent Detection. In Table 5, we present the results of PCMID with different combi- nations of options between CL and PCL and com- pare them on all of the datasets use. Given the results shown in Table 5, can see that by applying PCL, PCMID outperforms PCMID with- out any CL strategy on the MiXSNIPS, FSPS and CREDITI6 datasets. Especially on the CREDITI6 dataset, using PCL has an absolute advantage over PCMID alone. This indicates that PCL plays an important role in the PCMID model, especially when the amount of training data is limited. When compare the results of PCMID with CL only and PCMID without CL PCL. we also see that PCMID's performance is substantially improved on MIXATIS and MixSNIPS datasets, but not on the FSPS and CREDITI6 datasets. This suggests that PCMID with CL might be affected by data volume and complexity. Intents Visualization To illustrate the effective- ness of PCL optimization, Figure 2 shows the dis- tribution of label representations of the test samples
tual semantic information by applying the pre- trained sentence embeddings even though the model size smaller than other PCMID We also find that our PCMID large pooler achieves the highest performance on the MiXATIS dataset. PCMID iarge mean, which is PCMID with mean pooling using pre-trained BERT-large word embed- dings, achieves the highest accuracy and the best FI score both the CREDITI6 and FSPS datasets.
Our work, PCMID, outperforms previous base- line results on every public benchmark dataset with only single BERT encoder. Regarding our private CREDITI6 6 dataset, we compare our work with LA- BAN and BERT and show that our work still gets the best performance. However, we do not compare work with the other baseline models (Stack- Prop, Joint-MID-SS AGIF) on the CREDITIG dataset since CREDITI6 not designed for the joint multi-intent detection and slot filling task, and is therefore not comparable to these works.
Single-Inted Intent Results. In order to show the ro- bustness of our work, we also evaluate our ap- proach in the single-intent setting using ATTS? and
main/ data//taSA
"https: / thub. httbb..cm/jccccassssse//oo/rrree main/data/SNIPS

---

Dataset
MiXATIS
MixSNIPS
FSPS
CREDITI6
With CL
With PCL.
Acc
F1
Acc
Acc
Acc
No
Yes
0.773
0.912
0.960
0.983
0.913
0.941
0.471
0.759
Yes
No
0.783
0.923
0.959
0.982
0.904
0.940
0.437
0.740
Yes
Yes
0.788
0,919
0.961
0,982
0,909
0,939
0.466
0.761
No
No
0.767
0.905
0.958
0.981
0.911
0.940
0.448
0.756
Table Analysis of contrastive learning strategies for PCMID base- The first row default strategy through the
PCMIDbose pooler.
TSNE visualization of intent clusters
It indicates that the model can be further trained with some domain specific terms like location, com- pany, etc. Because the CREDITI6 dataset contains real world user conversations, the utterances in the dataset contain some noise. When faced with the imperfect case 3 (incomplete sentence) from the CREDITI6, PCMID is able to make the correct prediction. Even if human labels corrupted, in case PCMID makes a correct classification ignoring the noise. In summary, PCMID is robust model, but it can further improved by learning the representation relationships between specific terminologies and labels.
In terms of model size, LABAN's (Wu et al.. 2021) parameters are about twice that of our PCMID due to LABAN's use of separate BERT encoder for the label. But their performance is worse than our model. Additionally, BERT fine- tuning classification can yield decent perfor- mance gain on some tasks. By simply building a cross-attention layer of target and utterances on top of BERT, the model can be further boosted fol- lowing the PCL strategies. With small additional costs, the model can gain consistent improvements, especially in the multi-labed tasks.
Cambiie
yet pwent praveeergrooot
1o se, 300i
E. get contect
10 pernua
18 aer
giactions
penatio
get axdimatdd anival
13 iis ratk
21: snuppoted
gel _eeurtte__
Di oet kcaicce
22: snas
saliutad sunbtrr
kcation
Zo wngepatss..woett
gct picoreee
The distribution of FSPS intent representa- tions of test samples from the PCMID pooler.
Figure 2:
Conclusion
In our paper, we propose multi-label classifica- tion model PCMID with single BERT encoder handle the multi-intent detection task with smaller model size and higher computational efficiency but still achieve higher performance compared with previous statc-of-art work.
from FSPS. Each cluster is clearly separated with- out significant overlap. Compared with the plot clusters
diagram from (Wu et al., 2021), more concentrated within the class and more iso- lated from each other. other words, due to the PCL optimization, each utterance can be more eas- ily classified into their label clusters.
Our approach maps complex utterance states to an utterance-label representation with respect to joint label representation, and adapts proto- typical contrastive learning to optimize both the utterance encoder and the representation. Experi- ments and analysis demonstrate the effectiveness of our proposed model, which can further allevi- ate the discrepancies between utterance and label representations and improve the quality of label
Samples Analysis As motivational example, con- sider cases and 2 in the Table 3, from the Mix- ATIS dataset. PCMID is able to predict all labels correctly in case 1. However, also shows case 2 that PCMID cannot understand the representation "la", and misses the label "city" in its prediction.

---

representations and the performance of multi-label classification. In the future, we will explore how to further optimize the label representations in cross- domains and study the effectiveness of the model on domain transfer tasks.
Haihong Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. A novel bi-directional interrelated
model for joint intent detection and slot filling.
Rashmi Gangadharaiah and Balakrishnan Narayanaswamy. 2019. Joint multiple intent detection and slot labeling for goal-oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 564-569, Minneapolis, Minnesota. Association for Computational Linguistics.
Limitations
Although our work achieves good results the multi-intent detection tasks, we are focusing only on single-domain multi-intent detection and not on cross-domain domain settings; a fact which limits our work's generalizability.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021a. Simese: Simple contrastive learning of sentence em- boddings. arXiv preprint arXiv:2104.088
Through experiments, find that the perfor- mance of PCMID is sensitive to the initialization of label embeddings. When the label name is not well defined, the pre-trained label embedding through the BERT encoder can reduce the performance, which is even worse than with randomly initialized label embeddings. For example, if the intent labels are not clearly defined, the model will potentially be misled and optimize both utterance embeddings and label embeddings in the wrong direction the beginning. As result, utterances will be assigned to the wrong intent label clusters.
Tianyu Gao. Xingeheng Yao, and Danqi Chen. 2021b.
Simese: Simple contrastive learning of sentence em- beddings.
Chih-Wen Goo, Guang Gao, Yun-Kat Kai Hsu, Chih-Li Huo, Tsung-Chieh Chieh Chen, Keng- Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 753-757, New Orleans, Louisiana. Association for Computational Linguistics.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoy- anov. 2020. Supervised contrastive learning for pre- trained language model finettuiig. arXiv preprint arXiv:201..0140)
References
Tiago Almeida, José María Gómez Hidalgo, and Tiago Pasqualini Silva. 2013. Towards sms spam filtering: Results under a new dataset. International Journal of Information Security Science, 2(1):1-18.
Arshit Gupta, John Hewitt, and Katrin Kirchhoff. 2019. Simple, fast, accurate intent classification and slot labeling for goal orienteee dialogue systems. arXiv preprint
Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019. Multi-lingual intent detection and slot filling in a joint bert-based model. arXiv preprint 1XX7::907700288
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task ori- ented dialog using hierarchical representations. arXiv preprint urXiv:1810.0799
Junfan Chen, Richong Zhang, Yongyi Mao, Jie Xue. 2022. Contrastnet: A contrastive learning framework for shot text classification.
Homa Hashemi, Amir Asiace, and Reiner Kraft. 2016. Query intent detection using convolutional neural net- works. International Conference on Web Search and Data Mining, Workshop on Query Understand- ing.
Qian Chen, 7hu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. anXiv preprint 1902 10909 10909.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019. Momentum contrast for unsu- pervised visual representation learning.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips voice plat- form: an embedded spoken language understanding system for beyaate-b-desige voice interfaces. arXiv preprint arXiv: 100.:00000000
Jian Hu, Gang Wang, Fred Lochovsky, Jian-tao Sun, and Zheng Chen. 2009. Understanding user's query intent with wikipedia. Proceedings of the 18th international conference on World wide web, pages 471-480.
Son D. Dao, Ethan Zhao, Dinh Phung, and Jianfei Cai. 2021. Multi-label image classification with con- trastive learning.
Xuming Hu, Li Lin, Lijie Wen, et al. 2022. Pair-level supervised contrastive learning for natural language inference. In ICASSP 2022-2022 IEEE International

---

Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages
Yu Meng. Chenyan Xiong, Payal Bajaj. Paul Bennett, Jiawei Han, Xia Song, et al. 2021. Coco-Im: Cor- recting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems, 34.
LEEE.
Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu. 2021. Classic: Continual and contrastive learning of as- pect sentiment classification tasks. arXiv preprint arXiv:2112.02714.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan, 2020. Super- vised contrastive learning.
Libo Qin, Wanxiang Che, Yangming Li, Minheng Ni, and Ting Liu. 2020a. Der-net: A deep co-interactive relation network for joint dialog act recognition and sentiment classification.
Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability in- dex, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Re- search Branch.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. A stack-propagation framework with token-level intent detection for spoken language understanding. arXiv preprint 190.v1999.00//11
Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020b. Agif: An adaptive graph-interactive framework for joint multiple intent detection and slot filling.
Changliang Li, Liang Li, and Ji Qi. 2018a. A self- attentive model with gate mechanism for spoken lan- guage understanding. In EMNLP.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. 2020. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966,
Xiujun Li, Yu Wang, Siqi Sun, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018b. Microsoft dialogue challenge: Building end-to-end task-completion dia- logue systems.
Abhinav Rastogi, Xiaoxue Zang. Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.
Han Liu, Feng Zhang, Xiaotong Zhang, Siyang Zhao, and Xianchao Zhang. 2021. An explicit-joint and learning framework for few- shot intent classification and slot filling. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 1945-1955,
Suman Ravuri and Andreas Stolcke. 2015. Recurrent neural network and Istm models for lexical utterance classification. In Sixteenth Annual Conference of the International Speech Communication Association.
Nils Reimers and Iryna Gurevych. 2019. Sentence- Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Matthew Louis Mauriello, Thierry Lincoln, Grace Hon, Dorien Simon, Dan Jurafsky, and Pablo Paredes. 2021. Sad: A stress annotated dataset for recog- nizing everyday stressors in sms-like conversational systems. In Extended Abstracts of the 2021 CHI Con- ference on Human Factors in Computing Systems, pages 1-7.
Barbara Rychalska, Helena Glabska, and Anna Wrob- lewska. 2018. Multi-ine intent hierarchical natural lan- guage understanding for chatbots. In 2018 Fifth In- ternational Conference on Social Networks Analysis, Management and Security (SNAMS), pages 256-259
Carlos Medina, Arnout Devos, and Matthias Gross- glauser. 2020. Self-supervised prototypical transfer learning for few-shot classification. arXiv preprint (tl::2006.11225.
Elvis Saravia, Hsien- Chi Toby Liu, Yen-Hao Huang. Junlin Wu, and Yi-Shin Chen. 2018 CARER: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium. Association for Computational Linguistics.
Shikib Mehri and Mihail Eric. 2021. Example-driven intent prediction with observers. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Human Language Technologies, pages 2979-29999 2992, Online. Association for Computational Linguistics.
Lucas Shen. 2022. LexicalRichness: A small module to compute textual lexical richness.
Marcelo Mendoza and Juan Zamora. 2009, Identify- ing the intent of a user query using support vector machines. In International symposium on string pro- cessing and information retrieval, pages 131-142. Springer.
K Sreelakshmi, PC Rafeeque, S Sreetha, and ES Gay- athri 2018 Deep bi-directional Istm network for query intent detection. Procedia computer science, 143:939-946.

---

Mildred C Templin. 1957. Certain language skills children: Their development and interrelationships. volume 10. JSTOR.
Appendix
Ablation analysis
Gokhan Tur. 2011. Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. John Wiley and Sons.
We focus on the impact of choices in architec- ture and contrastive learning strategy the perfor- mance of models, and vary the pooling strategy choice of contrastive learning strategy and label pre-training strategy.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processsinngnss systems, 30.
vs Mean-Pooling vs Pooler
Liwen Wang. Xuefeng Li, Jiachi Liu, Keqing He, Yuan- meng Yan, and Weiran Xu. 2021. Bridge to target domain by prototypical contrastive learning and la- bel confusion: Re-explore zero-shot shot learning for slot filling. arxiv preprint arXiv:21 10.0:22.
We experiment with different choices of pooling strategy, as seen in Table 6 namely with mean pool- ing, self-attentive layer over the pooled output, and using a pooler layer. We find that at lower accuracy levels, such as seen for MiXATIS, mean- pooler layer, obtaining a
Ting- Wei Wu, Ruolin Su, and Biing Juang. 2021. label-aware BERT attention network for zero-shot multi-intent detection in spoken language understand- ing. Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pages 4896, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
pooling is superior gain of 1.9% in accuracy. Whereas for higher ac- curacy datasets such as MixSNIPS and FSPS, the pooling strategies are essentially equivalent.
In the Table 6, a comparison is made of the pool- ing strategies. The performance of three strategies are close for MiXSNIPS and FSPS. While using the mean-pon hean-pooling shows a slight better performance in both MixATIS and CREDITI6 than the other two datasets. Therefore, using the default pooler and mcan-pooling is recommended because the use of a self-attentive layer requires large number of model parameters.
Zhirong Wu. Yuanjun Xiong, Stella Yu, and Dahua Lin. 2018. Unsupervised feature learning via non- parametric instance-level discrimination.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2018. Zero-shot user intent de- tection via capsule neural networks. arXiv preprint arXiv: 003.00
Puyang Xu and Ruhi Sarikaya. 2013. Exploiting shared information for multi-intent natural language sen- tence classification. In Interspeech, pages 3785- 3789.
Label Pre-trained
We validate the effect of label training on our model, with the other loss and pooler strategies kept at the default values. We find that the accuracy is consistently higher when using label pretraining, ranging from a 0.2% accuracy boost on FSPS, to a 1.3% accuracy boost on MiXATIS.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self-supervised sentence repre- sentation transfer. arXiv preprint
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. 2019. Unsupervised embedding learning via invari- ant and spreading instance feature.
In the Table 7, the results show that our models can achieve a better result with pretrained label states in the MiXATIS, MiXSNIPS and FSPS. But it is not as good as that of using random initialization on the CREDIT16 dataset.
Xiangyu Yue, Zangwei Zheng, Shanghang Zhang. Yang Gao, Trevor Darrell, Kurt Keutzer, and Al- berto Sangiovanni Vincentelli. 2021. Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 53834-13844,
With pretrained label states, the model can han- dle unseen labels more confidently because unseen labels can be turned into processed reprexentation instead of being randomly initialized.
Zhichang Zhang, Zhenwen Zhang, Haoyuan Chen, and Zhiman Zhang. 2019. A joint learning framework with bert for spoken language understanding, IEEE
However, there are some cases in which the label names are too sparse represent the utterance's complex intents. In such scenarios, the model can show a better performance without pretrained label states, as seen for CREDITI6 in Table
Accexs, 7:1 2168849-166888

---

Dataset
MiXATIS
MixSNIPS
FSPS
CREDITI6
Models
Acc
F1
Acc
F1
Acc
F1
Acc
F1
PCMIDbase base: pooler
0.773
0.912
0.960
0.983
0.913
0.941
0.471
0.759
PCMID mean
0.792
0.911
0.959
0,982
0.910
0.940
0.482
0.747
PCMID self-attentive
0.777
0.905
0.960
0.982
0.910
0.939
0,481
0.741
Table 6: Ablation analysis of pooling strategies for label states.
Dataset
MixATIS
MixSNIPS
FSPS
CREDITI6
Uses Pretrained Label
Acc
Acc
F1
Acc
Fl
Acc
F1
Yes
0.773
0.912
0.960
0.983
0.913
0.941
0.472
0.742
No
0.760
0.899
0.957
0.981
0.911
0.940
0.486
0.747
Table 7: Ablation analysis of label pretraining for PCMID. The first row is the default strategy through PCMID
pooler.
MAB Analysis
More details on the CREDITI6 dataset
Similar datasets.
In the main experiments, we achieve good results on multi-intent detection tasks across different datasets by setting label embedding as query and utterance embedding as key&value in MAB. In or- der to ensure this is the best setting for MAB in our task, we also conduct experiments with the sce- nario of setting utterance embedding as query and label embedding key&value in MAB.
SMS focused datasets have been collected before, for example Almeida et al. (2013) collect dataset labeling spam messages, while Mauriello al. (2021) label different stressors in another SMS dataset. As far are aware however, dataset has been collected before that specifically deals with conversational sales interactions over SMS the credit repair domain. In particular, conduct- ing automated conversations over text message a fairly new endeavor, in which we hope this dataset will be useful for benchmarking trained models.
Based on the experiment results shown in Ta- ble 8, we can clearly observe that using label embedding query and utterance embedding valuvalue in MAB achieving significantly higher performance than the other way around. This fact happens because the intent label is much shorter and contains very limited semantic informa- tion compared to utterances, and usually we want to set the value as the one that contains richer se- mantic information to fully explore the multi-head attention mechanism. Therefore, by setting query intent label and utterance as key and values, the utterance-labed embedding, the output of MAB, can directly represent how much label semantic infor- mation is contained each corresponding utter- ance and PCL can further fine-grain the embedding to achieve higher classification results. However, if the query is set as utterance and the label is set key and value, then the output will focus more the token semantic information of utterances that associate with the limited label semantic in- formation, where the utterance-label embedding needs be obtained using additional pooling layer, resulting in worse utterance and label semantic as- sociation and lower classification results.
Initial dataset collection and domain description
We collected an initial dataset of 115,000 un- labelled utterances acquired through SMS text message.
The context of these conversations is agent reaching out to pre-subscribed customer, asking their interest in scheduling call with credit repair representative discuss how their credit scores may be improved.
As such, the utterances span discussion on cus- tomer credit scores, client interest in schedul- ing a call, as well as requests to remove erro- neously added customers from the automated texting list.
Pre-labelling and annotation of CREDIT16
For the domain, we have a pre-collected train- ing dataset with 130 intents for credit repair

---

MAB
MiXATIS
MiXSNIPS
FSPS
CREDITI6
Query/Key/Value Value
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Label/Utter/Utter
0.773
0.912
0.960
0.983
0.913
0.941
0.471
0.759
Utter/Label/LLbell
0.733
0.893
0.952
0.979
0.897
0.935
0.340
0.601
Table 8: Results of PCMID pooler by swapping label embedding and utterance embedding as Query, Key and value in MAB. Label indicates label embedding and Utter indicates utterance embedding.
Metric
Avg. # of sentences
Avg. # of tokens
Avg. # of chars
Avg. TTR
Med. Flesch reading ease
Value
2.63
36.4
174.2
0.84
68.1
Table 9: Length and token type derived statistics.
and general conversational texting. We trained 130 binary text classification models, where for each classifier the negative samples for an intent were randomly collected from all other 129 intents.
of names are replaced by "namex" ranging from "namel" to "name2" and so forth, where the num- bering indicates unique names in the utterance. Lo- cations are replaced with "locationI", "location2" and forth. Similarly, we replace credit card info and expiration with "creditcardx" and "credit- cardexpirationx" and email address with "emailad- dressx".
For pre-selection of which samples would be annotated by humans, we first restricted to which utterances would receive at least two la- bels under automated process above. (We note that not all utterances retained more than label by the end of our rounds of annotation.) Next, we split the dataset to be annotated by linguists contracted to our company for a first round of annotations.
Basic dataset statistics
We look at basic statistics of dataset quality. First we evaluate the average length (in characters, to- kens and sentences) of each phrase. Next, we com- pute the average type-token-ratio(TTR. introduced by Templin 1957 which is a measure of the lin- guistic complexity/quality? and the Flesch reading ease metric introduced by Kincaid et al. 1975
At this point, we ran preliminary unalysis and either removed or merged all labels with less than 100 examples. Merger of intents was based on discretion of which seemed com- mon, e.g., joining together different reasons to stop calling. or merging different intents expressing interest in the service. After this la- bel cleanup, we dropped any utterances which had no associated label.
The average TTR of 0.84 indicates the token repetition within phrases is limited, and median Flesch reading ease metric of 68.1 indicates the texts are easily readable by most English speakers of teenage age or older. See Table for precise figures.
Next we consider the emotional context of the text using standard T5 (Raffel et al., 2020) clas- sifier trained on the emotion dataset introduced by Saravia et al. 22018). We note that nearly 20% of examples are emotion laden, especially anger and sadness. This dataset may therefore also be useful for detection and analysis of emotion laden responses in short text conversations. See Table 10
Next, we rotated the assignment of utterances to the linguists that each could revise an- notations from the previous round under the revised label set. The intent label distribution visualization shown in Figure 3.
Privacy and anonymity
"We calculate the TTR via the Lexical Richness package hy Shen (2022)
All data was run through an automatic entity clas- sifier before even being presented to human anno- tators, removing all instances of PIID including name, phone number, address, email address and credit card number. The labelers then also manually reviewed given examples, removing any omissions found in the automatic redaction. All occurrences
(WWe use the median Flesch reading case metric, because we find the scores are not normalized on our dataset, has some large negative outliers that skew the distribution.
II 'See here for the emotion classifier: http://huginngfae...oommmmaeaeea 1oo//mmm8488/--bbse--iiee boase--inetund--emoon We threshold any examples assigned below a 0.9 score as neutral, and with an even higher threshold of 0.97 for joy and love due to oversensitivity of the model.

---

Emotion
Percentage of total
Example Utterance
Neutral Anger
81.34 %
Are you all able help all of my debts? I definitely terminated it twice now. If I find I'm still being charged somehow, I'm going to be very upset I'm not well at the moment and I' m off of work on sick leave from Covid call me tomorrow evening I' I'm resting my voice is horrible!.
8.19
Sadness
4.61 %
Fear
0.753 %
I would love fix my credit but I'm also afraid of spams and people stealing money.
Joy
4.90
Thank you for checking with me. I'm in good shape right now from your service to my past issues. have encouraged others to use your services also.
Table 10: Emotional content of Credit-16.
for more details.

---

Pie chart of CREDIT16 labels
phone call disconnected, answer
customer not
good good morning
6%
12
2.3%
labels
13.3%
168
interested
not happy
972
231
4.0%
service issue account issue
8.8%
am pm time _aay__yyy sel_dd_yllllleeeeee scheuuu
8.2%
601
640
don't oww anything
never, didn't call_out
credit report
work
6.1%
441
217
232
stop call stop calling
too expensive cancel, __cccunt service
sorry, sorry_apologize aapologizee thank you
8 8.0%
458
stop, stop calling don't owe aeyyannhinng
583
cancel
pm, day, call, schedule
3.9% 285
13. 7%
hi good hello good hello_geod morning
9.3% 682
5.1%
phone call, disconnected answer already. client
371
never didn't call, our
cost,
credit, report_score
sorry apologize thank
Figure 3: The distribution of CREDITI6 labels


Reviews:
[{"review_id": "PDhXnojEYM", "reviewer": "Reviewer_G3TU", "summary": "", "questions": "Please address the points under weakness along with the following questions:\n\n•\tCould you please explain how the slots are processed? as I found most of the discussions are based on intents. The examples from table-3 are also only intents. So curious to know, how slots are extracted.\n\n•\tQualitative studies over predicted outputs with examples (not only on numbers) will be better to understand", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer G3TU,\n\nThank you for the detailed review and recognizing the novelty and importance of our problem statement and approach. We appreciate your thoughtful comments and would like to address the concerns you raised.\n\n1. Dataflow Diagram: Your suggestion to include a dataflow diagram is well-received. While we believe our textual explanations are comprehensive, we acknowledge that a visual representation can provide additional clarity. The limited space in the current format was a constraint, but we will consider integrating such a diagram in extended versions or supplementary material.\n\n2. Supervised Nature of PCMID: While it's true that our method is supervised, it's important to note that most state-of-the-art methods for intent detection rely on supervised learning. The specificity of our method is not a drawback but a focus. Our method has been specifically tailored for multi-intent detection, which is an inherently complex task requiring clear intent boundaries. The supervised nature ensures accuracy in such complex scenarios.\n\n3. Data-collection and Annotation: The need for extensive data collection and annotation is not unique to our approach. Any supervised method requires annotated data. However, PCMID's strength lies in its efficiency and its ability to generalize over diverse datasets, as evidenced by our experiments across various public benchmarks and a real-world dataset.\n\n4. Comparison with LLMs: While direct comparisons with Language Model-based methods (LLMs) might provide additional insights, our main objective was to highlight the novelty and effectiveness of the PCMID framework. However, we acknowledge your point, and we will add the comparisons in the paper revision. \n\n5. Zero-shot Comparisons: The primary aim of our paper was to introduce and validate the PCMID framework. While zero-shot learning is an exciting direction, our focus was on establishing PCMID's effectiveness in a supervised setting. Nevertheless, we agree that zero-shot comparisons can offer additional robustness checks, and we will consider incorporating them in future studies.\n\n6. Performance improvement: Our experiments covered a range of datasets: MixATIS, MixSNIPS, FSPS, and a real-world private dataset, CREDIT16. Achieving consistent improvements across diverse datasets demonstrates the robustness of our approach.  \n    \n    Looking at Table 2, our model PCMID (in various configurations) consistently outperforms or matches previous state-of-the-art models in both accuracy and F1 score across almost all datasets. These improvements are not merely on the MixATIS dataset but are visible in others as well. \n\n    Also, When tested on CREDIT16, a real-world commercial dataset reflecting the genuine challenges of multi-intent detection, PCMID models showcased clear performance improvement. This underscores the model's potential real-world applicability and its capability to handle practical challenges.\n\n    While your observation about \"slight\" improvements might seem accurate at first glance, a deeper look reveals that these improvements are significant, consistent, and achieved with an efficient model structure. Our approach, PCMID, has demonstrated robustness across different datasets and settings, solidifying its position as a commendable advancement in the multi-intent detection task.\n\n7. Answer to Q1: Our paper's primary objective was to delve deep into the challenges and methodologies associated with multi-intent detection. As you noted, most of our discussions and examples are focused on intents. We choose to do so given the complexity and significance of detecting multiple intents in a single utterance. While slots and intents often co-exist in many dialogue systems, they address different challenges. Intents dictate the overall direction of an utterance or user input, while slots capture specific entities or details within that context. Given our paper's scope, we concentrated on the nuances of detecting multiple intents without diving into the intricacies of slot-filling. In particular, the real-world dataset CREDIT16 is a dataset without any slot annotations and we also do not use the slot annotations of MixAtis, MixSnips and FSPS.\n\n8. Answer to Q2: Section 6 \"Samples Analysis\", at line 564, indeed provides qualitative insights using concrete examples from Table 3. It not only identifies PCMID's correct predictions but also pinpoints where the model could be enhanced, giving a balanced view. The examples showcase PCMID's robustness in varied scenarios - from handling domain-specific terms to mitigating noise and incorrect human labels. This serves as a testament to our model's reliability in real-world situations.\n\n9. Typos Grammar Style And Presentation Improvements: We will fix the double quotation marks in our revision and make sure they stay consistent. Regarding your point on the redundant presentation in Section 3.3, the Section 3.3 introduces the core of our methodology: Multi-label Supervised Contrastive Learning. We intentionally outlined the evolution of Supervised Contrastive Learning (SCL) here to highlight the challenges and our unique solution, especially the utterance-label embedding.\n\n    Our aim was a cohesive narrative, emphasizing our novel contributions like the adaptation of Supervised Prototypical Contrastive Learning (SPCL) for multi-intent detection. While referencing existing methods, our main focus remains on our unique approach.\n\n    Moving foundational concepts like SCL to a separate background might disrupt this narrative flow. We've structured the section for clear understanding without fragmenting the content. However, we value your input and will reconsider our organization to maintain clarity and emphasis on our contributions.\n\n10. Reproducibility: We ensured transparency by detailing our model training statistics and hardware configurations in the \"Experiment Setup\" Section. These descriptions provide comprehensive insights into our experimental conditions. We have conducted a thorough analysis of the CREDIT16 dataset, detailed in the Appendix. To further the community's research endeavors, we're committed to releasing the dataset once the paper is finalized, ensuring researchers can access the exact data we worked with. Our parameter settings are empirical and based on best practices. If any parameter seemed underspecified, we are open to addressing it explicitly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tThe authors did a good job explaining the approach in detail, and the paper is written cleanly with very few minor typos – such as different types of double quotations.\n\n•\tThis is a very important problem statement and the conversational agents getting more complex day by day and the utterances becoming trickier. \n\n•\tThe method is novel.", "reasons_to_reject": "•\tIt’s better to have a dataflow diagram with examples from any of the datasets used, which helps to follow the paper better.\n\n•\tIt’s a supervised method, which is very specific to the tasks. It will be very hard to use it in any other tasks, as it requires an extensive amount of data-collection and annotation. No comparison is given with LLMs performance on these datasets or even zeroshot comparisons across the datasets (e.g., trained on mixATIS and tested on MixSNIPS). It helps to understand the robustness of this approach.  \n\n•\tPerformance improvements are very slight on all the datasets except MixATIS.", "questions_to_authors": "Please address the points under weakness along with the following questions:\n\n•\tCould you please explain how the slots are processed? as I found most of the discussions are based on intents. The examples from table-3 are also only intents. So curious to know, how slots are extracted.\n\n•\tQualitative studies over predicted outputs with examples (not only on numbers) will be better to understand", "paper_topic_and_main_contributions": "This paper talks about an important issue of real-world dialog systems, where the utterances can be crafted with multi-intent which may or may not be nested. This paper showcases a new way of doing multi-intent detection – PCMID that is a novel Multi-Intent Detection framework enabled by Prototypical Contrastive Learning under a supervised setting. The authors have clearly showcased their approach and explained the architecture. This work has been tested on 3 public datasets – MixATIS, MixSNIPS, and FSPS; and one private dataset CREDIT16. The authors have demonstrated slight to moderate improvement across all the datasets using this approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* double quotation marks are not consistent\n\n* Many concepts can be moved from section 3 to a separate Background section, such as SCL."}, "extra_text": {}}, {"review_id": "60E8WXFHcE", "reviewer": "Reviewer_NXkp", "summary": "", "questions": "When selecting the baseline, you chose a number of models that do joint training for intent detection and slot filling, so why not a few more models that only do multiple intent detection?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NXkp,\n\nThank you for your detailed feedback on our paper. We value and respect the time and effort you've taken in reviewing our work. Please allow us to address the concerns you've raised.\n\n1. Abstract's Description of Prior Work: The primary goal of an abstract is to provide a concise summary of the paper's main contributions. While we acknowledge the value of comparing our work to previous approaches in the abstract, the limited word count necessitated the omission of an extensive discussion of previous shortcomings. \n    \n    However, the main text does cover this in detail. We believe that presenting our main contribution was crucial for the abstract. That said, based on your feedback, we'll consider a revision to incorporate a brief mention of the uniqueness of our approach compared to previous works.\n\n2. Introduction's Summary of the Thesis:  Our introduction begins by highlighting the limitations of traditional single-intent detection, emphasizing the necessity for advanced multi-intent methods. We pinpointed specific shortcomings in prior research, such as Gangadharaiah and Narayanaswamy (2019)'s lack of fine-grained intent information and Wu et al. (2021)'s computational inefficiencies with dual BERT encoders. \n\n    We introduced PCMID's unique contributions: \n\n         a. Enhanced efficiency via a single BERT encoder. \n\n         b. Precise utterance-label attention mechanism. \n\n         c. An advanced learning approach, leveraging multi-label Supervised Prototypical Contrastive Learning.\n \n     We believe our introduction succinctly outlines PCMID's innovative nature against the backdrop of prior works. We're open to more detailed suggestions to enhance clarity.\n\n3. Redundancy and Methodology Section: Section 3.3 introduces the core of our methodology: Multi-label Supervised Contrastive Learning. We intentionally outlined the evolution of Supervised Contrastive Learning (SCL) here to highlight the challenges and our unique solution, especially the utterance-label embedding.\n\n     Our aim was a cohesive narrative, emphasizing our novel contributions like the adaptation of Supervised Prototypical Contrastive Learning (SPCL) for multi-intent detection. While referencing existing methods, our main focus remains on our unique approach.\n\n     Moving foundational concepts like SCL to a separate background might disrupt this narrative flow. We've structured the section for clear understanding without fragmenting the content. However, we value your input and will reconsider our organization to maintain clarity and emphasis on our contributions.\n\n4. Reporting Model Parameters for PCMID: Our PCMID model is architecturally simpler, as it incorporates a single BERT encoder, a multi-head attention block module, and a multi-label supervised contrastive learning module. The utterance-label encoder efficiently leverages a shared BERT encoder for both the utterance and intent labels, thereby reducing redundant parameters and computational overhead. \n\n     The main thrust of our approach is not just parameter reduction but achieving the task with streamlined model components. The focus of our model, as described in Section 3, emphasizes more on a modular architecture, enabling each component to effectively perform its task with the utmost efficiency. \n\n     While we detailed the functional aspects of each component, we understand the importance of quantifying the model's lightness through concrete metrics. We will ensure to include explicit parameter counts, computational requirements, and benchmark comparisons to provide a clearer picture of PCMID's advantages in terms of efficiency and lightness.\n\n5. Spelling issue in detail: We apologize for any oversights and will rectify them immediately. Such typographical mistakes, while regrettable, do not compromise the scientific validity of our research. Also, we aimed for clarity by reintroducing abbreviations. In our revisions, we will maintain consistency without redundancy.\n\n6. Answer to Q1: We focused on using intent annotations and labels exclusively for our study. The datasets we employed, such as Credit16, do not contain any slot annotations, and we intentionally avoided using the slot annotations available in MixAtis and MixSnips. Our primary criterion for selecting baselines was their compatibility with the type of annotations we were using. To the best of our knowledge, LABAN is the most recent state-of-the-art (SOTA) model that operates within this specific regime of annotations, and hence, we chose to compare against it. Our goal was to ensure a fair and direct comparison with models operating under similar constraints rather than including models that diverge significantly from our approach in terms of data requirements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The structure of the paper is clear and easy to read;\n2.The results of the experiment are detailed and the results have been analyzed in detail in various ways.", "reasons_to_reject": "1. Failed to write about the shortcomings of previous multi-intent detection work in the abstract, simply stating that multi-intent detection missions are closer to the real world, the abstract failed to excite me;\n2. The introduction does not summarize the work of the thesis well enough to indicate the innovative nature of the thesis;\n3. The paper is redundant, with large parts of previous work (e.g. 3.3, etc.) in the introduction to the methodology, which should focus on the problem to be solved and how it was solved;\n4. The introduction mentions that PCMID is lighter than previous frameworks, which should be reflected in the later paper by reporting the number of different model parameters and highlighting the advantages of PCMID;\n5. There are some errors in detail, such as: spelling of words, formulas without punctuation, and full names that appeared in the previous text do not need to be repeated and explained when abbreviations are used in the later text. It is recommended that you read the entire text carefully and check it thoroughly.", "questions_to_authors": "When selecting the baseline, you chose a number of models that do joint training for intent detection and slot filling, so why not a few more models that only do multiple intent detection?", "paper_topic_and_main_contributions": "Multi-intent detection is closer to the reality of complex situations and is more challenging than single-intent detection. This work proposes PCMID to model the semantics between individual utterance tokens and the intent label words, and achieves state-of-the-art performance on four datasets. In addition, this work constructs a multi-intent detection dataset", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZX90wMzgy3", "reviewer": "Reviewer_ZoeN", "summary": "", "questions": "Can you provide more analysis about figure 2?\nYou should compare your work with more recent SOTA to improve Soundness.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZoeN,\nThank you for your detailed feedback on our paper. We appreciate the time and effort you've taken in reviewing our work. Please allow us to discuss the concerns you've raised, and how we can address them.\n\n1. We will give the explicit formula of lbce as kindly suggested. We note however, that we have specified on line 393 that this symbol is the binary cross entropy loss, which is a standard formula in the field. We have defined fully the supervised contrastive learning loss l_scl, and supervised prototypical contrastive learning loss l_spcl, on lines 361 and 391 respectively, which are the lesser known formulas.\n\n2. Regarding hyperparameters, we commit again that we will share the full code publicly including all hyperparameters. We apologize for the current omission of τ, which is 0.5 and γ which is 0.5. These two parameters were held fixed for our experiments, and we did not optimize against these. We note that we have already specified the optimizer, learning rate, number of epochs, scheduler and batch size, which are more critical to the learning convergence.\n\n3. We have double-checked our Figure 1 and believe it is very clear and has high resolution to be viewed.\n\n4. For more insight on Figure 2, we have indicated on Line 558 that the label clusters are visually far tighter than the corresponding embeddings visualization in the previous work by Wu et. al, 2021. We could not include this figure directly due to space constraints, but it can be inspected here as Figure 3 on page 4892: https://aclanthology.org/2021.emnlp-main.399.pdf.\n\nOn comparisons:\n\nBenchmarks from all of the papers [1,2,3,4,5] suggested by ZoeN are not directly comparable in the scope of our work, because they require both intent and slot annotations - the latter of which are often unavailable in practice.\n\nIn contrast, we only use intent annotations and labels. In particular, Credit16 is a dataset without any slot annotations and we also do not use the slot annotations of MixAtis and MixSnips.\n\nAt least contemporaneous to our work, LABAN remains the prior SOTA in the regime of the annotations we use (as far as we are aware) and which we have compared against.\n\nOn prototypical contrastive learning:\n\nIn the context of the papers suggested by ZoeN (numbered as 6,7,8,9) that also employ contrastive learning, we further outline the novelty of our work:\n\nWe do not claim to have introduced any form of contrastive learning for the first time in intent detection. Rather, ours is the first paper to investigate supervised prototypical contrastive learning, where the intent labels are used as (trainable) prototypes for their classes.\nLiu et. al ([6])  uses a simple averaged embedding of each class instance as more naive prototype formulations, which is sensitive to the balance of data within an instance class. They do not leverage the class label text, nor any supervised prototypical contrastive learning, nor any mechanism comparable to the label-utterance attentive mechanism of PCMID.\n\nQin et. al ([7]) do not use any form of prototypical contrastive learning at all. Instead they use a more manual form of pairwise contrastive learning, with positive sample pairs generated by augmenting data with dictionary based word substitutions. And negative samples picked by a random selection scheme, with reuses of samples.\n\nLiang et. al ([8]) also do not use any form of prototypical contrastive learning - their formulation of contrastive loss uses standard pairwise construction of positive and negative examples. Also, their usage of label-aware learning is primarily to concatenate the slot labels alongwith the utterance for input representation, and to align representations across languages, since the slot labels are language invariant. They do not use a mechanism comparable to our attentive label-utterance mechanism.\n\nFinally, Chang and Chen ([9]) also do not use any form of prototypical contrastive learning at all, nor any label-aware mechanism.\nOf the papers mentioned, we would agree that Liu et. al [6] is most relevant for discussion since they use a form of PCL and thank the reviewer for this suggestion in particular, along with the others.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper conducts experiments on a real-word dataset.\n2. The paper is easy to follow.\n3. The paper provides the url of datasets.\n4. The paper provides introduction of baselines.", "reasons_to_reject": "1. You should compare your model with more recent models [1-5].\n2. Contrastive learning has been widely used in Intent Detection [6-9], although the tasks are not identical. I think the novelty of this simple modification is not suitable for EMNLP.\n3. You should provide more details about the formula in the text, e.g. $\\ell_{BCE}$ ,even if it is simple, give specific details.\n4. You don't provide the value of some hyper-parameters, such as τ.\n5. The Figure 1 is blurry, which affects reading.\n\n[1] Qin L, Wei F, Xie T, et al. GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Filling[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 178-188.\n\n[2] Xing B, Tsang I. Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 159-169.\n\n[3] Xing B, Tsang I. Group is better than individual: Exploiting Label Topologies and Label Relations for Joint Multiple Intent Detection and Slot Filling[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 3964-3975. \n\n[4] Song M, Yu B, Quangang L, et al. Enhancing Joint Multiple Intent Detection and Slot Filling with Global Intent-Slot Co-occurrence[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 7967-7977.\n\n[5] Cheng L, Yang W, Jia W. A Scope Sensitive and Result Attentive Model for Multi-Intent Spoken Language Understanding[J]. arXiv e-prints, 2022: arXiv: 2211.12220.\n\n[6] Liu H, Zhang F, Zhang X, et al. An Explicit-Joint and Supervised-Contrastive Learning Framework for Few-Shot Intent Classification and Slot Filling[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. 2021: 1945-1955.\n\n[7] Qin L, Chen Q, Xie T, et al. GL-CLeF: A Global–Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 2677-2686.\n\n[8] Liang S, Shou L, Pei J, et al. Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 9903-9918.\n\n[9] Chang Y H, Chen Y N. Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding[J]. arXiv preprint arXiv:2205.00693, 2022.", "questions_to_authors": "Can you provide more analysis about figure 2?\nYou should compare your work with more recent SOTA to improve Soundness.", "paper_topic_and_main_contributions": "The paper proposes PCMID, a novel Multi-Intent Detection framework enabled by Prototypical Contrastive Learning under a supervised setting. The PCMID model can learn multiple semantic representations of a given user utterance under the context of different intent labels in an optimized semantic space.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-"}, "extra_text": {}}, {"review_id": "SkuYMZFEYl", "reviewer": "Reviewer_XUFX", "summary": "", "questions": "- Are utterances and labels encoded jointly or separately? If jointly then this means that label representations cannot be re-used for new utterances.\n- Are MAB and the utterance encoder the same model?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive review and for acknowledging our problem statement and methodology. We are grateful for your insightful feedback and would like to address the points you've highlighted.\n\n1. We agree that using BERT directly for sentence embeddings is suboptimal. As a result, we have incorporated the Sentence-BERT model in the paper, specifically the S-PCMID base pooler/mean (referenced in lines 446-451). These models are fine-tuned on Sentence Semantics tasks, including NLI and STSB.  Additionally, we aim to compare our results with prior work (Cited in lines 438) that also employs a Bert-Based approach.\n\n2. While fine-tuning for new utterances for new datasets can potentially enhance performance, it's noteworthy that new label representations maintain good alignment even without such fine-tuning. This is primarily because they share the same encoding mechanism. Our results from the MixATIS dataset, which contains unknown labels in the test set, further substantiate this observation.\n\n3. It's crucial to note the distinction between the MAB and the utterance encoder. The utterance encoder is solely a Bert encoder. In contrast, the MAB consists of multiple attention block layers designed to align the utterance and label representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- PCMID outperforms existing approaches\n- The topic area is interesting and relevant to the dialogue systems community\n- Paper is easy to read and follow. Experiment settings are clearly defined and analysis is sufficient.", "reasons_to_reject": "- Contribution is incremental and relies heavily upon the initialized base encoder. BERT out-of-the-box is terrible at producing sentence embeddings and as such needs to be pre-trained. ", "questions_to_authors": "- Are utterances and labels encoded jointly or separately? If jointly then this means that label representations cannot be re-used for new utterances.\n- Are MAB and the utterance encoder the same model?", "paper_topic_and_main_contributions": "This paper focuses on the task of multi-intent detection for dialogue systems applications. The authors expand upon prior work that utilizes two BERT networks to capture the semantics of utterances and intent labels to calculate the similarity between them for multi-intent detection. The authors argue that prior work ignores the complexity of the intent label embedding space thus failing to capture the semantic relation between individual utterance tokens and the intent labels. To address, this limitation the authors propose Multi-Intent Detection through Supervised Prototypical Contrastive Learning (PCMID) which utilizes a single BERT Encoder for constructing utterance and label embeddings. Additionally, PCMID utilizes conservative learning to optimize the representation of both intent labels and the utterance samples in the embedding space.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the task of multi-label intent prediction and proposes SPCL - Supervised Prototypical Contrastive Learning, which is a mashup of PCL and SCL. 

Reviewers appreciated certain aspects of the paper which includes its presentation, description of the baseline approaches which the paper uses to build upon, and decent levels of analysis. 

Amongst concerns, primary points that were raised included:
- lack of relevant comparisons/discussions. For this point, the authors argue the recent works are not comparable due to slot-filling task. While that's true, a discussion of latest research is warranted so that reviewers can better evaluate the positioning of the paper. 
- lack of excitement. The paper largely fuses known ideas in the CV and NLP community. As such, reviewers were not highly excited about the proposed innovations.
- blurry figure. While a minor issue, the text in figures are indeed blurry which can (and should) be easily fixed using pdfs or other vector-based images.

The above issues are addressable and should be fixed in the revision as promised by the authors.
Decision: Accept-Findings