IntenDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery
Authors: Bhavuk Singhal
Ashim Gupta
Shivasankaran V P
Amrith Krishna
Venue: EMNLP 2023 Findings (2023)
Keywords: Intent Discovery
Intent Detection
Contrastive Learning
Modified Adsorption
Label Propagation

Abstract: Identifying intents from dialogue utterances forms an integral component of task-oriented dialogue systems. Intent-related tasks are typically formulated either as a classification task, where the utterances are classified into predefined categories or as a clustering task when new and previously unknown intent categories need to be discovered from these utterances. Further, the intent classification may be modeled in a multiclass (MC) or multilabel (ML) setup. While typically these tasks are modeled as separate tasks, we propose IntenDD a unified approach leveraging a shared utterance encoding backbone. IntenDD uses an entirely unsupervised contrastive learning strategy for representation learning, where  pseudo-labels for the unlabeled utterances are generated  based on their lexical features. Additionally, we introduce a two-step post-processing setup for the classification tasks using modified adsorption. Here, first, the residuals in the training data are propagated followed by smoothing the labels both modeled in a transductive setting. Through extensive evaluations on various benchmark datasets, we find that our approach consistently outperforms competitive baselines across all three tasks. On average, IntenDD reports percentage improvements of 2.32 %, 1.26 %, and 1.52 % in their respective metrics for few-shot MC, few-shot ML, and the intent discovery tasks respectively.

INTENDD: A Unified Contrastive Learning Approach for Intent Detection and Discovery
Bhavuk Singhal
Ashim Gupta
Uniphore Inc.
University of Utah
bhavuk. singhal@uniphore.ccmm com
ashimēcs, utah. edu
Shivasankaran V P
Amrith Krishna"
Stony Brook University
umrith.tech ask@amritt th. tech
svanajapandi@cs. htt.aaapanniiesssstttooorrrooo stonybrook. edu
Abstract
and semantically-similar concepts as separate in- tents (Zhang et al., 2021c). Overall, intent-related tasks typically are expected to be scalable and re- source efficient, to quickly bootstrap to new tasks and domains; lightweight and modular for main- tainability across domains and expressive to handle large, related often overlapping intent scenarios (Vulié et al., 2022; Zhang al., 2021a).
Identifying intents from dialogue utterances forms integral component of task-oriented dialogue systems. Intent-related tasks are typi- cally formulated either as a classification task, where the utterances classified into prede- fined categories or as a clustering task when new and previously unknown intent categories need to be discovered from these utterances. Further, the intent classification may be mod- eled in multiclass (MC) or multilabel (ML) setup. While typically these tasks are mod- eled as separate tasks, we propose INTENDD unified approach leveraging shared utterance encoding backbone. INTENDD uses an en- tirely unsupervised contrastive learning strat- egy for representation learning, where pscudo- labels for the unlabeled utterances are gener- ated based their lexical features. Addition- ally, we introduce a two-step procesrrcessing setup for the classification tasks using modi- fied adsorption. Here, first, the residuals in the training data are propagated followed by smoothing the labels both modeled trans- ductive setting. Through extensive evaluations on various benchmark datasets, we find that our approach consistently outperforms competitive baselines across all three tasks. On average, INTENDD reports percentage improvements of 2.32 %. 1.26 %. and 1.52 % in their respective metrics for few-shot MC, few-shot ML, and the intent discovery tasks respectively.
INTENDD proposes a unified framework for in- tent detection and discovery from dialogue utter- ances from ToD systems. The framework enables the modeling of various intent-related tasks such as intent classification, both multiclass and multil- abel, as well as intent discovery, both unsupervised and semi-supervised. In Intent detection (classifica- tion), we expect every class to have a few labeled instances, say 5 or 10. However, in intent discov- cry, not all classes are expected to have labeled instances and may even be completely unlabeled.
Recently, intent-related models focus more on contrastive representation learning, owing to the limited availability of labeled data and the pres- ence of semantically similar and fine-grained label space(Kumar et al., 2022: Zhang et al., 2021c). Similarly, common utterance encoder forms the backbone of INTENDD, irrespective of the task. The utterance encoder is learned by updating the parameters of a general-purpon pre-trained en- coder using a two-step contrastive representation learning process. First, we adapt general-purpose pre-trained encoder by using unlabelled informa- tion from various publicly available intent datasets. Second, update the parameters of the encoder using utterances from the target dataset, on which the task needs be performed, making the encoder specialize on the corpus. Here, we use both labeled and unlabelled utterances from the target dataset, where pseudo labels are assigned to the latter.
1 Introduction
Intents form a core natural language understand- ing component task-oriented dialogue (ToD) systems. Intent detection and discovery not only have immense utility but are also challenging due to numerous factors. Intent classes vary vastly from one use case to another, and often arise out of business needs specific a particular prod- uct or organization. Further, modeling require- ments might necessitate considering fine-grained
For intent classification, both multiclass and mul- tilabel, INTENDD consists of a three-step pipeline. includes training a classifier that uses the rep-
Work done while at Uniphore

---

resentation from the encoder as its feature repre- sentation, followed by two post-procsssing steps in a transductive setting. Specifically, a multilayer perceptron- based classifier is trained by stacking it on top of the utterance representation from our encoder. The post- processooss steps consider the tar- get corpus as graph in transductive setting. The first postprocessing step involves propagating the residual errors in the training data the neighbors. The second one further performs label smoothing by propagating the labels obtained from the pre- vious step. Both these steps are performed using Modified Adsorption, an iterative algorithm that enables controlling the propagation of information that passes through node more tightly (Talukdar and Pereira, 2010).
of K sentences, we compute the contrastive loss (Wu et al., 2020; Liu et al., 2021) as follows
K
exp(sim((h,hi))
Lucel
log
K
Σ E= 21=1 exp(sim( h2)/T)
(1)
For a sentence Ty we obtain a masked version of the sentence Ty, where few tokens of z; are randomly masked. Further, we dynamically mask tokens such that each sentence has different masked positions across different training epochs. In Laset, h4 the representation of the sentence z; and h, is the representation of the T₁. T is the tempera- ture parameter that controls the penalty to negative samples and sim(...) denotes the cosine similarity between two vectors. The final loss Lpretraining is computed as L-retrainiing = Leset + Xm Here, Lmtm is token level masked language modelling loss and À is a weight hyper-parameter.
INTENDD reports perfor- mance improvements compared to that of competi- tive baselines in all the tasks and settings we exper- imented with, including multiclass and multilabel classification in few-shot and high data settings; unsupervised and semi-supervised intent discov- setup for intent
Major contributions:
2.2 Corpus-specialized Representation Learning
ery. Our two-step classification leads statistically significant perfor- mance improvements to our base model. While ex- isting intent models focus primarily on better repre- sentation learning and data augmentation, we show that classical transductive learning approaches can help improve the performance of intent models even in fully supervised settings. Finally, we show that with careful construction of graph struc- ture in transductive learning setting in terms of both edge formation and edge weight formation can further improve our outcomes.
The pretraining step uses unlabelled sentences from publicly available intent datasets which should ide- ally expose pre-trained language model with utter- ances in the domain. Now, we consider contrastive representation learning using the target dataset on which the task needs to be performed.
Consider a dataset D with total of N unla- belled input utterances. Here, assuming 'D be completely unlabeled, we first assign pseudo labels to each of the utterances in D. Using the pseudo labels, we learn corpus-level contrastive represen- tation by using supervised contrastive loss (Khosla et al., 2020). The pseudo labels are assigned by first finding clusters of utterances by using commu- nity detection algorithm, 'Louvain' (Blondel et al., 2008). Community detection assumes the construc- tion of graph structure. We form connected weighted directed graph GD(VD, E, W), ) the input utterances in D form the nodes in Gp.. We identify lexical features in the form word-level n-grams.
2 INTENDD
INTENDD consists of two-step representation learning module, a classification module, and an intent detection module. We elaborate on each of these modules in this section.
2.1 Continued Pretraining
We start with a geurrae pre-trained model and use it as a cross-encoder for the continued pretraining (Gururangan et al., 2020). We start with standard general-purposss pre-trained model the encoder. We follow Zhang al. (2021c) for our pretraining phase where the model parameters are updated both using combination of token-level masked language modeling loss and a sentence- level sepe-super contrastive loss. For batch
We identify keyphrases that are representative of the target corpus on which the representation learn- ing is performed. The keyphrases are obtained by finding word-level n-grams that have high sociation with the target corpus, as compared the likelihood of finding those in other arbitrary corpora. Here, we obtain the pointwise mutual in- formation (PMD) of the n-grams in the target corpus, based on the likelihood the n-gram occurring in

---

the corpus, compared set of utterances formed via the union of the sentences in the target cor- pus and that in the corpora used during pretraining setup. Let P be the union of all the sentences in the corpora used in the pretraining step. Now, the PMI is calculated as
to iteratively maximize the quality function opti- mizes, generally modularity. While the approach may be started with any arbitrary partitioning of the graph, we start with each data point belonging to its own community (singleton communities). It then works itcratively in two phases. In the first phase, the algorithm tries to assign the nodes to their neighbors' community as long as that reas- signment leads to gain in the modularity value. (2)
PMI(kp, D) = log df (kp, DJ (kp, D) U log of (kp, P (DD
The second phase then aggregates the nodes within community and forms a super node, thus creat- ing a new graph where each community in the first phase becomes a node in the second phase. The process iteratively continues until the modularity value can no longer be improved.
Here, df (kp, D) is the count of utterances in D that contain the keyphrase kp. df (kp, IP U DI) is the frequency of the keyphrase from the combined collection D and P. Here, we only consider those keyphrases which is present at least five times in D. Moreover, the log frequency of the count of the keyphrase is also multiplied with PMI to avoid high scores for rare words (Jin et al., 2022). Further, the PMI value is multiplied by the square of the number of the words in the ngram so as to have higher scores for ngrams with larger values of (Banerjee and Pedersen, 2002). We validated this decision during preliminary experiments where we found that multiplying PMI with the square of the number of words generally worked better for the datasets considered in this work. That said, it's important to note that this design choice may vary in its necessity when applied to a different dataset, and its requirement should be established through empirical investigation.
Until now, we were assuming GD to be com- pletely unlabeled. However, we are yet to discuss two crucial questions. One, how to incorporate labeled information for an available subset of utter- ances in a semi-supervised setup. Here, we need to ensure that nodes belonging to the same true la- bel should not get partitioned into separate clusters, We merge those inputs with the same true label single node before constructing Gp, and initial- ize Louvain with the graph structure so obtained. The merging of the utterances with common la- bel into a single node trivially ensures that no two utterances of the same label get partitioned into different clusters. Hence, we ensure that no two nodes with the same true label are assigned with different pseudo labels. However, at this stage, the pseudo-labels are obtained purely for representa- tion learning. It is not intended to be representative of the real intent classes but rather simply a par- tition based on the keyphrases in the utterances. Finally, Using the pseudo labels obtained via Lou- vain, we learn corpus-level contrastive representa- tion by using supervised contrastive loss (Khosla et al., 2020). Here, during the representation learn- ing cach utterance is treated separately and we do not consider the merging that we performed for the community detection.
Now, the keyphrases are used to construct Gp. Two nodes have edges between them if they both have at least one common keyphrase. The edge weights are the sum of the keyphrase scores com- mon between two nodes. The weight matrix W is a N x N matrix representing the edge weights in the graph. W is row-normalized using min-max normalization, form of feature scaling. The graph Gp is then used to perform community detection using Louvain, modularity-baased community de- tection algorithm. Community membership used to form clusters of inputs. Here, all the nodes in GD that belong to the same cluster are assigned with common (pseudo)-label.
Keyphrase selection for constructing Gp: While we have list of n n-grams, along with their feature scores. Here, we employ recursive feature climination (RFE), greedy feature elimination ap- proach as our feature selection strategy. In RFE we start with a large set of features and greedily eliminate features, one at time. We start with the top k features and perform the community detec- tion using Louvain. We then start with the least
Louvain Method: is modularity- graph partitioning approach for detecting hierarchical community structure (Blondel et al., 2008). Here, each utterance is considered a node in a graph and the edge weights capture the strength of the rela- tion between node pairs. Louvain Method attempts

---

promising feature from our selected features and check if removing the feature leads an increase in the overall modularity of the graph, as com- pared to the modularity when the feature was in- cluded. Here, the number of nodes in GD remain the same, though the number of edges and their edge weights are dependent on the features. A sin- gle run of the Louvain algorithm has a complexity of Onn.logn), where is the number of nodes. So in worst case, the time complexity for graph con- struction is Onn.dloggn where d is the number of features. We perform the feature selection for a few fixed iterations. We incorporate some addi- tional constraints to keep track of for the feature selection, which are as follows: The graph needs to remain single connected graph and if the removal of feature violates it, then we keep the feature. Second, in all the tasks we consider, we assume the knowledge of the total number of intents. Hence a feature, whose presence, even if contributes pos- itively to modularity but results in increasing the gap between the total number of true intent classes and the number of clusters Louvain provides with it as the feature, then the feature is removed as well.
2.4 Intent Classification
Irrespective of whether multiclass or multilabel setup, our base classifier is a multilayer percep- tron comprising of a single hidden layer with non- linearity. It uses the utterance level representation, learned in $2.1 and $2.2, as its input feature, which remains frozen during the training of the classifier. The classifier is trained using cross-entropy loss with label smoothing (Vulié et al., 2022; Zhang et al., 2021c). The activation function at the output layer is set to softmax and sigmoid for multiclass and multilabel classification respectively.
Modified Adsorption (MAD) is a graph-based
suni-supe transductive learning approach (Talukdar and Crammer, 2009). MAD is a vari- ant of the label propagation approach. While label propagation (Zhu al., 2003) forces the unlabeled instances to agree with their neighboring labeled instances, MAD enables prediction on labeled in- stances to vary and incorporates node uncertainty (Yang et al., 2016). It is expressed as an uncon- strained optimization problem and solved using an iterative algorithm that guarantees convergence to local optima (Talukdar and Pereira, 2010; Sun et al., 2016). The graph typically contains few labeled nodes, referred to seed nodes, and a large set of unlabelled nodes. The graph structure can be explicitly designed MAD. The unlabelled nodes are typically assigned a dummy label. In MAD, a node actually is assigned label distribution than a hard assignment of label.
2.3 Intent Discovery
We perform intent discovery in both unsupervised and semi-supervised setups. Intent discovery is performed via clustering. Here, we start with the same graph construction as was used for Louvain in $2.2. The weight matrix W is row-normalized. Additionally, we obtain a similarity matrix A based on the cosine similarity between the utterance level encodings of two nodes. The encodings are ob- tained from the encoder learned in $2.2. We obtain a weighted average of the edge weights in W and A. Specifically, the weights for the average is ob- tained via grid search and selects the configuration that optimizes the silhouette score, an intrinsic mea- sure for clustering quality. The new graph will be referred to as Spred- With Spred, we perform Lou- vain again for intent discovery. The labeled nodes in semi- supervised setup would be merged as a single node before running Louvain. When new set of utterances arrive, these utterances added nodes in Spred- Their corresponding values in A are obtained based on their representation obtained from our encoder ($2.2). The corresponding val- ues in W are obtained based on the existing set of ngrams and no new feature selection is performed.
From random walk perspective, it can be seen as controlled random walk with three possible tions, each with predefined probabilities. all adding to one (Kirchhoff and Alexandrescu, 2011). The three actions involve a) continuing a random walk to the neighbors a node based on the transition matrix probability, b) stopping and returning the la- bel distribution for the node, and c) abandoning and returning an all-zero distribution or high probabil- ity to the dummy label. Each of these components forms part of the MAD objective in the form of seed label loss, smoothness loss across edges, and the label prior loss. The objective is:
K+1
ST SY, syi||²
arg min Y
1=1
Σ
Myy Yn
- + 02
4 R
111

---

Here M is the symmetrized weight matrix, Yι is the initial weight assignment the seed weight for label on node j. Y is the updated weight of the label / on node j. is diagonal matrix in- dicating seed nodes, and R is the regularization target for label / on node j. Here, we are assuming a classification task with K labels, and MAD intro- duces& dummy label as an initial assignment for the unlabeled nodes.
classification, the label with the maximum value for each node is predicted as the final class. In multilabel classification, all the labels with score above a threshold are predicted as the final labels.
3 Experimental Setup
We perform experiments for the three intent lated tasks Intent Discovery, Multiclass Intent Detection, and Multi-label Intent Detection. Here, we provide training and modeling details that are common to all three tasks and then mention task- specific details such as the baselines and evaluation metrics at appropriate sections.
We follow Huang et al. (2021) and perform two pootespocesssing steps. While the original ap- proach use label spreading (Zhou et al., 2003) for both steps, we replace it with MAD. Moreover, our graphs are constructed by combination of embedding- similarity and n-gram based sim- ilarity as described $2.3, i.e. Gpred- Both the postprocessing steps are applied on the same graph structure. However, the seed label initializations differ in both settings.
Pretraining Datasets. One feature of IN- TENDD is the unification of these three tasks via a common pretrained transformer backbone. This common pretraining step is performed CLINC-150 (Larson et al., 2019), BANK- ING77 (Casanueva et al., 2020), HWU64 (Liu et al., 2019a), NLU++ (Casanueva al., 2022), and StackOverflow (Xu et al., 2015). Following prior work on contrastive learning for intent detection by Zhang et al. (2021c), we additionally include TOP (Gupta et al., 2018), SNIP (Coucke et al., 2018), and ATIS (Tür et al., 2010). Table 4 shows some of the relevant statistics for the datasets.
Propagation of Residual Errors: We obtain the predictions from the base predictor, where each prediction is a distribution over the labels. Using the predictions, we compute the residual errors for the training nodes and propagate the residual errors through the edges of the graph. The unlabelled and validation nodes are initialized with zero value (or a dummy value), and the seed nodes are initialized with their residuals. Essentially Y initialized with a zero error for the training nodes with a non-zero residual error. With this initialization of Y we apply MAD on GMAD. The key assump- tion here is that the errors in the base prediction are positively correlated with the similarity neigh- borhood in the graph and hence the residuals need be propagated (Huang et al., 2021). Here, the residuals are propagated. Hence at the end of the propagation, each node has the smoothed errors as a distribution over the labels. To get the predictions after this step, the smoothed errors are added to predictions from the base predictor for each node.
Training and Modeling Details. We choose RoBERTa (Liu et al., 2019b) with the base con- figuration our common encoding backbone and pretrain with aforementioned datasets. For encod- ing the input utterances, we use cross-encoder architecture as detailed by (Mesgar et al., 2023). In this setup, the joint embedding for any pair of ut- terances (p, 7) -needed for contrastive learning for instance- is obtained by embedding it as "[CLS] [SEP] q" and the [CLS] representation is used as the representation for that pair. Mesgar et al. (2023) found that cross-encoder approach works much better than a Bi-encoder where any pair of utterances are independently embedded.
Smoothing Label Distribution The last step in our classification pipeline involves a smoothing step. Here, we make the fundamental assumption of homophily, where adjacent nodes tend to have similar labels. Here, Y is initialized follows: Seed labels are provided with their ground truth la- bels, the validation nodes and the unlabelled nodes are provided with initialized with the predictions after the error propagation step. With this initializa- tion, we perform MAD over 9 M AD- In multiclass
We perform all of our experiments using the tranformers library (Wolf et al., 2020) and the pytorch framework (Paszke et al., 2019). We train our models using the AdamW optimizer with learning rate set to 2c-5, warmup rate of 0.1, and weight decay of 0.01. We pretrain our model for 15 epochs, and thereafter perform task-specific training for another 20 epochs. All experiments are performed on a machine with NVIDIA 100 80GB and we choose the maximum batch size that

---

fits the GPU memory (= 96). We perform hy- peraparameter search for the temperature and lambda À over the ranges T E 0..1, 0.5), and À E 0.00,,0.03, 0.05)-
(via keyphrases), underscoring the importance of incorporating both these similarity measures. Furthermore, while our approach demonstrates notable enhancements across all configurations, these improvements are particularly pronounced when the amount of labeled data is limited, result- ing in an average increase of nearly 3% in accuracy for KIR values of 0% and 25%.
4 Experiments and Results
4.1 Intent Discovery
Datasets. We use three datasets for benchmark- ing INTENDD for Intent Discovery, namely, BANKING77, CLINC-150, and Stack Overflow. We assess the effectiveness of our proposed ID sys- tem two practical scenarios: unsupervised ID and semi-supervised ID. To ensure clarity, we in- troduce the term Known Intent Ratio (KIR), which represents the ratio of known intents in the train- ing data: the number of known intent categories ((Z&) divided by the sum of the known intent cate- gories and unknown categories ([zal + (. In this context, value of [& 0 corresponds to unsu- pervised ID, indicating the absence of any known intent classes. For semi-supervised ID, we adopt the approach outlined in previous works (Kumar et al., 2022; Zhang et al., 2021b), conducting exper- iments using three KIR values: (25%, 50%, 75%).
4.2 Multiclass Intent Detection
Datasets and Evaluation Metric. Following Zhang et al. (2021c), perform few-shot in- tent detection and select three challenging datasets for our experiments, namely, CLINC-150, BANK- ING77, and HWU64. We use the same training and test splits as specified in that paper, and use detection accuracy as our evaluation metric.
Due to space constraints, we provide detailed description of all baselines in the ap- pendix (please refer $A.1). We use the following baselines: RoBERTa-base (Zhang al., 2020), CONVBERT (Mehri al., 2020), CONVBERT Combined Mehri and Eric (2021), (Zhang et al., 2020), and CPFT (Zhang et al., 2021c, Contrastive Pre-training and Fine-Tuning) CPFT the cur- rent state-of-the-art employing self-supervised con- trastive pre-training on multiple intent detection datasets, followed by fine-tuning using supervised contrastive learning.
Baselines.
Evaluation Metrics. Following previous work (Zhang et al., 2021b), we report three metrics, namely Clustering Accuracy (ACC) (Yang al., 2010), Normalized Mutual Information (NMI) (Strehl and Ghosh, 2002), Adjusted Rand Index (ART) (Hubert and Arabie, 1985). All metrics range between O and 100 and larger values are more desirable.
Results. Table 2 shows the results of our exper- iments for multiclass intent detection. Our pro- posal, INTENDD demonstrates superior perfor- mance across all three sctups when compared to the baseline models in the 5-shot, 10-shot, and full data scenarios. In the 5-shot setting, exhibits an average absolute improvement of 2.47%, with the highest absolute improvement of 4.31% % observed in the BANKING77 dataset. Across all the datasets, IN- TENDD achieves average absolute improvements of 1.31% and 0.71% in the 10-shot and full data settings, respectively.
Baselines. We follow the recent work of Kumar et al. (2022) to select suitable baselines for unsu- pervised and semi-supervised scenarios. Due to space constraints, we detail these in the appendix.
Results. We report all the intent discovery results in table To begin with, it is important to highlight that our proposed method INTENDD consistently demonstrates superior performance surpassing all baseline techniques both unsupervised and semi- supervised settings across all three datasets. Specif- ically, in an entirely unsupervised scenario, SBERT- KM emerges as the most formidable baseline, yet approach significantly outperforms it. It should be noted that the fundamental distinction between INTENDD and SBERT-KM lies in our graph construction strategy for clustering. Our strategy relies on combination of semantic simi- larity (via embeddings) and n- a-gram based similarity
INTENDD currently does not incorporate any augmented data in its experimental setup. We do not compare our work with data augmenta- tion methods they are orthogonal to ours. One such example is that of ICDA (Lin et al., 2023), where large language model (OPT-66B) (Zhang et al., 2022) is used to augment the intent detection datasets for few-shot data settings. Nevertheless, we find that our method performs better than ICDA. We mention this comparison in the appendix B.

---

CLINC
BANKING
STACK OVERELOW
Method
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
BERT-KM
45.06
70.89
26.86
29.55
54.57
12.18
13.85
11.60
1.60
DAC
55.94
78.40
40.49
27.41
47.35
14.24
16.30
14.71
2.76
DCN
49.29
75.66
31.15
41.99
67.54
26.81
57.09
61.34
34.98
DEC
46.89
74.83
27.46
41.29
67.78
27.21
57.09
61.32
21.17
Unsupervised
SAE-KM
46.75
73.13
29.95
38.92
63.79
22.85
37.16
48.72
23.36
SBERT-KM
61.04
82.22
48.56
55.72
74.68
42.77
INTENDD (Ours)
63.87
83.12
51.76
58.74
75.91
47.88
79.32
73.88
62.49
CDAC+
64.64
84.25
50.35
48.71
69.78
35.09
74.30
74.33
39.44
DeepAligned
73.71
88.71
64.27
48.88
70.45
36.81
69.66
70.23
53.69
KIR 25%
DSSCCBERT
75.72
89.12
66.72
55.52
72.73
42.11
DSSCCSBERT
80.36
91.43
72.83
64.93
80.17
53.60
81.72
76.57
68.00
INTENDD (Ours)
83.11
92.32
76.31
67.50
76.79
57.85
84.82
78.93
71.64
CDAC+
69.02
86.18
54.15
53.34
71.53
40.42
76.30
76.18
41.92
DeepAligned
80.22
91.63
72.34
59.23
76.52
47.82
72.89
74.49
57.96
KIR 50%
DSSCCBERT
81.46
91.39
73.48
63.08
77.60
50.64
DSSCCSBERT
83.49
92.78
76.80
69.38
82.68
58.95
82.43
77.30
68.94
INTENDD (Ours)
84.57
93.91
78.42
71.16
84.56
63.17
85.01
79.14
72.49
CDAC+
69.89
86.65
54.33
53.83
72.25
40.97
75.34
76.68
43.97
DeepAligned
86.01
94.03
79.82
64.90
79.56
53.64
74.51
76.24
59.45
KIR 75%
DSSCCBERT
87.91
93.87
81.09
69.82
81.24
58.09
DSSCCSBERT
88.47
94.50
82.40
75.15
85.04
64.83
82.65
77.08
68.67
INTENDD (Ours)
90.99
96.29
83.62
77.08
87.39
68.69
85.47
77.12
72.90
Table 1: Results for Intent Discovery. First set of results are a completely unsupervised setting, while others when of the intent categories are known. KTR is used to represent the Known Intent Ratio. In all the experiments involving known intents classes, assume the proportion of labeled examples to be 10% (Kumar et al., 2022). Baseline results are taken from Kumar et al. (2022) and those marked with have not been reported in literature. DSSCC paper does not report results for DSSCCBERT on Stack Overflow, and we could not get access their code to independently run that model. The best results for each dataset and setting are marked in bold. We note that our proposed method consistently outperform recent baselines hy a significant margin.
Method
BANKING77
HWU64
CLINC150
5
10
Full
10
Full
5
10
Full
RoBERTa
74.65
84.67
93.08
76.75
83.42
90.97
88.27
91.21
96.46
CONVBERT
83.63
92.95
83.77
90.43
92.10
97.07
MLLM
83.99
93.44
84.52
92.38
92.75
97.11
MLM + Example
84.09
94.06
83.44
92.47
92.35
97.11
+ Combined
85.95
93.83
86.28
93.03
97.97
97.31
DNNC
80.40
86.71
80.46
84.72
91.02
93.76
CPFT
80.86
87.20
82.03
87.13
92.34
94.18
INTENDD-MLP (Ours)
82.17
88.70
93.63
81.27
85.32
92.89
91.34
93.66
96.92
INTENDD-EP (Ours)
83.25
88.96
94.18
83.17
86.35
93.31
92.70
92.24
97.93
INTENDD (Ours)
85.34
89.62
94.86
84.11
88.37
93.64
93.52
94.71
98.03
Table Results for Multiclass Intent Detection. We report intent detection accuracy for three data settings. We use the baseline numbers from (Lin et al., 2023). The best results for each dataset and setting marked in bold.
Modified Adsorption important for Intent Detection? INTENDD uses a pipeline of three
classification setups: one using the MLP, and two in a transductive setting using the Modified Ad-

---

sorption (MAD). We perform ablation experiments with these components and report results in the la- ble 2. We report results from three systems by progressively adding one component at a time. IN- TENDD-MLP denotes the results without using the two steps of Modified Adsorption, INTENDD-ER denotes the results with MAD but only the residual propagation step (i.e. without the label smoothing). We observe consistent performance improvements due to each of the components of the pipeline. No- tably, the label propagation step leads to more sig- nificant improvements and these gains are not only observed in the sho-shot setups but also in the fully data scenarios.
We find the results of our base predictor and our final classifier be statistically significant for all the settings of multi-class and multi-label intent detection using the t-test (p < 0.05).
5 Conclusion
In summary, this paper presents novel approach, INTENDD, for intent detection and discovery in task-oriented dialogue systems. By leverag- ing shared utterance encoding backbone, IN- TENDD unifies intent classification and novel in- tent discovery tasks. Through unsupervised con- trastive learning, the proposed approach learns representations by generating pseudo-labels based on lexical features of unlabeled utterances. Ad- ditionally, the paper introduces a two-step post- processing setup using modified adsorption for clas- sification tasks. While intent classification tasks typically focus contrastive representation Icarn- ing or data augmentation, show that two-step post-processee setup in a transductive setting leads to statistically significant improvements to our base classifier, often rivaling or at par with data aug- mentation approaches. Extensive evaluations on diverse benchmark datasets demonstrate the con- sistent improvements achieved by our system over competitive baselines.
4.3 Multilabel Intent Detection
Datasets and Evaluation Metric. Follow- ing Vulié et al. (2022), we use three datasets for multilabel intent detection: BANKING77, Mix- ATIS, and HOTELS subset is taken from NLU++ benchmark. MiXATIS consists of multilabel dataset synthetically obtained via concatenating single-label instances from the ATIS dataset. We do not perform experiments with InsuranceFAQ from that paper since it was an internal data. We report standard evaluation metrics: Fi and exact match accuracy (Acc). We report results on all datasets in two settings: low-data, and the high-data regimes, again replicating the experimental settings from Vulié et al. (2022).
Limitations
While our research provides valuable insights and contributions, acknowledge certain limitations that should be considered. In this section, we dis- cuss two main limitations that arise from our work.
Baselines. Our main baseline is the MultiCon- vFTT model proposed by Vulié et al. (2022) with two variants. MultiConvFiT (FT) where full fine- tuning along with the updating encoder parameters is performed. The second, more efficient alterna- tive MultiConvFiT (Ad) where an adapter is used instead of updating all parameters. Along with this, two other baselines from ConVFiT (Vulić et al., 2021) are adapted -DROB, and mini-LM. Please refer to Vulié et al. (2022) for more details on these methods.
First, a limitation of our proposed intent discov- ery algorithm is its reliance on prior knowledge of the number of intent clusters. This assumption may not hold in real-world scenarios where the under- lying intent structure is unknown may change dynamically. The requirement of knowing the ex- act number of intent clusters can be impractical and unrealistic, limiting the generalizability of our approach. However, we recognize that this limi- tation can addressed through modifications to our algorithm. Future investigations should explore techniques that allow for automated or adaptive de- termination of the number of intent clusters, mak- ing the approach more robust and applicable diverse real-world settings.
Results. The results of our experiments are shown in table 3. First, the results demonstrate consistent gains achieved by our method across all three datasets. Notably, in low-data scenarios, we observe an average increase of approximately 1% in F-scores. As anticipated, the performance enhancements are more substantial in low-data set- tings. However, it is noteworthy that our model out- performs MultiCon VFiT even in high-data setup.
The second limitation of our research lies in the reliance on the construction of a graph using ex- tracted keyphrases during the contrastive pretrain-

---

Method
BANKING77
HOTELS
MIXATIS
low-data
high-data
low-data
high-data
low-data
high-data
DRoB (Vulié et al., 2021)
70.6/31.0
/6.7
65.3 / 46.8
80.9 65.1
58.5/21.2
78.4 46.9
mini-LM (Vulić et al., 2021)
70.8/31.
86.7/58.1
64.2/46.0
80.3/65..
58.1/22.0
78.6 47.5
MultiConvFiT (Ad)
47.9
93.7/77.3
67.3
92.8/84.9
73.7 44.6
90.8 78.3
MultiConvFiT (FT)
81.9/49.1
94.3 90.5
70.2/51.1
75.5/514 93.4/84.0
91.5 / 88.1
INTENDD(Ours)
82.4/ 49.7
94.8/ 80,9
71.5 / 51.8
93.7 88.3
77.6/ 51.9
91.9/ / 81.5
Table 3: Results for Mult-label Intent Detection. We report both Fi score and Accuracy for all the settings. The first number in each cell is the Fi score and the second number is the accuracy. Vulić et al. (2022) proposed two variants for MultiConVFiT one with full fine-tuning (FT) and another with adapters (Ad). The ConvFiT model proposed by Vulié al. (2021) has been adapted for multi-label settings with DistilRoBERTa (DRoB), and mini-LM as backbones. The best results for each dataset and setting are marked in bold. To establish the statistical significance of our results, we performed the paired t-test between INTENDD and MultiConvFiT (FT) and found the p-value in all cases to be < 0.05.
ing step, which is a common requirement across all three tasks explored in our study. While this graph construction step facilitates the representa- tion learning process, it introduces a constraint on the flexibility of modifying the graph structure. Even minor modification to the graph construc- tion would necessitate retraining all systems, which can be time-consuming and resource-intensivee Currently, we mitigate the need for covering new ut- terances (with no overlapping keyphrases) by sim- ply relying on similarity from the encoder represen- tation itself. However, it still may still lead to con- cept drift over time, and the representation might need to be updated by retraining all the modules in INTENDD. In future work, intend to explore alternative approaches that offer more flexibility in graph construction, allowing for easier modifica- tions without the need for extensive retraining. By addressing this limitation, we aim to enhance the adaptability and scalability of our framework.
Processing for Conversational AI, pages 38-45, On- line. Association for Computational Linguistics.
Inigo Casamueva, Ivan Vulié, Georgios Spithourakis, and Pawet Budzianowski. 2022. NLU++: A multi- label, slot-rich, generalisable dataset for natural lan- guage understanding in task-triented dialogue. In Findings of the Association for Computational Lin- guisties; NAACL 2022, pages 1998-2013, Seattle, United States. Association for Computational Lin- guistics.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shim- ing Xiang, and Chunhong Pan. 2017. Deep adaptive image clustering. In Proceedings of the IEEE in- ternational conference on computer vision, pages 5879-5887.
A Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thihaut Lavril, et al. 2018. Snips voice plat- form: an embedded spoken language understanding system for privat-by-desige voice interfaces. arXiv preprint arXiv:1805. 10/90.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In I Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171 1171-118,, Minneapolis, Minnesota. Association for Computational Linguistics.
References
Satanjeev Banerjec and Ted Pedersen. 2002. An adapted lesk algorithm for word sense disambiguation using wordnet. In Proceedings of the Third International Conference on Computational Linguistics and Intelli- gent Text Processing, pages 136-145.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Ku- mar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representa- tions. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2787-2792 Brussels, Belgium. Association for Computational Linguistics.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lam- biotte, and Etienne Lefebvre. 2008. Fast unfold- ing of communities in large networks. Journal of Statistical Mechanics: Theary and Experiment, 2008(10):P0008..
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulié. 2020. Efficient intent detection with dual sentence encoders. Pro- ceedings of the 2nd Workshop on Natural Language
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In

---

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
lexical and sentence encoders. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442-1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. 2021. Combining label propa- gation and simple models out-performs graph neural networks. In International Conference on Learning Representations.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural lan- guage understanding services for building conversa- tional agents. arXiv preprint (rXivv19033055555
Lawrence Hubert and Phipps Arabic. 1985. Comparing partitions. Journal of classification, 2:193-218.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini urXiv:1907.11 1692.
Yiping Jin, Dittaya Wanvarie, and Phu TV Le. 2022. Learning from noisy out-of-domain corpus using dataless classification, Natural Language Engineer- ing, 2...
MacQueen. 1967. Classification and analysis of mul- tivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pages 281-297, University of California Los Angeles LA USA.
Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in neural information processing systems, 33:18661-1866
Shikib Mehri and Mihail Fric. 2021. Example-driven intent prediction with observers. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies. pages 2979-2992, Online. Association for Computational Linguistics.
Katrin Kirchhoff and Andrei Alexandrescu. 2011. Pho- netic classification using controlled random walks. In Twelfih Annual Conference of the International Speech Communication Association.
Rajat Kumar, Mayur Patidar, Vaibhav Varshney, Lovekesh Vig. and Gautam Shroff. 2022. Intent de- tection and discovery from user logs via deep semi- supervised contrastive clustering. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1836-1853, Seattle, United States. Association for Computational Linguistics.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tiu 2020. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. arXiv preprint 1eXiv:2009.13540
Mohsen Mesgar, Thy Thy Tran, Goran Glavaš, and Iryna Gurevych. 2023. The devil is in the details: On models and training regimes for few-shot intent classification. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics. pages 1846-1857, Dubrovnik, Croatia. Association for Computational Linguistics.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, 16, Hong Kong, China. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style. high- performafrcce deep learning library. Advances in neural information processing systems, 32.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP (ICNLP,, pages 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages -360-8367.
Yen-Ting -ing Lin, Alexandros Papangelis, Scokhwan Kim, Sungjin Lee. Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani- Tur. 2023. Selective in-context data augmentation for intent de- tection using pointwise V-information. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1463-1476, Dubrovnik, Croatia. Association for Computational Linguistics.
Alexander Strehl and Joydeep Ghosh. 2002. Cluster ensembles- a knowledge reuse framework for com- bining multiple partitions. Journal of machine learn- ing research, 3(Dec):583661 617.
Ming Sun, Aasish Pappu, Yun-Nung Chen, and Alexan- der I Rudnicky. 2016. Weakly supervised user intent detection for multi-domain dialogues. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 91-97. IEEE
Fangyu Liu, Ivan Vulié, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into universal
Partha Pratim Talukdar and Koby Crammer. 2009. New regularized al algorithms for transductive learning. In

---

Machine Learning and Knowledge Discovery in European Conference, ECML PKDD 2009, Bled, Slovenia, September 7-11, 2009, Pro- ceedings, Part II 20, pages 442-457. Springer.
multaneous deep learning and clustering. In interna- tional conference on machine learning, pages 3861- 3870. PMLR.
Databases:
Yi Yang. Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. 2010. Image clustering using local discriminant models and global integration. IEEE Transactions on Image Processing, 19( 19)):)2761- 2773.
Partha Pratim Talukdar and Fernando Pereira. 2010. Ex- periments in graph-based semi -uperviie learning methods for class-instance acquisition. In Proceed- ings of the 48th Annual Meeting of the Association for Computational Linguistics. pages 1473-1481, Up- psala, Sweden. Association for Computational Lin- guistics.
Zhilin Yang, William Cohen, and Ruslan Salakhudi- nov. 2016. Revisiting semi supi-siee learning with graph embeddings. In Proceedings of The 33rd In- ternational Conference on Machine Learning, vol- ume 48 of Proceedings of Machine Learning Re- search, pages 40-48, New York, New York, USA. PMLR.
Gökhan Tür, Dilek Z. Hakkani-Tünn and Larry Heck. 2010. What is left to be understood in atis? 2010 IEFE Spoken Language Technology Workshop, pages 19-24.
Ivan Vulić, lñigo Casanueva, Georgios Spithourakis. Avishek Mondal, Tsung-HHseen Wen, and Pawel Budzianowski. 2022. Multi-label intent detection via contrastive task specialization of sentence encoders. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7544-555 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021a. TEXTOIR: An integrated and visualized platform for text open in- tent recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing: System Demon- strations, pages 167-174, Online. Association for Computational Linguistics.
Ivan Vulié, Pei-Hao Su, Samuel Coope, Daniela Gerz, Pawel Budzianowski, Iñigo Casanueva, Nikola Mrkšić, and Tsinn Wen. 2021. ConvFiT: Con- versational fine-tuning of pretrained language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1151-1168, Online and Punta Cana, Dominican Re- public. Association for Computational I Linguisties.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. lb. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 14365- 14373.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021c. Few-shot intent detection via contrastive pre-training and fine-tuning, In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1912, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao. Sylvain Gugger, Mariama Drame Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by trans- ferring natural language inference. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5064- 5082, Online. Association for Computational Lin- guistics.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15666.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00 01068.
Junyuan Xie. Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analy- sis. In International conference on machine learning. pages 478 487. PMLR.
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural net- works. In Proceedings of the 1st Workshop on Vec- for Space Modeling for Natural Language Process- ing, pages 62-69, Denver, Colorado. Association for Computational Linguistics.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. 2003. Learning with local and global consistency. In Advances in Neural In Information Processing Systems, volume 16. MIT Press.
Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. 2003. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the Twentieth International Conference on International
Bo Yang, Xiao Fu, Nicholas Sidiropoulos, and Mingyi Hong. 2017. Towards k-means-friendly spaces: Si-

---

Name
[D]
#Intent
Tasks
IntenDD
CPFT
CLINC- 150
18,200
150
MC, ID
10
5
10
BANKING77
10,162
77
MC, ID
HWU64
10,030
64
MC
BANKING77
0.38
0.29
0.20
0.48
NLU++
3,080
62
ML
MIXATIS
20,000
18
ML
HWU
0.35
0.18
0.51
0.25
STACK OVERFLOW
20,000
20
ID
CLINCI50
0.32
0.21
0.39
0.18
Table 4: Dataset Statistics for the three Intent Identifi-
cation tasks explored in this work. Second column lists the number of intent classes in each of the datasets, Key: MC - Multiclass (Single label), ML Multi-label, ID Intent Discovery.
Table 5: Standard Deviation across different runs for Few-Shot Intent Detection. We observe that, compared CPFT, our method has lower variance across most settings.
Conference on Machine Learning, ICML'03, page 912 919. AAAI Press.
tection. The first bascline, RoBERTa-base, uti- lizes RoBERTa its base model, supplemented with linear classifier on top for classifica- tion purposes. Another baseline, CONVBERT, involves fine-tuning BERT using a vast open- domain dialogue corpus consisting of 700 mil- lion conversations (Mehri et al., 2020). Fur- thermore, CONVBERT + Combined, an intent detection model based on CONVBERT, adopts example-driven training with similarity match- ing and transformer attention observers, along with task-adaptive self-supervised learning using masked language modeling on intent detection datasets. The term "Combined" refers to the opti- mal MLM--Exappe=+Obsereess setting described in Mehri and Eric (2021). Another baseline model, DNNC (Discriminative Nearest-Neighbor Classifi- cation) (Zhang et al., 2020), employs discrimina- tive nearest-neighhor approach, matching training examples based on similarity and employing data augmentation during training. Additionally, en- hances performance through pre-training on three natural language inference tasks. Finally, CPFT (Contrastive Pre-training and Fine-Tuning) (Zhang et al., 2021c) represents the current state-of-the art in few-shot intent detection, employing self- supervised contrastive pre-training on multiple in- tent detection datasets, followed by fine-tuning ing supervised contrastive learning.
A Experimental Details
A.1 Baseline Description
Intent Discovery In the unsupervised setting, the first two baselines use K-means algorithm (Mac- Queen, 1967) on top of sentence embeddings from BERT (Devlin et al., 2019) and SBERT (Reimers and Gurevych, 2019) to cluster ances (BERT-KM, SBERT-KM respectively). DEC (Xie et al., 2016) a two step deep clustering approach involving a Stacked Autoencoder (SAE) along with confidence based cluster assignment. SAE-KM uses K-means with SAE (Xie et al., 2016), DCN (Yang et al., 2017) is method that performs dimensionality reduction and clustering using a joint objective function, and DAC (Chang al., 2017) treats the clustering problem pair- wise binary classification problem to learn cluster centers.
utter-
For the semi-supervised case, we use CDAC+ (Lin et al., 2020), in which the pairwise constraints from the labeled examples are incorporated into the clustering problem. DeepAligned (Zhang et al., 2021b) uses labeled data to generated pseudo labels well as pretrain BERT model followed by K-means clustering. Finally, we compare our method with a very recent method DSCC (Kumar et al., 2022) where the authors propose an end-to-end contrastive clustering algorithm to jointly learn cluster centers and utterance representations via combination of supervised and self-supervised methods. We report results with two backbone models used in the paper, BERT and S-BERT.
B Additional Results
Variance in Few-shot Intent Detection. In the few-shot settings, generally report lower vari- than CPFT, the system with the second-best results consistently. Table shows the standard deviation for INTENDD and CFPT, where CPFT has a lower variance than INTENDDonly two out of six settings.
Multiclass Intent Detection this study, we consider several baseline models for intent de-

---

B.1 Comparisong with a recent data augmentation strategy ICDA
INTENDD currently does not incorporate any aug- mented data in its experimental setup. We do not compare our work with data augmentation methods they are orthogonal to ours. One such example is that of ICDA (Lin al., 2023), where large language model (OPT-66B) (Zhang et al.. 2022) is used augment the intent detection datasets for few-shot data settings. Nevertheless, we find that our method performs better than ICDA.
INTENDD outperforms all the settings of ICDA in both 5-shot and Full data settings. In 10-shot settings, while INTENDDreports the best results HWU64, the largest configurations of ICDA ports better accuracy for the other two datasets. The largest configuration uses 128 times more aug- mented data than the available supervised data to report the best results. Overall, ICDA reports an accuracy of 89.79 % and 94.84 % on BANKING77 and CLINC150 respectively which is 0.17 % and 0.13 % more than INTENDD.
B.2 Computing Infrastructure Used
All of our experiments required access to GPU accelerators. We ran experiments three ma- chines: Nvidia Tesla A100 (80 GB VRAM), Nvidia Tesla V100 (16 GB VRAM), Tesla A100 (40 GB VRAM).


Reviews:
[{"review_id": "EkSypK4HTW", "reviewer": "Reviewer_cGUN", "summary": "", "questions": "1. Statistics analysis to support the hypothesis that every keyphrase is present at least five times in D\n2. More detailed analysis on why the proposed method fails to outperform DSSCCSBERT in BANKING and in KIR=25% setting", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and the thoughtful review. \n\n**Q- \"Statistics analysis to support the hypothesis that every keyphrase is present at least five times in D\"**\n**Response:** As stated in lines 189-191, our choice to incorporate key phrases with a minimum support of five from 𝒟 is a deliberate design choice rather than a hypothesis. This selection also serves as both a filtering and pre-processing measure. To eliminate any potential ambiguity, we will revise the phrasing of the aforementioned statement in the manuscript.\n\n**Q- \"More detailed analysis on why the proposed method fails to outperform DSSCCSBERT in BANKING and in KIR=25% setting\"**\n**Response:** Please note that even in this specific case, our method still significantly outperforms DSSCCSBERT for the BANKING dataset on 2/3 metrics (i.e. ACC, ARI). We will include a more detailed analysis of this specific case in the final draft. \n\n\"Reasons to reject\n1. More evaluations for backbones other than RoBERTa are necessary such as BERT to prove the general improvement of the method for most pre-trained models.\n2. More evaluations for backbones of more size are preferred.\"\n\nAlthough it is valuable to explore assessments using alternative backbones (varying models or sizes), our primary objective in this study is to ensure a direct comparison with the latest state-of-the-art models. Consequently, we meticulously adhere to the experimental parameters outlined in those baseline works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is well-defined and well address the problems. The paper introduces novel ideas about the construction of graph structures for pseudo label assignment and the method Modified Adsorption to perform graph-based semi-supervised transductive learning approach as post-processing of intent classification.\n2. Conduct comprehensive evaluations\n3. Evaluations on various benchmark datasets prove that the proposed method consistently outperforms competitive baselines across all three tasks.", "reasons_to_reject": "1. More evaluations for backbones other than RoBERTa is necessary such as BERT to prove the general improvement of the method for most pre-trained models.\n2. More evaluations for backbones of more size is preferred.", "questions_to_authors": "1. Statistics analysis to support the hypothesis that every keyphrase is present at least five times in D\n2. More detailed analysis on why the proposed method fails to outperform DSSCCSBERT in BANKING and in KIR=25% setting", "paper_topic_and_main_contributions": "This paper uses an unsupervised contrastive learning strategy for representation learning, where pseudo labels for the unlabeled utterances are generated based on lexical features. This paper also introduces a two-step post-processing setup for the classification tasks using modified adsorption. The residuals in the training data are propagated followed by smoothing the labels both modeled in a transductive setting.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. line 66: aA\n2. line 145: Here\n3. line 179\n4. line 190: ngram to n-grams?\n5. line 290"}, "extra_text": {}}, {"review_id": "LlNzBukIcR", "reviewer": "Reviewer_MW7X", "summary": "", "questions": "* Please clarify if the intent discovery baselines were also evaluated under the \"known number of intent classes\" assumption\n\n* Please elaborate on the graph construction complexity.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and the thoughtful review. \n\n**Q- \"Please clarify if the intent discovery baselines were also evaluated under the \"known number of intent classes\" assumption\"**\n\n**Response:** As mentioned in L474-477 of the submitted draft, we follow the exact same setting as Kumar et al. (2022), the current state of the art in intent discovery (without data augmentation). The same setting is followed by several other recent works such as Zhang et al., (2021a).\n\nWe primarily consider three different values for the Known intent ratio (25%, 50%, 75%) and each of them has 10 % of its samples labeled. For the unsupervised setup, no labeled information is used. Further, we assume the total number of intent classes is known apriori. This is a standard evaluation setup for intent discovery. For instance, *Zhang et al. (2022)*, Kumar et al. (2022), and  Zhang et al., (2021a) follow the same setup. In Kumar et al. (2022), they consider two scenarios. In the second scenario, they do not assume that the total number of intent classes is known beforehand. However, we have considered the results based on a comparable setting with ours, where the total number of classes is known beforehand.  Similarly, if we consider *Zhang et al. (2022)*, their experiments are solely based on intent discovery and they only consider the setting where the total number of classes is known beforehand. Since *Zhang et al. (2022)* use data augmentation, it is orthogonal to ours (please refer to lines 544 to 553) and hence we do not compare it with our approach.\n\nZhang et al. (2022) - https://aclanthology.org/2022.acl-long.21.pdf\n\n**\"Please elaborate on the graph construction complexity.\"**\n\nFor graph construction, the nodes remain constant and the edges are constructed by using a feature selection approach. For each iteration of the feature selection approach, we perform the Louvain algorithm. A single run of the Louvain algorithm has a complexity of $O(n.log n)$, where $n$ is the number of nodes. It is known to be a fast community Detection approach. We run that for $d$ iterations in the worst case, where $d$ is the number of features. Altogether, the time complexity for graph construction is $O(n.d.log n)$ . Now, The aforementioned time complexity is comparable to that of the K-Nearest neighbors (K-NN) algorithm. For K-NN the complexity is  $O(n.d.k)$, or a $O(nd + kn)$ for a faster version, where $k$ is the number of neighbors.  Moreover, the largest dataset we currently consider includes 20,000 nodes where a single run of Louvain is often completed in less than 10 seconds in an Intel i-5 CPU and 16 GB Ram, in terms of the wall clock time. Please note that in case of larger network sizes, say those with $10^7$ nodes ore more, we may alternatively use the Leiden algorithm to improve upon the speed. However, that comparison does not fall within the scope of the current work.\n\nWe will add these details to the paper.\n\nLeiden - https://www.nature.com/articles/s41598-019-41695-z", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper introduces several novel ideas, especially around the construction of graph structures for pseudo label assignment, and applying Modified Adsorption -- a graph-based semi-supervised transductive learning approach as post-processing of intent classification.\n\n* Empirical evaluation were carried out on multiple datasets to benchmark intent clustering and classification performance of IntenDD.", "reasons_to_reject": "* A key limitation of the work is the assumption of known number of intent classes during graph construction. Especially for the intent discovery evaluation, I am not sure if the IntenDD experiments reported in Table 1 are all under this assumption, and can be directly comparable to the cited baselines. While the authors acknowledged this as a limitation of the work, it would require further clarification on the intent discovery experiment setting to see if the reported SOTA is in fair comparison with baselines. \n\n* Graph construction procedure described in section 2.2 seems to be very compute-intensive and difficult to scale with the size of dataset D.", "questions_to_authors": "* Please clarify if the intent discovery baselines were also evaluated under the \"known number of intent classes\" assumption\n\n* Please elaborate on the graph construction complexity.", "paper_topic_and_main_contributions": "This paper presents IntenDD as a unified modeling approach for both intent discovery/clustering and intent detection/classification tasks. Both tasks in IntenDD share a common continually pretrained utterance encoder that is trained with the combination of contrastive and MLM losses, and then updated using utterances from the target dataset.  Further post-processing in the classification model through transductive learning on a graph structure. Evaluation on intent discovery and multiclass/multilabel intent detection benchmark tasks shows new state-of-the-art accuracy.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The paper should"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "p1. L66. aA -> a\np2. L166. form a connected a weighted directed graph -> form a connected weighted directed graph\np6. L514. zhang-etal-2021-shot <- reference format\np6. L525. zhang-etal-2020-discriminative <- reference format\np7. L570. fully data setups -> full data scenarios\np8. L579. AXIS -> ATIS"}}, {"review_id": "F0lty6TgAO", "reviewer": "Reviewer_DQWw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful review. \n\n**Re: PMI for identifying keyphrases:**\n1. We performed min-max normalization for each row – a form of feature scaling as a preprocessing step. For a given node, we wanted to capture the relative importance of other nodes in the graph and hence performed row-wise min-max normalization. During our initial experiments, we observed that we get better partitions when using normalized values over unnormalized values (no test data was used during these initial experiments). We will incorporate the aforementioned details in the draft to avoid any possible ambiguity.\n2. In our current approach, we used ngrams of up to 5-gram for all the datasets we reported. Moreover, we included an extended-lesk style approach, i.e. multiplying the square of the number of words in a phrase with its PMI, to favor longer keyphrases. In our initial experiments, we observed that we obtained better performance across all the datasets reported in this work when using keyphrases of up to 5-grams and the extended lesk style product (validated on development data).  That said, for the keyphrase scoring approach, we acknowledge that the design decisions may vary for new datasets depending on the nature of those corpora and need to be finalized empirically. Moreover,  the final set of keyphrases is decided not merely using these scores but also based on a feature selection approach (Lines 259-286). Our focus, in this work, is to show that a simple graph construction strategy based on lexical features works well across multiple datasets and we use PMI based scoring approach to capture this intuition. We will add the aforementioned details in the draft to avoid any possible ambiguity.\n3. Good point. In fact, one of the reasons to use counts of keyphrases is to ensure the relative efficiency of the graph construction procedure. This method requires one pass of D and P for collecting n-gram counts of keyphrases and another for calculating the PMI scores. For graph construction, the nodes remain constant and the edges are constructed by using a feature selection approach. For each iteration of the feature selection approach, we perform the Louvain algorithm. A single run of the Louvain algorithm has a complexity of $O(n.log n)$, where $n$ is the number of nodes. It is known to be a fast community Detection approach. We run that for $d$ iterations in the worst case, where $d$ is the number of features. Altogether, the time complexity for graph construction is $O(n.d.log n)$ . Now, The aforementioned time complexity is comparable to that of the K-Nearest neighbors (K-NN) algorithm. For K-NN the complexity is  $O(n.d.k)$, or a $O(nd + kn)$ for a faster version, where $k$ is the number of neighbors.  Moreover, the largest dataset we currently consider includes 20,000 nodes where a single run of Louvain is often completed in less than 10 seconds in an Intel i-5 CPU and 16 GB RAM, in terms of the wall clock time. Please note that in case of larger network sizes, say those with $10^7$ nodes ore more, we may alternatively use the Leiden algorithm to improve upon the speed. However, that comparison does not fall within the scope of the current work. Further, obtaining and scoring keyphrases does not take more than 14 minutes of wall clock time in a CPU with the aforementioned configuration. For larger datasets, we may even use parallelization that can substantially reduce the wall clock time computation. \n4. Duplicate of Point 2.\n\n**Constructing similarity matrix A**\n\nThe dimension of the utterance encoding for a node in the graph was 768. As previously mentioned, the time complexity for our approach is still close to that of a k-NN-based approach, which is widely used in Intent-related tasks in ToD systems.  Modern scientific libraries like pytorch and scipy handle such cases with little to no difficulty. \n\nThanks for pointing out the need for additional information on weighted averaging. We obtained the weights via parameter search (grid search) and selected the configuration that optimizes the silhouette score, an intrinsic measure for clustering quality.  We will add the necessary details regarding our weighted average process in the draft. \n\n**Limitation in case of new instances**\n\nGreat point. As we acknowledge in the Limitations (L645-651), MAD is a transductive learning approach. Here, we need to update the graph to add new instances which requires re-training the system. As far as generalization is concerned, the neighbors for a newly inserted node (new instance) are determined using the embedding similarity and keyphrae-based overlap. Like in other works on intent classification, we employ contrastive representation learning to learn our embedding. Hence, unseen instances can still leverage their embedding level similarity with instances used during the training for MAD. As stated in lines 655-657, our approach may lead to concept drift over time, and the representation might need to be updated by retraining.\n\n**Re: Evaluation**\n\nTo be consistent with the baselines, we choose our datasets from the recent works for the three tasks we considered: Kumar et al. (2022) for Intent Discovery, Lin et al. 2023 for Intent Detection, and Vulic et al. 2022 for multi-label ID.  \n\nRegarding the results in Table 3, we performed the paired t-test to establish the significance of results (p-value in all cases was < 0.05). We will mention this in the caption of table 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength 1: The experimentation is thorough, compares to strong baselines and the paper performs a good analysis.\n\nStrength 2: Detailed explanation of the approach.\n", "reasons_to_reject": "W1: In section 2.2, the authors describe the process of using PMI to identify keyphrases representative of a target corpus for representation learning. However, there are several limitations regarding this approach:\n\n1. The process includes row-normalizing the weight matrix W. However, it is not clear what specific normalization technique is used, or how it might affect the properties of the graph or the success of the community detection.\n\n2. The calculation of PMI includes additional terms to avoid bias toward rare words and to favor longer n-grams. These modifications might address some of the common problems with PMI but might introduce new biases or sensitivities. For instance, the square of the number of words in the n-gram could heavily bias the system towards very long phrases, which might or might not be appropriate depending on the nature of the corpus.\n\n3. Depending on the size of D and P, the graph G_D may be quite large, and constructing and analyzing it may be computationally expensive. \n\n4. The calculation of PMI includes additional terms to avoid bias toward rare words and to favor longer n-grams. These modifications might address some of the common problems with PMI but might introduce new biases or sensitivities. For instance, the square of the number of words in the n-gram could heavily bias the system towards very long phrases, which might or might not be appropriate depending on the nature of the corpus.\n\nW2: In section 2.3, the similarity matrix A is constructed based on the cosine similarity between utterance encoding of 2 nodes. However, if the underlying representations (e.g., embeddings) are high-dimensional, computing cosine similarities can be computationally expensive. Also, when creating a new graph G_pred using the weighted average between W and A, the process of conducting weighted averaging is missing. If the weighted averaging process is not thoughtfully designed, it could lead to loss of discriminative information.\n\nW3: In section 2.4,  as a transductive learning method, MAD requires access to all labeled and unlabeled data upfront. It may not generalize well to unseen data or new instances that weren't included in the original graph. Also, MAD introduces a dummy label for the unlabeled nodes. If not managed carefully, this can sometimes lead to ambiguity or confusion in the classification, especially if there's a large imbalance between labeled and unlabeled instances.\n\nW4: For evaluation on intent classification, more evaluations are required on other datasets including MixSNIPS, and FSPS. Also, compared with previous works' multi-label classification results on different benchmark datasets, IntenDD does not have significant improvements from the perspectives of F1 and accuracy shown in Table 3. The author may need to re-examine their approaches and the implementation part. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the detection and discovery of intents within dialogue utterances in task-oriented dialogue systems. It aims to unify different intent-related tasks, including multiclass and multilabel intent classification, as well as intent discovery in unsupervised and semi-supervised settings.\n\nContribution 1: IntenDD is introduced, combining intent detection and discovery through a shared utterance encoding backbone. The model leverages an unsupervised contrastive learning method for representation learning and uses pseudo-labeling for unlabeled utterances based on n-gram lexical features.\n\nContribution 2: A two-step post-processing setup is incorporated for classification tasks using Modified Adsorption (MAD). This process involves propagating residuals in the training data and smoothing the labels in a transductive setting.\n\nContribution 3: The paper highlights the application of classical transductive learning approaches and how the construction of a graph structure in a transductive setting might contribute to improving model performance.\n\nContribution 4: The evaluations presented in this work show that the proposed approach performs better than competitive baselines across various tasks under the selected evaluation metrics.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper is very hard to follow and read. Many paragraphs are repeated and incoherent with each other. For example, when presenting section 2.2, the authors should present the Louvain method and Keyphrase selection for constructing G_D first for avoiding the reader's confusion. \n\nThere are also many inconsistent use of abbreviations and acronyms in this paper. For example,  the terms like \"ID\" (for Intent Discovery), \"KIR\" (Known Intent Ratio) [line 461- 470], and others are used without clear definitions in the preceding text. This could lead to confusion. \n\nFinally, the proposed approach has a very complex design and needs to be visualized for the reader to better understand how each component works and is connected to each other. Otherwise, it would be too hard for the reader to follow. "}}, {"review_id": "EzTO1MZM1R", "reviewer": "Reviewer_WzPH", "summary": "", "questions": "1. About motivation, what is the intuition to unified tasks by introducing graphs with Louvain and MAD?  If I used K-means with representations learned by INTENDD, will it also work well?\n2. Significant tests are needed to prove the effectiveness, such as a single-tail t-test. From Table 2, adding the MAD increases merely about 1% in accuracy;\n3. Does the model require a separate graph to be constructed for each dataset?\n4. What are the time and space complexities associated with the construction of these graphs? \n5. In Table 4, Is the continued pre-training / graph constructed using the entire dataset or only the training set? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and the thoughtful review. \n\n**Loss functions used**\n\nThank you for pointing this out. Due to space constraints, we had to omit the formulae for loss functions, but we will add this in the final version. \n\n**Graph Construction Procedure**\nWe will add a pictorial illustration to clarify the graph construction procedure in the appendix of the paper to improve the readability. \n\n**Re: Corpus-specific representation learning**\n\nFirst, we want to emphasize that by unifying we simply mean that we are the first ones to propose a shared encoding backbone for the three tasks of intent detection and discovery. Prior works  in literature, such as the baselines we consider, use different backbones with different pre-training procedures for each of these tasks \n\n**Re: Missing Reference**\n\nThank you for pointing this out. We will add this to the final version. \n\n**For Q1:**  Good point. First, our motivation to unify the tasks is an important practical concern in the industry where separate pre-training procedures for each of the three tasks are undesirable and hard to maintain. We use Louvain and MAD since these are well-known graph construction methods (ex: for community detection). Using Louvain, we are able to additionally incorporate valuable lexical semantics-level information which is often not explicitly captured in contrastive representation learning approaches.  Hence it complements the information captured in contrastive learning techniques. Using MAD, we apply both error propagation (EP) and smoothing. Here, EP propagates residual errors in training data to correct errors in test data and the smoothing step enables to penalize similar models that are assigned with different labels, or more technically it smoothes the prediction in the test data based on the similarity between data points. In our few shot (5-shot) multiclass settings, the use of MAD has resulted in at least 2 % absolute improvements for all the three datasets we experimented with.\n\n We expect our method to work better than k-means since it involves a two-step process of label propagation and label smoothing (as part of MAD). \n\n**For Q2**: Indeed, we performed the paired t-test to verify the effectiveness of our method (p-value in all cases was < 0.05). We will mention this in the paper. \n**Regarding  MAD**: As compared to IntenDD-MLP, our configuration where MAD is not employed,  IntenDD uses a two-step pipeline application of MAD (error propagation step and smoothing step). In few-shot settings (5-shot), IntenDD has been shown to increase the performance by at least 2 % absolute improvement for all the datasets as compared to IntenDD-MLP, and the maximum being 3.17 % absolute improvement for Banking. We achieve statistically significant gains in all the cases in Table 2 (10-shot and full), though the gains are merely 1% for full shot setting as highlighted by you. Given that, intent classes can be quite high and can be highly dependent on a business use-case, few shot settings are desirable for intent-related tasks in ToD systems. Hence, we believe the gains by MAD in few shot settings is a valuable addition to the pipeline.\n\n**For Q3:** Yes, the model requires constructing a separate graph for each dataset. We follow this from previous works (like: Vulic et al. 2022 (https://aclanthology.org/2022.emnlp-main.512.pdf )) that use dataset specific procedures after common pre-training. \n\n\n**For Q4:** For graph construction, the nodes remain constant and the edges are constructed by using a feature selection approach. For each iteration of the feature selection approach, we perform the Louvain algorithm. A single run of the Louvain algorithm has a complexity of $O(n.log n)$, where $n$ is the number of nodes. It is known to be a fast community Detection approach. We run that for $d$ iterations in the worst case, where $d$ is the number of features. Altogether, the time complexity for graph construction is $O(n.d.log n)$ . Now, The aforementioned time complexity is comparable to that of the K-Nearest neighbors (K-NN) algorithm. For K-NN the complexity is  $O(n.d.k)$, or a $O(nd + kn)$ for a faster version, where $k$ is the number of neighbors.  Moreover, the largest dataset we currently consider includes 20,000 nodes where a single run of Louvain is often completed in less than 10 seconds in an Intel i-5 CPU and 16 GB RAM, in terms of the wall clock time. Please note that in case of larger network sizes, say those with $10^7$ nodes ore more, we may alternatively use the Leiden algorithm to improve upon the speed. However, that comparison does not fall within the scope of the current work. Further, obtaining and scoring keyphrases does not take more than 14 minutes of wall clock time in a CPU with the aforementioned configuration. This method requires one pass of the Dataset and the set of keyphrases (P)  for collecting n-gram counts of keyphrases and another for calculating the PMI scores.  For larger datasets, we may even use parallelization that can substantially reduce the wall clock time computation. \n\n**For Q5**: Using only the training set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel unified approach to intent detection (multi-class and multi-label) and discovery (unsup. and semi-sup).\n2. The introduction of the two-step post-processing method shows performance improvement, demonstrating the efficacy of the proposed model. \n3. The extensive experiments across various datasets and tasks makes it a valuable contribution to the research community.", "reasons_to_reject": "1. Some important details is unclear and should be clarified. For example: (1) needs to summarize the total loss function of each stage/task, including how to balance the loss. (2) The final prediction for intent classification (multi-class/label) and intent discovery (unsup., semi-sup.) should be formally defined in the formula.\n2. The absence of a comprehensive discussion of the construction of the graph structure in the transductive learning setting might be a concern for readers aiming to replicate or extend the model.\n3. The proposed method necessitates corpus-specific representation learning. It does not follow the pattern of unified training followed by direct inference across multiple tasks and datasets.", "questions_to_authors": "1. About motivation, what is the intuition to unified tasks by introducing graphs with Louvain and MAD?  If I used K-means with representations learned by INTENDD, will it also work well?\n2. Significant tests are needed to prove the effectiveness, such as a single-tail t-test. From Table 2, adding the MAD increases merely about 1% in accuracy;\n3. Does the model require a separate graph to be constructed for each dataset?\n4. What are the time and space complexities associated with the construction of these graphs? \n5. In Table 4, Is the continued pre-training / graph constructed using the entire dataset or only the training set? ", "paper_topic_and_main_contributions": "This paper is centered around INTENDD, a novel and unified contrastive learning approach for intent detection and discovery in task-oriented dialogue systems. \nThe primary contributions include (1) the formulation of INTENDD that unifies the tasks of intent classification and discovery using a shared utterance encoding backbone, (2) an entirely unsupervised contrastive learning strategy for representation learning, and (3) a two-step post-processing setup using modified adsorption(variant of label propagation) for intent classification. The study also presents performance improvements in comparison with other baselines across several benchmark datasets.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. It is advisable to add a figure of model architecture / framework to better illustrate the idea.\n2. Line 066: aA —> a\n3. Line 099: report —> reports\n4. Line 166: a connected a weighted"}, "extra_text": {"Missing_References": "Zhang, Hanlei, et al. \"TEXTOIR: An Integrated and Visualized Platform for Text Open Intent Recognition.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. 2021."}}]

Author rebuttal:

Meta-review: An unsupervised contrastive learning strategy for representation learning, even with interesting results, the paper lacks comprehensive experiments and a sound explanation of graph structure in the method section.
Decision: Accept-Findings