SlowGuess/NAACL_2022_OCR huggingface dataset:
Cluster Merger Algorithm After phase-2, we run DBSCAN and obtain a set of clusters including the outlier cluster. For each cluster (except the outlier) we randomly sample  $p$  utterances and use them as cluster descriptions. The cluster representation is obtained as the mean of these utterance representations. So, now each cluster has its own description and representation. Now we randomly pick one cluster as the query cluster  $(q)$  and get its  $s$  nearest neighbors based on cosine-similarity and ask "Which of these  $s$  clusters should be merged with  $q$ ? to a domain expert. Based on the domain expert's response we merge similar clusters, recalculate the cluster representations, and assign a cluster description of  $q$  to this newly created cluster. We repeat this process till the domain expert finds no candidate for thirty consecutive query clusters. One iteration of the cluster merging algorithm is illustrated in Fig 4.
Now we treat each cluster as an intent and train a logistic classifier to label utterances that belong to the outlier cluster. We use the same classifier for intent detection on the test set.
$$
\begin{array}{l} \mathcal {L} _ {s s c} ^ {\prime} = \sum_ {t \in \mathcal {B} ^ {\prime}} \frac {- 1}{| P (u _ {t}) |} \sum_ {p ^ {\prime} \in P (u _ {t})} \exp \left(u _ {t - 1}, u _ {t - 1}\right) / (\tau) (7) \\ \log \frac {\exp \left(u _ {t} \cdot u _ {p ^ {\prime}}\right) / \tau}{\sum_ {a \in N (t)} \exp \left(u _ {t} \cdot u _ {a}\right) / \tau}; (7) \\ \end{array}
$$
Figure 4: Cluster Merger Algorithm: In each iteration, a Domain Expert is shown (s = 5) Candidate Clusters which are nearest neighbors of Query Cluster (q), where per cluster, only (p = 2) utterances are shown to the expert. And based on the reply from the domain expert, some candidate clusters are merged into the query cluster and the cluster definition is updated.
In this section, we describe the various baselines, datasets, and evaluation metrics used in our experiments.
5.1 Baseline Approaches
We use unsupervised  $K$ -means (MacQueen, 1967), Agglomerative Clustering (AG) (Chidananda Gowda and Krishna, 1978), DEC (Xie et al., 2016b), SAE-KM (Xie et al., 2016b), DCN (Yang et al., 2017), DAC (Chang et al., 2017), DeepCluster (Caron et al., 2018), SCCL (Zhang et al., 2021a) and semi-supervised PCK-means (Basu et al., 2004), BERT-KCL (Hsu et al., 2018), BERTMCL (Hsu et al., 2019), BERT-DTC (Han et al., 2019), CDAC+ (Lin et al., 2020), DeepAligned (Zhang et al., 2021c) clustering approaches as baselines. SCCL and DeepAligned are the state-of-the-art approaches for unsupervised and semi-supervised clustering respectively. Details about unsupervised and semi-supervised baselines are included in Appendix A.1.
We evaluate DSSCC on five datasets with a varying number of intents. We use BANKING77 (Casanueva et al., 2020), CLINCI50 (and CLINC150  $OOS$ ), (Larson et al., 2019) SNIPS (Coucke et al., 2018), StackOverflow Xu et al. (2015) and, DBPedia (Zhang and LeCun, 2015). For BANKING77 and CLINCI50 we use the same train, val and test split as Zhang et al. (2021c) and for SNIPS, StackOverflow and DBPedia we follow the same split as Lin et al. (2020). For more details, please refer to Appendix A.3
Similar to previously reported results (Lin et al., 2020; Zhang et al., 2021c), we use Clustering Accuracy (ACC) (Yang et al., 2010), Normalized Mutual Information (NMI) (Strehl and Ghosh, 2002) and Adjusted Rand Index (ARI) (Hubert and Arabie, 1985) as evaluation metrics. All metrics range from 0 to 100 and higher values of a metric indicate superior clustering results. Further details of the experimental setup are provided in Appendix A.4. Due to space constraints, we report ACC, NMI only but in Appendix, we use all three metrics to report results.
6 Results And Discussion
We present results comparing DSSCC with unsupervised and semi-supervised clustering approaches for scenarios where 1) the system is aware of the number of unknown intents, 2) the system is unaware of the number of unknown intents. We further report results of DSSCC-HIL and compare it with DSSCC-E2E.
6.1 Intent detection and discovery when the number of new intents  $m$  is provided
For the semi-supervised scenario, we assume  $x\%$  of the total intents are known apriori, also referred to as the Known Intent Ratio (KIR). For each known intent,  $10\%$  of the utterances are labeled and the rest are unlabeled. For a fair comparison, we also use the same BERT-based PLM as other baselines. DSSCC outperforms all unsupervised baselines on both BANKING77 and CLINC150 datasets by significant margins suggesting that for known intents, supervision in the form of a few labeled utterances leads to better intent representations, as shown in Table 1. DSSCC also outperforms semi-supervised baselines on BANKING77 and CLINC150 for all cases except for the NMI metric on the CLINC150 dataset for KIRs  $50\%$  and  $75\%$ .
For K-meansSBERT, we use SBERT Reimers and Gurevych (2019) as our PLM instead of BERT to obtain utterance representations and find that it outperforms K-meansBERT by a significant margin, as shown in Table 1. This suggests that utterance representations from SBERT are more suitable for clustering than BERT-based representations. Motivated by these results we compared BERT and SBERT representations in Table 2 and found that DSSCC
KIR	Approach	ACC	NMI	ACC	NMI
0%	K-meansBERT	45.06	70.89	29.55	54.57
K-meansSBERT	61.04	82.22	55.72	74.68
AG	44.03	73.07	31.58	57.07
SAE-KM	46.75	73.13	38.92	63.79
DEC	46.89	74.83	41.29	67.78
DCN	49.29	75.66	41.99	67.54
DAC	55.94	78.40	27.41	47.35
DeepCluster	35.70	65.58	20.69	41.77
SCCL	33.52	66.63	13.41	34.14
25%	PCK-means	54.51	68.71	32.66	48.22
BERT-KCL	24.72	65.74	22.11	52.42
BERT-MCL	24.35	65.06	22.07	51.96
BERT-DTC	49.1	74.17	25.24	48.58
CDAC+	64.64	84.25	48.71	69.78
DeepAligned	73.71	88.71	48.88	70.45
DSSCCBERT	75.72	89.12	55.52	72.73
50%	PCK-means	54.51	68.62	32.26	48.11
BERT-KCL	46.91	78.45	40.97	65.22
BERT-MCL	47.21	78.39	41.43	65.68
BERT-DTC	71.68	86.20	53.59	71.40
CDAC+	69.02	86.18	53.34	71.53
DeepAligned	80.22	91.63	59.23	76.52
DSSCCBERT	81.46	91.39	63.08	77.60
75%	PCK-means	54.61	68.70	32.66	48.22
BERT-KCL	68.86	86.82	60.15	75.21
BERT-MCL	69.66	87.72	61.14	75.68
BERT-DTC	80.73	90.41	56.51	76.55
CDAC+	69.89	86.65	53.83	72.25
DeepAligned	86.01	94.03	64.90	79.56
DSSCCBERT	87.91	93.87	69.82	81.24
Table 1: We report ACC and NMI on CLINC150 and BANKING77 datasets in the semi-supervised scenario for three different known intent ratios (KIR). Except K-means  ${}_{SBERT}$  and SCCL,we take all baseline results from Zhang et al. (2021c).
KIR	Approch	ACC	NMI	ACC	NMI
25%	\( \mathbf{D_{ABERT}} \)	73.71	88.71	48.88	70.45
\( \mathbf{D_{ASBERT}} \)	67.78	86.50	57.0	75.0
\( \mathbf{DSSCC_{BERT}} \)	75.72	89.12	55.52	72.73
\( \mathbf{DSSCC_{SBERT}} \)	80.36	91.43	64.93	80.17
50%	\( \mathbf{D_{ABERT}} \)	80.22	91.63	59.23	76.52
\( \mathbf{D_{ASERT}} \)	77.69	91.40	64.14	79.30
\( \mathbf{DSSCC_{BERT}} \)	81.46	91.39	63.08	77.60
\( \mathbf{DSSCC_{SBERT}} \)	83.49	92.78	69.38	82.68
75%	\( \mathbf{D_{ABERT}} \)	86.01	94.03	64.90	79.56
\( \mathbf{D_{ASERT}} \)	85.89	94.20	74.08	83.80
\( \mathbf{DSSCC_{BERT}} \)	87.91	93.87	69.82	81.24
\( \mathbf{DSSCC_{SBERT}} \)	88.47	94.50	75.15	85.04
Table 2: DA vs DSSCC with BERT and SBERT as PLM
(Ours) $_{SBERT}$  outperforms DSSCC (Ours) $_{BERT}$ , DeepAligned $_{BERT}$  and DeepAligned $_{SBERT}$  by a significant margin on both datasets for all evaluated intent ratios. DeepAligned $_{SBERT}$  outperforms DeepAligned $_{BERT}$  on BANKING77 by a significant margin but we observe the opposite result on CLINC150. For the remaining experiments, we use SBERT as our PLM in DSSCC and compare it with DeepAligned $_{BERT}$  and DeepAligned $_{SBERT}$ .
We evaluated DSSCC on three other public datasets for intent detection and discovery, and found that DSSCC outperforms DeepAligned on all three datasets except for NMI on DBPedia for known intent ratios of  $75\%$ , as shown in Table 10. Results of  $DSSCC_{SBERT}$  on intent detection and
CLINC150  $(\mathcal{T} = 150, n = 112, m = 38)$  BANKING77  $(\mathcal{T} = 77, n = 58, m = 19)$
Approach	K'	KPred	ACC	NMI	K'	KPred	ACC	NMI
DABERT	300	130	77.18	92.5	154	65.1	62.49	78.88
DASBERT	300	129.6	76.87	92.61	154	66.9	63.53	80.84
DSSCCSBERT	81.37	92.97	71.77	84.29
DABERT	450	189.2	83.81	93.57	231	99.3	63.98	79.93
DASBERT	450	190.1	82.57	93.85	231	96.8	66.20	82.11
DSSCCSBERT	84.59	93.64	72.93	84.97
DABERT	600	258.6	72.22	91.8	308	121.9	61.05	79.95
DASBERT	600	255.9	72.29	92.18	308	118.1	62.67	82.05
DSSCCSBERT	80.83	92.04	67.56	84.43
DSSCCSBERT	150	150	88.47	94.50	77	77	75.15	85.04
discovery are separately reported in Table 11.
6.2 Intent detection and discovery with unknown number of new intents  $m$
We evaluate DSSCC for the realistic scenario when the number of new  $(m)$  intents present in user logs is not provided to the system apriori, i.e., the total number of intents  $(\mathcal{T} = n + m)$  is not known in advance and the system has to infer the number of clusters present in user logs. We use an existing algorithm proposed by Zhang et al. (2021c) which refines an initial guess  $K^{\prime}$  to arrive at the final estimate  $K_{pred}$ . As shown in the Table 3, DSSCC outperforms DeepAligned for  $K^{\prime} \in \{2 * \mathcal{T}, 3 * \mathcal{T}, 4 * \mathcal{T}\}$  except for the NMI metric on the CLINC150 dataset for  $K^{\prime} = 450$  and  $K^{\prime} = 600$  although results are competitive. As compared to the scenario where the total number of intents  $\mathcal{T}$  are known (last row in Table 3), on an average there is drop of  $6.20\%$  in ACC,  $1.61\%$  in NMI on CLINC150 and a drop of  $4.39\%$  in ACC,  $0.47\%$  in NMI on BANKING77. These results suggest that the performance of DSSCC does not drop significantly even when the total number of intents in user logs are not known in advance.
6.3 DSSCC-HIL
We evaluate DSSCC-HIL in a realistic scenario where the number of new intents is not known and KIR=75%. As shown in Table 4, we get 333 and 523.7 clusters after phase-2 of DSSCC-HIL with average cluster purity of 96.96% and 98.30% corresponding to B77 and C150 respectively. Average purity refers to average clustering accuracy where we use ground-truth labels and based on majority voting, assign an intent label to the predicted cluster. For merging similar clusters, we show (s=5) candidate clusters per query to the domain expert who is asked to choose clusters that are similar to the query cluster. Here, we have used oracle
ground truth cluster labels instead of a domain expert to answer these queries. For B77 and C150, 259.3 and 349.5 queries are required (avg over 10 runs) to merge similar clusters respectively where the domain expert has to read 12 utterances (2 per cluster) per query. As a result, we obtain 81 and 152.59 clusters (intent) for B77 and C150 which are close to the actual number of intents i.e., 77 and 150 respectively. Then a classifier is trained with these intents and prediction is done on the test set. DSSCC-HIL achieved an ACC of  $81.21\%$  and  $88.93\%$  on B77 and C150 respectively which is significantly higher than the ACC of DSSCC-E2E. Also, DSSCC-HIL is able to discover all intents in the ground truth. We also employ a cluster merging strategy with DSSCC-E2E i.e., DSSCC-E2E+HIL and got an improvement of  $2\%$  for C150 but negative results for B77. This is due to better initial cluster purity (P) of C150 (i.e.  $87.73\%$ ) versus B77 (i.e.  $79.92\%$ ), as the merging of noisy clusters intuitively leads to a decrease in ACC. This observation supports the fact that, for merging clusters by a domain expert, a good initial cluster purity is required. Due to comparatively low purity initial clusters, HIL does not help much in DSSCC-E2E.
Table 3: Intent detection and discovery with unknown value of  $m$  for KIR=75%. We obtain results corresponding to DeepAligned  ${}_{BERT}$  (DA  ${}_{BERT}$  ) (Zhang et al.,2021c) and report results for DeepAligned  ${}_{SBERT}$  (DA  ${}_{SBERT}$  ) using code provided by the authors of Zhang et al. (2021c) with SBERT as the PLM.
A	Q	P	K'	Kpred	ACC	NMI
E2E	0	NA	231	96.8	72.93	84.97
E2E+HIL	144.8	79.92	96.8	67	71.14	84.00
HIL	259.3	96.96	333	81	81.21	87.35
E2E	0	NA	450	190.1	84.59	93.64
E2E+HIL	218.2	87.73	190.1	148.6	86.10	93.83
HIL	349.5	98.30	523.7	152.6	88.93	95.19
Table 4: We compare DSSCC-E2E, DSSCC-E2E+HIL and DSSCC-HIL. P, Q refers to average cluster purity before merging and number of queries respectively. For DSSCC-E2E, we pick best results from Table 3. First row and second row of the table contains results on B77 and C150 respectively.
As part of the ablation study, we answer the following questions "Do we need both phase-1 and
phase-2 in DSSCC ?”, “Does joint training on multiple loss functions in phase-2 affect clustering accuracy ?”, “Do we need different values of Entropy Weight  $(\lambda)$  (Eq. 6) for different datasets?”, “How does the presence of out-of-scope (oos) utterances in user logs affect DSSCC performance”? and present our observations. To answer these questions we perform ablation studies on CLINC150 and BANKING77.
"Do we need both phase-1 and phase-2 in DSSCC?" As shown in Table 5, KIR=75%, finetuning of the PLM on labeled utterances from known intents in phase-1 is more vital for BANKING77 as compared to CLINC150 because without phase-1 there is a significant drop in performance on the BANKING77 dataset (DSSCC w/o phase-1 vs DSSCC). Whereas there is an improvement of 4.39% in ACC, 1.15% in NMI and 4.86% due to phase-2 on CLINC150 (DSSCC w/o phase-1 vs DSSCC), which suggests that both phase-1 and phase-2 of DSSCC are important for a generic solution. In the case of BANKING77, performance largely depends on labeled utterances from known intents and there is not much improvement due to phase-2, which may be attributed to its complexity as all intents are semantically closer and belong to the Banking domain as compared to CLINC150 where utterances belong to 10 different domains (utility, travel, etc.). But when we perform the same ablation with fewer known intents i.e., KIR=25%, then both phase-1 and phase-2 are equally important to achieve good performance. This suggests that DSSCC phase-2 is more important when there are fewer known intents and fewer labeled utterances per known intent.
"Does joint training on multiple loss functions in phase-2 affect clustering accuracy"? As shown in Table 5, we also perform an ablation on different loss functions used in phase-2 and found that Semi-supervised contrastive (ssc) loss (DSSCC w/o ssc vs DSSCC), entropy maximization (em) (DSSCC w/o em vs DSSCC) and supervised-representation learning (srl) loss (DSSCC w/o srl vs DSSCC) all affect the clustering performance on CLINC150 and BANKING77. The reason for the most drop in ACC and NMI in the case of (DSSCC w/o em) is that, when entropy maximization is not done, the probability distribution over clusters for unlabeled utterances is decided only by the supervised-representation learning (srl) loss. Therefore, all the utterances become part of known intent clusters
and the clusters for unknown intents remain empty. "Do we need different values of Entropy Weight  $(\lambda)$  (Eq. 6) for different datasets?" In Table 7, we report DSSCC-E2E results on CLINC150 and BANKING77 with different values of  $\lambda$ . And we observe that a value of  $\lambda \epsilon$  [10, 14] yields the best results across datasets.
"How does the presence of out-of-scope (oos) utterances in user logs affect DSSCC performance"? We use oos utterances as part of user logs in CLINC150 dataset and evaluate DSSCC. As shown in Table 6, there is a small drop in performance because a few oos utterances are classified to belong to the set of actual intents. This observation shows that even with presence of oos utterances, DSSCC is able to maintain its performance. Manual intervention may be required in practice to filter clusters containing oos queries.
KIR	Approach	CLINC150	BANKING77
ACC	NMI	ACC	NMI
75%	DSSCC w/o srl	85.99	93.68	70.24	81.78
DSSCC w/o ssc	73.87	87.07	67.97	78.81
DSSCC w/o em	71.05	91.88	64.55	81.56
DSSCC w/o phase-1	88.13	94.04	72.39	81.76
DSSCC w/o phase-2	84.08	93.35	74.09	84.56
DSSCC	88.47	94.50	75.17	83.69
25%	DSSCC w/o srl	78.60	90.24	54.93	73.05
DSSCC w/o ssc	40.32	71.09	37.17	61.15
DSSCC w/o em	24.86	75.02	23.36	64.10
DSSCC w/o phase-1	79.95	90.44	58.85	74.74
DSSCC w/o phase-2	72.51	88.38	58.62	76.40
DSSCC	80.36	91.43	61.91	77.44
Table 5: Ablation with KIR (75%) and (25%) on CLINC150 and BANKING77 with  $DSSCC_{SBERT}$  where we use DSSCC-CH for inference.
In this work, we propose a semi-supervised contrastive learning approach for intent detection and discovery from user logs. The proposed approach optimally utilizes both labeled and unlabeled utterances to outperform SOTA approaches for both scenarios where the total number of intents is either known in advance or has to be estimated. We also propose a variant of our approach which does not need to estimate the number of new intents and yields pure clusters which are merged by domain experts based on the cluster descriptions. Future work will focus on, (i) "How to get better cluster descriptions?" (ii) "How to optimally select query and corresponding candidate clusters?" and (iii) "How to discover new intents with minimum human effort with a long-tail distribution over new intents in user logs?"
Sugato Basu, Arindam Banerjee, and Raymond J. Mooney. 2004. Active semi-supervision for pairwise constrained clustering. In Proceedings of the Fourth SIAM International Conference on Data Mining, pages 333-344.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV).
Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020. Data available at https://github.com/PolyAI-LDN/task-specific-datasets.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. 2017. Deep adaptive image clustering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
Ajay Chatterjee and Shubhashis Sengupta. 2020. Intent mining from past conversations for conversational agent. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4140-4152, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
K. Chidananda Gowda and G. Krishna. 1978. Agglomorphic clustering using the concept of mutual nearest neighbourhood. Pattern Recognition, 10(2):105-112.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. CoRR, abs/1805.10190.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96, page 226-231. AAAI Press.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP).
Belize Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations.
Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. A self-training approach for short text clustering. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 194–199, Florence, Italy. Association for Computational Linguistics.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. 2018. Learning to cluster in order to transfer across domains and tasks. In ICLR.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. 2019. Multi-class classification without multi-class labels. In ICLR.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. 2017. Learning discrete representations via information maximizing self-augmented training. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 1558-1567. JMLR.org.
Lawrence J. Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of Classification, 2:193-218.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457, New Orleans, Louisiana. Association for Computational Linguistics.