SlowGuess/NAACL_2022_OCR huggingface dataset:
per McNemar's Test. It is observed that our approach also works well on the highly imbalanced ATIS dataset in which some of the classes have less than 10 data points and highly similar BANKING dataset in which the intents are closely related eg., 'top-up-reverted' and 'top-up-failed'. This is because although  $Gl_{F}$  chooses uniformly across all classes, MNID selectively labels datapoints having high uncertainty thus providing the classifier with the right ingredient to perform better. In IFSTC on SNIPS dataset, MNID underperforms as compared to  $Gl_{F}$  but with a very small margin. This happens because in the case of SNIPS dataset, the number of new classes is very less, hence  $Gl_{F}$  can choose ideal candidates. The best performance of MNID as well as the two baselines is in the PolyAI setting when it is used with Universal Sentence Encoders (USE). Since PolyAI performs the best, all our subsequent results are provided on PolyAI (USE).
(C) Distribution of gold annotated points: Fig 2 shows the count of the gold annotated points  $(Y - axis)$  for new classes (class indices on  $X -$  axis). The dotted line is at the frequency of 10, corresponding to the average annotations per class. For  $76.2\%$  (HWU),  $81.5\%$  (BANKING) and  $67.3\%$  (CLINC) classes in good clusters require  $\leq 10^{\prime}$  annotations. More than 10 annotations are needed for  $65.4\%$  (HWU),  $68.5\%$  (BANKING) and  $54.5\%$  (CLINC) classes in bad clusters.
(D) Budgets: For novel intent class discovery, a minimum number of human annotation is necessary. For NCD to work, at least 4 shot, 6-shot and 7-shot annotations are required for HWU, BANKING and CLINC datasets respectively.
Different Variations of MNID
MNID consists of three steps (a). novel class detection (NCD), (b). cluster quality based annotation (CQBA) and (c). post-processing annotation strat
egy (PPAS). In each of these steps, certain parameters can be varied. We systematically discuss the impact of these parameters on MNID performance.
Variations at NCD
(a) Performance of Clustering Algorithms: We explore different unsupervised and semi-supervised clustering algorithms in our MNID framework. Overall accuracy and F1-Score for open intent discovery by different approaches are shown in Table 4. From Table 4, it is seen that unsupervised approaches perform better than semi-supervised models. The semi-supervised techniques get biased by the initial seed and fail to discover diverse clusters needed to detect all the new intent classes. K-Means (KM) performs the best across all datasets in terms of accuracy and F1 score except for HWU dataset where DEC and DTC (F1 only) outperforms it. This is most probably due to its robustness and absence of any outlier in the dataset. So we use K-Means as the clustering algorithm for MNID.
(b) Class Discovery with number of clusters: From Fig 3a, we observe an increasing trend in the number of classes discovered with increasing number of clusters which show that classes get evenly distributed across clusters as the number of clusters increases. The rate at which new classes are discovered is linear with the new clusters until significant classes are detected. The horizontal lines represent the gold number of new intents.
(c) Effect of number of points  $(x)$  used in clustering: Fig 3b shows that the accuracy on all datasets drops as we increase the number of points used for new class discovery in clustering beyond  $x = 2$ . This is because most of the budget gets exhausted while clustering and we have a very small budget to annotate low-confidence points in the next steps. Note that at least two points from a cluster need to be annotated for new class discovery.
(a) HWU
(b) BANKING
(c) CLINC
Figure 2: Count of gold annotated points for newly discovered classes
Dataset	Unsupervised Clustering Algorithms	Semi-Supervised Clustering Algorithms
KM	AG	DCN	DEC	DAL	DTC	KCL	MCL	CDAC+
A	F1	A	F1	A	F1	A	F1	A	F1	A	F1	A	F1	A	F1	A	F1
SNIPS	95.1	94.9	92.7	92.9	89.2	88.7	89.6	88.2	92.2	92.2	87.6	87.2	73.3	70.4	78.2	74.1	80.4	79.2
ATIS	88.6	87.0	85.8	86.4	77.7	79.78	83.1	85.42	86.9	87.0	84.3	85.9	76.7	80.8	80.4	83.2	77.9	81.6
HWU	83.8	84.2	83.2	83.3	84.1	83.6	84.7	84.4	83.7	82.6	83.6	85.2	73.3	74.1	78.1	74.8	83.1	81.1
BANKING	84.7	84.4	84.2	84.1	80.1	83.2	80.1	80.5	80.5	81.1	79.9	78.2	71.8	72.4	74.2	73.1	83.4	82.6
CLINC	94.9	95.2	93.9	94.8	93.4	94.2	93.4	94.9	93.9	92.6	93.9	93.2	83.4	84.5	81.0	82.3	92.1	92.5
Table 4: Accuracy (A) and F1-Score (F1) in (%) for various Open Intent Discovery Based Clustering Algorithms across all datasets. The best results for each dataset in Bold.
p, q	2, 1	2, 2	2, 3	3, 0	3, 1	3, 2	4, 1
HWU	82.1	81.2	82.5	81.8	83.7	83.8	83.2
BANKING	84.1	83.0	84.2	82.9	84.0	84.7	84.1
CLINC	94.8	93.9	94.2	92.7	93.0	94.9	94.2
(a) Class discovery with number of clusters
Figure 3: Variations of NCD
(b) Accuracy vs points annotated  $(x)$  for clustering
Variations at CQBA
(a) Effect of number of points selected from Good and Bad clusters: We experiment with different values of point selection  $(p,q)$  for the module CQBA (Algo 4) and observe how accuracy changes for three larger datasets - HWU, BANKING and CLINC. We get the best accuracy for  $(p,q) = (3,2)$  i.e  $3(p)$  points from good cluster and  $5(p + q)$  points from bad cluster as shown in Table 6. Since, we perform gold annotation strategy on the bad clusters, a higher number of point selection is re
quired to identify classes.
(b) Distribution of good and bad clusters: For CLINC, BANKING and HWU we obtain 256, 128 and 64 clusters respectively by NCD. The percentage of good clusters obtained for CLINC, BANKING and HWU are  $70.70\%$  (181 out of 256),  $46.88\%$  (60 out of 128)  $56.25\%$  (36 out of 64), respectively. For BANKING, since the entire dataset is from a single domain with multiple intents being similar among themselves, we obtain more bad clusters than the good clusters. For SNIPS and ATIS, however, all the clusters are good clusters.
Variations at PPAS
(a) Different Variations of Gold and Silver Strategies: The results for different variations of MNID methods (based on Silver and Gold Strategy applications) for all the datasets are provided in Table 5. We observe that the best result is obtained on MNID-9, i.e., choosing high confidence points from the good clusters for silver strategy and low confidence points from the bad clusters (if detected or else from the good clusters) only for gold strategy. This strategy ensures that during silver annotation we choose points with high fidelity and side by side for gold annotation choose points with high uncertainty, both of which help in developing a highly accurate classifier. Silver strategy on
Table 6: Accuracy (%) based on point selections from Good and Bad clusters
Method	Silver Strategy	Gold Strategy	SNIPS	ATIS	HWU	BANKING	CLINC
A	F1	A	F1	A	F1	A	F1	A	F1
MNID-1	Good Clusters†	X	94.4	93.2	87.2	85.4	78.5	78.8	77.5	78.4	89.2	89.8
MNID-2	Good Clusters†	Any Point from Bad Clusters	94.4	93.2	87.2	85.4	80.9	80.9	79.3	80.0	90.8	90.7
MNID-3	X	Low-Conf from Any Cluster	94.7	94.0	87.9	86.1	81.2	81.1	81.7	81.1	91.3	91.0
MNID-4	High-Conf from Good Clusters	X	94.8	93.9	87.7	86.1	81.5	81.4	82.2	81.8	91.8	91.9
MNID-5	High-Conf from Good Clusters	Low-Conf from Any Cluster	95.1	94.9	88.6	87.0	82.9	82.2	82.7	81.8	92.1	93.5
MNID-6	Good Clusters†	Low-Conf from Bad Clusters	94.4	93.2	87.2	85.4	83.1	82.8	83.9	83.1	93.9	93.7
MNID-7	X	Low-Conf from Bad Clusters*	94.7	94.0	87.9	86.1	81.9	81.6	83.0	82.4	92.8	92.7
MNID-8	Good Clusters†	Low-Conf from Bad Clusters*	94.9	94.4	88.2	86.4	83.1	82.8	83.9	83.1	93.9	93.7
MNID-9	High-Conf from Good Clusters	Low-Conf from Bad Clusters*	95.1	94.9	88.6	87.0	83.8	83.2	84.7	84.4	94.9	95.2
Table 5: Accuracy (A) and F1-score (F1) in  $(\%)$  across all datasets for different variations of silver and gold strategy of MNID.  $[^{*}$  - If no bad cluster exists then the strategy is applied on good clusters (SNIPS, ATIS have no bad cluster). Detailed in line 6-11 of Algorithm 5.  $\dagger$  We use  $\mathcal{T}\mathcal{H} = 0$  in Algorithm 5.  $\pmb{x}$ : denotes we are not using this. Bold notifies the best for each dataset.]
Silver Strategy on	SNIPS	ATIS	HWU	BANKING	CLINC
All Clusters (%)	96.2	95.9	82.5	84.3	85.2
Good Clusters (%)	96.2	95.9	93.6	95.4	97.2
High-Conf from Good Clusters (%)	97.8	96.2	95.6	97.2	98.1
Average Points per Class	10.1	8.2	17.0	22.8	15.9
Table 7: Accuracy (in %) and usage of average number of datapoints per class in silver strategy
high confidence points from good cluster (7 vs 9) and gold strategy on low confidence points from bad cluster (4 vs 9) alone enhances  $\sim 1 - 3\%$  accuracy and F1 for the three large datasets.MNID-9 corresponds to our proposed approach, MNID.
(b) Silver Strategy Analysis: We inspect silver strategy based on cosine similarity, confidence score and strategy accuracy.
(i) Effect of Cosine Similarity and Confidence Score (CS): We study the effect of cosine similarity of silver strategy for MNID. From Fig. 4a, we observe that the best results are always obtained using a higher threshold of 0.8 cosine similarity. In case of BANKING, HWU and ATIS accuracy drops at 0.9 whereas for other datasets it remains almost identical. Fig. 4b shows how accuracy varies for different confidence scores. We observe that for all the datasets the best results are obtained at a threshold of 0.5. This is because a lower threshold allows more diverse datapoints to be selected using cosine similarity and this in turn improves the model performance. In both the cases if the cosine similarity or the threshold is increased beyond the optimal point, that results in selection of too less datapoints which is not enough for the classifier to do a meaningful learning. Hence accuracy drops. So we choose the parameters - cosine similarity = 0.8 and  $\tau = 0.5$  - while choosing high confidence point to be annotated by silver strategy.
(ii) Strategy Accuracy: The accuracy of data point selection by silver strategy for different MNID variations is shown in Table 7. We see the strategy of choosing high-confidence points from good clusters produce points with high fidelity. Table 7 also shows the average number of points per class as selected by this strategy for various datasets. Here we see that enough number of silver points are annotated even after considering a very strict criterion. Note, average points per intent class count is the highest for BANKING because multiple intents are very similar to each other and hence more points qualify the cosine similarity threshold,  $\tau$ .
(a) Cosine similarity threshold  $(\tau)$
Figure 4: Variations of PPAS
(b) Confidence score threshold  $(\mathcal{T}\mathcal{H})$
We have developed MNID (Multiple Novel Intent Detection), an end-to-end framework to identify multiple novel intents within a fixed annotation cost. The algorithm intelligently uses the concept of clusters to first discover the classes and then estimate the nature in which datapoints of a class is distributed, that is, whether the datapoints of a class congregate strongly within themselves and separate from other classes or are entangled with datapoints of other classes. In the two types of situations, we propose two different strategies, silver strategy to take advantage of the clusters so that we can annotate many points without any extra human cost and gold strategy to annotate highly uncertain points. This two-pronged approach helps us to annotate highly precise points automatically while annotating the most uncertain (with respect to the class it belongs) points using human assistance. We have done a very rigorous analysis/experimentation to establish the core idea of our algorithm. We observe that the accuracy of classifiers when fed with the dataset created by MNID can beat the standard best few-shot setting where it is assumed that  $\kappa$  instances of each class are provided and annotated by human whereas in our case we have to first discover the classes and then have to find the instances of each class.
One limitation of MNID is that it is not able to detect intents where classes are very similar to each other. For example, the query "Can you explain why my payment is still pending?" in BANKING dataset is from the "pending transfer" category but our system detects as "pending card payment" intent as both intents are quite similar. We shall try to address this issue in future. We have presently worked on a setting where novel intents appear in one step, we would strive to extend this framework to explore the dynamics of periodically evolving intents.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Mœl Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
K Chidananda Gowda and G Krishna. 1978. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):105-112.
Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020. Task-aware representation of sentences for generic text classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3202-3213.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409.
Ahsanul Haque, Latifur Khan, and Michael Baron. 2016. Sand: Semi-supervised adaptive novel class detection and classification over data stream. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
Dan Hendrycks and Kevin Gimpel. 2018. A baseline for detecting misclassified and out-of-distribution examples in neural networks.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. 2017. Learning to cluster in order to transfer across domains and tasks. arXiv preprint arXiv:1711.10125.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. 2019. Multi-class classification without multi-class labels. arXiv preprint arXiv:1901.00544.
Ilja Kuzborskij, Francesco Orabona, and Barbara Caputo. 2013. From n to n+ 1: Multiclass transfer incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3358-3365.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.02027.
Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. arXiv preprint arXiv:1906.00434.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Discovering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8360-8367.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural language understanding services for building conversational agents.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281-297. Oakland, CA, USA.
Mohammad Masud, Jing Gao, Latifur Khan, Jiawei Han, and Bhavani M Thuraisingham. 2010. Classification and novel class detection in concept-drifting data streams under time constraints. IEEE Transactions on Knowledge and Data Engineering, 23(6):859-874.
Xin Mu, Kai Ming Ting, and Zhi-Hua Zhou. 2017a. Classification under streaming emerging new classes: A solution using completely-random trees. IEEE Transactions on Knowledge and Data Engineering, 29(8):1605-1618.
Xin Mu, Feida Zhu, Juan Du, Ee-Peng Lim, and Zhi-Hua Zhou. 2017b. Streaming classification with emerging new class by class matrix sketching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. 2012. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7):1757-1772.
Lei Shu, Hu Xu, and Bing Liu. 2017. Doc: Deep open classification of text documents.
AB Siddique, Fuad Jamour, Luxun Xu, and Vagelis Hristidis. 2021. Generalized zero-shot intent detection via commonsense knowledge. In Proceedings of the 44th International ACM SIGIR Conference on
Research and Development in Information Retrieval, pages 1925-1929.
Yu Sun, Ke Tang, Leandro L Minku, Shuo Wang, and Xin Yao. 2016. Online ensemble learning of data streams with gradually evolved classes. IEEE Transactions on Knowledge and Data Engineering, 28(6):1532-1545.
Ming Tan, Yang Yu, Haoyu Wang, Dakuo Wang, Saloni Potdar, Shiyu Chang, and Mo Yu. 2019. Out-of-domain detection for low-resource text classification tasks. arXiv preprint arXiv:1909.05357.
Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2010. What is left to be understood in atis? In 2010 IEEE Spoken Language Technology Workshop, pages 19-24. IEEE.
Min Wang, Ke Fu, Fan Min, and Xiuyi Jia. 2020. Active learning through label error statistical methods. Knowledge-Based Systems, 189:105140.
Congying Xia, Wenpeng Yin, Yihao Feng, and Philip Yu. 2021. Incremental few-shot text classification with multi-round new classes: Formulation, dataset and system. arXiv preprint arXiv:2104.11882.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S Yu. 2018. Zero-shot user intent detection via capsule neural networks. arXiv preprint arXiv:1809.00385.
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487. PMLR.
Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020. A deep generative distance-based classifier for out-of-domain detection with mahalanobis space. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1452-1460.
Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. 2017. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In international conference on machine learning, pages 3861-3870. PMLR.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14365–14373.
Zhi-Hua Zhou and Zhao-Qian Chen. 2002. Hybrid decision tree. Knowledge-based systems, 15(8):515-528.