SlowGuess/NAACL_2022_OCR huggingface dataset:
H. W. Kuhn and Bryn Yaw. 1955. The hungarian method for the assignment problem. *Naval Res. Logist. Quart*, pages 83-97.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Soren Auer, and Christian Bizer. 2015. Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:167-195.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. 2021. Contrastive clustering. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8547-8555.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Discovering new intents via constrained deep adaptive clustering with cluster refinement. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8360-8367.
Samuel Louvan and Bernardo Magnini. 2020. Recent neural methods on slot filling and intent classification for task-oriented dialogue systems: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 480-496, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Edward Ma. 2019. Nlp augmentation. https://github.com/makcedward/nlpaug.
J. B. MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 281-297. University of California Press.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR.
Md Rashadul Hasan Rakib, Norbert Zeh, Magdalena Jankowska, and Evangelos Milios. 2020. Enhancement of short text clustering by iterative classification. In Natural Language Processing and Information Systems, pages 105-117, Cham. Springer International Publishing.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages
3982-3992, Hong Kong, China. Association for Computational Linguistics.
Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53-65.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked sequence to sequence pretraining for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936. PMLR.
Alexander Strehl and Joydeep Ghosh. 2002. Cluster ensembles - a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3:583-617.
Laurens van der Maaten and Geoffrey E. Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. 2020. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision.
Zhiguo Wang, Haitao Mi, and Abraham Ittycheriah. 2016. Semi-supervised clustering for short text via deep representation learning. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 31-39, Berlin, Germany. Association for Computational Linguistics.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. *ArXiv*, abs/2012.15466.
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016a. Unsupervised deep embedding for clustering analysis. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 478-487, New York, New York, USA. PMLR.
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016b. Unsupervised deep embedding for clustering analysis. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 478-487, New York, New York, USA. PMLR.
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 62-69, Denver, Colorado. Association for Computational Linguistics.
Jiaming Xu, Bo Xu, Peng Wang, Suncong Zheng, Guanhua Tian, and Jun Zhao. 2017. Self-taught convolutional neural networks for short text clustering. *Neural networks: the official journal of the International Neural Network Society*, 88:22-31.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. 2017. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 3861-3870. JMLR.org.
Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. 2010. Image clustering using local discriminant models and global integration. IEEE Transactions on Image Processing, 19(10):2761-2773.
Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021a. Supporting clustering with contrastive learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5419-5430, Online. Association for Computational Linguistics.
Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021b. TEXTOIR: An integrated and visualized platform for text open intent recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 167-174, Online. Association for Computational Linguistics.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021c. Discovering new intents with deep aligned clustering. Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14365-14373.
Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. CoRR, abs/1502.01710.
A.1 Unsupervised and Semi-supervised Clustering
A.1.1 Unsupervised Clustering
Building upon classical clustering techniques such as K-means (MacQueen, 1967) and Agglomerative Clustering (AG) (Chidananda Gowda and Krishna, 1978) several deep learning-based clustering techniques have recently been proposed in the literature. DEC (Xie et al., 2016b) is a two-step deep unsupervised clustering algorithm where the first step involves training a stacked autoencoder (SAE) and the second step involves updating
the SAE encoder and the cluster centers based on high confidence assignment of an utterance to cluster while using an auxiliary target distribution. In SAE-KM (Xie et al., 2016b), k-means is used to cluster the representations obtained from the encoder of the (SAE). DCN (Yang et al., 2017) is a joint deep unsupervised clustering algorithm performing joint representation learning and clustering. DAC (Chang et al., 2017) recasts the clustering problem into a binary pairwise classification framework to determine whether pairs of samples belong to the same cluster or not and the cosine similarity between pairs of samples is used to create pseudo-training data. DeepCluster (Caron et al., 2018) jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm such as k-means and uses the subsequent assignments as supervision to update the weights of the network. STC (Xu et al., 2017) is an approach for short-text clustering where learned representations are clustered using k-means. Self-Train (Hadifar et al., 2019) extends DEC for short-text clustering and uses weighted average of pre-trained word embeddings (Mikolov et al., 2013) to get text representations. Rakib et al. (2020) alternately use classification and outlier detection to improve the accuracy of existing short-text clustering algorithms. SCCL Zhang et al. (2021a) jointly perform representation learning and clustering via contrastive representation learning and minimize a modified version of the clustering loss proposed by Xie et al. (2016b).
A.1.2 Semi-supervised Clustering
PCK-means (Basu et al., 2004) is a semi-supervised clustering algorithm where labeled samples are used as pairwise constraints to improve clustering performance. KCL (Hsu et al., 2018) is a two-stage image clustering algorithm that uses a binary classification model trained on labeled data in the first phase to measure pair-wise image similarity and in the second stage, a clustering model is trained on unlabeled data by using the output of the binary classification model for supervision. The network is trained using a Kullback-Leibler divergence-based contrastive loss (KCL). Meta Classification Likelihood MCL (Hsu et al., 2019) leverages pairwise similarity between samples and optimizes a binary classifier for pairwise similarity prediction and through this process learns a multi-class classifier as a submodule. DTC
CLINC150  $(\mathcal{T} = 150, n = 112, m = 38)$  CLINC150  $OOS$ $(\mathcal{T} = 150, n = 112, m = 38)$
Approach	K'	KPred	ACC	NMI	K'	KPred	ACC	NMI
DSSCCSBERT	300	129.6	81.37	92.97	300	130.5	80.24	91.95
450	190.1	84.59	93.64	450	182.2	86.23	93.84
600	255.9	80.83	92.04	600	248.1	79.96	91.20
150	150	88.47	94.50	150	150	87.23	94.39
Table 6:  $DSSCC_{SBERT}$  with and without out-of-scope (OOS) utterances in user logs, where value of  $m$  is not known.
KIR	λ	ACC	NMI	ARI	ACC	NMI	ARI
75%	2	71.20	91.95	66.09	67.34	79.46	55.26
5	80.44	93.48	75.47	70.29	79.46	57.65
7	83.78	94.0	78.31	70.78	79.39	58.11
10	87.20	94.15	80.57	70.16	78.93	57.11
14	89.51	94.61	82.52	67.99	77.82	54.95
15	89.73	94.61	82.75	68.99	77.91	55.37
17	88.93	94.10	81.60	68.70	77.51	54.72
20	89.42	94.17	82.23	67.70	76.92	53.97
25	89.91	94.26	82.77	66.82	76.16	52.77
30	88.76	93.44	80.77	66.33	76.02	52.37
(Han et al., 2019) extend  $DEC$  to a semi-supervised scenario and also improve upon DEC by enforcing a representation bottleneck, temporal ensembling, and consistency.  $\mathbf{CDAC}+$  (Lin et al., 2020) is an end-to-end clustering method that incorporates pairwise constraints obtained from labeled utterances as prior knowledge to guide the clustering process and clusters are further refined by forcing the model to learn from high confidence assignments. DeepAligned (Zhang et al., 2021c) use labeled utterances from known intents to update BERT parameters and in the second step perform representation learning and K-means clustering alternately by minimizing cross-entropy loss over labeled and pseudo-labeled utterances treating each k-means cluster as one intent.
A.2 Representation of an utterance from PLM
We get the representation  $h_t =$  mean-pooling([CLS,  $T_1, T_2, \dots, T_l$ ]) of an utterance  $u_t$  consisting of  $l$  tokens from BERT/SBERT by applying mean-pooling over representations of all tokens including  $CLS$ , where  $T_j$  denotes representation corresponding to  $j^{th}$  token.
Table 7: Ablation with KIR (75%) on CLINC150 and BANKING77 where we try different values of  $\lambda$ . We use  $DSSCC_{SBERT}$  for ablation.
Dataset	Avg utterance length	z%
CLINC150	8.31	10
BANKING77	11.91	20
SNIPS	9.03	10
StackOverflow	9.18	10
DBPedia	29.97	30
Table 8: Data Augmentation Details.
A.3 Dataset Description and Details
We evaluate DSSCC on five datasets with a varying number of intents. And all of them are available in english language and released under creative Commons licences.
BANKING77 (Casanueva et al., 2020) is a fine-grained intent detection dataset from the banking domain comprising of 13,083 customer queries labelled with 77 intents.
CLINC150 (Larson et al., 2019) is a crowdsourced multi-domain (10 domains such as utility, travel etc.) intent detection dataset comprised of 23,700 queries with 22,500 in-scope queries labelled with 150 intents and 1,200 out-of-scope queries. For our experiments, we use both sets-CLINC150 which contains only in-scope queries, and CLINC-  $150_{\text{OOS}}$  which contains both in-scope and out-of-scope queries and we use the balanced version of the dataset.
SNIPS (Coucke et al., 2018) consists of 16000 crowd-sourced user utterances distributed across 7 intents. Out of 16k, 14484 utterances has been used for experimental purpose in the past.
StackOverflow: This dataset was originally released as part of a kaggle competition. Xu et al. (2015) used a subset of this dataset for short-text clustering. The dataset consists of 20,000 technical question titles distributed across 20 intents with 1k questions per intent.
DBPedia (Zhang and LeCun, 2015) is an ontology classification dataset constructed by picking 14
$^{2}$ https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow
non-overlapping classes from DBpedia 2014 Lehmann et al. (2015). Wang et al. (2016) used a subset of this dataset consisting of 14000 samples distributed across 14 classes.
We mention dataset statistics in table 9. We mention the train, validation, and test split details, that were used while performing the experiments for each dataset. Based, on the len and  $\mathcal{T}(n + m)$  attributes, we can see the variation and complexity of these datasets.
For the first phase, we follow the same pre-training steps outlined by Zhang et al. (2021c). For the second phase of DSSCC-E2E, an embedding of dimension  $(d*(n+m))$ , is used for the cluster centers where a  $d$  dimensional representation of an utterance  $u$  is obtained from the PLM and  $n+m$  represents the total number of clusters corresponding to known and new intents. For the HIL approach, two different linear heads/layers are used - one with dimension  $(d*(128))$  for Instance-Level Contrastive Learning and second with dimension  $(d*(n))$  for Supervised Representation Learning. For simulating a real-world problem of intent detection and discovery from user logs, we follow the experimental setting similar to Zhang et al. (2021c) where they assume that  $\mathrm{x}\% \in \{25\%, 50\%, 75\%\}$  of the total intents for a given dataset is known (we denote this number by  $n$ ) where  $\mathrm{x}$  is also referred to as the known intent ratio (KIR). The remaining number of intents  $(m)$  are considered novel. Accordingly, each dataset is divided into a  $\mathcal{D}_{train}$ ,  $\mathcal{D}_{val}$  and  $\mathcal{D}_{test}$  where  $\mathcal{D}_{train}$  contains  $10\%$  of labeled utterances per known intent and unlabeled utterances from both known and unknown intents.  $\mathcal{D}_{val}$ ,  $\mathcal{D}_{test}$  consists of utterances from known and new intents. We do two sets of experiments - one with a known value of  $n+m$  (number of total intents) and another one where the total number of intents is not known in advance. For a given dataset and KIR, we run the same experiment on ten different seeds and report the average ACC, NMI, and ARI on  $\mathcal{D}_{test}$ . For DSSCC-E2E, we use DSSCC-CH, DSSCC-KM for intent detection and discovery and report the best results based on majority voting over ACC, NMI and ARI. In a realistic scenario when ground truth is not available, one can use the Silhouette Score (Rousseeuw, 1987) to decide which inference strategy to use. For the
HIL approach, we get predictions from DBSCAN clustering after model convergence and perform inference after running the cluster merger algorithm. We use existing checkpoints of bert-base-uncased  $^{3}$  (Devlin et al., 2019) and stsb-roberta-base-v2  $^{4}$  (Reimers and Gurevych, 2019) as our PLM. Similar to (Zhang et al., 2021c), we freeze all but the last transformer layer parameters in our PLM for both phases to improve training efficiency and speed up the training process. We use the Adam optimizer (Kingma and Ba, 2014) to update PLM parameters and the learning rate is set to 5e-5 for PLM, 5e-3 for cluster centers in case of DSSCC-E2E and 5e-3 for both heads in case of HIL Approach. For all of our experiments, the batch size is kept at 400. For DSSCC-E2E, the entropy weight  $(\lambda)$  is set as 14.0. Whereas, For HIL, the minimum samples and epsilon for DBSCAN is kept as 3.0 and 0.09 respectively. We run all experiments on an Nvidia Titan A100 GPU. We use classification accuracy on the  $\mathcal{D}_{val}$  set for known intents as converge criteria for Phase 1. And for Phase 2 of DSSCC-E2E, we calculate the Silhouette Score Rousseeuw (1987) given utterance representations and corresponding predicted cluster-ids on  $\mathcal{D}_{train}$ . Whereas, for Phase 2 of HIL, we converge when the number of predicted clusters by DBSCAN clustering is minimum. We use early stopping with a patience value of 20.0 for both phases. For semi-supervised contrastive representation learning, similar to Zhang et al. (2021a), we use contextual augmenter Kobayashi (2018) to generate augmentations corresponding to unlabeled utterances where  $z\%$  of the words in an utterance are substituted with similar words. We use a suitable value of  $z\%$  for different datasets based on average utterance length as mentioned in the table 8 following the observations from Zhang et al. (2021a). We do this to preserve the semantics of an utterance while at the same time, substituting words in an utterance to create augmentations. We report the best results averaged over ten different seeds based on the inference details as mentioned in section 4.4. For our codebase, we have adapted existing SupContrast  $^{5}$  loss in the semi-supervised setting and also utilized data creation steps from Zhang et al. (2021c) $^{6}$ .
3https://huggingface.co/bert-base-uncased
4https://huggingface.co/sentence-transformers/stsb-roberta-base-v2
<sup>5</sup>https://github.com/HobbitLong/SupContrast
$^{6}$ https://github.com/thuiar/DeepAligned-Clustering
A.5 Results on Intent Detection and Discovery
We have also reported results on five public datasets with our proposed approach on intent detection and intent discovery separately for the case when  $m$  is known, are shown in table 11. If we assume that intent detection and discovery are two separate problems, we decouple the results from our joint approach (after training) to see the contribution of DSSCC on both tasks. It is clear from the results that, even with very few labeled utterances from known intents, our model maintains the performance on the known intents with at least  $83\%$  clustering accuracy on all five datasets. From the results on Intent Discovery, except for BANKING77, DSSCC (Ours) gets at least  $74\%$  clustering accuracy on 4 datasets. The low performance on intent discovery in BANKING77 is attributed to the complexity of the dataset where all intents are part of one larger domain, i.e., Banking. Whereas, in CLINC150, the intents belong to multiple domains as mentioned in section A.3.
A.6 DSSCC-KM vs DSSCC-CH
We use both inference strategies, i.e., DSSCC-KM and DSSCC-CH to obtain results for all experiments described in section 6 and report ACC, NMI and ARI as shown in Table 12, 13 and 15. DSSCC-CH outperforms DSSCC-KM on CLINC150, SNIPS and StackOverflow whereas DSSCC-KM gives better results on BANKING77 and DBPedia. This inconsistency between the behaviour of DSSCC-CH and DSSCC-KM can be attributed to complexity of a given dataset, i.e., DSSCC-KM outperforms DSSCC-CH on BANKING77 (single domain dataset) and DBpedia whereas DSSCC-CH outperforms DSSCC-KM on CLINC150 (multi-domain dataset).
In realistic scenario where ground truth is not available, one can use Silhouette Score (SS) to choose between DSSCC-CH and DSSCC-KM. As shown in Fig. 6, when the difference between SS corresponding to DSSCC-CH and DSSCC-KM is significant, then one should choose inference strategy which gives higher SS. And when the difference between SS score corresponding to DSSCC-CH and DSSCC-KM is not significant, then one can choose DSSCC-CH for inference, as shown in Fig 5. Above mentioned approach for inference strategy selection correlates with the selection done by majority voting (over ACC, NMI and ARI).
Figure 5: CLINC150 (KIR=75%) DSSCC-CH vs DSSCC-KM where we use SBERT as PLM. The left subfigure shows silhouette scores over different seeds and the right subfigure shows ACC, NMI, and ARI with DSSCC-KM and DSSCC-CH over these seeds.
Figure 6: BANKING77 (KIR=75%) DSSCC-CH vs DSSCC-KM where we use SBERT as PLM. The left subfigure shows silhouette scores over different seeds and the right subfigure shows ACC, NMI, and ARI with DSSCC-KM and DSSCC-CH over these seeds.
A.7 Representations from SBERT vs  $DSSCC_{SBERT}$
To showcase the effectiveness of representations learnt by  $DSSCC_{SBERT}$ , we plot the utterance embeddings  $(\pmb{h}_t)$  with ground truth labels for all five datasets as shown in Fig. 7 and Fig. 8. Initial and final representations correspond to utterance embeddings obtained from SBERT and  $DSSCC_{SBERT}$  respectively. It can be observed that for all five datasets, intents are clearly separable with final representations as compared to initial representations.
Figure 7: TSNE (van der Maaten and Hinton, 2008) plot for CLINC150 and Banking77 (75% KIR) before and after DSSCC Training.
Dataset	Dtrain	Dval	Dtest	len(max/mean)	T(n+m)
CLINC150	18,000	2,250	2,250	25/8.31	150(112+38)
CLINC150OOS	19,000	2,250	2,450	25/8.32	150(112+38)
BANKING77	9,003	1,000	3,080	79/11.91	77 (58+19)
SNIPS	13,084	700	700	35/9.03	7 (5+2)
StackOverflow	18,000	1,000	1,000	41/9.18	20 (15+5)
DBPedia	12,600	700	700	54/29.97	14 (11+3)
Table 9: Dataset Details.  $\mathcal{D}_{train}$ ,  $\mathcal{D}_{test}$ ,  $\mathcal{D}_{val}$ : number of examples in train, validation and test set respectively; len(max/mean): maximum sentence length / mean sentence length;  $\mathcal{T}(n + m)$ : Total number of classes (Known Intents + New Intents) in case of 75% KIR
SNIPS	StackOverflow	DBPedia
KIR	Approach	ACC	NMI	ACC	NMI	ACC	NMI
25%	\( \mathbf{DABERT} \)	86.21	80.42	69.66	70.23	85.89	88.98
\( \mathbf{DASBERT} \)	81.16	77.33	72.64	73.43	83.70	85.40
\( \mathbf{DSSCCSBERT} \)	94.33	89.30	81.72	76.57	89.44	89.25
50%	\( \mathbf{DABERT} \)	85.69	83.03	72.89	74.49	88.63	91.24
\( \mathbf{DASBERT} \)	88.83	84.19	73.07	74.08	87.29	88.80
\( \mathbf{DSSCCSBERT} \)	95.20	91.07	82.43	77.30	92.14	92.70
75%	\( \mathbf{DABERT} \)	90.10	86.94	74.51	76.24	92.17	93.25
\( \mathbf{DASBERT} \)	92.70	88.22	75.50	75.90	91.17	91.14
\( \mathbf{DSSCCSBERT} \)	94.87	90.44	82.65	77.08	92.73	92.58
Table 10: Intent detection and discovery results on three datasets, i.e., SNIPS, StackOverflow and DBPedia where we computed results corresponding to DeepAligned  ${}_{BERT}$  (DA  ${}_{BERT}$  ),DeepAligned  ${}_{SBERT}$  (DASBERT) using code provided by Zhang et al. (2021c)
Figure 8: TSNE plot for StackOverflow, DBPedia and SNIPS (75% KIR) before and after DSSCC Training.
Intent Detection Intent Discovery
KIR	Dataset	ACC	NMI	ARI	ACC	NMI	ARI
25%	CLINC150	91.14	96.87	92.84	76.70	90.66	72.14
BANKING77	78.11	88.53	79.27	60.61	79.09	53.37
SNIPS	94.36	87.36	88.98	94.32	88.85	89.0
StackOverflow	86.4	78.82	80.15	80.16	75.26	68.33
DBPedia	96.0	93.25	94.12	86.82	87.74	82.28
50%	CLINC150	92.03	96.59	91.13	74.97	91.02	74.62
BANKING77	79.66	88.66	77.19	59.35	80.18	58.0
SNIPS	95.27	90.78	91.07	95.33	89.90	91.21
StackOverflow	86.72	81.37	79.50	78.14	72.96	67.99
DBPedia	95.26	94.96	94.81	88.40	91.07	88.95
75%	CLINC150	92.73	96.44	89.89	75.91	91.52	78.77
BANKING77	83.60	87.61	74.99	57.72	80.71	62.39
SNIPS	95.54	91.99	91.84	93.17	81.92	84.77
StackOverflow	85.37	79.76	75.22	74.48	66.92	64.48
DBPedia	95.72	94.73	94.09	85.25	86.66	84.02