SlowGuess/NAACL_2021_OCR huggingface dataset:
We also evaluate our approaches on Quora Intent Corpus<sup>5</sup> (Haponchyk et al., 2018) based on 1,334 questions from Quora duplicate detection competition<sup>6</sup>. This corpus contains 270, 146 and 212 question clusters respectively in the training, dev. and test parts. The clusters in each part are split into samples: training – in 10 samples, both dev. and test – in 5. A part of the test set is also provided with an expert annotation. We refer to the whole test set with labels automatically derived from Quora annotation as automatic test set, and to its part with expert annotation – as manual
5https://ikernels-portal.disi-unitit/ repository/intent-ga
$^{6}$ https://www.kaggle.com/c/quora-question-pairs
Type of Supervision	Model	Clustering measure	CEAFe
Precision	Recall	F1
None	Spectral clustering	tfidf	78.44	78.38	78.41±0.24	71.55±0.57
CNN	75.75	75.75	75.75±0.29	69.09±0.54
Clustering Function (instance similarity)	BERT	75.74	75.74	75.74±0.21	69.02±0.36
Kruskal	CNN	97.72	79.00	87.28±0.82	80.23±1.15
BERT	79.95	91.30	85.25±0.46	78.59±0.61
Our Supervised Clustering	NSC-CNN	88.83	89.61	89.19±0.99	82.66±1.07
NSC-BERT	94.98	96.36	95.65±0.82	91.76±1.51
Table 1: Comparison of clustering models: completely unsupervised, using supervised instance similarity function, and our supervised clustering on the test set of IC&OOS by Larson et al. (2019); disjoint scenario.
Models: We experiment with two variants of our Neural model for Supervised Clustering (NSC), based on two ways to encode question pairs outlined in Sec. 5.1: (i) NSC-CNN, using word and word overlap embeddings, and (ii) NSC-BERT, using BERT embeddings of question pairs.
For NSC-CNN, we employ fastText $^7$  word embeddings, in dimension 300, pre-trained for English language on Wikipedia (Bojanowski et al., 2017). We set the max length of questions to 50 and pad the shorter questions on the right. The size of the hidden layer is set to  $\frac{1}{3}$  of the size of the input layer. The convolution filter width varies from 1 to 3.
For NSC-BERT, we use BERT<sub>BASE</sub> model, which we train for 3 epochs for fine-tuning on the question pair classification task.
Since the training samples vary in size, we clip the gradients to have their  $L_{\infty}$  norm less than or equal to 1. This is to prevent the updates being dominated by the samples of bigger size.
Evaluation: We follow the evaluation setting of Haponchyk et al. (2018). Thus, we compute (i) clustering F1 measure, based on assigning each cluster to the most frequent (gold/output) cluster, (ii) coreference resolution  $\mathrm{CEAF}_e$  score (Luo, 2005; Cai and Strube, 2010).
Parameterization: We use dev. set for tuning the loss parameter  $r$ , which takes values from  $\{0.1, 0.5, 1.0\}$ , and selecting the best epoch with respect to clustering F1.
Baselines We consider a number of baselines based on the pairwise query similarities. We experiment with the following sources of pairwise signals: (i) tfidf scores, (ii) outputs of the binary question pair classifier, which we train in two modalities, CNN and BERT.
We group the pairwise signals into a clustering output using spectral clustering algorithm (Ng et al.,
2001) (implementation from  $smile^8$  library), which we run on a matrix of pairwise similarities between data points. Spectral clustering is unsupervised, and requires the indication of the number of clusters  $k$ . For each sample, we set the parameter  $k$  to the gold number of clusters. This means that we are computing an upper bound and unrealistic performance, which can be used to provide a meaningful comparison (especially if our approach outperforms it). As an alternative to spectral clustering, we run Kruskal's algorithm on the graph of pairwise edges, using 0.5 as a threshold for the question pair classifier scores on them; pairs having the scores lower 0.5 are neglected.
7.2 Experiments on IC&OOS task
In Tab. 1, we present the results on IC&OOS dataset averaged over 10 different sample splits, obtained with 10 different random seeds. First, we note that NSC consistently improves over all the baselines in terms of both F1 and CEAF. It also shows a good precision/recall balance.
The significantly lower results of the unsupervised baseline, spectral clustering+tfidf, suggest that, in IC&OOS, we are supposedly dealing with a rather non-trivial task, where queries expressing the same intent do not necessarily have surface closeness. Even if the model is aware of the true number of present intents in a sample (gold  $k$ ).
The other four baselines capitalize on training a supervised scoring function for query pairs, to be further used as a clustering criterion. From our experiments with IC&OOS data, we conclude that it is also not trivial to train a pairwise classifier to convey a notion of semantic similarity to the pairs of queries from unseen classes. This is reflected in the results of using a supervised similarity function. The performance of spectral clustering on the output of the pairwise classifier, equally low for CNN and BERT, and lower than model using tfidf
7https://fasttext.cc/docs/en/pretrained-vectors.   
html
<sup>8</sup>http://haifengl.github.io/smile/
Model	Clustering measure	CEAFe
Precision	Recall	F1
LSSVM	84.92	51.76	64.32	49.72
LSP	71.36	89.45	79.38	59.99
LSPpy	70.80	90.00	79.22 ± 0.33	59.82
NSC-CNN	80.25	82.16	81.12 ± 1.76	62.80
NSC-BERT	86.93	72.96	79.19 ± 1.41	63.89
Table 2: Comparison of our neural models to the structural baselines on the manual test set of Quora Intent Corpus.
Model	Clustering measure	CEAFe
Precision	Recall	F1
LSSVM	80.16	77.81	78.96	63.68
LSP	66.06	91.64	76.78	51.50
LSPpy	65.32	91.47	76.18 ± 0.63	50.62
NSC-CNN	71.44	91.10	80.07 ± 0.28	59.08
NSC-BERT	88.01	96.96	92.23 ± 0.98	86.07
scores, is clear evidence for this. When we run Kruskal's on top of the output of the pairwise classifier, we observe a huge bias, towards precision in case of CNN and recall in case of BERT. The use of a threshold does not seem robust, when training examples (query pairs) are treated as independent.
In NSC, we assume, this problem is mitigated by "collaborative" updates of the structural loss. Overall, we note the impressive performance of NSC, especially when fed by BERT. A clustering F1 of 95.65 suggests that NSC can replicate the clusters of questions that the human annotator/knowledge engineer devised.
7.3 Experiments on Quora Intent Corpus
We run each model with 10 different random seeds for shuffling training examples and report the averaged results on the manual, Tab. 2, and automatic test sets, Tab. 3.
7.3.1 Baselines
We compare NSC with two state-of-the-art structural approaches, LSSVM and LSP proposed in (Haponchyk et al., 2018), reporting their numbers on the same data.  $\mathrm{LSP}_{py}$  is our LSP reimplementation in python using text similarity. We trained  $\mathrm{LSP}_{py}$  for 100 epochs.
7.3.2 Results
NSC-CNN improves over the state of the art on both the test sets for both measures. On the manual test set, NSC-BERT achieves, as expected, higher CEAF. One possible explanation for its lower F1 on the manual test set is its small size, which probably does not enable an accurate evaluation.
Table 3: Comparison of our neural models to the structural baselines on the automatic test set of Quora Intent corpus.
BiMPM	Our fine-tuned BERT
Accuracy	88.17	90.88
Table 4: Accuracy comparison on question duplicate detection task on Quora split by Wang et al. (2017).
Model	Inclass Acc.	OOS Recall
Larson et al. (2019)
NN + avg. FastText emb.	84.50	23.20
CNN + Glove emb.	88.90	22.20
BERT	96.40	40.90
Our Models
CNN	80.20	28.88
BERT	94.87	38.80
Table 5: Comparison of our intent classification baselines to the intent classification models from Larson et al. (2019) on IC&OOS test set.
In contrast, on the evaluation over the automatic test set, NSC-BERT largely outperforms any model, according to any measure. This is mainly due to the fact that the automatic clusters are more consistent with the information present in the training data, used for fine-tuning BERT. Indeed, we fine-tuned  $\mathrm{BERT}_{\mathrm{BASE}}$  on the full Quora dataset of question pairs<sup>9</sup> (except for the pairs containing questions from development and test parts of Quora Intent Corpus). For the sake of transparency, in Tab. 4, we report the accuracy of the fine-tuned BERT on Quora split by Wang et al. (2017) compared to the official results of their BiMPM model.
We believe the result of NSC-BERT is promising, and, in the scope of intent detection, by not being bounded to a particular set of intents, it contributes to the existing neural solutions (Xia et al., 2018; Lin and Xu, 2019; Lin et al., 2020).
7.4 Deeper Analysis
In this section, we investigate the general clustering ability of NSC, and in this way, enable the comparison to the upper bound of intent detection, i.e., the intent classifier, and list its most common mistakes.
7.4.1 How good is the clustering per se?
Here, we address the standard IC&OOS scenario with the original class distribution of dataset, where all the 150 intent classes are equally presented in the data. Moreover, we explore the upper bound to any clustering algorithm, i.e., the use of a supervised classifier in an unrealistic (useless for clustering) scenario, that is, having in the training data, all the clusters (classes) to be discovered. To carry out this comparison, we trained two intent classifier
$^{9}$ https://www.quora.com/q/quoradata/ First-Quora-Dataset-Release-Question-Pairs
Model	Clustering measure	CEAFe	OOS Recall
Precision	Recall	F1
CNN intent classifier	88.70	93.14	90.76±0.93	86.12±0.75	28.88
BERT intent classifier	90.53	98.36	94.29±0.24	89.10±0.40	38.80
NSC-CNN	89.32	91.25	90.24±1.09	84.86±1.34	85.49±3.94
NSC-BERT	93.58	97.27	95.38±0.34	92.05±0.58	71.45±4.35
Table 6: Comparison of the neural clusterings models to the classification baselines on the test set of IC&OOS dataset by Larson et al. (2019); full scenario.
models, CNN and BERT, with  $150 + 1$  (OOS) target classes. In Tab. 5, we report their performance in terms of in-class accuracy and OOS recall. We also report the performance of the classification models from Larson et al. (2019) for reference. As it can be observed, our models perform comparably, e.g., our BERT model is just 1.5 points behind.
We trained NSC on the same data, split into samples, so that we could compare to the above classifiers. For this purpose, we follow our sampling procedure in Sec. 6.1, this time keeping all the classes, which gives us around 79 and 32 samples for training and dev., respectively, and 66 samples for test. To keep the two types of systems aligned, we evaluate the classifiers also in terms of clustering F1 on the same test samples (of size  $M$ ), which are then averaged. Namely, we consider queries, within a sample, predicted by a classifier model, as the same class to form a distinct cluster, while those predicted as OOS - singletons.
We note that (i) as expected, the results of NSC in Tab. 6 improve with respect to the completely disjoint setting (Tab.1). (ii) NSC-CNN is able to almost replicate the result of the CNN classifier in terms of F1, yielding only 1.3 of a point in terms of CEAF. (iii) Interestingly, the OOS Recall is more than  $85\%$  (2-3 times the one of the classifiers), which means that  $85\%$  of all OOS queries were detected by NSC-CNN (predicted singletons in their corresponding samples). Although, we recognize that it can be easier to detect OOS queries in a small sample than in a big set. (iv) NSC-BERT improves over the classifier model on the test samples by 1.5 in terms of clustering F1 and by more than 3 CEAF points, also achieving a better precision/recall balance (same as for CNN modality). We hypothesize here an advantage of the supervised clustering model might lie over the classification models, which are generally not as well adaptive to class imbalance in data. (v) Again, the NSC-BERT highly improves (at least 2 times) the recall of the classifier for the OOS task.
7.4.2 Error analysis
Analysing the output of NSC (here, we limit the discussion to NSC-BERT in the disjoint scenario of Sec. 7.2), we discovered that the majority of the mistakes made by the clustering algorithm can be traced back to several interpretable causes.
A trivial case of word overlap or generally string matching in Ex. 4 made NSC put the examples of seemingly distinct classes together. Actual ground truth intent classes are denoted in parentheses.
(4) cluster:
(1) what is the reason humans even exist (meaning_of_life)
(2) let me know if you are a human or are a computer (are_you_a.Bot)
Next, we find the presence of the word-indicators of the same semantic category, i.e., SPEED, in Ex. 5, that misled NSC.
(5) cluster:
(1) speak more quickly (change_speed)
(2) i'm in the mood for slow songs and nothing else (play_music)
(3) talk faster (change_speed)
A frequent type of NSC's mistakes is merging together instances of different intent classes which belong to the same topic domain, especially in case of rather close subtopics as in cluster Ex.-s 6-7.
(6) cluster:
(1) put on my 90s playlist (play_music)
(2) put on some metallica music (play_music)
(3) what kind of music on the speaker now (what_song)
(7) cluster:
(1) how do i freeze my bank account (freeze_account)
(2) why is there a stop on my deposit account (accountBlocked)
In addition, Ex. 7 has another complicating factor of using semantically very close expressions for distinct intent concepts. Right the opposite situation of erroneously splitting the instances of the same intent class is also common, as in Ex.-s 8-9.
(8) cluster:
(1) bye bye then (goodbye)
(9) cluster:
(1) good speaking to you (goodbye)
(2) it was great talking to you (goodbye)
In general, we assume, that the last two types of mistakes can be reduced if the model sees on training the data from the corresponding intent classes.
NSC also drew some (not absolutely meaningless) connections between OOS queries (Ex. 10).
(10) cluster:
(1) what is the highest quality carpet available (oos)
(2) find schematics for ikea desk assembly (oos)
(3) i have a super runny nose and want to find a doctor (oos)
(4) what was the latest tremor on the richter scale (oos)
And finally, the clustering decision in Ex. 11 potentially highlights an annotation error of query (2) being a false positive OOS.
(11) cluster:
(1) where is the closest mcdonald's to foxwoods casino (directions)
(2) where is the closest driving range (oos)
7.5 Discussion
In this section, we discuss some of the important findings of our paper: First, our experiments suggest that the transformer model boosts the performance of our clustering approach. This is justified by the mainstream research: with respect to the standard embeddings (word2vec, glove,.), transformer models provide contextual representation of words, i.e., the embedding of a word is defined with respect to the others that are in the same piece of text. They provide a very powerful representation of pieces of text. Thus, we can obtain a precise similarity between pairs of questions. Thanks to our structural loss function, we can back-propagate structure properties of the entire cluster back to the transformer models so that we enrich even more its contextual similarity.
Second, in the field of dialog systems, our approach can be extended to jointly predict intent and slot attributes. NSC can use information about slots and the background knowledge given by attributes and values, to cluster questions into intents. The latter will be then more related to the specific task defined by the available slot information. Conversely, if we suppose the developer has already the intents, our clustering algorithm could be used to cluster values into attributes. Then, since NSC can reach performance similar to supervised classification methods, it would be interesting to see if it can be more accurate than them, considering the
critical problems of transfer learning (i.e., when the data for training is different from the one the deployed system receives).
Third, we showed the performance on NSC exactly on unseen clusters. Our approach only uses some clusters of the data for training (each cluster is a training example). Then, it can predict unseen clusters in the test set. In other words, our models generalize what they learn from some clusters to unseen clusters.
Finally, given one of our models trained on a set of clusters, we can easily continue its training with new examples, i.e., new training clusters, as our neural architecture is an online framework. One main scalability question could be: Given one domain for which we have clusters to train our approach, how can we scale to other domains?
We will need new clusters for the new domains, i.e., target domain data, which is typically used for effective transfer learning. This does not mean that we need a large number of clusters, we just need some of them to transfer our clustering model from one domain to another. The transferred models will be able to predict many more new clusters from the new target domain.
In this work, we firstly proposed supervised neural clustering based on traditional LSSVM and LSP models, which hinge on optimizing the structural margin loss. This extends the structured prediction methods for supervised clustering to a neural setting. Our experiments on IC&OOS and Quora Intent Corpora show an impressive improvement over the state of the art,  $17.24\%$  absolute over unsupervised models, and  $8\%$  points more than our proposed semi-supervised approaches. This suggests that our neural structured prediction can (i) effectively optimize a structural clustering objective function on structured examples, such as sets of questions for intent detection, and (ii) uncover clusters of questions of unseen classes, i.e., potential intents not seen in training.
We would like to thank the anonymous reviewers as well as the entire PC for their valuable work.
Steven M Beitzel, Eric C Jensen, David D Lewis, Abdur Chowdhury, and Ophir Frieder. 2007. Automatic classification of web queries using very large unlabeled query logs. ACM Transactions on Information Systems (TOIS), 25(2):9.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL '10, pages 28-36, Stroudsburg, PA, USA. Association for Computational Linguistics.
P. Deepak. 2016. Mixkmeans: Clustering question-answer archives. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1576-1585. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Greg Durrett and Dan Klein. 2015. Neural crf parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 302-312. Association for Computational Linguistics.
Eraldo Rezende Fernandes, Cicero Nogueira dos Santos, and Ruy Luiz Milidiú. 2014. Latent trees for coreference resolution. Computational Linguistics, 40(4):801-835.
Thomas Finley and Thorsten Joachims. 2005. Supervised clustering with support vector machines. In ICML '05: Proceedings of the 22nd international conference on Machine learning, pages 217-224, New York, NY, USA. ACM.
Iryna Haponchyk, Antonio Uva, Seunghak Yu, Olga Uryupina, and Alessandro Moschitti. 2018. Supervised clustering of questions into intents for dialog system applications. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2310-2321. Association for Computational Linguistics.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions
of the Association for Computational Linguistics, 4:313-327.
Joseph Bernard Kruskal. 1956. On the Shortest Spanning Subtree of a Graph and the Traveling Salesman Problem. In Proceedings of the American Mathematical Society, 7.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
Yann LeCun, Sumit Chopra, Raia Hadsell, Fu Jie Huang, and et al. 2006. A tutorial on energy-based learning. In PREDICTING STRUCTURED DATA. MIT Press.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197. Association for Computational Linguistics.
Xiao Li, Ye-Yi Wang, and Alex Acero. 2008. Learning query intent from regularized click graphs. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 339-346. ACM.
Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5491-5496, Florence, Italy. Association for Computational Linguistics.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Discovering new intents via constrained deep adaptive clustering with cluster refinement. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8360-8367. AAAI Press.
Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT '05, pages 25-32, Stroudsburg, PA, USA. Association for Computational Linguistics.
Ruy Luiz Milidiú and Rafael Rocha. 2018. Structured prediction networks through latent cost learning. 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pages 645-649.