SlowGuess/NAACL_2021_OCR huggingface dataset:
Iryna Haponchyk
DISI, University of Trento
Povo (TN), Italy
gaponchik. irina@gmail.com
Previous pre-neural work on structured prediction has produced very effective supervised clustering algorithms using linear classifiers, e.g., structured SVM or perceptron. However, these cannot exploit the representation learning ability of neural networks, which would make supervised clustering even more powerful, i.e., general clustering patterns can be learned automatically. In this paper, we design neural networks based on latent structured prediction loss and Transformer models to approach supervised clustering. We tested our methods on the task of automatically recreating categories of intents from publicly available question intent corpora. The results show that our approach delivers  $95.65\%$  of F1, outperforming the state of the art by  $17.24\%$ .
Recent years have witnessed a vast spread of virtual assistants, such as Google Home, Siri and Alexa, which are based on the research areas of Conversational Agents and Question Answering. When designing such systems, the creation of classes of expected questions, aka intents, is essential for building the main states of a dialog manager. In particular, when an assistant is designed for a specific domain, a knowledge engineer needs to analyze typical user's questions, answered by human operators. This work would be greatly sped up, if the engineers could have questions clustered according to the different topics they ask for. For example, the following questions/request from the intent dataset by Larson et al. (2019):
i want to switch to direct deposit
set up direct deposit for me
how do i go about setting up direct deposit
all have a common intent of making a direct deposit. Thus, the dialog designer will create this intent, if the cluster captures a large number of requests.
However, for being effective, the clustering algorithm must demonstrate a sufficient accuracy, which is often not the case for completely unsupervised methods. Thus, supervised clustering (Finley and Joachims, 2005), which exploits some training data of the target domain, e.g., previously designed clusters, to discover new clusters, is a viable approach. A seminal work on structured prediction was Latent Structural Support Vector Machines (LSSVM) by Yu and Joachims (2009). Recently, Haponchyk et al. (2018) have shown that LSSVM as well as the Latent Structured Perceptron (LSP) by Fernandes et al. (2014), originally designed for coreference resolution, were also effective, when provided with the appropriate node similarity function, for clustering questions into intents. These approaches used traditional feature engineering (question similarity) and a linear classifier, i.e., SVM, which can be highly improved by neural networks, and pre-trained Transformers, e.g., Devlin et al. (2019). Indeed, neural models enable representation learning, which can amplify the generalization ability of supervised clustering algorithms.
In this paper, we design neural supervised clustering (NSC) models using the structured prediction algorithms, LSSVM and LSP. These are based on a latent representation of clusters using graph structures, which are used to compute an augmented loss. The latter, in turn, is used together with the model score to globally select the max-violating constraint at each learning step. This is the clustering that maximally penalizes the current model, which is used for a model update. We apply the same idea by computing the margin loss for our neural model and then back-propagating it, as any other differentiable loss. The augmented loss does not depend on the neural model, thus our approach can be applied to arbitrary learning settings.
We applied NSC to two different question intent clustering tasks, defined by two datasets: IC&OOS (Larson et al., 2019), which is an intent classifica
Association for Computational Linguistics: Human Language Technologies, pages 3364-3374
tion corpus, and Quora Intent Corpus (Haponchyk et al., 2018). Another interesting contribution of our work is the creation of a training set for clustering from IC&OOS, which enables an effective training of NSC. Our corpus and software are available to the research community<sup>1</sup>.
The comparative results of NSC using traditional CNN networks and Transformer models against traditional methods, e.g., spectral clustering, show an impressive boost in F1 of our NSC-BERT model, e.g.,  $95.65\%$  vs.  $78.38\%$ , more than  $17\%$  of improvement over the best spectral clustering method. This accuracy enables the use of our approach for dialog applications and opens up further directions for other clustering tasks.
This paper touches two main research areas: structured prediction, in particular with neural models, and intent clustering, which are described below.
Structured prediction has shown powerful machine learning algorithms for solving NLP tasks requiring complex output, e.g., syntactic parsing (Smith, 2011), coreference resolution (Yu and Joachims, 2009; Fernandes et al., 2014). This work has mainly regarded traditional frameworks, e.g., SVMs, CRF, perceptron. Only little work has been devoted to the integration of the above theory in neural networks (LeCun et al., 2006; Durrett and Klein, 2015; Weiss et al., 2015; Kiperwasser and Goldberg, 2016; Peng et al., 2018; Milidiú and Rocha, 2018; Xu et al., 2018; Wang et al., 2019), and, to the best of our knowledge, none to supervised clustering.
This is partially due to the fact that local solutions have usually produced optimal results. For example, in case of supervised clustering, it is difficult to design a loss function that captures the global information about the clusters. Work in neural coreference resolution, e.g., (Lee et al., 2017), uses simple losses, which deliver state-of-the-art results but do not strictly take into account the cluster structure. Secondly, this is also due to the complexity associated with adapting the methods from previous work to neural frameworks. For example, using ILP (Roth and Yih, 2004) for clustering inference in SFIGOT (Peng et al., 2018), which facilitates the backpropagation through argmax based on a projection onto the feasible set of structured outputs, would inevitably require reducing the com
putational overhead (Miyauchi et al., 2018).
On the line of research of question clustering, Wen et al. (2001) proposed to cluster queries with respect to a group of web pages frequently selected by users. Deepak (2016) describes a k-means like algorithm, MiXKmeans, that can cluster threads in Community Question Answering websites. These methods are unsupervised and, thus, are likely sensitive to setting the optimal number of clusters or to a heuristic adopted for the clustering criterion.
Also among the classification approaches, there are semi-supervised and mixed classification methods which advance on the use of vast amounts of unlabelled queries. Li et al. (2008) classify unlabeled queries using their proximity to labeled queries in a click graph. Beitzel et al. (2007) classify queries from logs into topics using supervised or unsupervised methods.
The following classification approaches address new emerging intents. Lin and Xu (2019) enable a neural model to detect unknown intents as outliers using a novelty detection technique. This model, however, does not have the capability to distinguish between different unknown intents. Xia et al. (2018) devise a capsule neural network able to discriminate between different emerging intents. Its zero-shot learning ability critically depends on the definition of a similarity between existing and new intents. Our approach does not hold any explicit representation of intents.
The recent work by Lin et al. (2020) proposes a deep intent clustering model which takes advantage of labeled data for discovering new user intents, but it requires the indication of the exact number of output clusters. Finally, Zhang et al. (2021) propose Deep Aligned Clustering, a semisupervised method, to discover new intents using limited knowledge over intent data. We believe their approach is completely compatible with ours, i.e., our supervised clustering models can be integrated in their approach to improve intent discovering.
3 Structured Output for Clustering
LSSVMs train a clustering function from a series of training examples  $\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i = 1}^n$ , where  $\mathbf{x}_i$  are input sets of elements, and  $\mathbf{y}_i$  are structured outputs, i.e., gold clusters. This function applied to unseen elements  $\mathbf{x}$  predicts their clusters  $\mathbf{y}$ .
3.1 Cluster inference
The clustering  $\mathbf{y}$  of a set  $\mathbf{x}$  is inferred over a fully-connected graph  $G$ , which nodes represent the el
$^{1}$ https://github.com/iKernels/intent-qa
ements  $x_{k}$  of  $\mathbf{x}$ , and edges  $\mathbf{e} = (x_i, x_j)$  - the pairwise links between them. The inference step consists in finding a maximum spanning forest  $\mathbf{h}$  on  $G$ , e.g., using Kruskal's algorithm (Kruskal, 1956). The nodes appearing in the same connected component (tree) in  $\mathbf{h}$  are placed together in the same cluster in  $\mathbf{y}$  (deterministically obtained from  $\mathbf{h}$ ).
3.2 Learning
The approach learns a linear scoring function which decomposes over the edges of  $\mathbf{h}$ :
$$
s _ {\mathbf {w}} (\mathbf {x}, \mathbf {y}, \mathbf {h}) = \sum_ {\mathbf {e} = (x _ {i}, x _ {j}) \in \mathbf {h}} \mathbf {w} \cdot \phi (\mathbf {e}), \tag {1}
$$
where  $\phi (\mathbf{e})$  is a feature representation for edge e, describing a pair of elements of  $\mathbf{x}$ . Graph structures  $\mathbf{h}$  are incorporated as latent variables into the latent formulation of LSSVM. Haponchyk et al. (2018) adapt the latent structured perceptron (LSP) by Fernandes et al. (2014) to the graph structures  $\mathbf{h}$  and apply the approach to question pairs,  $(q_{i}, q_{j})$ , to cluster sets of questions into different user intents; we compare to these methods.
4 Neural Structured Output Clustering
We propose a model for optimizing a structural clustering loss with neural networks.
4.1 Global max-margin objective
As a standard practice in structured prediction, our goal is to train a model with a scoring function  $s_{\theta}$  such that the correct clustering  $\mathbf{y}$  is scored higher than incorrect clusterings  $\hat{\mathbf{y}}$ . LSSVM optimizes an upper bound,  $E$ , on the structural loss  $\Delta$ , which, in general terms, can be rewritten using the parameters  $\theta$  as:
$$
\begin{array}{l} \Delta (\mathbf {y}, \hat {\mathbf {y}} (\theta), \hat {\mathbf {h}} (\theta)) \leq E (\mathbf {y}, \hat {\mathbf {y}} (\theta), \hat {\mathbf {h}} (\theta)) = \\ \max  _ {\hat {\mathbf {y}}} \left[ \Delta (\mathbf {y}, \hat {\mathbf {y}}, \hat {\mathbf {h}}) + s _ {\theta} (\mathbf {x}, \hat {\mathbf {y}}, \hat {\mathbf {h}}) \right] - \tag {2} \\ (\hat {\mathbf {y}}, \mathbf {h}) \in Y \times H \qquad \qquad \max  _ {\mathbf {h} \in H} s _ {\theta} (\mathbf {x}, \mathbf {y}, \mathbf {h}), \\ \end{array}
$$
where  $\hat{\mathbf{y}} (\theta)$  is an output of the model with its auxiliary latent structure  $\hat{\mathbf{h}} (\theta)$  .  $Y$  and  $H$  are the spaces of all possible clusters and latent trees;  $\Delta (\mathbf{y},\hat{\mathbf{y}},\hat{\mathbf{h}})$  is a standard structural loss, measuring the difference between the gold y and the output  $\hat{\mathbf{y}}$  clusters; and  $(\hat{\mathbf{y}} (\theta),\hat{\mathbf{h}} (\theta)) = \operatorname *{argmax}_{(\mathbf{y},\mathbf{h})\in Y\times H}s_{\theta}(\mathbf{x},\mathbf{y},\mathbf{h}).$
The right-hand side of Eq. 2 is essentially a margin-based objective with margin rescaled by the loss  $\Delta$ . Its minimization forces maximum
weighted incorrect  $\hat{\mathbf{h}}$  score lower than the maximum weighted correct  $\mathbf{h}$  by at least the value of the loss  $\Delta (\mathbf{y},\hat{\mathbf{y}},\hat{\mathbf{h}})$  on that  $\hat{\mathbf{h}}$  under  $\theta$  parameters. Our neural model optimizes the objective  $E$  defined in Eq. 2. We use the loss  $\Delta$  of Yu and Joachims (2009) based on computing edge mistakes in  $\hat{\mathbf{h}}$ , in which negative edge penalties are scaled with an  $r$ -parameter.
4.2 Differentiable scoring function
For optimizing  $E$  in Eq. 2, we define a scoring function that decomposes over the edges of  $\mathbf{h}$ :
$$
s _ {\theta} (\mathbf {x}, \mathbf {y}, \mathbf {h}) = \sum_ {\mathbf {e} = (x _ {i}, x _ {j}) \in \mathbf {h}} n e t _ {\theta} (\mathbf {e}). \tag {3}
$$
This enables the inference by Kruskal's algorithm, where the network  $net_{\theta}$  activates on edge representations<sup>2</sup>. Eq. 3 indicates that our approach is applicable to any network  $net_{\theta}$ .
Our work is inspired by Kiperwasser and Goldberg (2016), who pass the arcs of dependency parses through a multi-layer perceptron and optimize a structured margin loss. Differently to them, we elaborate on the case of the margin rescaled with the structural loss, which includes max-violating inference. The objective  $E$  in Eq. 2 is sub-differentiable as a summation of edge networks;  $\Delta$  in  $E$  does not depend on  $\theta$ . We propagate the error from the margin loss  $E$  in Eq. 2 back to input layer of  $net_{\theta}$ .
One iteration of the algorithm operates on one sample of training data  $(\mathbf{x},\mathbf{y})$ , where, in the context of intent task,  $\mathbf{x} = \{x_{i}\}$  is a set of questions,  $\mathbf{y}$  are gold clusters of the questions in  $\mathbf{x}$ . We pass all the pairs of questions, i.e., edges  $\mathbf{e} = (x_i,x_j)$ ,  $i < j$  of a fully connected graph  $G$ , through  $net_{\theta}$ , and compute the global error  $E$ . The error computation includes finding (i) the max-violating graph,  $\hat{\mathbf{h}}$ , among all possible spanning graphs  $\mathbf{h}$  of  $G$ ; and (ii) the max-scoring correct spanning graph,  $\mathbf{h}^*$ , over the set of graphs that comply with the gold label  $\mathbf{y}$ . If  $E > 0$ , the backward pass of the model computes gradients for an update of the model. The partial derivatives with respect to the parameters  $\theta_{j}$  in the last layer of the network are
$$
\frac {\partial E}{\partial \theta_ {j}} = \sum_ {\mathbf {e} \in \hat {\mathbf {h}}} \frac {\partial n e t _ {\theta} (\mathbf {e})}{\partial \theta_ {j}} - \sum_ {\mathbf {e} \in \mathbf {h} ^ {*}} \frac {\partial n e t _ {\theta} (\mathbf {e})}{\partial \theta_ {j}},
$$
2The scoring function follows the standard formulation of structured prediction tasks, where the score of a structure is computed by aggregating the scores of its constituent parts. In our case, it is a summation of edge scores. The reader may refer to the work on dependency parsing by Kiperwasser and Goldberg (2016).
and so on down to the input nodes of  $net_{\theta}$  following the chain rule.
5 Our Baseline Network
We intentionally do not specify the architecture of  $net_{\theta}$  in Sec. 4.2 as it could be of any form once encoding a pair of questions  $(x_i, x_j)$ . However, we further describe the architecture with which we experiment in this work.
We use a simple feedforward neural network, which consists of (i) an input layer encoding a pair  $(x_{i},x_{j})$ , (ii) one fully connected hidden layer with ReLU activation functions, and (iii) an output layer, which is a linear operation over the outputs of the hidden layer. This way, for edge  $\mathbf{e}$ ,  $net_{\theta}(\mathbf{e})$  is a real number without any restriction on its range.
The pairwise encoder is practically trained to score good edges higher than bad edges. However, doing it jointly for all the edges over a sample, in a structured way, has the goal of producing a more consistent decision in terms of clustering.
5.1 Input layer
We consider two ways for representing question pairs  $\phi (\mathbf{e}) = \phi (x_i,x_j)\in \mathbb{R}^d$ , using embeddings:
(1) We use a sentence encoder (Severyn and Moschitti, 2016) to map each question  $x_{i}$  into a fixed-size intermediate vector representation  $\psi(x_{i})$ . The encoder operates on a sentence matrix  $S$ , in which the  $k$ -th column corresponds to the  $k$ -th word in  $x_{i}$  and is a concatenation of the word embedding and overlap embedding:  $S_{k} = [word\_emb(w_{k}), ov\_emb(w_{k})]$ . The  $ov\_emb$  part for  $x_{i}$ , in each pair, is formed in association with the other question of the pair,  $x_{j}$ .  $S$  is given as input to a series of convolution operations with ReLU activations followed by a max-pooling layer. From the obtained question representations  $\psi(x_{i})$  and  $\psi(x_{j})$ , we compose a symmetrical pairwise representation as  $\phi(x_{i}, x_{j}) = [\max(\psi(x_{i}), \psi(x_{j})), \min(\psi(x_{i}), \psi(x_{j}))]$ , where max and min are component-wise vector operations, i.e., max and min are applied to pair of components, so that two final vectors are obtained.
(2) We exploit BERT (Devlin et al., 2019) embeddings:  $\phi(x_i, x_j) = \frac{1}{2} (\text{bert\_emb}(x_i, x_j) + \text{bert\_emb}(x_j, x_i))$ , where  $\text{bert\_emb}$  for a pair of questions comes from the final hidden layer representations, i.e., [CLS] token from the BERT model.
6 Question Clustering Data
In this section, we describe two datasets: (i) IC&OOS for intent classification; and (ii) Quora for intent clustering. We illustrate our procedure to transform the former into a dataset for clustering.
6.1 Intent clustering with IC&OOS
The dataset for Intent Classification and Out-Of-Scope prediction by Larson et al. (2019), which we denote IC&OOS, is a classification dataset, composed of user's queries distributed into 150 different intent classes over 10 domains, plus out-of-scope (OOS) queries falling outside the pre-defined classes. The data<sup>3</sup> contains 50, 20 and 30 user's queries per intent class in training, dev. and test parts, respectively. Plus 100 OOS queries for training, 100 – for development, and 1000 for test.
For example, we may see class categories such as MEAL SUGGESTION, with queries such as, 'suggestions for thai food' or 'help me find some new dinner recipes', which may be challenging to separate from the items of RESTAURANT REVIEWS, e.g., 'at yakamoto how is their sushi', and even more difficult to discern from 'what are some good sushi restaurants in reno', belonging to RESTAURANT SUGGESTION class.
The data from all of the pre-defined classes are present in training, dev. and test parts. The main steps for transforming this dataset in one for clustering are (i) merging the items of all the categories together; and (ii) using the original class labels as indication of belonging to different clusters.
However, a real-world application scenario of automatic clustering would entail that new incoming data can contain items which constitute new clusters (class categories). Thus, in order to demonstrate the capability of the supervised clustering models to group together the items of unseen clusters, we use one set of intent classes for training and another set for evaluation, which is constituted by a completely different set of intent classes and questions. This way, we retain the queries from one third of intent classes, i.e., 50, from the training part, the dev. queries from another third of intent classes, and the test queries from the remaining one third $^4$  of classes, and use them as new training, dev. and test parts, respectively.
Additionally, it should be noted that the original dataset contains OOS queries, which we keep all.
<sup>3</sup>We use the Small variant of the dataset.
4The split of the classes into three sets is done randomly without reference to the 10 original topic domains.
Thus, our new split can be also used to analyze OOS queries, which might not be put in any semantically meaningful cluster, as well as unseen intent items, for which, we know that their natural clusters (original categories) exist.
Data sampling and instance creation Training and test examples in a clustering problem are sets of items. In order to be able to effectively update the structural clustering objective in Eq. 2 with NNs, we need to limit the size of training examples  $(\mathbf{x},\mathbf{y})$ . Precisely we need to limit the size of input query sets  $\mathbf{x}$ , as the number of edges e (Eq. 3) to be passed trough the network grows exponentially with it. Thus, we split the data into samples: (i) from test set, we just extract random disjoint samples of equal size  $M$ ; and (ii) from training and dev. sets, we form samples according to a more elaborated procedure to avoid having too many singletons in an instance.
More specifically, for each class  $C$ : (i) we shuffle its items in a random order; and (ii) we split them into a set  $P$  of  $m$  disjoint parts (mini-clusters) of random sizes (different sizes, necessarily  $\geq 2$ ) each, s.t.,  $\cup_{p\in P} = C$ . Then, to build the training clustering examples,  $\mathbf{x}$ , we iterate for several times over the entire list of classes  $\mathcal{C}$ , in a random order, and,  $\forall C\in \mathcal{C}$ , we select a mini-cluster  $p\in P_C$  which we append to a current sample  $S$  (initialized as empty), if  $|S\cup p|\leq M$ . Otherwise, we start a new sample with  $S = \{p\}$ , and go to the next category, until all  $P_C$  are exhausted (this happens simultaneously  $\forall C$ , as  $P_C$  have the same size). Now, our  $\mathbf{x}$  sets consist of items contained in  $S$ 's.
This procedure makes the presence of each category uniform (binary presence, yes/no): after seeing each  $N$  number of samples  $S$ , we encounter elements of all the classes, however, without preserving the original relative proportions of the class distribution. This way, by setting the sample size limit  $M = 100$ , we obtain around 28 and 12 clustering examples from training and dev. sets, respectively, and 35 examples from the test set.
6.2 Quora question clustering
The Quora dataset, made available by Lee et al. (2017), was designed to learn and test question duplication classifiers. That is, for automatically detecting if two questions are semantically duplicate or not. We use the Quora Intent Corpus by Haponchyk et al. (2018) based on a sample of questions from Quora.
One main difference with IC&OOS is the fact that the negative examples selected by the organizers of the Quora challenge refer to pairs of questions that have always some degree of lexical overlap. For example, the following pair How much water on earth is consumable? and How much water is on earth? is not duplicate. On the other hand, the two questions could be surely put in the same cluster WATER OF EARTH. This means that a similarity function learned from Quora labels may not be enough accurate for clustering. Also a simple scalar product between two embeddings would not be enough as it can only capture lexical overlap. The latter would surely fail on the pairs of the following questions What is a recursion tree?, What does your family tree look like?, How does your Christmas tree look like? since their specific semantics is different but the overlap is large. These examples suggest that a clustering algorithm must learn a similarity that looks to the entire set of items to be clustered, not just to single pairs. This requirement is inline with the characteristics of the methods we presented in Sec 4.
We present the results of our empirical evaluation of the neural clustering models using IC&OOS data, followed by that using Quora Intent Corpus. We summarize afterwards the highlights of a deeper investigation, we conducted on the performance of our approach and of its errors.
7.1 Setup
Data: We created data from IC&OOS as discussed in Sec 6.1, which contains, 2650, 1100, and 2470 queries and 28, 12, 35 clustering examples, in training, dev., and test sets, respectively.