SlowGuess/NAACL_2022_OCR huggingface dataset:
Rajat Kumar, Mayur Patidar, Vaibhav Varshney, Lovekesh Vig, Gautam Shroff TCS Research, New Delhi, India
{k.rajat2, patidar.mayur, varshney.v, lovekesh.vig, gautam.shroff}@tcs.com
Intent Detection is a crucial component of Dialogue Systems wherein the objective is to classify a user utterance into one of the multiple pre-defined intents. A pre-requisite for developing an effective intent identifier is a training dataset labeled with all possible user intents. However, even skilled domain experts are often unable to foresee all possible user intents at design time and for practical applications, novel intents may have to be inferred incrementally on-the-fly from user utterances. Therefore, for any real-world dialogue system, the number of intents increases over time and new intents have to be discovered by analyzing the utterances outside the existing set of intents. In this paper, our objective is to i) detect known intent utterances from a large number of unlabeled utterance samples given a few labeled samples and ii) discover new unknown intents from the remaining unlabeled samples. Existing SOTA approaches address this problem via alternate representation learning and clustering wherein pseudo labels are used for updating the representations and clustering is used for generating the pseudo labels. Unlike existing approaches that rely on epoch-wise cluster alignment, we propose an end-to-end deep contrastive clustering algorithm that jointly updates model parameters and cluster centers via supervised and self-supervised learning and optimally utilizes both labeled and unlabeled data. Our proposed approach outperforms competitive baselines on five public datasets for both settings: (i) where the number of undiscovered intents is known in advance, and (ii) where the number of intents is estimated by an algorithm. We also propose a human-in-the-loop variant of our approach for practical deployment which does not require an estimate of new intents and outperforms the end-to-end approach.
Modern dialogue systems (Louvan and Magnini, 2020) are increasingly reliant on intent detection
Figure 1: An instance of user logs, where the intent detection model is trained on two known intents, i.e.,  $i_1$  and  $i_2$ . After manual analysis of user logs a human reviewer has discovered two new intents  $i_3$  and  $i_4$  and assigned utterance  $u21$  to an existing intent, i.e.,  $i_2$ .
to classify a user utterance into one of the multiple known user intents. Intent detection is typically modeled as a multi-class classification problem where labeled data comprising of utterances for each known intent is manually created by domain experts. However, most real-world applications have to cope with evolving user needs and new functionality is routinely introduced into the dialogue system resulting in a continuously increasing number of intents over time. Even for seasoned domain experts estimating future user requirements at design time is challenging and these often have to be discovered from recent user logs which contain information corresponding to past user utterances, model response (i.e., predicted intent), implicit (confidence or softmax probability), and explicit (user clicks on a thumbs up or thumbs down icon) feedback as shown in Fig 1. The intent detection model presented in Fig. 1 is trained on two initial intents  $(I_1, I_2)$  from the banking domain using labeled data created by domain experts. Filtered user logs containing implicit and explicit feedback were shared with domain experts, who, discovered two new intents  $(I_3, I_4)$  and mapped the filtered utterances to these new intents. Additionally, experts also have to identify and discard utterances that are outside the domain of the dialog system.
Human Language Technologies, pages 1836 - 1853
The remaining user logs are the primary source of evolving user needs and the process of identifying utterances belonging to "new intents"/"unknown intents" from user logs is referred to as Intent Discovery/Intent Mining (Chatterjee and Sengupta, 2020; Zhang et al., 2021b). However, manually monitoring user logs is not scalable and our objective in this paper is to present novel SOTA techniques to perform automated and semi-automated intent detection and discovery over user logs given only a few labeled utterances from known intents in addition to unlabeled utterances from both known and unknown intents.
Several classical (MacQueen, 1967; Chidananda Gowda and Krishna, 1978; Ester et al., 1996) and deep learning (Xie et al., 2016a; Zhang et al., 2021a) based clustering methods have been used for intent discovery. Chatterjee and Sengupta (2020) modeled intent discovery from unlabeled utterances as an unsupervised clustering problem and proposed a variant of DBSCAN (Ester et al., 1996) for clustering but do not employ any representation learning and rely heavily on manual evaluation. Zhang et al. (2021a) use a contrastive learning (Chen et al., 2020) based unsupervised approach for joint representation learning and clustering where performance largely depends on the quality of an auxiliary target distribution (Xie et al., 2016a). Lin et al. (2020) and Zhang et al. (2021c) model intent detection and discovery as a semi-supervised learning problem where the objective is to detect known intents and discover new intents given 1) a few labeled utterances from known intents along with 2) unlabeled utterances from known and new intents. This is similar to our problem of intent detection and discovery from user logs. Deep-Aligned (Zhang et al., 2021c) is the current SOTA approach for intent detection and discovery alternately performing representation learning and clustering by utilizing pseudo-labeled data obtained from clustering for representation learning. Deep-Aligned uses k-means (MacQueen, 1967) as the clustering algorithm of choice and updates a BERT (Devlin et al., 2019) backbone's parameters in the process. As k-means may assign different cluster ids to the same set of data points over different iterations the authors propose an alignment algorithm to align clusters obtained in consecutive epochs. Thus, an incorrect cluster alignment over epochs may lead to a significant drop in clustering accuracy. Additionally, they make the unrealistic assumption of
a uniform distribution over intents to estimate the number of intents.
In this paper, we propose a novel end-to-end Deep Semi-Supervised Contrastive Clustering (DSSCC-E2E) algorithm for intent detection and discovery from user logs. DSSCC-E2E is motivated by recent advances in self-supervised (Chen et al., 2020; Wu et al., 2020; Li et al., 2021) and supervised contrastive learning (Khosla et al., 2020; Gunel et al., 2021) applied to Computer vision and natural language processing. We model intent detection and discovery as a form of semi-supervised contrastive clustering wherein we jointly update backbone representations and cluster centers by minimizing the distance between the distribution over clusters of similar utterances and maximizing the same for dissimilar utterances via a semi-supervised variant of supervised contrastive (SupCon) loss(Khosla et al., 2020). For contrastive learning, we use the contextual augmenter (Ma, 2019) to create pairs of augmentations (positive pairs/ similar utterances) corresponding to each unlabeled utterance from known and unknown intents. For labeled utterances, we use pairs of utterances with the same intent to create positive pairs. To improve the accuracy of intent detection, we update representations based on labeled utterances by minimizing the cross-entropy loss. To avoid the trivial solution that assigns all utterances to the same cluster (Hu et al., 2017), similar to Van Gansbeke et al. (2020); Li et al. (2021) we also add an entropy term to the loss function which distributes utterances uniformly across the clusters.
Existing semi-supervised approaches(Lin et al., 2020; Zhang et al., 2021c) including DSSCC-E2E and some unsupervised approaches (MacQueen, 1967; Zhang et al., 2021a) for intent discovery require an estimate of the number of new intents  $(\mathfrak{m})$  present in the user logs. Incorrect estimates for  $m$  can lead to noisy clusters (i.e., a cluster which contains utterances from multiple intents), which then require substantial manual effort to split cleanly. Unsupervised approaches (Ester et al., 1996; Chatterjee and Sengupta, 2020) often lead to a large number of clusters due to poor semantic utterance representations. For practical deployment, we propose a human-in-loop variant of DSSCC-E2E called DSSCC-HIL which does not estimate  $m$  and instead creates multiple dense clusters via DBSCAN. Domain experts can then merge these clusters with minimal manual effort such that each
merged cluster represents an intent.
Our key contributions are: (1) We propose a novel deep semi-supervised contrastive clustering-based approach for intent detection and discovery from user logs. (2) DSSCC-E2E does not require epoch-wise cluster alignment, is end-to-end trainable, and fully utilizes both the labeled and unlabeled utterances for novel intent discovery. (3) DSSCC-E2E outperforms competitive baselines on five public datasets in the following settings: (3.1) Number of intents are known in advance. (3.2) Number of intents is estimated by an algorithm. (4) For realistic deployment, we propose a human in the loop intent detector DSSCC-HIL, which does not need to estimate the number of new intents. (5) With minimal manual effort, DSSCC-HIL outperforms existing approaches on the intent discovery task.
2.1 Self-supervised and Supervised Representation Learning
Different self-supervised pre-training tasks have been proposed to pre-train PLMs, such as Masked language modeling (MLM) (Devlin et al., 2019), and MAsked Sequence to Sequence pre-training (MASS)(Song et al., 2019). Reimers and Gurevych (2019); Gao et al. (2021) fine-tune BERT on a supervised contrastive learning objective to learn better sentence embeddings. Zhang et al. (2021a) use SBERT (Reimers and Gurevych, 2019) as a PLM to learn clustering friendly representations in an unsupervised scenario by using instance-level contrastive representation learning (Chen et al., 2020) and clustering in a joint fashion. We propose a deep semi-supervised contrastive learning approach for intent detection and discovery where we jointly update PLM representations and cluster centers. We leverage the idea of unsupervised contrastive clustering (Li et al., 2021) for images to semi-supervised contrastive clustering of text and use a modified version of the supervised contrastive loss function (Khosla et al., 2020) to jointly update model parameters and cluster centers.
We also compare the proposed approach with existing unsupervised and semi-supervised approaches for clustering in Appendix A.1.
3 Problem Description
Consider a set of  $n$  known intents with a few labeled utterances per intent  $I_{known} =$
Figure 2: Proposed Approach - Deep Semi-Supervised Contrastive Clustering (DSSCC)
$\{I_1, I_2, \dots, I_n\}$  where  $I_i = \{\bigcup_{k=1}^r (u_{ik}, y_i)\}$ ,  $\mathcal{D}_l = \bigcup_{i=1}^n I_i$  and user logs  $\mathcal{D}_{user\_logs} = \{\bigcup_{v=1}^S (u_v)\}$  which contain unlabeled user utterances from both known and unknown intents. The objective is to detect and discover existing and new/unknown intents from a test set,  $\mathcal{D}_{test}$  given a training set  $\mathcal{D}_{train} = \mathcal{D}_l \cup \mathcal{D}_{user\_logs}$  (and  $\mathcal{D}_{val}$ ) which contains utterances from both known and unknown intents.  $\mathcal{D}_{test} = \{I_{known}' \cup I_{new}\}$  and  $I_{new} = \{I_{n+1}, I_{n+2}, \dots, I_m\}$  where  $m$  represents the number of new intents,  $I = I_{known} \cup I_{new}$  and  $|I| = n + m$  represents the total number of known and unknown intents.  $I_{known}'$  is similar to  $I_{known}$  except it only contains the new set of utterances for known intents from user logs. In a realistic scenario the number of "new intents"  $m$  present in user logs are not known apriori. We perform experiments in two scenarios (i) The number of known and new intents is known in advance and (ii) The number of new intents has to be inferred from the user logs.
As shown in Fig. 2, we propose a two-phase algorithm for intent detection and discovery from user logs. In the first phase of  $DSSCC$ , the parameters of a pre-trained language model (PLM) are updated based on labeled data from known intents. In the second phase, we perform joint representation learning and clustering by updating the cluster centers and PLM parameters via semi-supervised contrastive learning. In addition to contrastive learning,  $DSSCC$  also uses labeled utterances to update PLM representations via cross-entropy loss. Further, the entropy of the intent distribution is maximized to
For labeled utterance,  $u_{t}$ :  $u_{t1}$  and  $u_{t2}$  are utterances from the same intent. For unlabeled utterance,  $u_{t}$ :  $u_{t1}$  and  $u_{t2}$  are generated by contextual augmentor.
Figure 3: Utterance augmentations for semi-supervised contrastive learning.
alleviate the issue of empty clusters, i.e., when all utterances become part of a single cluster.
In a realistic scenario, the number of unknown intents  $m$  present in user logs is not known in advance so we use the approach proposed by Zhang et al. (2021c) to estimate  $m$  and use DSSCC-E2E for intent detection and discovery. The approach uses a PLM to get utterance representations and run K-means with a very high value of  $k = K'$  and only count clusters with cluster cardinality greater than  $|\mathcal{D}_{train}| / K'$  to estimate  $k$ . However in practice, we find that even this estimation method is often incorrect, and in section 4.5, we describe how we derive DSSCC-HIL from DSSCC-E2E to address this problem with a small amount of manual supervision. We use the Hungarian (Kuhn and Yaw, 1955) algorithm to align clusters with known and unknown intents and report performance. In the rest of the paper, we use DSSCC and DSSCC-E2E interchangeably and use DSSCC-HIL to refer to the human-in-the-loop variant of our approach.
4.1 Phase-1: Fine-tuning of PLM using labeled utterances from known intents
To leverage the labeled utterances from known intents for intent detection and discovery, in the first phase of  $DSSCC$ , we use these to update the parameters of the PLM, as shown in Phase-1 of Fig. 2. We fine-tune the PLM by minimizing the cross-entropy loss over a batch  $\mathcal{B}$  of size  $N$  consisting of labeled utterances from known intents, as shown in Eq. 1 and 2.
$$
p \left(I _ {\text {k n o w n}} \mid u _ {t}\right) = \operatorname {s o f t m a x} \left(\boldsymbol {h} _ {\boldsymbol {t}} * \boldsymbol {W} + b\right) \tag {1}
$$
$$
\mathcal {L} _ {C E} = - \frac {1}{N} \sum_ {t \in \mathcal {B}} \sum_ {i = 1} ^ {n} y \cdot \log \left(p \left(I _ {\text {k n o w n}} ^ {i} \mid u _ {t}\right)\right) \tag {2}
$$
In Eq. 1,  $h_t$  denotes a d-dimensional representation of the  $t^{th}$  utterance  $(u_t)$  in a batch  $\mathcal{B}$  obtained from the PLM and  $W \in \mathbb{R}^{d * m}$ ,  $b$  represent the weights and bias of a linear layer respectively. In Eq. 2,  $p(I_{known}^i \mid u_t)$  denotes the probability of assigning
$u_{t}$  to the  $i^{th}$  known intent and  $y$  is 1 only for the true intent and zero otherwise. After fine-tuning the PLM on labeled utterances, we discard the linear layer and use the PLM with updated weights in the second phase of DSSCC.
4.2 Phase-2: Deep Clustering
In the second phase, we use both labeled and unlabeled utterances to perform representation learning and clustering jointly via semi-supervised contrastive learning. To maintain intent detection accuracy on known intents, we also update representations and cluster centers by minimizing cross-entropy loss on labeled utterances from known intent.
4.2.1 Semi-supervised Representation Learning and Clustering
In addition to labeled utterances from known intents, we also use unlabeled utterances from both known and unknown intents to improve performance on intent detection and discovery, as shown in Phase-2 of Fig. 2. To learn better cluster representations, instead of minimizing the distance between utterances belonging to the same intent, we minimize the distance between their corresponding cluster distributions. Conversely, we maximize the distance between cluster distributions for clusters corresponding to different intents. In contrast to self-supervised (Chen et al., 2020) or supervised contrastive (Khosla et al., 2020) learning, for semi-supervised learning a batch  $\mathcal{B}$  of size  $N$  may contain both labeled and unlabeled utterances. As shown in Fig. 3, similar to self-supervised contrastive learning, we create a pair of augmentations  $(u_{t1}, u_{t2})$  or positive pairs corresponding to the  $t^{th}$  or anchor utterance  $(u_t)$  in  $\mathcal{B}$  to obtain  $\mathcal{B}'$ , which contains two augmented utterances corresponding to each utterance in  $\mathcal{B}$ . To generate augmentations for a labeled utterance, we randomly sample two utterances from the same intent and use them as augmentations whereas for an unlabeled utterance, we generate two augmented pairs by performing contextual augmentation<sup>1</sup> (Kobayashi, 2018; Ma, 2019), as shown in Fig. 3. In contextual augmentation, given an utterance, we randomly mask a few words and use BERT's masked-language modeling (MLM) objective to generate words corresponding to masked positions. If  $u_{t1}, u_{t2}$  are augmentations of a labeled utterance  $u_t$  then  $P(u_{t1})$  is defined as
$^{1}$ https://github.com/makcedward/nlpaug
the set of utterances belonging to the same intent as  $u_{t1}$  in  $\mathcal{B}'$  whereas  $N(u_{t1})$  will contain the all  $2N - 1$  utterances excluding  $u_{t1}$  (note that  $N(u_{t})$  and  $P(u_{t})$ ) may have utterances in common). If  $u_{t1}, u_{t2}$  are augmentations of an unlabeled utterance  $u_{t}$  then  $P(u_{t1})$  will only contain  $u_{t2}$  and  $N(u_{t1})$  will contain all  $2N - 1$  utterances excluding  $u_{t1}$ . We update PLM parameters and cluster centers by minimizing the Semi-Supervised Contrastive (SSC) loss as shown in Eq. 3.
$$
\begin{array}{l} \mathcal {L} _ {s s c} = \sum_ {t \in \mathcal {B} ^ {\prime}} \frac {- 1}{| P (u _ {t}) |} \sum_ {p ^ {\prime} \in P (u _ {t})} \\ \log \frac {\exp (p (I \mid u _ {t}) \cdot p (I \mid u _ {p ^ {\prime}}) / \tau)}{\sum_ {a \in N (t)} \exp (p (I \mid u _ {t}) \cdot p (I \mid u _ {a}) / \tau)}; \tag {3} \\ \end{array}
$$
In Eq. 3,  $(\cdot)$  symbol and  $\tau$  denotes dot product and scalar temperature parameter respectively. To get the distribution over intents/clusters  $(I / C)$ , i.e.,  $p(I \mid u_t)$  for the  $t^{th}$  utterance in  $\mathcal{B}'$  we apply a linear transformation over  $h_t$  and normalize via softmax,  $p(I \mid u_t) = \mathrm{softmax}(h_t * C + b)$ . Each column  $C_i$  of a linear layer  $C \in \mathbb{R}^{d \times (n + m)}$  acts as a cluster center, where  $C_i$  is the  $d$ -dimensional representation of the  $i^{th}$  cluster and  $(m + n)$  represents the total number of known and unknown intents.
To avoid the trivial solution that assigns all utterances to the same cluster Hu et al. (2017), similar to Van Gansbeke et al. (2020); Li et al. (2021) we also add an entropy term to the loss function and maximize it which distributes utterances uniformly across the clusters, as shown in Eq. 4 and Eq. 6.
$$
\mathcal {L} _ {e m} = - \sum_ {i \in \mathcal {T}} p \left(I ^ {\prime i}\right) * \log p \left(I ^ {\prime i}\right) \tag {4}
$$
$p(I^{\prime i}) = \frac{1}{|\mathcal{B}|}\sum_{t\in \mathcal{B}}p(I^{i} \mid u_{t})$  where  $p(I^i \mid u_t)$  denotes the probability of an utterance  $u_{t}$  being assigned to the  $i^{th}$  intent.
4.2.2 Supervised Representation Learning
To maintain intent detection accuracy on known intents, we also update the cluster centers and PLM parameters by minimizing cross-entropy loss over labeled utterances from known intents, as shown in Eq. 5 where  $y$  is 1 only for the target intent and zero otherwise. Unlike in phase-1,  $\mathcal{B}$  can contain labeled and unlabeled utterances from known intents and unlabeled utterances from unknown intents but we ignore the unlabeled utterances during
backpropagation.
$$
\mathcal {L} _ {s r l} = - \frac {1}{N} \sum_ {t \in \mathcal {B}} \sum_ {i = 1} ^ {n} y \cdot \log \left(p \left(I ^ {i} \mid u _ {t}\right)\right) \tag {5}
$$
We train  $DSSCC$  in two phases, in the first phase we fine-tune the PLM using labeled utterances from known intents as mentioned in Eq. 2. In the second phase, we do representation learning and clustering jointly by minimizing a combination of SSC loss, and cross-entropy loss. Further, to avoid the trivial solution of all the utterances getting assigned to a single cluster, the entropy over the intent distribution is maximized, as shown in Eq. 6.  $\lambda$  is a scalar hyper-parameter which controls the contribution of  $\mathcal{L}_{em}$  in  $\mathcal{L}$ .
$$
\mathcal {L} = \mathcal {L} _ {s s c} + \mathcal {L} _ {s r l} - \lambda * \mathcal {L} _ {e m} \tag {6}
$$
4.4 Inference
We propose two ways of utilizing representations learned by DSSCC for intent detection and discovery.
Clustering via learned cluster centers (DSSCCCH) Cluster assignment is based on similarity between cluster and utterance representations, i.e.,  $\operatorname{argmax} p(I^i \mid u_t)$ .
$$
1 \leq i \leq n + m
$$
K-means over representations (DSSCC-KM) We get the representations  $(\pmb{h}_t)$  for each utterance in  $\mathcal{D}_{test}$  from the PLM and use K-means for clustering.
4.5 Intent discovery with Human-in-the-loop (DSSCC-HIL)
Existing approaches including DSSCC assume knowledge of the number of unknown intents  $(m)$  to achieve good performance. This is not a realistic assumption and despite SOTA performance, significant manual effort is still needed to denoise the discovered clusters. However, the manual effort can be drastically reduced if we can generate dense clusters with high purity and assign a natural language description to each cluster. These descriptions can be used by a domain expert to merge similar clusters (i.e., a set of clusters with similar descriptions) or split a noisy cluster into multiple sub-clusters. Merging clusters requires much less manual effort than splitting as merging does not require examining every utterance in the cluster. Thus, if we can obtain
a set of pure clusters (i.e., a cluster where the majority of the utterances belong to a single intent) then the domain expert only needs to examine a few representative utterances per cluster before merging similar clusters.
To obtain better representations for clustering, we use phase-1 of  $DSSCC$  without any modification. To obtain pure clusters, unlike phase-2 of  $DSSCC$ , we perform representation learning and clustering in an alternate fashion and use DBSCAN for generating a large number of pure clusters. We assume that the utterances which are part of the same cluster belong to the same intent and use this fact to create a pair of augmentations for a given utterance along with contextual augmentation. Due to the unknown value of  $m$ , we perform semi-supervised contrastive representation learning at the utterance level, as shown in Eq. 7. The model parameters are updated according to  $\mathcal{L} = \mathcal{L}_{ssc}^{\prime} + \mathcal{L}_{srl}$ , where  $\mathcal{L}_{srl}$  and  $\mathcal{L}_{ssc}^{\prime}$  is defined in Eq. 5 and 7 respectively.