SlowGuess/ACL_2020_OCR huggingface dataset:
Lu Fan $^{1*}$  Guangfeng Yan $^{1*}$  Qimai Li $^{1*}$
Han Liu $^{1}$  Xiaotong Zhang $^{1}$  Albert Y.S. Lam $^{2}$  Xiao-Ming Wu $^{1\dagger}$
Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R.1
Fano Labs, Hong Kong S.A.R.2
{csgfyan, cslfan, csqmli}@comp.polyu.edu.hk
{cshliu, csxtzhang}@comp.polyu.edu.hk
albert@fano.ai, csxmwu@comp.polyu.edu.hk
User intent classification plays a vital role in dialogue systems. Since user intent may frequently change over time in many realistic scenarios, unknown (new) intent detection has become an essential problem, where the study has just begun. This paper proposes a semantic-enhanced Gaussian mixture model (SEG) for unknown intent detection. In particular, we model utterance embeddings with a Gaussian mixture distribution and inject dynamic class semantic information into Gaussian means, which enables learning more class-concentrated embeddings that help to facilitate downstream outlier detection. Coupled with a density-based outlier detection algorithm, SEG achieves competitive results on three real task-oriented dialogue datasets in two languages for unknown intent detection. On top of that, we propose to integrate SEG as an unknown intent identifier into existing generalized zero-shot intent classification models to improve their performance. A case study on a state-of-the-art method, ReCapsNet, shows that SEG can push the classification performance to a significantly higher level.
Understanding user intent is crucial for developing conversational and dialogue systems. It is essential to accurately identify the intent behind a user utterance to better guide downstream decisions and policies. With the advent of conversational AI, dialogue systems are becoming central tools in many applications such as mobile apps, companion bots, virtual assistants and so on. Since user interests may change frequently over time, the AI agents may continuously see unknown (new) user intents. Manual annotation can hardly catch up with such rapid development, which motivates the problem
of unknown intent detection that has recently attracted increasing interest from both academia and industry.
While there have been some pioneering works studying the open-world classification problem in natural language processing (Fei and Liu, 2016; Shu et al., 2017), very few methods are designed for unknown intent detection. To our knowledge, the first work is by Lin and Xu (2019), in which the authors use large margin cosine loss (LMCL) to learn deep discriminative features and then feed them to a density-based outlier detection algorithm to identify unknown intents. Although this method performs well on some benchmark datasets, it has two limitations. (1) In training, LMCL ignores the prior knowledge of class labels, while it has been shown that label correlations captured in the embedding space can improve prediction performance, especially in the zero-shot learning scenarios (Palatucci et al., 2009; Ma et al., 2016). (2) LMCL computes the cosine distance between embeddings in the feature space and trains with a softmax cross-entropy loss, making the embedding distribution of each class long and narrow (Wan et al., 2018), which may be less suitable for applying density-based anomaly detection algorithms to detect unknown intents.
In this paper, we aim to address these limitations and propose a novel semantic-enhanced Gaussian mixture model (SEG) for unknown intent detection. In contrast to the softmax function, the Gaussian mixture model enforces embeddings to form ball-like dense clusters in the feature space, which may be more desirable for outlier detection, especially when using density-based outlier detection algorithms. Furthermore, we propose to inject the semantic information of class labels into the Gaussian mixture distribution by assigning the embeddings of class labels or descriptions to be the means of the Gaussians. This enables SEG to learn
more class-concentrated embeddings that can benefit downstream outlier detection. We further use a large margin loss to make SEG learn more discriminative features and employ a density-based outlier detection algorithm LOF (Breunig et al., 2000) to detect unknown intents.
Identifying unknown intents is not enough for some application scenarios where it is important to know what exactly the new intents are, e.g., zero-shot intent classification. Current generalized zero-shot intent classification methods (Chen et al., 2016; Kumar et al., 2017; Xia et al., 2018; Liu et al., 2019) attempt to classify test instances directly by making predictions in the pool of all the seen and unseen intents. However, their prediction performances are quite low, and they are still far from practical use. In this work, we propose to integrate SEG as an unknown intent identifier into the generalized zero-shot intent classification pipeline. The basic idea is that correctly identifying if the intent of an utterance is known or unknown will make the subsequent intent classification task much easier. We conduct a case study on a state-of-the-art zero-shot intent classification method ReCapsNet (Liu et al., 2019). The results show that incorporating SEG successfully improves the performance of ReCapsNet by a large margin. It even pushes the performance to a practical level on the SNIPS dataset (Coucke et al., 2018).
- We propose a semantic-enhanced Gaussian mixture model (SEG) for unknown intent detection by incorporating class semantic information into a Gaussian mixture distribution.
- We explore to improve existing generalized zero-shot intent classification systems with an unknown intent identifier. To the best of our knowledge, this is the first attempt to apply unknown intent detection in this task.
- We conduct extensive experiments on three real-world datasets to validate the effectiveness of the proposed SEG model for unknown intent detection and its application in generalized zero-shot intent classification.
The rest of the paper is organized as follows. In Section 2, we review related works on intent classification and open-world classification. In Section 3, we discuss the proposed SEG model in details.
In Section 4, we present experimental results on unknown intent detection. In Section 5, we apply SEG to improve generalized zero-shot intent classification and conduct a case study. Finally, Section 6 concludes the paper.
2.1 Intent Classification
User intent classification is an important component of dialogue systems. Great effort has been made to understand user intent across various domains, ranging from search engine questions (Hu et al., 2009) to medical queries (Zhang et al., 2016). Deep learning models including convolutional neural networks (CNN) (Xu and Sarikaya, 2013) and attention-based recurrent neural networks (RNN) (Ravuri and Stolcke, 2015; Liu and Lane, 2016) are commonly used for intent classification. CNN based methods build sentence embeddings by aggregating embeddings of adjacent words, while RNN based methods extract sentence embeddings via encoding word embeddings sequentially. Both types of methods have shown promising results in practice (Yin et al., 2017).
Traditional intent classification methods require considerable amount of labeled data for each class to train a discriminative classifier, while zero-shot intent classification (Sappadla et al., 2016; Zhang et al., 2019) addresses the problem that not all intent categories are seen during the training phase, which is an important task in natural language understanding as novel intents may continuously emerge in dialogue systems (Liu and Lane, 2016; Nam et al., 2016; Xu and Sarikaya, 2013). Zero-shot intent classification aims to generalize knowledge and concepts learned from seen intents to recognize unseen intents. Early methods (Ferreira et al., 2015a,b; Yazdani and Henderson, 2015) explore the relationship between seen and unseen intents by introducing external resources such as manually defined attributes or label ontologies, but they are usually expensive to obtain. To deal with this, some methods (Chen et al., 2016; Kumar et al., 2017) map the utterances and intent labels to an embedding space and then model their relations in the space. Recently, IntentCapsNet-ZS (Xia et al., 2018) extends capsule networks (Sabour et al., 2017) for zero-shot intent classification by transferring the prediction vectors from seen classes to unseen classes. ReCapsNet (Liu et al., 2019) shows that IntentCapsNet-ZS hardly recognizes
Figure 1: Illustration of the proposed framework for unknown intent classification. The backbone network is a self-attention Bi-LSTM encoder, which is trained by the proposed semantic-enhanced large margin Gaussian mixture loss (SEG classifier). In the testing phase, LOF is employed to detect outliers. The predicted outliers will be considered as unseen intent class instances, while the inliers will be classified by the SEG classifier.
utterances from unseen intents in the generalized zero-shot classification scenario, and proposes to solve this issue by transferring the transformation matrices from seen intents to unseen intents. In this paper, we use ReCapsNet as an example to show that incorporating an unknown intent identifier in the generalized zero-shot classification pipeline can significantly improve the prediction performance on unseen intents and the overall performance.
2.2 Open-world Classification
Most of existing classification methods make the closed-world assumption, that is, no new classes can appear in testing. However, the real world is open and dynamic, and in many applications, the AI agent cannot expect it sees everything in training, which makes open-world learning or classification an important problem.
There are two major approaches to tackle open-world classification. One is to use the classifier to output an additional confidence score to measure the probability that a test sample is seen or unseen. cbsSVM (Fei and Liu, 2016) proposes a center-based similarity (CBS) learning strategy and employs SVM to build 1-vs-rest CBS classifiers. MSP (Hendrycks and Gimpel, 2017) proposes to use the maximum softmax probability as the confidence score. Instead of using Softmax as the final output layer, DOC (Shu et al., 2017) builds a multiclass classifier with a 1-vs-rest final layer which contains a sigmoid function for each seen class to reduce the open space risk.
The other approach is to treat the open-world classification as an outlier detection problem by ex
ploiting anomaly detection methods such as robust covariance estimators (Rousseeuw and Driessen, 1999), one-class SVM (Scholkopf et al., 2001), isolation forest (Liu et al., 2008) and local outlier factor (Breunig et al., 2000). Robust covariance estimators assume data follows a Gaussian mixture distribution. Based on this, it tries to fit an elliptic envelope, and outliers can be defined as points standing far enough from the fit shape. One-class SVM finds a hyperplane that circles the positive samples as the decision boundary. Isolation forest uses a binary search tree (isolated tree) to isolate samples. Due to the small number of outliers and their alienation from most samples, outliers will be isolated earlier and be closer to the root node of the isolated tree. Local outlier factor (LOF) is a density-based algorithm, which compares the density of a point and its neighbors to determine whether it is an abnormal point. Lower density means it is more likely to be identified as an abnormal point. In addition, to facilitate anomaly detection, some methods (Lin and Xu, 2019; Wan et al., 2018) use large margin loss functions to learn more discriminative feature representations.
Given an utterance  $\pmb{x} = \{\pmb{w}_1, \pmb{w}_2, \dots, \pmb{w}_T\}$  with  $T$  words, where  $\pmb{w}_t \in \mathbb{R}^{d_w}$  is the embedding of the  $t$ -th word. Each word can be further encoded sequentially using a bidirectional LSTM (BiLSTM),
$$
\begin{array}{l} \begin{array}{c} \overrightarrow {\boldsymbol {h}} _ {t} = \operatorname {L S T M} _ {f w} (\boldsymbol {w} _ {t}, \overrightarrow {\boldsymbol {h}} _ {t - 1}), \\ \leftarrow \end{array} \tag {1} \\ \stackrel {\leftarrow} {h} _ {t} = \operatorname {L S T M} _ {b w} \left(\boldsymbol {w} _ {t}, \stackrel {\leftarrow} {h} _ {t + 1}\right), \\ \end{array}
$$
where  $\overrightarrow{\pmb{h}}_t, \overleftarrow{\pmb{h}}_t \in \mathbb{R}^{d_h}$  are the hidden states of the word  $\pmb{w}_t$  by forward  $\mathrm{LSTM}_{fw}$  and backward  $\mathrm{LSTM}_{bw}$  respectively. The word  $\pmb{w}_t$  is encoded as the entire hidden state, which is represented by concatenating  $\overrightarrow{\pmb{h}}_t$  and  $\overleftarrow{\pmb{h}}_t$ , i.e.  $\pmb{h}_t = [\overrightarrow{\pmb{h}}_t; \overleftarrow{\pmb{h}}_t]$ , and the hidden state matrix of the utterance can be represented as  $\pmb{H} = [h_1, h_2, \dots, h_T] \in \mathbb{R}^{2d_h \times T}$ . Furthermore, we use the self-attention mechanism to obtain the sentence embedding. Specifically,
$$
\begin{array}{l} \boldsymbol {a} = \operatorname {s o f t m a x} \left(\boldsymbol {W} _ {s 2} \tanh  \left(\boldsymbol {W} _ {s 1} \boldsymbol {H}\right)\right), \\ z = \boldsymbol {W} \boldsymbol {H} \boldsymbol {a}. \end{array} \tag {2}
$$
where  $\pmb{a} \in \mathbb{R}^T$  is the self-attention weight vector,  $\pmb{W}_{s1} \in \mathbb{R}^{d_a \times 2d_h}$  and  $\pmb{W}_{s2} \in \mathbb{R}^{1 \times D_a}$  are trainable parameters,  $\pmb{W} \in \mathbb{R}^{d_z \times 2d_h}$  is also trainable feedforward weight parameter, and  $\pmb{z} \in \mathbb{R}^{d_z}$  is the final representation of the utterance  $\pmb{x}$ .
3.2 Semantic-Enhanced Large Margin Gaussian Mixture Loss
The softmax cross-entropy loss is widely used in many machine learning problems. However, the embedding distribution of each class learned by the softmax cross-entropy loss tends to be long, narrow, and radiating from the center, with different classes distributed next to each other closely (Wan et al., 2018). Such embedding distribution may not be ideal for detecting new intent classes, as there might not be much space for new classes. Nevertheless, the Gaussian mixture loss can enforce each class to gather into a dense and small cluster, which may be more desirable for detecting new intents. Here, we design a semantic-enhanced large margin Gaussian mixture loss for embedding learning.
Large-Margin Cross-Entropy Loss Given a  $K$ -way classification task, we assume the extracted feature vector (embedding)  $\mathbf{z}$  of the training samples follows a Gaussian mixture distribution, where  $\pmb{\mu}_k$  and  $\pmb{\Sigma}_k$  are the mean and covariance of class  $k$  in the embedding space respectively and  $p(k)$  is the prior probability of class  $k$ . The probability density function of  $\mathbf{z}$  is given by
$$
p (\boldsymbol {z}) = \sum_ {k} \mathcal {N} (\boldsymbol {z}; \boldsymbol {\mu} _ {k}, \boldsymbol {\Sigma} _ {k}) p (k), \tag {3}
$$
where  $\mathcal{N}(\pmb {z};\pmb {\mu}_k,\pmb {\Sigma}_k)$  is the Gaussian distribution.
For the embedding  $\mathbf{z}_i$  of any training sample  $\mathbf{x}_i$ , the posterior probability that  $\mathbf{z}_i$  belongs to its class  $y_i$  can be expressed as
$$
p \left(y _ {i} \mid \boldsymbol {z} _ {i}\right) = \frac {\mathcal {N} \left(\boldsymbol {z} _ {i} ; \boldsymbol {\mu} _ {y _ {i}} , \boldsymbol {\Sigma} _ {y _ {i}}\right) p \left(y _ {i}\right)}{\sum_ {k} \mathcal {N} \left(\boldsymbol {z} _ {i} ; \boldsymbol {\mu} _ {k} , \boldsymbol {\Sigma} _ {k}\right) p (k)}. \tag {4}
$$
The cross-entropy loss of  $\mathbf{z}_i$  between the true class label  $y_i$  and the inference  $p(y_i | \mathbf{z}_i)$  can then be computed as:
$$
\mathcal {L} _ {c e, i} = - \log p \left(y _ {i} \mid \boldsymbol {z} _ {i}\right), \tag {5}
$$
and the total loss of  $N$  training samples is
$$
\mathcal {L} _ {c e} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} _ {c e, i}. \tag {6}
$$
Let  $d_{k}$  be the Mahalanobis distance between  $\mathbf{z}_i$  and  $\pmb{\mu}_{k}$ , i.e.,
$$
d _ {k} = \left(\boldsymbol {z} _ {i} - \boldsymbol {\mu} _ {k}\right) ^ {\top} \boldsymbol {\Sigma} _ {k} ^ {- 1} \left(\boldsymbol {z} _ {i} - \boldsymbol {\mu} _ {k}\right) / 2. \tag {7}
$$
Then  $\mathcal{L}_{ce,i}$  can be expressed as
$$
\mathcal {L} _ {c e, i} = - \log \frac {p \left(y _ {i}\right) \left| \boldsymbol {\Sigma} _ {y _ {i}} \right| ^ {- \frac {1}{2}} e ^ {- d _ {y _ {i}}}}{\sum_ {k} p (k) \left| \boldsymbol {\Sigma} _ {k} \right| ^ {- \frac {1}{2}} e ^ {- d _ {k}}}. \tag {8}
$$
Consider a simplified case where  $p(k)$  and  $\Sigma_{k}$  are identical for all classes. In this case, the model will give a correct prediction of  $z_{i}$  if the distance of  $z_{i}$  to its class mean  $\mu_{y_i}$  is less than or equal to its distance to any other class mean.
In general, large margin loss helps to improve classification performance. Here, we also introduce a classification margin  $m \in [1, +\infty)$  into the cross-entropy loss, which then becomes:
$$
\mathcal {L} _ {c e} ^ {m} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} _ {c e, i} ^ {m}, \tag {9}
$$
$$
\mathcal {L} _ {c e, i} ^ {m} = - \log \frac {p (y _ {i}) | \boldsymbol {\Sigma} _ {y _ {i}} | ^ {- \frac {1}{2}} e ^ {- m d _ {y _ {i}}}}{\sum_ {k} p (k) | \boldsymbol {\Sigma} _ {k} | ^ {- \frac {1}{2}} e ^ {- d _ {k}}}.
$$
With the large margin loss,  $\mathbf{z}_i$  is correctly classified only when its distance to class mean  $\mu_{y_i}$  is significantly less than (no more than  $\frac{1}{m}$  of) its distance to any other class mean.
Semantic Enhancement via Class Description This is one of the key features of our proposed method. We inject the semantic information of each class into the Gaussian mixture model by assigning
the embedding learned from the text description  $d_{k}$  of class  $k$  to be the class centroid  $\pmb{\mu}_{k}$ . The text description  $d_{k}$  can either be a single-word class name or a sentence or paragraph that describes the class. That is,
$$
\boldsymbol {\mu} _ {k} = \text {f e a t u r e} _ {-} \text {e x t r a c t} \left(d _ {k}\right), \tag {10}
$$
where featureextract  $(\cdot)$  indicates the feature extraction module in Section 3.1.
Generation Loss In addition to the cross-entropy loss, we want to maximize the observed likelihood of the embeddings with the Gaussian mixture distribution. Specifically, we minimize the following negative logarithm likelihood,
$$
\begin{array}{l} \mathcal {L} _ {g} = - \sum_ {i = 1} ^ {N} \log \mathcal {N} (\boldsymbol {z} _ {i}; \boldsymbol {\mu} _ {y _ {i}}, \boldsymbol {\Sigma} _ {y _ {i}}) \\ = \frac {1}{2} \sum_ {i = 1} ^ {N} \left(\boldsymbol {z} _ {i} - \boldsymbol {\mu} _ {y _ {i}}\right) ^ {\top} \boldsymbol {\Sigma} _ {y _ {i}} ^ {- 1} \left(\boldsymbol {z} _ {i} - \boldsymbol {\mu} _ {y _ {i}}\right) \tag {11} \\ + \text {c o n s t}, \\ \end{array}
$$
where const means a constant number. As shown in Eq. (11), the generation loss  $\mathcal{L}_g$  encourages the embedding  $z_{i}$  to be close to its class centroid  $\mu_{y_i}$  which facilitates learning a more class-concentrated embedding distribution that may benefit the downstream outlier detection task.
By combining the cross-entropy loss and the generation loss, the total objective function is:
$$
\mathcal {L} = \mathcal {L} _ {c e} ^ {m} + \lambda \mathcal {L} _ {g}, \tag {12}
$$
where  $\lambda$  is a trade-off parameter.
3.3 Outlier Detection
By the above feature learning procedure, each utterance  $x$  can be encoded as an embedding  $z$ . Then, the embedding  $z$  is fed to a well-known outlier detection algorithm LOF (Breunig et al., 2000) to detect new or unknown intents (outliers). LOF is an unsupervised density-based anomaly detection method based on the following intuition. By comparing the local density of an object to those of its neighbors, it can identify regions of similar density. The objects with substantially lower density than their neighbors' are considered to be outliers.
LOF defines the local outlier factor of an object  $z$  as
$$
\operatorname {L O F} _ {k} (z) = \frac {1}{| N _ {k} (z) |} \sum_ {\boldsymbol {o} \in N _ {k} (z)} \frac {\operatorname {l r d} _ {k} (\boldsymbol {o})}{\operatorname {l r d} _ {k} (z)}, \tag {13}
$$
Dataset	SNIPS	ATIS	SMP-2018
Vocab Size	11642	938	3189
Avg. Length	9.05	11.37	4.87
# of Samples	13802	6371	2460
# of Classes	7	18	30
where  $N_{k}(z)$  denotes the set of  $k$ -nearest neighbors of  $z$ , and "Ird" denotes the local reachability density which measures the local density around an object. The local reachability density is defined as the inverse of the average reachability distance between  $z$  and its neighbors, i.e.,
$$
\operatorname {l r d} _ {k} (z) = \frac {\left| N _ {k} (z) \right|}{\sum_ {\boldsymbol {o} \in N _ {k} (z)} \operatorname {r e a c h - d i s t} _ {k} (\boldsymbol {z} , \boldsymbol {o})}. \tag {14}
$$
Here, the reachability distance reach-distk  $(z,o)$  is defined as
$$
\operatorname {r e a c h - d i s t} _ {k} (\boldsymbol {z}, \boldsymbol {o}) = \max  \left\{\mathrm {k - d i s t} (\boldsymbol {o}), d (\boldsymbol {z}, \boldsymbol {o}) \right\}, \tag {15}
$$
where  $\mathbf{k}\text{-dist} (\pmb {o})$  denotes the distance of the object  $\pmb{o}$  to its  $k$  -th nearest neighbor, and  $d(z,o)$  is the distance between  $\pmb{z}$  and  $\pmb{o}$
If the LOF factor of an utterance is much larger than 1, it has substantially lower local density than its neighbors', which means the utterance embedding is relatively distant from its neighbors. Hence, it can be inferred the utterance is likely to belong to an unknown intent class.
3.4 Overall Procedures
Figure 1 illustrates the overall training and testing procedures of the proposed framework for unknown intent detection. The backbone network is a self-attention Bi-LSTM encoder. In the training phase, the encoder is trained by minimizing the semantic-enhanced large margin Gaussian mixture loss (SEG classifier) as in Eq. (12) on the training samples (seen intent class instances). In the testing phase, user utterances may come from both seen and unseen intent classes. Given an utterance, we first obtain its feature representation  $z$  with the trained encoder, then we use LOF to decide whether  $z$  is an outlier or not. If  $z$  is an outlier, we take it as an instance of some new intent class. Otherwise, we classify  $z$  to one of the seen intent classes using the SEG classifier.
In this section, we present experimental results on unknown intent detection. Formally, we train an
Dataset	SNIPS	ATIS	SMP-2018
% of known intents	25%	50%	75%	25%	50%	75%	25%	50%	75%
MSP	0.5543	0.8060	0.8585	0.6848	0.5158	0.3853	0.6132	0.7089	0.7716
DOC	0.5462	0.7962	0.8564	0.7007	0.5073	0.3659	0.6095	0.7197	0.7642
Softmax	0.5508	0.8036	0.8393	0.6597	0.6310	0.5732	0.5818	0.6860	0.7351
LMCL	0.5489	0.8041	0.8458	0.6763	0.6778	0.6110	0.6059	0.7094	0.7580
SEG/o	0.5440	0.8067	0.8474	0.6768	0.6699	0.5918	0.6734	0.7676	0.8128
SEG	0.5599	0.8193	0.8612	0.6410	0.6700	0.6466	0.6966	0.7895	0.8205
Table 2: Macro F1-score of unknown intent detection with different proportion of seen classes. The top 2 results for each metric are marked in bold.
unknown intent detection system with training data  $D^{tr} = (X^{tr}, Y^{tr})$ , where  $Y^{tr} \in \{l_1, \dots, l_K\} = C_{\mathrm{seen}}$  (the set of seen intent classes). For test utterances of seen intents, the unknown intent detection system aims to assign correct intent labels to them. For test utterances of unseen intents, the system is expected to identify them as outliers.
4.1 Datasets and Baselines
We evaluate our method SEG for unknown intent detection on 3 real task-oriented dialogue datasets: SNIPS (Coucke et al., 2018), ATIS (Hemphill et al., 1990) and SMP-2018 (Zhang et al., 2017). SNIPS is an open-source single-turn English corpus, which contains 7 types of user intents across different domains. ATIS is also an English dataset, which contains 18 types of user intent in the airline travel domain. SMP-2018 is a Chinese dialogue corpus for user intent recognition, which contains 30 different types of user intents. The statistics of the datasets are summarized in Table 1.
We compare SEG with the following unknown intent detection methods.