SlowGuess/ACL_2020_OCR huggingface dataset:
Maximum Softmax Probability (MSP)
(Hendrycks and Gimpel, 2017) considers the maximum softmax probability of a sample as the confidence score to measure the probability that it belongs to a seen intent. The smaller the confidence score is, the more likely it belongs to an unknown intent.
- DOC (Shu et al., 2017) builds  $m$  1-vs-rest sigmoid classifiers for  $m$  seen classes respectively. The maximum probability is considered as the confidence of whether the sample belongs to the seen intent.
- Softmax. It can be considered as an ablation study of our method SEG, which uses softmax
instead of Gaussian mixture distribution to learn discriminative features.
- LMCL (Lin and Xu, 2019) uses large margin cosine loss instead of Gaussian mixture distribution to learn discriminative embeddings.
- SEG/o. A variant of our method SEG. It does not inject the class semantic information into the Gaussian mixture model.
We follow the setting in LCML (Lin and Xu, 2019) for unknown intent detection. Considering that some datasets may be unbalanced, we randomly select seen intents by a weighted random sampling over the entire intent set. The rest of the intents are regarded as unknown. We randomly select  $30\%$  samples of each intent to form the test set. The rest of each seen intent is added to the training set. We also follow LMCL to use macro f1-score as the evaluation metric, which makes sense because the ATIS dataset is extremely unbalanced.
For SNIPS, ATIS and SMP-2018, we use 300-dim embeddings pre-trained on Fasttext, Glove, and Chinese-Word-Vectors respectively. For BiLSTM, we set the number of layers as 2 and the output dimension as 128. In the self-attention layer, we set the attention dimension  $d_{a} = 10$ . After the self-attention layer, we project the feature vector to a  $d_{z}$ -dimension vector via a linear layer. We set  $d_{z} = 12$  for SNIPS and SMP-2018, and  $d_{z} = 4$  for ATIS. We report the average results over 10 runs. For the loss function, we set the margin  $m = 1$  and the trade-off parameter  $\lambda = 0.5$ .
For MSP, we set the threshold as 0.5 following Lin and Xu (2019). For DOC, we set the threshold as 0.5 as used in the original paper. During training of MSP and DOC, we clip the gradient norm to avoid gradient exploding. For LMCL, we follow
the original paper to set the scaling factor  $s = 30$  and the cosine margin  $m = 0.35$ . Softmax, LMCL, SEG/o and SEG all use LOF as the outlier detector, and we use the same set of parameters for LOF.
4.3 Result Analysis
From Table 2, it can be seen that our method SEG outperforms the baselines in most cases. Especially, on the most challenging dataset SMP-2018, SEG and SEG/o outperform others by a large margin, demonstrating its high effectiveness. Moreover, we can make the following observations:
(1) SEG consistently outperforms SEG/o in most cases, which proves the effectiveness of the proposed semantic enhancement mechanism.
(2) SEG/o generally has higher scores than Softmax and LMCL, especially on the more complex dataset SMP-2018, where significant gaps can be observed. The results indicate the advantage of Gaussian mixture model over Softmax and the variant LMCL for learning class-concentrated embeddings, which are more suitable to be coupled with the outlier detector LOF.
(3) All the methods work well on SNIPS, which is a simple dataset. MSP and DOC outperform other methods on ATIS with only  $25\%$  seen classes. However, as the proportion of seen class increases, we can see a significant decline in their performance. This is because ATIS is severely imbalanced where one intent accounts for  $96\%$  of the entire data. When there are many seen classes, DOC and MSP cannot learn an effective supervised classifier due to the dominance of one class.
5 Application in Generalized Zero-shot Intent Classification
In this section, we apply our method SEG to an extended application of unknown intent classification - zero-shot intent classification. It aims to discriminate unseen intents, which is beyond only detecting their existence. Specifically, given the training data  $D^{tr} = (X^{tr}, Y^{tr})$  where  $Y^{tr} \in C_{\mathrm{seen}}$ , a zero-shot classification system is trained to predict the label  $\hat{y}^{te}$  of any test sample which may belong to an unseen class, using the knowledge transferred from the seen data. There are two common settings for zero-shot learning, generalized zero-shot classification, where  $\hat{y}^{te} \in \{C_{\mathrm{seen}}, C_{\mathrm{unseen}}\}$ , and standard zero-shot classification, where  $\hat{y}^{te} \in C_{\mathrm{unseen}}$ . Here,  $C_{\mathrm{unseen}}$  is the set of unseen intent classes.
Previous attempts try to tackle the challenge of
Figure 2: A typical generalized zero-shot intent classification pipeline.
zero-shot intent classification from three directions. (1) What prior knowledge is more supportive, such as morphology (character-level embedding), class descriptions, and knowledge-based entity attributes (Ferreira et al., 2015a,b; Chen et al., 2016; Kumar et al., 2017). (2) How to better utilize these prior knowledge to extract more informative semantic representations, such as data augmentation and hierarchical representations learned by capsule networks (Xia et al., 2018). (3) With the extracted semantic features, how to design a better zero-shot learning strategy, such as reconstructing weight matrix for unseen intents through relation learning (Liu et al., 2019).
In this work, we improve generalized zero-shot intent classification by integrating the proposed SEG model as a binary unknown intent identifier into the original pipeline. We explore multiple ways of integration and conduct a case study based on a state-of-the-art method ReCapsNet (Liu et al., 2019).
5.1 Integrating Unknown Intent Identifier
As shown in Figure 2, a typical generalized zero-shot classification framework can be abstracted into two layers, the encoder layer and the zero-shot classifier layer. In the encoder layer, a user utterance  $x$  in the text format needs to be first mapped to the semantic representation  $z_{x}^{ZS}$ . In addition, it is common to encode class information as  $S$  for better semantic learning or knowledge transfer. In order to learn better semantic representation, prior knowledge is usually incorporated at this stage. Then, the learned representation will be fed to the zero-shot classifier layer. Various zero-shot classification strategies have been proposed to transfer knowledge to new categories. Finally, the system outputs the prediction  $\hat{y}^{te} \in \{C_{\mathrm{seen}}, C_{\mathrm{unseen}}\}$  for the utterance  $x$ .
We integrate SEG into the pipeline between the encoder layer and the classifier layer as shown in Figure 3. With the semantic feature  $z_{x}$ , we predict if the utterance  $x$  is an outlier via:
$$
p (g \mid z _ {x}), g \in \{\text {" s e e n " , " u n s e e n "} \}. \tag {16}
$$
Figure 3: Integration of the new intent identifier (SEG) into the generalized zero-shot intent classification pipeline.
For the case  $g =$  "seen", the intent of the utterance is considered to be a seen one. We then predict the intent by  $p(y|z_x,y\in C_{\mathrm{seen}},X^{tr},\theta)$  where  $\pmb{\theta}$  denotes the parameters of the original framework. Otherwise, the intent of the utterance is considered to be unseen, and we predict it via  $p(y|z_x,y\in C_{\mathrm{unseen}},X^{tr},\theta)$ .
Feature Assemble We adopt two ways "Separate" and "Combine" to assemble features for the following outlier detection task.
- Separate (Sep). We directly feed the output of the pre-trained SEG encoder  $\mathbf{z}_x^{SEG}$  to LOF for outlier detection, i.e.,
$$
z _ {x} = z _ {x} ^ {\text {S E G}}. \tag {17}
$$
- Combine. To take advantage of the original model, we first obtain the original semantic feature representation  $\boldsymbol{z}_x^{ZS}$  and define a transform function  $f$ . Then,  $f(\boldsymbol{z}_x^{ZS})$  is concatenated with the pre-trained features by SEG,  $\boldsymbol{z}_x^{SEG}$ , to make a combined feature representation:
$$
\boldsymbol {z} _ {x} = \left[ \boldsymbol {z} _ {x} ^ {S E G} \right| | f \left(\boldsymbol {z} _ {x} ^ {Z S}\right) ]. \tag {18}
$$
5.2 A Case Study on ReCapsNet
ReCapsNet Recently, ReCapsNet-ZS (Liu et al., 2019) demonstrates state-of-the-art performance in generalized zero-shot intent classification. In this section, we conduct a case study on integrating the new intent identifier into ReCapsNet.
The framework of ReCapsNet is illustrated in Figure 4. In the encoder layer, each utterance  $x$  is encoded with  $R$  semantic capsules  $[m_1, m_2, \dots, m_R]$  as the representations in  $R$  different semantic spaces. In addition, the training set  $D^{tr}$  and class labels  $L$  are encoded as  $S^{tr}$  and
Figure 4: The framework of ReCapsNet.
$S^C$ , respectively. In the zero-shot classifier layer,  $z_x^{ZS}$  is fed to a capsule network to make prediction. Each seen class  $k$  has  $R$  transformation matrices  $\{\pmb{W}_{kr}\}_{r=1}^{R}$ . In the testing phase, ReCapsNet reconstructs the  $r$ -th transformation matrix for each unseen class  $l$  as  $\pmb{W}_{lr} = \sum_{k} q_{lk} \pmb{W}_{kr}$ , where  $q_{lk}$  is the relation between unseen class  $l$  and seen class  $k$  learned from  $(\pmb{S}^{tr}, \pmb{Y}^{tr})$  and  $\pmb{S}^C$  by metric learning.
For the variant "Combine", to exploit the property that each utterance is variously represented in different semantic spaces as discussed in Liu et al. (2019), we define the semantic feature representation of ReCapsNet as
$$
f \left(\boldsymbol {z} _ {x} ^ {Z S}\right) = \left[ \| \boldsymbol {m} _ {1} \| _ {2}, \| \boldsymbol {m} _ {2} \| _ {2}, \dots , \| \boldsymbol {m} _ {R} \| _ {2} \right]. \tag {19}
$$
Experimental Setup We integrate SEG into the ReCapsNet pipeline with both "Sep" and "Combine" variants and test the performance of generalized zero-shot classification.
Following the settings of generalized zero-shot classification in Liu et al. (2019), we test our methods on two datasets SNIPS (Coucke et al., 2018) and SMP-2018 (Zhang et al., 2017) and report the micro-averaged recall (accuracy) and F1 scores. The baselines include DeVISE (Frome et al., 2013), CMT (Socher et al., 2013), CDSSM (Chen et al., 2016), Zero-shot DNN (Kumar et al., 2017), Intent
Method	SNIPS	SMP-2018
Seen	Unseen	Overall	Seen	Unseen	Overall
Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
DeViSE	0.9481	0.6536	0.0211	0.0398	0.4215	0.3049	0.8040	0.6740	0.0270	0.0310	0.5030	0.4250
CMT	0.9755	0.6648	0.0397	0.0704	0.4438	0.3271	0.8314	0.7221	0.0798	0.1069	0.5398	0.4834
CDSSM	0.9549	0.7033	0.0111	0.0218	0.4234	0.3194	0.6653	0.5540	0.1436	0.1200	0.4864	0.4052
Zero-shot DNN	0.9432	0.6679	0.0682	0.1041	0.4488	0.3493	0.7323	0.6116	0.0590	0.0869	0.5013	0.4316
IntentCapsNet	0.9741	0.6517	0.0000	0.0000	0.4200	0.2810	0.8850	0.7281	0.0000	0.0000	0.5375	0.4423
ReCapsNet	0.9511	0.6777	0.0994	0.1594	0.4705	0.3826	0.8107	0.7417	0.1959	0.1727	0.5692	0.5182
SEG (Sep / o)	0.9308	0.7501	0.3523	0.4514	0.6014	0.5800	0.7066	0.7391	0.3848	0.3038	0.5802	0.5681
SEG (Combine / o)	0.9217	0.7924	0.4642	0.5321	0.6612	0.6441	0.7054	0.7326	0.3888	0.3116	0.5811	0.5672
SEG (Sep / w)	0.7898	0.8335	0.6728	0.6420	0.7232	0.7245	0.6624	0.7243	0.4779	0.3627	0.5899	0.5823
SEG (Combine / w)	0.8644	0.8658	0.6961	0.6931	0.7685	0.7674	0.6821	0.7359	0.4848	0.3806	0.6046	0.5963
Table 3: Results of generalized zero-shot intent classification equipped with our new intent identifier SEG. "Seen", "Unseen" and "Overall" respectively denote the prediction performance on the utterances from seen intents, unseen intents, and both seen and unseen intents. The suffixes "/w" and "/o" stand for with and without semantic enhancement, respectively. The top 2 results for each metric are marked in bold.
CapsNet (Xia et al., 2018), and ReCapsNet (Liu et al., 2019). The average results over 10 runs of our methods and ReCapsNet are reported in Table 3, where the results of other baselines are taken from Liu et al. (2019).
We use the same setting and hyper-parameters as in ReCapsNet (Liu et al., 2019). We set  $d_z = 4$  for SNIPS and  $d_z = 12$  for SMP-2018. The rest of the parameters of SEG are the same as those used in Section 4.2. In addition, we also conduct an ablation study to demonstrate the effectiveness of the proposed semantic enhancement mechanism by testing two variants of our integration ("Sep / o" and "Combine / o") without using it.
Result Analysis From the results in Table 3, we can make the following observations:
(1) All variants of our integration achieve a significant boost in the overall accuracy and F1 scores on the two datasets, especially on SNIPS, where the performance increase is huge. Each variant leads to a qualitative leap in the performance on unseen intents. The prediction accuracy (microaveraged recall) on seen intents may be reduced compared to ReCapsNet and other baselines, since some utterances of seen intents are classified to unseen intents. However, the F1 score on seen intents increases significantly, indicating that it has much higher precision than that of the baselines.
(2) The variants of our integration with semantic enhancement significantly outperform those without using it on predicting unseen intents by very large margins. Although their accuracy scores on seen intents are lower, their overall accuracy and F1 scores are consistently better, which confirms
the effectiveness of semantic enhancement.
(3) It can be seen that the "Combine" variants generally perform much better than the "Sep" variants, especially the one with semantic enhancement ("Combine / w"), which performs outstandingly. It surpasses the performance of "Sep / w" in every metric, demonstrating the usefulness of the simple feature assemble strategy of concatenating the feature representations of ReCapsNet and SEG.
In this paper, we have proposed SEG, a semantic-enhanced Gaussian mixture model coupled with a LOF outlier detector, for unknown (new) intent detection. We empirically verified the effectiveness of SEG for unknown intent detection on real dialogue datasets in English and Chinese. Furthermore, we successfully applied SEG to improve generalized zero-shot intent classification and achieved remarkable performance gain over a most recent competitive method ReCapsNet. In future work, we plan to conduct more empirical studies on SEG and further improve its performance on new intent identification. We also plan to conduct more case studies in applying SEG to boost the performance of current zero-shot intent classification methods.
We would like to thank the anonymous reviewers for their helpful comments. This research was supported by the grant HK ITF UIM/377.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jorg Sander. 2000. Lof: identifying density-based local outliers. In ACM SIGMOD RECORD, volume 29, pages 93-104.
Yun-Nung Chen, Dilek Z. Hakkani-Tür, and Xiaodong He. 2016. Zero-shot learning of intent embeddings for expansion by convolutional deep structured semantic models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6045-6049.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv, abs/1805.10190.
Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 506-514.
Emmanuel Ferreira, Bassam Jabaian, and Fabrice Lefèvre. 2015a. Online adaptative zero-shot learning spoken language understanding using word-embedding. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5321-5325.
Emmanuel Ferreira, Bassam Jabaian, and Fabrice Lefèvre. 2015b. Zero-shot semantic parser for spoken language understanding. In Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 1403-1407.
Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems (NeurIPS), pages 2121-2129.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, USA, June 24-27, 1990. Morgan Kaufmann.
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (ICLR).
Jian Hu, Gang Wang, Fred Lochovsky, Jian-tao Sun, and Zheng Chen. 2009. Understanding user's query intent with wikipedia. In International Conference on World Wide Web (WWW), pages 471-480.
Anjishnu Kumar, Pavankumar Reddy Muddireddy, Markus Dreyer, and Björn Hoffmeister. 2017. Zero-shot learning across heterogeneous overlapping domains. In Annual Conference of the International Speech Communication Association (INTER-SPEECH), pages 2914-2918.
Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 5491-5496. Association for Computational Linguistics.
Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. In Annual Conference of the International Speech Communication Association (INTERN-SPEECH), pages 685-689.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation forest. In IEEE International Conference on Data Mining (ICDM), pages 413-422.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert YS Lam. 2019. Reconstructing capsule networks for zero-shot intent classification. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4798-4808.
Yukun Ma, Erik Cambria, and Sa Gao. 2016. Label embedding for zero-shot fine-grained named entity typing. In International Conference on Computational Linguistics (COLING), pages 171-180.
Jinseok Nam, Eneldo Loza Menc'ia, and Johannes Furnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In AAAI Conference on Artificial Intelligence (AAAI), pages 1948-1954.
Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, and Tom M. Mitchell. 2009. Zero-shot learning with semantic output codes. In Advances in Neural Information Processing Systems (NeurIPS), pages 1410-1418.
Suman Ravuri and Andreas Stolcke. 2015. Recurrent neural network and LSTM models for lexical utterance classification. In Annual Conference of the International Speech Communication Association (INTERN-SPEECH), pages 135-139.
Peter J Rousseeuw and Katrien Van Driessen. 1999. A fast algorithm for the minimum covariance determinant estimator. Technometrics, 41(3):212-223.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017. Dynamic routing between capsules. In Advances in Neural Information Processing Systems (NeurIPS), pages 3859-3869.
Prateek Veeranna Sappadla, Jinseok Nam, Eneldo Loza Menc'ia, and Johannes Furnkranz. 2016. Using semantic similarity for multi-label zero-shot classification of text documents. In European Symposium on Artificial Neural Networks (ESANN).
Bernhard Scholkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443-1471.
Lei Shu, Hu Xu, and Bing Liu. 2017. DOC: deep open classification of text documents. In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2911-2916. Association for Computational Linguistics.
Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems (NeurIPS), pages 935-943.
Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. 2018. Rethinking feature distribution for loss functions in image classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9117-9126.
Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S. Yu. 2018. Zero-shot user intent detection via capsule neural networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3090-3099.
Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular CRF for joint intent detection and slot filling. In IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU Workshop), pages 78-83.
Majid Yazdani and James Henderson. 2015. A model of zero-shot learning of spoken language understanding. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 244-249.
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. 2017. Comparative study of CNN and RNN for natural language processing. arXiv, abs/1702.01923.
Chenwei Zhang, Wei Fan, Nan Du, and Philip S. Yu. 2016. Mining user intentions from medical queries: A neural network based heterogeneous jointly modeling approach. In International Conference on World Wide Web (WWW), pages 1373-1384.
Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019. Integrating semantic knowledge to tackle zero-shot text classification. In *Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies* (NAACL-HLT), pages 1031-1040.
Weinan Zhang, Zhigang Chen, Wanxiang Che, Guoping Hu, and Ting Liu. 2017. The first evaluation of chinese human-computer dialogue technology. arXiv, abs/1709.10217.