SlowGuess/NAACL_2022_OCR huggingface dataset:
Our method achieves the best results under all detection algorithms. Using RCL+Local threshold significantly outperforms all the baselines under different OOD detection algorithms. Specifically, our method achieves  $10.44\%$ ,  $5.84\%$ ,  $4.20\%$  improvements over SCL+Global threshold on OOD F1 under three OOD detection algorithms on CLINC-full dataset and  $7.30\%$ ,  $7.64\%$  and  $7.49\%$  on Snips dataset. We also observe that F1 score of IND classification with our approaches can keep comparable or slightly outperform the baselines, showing that RCL and class-dependent local threshold effectively improve OOD detection without harming the performance of IND classification. We show similar results on CLINC-Small dataset in the Appendix Table 7.
RCL consistently outperforms CE and SCL. On CLINC-Full, RCL achieves  $4.73\%$ ,  $4.57\%$  and  $4.18\%$  improvements over SCL on OOD F1 under three OOD detection settings and  $4.77\%$ ,  $4.97\%$  and  $4.69\%$  on Snips dataset, respectively. It demonstrates that RCL can stably discriminate the representation space by separating easily-confused classes and thus improve the performance of OOD detection. We also find compared with local threshold, RCL improved more significantly on global threshold. We argue this is due to the local threshold has alleviated part of overconfidence problem.
Local threshold consistently outperforms global threshold. We find the local threshold consistently wins the global one with a significant margin, especially in MSP. We argue MSP suffers from more serious over-confident IND issue than GDA and energy. This reveals that the local threshold can alleviate the over-confident IND issue by assigning an adaptive threshold for each IND class.
Comparing with previous baselines. To make a
Models	Min	Median	Max	Mean
CE	162.9	252.6	445.5	261.1
SCL	125.0	228.8	408.7	235.9
RCL(ours)	96.4	177.9	390.2	190.6
Figure 5: Inter-class distance statistics with different number of the nearest class centers.
fair comparison with previous baselines, we follow the same setting as Zhang et al. (2021). Specifically, we use the BERT as the backbone of the model, and sample  $25\%$ ,  $50\%$ ,  $75\%$  or  $100\%$  (Full) classes of the CLINC-Full dataset as the IND classes, the other as OOD. From Table 3 and 10 (in the Appendix), we can observe that our approach outperforms previous baselines on F1 scores of both OOD detection and IND classification. We achieve  $4.56\%$ ,  $3.51\%$ ,  $4.03\%$ , and  $5.51\%$  performance gain on OOD F1 for the  $25\%$ ,  $50\%$ ,  $75\%$  and Full settings.
5.1 Analysis of IND Representations
To analyse how our RCL method affects the representation space and improves the performance of OOD detection, we perform a statistical analysis of the intra-class variance within each class as well as the inter-class distance between multiple classes.
RCL leads to smaller intra-class variance. To calculate the intra-class variance, we first compute the class center by averaging all samples representation corresponding to same class, then we calculate the variance of all samples to corresponding class center as the corresponding intra-class variance. In Table 4, we show the intra-class variance statistics among all classes with different training strategies: CE, SCL and our proposed RCL. We can find that all of the minimum, median, mean and maximum values of our approach are lower than CE and SCL, demonstrating that the RCL makes the representations within a single class tighter.
RCL leads to larger inter-class distance. We
Table 4: Intra-class variance statistics.
Type 1	Type 2	Models	Inter-class Distance
change_user_name	change.ai_name	CE	0.526
SCL	0.573
Ours	0.900
change_user_name	payday	CE	1.023
SCL	1.151
Ours	1.181
what_is_your_name	user_name	CE	0.449
SCL	0.466
Ours	0.789
change_speed	play_music	CE	0.774
SCL	0.925
Ours	1.094
rewards_balance	redeem_rewards	CE	0.511
SCL	0.572
Ours	0.666
Table 5: Inter-class distance statistics between Confused Pairs.
calculate the inter-class distance by averaging the euclidean distance from the center class to its K nearest classes. For each class, we take the class center by averaging all representations that belong to this class. Figure 5 show that our RCL approach consistently obtains larger inter-class distance compared to CE and SCL. This phenomena shows that our approach improves the OOD detection performance by separating among classes while maintaining intra-class high cohesion. It effectively improves the uniformity of the representation space.
5.2 Analysis of Confusing IND Categories
To further verify the effect of RCL on easily-confused classes, we select 4 label pairs that are easily confused by the baseline model (change username v.s. change ai name, what is your name v.s. user name, change speed v.s. play music, and rewards balance v.s. redeem rewards). In addition, we also select one label pair that is semantically unrelated (change username v.s. payday). For each class pair, we compare the inter-class of models that are trained with CE, SCL and our RCL.
We can observe from the group that training with RCL greatly increases the inter-class distance between confusable classes in Table 5. Take the first class pair as an example, our RCL achieves  $57\%$  gain of inter-class distance over the SCL baseline, while the SCL only achieves  $9\%$  gain based on CE. On the other hand, when comparing between the group 1 and 2, we can conclude that our method works better on pushing away confusable class pairs, while for semantically unrelated ones, the effect is not obvious  $(+57\% \mathrm{v.s.} + 2.6\%)$ .
5.3 Effect of Local Threshold
As we discussed above, setting a general global threshold for OOD detection is a straightforward
Figure 6: Local Threshold vs Global Threshold.
(1)vaccines (2)credit_limit_change (3)travelNotification (4)accountBlocked (5)order (6)timezone
(7)play_music (8)maybe (9)change_user_name (10)bill_balance
idea and the common practice in reality. However, it can not be applied to the situation where the variance of each IND class is various. If the global threshold is not suitable for some specific IND classes, the probability of true OOD samples may exceed the global threshold, resulting in overconfident IND samples.
To show the effect of class-dependent local threshold, we randomly select 20 IND classes, and give the mean softmax probability of the IND samples of each class as well as the OOD samples that have the same class with maximum softmax probability (i.e., these samples may be misclassified to this IND class or correctly identified as OOD, depending on the threshold). Then, we give the global threshold and the class-dependent local threshold for each IND class. For simplicity, we only show the mean value but variance. Note that the threshold between IND and OOD means most of the judgments are correct, but not necessarily all of them. We show half of results in Figure 6 and the rest in the Appendix(Figure 10).
For IND classes (1) - (4), the averaged maximum softmax probabilities of OOD samples are all beyond the threshold, indicating that a large proportion of the OOD samples are over-confident and will be classified to the corresponding IND class. However, with our class-dependent local threshold, the threshold is greater than the global threshold and can distinguish the IND and OOD samples more precisely.
For IND classes (5) - (6), the global threshold is lucky to distinguish the IND and OOD samples. However, we find these classes only make up a small proportion of all IND classes (about  $20\%$  since only 2 classes among 10 belong to this case).
For IND classes (7) - (10), we find even the averaged maximum softmax probability of IND samples is below the global threshold. It means that these IND classes can be easily confused with other
Figure 7: Effect of out-of-domain data size in development set.
semantically related IND classes, and thus degrades their maximum softmax probabilities, resulting in over-confident OOD samples. In contrast, our local threshold can automatically select the proper threshold, adaptively degrading the threshold correspondingly so that the threshold can exactly separate the IND and OOD samples.
5.4 Effect of Development Set Size
Figure 7 shows the effect of development set size. Using a small amount of OOD samples in development set to get a more suitable threshold is inevitable in OOD detection task. To show the effect of development set size, we randomly choose development data with a certain proportion from CLINC-Full OOD labeled development set and use the original test set for evaluation. We use the LSTM+Energy and LSTM+GDA settings.
Compare local and global thresholds. We find that the local threshold consistently outperforms global threshold with a significant margin, regardless of the OOD detection methods and training strategies we applied. Specifically, the performance drops are  $-4.5\% (-2.8\%)$ ,  $-10.5\% (-4.4\%)$  and  $-22.8\% (-13.4\%)$  over global(local) thresholds with three training strategies, respectively. Besides, we also find that the CE+Global collapses in performance when there are only 20 samples in dev set, while CE+Local is much better. We speculate that this phenomenon occurs because the OOD samples in the CLINC-Full dataset are diverse and the global threshold tends to overfit the data. It confirms that our proposed local threshold method can alleviate the reliance on development size by
Models	IND<>OOD	IND<>IND	All Errors
IND->OOD	OOD->IND	SUM
MSP	156	576	732	290	1022
MSP + RCL	122	542	664	233	897
MSP + Local Threshold	284	378	662	234	896
MSP + RCL + Local Threshold	229	390	619	194	813
Table 6: Statistics of different error types using RCL and Local Threshold.
assigning a specific threshold for each class.
Compare RCL with other training strategies. We find that the RCL consistently outperforms SCL and CE. Specifically, under CE, SCL and RCL, the performance drops are  $-22.8\%$ ,  $-10.5\%$  and  $-4.5\%$ , respectively. Besides, with the decrease of development data size, the corresponding difference gradually increases. It demonstrates that RCL can effectively relieve the dependence on development size by separating easily misclassified classes.
We also observe that in the range of 40-100, the performance of RCL hardly degrades. To be more specific, when only 40 OOD labeled data provided, the OOD F1 score of RCL+Local threshold model is still  $74.58(-0.74\%)$ . And even when only 20 OOD labeled data are available (about  $0.13\%$  training data), our proposed model still outperforms the best baseline(SCL+Global threshold) with 100 OOD labeled data by  $1.35\%$ . This reveals that our model is more robust and less dependent on development set size. And the result of our model is significantly improved under four datasets further proves that our proposed methods have strong robustness and generalization capability.
5.5 Analysis of Error Types
In order to explore the effect of our method on different error types. We divide all the error samples into three categories: confusion between IND(IND<->IND), over-confident OOD(IND<->OOD) and over-confident IND(OOD<->IND).
Figure 8(a) indicates that RCL can achieve consistently improvements in all the three error types. Figure 8(b) indicates that local threshold outperforms global threshold both in error of IND<->IND and OOD->IND. While greatly reducing OOD>IND errors, local threshold inevitably increases IND->>OOD errors (explained by Figure 2). We hypothesize this is due to the high semantic similarity between IND classes and labeling noise in dataset, which can be mitigated by RCL(see Table 6). We will leave more possible solutions for future work. Combined with Table 6, compared to global threshold, our local threshold can obtain smaller overall errors. In general, local threshold is better
Figure 8: Statistics of different error types using different training objectives and different thresholds.
for the overall performance.
Table 6 displays the comparison between RCL and local threshold. We find our RCL and local threshold both outperform MSP baseline and RCL+local threshold achieves the best performance. Comparing RCL and local threshold, RCL targets at IND->OOD errors (from 156 to 122) for over-confident OOD, while local threshold helps reduce OOD->IND errors (from 576 to 378) for over-confident IND. Besides, they both help reduce overall IND&OOD errors. On the other hand, the performance improvement on OOD detection always helps IND classification.
In this paper, we focus on the overconfidence issue of unsupervised OOD detection. We find the reasons for overconfidence arise from two aspects: (1) IND classes have high semantic similarity. (2) IND and OOD intents have spurious correlations. According to the two reasons, we respectively propose a novel reassigned contrastive learning (RCL) to discriminate IND intents for over-confident OOD and an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents for over-confident IND. We perform extensive experiments and comprehensive analyses to demonstrate the effectiveness of our approach. We hope to provide new guidance for future work.
We thank all anonymous reviewers for their helpful comments and suggestions. This work was partially supported by National Key R&D Program of China No. 2019YFF0303300 and Subject II No. 2019YFF0303302, DOCOMO Beijing Communications Laboratories Co., Ltd, MoE-CMCC "Artificial Intelligence" Project No. MCM20190701.
Satoshi Akasaki and Nobuhiro Kaji. 2017. Chat detection in an intelligent assistant: Combining task-oriented and non-task-oriented spoken dialogue systems. ArXiv, abs/1705.00746.
Abhijit Bendale and Terrance E Boult. 2016a. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572.
Abhijit Bendale and Terrance E. Boult. 2016b. Towards open set deep networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1563-1572.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. *ArXiv*, abs/2002.05709.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 506-514, San Diego, California. Association for Computational Linguistics.
Ulrich Gnewuch, S. Morana, and A. Maedche. 2017. Towards designing cooperative and social conversational agents for customer service. In ICIS.
Izhak Golan and Ran El-Yaniv. 2018. Deep anomaly detection using geometric transformations. In NeurIPS.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning. *ArXiv*, abs/2011.01403.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In ICML.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735.
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ArXiv, abs/1610.02136.
S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735-1780.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, A. Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. ArXiv, abs/2004.11362.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018. Training confidence-calibrated classifiers for detecting out-of-distribution samples. ArXiv, abs/1711.09325.
Shiyu Liang, Yixuan Li, and R. Srikant. 2017. Principled detection of out-of-distribution examples in neural networks. *ArXiv*, abs/1706.02690.
Shiyu Liang, Yixuan Li, and R. Srikant. 2018. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv: Learning.
Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5491-5496, Florence, Italy. Association for Computational Linguistics.
Weitang Liu, Xiaoyun Wang, John Douglas Owens, and Yixuan Li. 2020. Energy-based out-of-distribution detection. *ArXiv*, abs/2010.03759.
Andrey Malinin and Mark John Francis Gales. 2018. Predictive uncertainty estimation via prior networks. In NeurIPS.
Yawen Ouyang, Jiasheng Ye, Yu Chen, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2021. Energy-based unknown intent detection with data manipulation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2852-2861, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing
(EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Stanislav Pidhorskyi, Ranya Almohsen, Donald A. Adjeroh, and Gianfranco Doretto. 2018. Generative probabilistic novelty detection with adversarial autoencoders. In NeurIPS.
J. Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. 2019. Likelihood ratios for out-of-distribution detection. In NeurIPS.
Lei Shu, Hu Xu, and Bing Liu. 2017. Doc: Deep open classification of text documents. In EMNLP.
H. Shum, X. He, and Di Li. 2018. From eliza to xiaoice: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19:10-26.
Amrita S. Tulshan and Sudhir N. Dhage. 2019. Survey on virtual assistant: Google assistant, siri, cortana, alexa. Communications in Computer and Information Science.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. 2018. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In ECCV.
Feng Wang and Huaping Liu. 2021. Understanding the behaviour of contrastive loss. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2495-2504.
J. Winkens, R. Bunel, Abhijit Guha Roy, Robert Stanforth, V. Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, T. cemgil, S. Eslami, and O. Ronneberger. 2020. Contrastive training for improved out-of-distribution detection. ArXiv, abs/2007.05566.
Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020. A deep generative distance-based classifier for out-of-domain detection with mahalanobis space. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1452-1460, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 870-878, Online. Association for Computational Linguistics.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021b. Adversarial self-supervised learning for out-of-domain detection. In NAACL.
Zhiyuan Zeng, Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2021c. Adversarial generative distance-based classifier for robust out-of-domain detection. ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7658-7662.
Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiaoming Wu, and Albert Y.S. Lam. 2021. Out-of-scope intent detection with self-supervision and discriminative training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3521-3532, Online. Association for Computational Linguistics.
Hanlei Zhang, Hua Xu, and Ting-En Lin. 2021. Deep open intent classification with adaptive decision boundary. In AAAI.
Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020. Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1198-1209.
Chengxu Zhuang, Alex Zhai, and Daniel Yamins. 2019. Local aggregation for unsupervised learning of visual embeddings. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6001-6011.
A Comparison of CL, SCL and RCL
Contrastive learning (CL) methods (Chen et al., 2020; He et al., 2020) have been proven effective to learn unsupervised representations for downstream tasks. Winkens et al. (2020); Zeng et al. (2021b) propose to apply contrastive learning (CL) to OOD detection task. They first perform CL on unlabeled data (including unlabeled IND and OOD intents) to learn OOD representations, then use cross-entropy loss on labeled IND data to learn an IND intent classifier. But the unlabeled data is not always available. More importantly, CL can only independently learn the OOD and IND representations (because the CL loss is built in an instance-wise way), but not explicitly distinguish different intent types in a class-wise way. Further, Zeng et al. (2021a) uses supervised contrastive learning (SCL) (Khosla et al., 2020) to learn discriminative intent representations only using labeled IND data. Compared to CL, SCL regards all the IND intents from the same class as positive pairs and samples from different classes as negative pairs. SCL aims to learn tight intent representations for each intent type and tries to distinguish different intent types. However, we find intents within similar categories
Detection	Training	Global Threshold	Local Threshold(ours)
OOD	IND	OOD	IND
F1	Recall	F1	F1	Recall	F1
MSP	CE	48.89	37.40	83.94	62.94	61.42	85.61
SCL	55.79	47.20	86.25	63.57	61.10	86.80
RCL(ours)	61.54	53.23	86.26	65.33	62.05	86.81
GDA	CE	61.89	60.72	84.87	63.88	66.60	84.40
SCL	64.70	64.16	87.02	66.64	71.69	87.17
RCL(ours)	70.56	68.92	87.27	71.69	72.80	87.54
Energy	CE	63.57	58.10	84.70	66.63	66.69	84.74
SCL	67.64	65.10	85.77	70.67	70.16	86.89
RCL(ours)	71.46	71.49	87.69	74.19	77.90	87.86
Table 7: The performance of OOD detection and IND classification with different OOD detection methods on CLINC-Small dataset (p<0.05 under t-test).
Training	Test
SCL	x1.00	-
RCL(ours)	x1.10	-