SlowGuess/NAACL_2022_OCR huggingface dataset:
RQ1: D2U in zero-shot setup. In Table 2, we report ROC AUC, FPR90, and FNR90 scores for different post-processing methods applied to a BERT-based INS classifier with no OOS training data. Our proposed method, D2U-zero, statistically significantly outperforms all baselines in all datasets with respect to ROC AUC score. Using cross-entropy for D2U-zero has better performance in majority of cases, compared to other distance metrics. The reason for its success might be that cross-entropy is the loss function used in the training procedure of the model. In terms of FPR90 and FNR90, D2U-zero does not always outperform all baselines. Though, the cases when baselines outperform are not statistically significant. This shows that the baseline methods can optimize FPR90 and FNR90 individually but cannot outperform D2U in terms of ROC which considers Type I and Type II
Table 3: RQ2: D2U training compared to zero-shot. "●" indicates statistically significant differences with the two-tailed paired t-test at a  $95\%$  interval in pairwise comparison between D2U-zero and best supervised.
Data	Method	ROC↑	FPR90↓	FNR90↓	REC↑	F1↑
ACID	D2U-zero	92.01	21.40	21.46	86.43	88.69
D2U-CE	96.75	7.30●	7.96	95.98	95.55
D2U-KL	96.78●	7.90	7.76●	96.31●	96.01●
D2U-S	95.88	8.80	9.54	93.18	93.78
Banking	D2U-zero	97.03	7.30	8.01	91.47	91.67
D2U-CE	99.36●	1.00●	0.23●	96.66	96.55
D2U-KL	99.25	1.70	0.39	97.47●	97.42●
D2U-S	98.79	2.00	2.12	95.90	95.88
CLINC	D2U-zero	96.32	7.50	8.36	91.31	91.75
D2U-CE	97.48	5.10	6.18	93.27	92.84
D2U-KL	97.29	5.20	4.93●	93.33	92.91
D2U-S	97.69●	3.90●	5.36	94.53●	94.53●
HWU64	D2U-zero	80.32	52.50	52.01	76.83	76.03●
D2U-CE	87.37●	31.70	37.05●	74.58	68.18
D2U-KL	87.19	30.30●	41.50	75.27	69.41
D2U-S	82.23	47.80	49.10	74.28	68.30
SNIPS	D2U-zero	96.33	8.40	9.29	88.35	88.43
D2U-CE	98.61	2.70	2.86	89.47	89.52
D2U-KL	99.16●	1.60●	1.57●	88.59	88.64
D2U-S	98.39	2.80	2.29	90.29	90.36
TOP	D2U-zero	73.24	53.00	69.07	84.54	86.14
D2U-CE	97.42	6.25	4.03●	94.55	95.01
D2U-KL	97.50●	5.88●	4.10	95.17●	95.51●
D2U-S	94.94	12.00	15.61	92.13	92.95
errors simultaneously. Entropy (Shen et al., 2021) is a strong baseline that performs better than other baselines with respect to all performance metrics.
RQ2: D2U in supervised setup. Next, we report the effect of D2U training on OOS detection in Table 3. Since our concern here is to ob
Table 4: RQ3: D2U vs. OOS detection baselines. The bold score is the best. The underlined score is the best that baseline achieves when D2U outperforms, or vice versa. "●" indicates statistically significant differences with the two-tailed paired t-test at a  $95\%$  interval (with Bonferroni correction  $p < 0.0071$ ) in pairwise comparisons between D2U and all baselines except the ones marked with "○". If baseline outperforms, "●" indicates the difference (with Bonferroni correction  $p < 0.0167$ ) in pairwise comparisons between the baseline and our best version.
Train	ROC	FPR	ACID	Banking	CLINC
FNR	REC	F1	ROC	FPR	FNR	REC	F1	ROC	FPR	FNR	REC	F1
MLE (Hendrycks et al., 2020)	90.9	25.0	67.9	84.9	87.5	94.9	13.3	14.4	89.4	89.6	95.3	9.3	11.8	90.2	90.8
Temp. (Liang et al., 2018)	91.8	22.1	69.1	85.8	88.2	96.2	10.0	10.5	90.3	90.5	96.2	7.8	9.2	90.1	90.7
Entropy (Shen et al., 2021)	92.0	19.9	51.5	86.9	89.0	96.7	7.9	7.2	91.6	91.8	95.9	8.0	8.9	90.3	90.8
Binary (Devlin et al., 2019)	97.2	6.2	6.7	96.5	96.1	99.9	0.2	0.2	97.9	97.8	85.6	48.6	31.4	88.3	86.1
LMCL (Zeng et al., 2021a)	94.1	15.6	66.5	88.4	90.1	97.2	6.3	8.1	92.6	92.6	96.3	7.4	9.9	90.8	91.3
DRM (Shen et al., 2021)	93.2	19.9	62.5	86.8	89.0	96.1	13.1	11.4	90.6	90.5	95.9	8.5	9.7	91.0	91.4
Reg. (Zheng et al., 2020)	96.0	10.3	7.1	95.6	95.1	99.0	2.4	0.9	96.8	96.8	97.3	6.5	6.8	93.3	92.9
D2U-CE-CE (ours)	96.8	7.3	8.0	96.0	95.6	99.4	1.0	0.2	96.7	96.6	97.5	5.1	6.2	92.3	92.8
D2U-KL-CE (ours)	96.8	7.9	7.8	96.3	96.0	99.3	1.7	0.4	97.5	97.4	97.3	5.2	4.9	93.3	92.9
D2U-S-CE (ours)	95.9	8.8	9.5	93.2	93.8	98.8	2.3	2.1	95.9	95.9	97.7	3.9	5.4	94.5	94.5
Train	ROC	FPR	HWU64	SNIPS	TOP
FNR	REC	F1	ROC	FPR	FNR	REC	F1	ROC	FPR	FNR	REC	F1
MLE (Hendrycks et al., 2020)	79.3	58.9	52.0	73.0	73.7	95.5	10.4	11.1	88.3	88.4	74.2	51.4	67.9	84.9	86.6
Temp. (Liang et al., 2018)	80.5	54.1	52.3	76.7	76.5	96.2	8.4	9.3	88.4	88.5	73.2	53.0	69.1	84.5	86.1
Entropy (Shen et al., 2021)	80.9	48.1	52.8	77.7	77.3	95.7	10.7	11.8	88.2	88.3	74.2	66.7	51.5	84.8	86.5
Binary (Devlin et al., 2019)	88.0	35.8	31.0	74.7	67.8	98.9	1.7	2.0	86.2	86.2	97.3	4.4	5.8	97.0	97.0
LMCL (Zeng et al., 2021a)	84.3	43.0	49.0	80.3	80.2	85.2	49.5	31.9	67.8	66.3	70.6	69.8	66.5	57.8	66.3
DRM (Shen et al., 2021)	79.3	56.9	50.3	73.4	74.0	93.6	13.4	12.0	87.9	87.9	77.0	50.1	62.5	81.7	84.4
Reg. (Zheng et al., 2020)	83.4	46.5	45.2	74.0	67.0	98.6	2.5	2.7	88.4	88.5	96.5	7.5	7.1	94.5	94.9
D2U-CE-CE (ours)	87.4	31.7	37.1	74.6	68.2	98.6	2.7	2.9	89.5	89.5	97.4	6.3	4.0	94.6	95.0
D2U-KL-CE (ours)	87.2	30.3	41.5	75.3	69.4	99.2	1.6	1.6	88.6	88.6	97.5	5.9	4.1	95.2	95.5
D2U-S-CE (ours)	82.2	47.8	49.1	74.3	68.3	98.4	2.8	2.3	90.3	90.4	94.9	12.0	15.6	92.1	93.0
serve any improvement over zero-shot setup, we fix post-processing method as cross-entropy for all methods due to its performance in the previous experiment. The results show that using D2U as a loss function statistically significantly improves the performance of D2U-zero in almost all cases. KL divergence loss (D2U-KL) and Cross-Entropy loss (D2U-CE) are effective D2U methods in all datasets, except that Sinkhorn distance (D2U-S) is effective in CLINC dataset. The choice of loss function is a hyperparameter that can be tuned according to specific use cases and datasets.
RQ3: D2U versus state-of-the-art. The performances of state-of-the-art baseline OOS detection models, regardless of zero-shot or supervised, and D2U methods are compared in Table 4, with extensive results reported in the Appendix. MLE, softmax temperature (Temp.), Entropy, LMCL, and DRM are zero-shot OOS detection setups, whereas entropy regularization (Reg.) and BERT-Binary (Binary) are supervised setups. D2U statistically significantly outperforms most baselines, although Binary is a strong baseline method that outperforms D2U in ACID and Banking datasets and challenges it in HWU64 and TOP, which is not statistically significant. The reason for this might be the prevalent domain difference between INS and OOS utterances in ACID, Banking and TOP datasets; which
belong to the insurance, banking, and navigation applications, respectively. It causes a trivial detection for the BERT-based binary classifier. HWU64 contains generic utterances like queries and questions which may coincide with the OOS split and disturb the training process of D2U. The combination of D2U training and D2U post-processing demonstrates its advantage in CLINC where INS and OOS utterances span a wide spectrum.
5.1 Domain analysis
To validate our hypothesis that domain-specific datasets provide an advantage to the Binary method, we apply UMAP (Becht et al., 2019) dimension reduction on the CLS embeddings of Binary and D2U CE models and plot them in Figure 5. It is apparent that the OOS utterances are separated from INS utterances when there is a clear domain difference as in ACID and Banking. However, when this separation becomes fuzzy, Binary fails to properly distinguish INS and OOS utterances as in CLINC. There is also an overlapping set of INS and OOS utterances in SNIPS for Binary.
In D2U-CE plots, the clusters of INS intents are easily identifiable since the model is trained for intent detection, however, OOS utterances do not
Figure 5: UMAP distributions of CLS embeddings.
form a separate cluster. We see that the training procedure does not necessarily enforce a clustering on the OOS embeddings since the model is trained to output a uniform distribution for OOS inputs. D2U achieves competitive performance even though there are overlapping embeddings of INS and OOS utterances, highlighting the importance of D2U post-processing in the supervised setup.
We provide a qualitative analysis on the effect of D2U training. We illustrate the model output distributions for INS utterance "get me to ritzville by 4 via the freeway." belonging to the "GET_DIRECTIONS" intent, and the OOS utterance "how many skating rinks are available in the south pacific tomorrow at  $10^{\prime \prime}$  taken from the TOP dataset in Figure 6. We observe that the OOS utterance results in an overconfident prediction in the BERT MLE model whereas the prediction distribution of D2U-CE is similar to uniform distribution.
5.3 INS performance
In Table 5, we analyze if OOS detection models deteriorate INS performance. MLE baseline does not modify the training procedure. The results show that INS classification performance is not dramatically deteriorated by the supervised models including D2U in SNIPS and TOP, whereas it is even improved in the remaining datasets. Although D2U's INS performance is similar to other supervised models, D2U has better OOS performance than others, as observed in Table 4. The reason for the increase in INS detection performance could be the regularization signal provided by the OOS loss as observed by Shen et al. (2021). Note that this effect becomes more apparent when domain difference is prevalent (in ACID and Banking).
We do not include Binary, which has no capability of INS classification. Binary has a challenging OOS performance in Table 4, but D2U has advan
Figure 6: The effect of D2U on prediction distributions.
Table 5: Weighted F1 score for INS classification.
Method	Datasets
ACID Banking	CLINC HWU64	SNIPS	TOP
MLE	80.74	84.91	95.67	81.97	98.14	98.60
LMCL	85.69	89.64	95.83	82.08	97.86	98.56
DRM	88.70	89.89	96.25	82.49	98.14	98.68
Reg.	86.60	91.22	96.38	82.55	97.43	98.32
D2U-CE	86.40	90.95	96.42	82.31	97.84	98.26
D2U-KL	86.26	90.69	96.22	82.72	98.01	98.29
D2U-S	86.50	91.52	96.34	82.16	97.43	98.37
tage of showing state-of-the-art performances for both INS and OOS detection.
We acknowledge some limitations to our study. Except for CLINC and TOP, the datasets we use are augmented with the OOS data from CLINC. However, we argue that the majority of the data remains OOS for other datasets since it is sampled from Wikipedia (Larson et al., 2019). Moreover, D2U has effective performance on CLINC and TOP datasets which are designed with OOS utterances.
We leave the selection of the distance metric in post-processing and supervised learning as a hyperparameter of D2U. In the results, this might provide an advantage to D2U in comparisons since we do not apply hyperparameter tuning for baselines. However, we use default or suggested parameter settings for baselines. We adopt transparency in reporting the results that are also detailed in Appendix.
Zero-shot D2U post-processing emphasizes the distinction between INS and OOS utterances when confidence score becomes misleading. However, D2U struggles in the ultimate case where an OOS utterance is mapped to an INS class with  $\sim 100\%$  confidence (see Figure 6 BERT MLE). Nonetheless, D2U suffers from such overconfident predictions less than existing methods (see Table 2).
5.5 Ethical considerations
We list a number of ethical concerns related to environmental impact, explainability, and transparency in this section. We employ BERT fine-tuning with
small modifications, therefore the environmental impact can be considered small. Our work focuses on well-known OOS detection task with established use cases, therefore there would be no risk for unintended use. We use publicly available datasets with licences suitable for academic research.
To assure explainability and transparency, we report the length of utterances and domains of datasets in Section 4.1. We report the statistics of datasets and figuratively report the split strategy used in the experiments in Section 4.1. There are two setups in our study. The zero-shot setting does not include any training. In the supervised setup, the complexity of our method is quite similar to regular fine-tuning procedure of BERT. The thresholding hyperparameter is decided by maximizing the Youden's J statistic as explained in Section 4.2. In Section 4.3, we also report the hyperparameters of the baseline methods. We employ a modified 10-fold cross-validation strategy as explained in Section 4.4 and apply t-test with Bonferroni correction to all experimental results.
We propose an OOS detection pipeline with a distance calculation between classifier prediction and uniform distribution, called D2U. In the zero-shot setup, D2U serves as an architecture-agnostic post-processing step to emphasize the distinction between INS and OOS. In the supervised setup, we bring closer OOS predictions to uniform distribution with a modified loss function. Experimental results, supported by statistical tests, show that D2U outperforms existing baselines in zero-shot, and has challenging performance in the supervised setup. We plan to extend our study to different architectures and deep learning tasks in the future.
Shailesh Acharya and Glenn Fung. 2020. Using optimal embeddings to learn new intents with few examples: An application in the insurance domain.
Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng, Florent Ginghoux, and Evan W Newell. 2019. Dimensionality reduction for visualizing single-cell data using umap. Nature Biotechnology, 37(1):38-44.
Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence en
coders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020. Data available at https://github.com/PolyAI-LDN/task-specific-datasets.
Paulo Cavalin, Victor Henrique Alves Ribeiro, Ana Appel, and Claudio Pinhanez. 2020. Improving out-of-scope detection in intent classification by using embeddings of the word graph space of the classes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3952-3961, Online. Association for Computational Linguistics.
Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021. *ProtAugment: Intent detection meta-learning through unsupervised diverse paraphrasing*. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics* and the 11th International Joint Conference on *Natural Language Processing* (Volume 1: Long Papers), pages 2454–2466, Online. Association for Computational Linguistics.
Yulan Feng, Shikib Mehri, Maxine Eskenazi, and Tiancheng Zhao. 2020. "none of the above": Measure uncertainty in dialog response retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2013-2020, Online. Association for Computational Linguistics.
Varun Gangal, Abhinav Arora, Arash Einolghozati, and Sonal Gupta. 2020. Likelihood ratios and generative classifiers for unsupervised out-of-domain detection in task oriented dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7764-7771.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
pages 2787-2792, Brussels, Belgium. Association for Computational Linguistics.
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744-2751.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations.
Chaojie Liang, Peijie Huang, Wenbin Lai, and Ziheng Ruan. 2021. Gan-based out-of-domain detection using both in-domain and out-of-domain samples. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7663-7667. IEEE.
Shiyu Liang, Yixuan Li, and R Srikant. 2018. Enhancing the reliability of out-of-distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018.
Ting-En Lin and Hua Xu. 2019a. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5491-5496, Florence, Italy. Association for Computational Linguistics.
Ting-En Lin and Hua Xu. 2019b. A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier. Knowledge-Based Systems, 186:104979.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In 10th International Workshop on Spoken Dialogue Systems Technology 2019.
Petr Marek, Vishal Ishwar Naik, Anuj Goyal, and Vincent Auvray. 2021. Oodgan: Generative adversarial
network for out-of-domain data generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pages 238-245.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020. Dialogue: A natural language understanding benchmark for task-oriented dialogue. arXiv preprint arXiv:2009.13570.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanxiang Che, and Ting Liu. 2021. GL-GIN: Fast and accurate non-autoregressive model for joint multiple intent detection and slot filling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 178-188, Online. Association for Computational Linguistics.
Yilin Shen, Yen-Chang Hsu, Avik Ray, and Hongxia Jin. 2021. Enhancing the generalization for intent classification and out-of-domain detection in SLU. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2443-2453, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020. A deep generative distance-based classifier for out-of-domain detection with mahalanobis space. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1452-1460.
Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. 2021. Unsupervised out-of-domain detection via pre-trained transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pages 1052-1061, Online. Association for Computational Linguistics.
Eyup Halit Yilmaz and Cagri Toraman. 2020. KLOOS: KL Divergence-Based Out-of-Scope Intent Detection in Human-to-Machine Conversations, page 2105-2108. Association for Computing Machinery, New York, NY, USA.
WJ Youden. 1950. Index for rating diagnostic tests. Cancer, 3(1):32-35.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 870-878, Online. Association for Computational Linguistics.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021b. Adversarial self-supervised learning for out-of-domain detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5631-5639.
Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiaoming Wu, and Albert Y.S. Lam. 2021. Out-of-scope intent detection with self-supervision and discriminative training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3521-3532, Online. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, S Yu Philip, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by transferring natural language inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5064-5082.
Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020. Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1198-1209.
We report Receiver Operating Curve Area Under Curve, False Positive Rate at  $90\%$  OOS True Positive Rate, False Negative Rate at  $90\%$  OOS True
Negative Rate, weighted OOS Recall, and weighted OOS F1 scores in Tables 6, 7, 8, 9, 10 respectively. Different training procedures, baseline and proposed, are reported in rows and different post-processing methods, baseline and proposed, are reported in columns.