Venue: NAACL 2022

# D2U: Distance-to-Uniform Learning for Out-of-Scope Detection

Eyup Halit Yilmaz

Cagri Toraman

Aselsan Research Center

Ankara, Turkey {ehyilmaz,ctoraman}@aselsen.com.tr

###### Abstract

Supervised training with cross-entropy loss implicitly forces models to produce probability distributions that follow a discrete delta distribution. Model predictions in test time are expected to be similar to delta distributions if the classifier determines the class of an input correctly. However, the shape of the predicted probability distribution can become similar to the uniform distribution when the model cannot infer properly. We exploit this observation for detecting out-of-scope (OOS) utterances in conversational systems. Specifically, we propose a zero-shot post-processing step, called Distance-to-Uniform (D2U), exploiting not only the classification confidence score, but the shape of the entire output distribution. We later combine it with a learning procedure that uses D2U for loss calculation in the supervised setup. We conduct experiments using six publicly available datasets. Experimental results show that the performance of OOS detection is improved with our post-processing when there is no OOS training data, as well as with D2U learning procedure when OOS training data is available.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Automated conversational systems have recently received attention from the research community [3, 4, 7]. In applications such as voice assistants, Spoken Language Understanding [6] aims to extract meaning from the user inputs, called _utterances_, in order to process and execute desired functionalities. The task of Intent Detection, or Intent Classification, aims to classify user utterance into a set of system-identifiable intents. However, supervised training of such systems can only cover a restricted set of classes, i.e. in-scope (INS) classes. To enhance user experience, the task of Out-of-Scope (OOS) detection [8, 9, 10, 11, 12] distinguishes INS utterances from those that do not belong to the scope of the classifier with dedicated model architectures and loss functions.

Existing methods in OOS detection utilize the classifier confidence score for a given utterance with thresholding to classify highly confident predictions as INS and lower confidence predictions as OOS. However, softmax classifiers suffer from overconfident predictions for OOS data [1], which makes it difficult to accurately determine a threshold value. Confidence loss [10] mitigates this by calculating the KL Divergence between model prediction and the uniform distribution to decrease the confidence for OOS input. We adapt a similar idea to the zero-shot setup with a novel post-processing step and exploit it jointly in the supervised setup with a learning procedure. The joint application of supervised D2U learning and D2U post-processing forms a novel OOS detection pipeline.

Figure 1 illustrates output probability distributions of a classifier for predicting the intent class for an INS and OOS utterance. The classifier, trained only on INS utterances, is confused when OOS utterance is given. The model assigns closer probabilities for different classes since there is no _correct_ class for this OOS utterance, hence the resulting distribution gets closer to a uniform distribution

Figure 1: Sample output distributions of an INS classifier predicting the intent of an INS and OOS utterance. Since OOS utterances do not belong to any intent, the prediction gets closer to the uniform distribution.



than a discrete delta distribution.

Based on this observation, we propose to measure the dissimilarity from or distance to the uniform distribution (D2U). Statistical distance calculations between the prediction and uniform distribution enable the decision boundary to be more accurate. Figure 2 illustrates possible benefits of using distance to the uniform distribution with cross-entropy. The subplot at the left shows the distribution of the number of utterances according to their Maximum Likelihood Estimate (MLE) score. The subplot at the right shows the same distribution according to cross-entropy score between prediction probability and uniform distribution. A decision boundary or threshold can be easily determined using D2U's cross-entropy as a post-processing step without any OOS training data.

When OOS training data is available Larson et al. (2019), D2U can be used as a loss function to minimize the distance between OOS predictions and the uniform distribution. Such a loss function forces OOS predictions to be less confident, and benefit D2U post-processing further. To test our hypothesis that D2U is a useful method for OOS detection, we answer following research questions:

* **RQ1:** Does the application of D2U as a post-processing step on INS classifier predictions increase OOS detection performance when there is no OOS training data?
* **RQ2:** Does incorporating D2U into the training procedure as a particular loss function boost performance when OOS training data is available?
* **RQ3:** Is the performance of OOS detection significantly improved by D2U over existing state-of-the-art methods?

## 2 Related Work

We divide OOS detection studies into three categories: (i) Confidence-based, (ii) representation-based, and (iii) distance-based methods.

### Confidence-based OOS detection Threshold-based Methods

Threshold-based MethodsThesholding is a common approach in OOS detection Larson et al. (2019); Feng et al. (2020); Zhang et al. (2020), which reflects the intuition that a classifier output is more confident for a sample that follows its training distribution. The overconfidence problem of softmax classifiers Hendrycks and Gimpel (2017), although less apparent in Transformer-based Vaswani et al. (2017) models Hendrycks et al. (2020), hinders threshold-based OOS detection performance.

Post-processing MethodsThe overconfidence problem of softmax classifiers is tackled by post-processing predictions. ODIN Liang et al. (2018) and SofterMax Lin and Xu (2019) apply temperature scaling for enlarging the confidence gap between INS and OOS instances, since INS logits are ideally further away on the positive axis of the softmax input. Gangal et al. (2020) utilize likelihood ratios with generative classifiers to distinguish OOS predictions. Our method, D2U, employs a confidence-based post-processing method.

### Representation-based OOS detection

Dedicated model architectures or loss functions help represent utterances in a high-dimensional space suitable for OOS detection. Large Margin Cosine Loss (LMCL) ensures that INS intents are tightly clustered Zeng et al. (2021), so that OOS utterances are exposed for outlier detection algorithms, such as Local Outlier Factor Lin and Xu (2019). Intent class embeddings Cavalain et al. (2020) model OOS detection as a reverse dictionary task by mapping intent classes and utterances to the same space. Yilmaz and Toraman (2020) propose a feature representation mechanism that uses KL Divergence to capture the changes in model predictions during sequential processing of utterances.

In order to mitigate data scarcity Marek et al. (2021) propose a method to generate OOS data with Generative Adversarial Networks. GANs are also utilized to generate high-dimensional representations that are hard to distinguish from that of real utterances, providing adversarial signals to the INS classifier which increases the robustness of the model Zeng et al. (2021b); Liang et al. (2021).

Figure 2: The histogram of prediction probability scores for INS and OOS utterances (MLE) by using a classifier trained on only INS utterances at the left. Instead of MLE, for the same classifier, cross-entropy score between prediction distribution and uniform distribution (D2U) is given at the right. A vertical decision boundary is more accurately determined with D2U.

 



### Distance-based OOS detection

Distances and divergences are useful tools in OOS detection, since they provide a measure of dissimilarity that can distinguish INS and OOS samples. Xu et al. (2020) utilize Euclidean and Mahalanobis distances with generative classifiers to identify outliers with Gaussian Discriminative Analysis. Mahalanobis distance calculated using representations from the intermediate layers of BERT Devlin et al. (2019) increases OOS detection performance Shen et al. (2021). Lee et al. (2018) introduce the confidence loss in Computer Vision for GANs that calculates KL Divergence between the training predictions for OOS samples and uniform distribution.

The idea of measuring the distance between prediction distribution and uniform distribution is utilized in different learning architectures Lee et al. (2018); Gangal et al. (2020), but not extensively studied for OOS intent detection. Besides, we explore various distance metrics in zero-shot OOS detection and different distance-to-uniform training procedures in supervised setup.

## 3 Distance-to-Uniform OOS Detection

### D2U post-processing for zero-shot setup

Supervised classifiers trained on INS data model the ground truth labels with a discrete delta function that corresponds to the label, given as follows.

\[\delta_{c_{i}}(x)=\begin{cases}1,&\text{if $x=c_{i}$}\\ 0,&\text{otherwise}\end{cases} \tag{1}\]

For the data instance \(i\), \(c_{i}\) is the ground-truth label indicating the correct class. The cross-entropy loss between softmax model output and discrete delta function is given as follows.

\[L_{CE}=-\frac{1}{N}\sum_{i=1}^{N}\delta_{c_{i}}(x)\log\hat{P}(u_{i}) \tag{2}\]

Here, \(\hat{P}(u_{i})\) is the output probability distribution of the model for utterance \(u_{i}\) in a batch of \(N\) utterances, and \(c_{i}\) is the correct class label for the given utterance. This criterion implicitly forces the model to generate confident predictions for a given data point with maximal confidence score assigned to the ground-truth class label, and low prediction scores for the other classes. When an OOS utterance is given to an intent classifier that is trained using only INS data, the classifier gets confused, i.e., the output probability distribution is more dissimilar to a delta distribution than what an INS utterance would result in. In other words, output distributions of OOS samples get closer to the uniform distribution than that of INS samples, an observation that we exploit for OOS detection.

The conventional methods for OOS detection make use of a pre-determined threshold value on the Maximum Likelihood Estimate (MLE) score assigned to the predicted label, given as follows.

\[OOS(u_{i})=\begin{cases}1,&\text{if $max(\hat{P}(u_{i}))<\theta$}\\ 0,&\text{otherwise}\end{cases} \tag{3}\]

Here, \(\theta\) is a pre-defined threshold value between 0 and 1, and \(max(\hat{P}(u_{i}))\) is the MLE score, which considers only the confidence and ignores the shape of the distribution. We exploit the information conveyed by the shape of the entire prediction distribution by first calculating a distance between the output distribution \(\hat{P}\) and the uniform distribution \(U\) before applying the threshold, given as follows.

\[OOS(u_{i})=\begin{cases}1,&\text{if $dst(\hat{P}(u_{i}),U)<\theta$}\\ 0,&\text{otherwise}\end{cases} \tag{4}\]

The distance determined by the \(dst(.)\) function between \(\hat{P}(u_{i})\) and \(U\) can be calculated with various distance metrics. We experiment with geometric distance calculations, such as Euclidean distance and Cosine distance; as well as statistical distance calculations, such as Jensen-Shannon distance and symmetrized Kullback-Leibler divergence. The distance value calculated by the \(dst(.)\) function can be intuitively interpreted as the level of confidence of the model. When the distance value is low, the model is less confident and more confused, since the output distribution assigns closer scores for each class.

This is an architecture-agnostic zero-shot post-processing step which can be generalized to any classification model trained with cross-entropy loss with no need for OOS training data. OOS detection in test time is achieved by a function of the prediction distribution given by D2U.

### Distance metrics for post-processing

We examine a number of geometric and statistical distance measures listed as follows.

* Bray Curtis Distance (BC): For two probability distributions, \(u\) and \(v\), the Bray Curtis distance is given as \(\sum_{i}|u_{i}-v_{i}|/\sum_{i}|u_{i}+v_{i}|\)

* Canberra Distance (Cbr): Canberra distance between \(u\) and \(v\) is \(\sum_{i}{(|u_{i}-v_{i}|/(u_{i}+v_{i}))}\).
* Cosine Distance (Cos): Derived from the Cosine similarity, the cosine distance is formulated as \(1-(u\cdot v/||u||_{2}||v||_{2})\) where \(||.||_{2}\) is the \(L_{2}\) norm.
* Euclidean Distance (Euc): The Euclidean distance between \(u\) and \(v\) is given as \(||u-v||_{2}\).
* Hellinger Distance (Helng): The Hellinger distance between \(u\) and \(v\) is \(||\sqrt{u}-\sqrt{v}||_{2}/\sqrt{2}\).
* Cross-Entropy (CE): Cross-Entropy is a measure of dissimilarity between distributions \(u\) and \(v\) given as \(-\sum_{i}{u_{i}\log v_{i}}\).
* Symmetrized KL Divergence (KL): The symmetrized Kullback-Leibler divergence is given as \([KL(u,v)+KL(v,u)]/2\) where \(KL(u,v)=\sum_{i}{u_{i}\log{(u_{i}/v_{i})}}\).
* Jenshen Shannon Distance (JS): JS distance between \(u\text{ and }v\) is \(KL(u,m)/2+KL(v,m)/2\) where \(m\) is the mean of two distributions.

### D2U training for supervised setup

When OOS training data is available, we modify the fine-tuning procedure as in Figure 3, to increase the similarity between OOS prediction and uniform distribution. We use pretrained BERT Devlin et al. (2019) as the classifier network. The loss function for INS utterances, \(L_{ins}\), is still cross-entropy between true label and prediction, given as follows.

\[L_{ins}=-\frac{1}{N_{ins}}\sum_{i=1}^{N_{ins}}\delta(c_{i})\log\hat{P}(u_{i}) \tag{5}\]

For OOS utterances, the loss \(L_{oos}\), is calculated against the uniform distribution, given as follows.

\[L_{oos}=\frac{1}{N_{oos}}\sum_{i=1}^{N_{oos}}dst(\hat{P}(u_{i}),U) \tag{6}\]

The total loss is the weighted average over a batch of utterances containing \(N_{ins}\) number of INS utterances and \(N_{oos}\) number of OOS utterances, given as follows.

\[L_{total}=\frac{N_{ins}L_{ins}+N_{oos}L_{oos}}{N_{ins}+N_{oos}} \tag{7}\]

As the \(dst(.)\) function in Equation 6, we experiment with differentiable functions; such as cross-entropy, KL divergence, and Sinkhorn distance Cuturi (2013), named as D2U-CE, D2U-KL, and D2U-S, respectively. These functions treat the model output and ground truth as probability distributions and provide a differentiable measure. We do not modify the loss calculation for INS utterances so as not to affect the INS classification performance.

Note that this architecture does not model the OOS intent as a separate class. Therefore, post-processing is applied as described in Section 3.1 in test time. Since the loss function incorporates D2U into training, the performance gain by the post-processing is expected to increase.

## 4 Experiments

### Datasets

We use six publicly available intent classification datasets, some of which include labeled OOS data. We give the main statistics of the datasets in Table 1. CLINC Larson et al. (2019) is a dataset with 150 INS intent classes targeting various domains with curated OOS data. We use the OOS split of CLINC to augment other existing intent detection datasets that do not include labeled OOS data; which are ACID Acharya and Fung (2020), Banking Casanueva et al. (2020), HWU64 Liu et al. (2019), and SNIPS Coucke et al. (2018). We observe that HWU64 has many short and noisy utterances, we therefore remove any utterances with length less than or equal to three words.

TOP Gupta et al. (2018) is an intent detection dataset that generalizes conventional intent labeling with semantic parsing. The intent labels follow a hierarchical structure with potentially many

Figure 3: The supervised learning architecture of D2U. INS utterances are learned with conventional cross-entropy loss against true label distribution, while OOS loss is calculated against the uniform distribution.

 

labels for an utterance. However, we take only root intent class label into account to be consistent with other datasets. The utterances with the intent labels "UNSUPPORTED" and "UNSUPPORTED_NAVIGATION" are treated as OOS.

The variety of the number of classes, average length (number of words), and vocabulary size provide a wide spectrum for understanding different OOS detection scenarios. For instance, TOP dataset can be considered a low resource setup since the number of OOS utterances is significantly lower than INS utterances.

### Evaluation metrics

To assess the performance of OOS detection, we report the scores of Receiver Operating Curve Area Under Curve (ROC AUC), False Positive Rate at 90% OOS True Positive Rate (FPR90), and False Negative Rate at 90% OOS True Negative Rate (FNR90) using sklearn Pedregosa et al. (2011). These metrics are independent of the threshold value used for decision boundary, providing a means of fair comparison. We also report weighted OOS Recall and weighted OOS F1 based on the threshold value that maximizes the Youden's J statistic Youden (1950) on a validation set.

Compared to Precision, Recall is arguably a more critical performance metric for OOS detection; since Recall considers Type II error, meaning that OOS utterances are mislabeled as INS. In this case, the voice assistant would execute a task that the user does not intent to do. We argue that ROC is a more generic measure that considers the performances of varying thresholds, than Recall and F1 considering only a fixed threshold.

### Baseline approaches

In the experiments, BERT Devlin et al. (2019) with softmax layer is used as the classifier network. For RQ1, the baseline zero-shot post-processing approaches are listed below.

* **MLE Hendrycks and Gimpel (2017); Hendrycks et al. (2020): The confidence score of a classifier trained only on INS data is used for thresholding.
* **Softmax temperature scaling (Temp)**Liang et al. (2018); Lin and Xu (2019): As a modification to the MLE setup, the softmax input is applied a temperature value of \(10^{3}\).
* **Standard deviation (Stdev)**: We use the stdev of the distribution before thresholding since OOS predictions would have lower standard deviation.
* **Entropy (Ent)** Shen et al. (2021): The entropy of the prediction distribution, \(H(\hat{P}(u_{i}))\), is calculated before applying the threshold, as follows. \[OOS(u_{i})=\begin{cases}1,&\text{if $H(\hat{P}(u_{i}))>\theta$}\\ 0,&\text{otherwise}\end{cases}\] (8) For RQ2, we use D2U zero-shot cross-entropy post-processing (D2U-zero) as the baseline method, since we examine any improvement in supervised setup over zero-shot. For RQ3, we compare supervised D2U with the following baselines.
* **Large Margin Cosine Loss (LMCL)**Zeng et al. (2021b): Cosine distance among INS class centroids is increased up to a margin. We set the margin as 0.35, and scaling factor as 30.
* **Domain Regularization Module (DRM)**Shen et al. (2021): DRM introduces domain logits for regularization during INS training. We slightly modify the design and apply sigmoid to domain logits before dividing the classification logits for training stability.
* **BERT-Binary (Binary)**Devlin et al. (2019): The "bert-base-uncased" model fine-tuned as a binary classifier for OOS detection.
* **Entropy Regularization (Reg.)**Zheng et al. (2020): Entropy of OOS predictions are maximized while minimizing INS training loss.

### Experimental design

To avoid potential annotator-dependent effects as noted by Larson et al. (2019) and comply with the original splits, we modify 10-fold leave-one-out cross-validation as illustrated in Figure 4. The validation splits are used to find confidence threshold values for Recall and F1 calculations. We validate statistically significant differences in the average performances of 10-folds with the two-tailed paired t-test at a 95% interval w

Figure 4: Modified leave-one-out 10-fold split strategy that complies with original splits. At each fold, only 10% of test data is included, while 90% of training data is retained and the remaining 10% is used as validation.

 

Note that the test splits do not overlap in order to satisfy the independence criterion of t-test.

The experiments are designed with respect to our research questions (RQ 1-3). First, we fine-tune a BERT classifier Devlin et al. (2019) using huggingface implementation Wolf et al. (2019) for INS intent detection with cross-entropy loss, and apply different D2U post-processing methods for RQ1. Then, we fix the post-processing method, and examine the effect of supervised D2U training for RQ2. Lastly, we compare D2U with state-of-the art baselines for RQ3 to assess the performance gain of our method.

### Experimental results

**RQ1: D2U in zero-shot setup.** In Table 2, we report ROC AUC, FPR90, and FNR90 scores for different post-processing methods applied to a BERT-based INS classifier with no OOS training data. Our proposed method, D2U-zero, statistically significantly outperforms all baselines in all datasets with respect to ROC AUC score. Using cross-entropy for D2U-zero has better performance in majority of cases, compared to other distance metrics. The reason for its success might be that cross-entropy is the loss function used in the training procedure of the model. In terms of FPR90 and FNR90, D2U-zero does not always outperform all baselines. Though, the cases when baselines outperform are not statistically significant. This shows that the baseline methods can optimize FPR90 and FNR90 individually but cannot outperform D2U in terms of ROC which considers Type I and Type II errors simultaneously. Entropy Shen et al. (2021) is a strong baseline that performs better than other baselines with respect to all performance metrics.

**RQ2: D2U in supervised setup.** Next, we report the effect of D2U training on OOS detection in Table 3. Since our concern here is to ob

\begin{table}
\begin{tabular}{l l c r r r|r r r r r r r r} \hline \hline Metric & Dataset & \multicolumn{4}{c|}{Baselines} & \multicolumn{6}{c}{D2U-zero} \\  & MLE & Temp & Stdev & Ent & CE & BC & Chr & Cos & JS & Euc & KL & Helng. \\ \hline \multirow{5}{*}{ROC AUC (\(\uparrow\))} & ACID & 90.93 & 91.83o & 91.54 & 91.98 & 92.01 & 91.40 & 90.18 & 91.54 & 92.08 & 91.54 & **92.14*** & 92.13 \\  & Banking & 94.92 & 96.15 & 95.88 & 96.66 & **97.03*** & 96.89 & 96.09 & 95.88 & 96.93 & 95.86 & 96.97 \\  & CLINC & 95.94 & 96.16 & 95.52 & 95.90 & **96.32*** & 96.09 & 96.26 & 95.52 & 96.24 & 95.52 & 96.20 & 96.25 \\  & HVHUG & 92.90 & 80.46 & 79.95 & 80.00 & 80.32 & **81.49*** & 80.49 & 79.95 & 81.16 & 79.95 & 80.91 & 81.05 \\  & SNIPS & 95.45 & 96.29 & 95.50 & 95.70 & **96.33*** & 95.61 & 95.83 & 95.90 & 95.86 & 95.50 & 96.15 & 96.04 \\  & TOP & 74.23 & 73.23 & 74.26 & 74.19 & 73.24 & 74.32 & **74.36*** & 74.26 & 74.18 & 74.26 & 73.25 & 73.76 \\ \hline \hline \multirow{5}{*}{FPR90 (\(\downarrow\))} & ACID & 25.00 & 22.10e & 20.70o & 19.85o & 21.40 & 24.80 & 28.60 & 20.70 & 20.90 & 20.70 & **19.70*** & 20.70 \\  & Banking & 13.00 & 10.00 & 11.30 & 7.85 & 7.30 & 6.60 & 12.00 & 11.30 & 6.20 & 11.30 & 6.30 & **6.10*** \\  & CLINC & 93.0 & 78.00 & 93.00 & 8.000 & **7.50*** & 8.10 & 7.00 & 93.00 & 7.80 & 7.60 \\  & HWU64 & 58.90 & 54.10 & 55.70 & **48.12** & 52.50 & 51.60 & 55.00 & 55.70 & 52.20 & 55.70 & 52.60 & 51.90 \\  & SNIPS & 10.40 & **8.40** & 10.30 & 10.71 & **8.40** & 10.40 & 10.30 & 10.30 & 9.90 & 10.30 & 8.60 & 8.70 \\  & TOP & **51.83** & 53.00 & **51.38** & 66.65 & 35.00 & **51.38** & **51.38** & **51.35** & 51.50 & **51.38** & 53.50 & 51.75 \\ \hline \hline \multirow{5}{*}{FNR90 (\(\downarrow\))} & ACID & 27.30 & 25.67 & 26.22 & **19.70*** & 21.46 & 20.60o & 26.70 & 26.22 & 20.45o & 26.22 & 21.38o & 20.75o \\  & Banking & 14.36 & 10.49 & 11.37 & **7.20*** & 8.010 & 8.79o & 13.68 & 11.37 & 7.88o & 11.37 & 7.69o & 7.79o \\  & CLINC & 11.80 & 9.16 & 11.60 & 8.90 & 8.36 & 8.11 & **7.56e** & 11.60 & 7.98 & 11.60 & 8.67 & 8.31 \\  & WHU64 & 54.97 & 52.66 & 51.75 & 52.80 & 52.01 & 47.14 & 47.14 & 51.75 & 48.63 & 51.75 & 51.03 & 49.57 \\  & SNIPS & 11.14 & **9.29** & 11.14 & 11.80 & **9.29** & 11.14 & 11.14 & 11.14 & 10.71 & 11.14 & 9.71 & 10.29 \\  & TOP & 67.88 & 69.05 & 67.89 & **51.50** & 69.07 & 67.89 & 67.90 & 68.17 & 67.89 & 69.04 & 68.46 \\ \hline \hline \end{tabular}
\end{table}
Table 2: **RQ1:** D2U-zero with various distance metrics vs. post-processing baselines in zero-shot setup. Row-wise best scores are given in bold. (\(\uparrow\)) and (\(\downarrow\)) indicate that higher and lower scores are better, respectively. “\(\bullet\)” indicates statistically significant differences with two-tailed paired t-test at a 95% interval (with Bonferroni correction \(p\) < 0.0125) in pairwise comparison between D2U-zero and all baselines except the ones marked with ”\(\circ\)".

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline Data & Method & ROC\(\uparrow\) & FPR90\(\downarrow\) & FNR90\(\downarrow\) & REC\(\uparrow\) & F1\(\uparrow\) \\ \hline \multirow{5}{*}{FNRP90 (\(\downarrow\))} & D2U-zero & 92.01 & 21.40 & 21.46 & 86.43 & 88.69 \\  & D2U-CE & 96.75 & **7.30*** & 7.96 & 95.98 & 95.55 \\  & D2U-KL & **96.78*** & 7.90 & **7.76*** & **96.31*** & **96.01*** \\  & D2U-S & 95.88 & 8.80 & 9.54 & 93.18 & 93.78 \\ \hline \hline \multirow{5}{*}{FNR90 (\(\downarrow\))} & D2U-zero & 97.03 & 7.30 & 8.01 & 91.47 & 91.67 \\  & D2U-CE & **99.36*** & **1.00*** & **0.23*** & 96.66 & 96.55 \\  & D2U-KL & **99.95** & 2.50 & 1.70 & **93.97*** & **97.47*** & **97.42*** \\  & D2U-S & 98.79 & 2.00 & 2.12 & 95.90 & 95.88 \\ \hline \hline \multirow{5}{*}{FNR90 (\(\downarrow\))} & D2U-zero & 96.32 & 7.50 & 8.36 & 91.31 & 91.75 \\ \cline{2-11}  & D2U-CE & 97.48 & 5.10 & 6.18 & 93.27 & 92.84 \\ \cline{1-1}  & D2U-KL & 97.29 & 5.20 & **4.93*** & 93.33 & 92.91 \\ \cline{1-1}  & D2U-S & **97.69*** & **3.90*** & 5.36 & **94.53*** & **94.53*** \\ \hline \hline \multirow{5}{*}{FNR90 (\(\downarrow\))} & D2U-zero & 

serve any improvement over zero-shot setup, we fix post-processing method as cross-entropy for all methods due to its performance in the previous experiment. The results show that using D2U as a loss function statistically significantly improves the performance of D2U-zero in almost all cases. KL divergence loss (D2U-KL) and Cross-Entropy loss (D2U-CE) are effective D2U methods in all datasets, except that Sinkhorn distance (D2U-S) is effective in CLINC dataset. The choice of loss function is a hyperparameter that can be tuned according to specific use cases and datasets.

**RQ3: D2U versus state-of-the-art.** The performances of state-of-the-art baseline OOS detection models, regardless of zero-shot or supervised, and D2U methods are compared in Table 4, with extensive results reported in the Appendix. MLE, softmax temperature (Temp.), Entropy, LMCL, and DRM are zero-shot OOS detection setups, whereas entropy regularization (Reg.) and BERT-Binary (Binary) are supervised setups. D2U statistically significantly outperforms most baselines, although Binary is a strong baseline method that outperforms D2U in ACID and Banking datasets and challenges it in HWU64 and TOP, which is not statistically significant. The reason for this might be the prevalent domain difference between INS and OOS utterances in ACID, Banking and TOP datasets; which belong to the insurance, banking, and navigation applications, respectively. It causes a trivial detection for the BERT-based binary classifier. HWU64 contains generic utterances like queries and questions which may coincide with the OOS split and disturb the training process of D2U. The combination of D2U training and D2U post-processing demonstrates its advantage in CLINC where INS and OOS utterances span a wide spectrum.

## 5 Discussion

### Domain analysis

To validate our hypothesis that domain-specific datasets provide an advantage to the Binary method, we apply UMAP [1] dimension reduction on the CLS embeddings of Binary and D2U CE models and plot them in Figure 5. It is apparent that the OOS utterances are separated from INS utterances when there is a clear domain difference as in ACID and Banking. However, when this separation becomes fuzzy, Binary fails to properly distinguish INS and OOS utterances as in CLINC. There is also an overlapping set of INS and OOS utterances in SNIPS for Binary.

In D2U-CE plots, the clusters of INS intents are easily identifiable since the model is trained for intent detection, however, OOS utterances do not

\begin{table}
\begin{tabular}{l|c c c c c|c c c c c c c c} \hline \hline Train & \multicolumn{8}{c|}{ACID} & \multicolumn{8}{c|}{Banking} & \multicolumn{8}{c}{CLINC} \\  & ROC FPR & FNR REC & F1 & ROC & FPRF & FNR REC & F1 & ROC & FPRF & FNR REC & F1 \\ \hline MLE [7] & 90.9 & 25.0 & 67.9 & 84.9 & 87.5 & 94.9 & 13.3 & 14.4 & 89.4 & 89.6 & 95.3 & 9.3 & 11.8 & 90.2 & 90.8 \\ Temp. [10] & 91.8 & 22.1 & 69.1 & 85.8 & 88.2 & 96.2 & 10.0 & 10.5 & 90.3 & 90.5 & 96.2 & 7.8 & 9.2 & 90.1 & 90.7 \\ Entropy [10] & 92.0 & 19.9 & 51.5 & 86.9 & 89.0 & 96.7 & 7.2 & 91.6 & 91.8 & 95.9 & 8.0 & 8.9 & 90.3 & 90.8 \\ Binary [11] & **97.2** & **6.2** & **6.7** & **96.5** & **96.1** & **99.9** & **0.2** & **0.2** & **97.9** & **97.8** & 85.6 & 48.6 & 31.4 & 88.3 & 86.1 \\ LMCL [12] & 94.1 & 15.6 & 66.5 & 88.4 & 90.1 & 97.2 & 6.3 & 8.1 & 92.6 & 92.6 & 96.3 & 7.4 & 9.9 & 90.8 & 91.3 \\ DRM [12] & 93.2 & 19.9 & 62.5 & 86.8 & 89.0 & 96.1 & 13.1 & 11.4 & 90.6 & 90.9 & 95.9 & 8.5 & 9.7 & 91.0 & 91.4 \\ Reg. [12] & 96.0 & 10.3 & 7.1 & 95.6 & 95.1 & 99.0 & 2.4 & 0.9 & 96.8 & 96.8 & 97.3 & 6.5 & 6.8 & 93.3o & 92.9o \\ \hline D2U-CE-CE (ours) & 96.8 & 7.3 & 8.0 & 96.0 & 95.6 & 99.4 & 1.0 & 0.2 & 96.7 & 96.6 & 97.5 & 5.1 & 6.2 & 92.3 & 92.8 \\ D2U-KL-CE (ours) & 96.8 & 7.9 & 7.8 & 96.3 & 96.0 & 99.3 & 1.7 & 0.4 & 97.5 & 97.4 & 97.3 & 5.2 & 4.9 & 93.3 & 92.9 \\ D2U-SC-GE (ours) & 95.9 & 8.8 & 9.5 & 93.2 & 93.8 & 98.8 & 2.3 & 2.1 & 95.9 & 95.9 & **97.4** & **39.5** & 4.9 & **94.5*** \\ \hline \hline Train & \multicolumn{8}{c|}{HWU64} & \multicolumn{8}{c|}{SNIPS} & \multicolumn{8}{c}{TOP} \\  & ROC FPR & FNR REC & F1 & ROC & FPR & FNR REC & F1 & ROC & FPRF & FNR REC & F1 \\ \hline MLE [7] & 79.3 & 58.9 & 52.0 & 73.0 & 73.7 & 95.5 & 10.4 & 11.1 & 88.3o & 88.4o & 74.2 & 51.4 & 67.9 & 84.9 & 86.6 \\ Temp. [10] & 80.5 & 54.1 & 52.3 & 76.7 & 76.5 & 96.2 & 8.4 & 9.3 & 88.4o & 88.5o & 73.2 & 53.0 & 69.1 & 84.5 & 86.1 \\ Entropy [10] & 80.9 & 48.1 & 52.8 & 77.7 & 77.3 & 95.7 & 10.7 & 11.8 & 88.8o & 73.2 & 74.6 & 65.7 & 51.5 & 84.8 & 86.5 \\ Binary [11] & **88.0** & 35.8 & **31.0** & 74.7 & 67.8 & 98.9 & 17.0 & 2.0 & 86.2 & 86.2o & 97.3 & **44.4** & 5.8 & **97.0*** & **97.0*** \\ LMCL [12] & 84.3 & 43.0 & 49.0 & **80.a** & **80.e** & 82.5 & 49.5 & 31.9 & 67.8 & 66.3 & 70.6 & 69.8 & 66.5 & 67.5 & 87.6 \\ DRM [12] & 79.3 & 56.9 & 59.0 & 73.4 & 74.0 & 93.6 & 13.9 & 14.2 & 70.9 & 89.0 & 77.0 & 50.1 & 62.5 & 81.7 & 84.4 \\ Reg. [13] & 83.4 & 46.5 & 45.2 & 74.0 & 67.0 & 98.6 & 25.0 & 27.8 & 88.4o & 88.5o & 96.5 & 7.5 & 7.1o & 94.5 & 94.9 \\ \hline D2U-CE-CE (ours) & 87.4 & 31.7 & 37.1 & 74.6 & 68.2 & 98.6 & 2.7 & 2.9 & 89.5 & 89.5 & 97.4 & 6.3 & **4.0*** & 94.6 & 95.0 \\ D2U-KL-CE (ours) & 87.2 & **30.3*** & 41.5 & 75.3 & 69.4 & **99.2*** & **1.6***** & **1.6***** & 88.6 & 88.6 & **97.5*** & 5.9 & 4.1 & 95.2 & 95.5 \\ D2U-SL-CE (ours) & 82.2 & 47.8 & 49.1 & 74.3 & 68.3 & 98.4 & 2.8 & 2.3 & **90.3*** & **90.4*** & 94.9 & 12.0 & 15.6 & 92.1 & 93.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: **RQ3: D2U vs. OOS detection baselines. The bold score is the best. The underlined score is the best that baseline achieves when D2U outperforms, or vice versa. “\(\bullet\)” indicates statistically significant differences with the two-tailed paired t-test at a 95% interval (with Bonferroni correction \(p\) < 0.0071) in pairwise comparisons between D2U and all baselines except the ones marked with ”\(\circ\)". If baseline outperforms, ”\(\bullet\)” indicates the difference (with Bonferroni correction \(p\) < 0.0167) in pairwise comparisons between the baseline and our best version.** 

form a separate cluster. We see that the training procedure does not necessarily enforce a clustering on the OOS embeddings since the model is trained to output a uniform distribution for OOS inputs. D2U achieves competitive performance even though there are overlapping embeddings of INS and OOS utterances, highlighting the importance of D2U post-processing in the supervised setup.

### Qualitative analysis

We provide a qualitative analysis on the effect of D2U training. We illustrate the model output distributions for INS utterance _"get me to ritzville by 4 via the freeway."_ belonging to the "GET_DIRECTIONS" intent, and the OOS utterance _"how many skating riks are available in the south pacific tomorrow at 10"_ taken from the TOP dataset in Figure 6. We observe that the OOS utterance results in an overconfident prediction in the BERT MLE model whereas the prediction distribution of D2U-CE is similar to uniform distribution.

### INS performance

In Table 5, we analyze if OOS detection models deteriorate INS performance. MLE baseline does not modify the training procedure. The results show that INS classification performance is not dramatically deteriorated by the supervised models including D2U in SNIPS and TOP, whereas it is even improved in the remaining datasets. Although D2U's INS performance is similar to other supervised models, D2U has better OOS performance than others, as observed in Table 4. The reason for the increase in INS detection performance could be the regularization signal provided by the OOS loss as observed by Shen et al. (2021). Note that this effect becomes more apparent when domain difference is prevalent (in ACID and Banking).

We do not include Binary, which has no capability of INS classification. Binary has a challenging OOS performance in Table 4, but D2U has advantage of showing state-of-the-art performances for both INS and OOS detection.

### Limitations

We acknowledge some limitations to our study. Except for CLINC and TOP, the datasets we use are augmented with the OOS data from CLINC. However, we argue that the majority of the data remains OOS for other datasets since it is sampled from Wikipedia Larson et al. (2019). Moreover, D2U has effective performance on CLINC and TOP datasets which are designed with OOS utterances.

We leave the selection of the distance metric in post-processing and supervised learning as a hyperparameter of D2U. In the results, this might provide an advantage to D2U in comparisons since we do not apply hyperparameter tuning for baselines. However, we use default or suggested parameter settings for baselines. We adopt transparency in reporting the results that are also detailed in Appendix.

Zero-shot D2U post-processing emphasizes the distinction between INS and OOS utterances when confidence score becomes misleading. However, D2U struggles in the ultimate case where an OOS utterance is mapped to an INS class with \(\sim\)100% confidence (see Figure 6 BERT MLE). Nonetheless, D2U suffers from such overconfident predictions less than existing methods (see Table 2).

### Ethical considerations

We list a number of ethical concerns related to environmental impact, explainability, and transparency in this section. We employ BERT fine-tuning with

Figure 5: UMAP distributions of CLS embeddings.

Figure 6: The effect of D2U on prediction distributions.

 

small modifications, therefore the environmental impact can be considered small. Our work focuses on well-known OOS detection task with established use cases, therefore there would be no risk for unintended use. We use publicly available datasets with licences suitable for academic research.

To assure explainability and transparency, we report the length of utterances and domains of datasets in Section 4.1. We report the statistics of datasets and figuratively report the split strategy used in the experiments in Section 4.1. There are two setups in our study. The zero-shot setting does not include any training. In the supervised setup, the complexity of our method is quite similar to regular fine-tuning procedure of BERT. The thresholding hyperparameter is decided by maximizing the Youden's J statistic as explained in Section 4.2. In Section 4.3, we also report the hyperparameters of the baseline methods. We employ a modified 10-fold cross-validation strategy as explained in Section 4.4 and apply t-test with Bonferroni correction to all experimental results.

## 6 Conclusion

We propose an OOS detection pipeline with a distance calculation between classifier prediction and uniform distribution, called D2U. In the zero-shot setup, D2U serves as an architecture-agnostic postprocessing step to emphasize the distinction between INS and OOS. In the supervised setup, we bring closer OOS predictions to uniform distribution with a modified loss function. Experimental results, supported by statistical tests, show that D2U outperforms existing baselines in zero-shot, and has challenging performance in the supervised setup. We plan to extend our study to different architectures and deep learning tasks in the future.

## References

* Acharya and Fung (2020) Shailesh Acharya and Glenn Fung. 2020. Using optimal embeddings to learn new intents with few examples: An application in the insurance domain. Cited by: SS5.
* E. Becht, L. McInnes, J. Healy, C. Duetterre, I. W. Kwok, L. Guan Ng, F. Ginhoux, and E. W. Newell (2019) Dimensionality reduction for visualizing single-cell data using umap. Nature Biotechnology37 (1), pp. 38-44. External Links: Document, ISSN 0037-64X, Link Cited by: SS1.
* ACL 2020, External Links: Link Cited by: SS1.
* P. Cavalin, V. H. Alves Ribeiro, A. Appel, and C. Pinhanez (2020) Improving out-of-scope detection in intent classification by using embeddings of the word graph space of the classes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 3952-3961. External Links: Link Cited by: SS1.
* A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, Thibault Gisselbrecht, F. Caltagirone, T. Lavril, et al. (2018) Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190. Cited by: SS1.
* M. Cuturi (2013) Sinkhorn distances: lightspeed computation of optimal transport. Advances in neural information processing systems26, pp. 2292-2300. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Vol. , pp. 4171-4186. External Links: Link Cited by: SS1.
* T. Dopiere, C. Gravier, and W. Logerais (2021) ProtAugment: latent detection meta-learning through unsupervised diverse paraphrasing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 2454-2466. External Links: Link Cited by: SS1.
* Y. Feng, S. Mehri, M. Eskenazi, and T. Zhao (2020) "none of the above": measure uncertainty in dialog response retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2013-2020. External Links: Link Cited by: SS1.
* V. Gangal, A. Arora, A. Einolghozati, and S. Gupta (2020) Likelihood ratios and generative classifiers for unsupervised out-of-domain detection in task oriented dialog. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 7764-7771. External Links: Link Cited by: SS1.
* S. Gupta, R. Shah, M. Mohit, A. Kumar, and M. Lewis (2018) Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing 

pages 2787-2792, Brussels, Belgium. Association for Computational Linguistics.
* Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_.
* Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2744-2751.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1311-1316.
* Lee et al. (2018) Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In _International Conference on Learning Representations_.
* Liang et al. (2021) Chaojiie Liang, Peijie Huang, Wenbin Lai, and Ziheng Ruan. 2021. Gan-based out-of-domain detection using both in-domain and out-of-domain samples. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7663-7667. IEEE.
* Liang et al. (2018) Shiyu Liang, Yixuan Li, and R Srikant. 2018. Enhancing the reliability of out-of-distribution image detection in neural networks. In _6th International Conference on Learning Representations, ICLR 2018_.
* Lin and Xu (2019a) Ting-En Lin and Hua Xu. 2019a. Deep unknown intent detection with margin loss. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5491-5496, Florence, Italy. Association for Computational Linguistics.
* Lin and Xu (2019b) Ting-En Lin and Hua Xu. 2019b. A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier. _Knowledge-Based Systems_, 186:104979.
* Liu et al. (2019) Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In _10th International Workshop on Spoken Dialogue Systems Technology 2019_.
* Marek et al. (2021) Petr Marek, Vishal Ishwar Naik, Anuj Goyal, and Vincent Auvray. 2021. Oodgan: Generative adversarial network for out-of-domain data generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers_, pages 238-245.
* Mehri et al. (2020) Shikhib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020. Dialogblue: A natural language understanding benchmark for task-oriented dialogue. _arXiv preprint arXiv:2009.13570_.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnav. 2011. Scikit-learn: Machine learning in Python. _Journal of Machine Learning Research_, 12:2825-2830.
* Qin et al. (2021) Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanxiang Che, and Ting Liu. 2021. GL-GIN: Fast and accurate non-autoregressive model for joint multiple intent detection and slot filling. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 178-188, Online. Association for Computational Linguistics.
* Shen et al. (2021) Yilin Shen, Yen-Chang Hsu, Avik Ray, and Hongxia Jin. 2021. Enhancing the generalization for intent classification and out-of-domain detection in SLU. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2443-2453, Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Futowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Xu et al. (2020) Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020. A deep generative distance-based classifier for out-of-domain detection with mahalanobis space. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 1452-1460.
* Xu et al. (2021) Keyang Xu, Tongzheneg Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. 2021. Unsupervised out-of-domain detection via pre-trained transformers. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, 

pages 1052-1061, Online. Association for Computational Linguistics.
* Yilmaz and Toraman (2020) Eyup Halit Yilmaz and Cagri Toraman. 2020. _KLOOS: KL Divergence-Based Out-of-Scope Intent Detection in Human-to-Machine Conversations_, page 2105-2108. Association for Computing Machinery, New York, NY, USA.
* Jouden (1950) WJ Jouden. 1950. Index for rating diagnostic tests. _Cancer_, 3(1):32-35.
* Young et al. (2013) Steve Young, Milica Gasic, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. _Proceedings of the IEEE_, 101(5):1160-1179.
* Zeng et al. (2021a) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 870-878, Online. Association for Computational Linguistics.
* Zeng et al. (2021b) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021b. Adversarial self-supervised learning for out-of-domain detection. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5631-5639.
* Zhan et al. (2021) Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao-Ming Wu, and Albert Y.S. Lam. 2021. Out-of-scope intent detection with self-supervision and discriminative training. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3521-3532, Online. Association for Computational Linguistics.
* Zhang et al. (2020) Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, S Yu Philip, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by transferring natural language inference. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5064-5082.
* Zheng et al. (2020) Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020. Out-of-domain detection for natural language understanding in dialog systems. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 28:1198-1209.

## Appendix A Appendix

We report Receiver Operating Curve Area Under Curve, False Positive Rate at 90% OOS True Positive Rate, False Negative Rate at 90% OOS True Negative Rate, weighted OOS Recall, and weighted OOS F1 scores in Tables 6, 7, 8, 9, 10 respectively. Different training procedures, baseline and proposed, are reported in rows and different post-processing methods, baseline and proposed, are reported in columns.

Baseline training methods are BERT-based isotope classifier (MLE) Larson et al. (2019); Devlin et al. (2019), Large Margin Cosine Loss (LMCL) Zeng et al. (2021a), Domain Regularization Module (DRM) Shen et al. (2021), entropy regularization (Reg.) Zheng et al. (2020), and BERT-binary classifier (Binary)Devlin et al. (2019). Post-processing methods are not applicable for Binary training since it models OOS detection as a binary classification problem. Baseline post-processing methods are Maximum Likelihood Estimate (MLE) Gangal et al. (2020); Zhang et al. (2020), softmax temperature (Temp) Liang et al. (2018); Lin and Xu (2019), standard deviation (Stdev), and entropy (Ent) Shen et al. (2021).