SlowGuess/NAACL_2022_OCR huggingface dataset:
are still easily misclassified (see Section E). Intuitively, these easily-confused IND classes can be regarded as hard examples (Zhuang et al., 2019) for existing CL-based methods. Inspired by the idea, we propose a simple but strong reassigned contrastive learning (RCL) framework to give more penalty on these easily-confused IND classes to explicitly distinguish them. The main difference is the confused-label pair contrasting process (we provide an example to show how to construct hard positives and negatives in Figure 4) and please see details in the Confused-label pair contrasting section(line 241-267). Generally, RCL aims to learn discriminative intent representations for OOD detection, especially for these easily-confused intent classes. Compared to Zhuang et al. (2019); Wang and Liu (2021), these work only mines negatives close to the anchor sample as hard negatives by computing representation cosine similarity, but RCL uses the model's wrong predictions as supervised positives and negatives. Our method is more accurate because estimating representation similarity may be biased and we can construct both hard positives and negatives. Note that RCL only increases the training cost for a little but requires no extra inference budget and uses the same model size as SCL. Please see the following section for details.
B Comparison of time complexity and space complexity
We discuss the time and space complexity of RCL and Local threshold in Table 8 and Table 9. In terms
Table 8: The radio of training time and test time compared with SCL model.
Training	Test
Global Threshold	-	x1.00
Local Threshold(ours)	-	x1.06
Table 9: The radio of training time and test time compared with global threshold.
Models	CLINC-25%	CLINC-75%
OOD F1	IND F1	OOD F1	IND F1
OpenMax(Bendale and Boult, 2016a)	75.76	61.62	76.35	73.13
DeepUnk(Lin and Xu, 2019)	87.33	70.73	81.15	86.27
Energy(Ouyang et al., 2021)	91.09	72.68	71.43	78.07
SCL(Zeng et al., 2021a)	93.40	77.16	73.98	86.89
ADB(Zhang et al., 2021)	91.84	76.80	83.92	88.58
Ours	96.40	83.56	87.95	89.67
Table 10: The performance of OOD detection and IND classification compared with previous state-of-the-art baselines for the BERT-based model.
of time complexity, we set the epoch of SCL and RCL to 15. The training time for RCL is about 15 minutes using BERT in CLINC-Full dataset which has 15,000 training samples. RCL have similar training time with SCL. And the test stage of global threshold lasts about 3 seconds using BERT. The local threshold has almost the same test time as the global threshold, while the performance can even be increased by  $9\%$ . From the perspective of space complexity, RCL and SCL utilize the same encoder structure. The size of the model parameters of RCL is equal to that of SCL. Besides, the global threshold and the local threshold are only the differences of the algorithm, and there are no extra parameters.
We show the training procedure of RCL in Algorithm 1.  $E_0, E_1$  and  $E_2$  are the training epochs of confused-label pair contrasting, global contrasting and cross-entropy classification processes, respectively. In practice, we set  $E_0, E_1$  and  $E_2$  to 5, 10 and 20, respectively.  $n$  is the number of training samples. First, we construct a set of confused la
bel pairs by combining the ground truth labels  $y$  and predicted labels  $\hat{y}$  of the training data. Taking Figure 4 as an example,  $(A, B)$  is one of confusing label pairs in  $P$ .  $m$  is the number of confusing label pairs. And the remaining labels that never confused with other labels, called clean labels, are collected in the set  $S$ .  $k$  is the number of clean labels. Then, we sample confused mini-batch  $\hat{B}$  following  $P$  restrainedly. Note that a mini-batch  $\hat{B}_j$  will only contain the samples with ground truth  $y_j$  or  $\hat{y}_j$ . We also sample clean mini-batch  $B$  following  $S$  restrainedly. Different from the confused mini-batch, a clean mini-batch  $B_j$  can contain any sample whose label belongs to  $S$ . We will compute the supervised contrastive loss iteratively, on confused mini-batch or clean mini-batch. Apart from confused-label pair contrasting, we also employ global contrasting by randomly sampling batches on all IND samples to avoid knowledge forgetting. Finally, we compute cross-entropy loss to fine-tune the model.
We compare many types of unsupervised OOD detection models. For feature extractor, we use LSTM(Long Short Term Memory)(Hochreiter and Schmidhuber, 1997) or BERT(Bidirectional Encoder Representations from Transformers)(Devlin et al., 2019). For training objection, we compare RCL with CE and SCL. For detection algo
rithms, to verify the generalization of our proposed models, we use MSP(Maximum Softmax Probability)(Hendrycks and Gimpel, 2017), GDA(Gaussian Discriminant Analysis)(Xu et al., 2020) and Energy(Ouyang et al., 2021). Besides, we compare our models with the following state-of-the-art baselines, OpenMax(Bendale and Boult, 2016a), DeepUnk(Lin and Xu, 2019), Energy(Ouyang et al., 2021), SCL(Zeng et al., 2021a), ADB(Zhang et al., 2021). We supplement the relevant baseline details as follows:
MSP (Maximum Softmax Probability)(Hendrycks and Gimpel, 2017) applies a threshold on the maximum softmax probability. We use the best F1 scores on the validation set to calculate the threshold adaptively.
GDA (Gaussian Discriminant Analysis)(Xu et al., 2020) is a generative distance-based classifier for out-of-domain detection with Euclidean space. It estimates the class-conditional distribution on feature spaces of DNNs via Gaussian discriminant analysis to avoid over-confidence problems and use Mahalanobis distance to measure the confidence score of whether a test sample belongs to OOD.
Energy(Ouyang et al., 2021) maps a sample  $\mathbf{x}$  to a single scalar called the energy. We use the threshold on the energy score to consider whether a test query belongs to OOD.
OpenMax(Bendale and Boult, 2016b) is an open set detection method in computer vision, we adapt
Figure 9: Confusion matrix of CE, SCL and RCL based on MSP.
(a)change_user_name (b)user_name (c)change_ai_name (d)what_is_your_name (e)redeem_rewards (f)rewards_balance (g)ingredients_list (h)recipe (i)play_music (g)change_speed (k)unseen
it for OOD. We firstly use the CE loss to train a classifier on in-domain intents, then fit a Weibull distribution to the classifier's output logits.
DeepUnk(Lin and Xu, 2019) learns the deep intent features with the margin loss and detects the unknown intent with local outlier factor.
SCL(Zeng et al., 2021a) uses a supervised contrastive learning objective to learn discriminative intent features. We conduct many experiments and from multiple perspectives prove that our method can consistently outperform SCL.
ADB(Zhang et al., 2021) learns adaptive decision boundary using a loss function to balance both the empirical risk and the open space risk. It is still a time-consuming process. Compare with ADB, our method can achieve the best performance.
E IND Confusion Matrix
To demonstrate how our proposed RCL approach improves the performance by decreasing the number of error cases, we show the confusion matrix among 10 IND classes as well as the unseen OOD class. Specially, we pick 10 easily-confused classes by the baseline model as the IND classes, and show the confusion matrices of CE, SCL and RCL in Figure 9. From the figure, we can make the following observations.
On the one hand, we find the confusion of easilyconfused IND classes is significantly mitigated. Take the beginning four IND classes change user name (a),user name (b),change ai name (c) and what is your name (d) as an example, when training with CE, there are totally 34 misclassified samples. However, after applying our RCL approach, the number misclassified samples decreases to 11, which is a  $68\%$  reduction. It indicates that the confusion among those semantically close IND classes
Figure 10: Local Threshold vs Global Threshold.
is effectively alleviated by our RCL training strategy.
On the other hand, we focus on the confusion between the unseen OOD and the IND classes. The last row of the confusion matrix indicates the overconfident IND (truly OOD sample, but predicted as IND), while the last column of the confusion matrix indicates the over-confident OOD (truly IND sample, but predicted as OOD). We can find that our RCL model reduces the number of over-confident IND samples by  $69.6\%$  and  $70.0\%$  compared to the CE and SCL models, respectively. Meanwhile, our RCL model also reduces the number of over-confident OOD samples by  $60.0\%$  and  $47.4\%$  compared to the CE and SCL models, respectively. This phenomena proves that our RCL model improves the OOD detection performance through solving the over-confident IND and OOD issues.
F Effect of Local Threshold
Figure 10 shows the result of the additional 10 IND classes. Limited by the degree of semantic similarity between OOD and IND, there may be no OOD samples on some classes. In this case, we will set the local threshold to be the same as the global threshold, as IND class (3).
For IND classes (1), (6), (7) and (8), both the
global threshold and local threshold are between the mean softmax probability of the IND samples and OOD samples. Analyze with specific examples, we find that the probability of OOD samples in (6) is more concentrated on small probability values, while (7) and (8) have greater variance. Compared with the global threshold, the local threshold can better alleviate the over-confident problem by choosing a more appropriate threshold boundary.
For IND classes (2), (4) and (5), the mean maximum softmax probabilities of OOD samples are all beyond the global threshold. For category (2), the local threshold falls exactly between IND and OOD while global does not. For (4) and (5), although both the local threshold and the global threshold are below OOD, it is clear that the local threshold is more reasonable.
For IND classes (9) and (10), both the averaged max softmax probability of IND and OOD are lower than the global threshold, which means that a large portion of IND samples will be misclassified as OOD. On the contrary, our local threshold method can adaptively select more appropriate thresholds, which largely eliminates the problem of overconfident OOD.