SlowGuess/NAACL_2022_OCR huggingface dataset:
Yanan Wu $^{1*}$ , Keqing He $^{2*}$ , Yuanmeng Yan $^{1}$ , Qixiang Gao $^{1}$ , Zhiyuan Zeng $^{1}$ , Fujia Zheng $^{1}$ , Lulu Zhao $^{1}$ , Huixing Jiang $^{2}$ , Wei Wu $^{2}$ , Weiran Xu $^{1*}$
<sup>1</sup>Beijing University of Posts and Telecommunications, Beijing, China  
<sup>2</sup>Meituan Group, Beijing, China
{yanan.wu, yanyuanmeng, gqx, zengzhiyuan, fujia_zheng, zhaoll, xuweiran}@bupt.edu.cn
{hekeqing,jianghuixing,wuwei30}@meituan.com
Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a task-oriented dialog system. A key challenge of OOD detection is the overconfidence of neural models. In this paper, we comprehensively analyze overconfidence and classify it into two perspectives: over-confident OOD and in-domain (IND). Then according to intrinsic reasons, we respectively propose a novel reassigned contrastive learning (RCL) to discriminate IND intents for over-confident OOD and an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents for over-confident IND. Experiments and analyses show the effectiveness of our proposed method for both aspects of overconfidence issues.
Out-of-domain (OOD) detection is a key component of the task-oriented dialogue system(Gnewuch et al., 2017; Akasaki and Kaji, 2017; Shum et al., 2018; Tulshan and Dhage, 2019). It aims to decide whether a user query falls outside the range of predefined supported intents and avoid performing wrong operations (Lin and Xu, 2019; Xu et al., 2020; Zeng et al., 2021a). Due to the complexity of annotating OOD intents, most work focuses on unsupervised OOD detection where there is no labeled OOD data but only labeled in-domain (IND) data (Xu et al., 2020). No prior knowledge about OOD intents makes it challenging to identify these unknown samples in the dialog system.
Existing unsupervised OOD detection methods mostly follow the same framework: firstly learn intent representations via labeled in-domain (IND) data then employ detecting algorithms, such as Maximum Softmax Probability (MSP) (Hendrycks
Figure 1: Examples of two kinds of overconfidence.
Figure 2: Number of error cases comparing overconfident IND and OOD. Global threshold denotes the best overall performance.
and Gimpel, 2017), Local Outlier Factor (LOF) (Lin and Xu, 2019), Gaussian Discriminant Analysis (GDA) (Xu et al., 2020) to compute the similarity of features between OOD samples and IND samples. For example, Hendrycks and Gimpel (2017) simply uses a fixed threshold on the IND classifier's probability estimate and predicts a query as OOD only if its max logit is below the threshold. Lin and Xu (2019) employs an unsupervised density-based novelty detection algorithm, local outlier factor (LOF) to detect OOD intents. Further, Zeng et al. (2021a) proposes a supervised contrastive learning objective to learn discriminative intent features.
However, these methods ignore the key challenge of OOD detection, over-confidence. Guo et al. (2017); Liang et al. (2017, 2018) have theoretically proved that deep neural networks with the softmax classifier are prone to produce highly
*The first two authors contribute equally. Weiran Xu is the corresponding author.
1We release our code at https://github. com/pris-nlp/NAACL2022-Reassigned_ Contrastive_Learning_OOD.
Human Language Technologies, pages 4165 - 4179
over-confident posterior distributions even for such abnormal OOD samples. In this paper, we define the over-confidence issue from two aspects. (1) IND  $\rightarrow$  OOD: Given an in-domain test sample, the pre-trained IND classifier predicts a lower confidence score than a fixed threshold $^2$  and incorrectly regards it as an OOD intent, which we name as over-confident OOD. We argue it's because IND classes have high semantic similarity then scatter and lower the max confidence score. For example, in Figure 1, the IND test query "can you call me a different name" is wrongly detected as OOD intent because its ground-truth IND label change_user_name is similar to the other IND categories what_is_your_name and change.ai_name. Thus the query gets comparable probability scores among the three IND classes, resulting in a lower max probability score than the threshold. (2) OOD  $\rightarrow$  IND: Given an OOD test sample, the same classifier instead predicts a higher confidence score than the threshold and wrongly regards it as an IND, which we name as over-confident IND. The reason is the spurious correlation between OOD and IND intents, such as similar syntactic structure, entities, etc. For example, the OOD query "please clean the car windows" is classified into the IND category smart_home because plausibly similar examples like "please lock the doors" also exist in this category. The spurious correlation is frequent since humans define OOD without a clear and standard principle. Existing models (Lee et al., 2018; Ren et al., 2019; Zheng et al., 2020; Xu et al., 2020) mainly focus on the latter aspect over-confident IND, but we find the former over-confident OOD also makes a side effect on OOD detection. As Figure 2 shows, as the threshold rises, the number of negative OOD samples also increases which denotes the over-confident OOD issue gets worse but the over-confident IND issue gets better. We need to consider both two aspects of overconfidence.
In this paper, we propose a novel reassigned contrastive learning (RCL) to discriminate intent representations between semantically similar IND categories and an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents. Specifically, for over-confident OOD, we first construct hard contrastive pairs among easily misclassified IND types using a pre-trained intent classifier. Then we train a new model to learn
discriminative intent representations for similar IND categories via supervised contrastive learning (Khosla et al., 2020; Gunel et al., 2020). We aim to sample hard contrastive batches where anchors and positives have the same class label but different classifier outputs (hard positives), and anchors and negatives have the same classifier output but different class labels (hard negatives). For over-confident IND, we propose an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents. Traditional detection methods like MSP, GDA, Energy (Liu et al., 2020) use a global threshold to identify the confidence score of a test query, ignoring the difference between each IND class with OOD samples. We aim to adjust the class-dependent local threshold so that semantically correlated OOD and IND classes have a higher threshold to mitigate the model's overconfidence to IND.
Our contributions are three-fold: (1) We perform a comprehensive study on the overconfidence issue of OOD detection and analyze two-aspect reasons. (2) We propose a novel reassigned contrastive learning (RCL) to discriminate IND intents for overconfident OOD and an adaptive class-dependent local threshold mechanism to separate similar IND and OOD intents for over-confident IND. (3) Experiments and detailed analyses demonstrate the effectiveness of our proposed method for both aspects of overconfidence issues.
OD Detection Unsupervised models use only IND data for OOD detection following the threshold-based protocol, including modeling the probability density (Pidhorskyi et al., 2018), reconstruction (Golan and El-Yaniv, 2018), using classifier ensembles (Vyas et al., 2018; Shu et al., 2017), Bayesian models (Malinin and Gales, 2018), likelihood ratios (Ren et al., 2019). Note that all these methods require a dev set of labeled OOD intents to tune a fixed global threshold hyperparameter. We propose a more robust and efficient local threshold mechanism both to improve OOD performance and reduce the need for a large dev set of labeled OOD data. Another type of OOD detection model aims to utilize a set of OOD data in the training phase, including  $\mathrm{N + 1}$  classifier (Fei and Liu, 2016; Zhan et al., 2021), entropy regularization (Zheng et al., 2020), adversarial augmentation (Zeng et al., 2021c).
2The threshold is tuned via the dev set. We will discuss it later.
Figure 3: The overall architecture of our proposed approach. Reassigned contrastive learning (RCL) discriminates intent representations between semantically similar IND categories for over-confident OOD and adaptive class-dependent local threshold separates similar IND and OOD intents for over-confident IND.
Contrastive Learning Recent contrastive learning methods (Chen et al., 2020; He et al., 2020) have proven effective to learn unsupervised representations for downstream tasks. Winkens et al. (2020); Zeng et al. (2021b) combine cross-entropy loss on labeled IND data and instance-wise contrastive learning (CL) loss on unlabeled data (including unlabeled IND and OOD intents). They require a large amount of unlabeled corpus and can't explicitly distinguish different intent types. Further, Zeng et al. (2021a) uses supervised contrastive learning (SCL) (Khosla et al., 2020) to learn discriminative intent representations only using labeled IND data. Compared to CL, SCL regards all the IND intents from the same class as positive pairs and samples from different classes as negative pairs. However, we find intents within similar categories are still easily misclassified (see Section E). Thus, we propose a simple but strong reassigned contrastive learning (RCL) framework to give more penalty on these easily-confused IND classes to explicitly distinguish them. RCL aims to learn discriminative intent representations for OOD detection. Zhuang et al. (2019); Wang and Liu (2021) mines negatives close to the anchor sample as hard negatives by computing representation cosine similarity, but RCL uses the model's wrong predictions as supervised positives and negatives. Our method is more accurate because estimating representation similarity may be biased and we can construct both hard positives and negatives.
Figure 3 shows the overall architecture of our proposed RCL and class-dependent local threshold
where RCL discriminates easily-confused IND intents and local threshold separates similar IND and OOD intents. We follow a two-stage framework: first train an in-domain intent classifier in the training stage, then extract the intent feature of a test sample and employ the detection methods in the test stage.
3.1 Reassigned Contrastive Learning
Traditional models (Hendrycks and Gimpel, 2017) use cross-entropy (CE) loss to train an IND intent classifier which does not explicitly distinguish the margins between IND categories. Later, Lin and Xu (2019) and Zeng et al. (2021a) respectively propose a large margin cosine loss (LMCL) and a supervised contrastive learning (SCL) loss to minimize intra-class variance and maximize inter-class distance. However, we find IND intents within similar categories are still easily misclassified (see Section E). Therefore, we aim to give more penalty on these easily-confused IND classes to learn discriminative intent representations.
We first review the original contrastive learning (CL) and supervised contrastive learning (SCL) then introduce our RCL framework. Given an IND sample  $x_{i}$  and its intent label  $y_{i}$ , we adopt a BiLSTM (Hochreiter and Schmidhuber, 1997) or BERT (Devlin et al., 2019) encoder to get the intent representation  $z_{i}$ . Following Chen et al. (2020); Zeng et al. (2021b), we formulate CL loss for a positive pair of examples  $(i,j)$  as:
$$
\ell_ {i, j} = - \log \frac {\exp \left(\operatorname {s i m} \left(z _ {i} , z _ {j}\right) / \tau\right)}{\sum_ {k = 1} ^ {2 N} \mathbb {1} _ {[ k \neq i ]} \exp \left(\operatorname {s i m} \left(z _ {i} , z _ {k}\right) / \tau\right)} \tag {1}
$$
where  $\mathbb{1}_{[k\neq i]}\in \{0,1\}$  is an indicator function
evaluating to 1 if  $k \neq i$ .  $\tau$  denotes a temperature parameter. The final loss is computed across all positive pairs, both  $(i,j)$  and  $(j,i)$  in a minibatch of  $N$  examples. CL regards two augmented views of a sample as positive pairs and views of different samples as negative pairs. Further, SCL extends the positive set by adding views of different samples from the same IND intent class and vice versa. However, these methods ignore easily-confused relations between semantically similar IND classes and can't separate them in the latent space. Therefore, we add more penalty to these easily-confused IND classes to learn discriminative intent representations. Specifically, our proposed reassigned contrastive learning (RCL) framework includes three stages.
IND classifier training First, we train an initial intent classification model  $M_{init}$  on the labeled IND dataset  $\{(x_i, y_i)\}_{i=1}^n$  using CE, and save its predictions  $\{\hat{y}_i\}_{i=1}^n$  on the training IND datapoints. We will use these outputs to train a more discriminated model.
Confused-label pair contrasting Here we aim to separate easily misclassified IND types. Thus we use the  $M_{init}$  outputs to obtain confused label pairs of the training data. For example, if class A and B have misclassified error cases, we use all the A and B's samples to construct contrastive batches. Then we perform SCL on these batches to train a new intent model  $M$  from scratch<sup>4</sup>. The intuition is that we treat samples that have the same class label but different classifier outputs as hard positives, and samples that have the same classifier output but different class labels as hard negatives. We display an example as Figure 4. Essentially, we restrict the new model to focus on misclassified intent classes by adding hard positives and negatives, further to learn discriminative intent representations. Different from existing hard CL work (Zhuang et al., 2019; Wang and Liu, 2021) which only consider close negatives as hard negatives using representation similarity, RCL uses the model's wrong predictions as supervised positives and negatives. Our method is more accurate because estimating representation similarity may be biased and we can construct both hard positives and negatives. We also perform SCL on other clean IND types.
Figure 4: An example of confused label pair (A, B) where the same color denotes the same prediction class of the initial model.
In the experiments, we iteratively repeat the two processes for 5 epochs.
Global contrasting Apart from the confused label pairs, we also employ SCL on all the IND samples to avoid knowledge forgetting. Following Zeng et al. (2021a), we regard views of different samples from the same IND intent class as positives and views of different samples from the different IND intent class as negatives. Finally, we use CE to fine-tune the model  $M$ . In the experiments, we set the training epoch of SCL and CE to 10 and 20.
3.2 Adaptive Class-Dependent Local Threshold
Previous detection methods usually use a global threshold to identify the confidence score of a test query, ignoring the difference between individual IND classes with OOD samples. For example, if a test OOD query is similar to an IND type, it may obtain a high confidence score on this IND category and be wrongly regarded as IND. Therefore, we aim to set adaptive class-dependent local thresholds to avoid over-confident IND.
Previous methods using global threshold (Xuet al., 2020; Zeng et al., 2021a) first compute the max confidence scores of all the OOD and IND intents on the dev set, like max probability score, then adjust the threshold to maximize OOD F1 on the dev set. Notice it's a general and standard setting where a few labeled OOD data exists in the dev set for hyperparameter tuning for the OOD detection task. For the local threshold, we input all OOD and IND queries to the pre-trained classifier and then get the confidence scores belonging to each IND type. Here we can use existing detection
<sup>3</sup>In this paper, we use adversarial augmentation as Zeng et al. (2021a). Please see more details in the original paper.
<sup>4</sup>We find training a new model  $M$  from scratch is much better than continual finetuning  $M_{init}$ . We argue it's hard to remove intrinsic knowledge existing in a model.
<sup>5</sup>Section 5.4 shows our local threshold requires only 20 OOD intents on CLINC to achieve excellent metrics compared to 15,000 IND intents in the training set, which is more robust and efficient.
CLINC-Full	CLINC-Small	Snips
IND	OOD	IND	OOD	IND	OOD
Vocabulary size	6240	5725	11241
Avg utterance length	9	9	9
Intent	150	-	150	-	5	2
Training Set Size	15000	-	7600	-	9345	-
Samples per class	100	-	50	-	1869	-
Development Set Size	3000	100	3000	100	500	20
Samples per class	20	-	20	-	20	-
Testing Set Size	5500	1000	5500	1000	500	200
Samples per class	30	-	30	-	100	-
Table 1: Full statistics of OOD datasets, CLINC-Full, CLINC-Small and Snips.
methods to get confidence scores, such as MSP, GDA, Energy, etc. Taking MSP as an example, we get probability scores of all the intents. Then we group all the OOD samples into the corresponding IND type according to the max probability. So in each group, we have both OOD and IND intents and learn the class-dependent threshold by maximizing OOD F1 on each group. If no OOD sample is grouped into an IND type, we simply select the global threshold as the local threshold for this IND type. For inference, we select the local threshold of corresponding IND category where the test query gets the max probability score and predict it as OOD if the score is below the local threshold, Otherwise IND. Note that tuning local threshold doesn't increase too much computation cost but only multiple judgments and no extra cost in the inference. Local threshold achieves the best performance on CLINC and Snips even only using 20 OOD intents and is robust to different datasets and different number of OOD intents.
We perform experiments on three public benchmark OOD datasets, including CLINC-Full, CLINC-Small (Larson et al., 2019) and Snips (Coucke et al., 2018). We show the detailed statistic of these datasets in Table 1. Snips is a personal voice assistant dataset which contains 7 types of user intents across different domains. We randomly sample two classes among all classes in Snips, regarding them as OOD classes and the rest as IND classes. CLINC-Full and CLINC-Small both contain 150 IND intents across 10 domains. CLINC-Full has 100 training samples for each IND type, while CLINC-Small contains 50. We follow the standard dataset split Larson et al. (2019) and use the collected OOD test queries for evaluation. Note that all the datasets we used have a fixed set of labeled OOD data but we don't use it for training.
We notice some work (Zhang et al., 2021) use a different split in CLINC-Full dataset where they sample  $25\%$ ,  $50\%$ ,  $75\%$  of all IND classes as IND, the other IND classes as OOD. The simulated split makes OOD data similar to IND data which class clusters are more compact thus get higher metrics. In this paper, we mainly follow the standard dataset split unless otherwise stated. For fair comparison, we also perform the same dataset split  $25\%$ ,  $50\%$ ,  $75\%$  in Table 3 and Table 10.
We report both OOD metrics: Recall and F1-score (F1) and in-domain metrics: F1-score (F1). Since, we aims to improve the performance of detecting out-of-domain intents from user queries, OOD Recall and F1 are the main evaluation metrics in this paper.
In training stage, we compare RCL with CE and SCL. In detection stage, we compare local threshold with global threshold. To verify the generalization of our proposed models, we use three OOD detection algorithms MSP (Maximum Softmax Probability)(Hendrycks and Gimpel, 2017), GDA (Gaussian Discriminant Analysis)(Xu et al., 2020) and Energy(Ouyang et al., 2021). Besides, we compare our models with the following state-of-the-art baselines, OpenMax(Bendale and Boult, 2016a), DeepUnk(Lin and Xu, 2019), Energy(Ouyang et al., 2021), SCL(Zeng et al., 2021a) and ADB(Zhang et al., 2021). We provide a more comprehensive comparison and implementation details of these models in the Appendix.
To conduct a fair comparison, we follow a similar evaluation setting as (Zeng et al., 2021a) and (Zhang et al., 2021). We use the public pre-trained GloVe embeddings (Pennington et al., 2014) and BERT-uncased (Devlin et al., 2019) (with 12-layer transformer, implemented in PyTorch) to embed tokens. We set the learning rate to 1e-03 for LSTM and 2e-05 for BERT. To speed up the training procedure and achieve better performance, we freeze all but the last transformer layer parameters of BERT. We use Adam optimizer (Kingma and Ba, 2014) to train our model and set the dropout rate to 0.5. We use the best F1 scores on the development set to calculate the MSP, GDA and Energy thresholds adaptively. Each result of the experiments is tested
Detection	Training	CLINC-Full	Snips
Global Threshold	Local Threshold(ours)	Global Threshold	Local Threshold(ours)
OOD	IND	OOD	IND	OOD	IND	OOD	IND
F1	Recall	F1	F1	Recall	F1	F1	Recall	F1	F1	Recall	F1
MSP	CE	54.70	44.50	86.28	64.35	62.00	87.06	74.39	80.19	88.37	78.03	82.46	90.21
SCL	56.98	46.34	87.94	65.88	64.31	88.00	76.23	80.57	89.60	79.25	82.57	91.30
RCL(ours)	61.71	53.90	88.45	67.43	64.92	88.76	81.00	81.52	91.71	83.53	82.94	93.28
GDA	CE	65.79	64.14	87.90	68.06	73.50	87.95	77.33	79.23	90.08	81.96	84.52	90.11
SCL	68.04	66.92	88.60	70.85	73.40	88.63	80.27	82.46	91.19	83.45	87.20	92.58
RCL(ours)	72.61	70.00	88.98	73.88	75.10	89.03	85.24	86.95	93.89	87.91	88.57	94.65
Energy	CE	68.87	66.30	88.02	71.67	72.50	88.78	78.75	79.27	91.00	82.65	84.70	92.58
SCL	71.12	71.01	88.59	73.15	73.20	88.98	81.72	81.99	91.27	85.04	85.83	93.47
RCL(ours)	74.30	72.03	89.56	75.32	78.60	89.67	86.41	87.16	94.40	89.21	89.45	95.42
Table 2: The performance of OOD detection and IND classification with different OOD detection methods on CLINC-Full and Snips datasets for the BiLSTM-based model(p<0.05 under t-test).
Models	CLINC-50%	CLINC-Full
OOD F1	IND F1	OOD F1	IND F1
OpenMax(Bendale and Boult, 2016a)	81.89	80.54	-	-
DeepUnk(Lin and Xu, 2019)	85.85	82.11	-	-
Energy(Ouyang et al., 2021)	84.34	82.61	75.93	91.23
SCL(Zeng et al., 2021a)	86.42	84.55	68.21	89.57
ADB(Zhang et al., 2021)	88.65	85.00	76.52	90.94
Ours	92.16	86.05	82.03	92.00
Table 3: The performance of OOD detection and IND classification compared with previous state-of-the-art baselines for the BERT-based model.
for 10 times under the same setting and reports the average value. In the training stage, for our model, we conduct 5 epochs of confused-label pair contrasting on designative batches of IND data, and then 10 epochs of global contrasting on randomly sampled IND data. Finally, we used CE to finetune the previous model with the epoch to 20. For the baselines in Table 2, we set the training epoch to 20 for CE and 15 for SCL. For fair comparison, we adopt the same data augmentation method as (Zeng et al., 2021a). Specifically, we apply adversarial attack to generate pseudo positive samples to increase the diversity of views for contrastive learning. The training time for CE is about 1.6 minutes using Glove+LSTM, and 12 minutes using BERT. The training stage of our model lasts about 2 minutes using Glove+LSTM, and 15 minutes using BERT on single Tesla T4 GPU (16 GB of memory) in CLINC-Full dataset which has 15,000 training samples. And the test stage of our model lasts about 1 second using Glove+LSTM, and 3 seconds using BERT. We have similar training time with SCL. And the test time of RCL is the same as that of CE and SCL.
Table 2 displays our experimental results on datasets of CLINC-full and Snips with three different OOD detection algorithms: MSP, GDA and Energy. We show similar results on CLINC-Small
dataset in the Appendix Table 7. From Table 2, we can make the following observations.