STEP: Out-of-Distribution Detection in the Presence of Limited In-Distribution Labeled Data
Authors: Zhi Zhou
Lan-Zhe Guo
Zhanzhan Cheng
Yu-Feng Li
Shiliang Pu
Venue: NeurIPS 2021 Poster (2021)
Keywords: weakly supervised learning
semi-supervised learning
out-of-distribution detection

Abstract: Existing semi-supervised learning (SSL) studies typically assume that unlabeled and test data are drawn from the same distribution as labeled data. However, in many real-world applications, it is desirable to have SSL algorithms that not only classify the samples drawn from the same distribution of labeled data but also detect out-of-distribution (OOD) samples drawn from an unknown distribution. In this paper, we study a setting called semi-supervised OOD detection. Two main challenges compared with previous OOD detection settings are i) the lack of labeled data and in-distribution data; ii) OOD samples could be unseen during training. Efforts on this direction remain limited. In this paper, we present an approach STEP significantly improving OOD detection performance by introducing a new technique: Structure-Keep Unzipping. It learns a new representation space in which OOD samples could be separated well. An efficient optimization algorithm is derived to solve the objective. Comprehensive experiments across various OOD detection benchmarks clearly show that our STEP approach outperforms other methods by a large margin and achieves remarkable detection performance on several benchmarks. 

STEP Out-of-Distribution Detection in the Presence of Limited In-distribution Labeled Data
Zhi Zhou', Lan-Zhe Guol,, Zhanzhan Cheng", Yu-Feng Li Shiliang Pu²
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China {zhouz, guolz, 1iyf)=lamda.nni .jj. www..du..uu.. edu. cn Hikvision Research Institute, Hangzhou, China (chengzhanzhan, pushiliang hrri)hhikvssion.com
Abstract
Existing semi-supervised learning (SSL) studies typically assume that unlabeled and test data are drawn from the same distribution as labeled data. However, many weal-world applications, it is desirable to have SSL algorithms that not only classify the samples drawn from the same distribution of labeled data but also detect -uttr--distribution (OOD) samples drawn from an unknown distribution. In this paper, we study a setting called semi-supervised OOD detection. Two main challenges compared with previous OOD detection settings are the lack of labeled data and in in-distribution data; ii) OOD samples could be unseen during training. Efforts on this direction remain limited. In this paper, we present an approach STEP significantly improving OOD detection performance by introducing new technique: Structure-Kest Unzipping. It learns a new representation space which OOD samples could be separated well. An efficient optimization algorithm is derived solve the objective. Comprehensive experiments across various OOD detection benchmarks clearly show that our STEP approach outperforms other methods by a large margin and achieves remarkable detection performance on several benchmarks.
Introduction
Deep learning has achieved success in many application scenarios, such as computer vision, speech recognition, natural language processing [10]. The excellent performance typically rely on sufficient supervised information. However, collecting large amounts of well-labeled training data is not always available in real-world applications due to the expensive cost of the labeling process. Therefore, tremendous efforts have been devoted semi-supervised learning (SSL) [36] 30 which aims to enhance the model performance by exploiting much cheaper unlabeled data, and have shown
promising performance [52 37 29]
Previous SSL studies 33 41 typically work on the assumption that unlabeled data and test data drawn from the same distribution as labeled data. However, is often the case that such an assumption fails in practical applications [13 14]. For example, in document classification [9]. irrelevant documents readily occur in the testing data leading high-confidence misclassification. Similar cases commonly appear in other applications, such as medical diagnosis 44 and autonomous driving [ In such applications, it is desirable to have SSL algorithms which could not only classify samples from known distributions accurately but also be equipped with the ability to detect out-of-distribution (OOD) samples from unknown distributions precisely.
"Contribute to this work equally I CConesponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

OOD detection has been studied for a long history with numerous methods proposed, such as ODIN BT, Mahalanobis [27]. DeConf [20). ELOC 44 These methods perform OOD detection based on the logits of the model or the Mahalanobis distance in the feature space. However, it is hard to adapt these methods to semi-supervised settings because they all rely on massive labeled data, There are some methods associating with unlabeled data, such as UOOD 49). CSI [40]. SSD (38) have been proposed recently. These methods assume that the model can obtain sufficient in-distribution (ID) labeled data ID unlabeled data during the training process. Such an assumption also limits their ability to practical problems.
Therefore, we study a novel setting called sami-supervised OOD detection. Specifically, only tiny subset of ID labeled data is observed. The other ones remain unlabeled and may belong to ID or OOD. Here, we assume that abundant ID data is contained in the unlabeled data for extracting ID information. This setting ubiquitous in real-world applications. For example, in web page classification [47] acquiring large numbers of web pages annotated with relevant categories is very expensive, and unlabeled web pages crawled from the Internet according to keywords usually contain irrelevant pages that belong to unseen categories. In medical diagnosis 4 warning users of the model's s uncertainty is crucial because any unfaithful diagnosis will bring unimaginable disasters to the patients' health. In ride-sharing liability judgment [1], detecting abnormal orders is of significant value, while collecting training data will meet similar problems stated above. Similar cases often occur in other real-world applications, such as crowdsourcing [45] 28). There are two main challenges for us compared with previous OOD detection settings. First, both the labeled data and directly available ID data are limited, while sufficient unlabeled data is mixed with ID and OOD samples. Second, OOD samples could be unseen during the training, requiring more stringent generalization of the model.
Focusing on semi-supervised OOD detection, we find that the widely-used Mahalanobis distance no longer suitable the confidence score for OOD detection. This is because the necessary covariance matrix Σ for calculating Mahalanobis distance is hard to estimate accurately with limited ID samples which will severely affect the performance of OOD detection. To alleviate this issue, we propose a novel STEP (STructure-keEP approach. The idea is to detect OOD samples in a detection-speciffe space where we maintain the same local topological structures as the original feature space, because the relationships between samples need to be confirmed through local topological structures. We introduce a new objective and optimize it efficiently. The experiments prove that our STEP approach outperforms previous methods by large margin diverse data sets.
The contributions of our paper are summarized as follows:
We propose practical setting for OOD detection, called semi-supervised OOD detection.
To alleviate the problem of Mahalanobis distance that the necessary covariance matrix S is hard to be estimated with limited ID samples, we present new distance calculated in a detection-speeifie space as OOD confidence scores.
We evaluate our approach with comprehensive experiments across various OOD detection benchmarks. Our proposal outperforms previous methods by a large margin and achieves remarkable detection performance on several benchmarks.
2 Method
2.1 Notations and Setting
In the semi-supervised OOD detection setting, we assume that limited label data D) {(x;, (x)..))))) y consisting n samples with labels drawn from ID, and an unlabeled data set D., (xx) consisting m unlabeled samples drawn both ID and OOD, are accessible during the train- ing phase. We denote the set of ground-truth classes in the labeled data set D) and unlabeled data set Dr, as C₁ and C,,, respectively. The labeled samples can be classified into one of K classes denoted by C. [c1, ... (x). ) and the unlabeled samples can be classified into the seen K classes C₁ and some unseen classes denoted by C, = C,, (C, The goal to distinguish whether a sample in D., or an unknown testing sample is drawn from ID or not.

---

2.2 Inaccurate Mahalanobis Distance and Our Approach
Mahalanobis distance which is widely used in previous studies 27 38 has been proven
powerful metric in OOD detection. MD(x,, xj) denotes the function measuring the Mahalanobis distance between sample x; and sample X, based estimated covariance matrix Σ:
MD(x;; V(x (x) -x)
Previous methods mentioned above calculate the minimum Mahalanobis distance between target sample and each class center the confidence score:
SCOREMD(X)) oECr 0EC,,.2
min
MD(x,, 1b)
where [4, denotes the center of samples which belong to class e and Σ the covariance matrix estimated all ID samples.
However, Σ is hard to be accurately estimated in semi-supervised OOD detection setting because the available ID labeled data set Dj is insufficient. Inaccurate estimation of Σ will affect the calculation of Mahalanobis distance. This makes difficult for the algorithm to distinguish OOD samples and ID samples near the cluster boundary.
project samples into space where large margin separates ID samples and OOD samples. Inspired by the topological technology 44 used in noisy label problems and cluster assumption (36 used SSL, we hope that the projected samples can maintain the same local topological structure as the original space while increasing the distance between samples not directly topologically connected. Because the inaccurate estimation of Σ, we consider that relationship between samples that are not topologically adjacent is uncertain. Their relationships need be confirmed through each local topological structure. We formulate goal into the objective:
Instead of using inaccurate Mahalanobis distance, we decide to learn
max
Σ
Px Px2l12
X, ED;UD,
s.t.
PPx Px,l|2 MD(x; xn),
if
Xn E (x.)
where, MMx;, x is the Mahalanobis distance between and xy in the feature space and By (x,) is the set of k nearest neighbours of x4.
Finally, our detection-specific metric can directly calculate as L2 distance in the projected space:
N(x,, = |Px; Px,l|2
(4)
2.3 Backbone Pretraining
Our semi-supervised OOD detection task considers OOD detection as a clustering problem based on the feature space. Therefore, reliable feature representations essential. Benefiting from recent progress on self-supervised learning, we adopt simple contrastive learning method SimCLR (5) pre-train backbone network on the whole dataset D. U Di in unsupervised fashion. We find that representations obtained by SimCLR have a reasonable ability to distinguish ID and OOD samples. Notably, the learned representations could be not only used for STEP approach but also used as the initialization of downstream tasks.
2.4 Structure-Keep Unzipping
Based on the representations obtained by SimCLR, we further train to project samples into detection-specific space via our objective formulated in (3) However, there are two main difficulties: i) Building a KNN graph for extracting topological structure needs dn))d)) time complexity to calculate Mahalanobis distance between each pair of samples. This step is very time- consuming because we use an ensemble of representations from each backbone network's layer, and feature dimension d is relatively large. ii) The constraint in Eq. can not be directly optimized.

---

ID Labeled Data
ODD Unlabcled Data
OOOD Unlubeled data
Projector
Initial Step
Step A
Step
Figure The overall of STEP approach: (a) In initial step, we use contrastive learning to train initial representations. (b) In step A, we estimated statistics information via limited labeled data and extracted topological structure via the KNN algorithm. (c) In step B, we train P project all the samples into a detection-specife space where we can use 1.2 distances as OOD scores.
First, we transform the process of calculating pairwise Mahalanobis distance into calculating pairwise Euclidean distance in projection space. The time complexity of this step reduces from (n?dd) to O(n²d). Specifically, as shown in 5). we can perform cholesky decomposition on --1 to get linear projector the PMDD Then, we multiply all samples by PMD to project them into new space where Euclidean distance equals to original Mahalanobis distance between each pair of samples. There are n² pairwise Euclidean distances to calculate, and each calculation costs ((d)
time complexity. Therefore, the total time complexity of this step is (ndd).
MD(x;,,,) = (x, x,) (x,-xxx)) PMDX P PMDX211
(5)
s. PMD
After converting Mahalanobis distance to Euclidean distance, we can further use the advanced KNN toolkit, such Faiss [22] to speed up the entire process.
Second, we define Leee and LUnsip that can be directly optimized to approximately achieve our objective shown in Eq. 3. Both LKeep and LUnsip are shown in Eq.
[.Keep max(0, |Px; Px,,12 MD(xx, x.)),
(6)
LUnzip
Px, Px,112-
where X;, Xj randomly sampled from Dj U Du. and vn is randomly sampled from B,(x,), The final loss to optimize Loss Keep + LU'sip- The overall our STEP approach is summarized in Fig.(T), and the pseudo-code of our approach is shown in Algo.
In the detection stage, we directly use the minimum I.2 distance between the target sample and each class center in the detection-specifie space as the confidence score:
Score(x)
min
N(x, 14,))
cEfcr cx....c
where the n. is the center of class e the original feature space.
3 Experiments
3.1 Experimental Setup
In-distribution Data Set. We use CIFAR-10 and CIFAR-100 100 25 as data sets in our experiments. They both contain 50,000 training images and 10,000 testing images. The image size of these two data sets is 32 32. For CIFAR-10, each image belongs to one of 10 classes, and we randomly sample 250 training images as ID labeled data D,. For CIFAR-100, the size of image classes is 100, and we randomly sample 400 training images as ID labeled data. Dr. We add the remaining training images to the unlabeled data D...

---

Algorithm Training Phase of STEP
Input: Dj: ID labeled data set; Du: unlabeled mixed data set; K: number of neighbours
Output: pre-trained backbone fo(-): projctor P
I: train backbone fo(-) via contrastive learning D U Du
2: estimate Σ on Dj with fe(-)
3: calculate P.D based on Σ-1
4: build KNN on D JUD, D. with PMD and fo(.)
5: for epoch E (p.cch....pochmma
do
6: randomly sample X, from Dj U Du
7: randomly sample Xa from (x,)
8: calculate Loss based on Eq.(6)
9: optimize P via SGD according to Loss
10: end for
11: return fa(+) and
Out-of-distribution Data Set. We use Tiny ImageNet data set [6] and Large-seale Scene Understand- ing data set [48] as OOD data sets. The Tiny ImageNet data set (TIN) is subset of ImageNet, which contains 10,000 test images, includes 200 different classes. Following the settings used by previous studies [31 43 49 we two variants of TIN: T (TINc) and TinylmageNet-resizee (TINr), by randomly cropping or downsampling each image to 32 x 32, respectively. The Large-scale Scene Understanding data set (LSUN) contains 10,000 testing images belonging scene categories. Similarly, we use two variants of LSUN: LSUN-crop (LSUNc) and LSUN-resize (LSUNr). Because some comparison methods in our experiments heavily rely on OOD validation. We randomly draw several images from ID testing images and OOD images as the OOD validation set. The rest of the OOD images are added unlabeled data Du and used as testing data. These OOD data sets are released by ODIN [1 with their code'.
10 different
Comparsion Methods. We compare our STEP approach with representative OOD detection methods, including the state-of-the-art UOOD method. ODIN (31 is a common baseline of OOD detection. It uses maximal softmax score combining temperature scaling and input preprocessing tricks to distinguish ID and OOD samples. MAH (27) uses Mahalanobis distance as the OOD confidence score. For features of each layer in the backbone model, independently calculates the Mahalanobis distances between the target sample and each known class center. Then it integrates them by weighted averaging via an extra OOD validation set. We denote it as MAII because uses validation set when training. UOOD 49 utilizes a two-head CNN consisting of one common feature extractor and two classifiers which has different decision boundaries detect OOD samples. This method optimizes a discrepancy loss between two classifiers during the training stage and uses this discrepancy as the OOD score when testing. However, this method relies on extra OOD validation to perform model selection. Therefore, we denoted it as UOoD in our experiments For fair comparisons, we also implement a variant of it denoting as UOOD. UOOD that uses discrepancy loss to perform model selection instead of the performance on an extra OOD validation set.
we evaluate our approach with five common metrics: AUROC, FPR at 95% TPR, Detection Error, AUPR-In, and AUPR-Out. More details about evaluation metrics are presented in the supplementary material.
Evaluation Metrics. Following the settings used by previous studies 49 43
Implementation Details. In all experiments, we adopt the Densenet-Bi [21] as the backbone since it is widely used in previous studies [49 4 BI] Our backbone is trained by SOTA contrastive learning method SimCLR (5] for 500 epochs. We set the learning rate to 10-3 with a cosine annealing strategy. For fair comparisons, each comparison method can the pre-trained backbone model. MAH (27) uses the features from different layers extracted from the pre-trained backbone model. A well-trained linear classifier with a pre-traine pre-trained backbone model is provided for ODIN [3T] and UOOD 49 The hyper-parameter K for STEP is set to 12 for all data set pairs. All experiments are performed on one single NVIDIA 3090 graphics card. More details on implementation are provided in the supplementary material and our code has been open source 2
5

---

3.2 Experiment Results
OOD Detection Performance. We evaluate STEP with compared methods various OOD bench- marks. Analyzed by five common metries, the results are shown Tab.TT). From the results, we observe that ODIN suffers from severe performance degradation. Moreover, its performance is close random guessing some cases. The limitation of labeled data mainly causes this. We can hardly train a high-quality classification model to provide accurate logits for ODIN. Hence, ODIN can not give the correct judgment based on inaccurate logits. Our STEP approach outperforms methods that do not heavily rely on OOD validation set by large margin, Even compared with those methods that heavily rely on the OOD validation set, such as UOOD f and MAH our STEP approach is still better than them in most cases. However, good OOD validation set is expensive and nearly impossible to build in the real world. The number of OOD samples can be infinitely many, and fixed-size validation set cannot capture the complete OOD information. Therefore, introducing the validation set during training will reduce the model's generalization the real environment. We will verify this in detail in subsequent experiments.
Table 1: Performance comparison on various OOD benchmarks evaluated by common metrics. Methods with use extra OOD validation set. The best results are indicated in bold. Our approach outperforms other methods in most cases, even though they use an extra OOD validation set.
ID
OOD
Metrics
ODIN
MAH
UOOD
UooD
STEP
Dataset
Dutaset
Cifar10
TINe
81.00
6.30
87.67 + 2.47
90.46 + 9.74
99.07 + 0.48
99.99
0.00
TINr
59.10
86.88 0.87
84.67 9.41
92.63
3.42
95.61
0.36
AUROC
LSUNc
76.17 5.37
97.68 + 0.09
96.92 + 2.04
98.79 = 0.67
99.99
0.00
LSUNr
69.05 3.49
90.41 1: 1.00
80.87 24.45
97.81 1 0.94
99.07
0.20
10
TINc
61.65 6.71
71.15 = 2.20
98.34 + 1.57
98.84 = 0.83
99.99
0.01
TINr
54.46 + 0.74
73.94 + 1.79
84.80 + 8.87
95.31 + 0.93
93.51 + 1.17
LSUNc
46.99 = 4.99
93,91 + 3,41
97.49 + 1.48
99.31 = 0.62
99.99 = 0.00
LSUNr
52.06 1: 2.24
78.45 1: 1.11
97.61 1: 0.55
98.96 0.40
98. 28.20 0.56
Cifarl0
TINe
53.37 10.55
44.17 + 6.43
29.35 30,05
2.75 1.65
0.00 0.00
FPR at 95%TTP
TINr
89.76
1.45
58.57 3.09
31.72 11.50
19.61
9.50
17.63 1.10
LSUNc
64.06
9.12
7.73 0.46
6.59 1 3.22
3.56 + 1.93
0.00 1: 0.00
LSUNr
76.89 5.04
45.41 + 3.87
32.69 1 31.93
6.49 2.89
4,48 + 1.02
Cifur100
TINc
84.24 8.02
90.15 + 1.99
5.22 5.59
2.225
0,00 + 0.01
TINr
40.10
80.55 1.89
29.09 = 15.68
11.10 = 4.21
23.21 = 4.14
LSUNc
2.42 93.49
24.93 1: 21.75
6.24 3.80
1.93 2.43
0.00 1: 0.00
LSUNr
89.79 0.79
69.69 + 2.42
4.92 = 1.33
2.39 + 0.74
8.25 + 3.14
Cifar10
TINc
25,53 4.67
19.93 +: 2.63
11.59 11.35
2.54 1.27
0.12 1: 0.01
Detection Error
TINr
43.04 + 1.48
20.14 + 0.82
18.07 +: 5.55
11.71 + 4.56
10.77 0.52
LSUNc
29.57 3.822
6.28 + 0.25
4.20 2.12
2.58 + 1.32
0.11 + 0.01
LSUNr
16.23 + 0.95
18.40 = 15.68
4.99 1.91
4.66 + 0.57
Cifar100
TINc
40.95 +
32.58 + 1.64
3.67 3,62
2.76 1.00
0.32 1: 0.06
TINr
46.36 +.:66+0000 0.56
31.09 + 1.44
16.53 7.87
6,88 + 2.33
13.26 = 1.61
LSUNe
48.47 1.61
11.20
4.24 2.34
2.06 + 1.54
0.23 + 0.04
LSUNr
46.73 - 0.66
27.33 + 1.03
3.11 = 0.78
1,90 + 0.51
6.40 + 1.32
Cifar10
TINc
76.80 + 8.20
85.35 + 2.86
89.31 + 10.05
98.59 + 0.67
99.99 0.00
TINr
2.110
86.79 + 1.17
79.02 = 12.17
88.72 = 4.93
94.71 = 0.51
AUPR-In
LSUNe
72.16
1.66
96.70 1 0.21
94.78 1: 4.07
98.31 1 0,92
100.00 1 0.00
LSUNr
65.37 4.39
89.93 +: 1.23
79.41 + 19.89
96.86 + 1.27
99.02 - 0.20
00
TINc
58.29 + 3.01
71.18 + 2.69
97.35 + 2.04
98.24 + 1.50
99.99 0.01
TINr
52.96
0.59
70.95 + 2.20
77.32 + 9.81
91.67 = 1.29
91.91 + 1.34
LSUNe
47.41
2.86
2.86
92.26 2.17
95.45 1 2.32
99.09 1 0.88
99,99 1 0.00
LSUNr
50.47 1: 1.75
74.22 +: 1.14
95.53 1: 0.95
98.11 - 0.78
98.07 = 0.52
Cifar10
TINe
83.63 + 5.11
88.67 + 2.28
91.34 + 8.69
99.32 = 0.35
99,99 = 0.00
TINr
58.83 + 1.77
84.26 1 0.95
89.21 1 6.22
94.60 1 2.70
96.31 0.28
AUPR-OII Out
LSUNe
78.43 1 5.12
98.16 1 0.12 0.12
98.01 1 1.18
99.14 1 0.48
99,99 0.00
I.SUNr
70.51 + 3.97
88.84 + 1,20
84.45 21.48
98.41 + 0.70
99.14 0.19
Cifar100
TINe
62.88 + 7,90
2.21
98.77 + 1.23
99.08 = 0.51
99,99
0.01
TINr
55.94
1 0.71
71.57 1: 1.71
89.44 1 6.96
96.84 1 0.82
94.66
1.07
LSUNe
49.91 4.42
93.77 1: 5.30
98.33 1 0.99
99.39 1 0.48
99,99 0.00
L.UNNr
55.18 1.56
78.19 + 1.33
98.49 + 0.37
0.24 99.32
98.35 0.56

---

Generalization of OOD Detection. Our STEP approach and some previous studies (e.g.. UOOD MAH) ) could utilize OOD samples during the training phase. For example, our STEP performs contrastive learning on both ID) and OOD data, UOOD optimizes the discrepancy loss ID and OOD unlabeled data and selects the final model with an extra OOD validation set, MAH tunes their weighting parameters on an OOD validation set. Let known OOD samples denote the OOD samples that the algorithm used during the training phase, contrasting to unknown OOD samples. We want to explore whether the use of known OOD samples will reduce the performance of the model on unknown OOD samples. Therefore, we design a novel experiment for algorithms using OOD samples, in which the model is trained with ID dataset and known OOD samples while tested with unknown samples. As an example, we train the model on the ID data set (CIFAR-10) and OOD data set (TINr) and replace all OOD samples in the test set with OOD detection model with strong generalization should obtain consistent performance, no matter what OOD data CIFAR-10 with two different OOD data set pairs. From the results shown in Fig. we found that UOOD and MAH have severe performance degradation when detecting unknown OOD samples. This phenomenon is because the OOD validation set used by these methods introduces a severe bias their models. Furthermore, we also conducted experiments analyze the relationship between OOD detection performance and loss and verified the instability in the training process of the SOTA UOOD method. We put the extra experiments the supplementary material. ODIN's performance changes very randomly, which is also in line with expectations because it has only seen ID data during the training process. Our STEP approach gives a high and relatively close performance on both known and unknown OOD data sets, which proves the effectiveness and strong generalization of our approach. Further, we suggest that the experimental method proposed should be verified in all future OOD detection studies that use OOD samples in the training process.
OOD data set (TINc) when testing. An
used construct the testing set. We tested four OOD detection methods
AlIBut un
Ditaee
Ditani
AUPR-IK Wikkwwr
137%
AT30C
00%
6076
201
2014
27%
Mihilarchhll LO00S
700 vilittee EX000
ATRROC Kww.sTrrniannncoommmmmm OND Tattee
A119R.fe.m EantrsTittrom DCn Thraue
A1TFR Knvanrrrinvan Tambur
3orma3 SUNK
100%
Kavmat
1074
667
30%
Okm
000Y
Chev
Mith-de
Mith-ds
Figure 2: Performance of different methods on Known Unknown OOD data set evaluated by various metrics. The results shows that our STEP approach not only has very good OOD detection performance, but also can generalize to unknown OOD samples.
Ablation Study. As introduced in Section 2, our STEP approach contains four components in total: MAH, KNN, Unzipping, and Structure-Keep. Comprehensive ablation studies conducted to verify the effectiveness of each component. As shown in Tab. we sequentially add the components of STEP and verify the performance of each model on two OOD benchmarks. The first line in the table shows the results of directly distinguishing the minimum Mahalanobis distance from the target sample to each class center. Since necessary Σ cannot be accurately estimated, the detection performance is ideal. The second line proves that the geodetic distance can alleviate the inaccurate estimation problem to a certain extent, thereby improving the detection performance. The third line is the incomplete version our STEP approach to remove Structure-Keep. The result this line proves that the Structure-KKnd Keep technique is very important. Otherwise, the detection performance will be greatly reduced. The fourth line, our STEP approach, gives the best results. This proves that the four steps proposed in this article can only be integrated together to get the best results.
7

---

Table 2: Ablation Study of our STEP approach evaluated by AUROC. This table proves that every part of our approach indispensable.
Different parts of STEP
Data set pair
MAH KNN
Unzipping Sturture-Keep
CifarIO-TINr
Cilarl0-LSUNr
90.96 + 0.28
93,46 = 0.51
91.26 + 1.74
97.35 + 0.45
7
79.58 + 0.69
80.38 + 0.95
V
95.62 + 0.39
99.07 + 0.20
Robustness. In this paragraph, we verify the robustness of STEP to hyper-parameter K and the number of labeled data [Pr]. We test the performance of STEP on different OOD data sets for different choices of hyper-parameter K (the number of neighbors when KNN is built). Furthermore, we find that choosing smaller K helps improve the detection performance. Then we test how the amount ID labeled
large range from 2 to 18. Fig. (3a) shows that STEP is not sensitive the
data affects the performance of different methods. From the results shown in Fig, ()),, we find that
our STEP is very tolerant of the amount of ID labeled data. Even in the case of extremely insufficient ID labeled data, an acceptable performance still can be achieved by our STEP approach.
AUROC with Various
AUROC with Various Sizes of Labeled
100%
90%6
99%
TNN
AUROC
90%
IINe
LSUNG
50%
97%
46%
ODIN
30/
Mahnslucbis
967
UOOD*
Ous
95%
06/
100
150
200
250
K
Size of Labeled Data
(a) AUROC with various K on different OOD bench- marks.
(b) AUROC of different methods with various sizes of labeled data.
Figure 3: The robustness of STEP approach. (a), (b) show that STEP approach robust on K and size of labeled data, respectively.
4 Related Work
This work is mainly related to self-supervised learning, semi-supervised learning, positive-unlabeled learning, and OOD detection.
Self-supervised learning. Self-supervised learning a powerful framework to learn discriminative feature representations in an unsupervised fashion via artificially designed auxiliary tasks. Recently, contrastive learning [17 shows remarkable progress on it. Benefiting from the progress, some studies 42 [6] utilize the learned representations to cluster samples with unseen labels. STEP proposed this paper takes advantage of the powerful features derived from the use of contrastive learning. Any progress in comparative learning can be used by STEP to further improve OOD detection performance.
Semi-supervised learning. SSL 36 aims to leverage unlabeled data to improve the performance of the model when plenty of labeled data inconvenient and expensive access. Our paper is mainly related to deep SSL. The combining of SSL technology and DNNs has significantly improved classification accuracy. Many excellent studies, such as consistency regularization based methods entropy minimization based methods [I] and holistic methods 3 39 have been proposed recent years. There are also some studies Specifically, they aim to ensure the performance of SSL when unlabeled data contains OOD samples.
41 35
[3 50 that focus on improving the safeness of SSL..

---

However, these studies all consider the classification performance of the model for known categories under the semi-supervised setting and ignore the problem of overeonfidence in the OOD sample when testing. Efforts on this issue remain limited. Therefore, we propose the semi-supervised OOD detection setting and design STEP approach for it.
Positive-unlabeled learning. PU learning [1] is the setting where learner only has access to positive examples and unlabeled data. Studies in this direction can be mainly divided into three categories:
two-step techniques [32,. biased learning [33] and class prior incorporation 77]. Some recent studies expand this technique into the situation that includes anomalies and OOD samples. ADOA [51] considers the anomaly detection problem where we only can observe some labeled anomalies along
with unlabeled data. PUC [46] aims select data for network compression from massive unlabeled data that may contain OOD samples. However, these methods only consider the known distributions which pay little attention on the detection performance of unknown distribution.
OOD detection. OOD detection has been studied for a long history. The baseline 18 of this problem attempts to detect OOD samples depending on the predicted softmax class probability. Modified generative adversarial networks [26) are used to generate challenging OOD samples during the training stage, and the algorithm encourages the classifier to assign OOD samples uniform class probabilities. ODIN [3 applies the temperature scaling and input preprocessing to further strengthen the difference between ID samples and OOD samples. ELOC 4 uses the ensemble of K leave-out classifiers to detect OOD samples. There are some other studies that use energy-based models B4 12, hierarchical relations 23
24], . and so on. The current state-of-the-art method 49] for OOD detection utilizes the discrepancy between two classifiers to separate II) and OOD samples. Nevertheless, these studies either assume that there is an accurate classification model or assume that there is sufficient labeled data, which limits their application in the real world. There are also some unsupervised OOD detection studies [40] 19 2 38 utilizing the power of the contrastive learning framework. However, although these studies do not require labels, they still need large amount of ID data for training. Previous studies [3] have reported that collecting clean unlabeled data is also very difficult in the real world. Hence, we study a more general setting that is very common in real-world applications in this paper.
5 Conclusions
In this paper. we propose novel OOD detection setting, called semi-supervised OOD detection. In this setting, we aim to distinguish ID and OOD samples by using limited ID labeled data and large amounts of mixed unlabeled data. Due to the generality of this setting, it commonly occurs in real-world applications. In the case of only having limited ID labeled data, we find that the previous studies have suffered performance degeneration, mainly due the inaccurate estimation from the limited ID data. Focusing on this setting, we propose a novel STEP approach. Our main idea is to detect OOD samples in detection-specinf topological structures as the original feature space. Our STEP approach outperforms other methods by large margin in most cases and achieves remarkable detection performance on several benchmarks. Meanwhile, we also conduct comprehensive experiments to verify the robustness and generalization of our STEP approach. The limitation of our work the lack of solid theoretical results. Broadly speaking, other OOD detection methods also have similar problems. We will put efforts into the theoretical understanding of OOD detection in future work.
space where maintain the same local
Broader Impact
In this work, we study OOD detection, which is a fundamental problem in deep learning. Specifically, we first proposed a novel OOD detection setting. In this setting, only limited ID labeled data and many mixed unlabeled data can be used for OOD detection. This is a novel and practical setting commonly appearing in real-worle applications because under this setting, we neither require large amount of labeled data nor clean unlabeled data. We propose the STEP approach to detect OOD samples in a detection-speciff space, greatly improving the performance of OOD detection. Our work will give instructions for those applications having difficulties collecting large quantities of pure ID labeled data while demanding detecting OOD samples to prevent potential dangers in real-world applications. At the same time, there is still much room for exploration in this setting, We hope our

---

work can inspire more discussions about OOD detection in real scenarios and drive more researchers build practical and robust OOD detection algorithms.
Meanwhile, we are aware that abuse of this technology can pose ethical issues. In particular, we note that people expect that real people rather than algorithms make the judgments behind the system. Despite the risks of such AI research, developing and demonstrating such technologies is essential to understand the technology's practical and potentially troublesome applications. We hope that the responsible use of technology will stimulate discussion about these methods' practices and controls.
Acknowledgment
This research was supported by the NSFC (62176118, 61921006), and the Hikvision Cooperation Fund.
References
[1] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: survey, Machine Learning. 109(4):71--70, 2020.
[2] Liron Bergman and Yedid Hoshen, Classification- -haiiioon--aeedddd anomaly detection for general data. In Proceedings of the 8th International Conference on Learning Representations, pages 1-12, 2020.
[3] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: comi-superiised learning with distribution matching and augmentation anchoring. In Proceedings of the 8th International Conference on Learning Representations, pages 1-13, 2020.
[4] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1721-1730, 2015.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. Proceedings of the 37th International Conference on Machine Learning, pages 1597-1607, 2020.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Ying Li, Bing Liu, and Sunita Sarawagi, editors, Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 113-220, 2008.
[8] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-wond attacks deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1625-1634,
2018.
[9] Geli and Bing Liu. Breaking the closed world assumption in text classification. In Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 506-514, 2016.
[10] lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[11] Yves Grandvalet and Yoshua Bengio. Semi-super learning by entropy minimization. In Advances in Neural Information Processing Systems, pages 536, 2004.
[12] Will Grathwohl, Kuan-Chieh Wang, Joern-Hennik Jacobsen, David Duvenaud, Mohammad Norouzi. and Kevin Swersky. Your classifier is secretly an energy based model you should treat one. In Proceedings of the 8th International Conference on Learning Representations, pages 1-23, 2020.
[13] Lan -he Guo, Zhen Yu Zhang, Yuan Jiang, Yu-Feng Li, and Chi-Hua Zhou. Safe deep remi-supervised learning for unseen-class class unlabeled data. In Proceedings of the 37th International Conference on Machine Learning, pages 3897-3906, 2020.
10

---

[14] Lan-Zhe Guo. Zhi Zhou, and Yu-Feng Li. RECORD: resource constrained semi-supervised learning under distribution shift. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1636-1644, 2020.
[15] Lan- au-Zhe Guo, Zhi Zhou, Jie-Jing Shao, Qi Zhang, Feng Kuang, Gao-Le Li. Zhang-Xur Liu, Guobin Wu, Nan Ma, Qun Li, and Yu-Feng Li. Learning from imbalanced and incomplete supervision with its application to ride-sharing liability judgment. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 487-495, 2021.
[16] Kai Han, Sylvestre-Alvise Rebuffi, Sehastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automat- ically discovering and learning new visual categories with ranking statistics. In Proceedings of the 8th International Conference on Learning Representations, pages 1-13, 2020.
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
[18] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the 5th International Conference on Learning Representations, pages 1- 2017.
[19] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pages 15637-15648, 2019.
[20] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of-distribution image without learning from Vision and Pattern Recognition. pages 10951-10960, 2020.
data. Proceedings of the IEEE Conference on Computer
[21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
[22] Jeff Johnson, Mathhis Douze, and Herve Jegou. Billion-scale similarity search with gpus. IEFE Transac- tions on Big Data, pages 1-1, 2019.
[23] Josef Kittler and Cemre Zor Delta divergence: novel decision cognizant measure of classifier incongru- ence. IEEE Transactions on Cybernetics, 49(6):2331-25) 2223,,,233,,22 2018.
[24] Josef Kittler, Cemre Zor, Ioannis Kaloskampis, Yulia Hicks, and Wenwu Wang. Error sensitivity analysis of delta divergence-a novel measure for classifier incongruence detection. Pattern Recognition, 77:30 44, 2018.
[25] Alex Krizhevsky and Hinton Geoffrey. Learning multiple layers of features from tiny images. Tech Report, 2009.
[26] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training comfidence--alibrated classifiers for detecting out-of-distribation samples. In Proceedings of the 6th International Conference on Learning Representations, pages 1-16, 2018.
[27] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[28] Shao- Yuan Li, Sheng-Jun Huang, and Songcan Chen. Crowdsourcing aggregation with deep bayesian learning. Science China Information Sciences, 64(3), 2021.
[29] Yu-Feng Li and De-Ming Liang. Safe semi-supervised learning: a brief introduction. Frontiers of Computer Science, 13(4)x669-676, 2019.
[30] Yu-Feng Li, Lan-Zhe Guo, and Zhi-Hua Zhou. Towards safe weakly supervised learning. IEEE Transac- tions on Pattern Analysis and Muchine Intelligence, 433443334-34 2021.
[31] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the 6th International Conference on Learning Representations, pages 2018.
[32] Bing Liu, Wee Sun Lee, Philip S. Yu, and Xiaoli Li. Partially supervised classification of text documents. In Proceedings of the 19th International Conference Machine Learning, pages 387-394, 2002.
11

---

[33] Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S. Yu. Building text classifiers using positive and unlabeled examples. In Proceedings of the 3rd IEEE International Conference Data Mining, pages 179-188, 2003.
[34] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-t--distribution detection. In Advances in Neural Information Processing Systems, pages 21464-21475, 2020.
[35] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pottern Analysis and Machine Intelligence, 41(8):1979-199) 2018
[36] Avital Oliver, Augustus Odena, Colin Rafel, Ekin Dogus Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep supervssde learning algorithms. In Advances in Neural Information Processing Systems, pages 9239-3250, 2018.
[37] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep supervisee learning algorithms. In Advances in Neural Information Processing Systems, pages 3235-3246, 2018.
[38] Vikash Sehwag, Mung Chiang. and Prateek Mittll, Ssd: A unified framework for self- supervisd outlier detection. In Proceedings of the 9th International Conference on Learning Representations, pages 1-17, 2021.
[39] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, pages 396-608, 2020.
[40] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, pages 11839 1199-11111 11852, 2020.
[41] Antti Tarvainen and Harri Valpola. Mcan teachers are hetter role models: Weight-avcraged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pages 1195-1204, 2017.
[42] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision, pages 268-200 288, 2020.
[43] Apoory Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. -ut-o--distiibution detection using an ensemble of self supervised leave-out classifiers. In Proceedings the European Conference on Computer Vision, pages 550 564, 2018.
[44] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Mctaxas, and Chao Chen. A topological filter for learning with label noise. In Advances in Neural Information Processing Systems, pages 21382-21393, 2020.
[45] Miao Xu and I Lan-Zhe Guo. Learning from group supervision: the impact of supervision deficiency on lubti-label learning. Science China Information Sciences, 64(3), 2021.
[46] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. In Advances in Neural Information Processing Systems, pages 2561-2570 2019.
[47] Haiqin Yang, Shenghuo Zhu, Irwin King, and Michael Lyu. Can irrelevant data help semi-supervised learning. why and how? Proceedings of the 20th ACM International Conference on Information and Knowledge Management, pages 937-946, 2011.
[48] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image datasct using deep learning with humans in the loop. arXiv preprint 2015.
150iv
[49] Qing Yu and Kiyoharu Aizawa. Unsupervised out-s--diit detection by maximum classifier dis- crepancy. In Proceedings of the IEFE International Conference on Computer Vision, pages 9517-9525, 2019.
[50] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriuluum framework for open-set semi-supervised learning. In Proceedings of the European Conference on Computer Vision, pages 438 454, 2020.
12

---

[51] Ya-Lin Zhang, Longfei Li, Jun Zhou, Xiaolong Li. and Zhi-HHa Zhou. Anomaly detection with partially observed anomalies. In Proceedings the 27th International World Wide Web Conferences, pages 639-6466 2018.
[52] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44 53, 2018
13


Reviews:
[{"review_id": "ljx7R9M95Sd", "reviewer": "Reviewer_oqBv", "summary": "**Summary**\nThis paper proposes a new OOD detection method in a semi-supervised OOD detection setting. The problem formulation is based on two characteristics: (1) labeled ID data is limited and (2) unlabeled data can contain a mixture of ID and OOD data. The setting is interesting and more practical in real-world applications. The proposed method STEP is based on the projected space where local topological structures are maintained for the semi-supervised setting.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is thoughtfully written and easy to follow.\n- The proposed method and the setting of semi-supervised OOD detection is novel\n- The reported experimental results seem quite promising (with a few caveats, as described below).\n\n**Weaknesses**\n- [validity of OOD during training] The training setup is not completely OOD agnostic, as acknowledged by the authors. In real applications, one can not assume the availability of test OOD during training. The proposed setting had been studied in [1], where a similar assumption is made that unlabeled data is a mixture of ID and OOD. The reviewer thinks a more novel setting is OOD-agnostic, which makes the generalization to unseen OOD critical. Recent works [2][3] also leverage a large pool of unlabeled data for model regularization, which can be relevant. The subtle differences being that [2][3] do not consider limited ID data, and the unlabeled data does not assume the mixture of ID and OOD (which seems more realistic and less restrictive than the proposed setting).\n- [Validity of Generalization of OOD Detection] The review is glad that the authors made an effort to address the concern above by testing the OOD detecting performance on an unseen set. However, the experiments are very limited because (TINc, TINr) and (LSUNr, LSUNc) are semantically similar. It would be more convincing if the authors can report the performance on a broader range of OOD test sets. \n- [high time complexity] Training with SimCLR for 500 epochs itself is super expensive. To ensure a fair comparison, can the authors provide the training and evaluation time for each method and start from a representation not obtained by SimCLR?\n- [Need for labeled data] From L239 - L246, the detection performance is largely unaffected even if the size of the labeled data is only 10. It seems that the label information is redundant. Perhaps this makes the method in spirit closer to the self-supervised setting, and may deem a careful comparison with recent methods such as CSI [4]and SSD [5]. In comparison with UOOD* (with OOD validation data), the performance gain seems quite marginal.\n- [Visualization] Besides the illustration in Figure 1, it might be helpful to provide the visualization of the representation space, especially when the training OOD and test OOD are semantically different? In which case, by viewing the decision boundary, it may be more convincing to see that the proposed method can indeed generalize to unseen OOD distribution.\n\n----\nPost rebuttal:\nI raised my score since the authors have addressed the major concerns. In the final version, I recommend the authors include the complete results on broader OOD datasets (including some of the more challenging ones such as Textures and CIFAR-10 vs. CIFAR-100), together with self-supervised baselines. \n\n**References**\n[1] Blanchard, Gilles, Gyemin Lee, and Clayton Scott. \"Semi-supervised novelty detection.\" The Journal of Machine Learning Research 11 (2010): 2973-3009.\n\n[2]Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. Deep Anomaly Detection with Outlier Exposure. International Conference on Learning Representations. 2018.\n\n[3] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Out-of-distribution Detection. In Advances in Neural Information Processing Systems, 2020\n\n[4] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020\n\n[5] Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for self-supervised outlier detection. In Proceedings of the 9th International Conference on Learning Representations.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have provided a broader impact statement, which seems sufficient. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UF7jBF9P_Xi", "reviewer": "Reviewer_cW9s", "summary": "This paper introduces a novel out-of-distribution detection setting, where only a tiny subset of labeled ID data is observed. The others remain unlabeled and may belong to ID or OOD. A novel approach is proposed to address the lack of labeled in-distribution data by performing a structure-keep unzipping procedure. The main contribution of the paper is to discuss the new OOD detection setting caused by lacking in-distribution data and propose a new learnable metric to improve the original Mahalanobis distance in this case. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed setting is novel and the proposed approach reasonable, and the experiments are also convincing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The entire training process is to maintain the distance between connected pairs of samples, and at the same time to extend the distance between unrelated samples. How is the training stability of the algorithm, and whether the final performance is seriously affected by the training epochs?\n\n2. The paper written needs to be further improved. For example, there are some overlaps between the content of the related work and the comparison method. Moreover, the font in Figure 2 is too small.\n\n3.  The whole algorithm is based on the features obtained by unsupervised learning. Therefore, it is better to discuss how the quality of the representation learning affects the final performance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "ITas_6mZp_e", "reviewer": "Reviewer_Wkgz", "summary": "This paper proposes a novel setting for semi-supervised learning (SSL): SSL with out-of-distribution unlabeled data. In this setting, a limited amount of labelled in-distribution data and a large amount of unlabeled data, a mixture of both in-distribution and out-of-distribution data, are given.  For the labelled in-distribution data is limited, existing methods cannot work well. The paper proposes a method based on self-supervised learned features to project the data into a new space, where the nearest neighbours' distance is preserved but the non-nearest neighbours' distance is maximized. An efficient optimization problem is proposed and the distance in the new space is then used as a criterion to differentiate out-of-distribution data from in-distribution one. Experimental results are shown to verify the effectiveness of the proposed method compared to baselines. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Both the problem setting and the technologies proposed are novel. The problem setting is a combination of semi-supervised learning and out-of-distribution detection. The combination of the two settings has practical consideration, as many real-world problems listed in the paper encounter such kind of problem. It is non-trivial to be solved by existing methods since the labelled data is limited. In such a setting, how to make the best use of these limited labelled data becomes a challenge. The paper proposes a new method to tackle the challenge from the way of topological preserving. An efficient optimization is also proposed and can be combined easily with deep neural networks. \n\nThe paper uses abundant experimental results to support the effectiveness of the proposed method. With four different combinations of data, the proposal achieves the best performance in almost all settings.  Its performance is significantly better than other baselines with or without known that the out-of-distribution data exist. The ablation study shows the key components of the proposed method are necessary. The paper also shows that the proposal is less sensitive to hyperparameters. \n\nOverall, the paper proposes a novel learning setting with a novel solution. The novel learning setting is of practical value. The methods are appropriately used, and the experimental results have demonstrated the effectiveness of the proposed method from various aspects. \n\nI appreciate that the paper has honestly admitted in the conclusion that a theoretical study is lacking for the current paper and promise to explore further in this direction. It is quite valuable if some theoretical results can be given, showing how the amount of out-of-distribution data will impact the performance of the proposal compared to baselines, as well as how the amount of in-distribution labelled data will impact the performance. Before further exploring this direction in future work, I suggest the paper show some empirical studies by varying the amount of out-of-distribution or in-distribution data and plotting out how the proposal’s and baselines' performance change with that. \n\nAnother thing is the clarity may need further improvement in some parts of the paper. It is a bit vague in the Introduction whether the setting is of “limited in-distribution unlabeled data”, or “abundant in-distribution unlabeled data”. I suggest the paper clarifies this point. Other parts of the paper are well-written and easy to follow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Lacking theoretical studies is one limitation. The paper has also pointed it out in the conclusion. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Ehpr3vWUgUS", "reviewer": "Reviewer_vv5y", "summary": "This paper studies a novel setting called semi-supervised out-of-distribution detection. Focusing on the inaccurate estimation problem in this setting, the author proposes a STEP approach to learning an OOD detection-specific space where the model can better separate the in-distribution samples and OOD samples. The comprehensive experiments prove the effectiveness of the proposed approach in this paper.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper analyzes the problem that the OOD detection algorithm potentially encounters in the semi-supervised out-of-distribution detection setting, that is the inaccurate estimation of in-distribution samples. The solution using unlabeled data to compensate for missing in-distribution data seems reasonable and insightful.\n2. The experiment part includes comparisons with the SOTA methods, ablation experiments, and hyper-parameter experiments. The experimental content is relatively substantial and basically demonstrates the viewpoints put forward in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The quality of the features obtained by unsupervised training seems to have a quiet impact on the success of the algorithm. Does it infer that the features obtained by unsupervised learning already have a certain OOD detection ability?\n2. Although the experimental part of the comparison method is relatively sufficient, does the proposal outperforms unsupervised OOD detection methods?\n3. The histogram of Figure.2 does not intuitively shows extract change of performance changes. It's better to add the percentage of performance change.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studied out-of-distribution detection problem and proposed semi-supervised OOD detection setting with a new technique called structure-keep unzipping to solve the new setting. The writing is clear, the motivation is strong, the idea is novel, and the results are significant. Thus, I think it should be accepted for publication.

A reviewer had three concerns but the authors wrote a nice rebuttal which addressed two concerns. I think the third concern, which was discussed internally, is also sensible. The problem setting is closely related to positive-unlabeled learning; see [1]. If PU learning can be used to detect in-distribution data, it can of course be used to detect out-of-distribution data. Please add PU learning into the section 4 related work.

A minor comment is that OOD in the title should be given the full term instead of the abbreviation --- first, OOD is not as well known as SVM and it may hurt the understanding (the title should be understandable to every NeurIPS participant), and second, in-distribution is not abbreviated.

[1] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive-Unlabeled Compression on the Cloud. NeurIPS 2019.

Decision: Accept (Poster)