STEP: Out-of-Distribution Detection in the Presence of Limited In-Distribution Labeled Data
Authors: Zhi Zhou
Lan-Zhe Guo
Zhanzhan Cheng
Yu-Feng Li
Shiliang Pu
Venue: NeurIPS 2021 Poster (2021)
Keywords: weakly supervised learning
semi-supervised learning
out-of-distribution detection

Abstract: Existing semi-supervised learning (SSL) studies typically assume that unlabeled and test data are drawn from the same distribution as labeled data. However, in many real-world applications, it is desirable to have SSL algorithms that not only classify the samples drawn from the same distribution of labeled data but also detect out-of-distribution (OOD) samples drawn from an unknown distribution. In this paper, we study a setting called semi-supervised OOD detection. Two main challenges compared with previous OOD detection settings are i) the lack of labeled data and in-distribution data; ii) OOD samples could be unseen during training. Efforts on this direction remain limited. In this paper, we present an approach STEP significantly improving OOD detection performance by introducing a new technique: Structure-Keep Unzipping. It learns a new representation space in which OOD samples could be separated well. An efficient optimization algorithm is derived to solve the objective. Comprehensive experiments across various OOD detection benchmarks clearly show that our STEP approach outperforms other methods by a large margin and achieves remarkable detection performance on several benchmarks. 

STEP Out-of-Distribution Detection in the Presence of Limited In-distribution Labeled Data
Zhi Zhou', Lan-Zhe Guol,, Zhanzhan Cheng", Yu-Feng Li Shiliang Pu²
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China {zhouz, guolz, 1iyf)=lamda.nni .nj. www..du..uu.. edu. cn Hikvision Research Institute, Hangzhou, China (chengzhanzhan, pushiliang hrri)hhikvssion.com
Abstract
Existing semi-supervised learning (SSL) studies typically assume that unlabeled and test data are drawn from the same distribution as labeled data. However, many weal-world applications, it is desirable to have SSL algorithms that not only classify the samples drawn from the same distribution of labeled data but also detect -uttr--distribution (OOD) samples drawn from an unknown distribution. In this paper, we study a setting called semi-supervised OOD detection. Two main challenges compared with previous OOD detection settings are the lack of labeled data and in in-distribution data; ii) OOD samples could be unseen during training. Efforts on this direction remain limited. In this paper, we present an approach STEP significantly improving OOD detection performance by introducing new technique: Structure-Kest Unzipping. It learns a new representation space which OOD samples could be separated well. An efficient optimization algorithm is derived solve the objective. Comprehensive experiments across various OOD detection benchmarks clearly show that our STEP approach outperforms other methods by a large margin and achieves remarkable detection performance on several benchmarks.
Introduction
Deep learning has achieved success in many application scenarios, such as computer vision, speech recognition, natural language processing [10]. The excellent performance typically rely on sufficient supervised information. However, collecting large amounts of well-labeled training data is not always available in real-world applications due to the expensive cost of the labeling process. Therefore, tremendous efforts have been devoted semi-supervised learning (SSL) [36] 30 which aims to enhance the model performance by exploiting much cheaper unlabeled data, and have shown
promising performance [52 37 29]
Previous SSL studies 33 41 typically work on the assumption that unlabeled data and test data drawn from the same distribution as labeled data. However, is often the case that such an assumption fails in practical applications [13 14]. For example, in document classification [9]. irrelevant documents readily occur in the testing data leading high-confidence misclassification. Similar cases commonly appear in other applications, such as medical diagnosis 44 and autonomous driving [ In such applications, it is desirable to have SSL algorithms which could not only classify samples from known distributions accurately but also be equipped with the ability to detect out-of-distribution (OOD) samples from unknown distributions precisely.
"Contribute to this work equally I CConesponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

OOD detection has been studied for a long history with numerous methods proposed, such as ODIN BT, Mahalanobis (27]. DeConf [20). ELOC 44 These methods perform OOD detection based on the logits of the model or the Mahalanobis distance in the feature space. However, it is hard to adapt these methods to semi-supervised settings because they all rely on massive labeled data, There are some methods associating with unlabeled data, such as UOOD 49]. CSI [40]. SSD (38) have been proposed recently. These methods assume that the model can obtain sufficient in-distribution (ID) labeled data ID unlabeled data during the training process. Such an assumption also limits their ability to practical problems.
Therefore, we study a novel setting called sami-supervised OOD detection. Specifically, only tiny subset of ID labeled data is observed. The other ones remain unlabeled and may belong to ID or OOD. Here, we assume that abundant ID data is contained in the unlabeled data for extracting ID information. This setting ubiquitous in real-world applications. For example, in web page classification [47] acquiring large numbers of web pages annotated with relevant categories is very expensive, and unlabeled web pages crawled from the Internet according to keywords usually contain irrelevant pages that belong to unseen categories. In medical diagnosis 4 warning users of the model's s uncertainty is crucial because any unfaithful diagnosis will bring unimaginable disasters to the patients' health. In ride-sharing liability judgment [1], detecting abnormal orders is of significant value, while collecting training data will meet similar problems stated above. Similar cases often occur in other real-world applications, such as crowdsourcing [45] 28). There are two main challenges for us compared with previous OOD detection settings. First, both the labeled data and directly available ID data are limited, while sufficient unlabeled data is mixed with ID and OOD samples. Second, OOD samples could be unseen during the training, requiring more stringent generalization of the model.
Focusing on semi-supervised OOD detection, we find that the widely-used Mahalanobis distance no longer suitable the confidence score for OOD detection. This is because the necessary covariance matrix Σ for calculating Mahalanobis distance is hard to estimate accurately with limited ID samples which will severely affect the performance of OOD detection. To alleviate this issue, we propose a novel STEP (STructure-keEP approach. The idea is to detect OOD samples in a detection-speciffe space where we maintain the same local topological structures as the original feature space, because the relationships between samples need to be confirmed through local topological structures. We introduce a new objective and optimize it efficiently. The experiments prove that our STEP approach outperforms previous methods by large margin diverse data sets.
The contributions of our paper are summarized as follows:
We propose practical setting for OOD detection, called semi-supervised OOD detection.
To alleviate the problem of Mahalanobis distance that the necessary covariance matrix S is hard to be estimated with limited ID samples, we present new distance calculated in a detection-speeifie space as OOD confidence scores.
We evaluate our approach with comprehensive experiments across various OOD detection benchmarks. Our proposal outperforms previous methods by a large margin and achieves remarkable detection performance on several benchmarks.
2 Method
2.1 Notations and Setting
In the semi-supervised OOD detection setting, we assume that limited label data D) {(x;, (x)..))))) y consisting n samples with labels drawn from ID, and an unlabeled data set D., (xx) consisting m unlabeled samples drawn both ID and OOD, are accessible during the train- ing phase. We denote the set of ground-truth classes in the labeled data set D) and unlabeled data set Dr, as C₁ and C,,, respectively. The labeled samples can be classified into one of K classes denoted by C. [c1, ... (x). ) and the unlabeled samples can be classified into the seen K classes C₁ and some unseen classes denoted by C, = C,, (C, The goal to distinguish whether a sample in D., or an unknown testing sample is drawn from ID or not.

---

2.2 Inaccurate Mahalanobis Distance and Our Approach
Mahalanobis distance which is widely used in previous studies 27 38 has been proven
powerful metric in OOD detection. MD(x,, xj) denotes the function measuring the Mahalanobis distance between sample x; and sample X, based estimated covariance matrix Σ:
MD(x;; V(x (x) -x)
Previous methods mentioned above calculate the minimum Mahalanobis distance between target sample and each class center the confidence score:
SCOREMD(X)) oECr 0EC,,.2
min
MD(x,, 1b)
where [4, denotes the center of samples which belong to class e and Σ the covariance matrix estimated all ID samples.
However, Σ is hard to be accurately estimated in semi-supervised OOD detection setting because the available ID labeled data set Dj is insufficient. Inaccurate estimation of Σ will affect the calculation of Mahalanobis distance. This makes difficult for the algorithm to distinguish OOD samples and ID samples near the cluster boundary.
project samples into space where large margin separates ID samples and OOD samples. Inspired by the topological technology 44 used in noisy label problems and cluster assumption (36 used SSL, we hope that the projected samples can maintain the same local topological structure as the original space while increasing the distance between samples not directly topologically connected. Because the inaccurate estimation of Σ, we consider that relationship between samples that are not topologically adjacent is uncertain. Their relationships need be confirmed through each local topological structure. We formulate goal into the objective:
Instead of using inaccurate Mahalanobis distance, we decide to learn
max
Σ
Px Px2l12
X, ED;UD, PPx Px,l|2 MD(x; xn),
s.t.
if
Xn E (x.)
where, MMx;, x is the Mahalanobis distance between and xy in the feature space and By (x,) is the set of k nearest neighbours of x4.
Finally, our detection-specific metric can directly calculate as L2 distance in the projected space:
N(x,, = |Px; Px,l|2
(4)
2.3 Backbone Pretraining
Our semi-supervised OOD detection task considers OOD detection as a clustering problem based on the feature space. Therefore, reliable feature representations essential. Benefiting from recent progress on self-supervised learning, we adopt simple contrastive learning method SimCLR (5) pre-train backbone network on the whole dataset D. U Di in unsupervised fashion. We find that representations obtained by SimCLR have a reasonable ability to distinguish ID and OOD samples. Notably, the learned representations could be not only used for STEP approach but also used as the initialization of downstream tasks.
2.4 Structure-Keep Unzipping
Based on the representations obtained by SimCLR, we further train to project samples into detection-specific space via our objective formulated in (3) However, there are two main difficulties: i) Building a KNN graph for extracting topological structure needs dn))d)) time complexity to calculate Mahalanobis distance between each pair of samples. This step is very time- consuming because we use an ensemble of representations from each backbone network's layer, and feature dimension d is relatively large. ii) The constraint in Eq. can not be directly optimized.

---

ID Labeled Data
ODD Unlabcled Data
OOOD Unlubeled data
Projector
Initial Step
Step A
Step
Figure The overall of STEP approach: (a) In initial step, we use contrastive learning to train initial representations. (b) In step A, we estimated statistics information via limited labeled data and extracted topological structure via the KNN algorithm. (c) In step B, we train P project all the samples into a detection-specife space where we can use 1.2 distances as OOD scores.
First, we transform the process of calculating pairwise Mahalanobis distance into calculating pairwise Euclidean distance in projection space. The time complexity of this step reduces from (n?dd) to O(n²d). Specifically, as shown in 5). we can perform cholesky decomposition on --1 to get linear projector the PMDD Then, we multiply all samples by PMD to project them into new space where Euclidean distance equals to original Mahalanobis distance between each pair of samples. There are n² pairwise Euclidean distances to calculate, and each calculation costs ((d)
time complexity. Therefore, the total time complexity of this step is (ndd).
MD(x;,,,) = (x, x,) (x,-xxx)) PMDX P PMDX211
(5)
s. PMD
After converting Mahalanobis distance to Euclidean distance, we can further use the advanced KNN toolkit, such Faiss [22] to speed up the entire process.
Second, we define Leee and LUnsip that can be directly optimized to approximately achieve our objective shown in Eq. 3. Both LKeep and LUnsip are shown in Eq.
[.Keep max(0, |Px; Px,,12 MD(xx, x.)),
(6)
LUnzip
Px, Px,112-
where X;, Xj randomly sampled from Dj U Du. and vn is randomly sampled from B,(x,), The final loss to optimize Loss Keep + LU'sip- The overall our STEP approach is summarized in Fig.(T), and the pseudo-code of our approach is shown in Algo.
In the detection stage, we directly use the minimum I.2 distance between the target sample and each class center in the detection-specifie space as the confidence score:
Score(x)
min
N(x, 14,))
cEfcr cx....c
where the n. is the center of class e the original feature space.
3 Experiments
3.1 Experimental Setup
In-distribution Data Set. We use CIFAR-10 and CIFAR-100 100 25 as data sets in our experiments. They both contain 50,000 training images and 10,000 testing images. The image size of these two data sets is 32 32. For CIFAR-10, each image belongs to one of 10 classes, and we randomly sample 250 training images as ID labeled data D,. For CIFAR-100, the size of image classes is 100, and we randomly sample 400 training images as ID labeled data. Dr. We add the remaining training images to the unlabeled data D...

---

Algorithm Training Phase of STEP
Input: Dj: ID labeled data set; Du: unlabeled mixed data set; K: number of neighbours
Output: pre-trained backbone fo(-): projctor P
I: train backbone fo(-) via contrastive learning D U Du
2: estimate Σ on Dj with fe(-)
3: calculate P.D based on Σ-1
4: build KNN on D JUD, D. with PMD and fo(.)
5: for epoch E (p.cch....pochmma
do
6: randomly sample X, from Dj U Du
7: randomly sample Xa from (x,)
8: calculate Loss based on Eq.(6)
9: optimize P via SGD according to Loss
10: end for
11: return fa(+) and
Out-of-distribution Data Set. We use Tiny ImageNet data set [6] and Large-seale Scene Understand- ing data set [48] as OOD data sets. The Tiny ImageNet data set (TIN) is subset of ImageNet, which contains 10,000 test images, includes 200 different classes. Following the settings used by previous studies [31 43 49 we two variants of TIN: T (TINc) and TinylmageNet-resizee (TINr), by randomly cropping or downsampling each image to 32 x 32, respectively. The Large-scale Scene Understanding data set (LSUN) contains 10,000 testing images belonging scene categories. Similarly, we use two variants of LSUN: LSUN-crop (LSUNc) and LSUN-resize (LSUNr). Because some comparison methods in our experiments heavily rely on OOD validation. We randomly draw several images from ID testing images and OOD images as the OOD validation set. The rest of the OOD images are added unlabeled data Du and used as testing data. These OOD data sets are released by ODIN [1 with their code'.
10 different
Comparsion Methods. We compare our STEP approach with representative OOD detection methods, including the state-of-the-art UOOD method. ODIN (31 is a common baseline of OOD detection. It uses maximal softmax score combining temperature scaling and input preprocessing tricks to distinguish ID and OOD samples. MAH (27) uses Mahalanobis distance as the OOD confidence score. For features of each layer in the backbone model, independently calculates the Mahalanobis distances between the target sample and each known class center. Then it integrates them by weighted averaging via an extra OOD validation set. We denote it as MAII because uses validation set when training. UOOD 49 utilizes a two-head CNN consisting of one common feature extractor and two classifiers which has different decision boundaries detect OOD samples. This method optimizes a discrepancy loss between two classifiers during the training stage and uses this discrepancy as the OOD score when testing. However, this method relies on extra OOD validation to perform model selection. Therefore, we denoted it as UOoD in our experiments For fair comparisons, we also implement a variant of it denoting as UOOD. UOOD that uses discrepancy loss to perform model selection instead of the performance on an extra OOD validation set.
we evaluate our approach with five common metrics: AUROC, FPR at 95% TPR, Detection Error, AUPR-In, and AUPR-Out. More details about evaluation metrics are presented in the supplementary material.
Evaluation Metrics. Following the settings used by previous studies 49 43
Implementation Details. In all experiments, we adopt the Densenet-Bi [21] as the backbone since it is widely used in previous studies [49 4 BI] Our backbone is trained by SOTA contrastive learning method SimCLR (5] for 500 epochs. We set the learning rate to 10-3 with a cosine annealing strategy. For fair comparisons, each comparison method can the pre-trained backbone model. MAH (27) uses the features from different layers extracted from the pre-trained backbone model. A well-trained linear classifier with a pre-traine pre-trained backbone model is provided for ODIN [3T] and UOOD 49 The hyper-parameter K for STEP is set to 12 for all data set pairs. All experiments are performed on one single NVIDIA 3090 graphics card. More details on implementation are provided in the supplementary material and our code has been open source 2
5

---

3.2 Experiment Results
OOD Detection Performance. We evaluate STEP with compared methods various OOD bench- marks. Analyzed by five common metries, the results are shown Tab.TT). From the results, we observe that ODIN suffers from severe performance degradation. Moreover, its performance is close random guessing some cases. The limitation of labeled data mainly causes this. We can hardly train a high-quality classification model to provide accurate logits for ODIN. Hence, ODIN can not give the correct judgment based on inaccurate logits. Our STEP approach outperforms methods that do not heavily rely on OOD validation set by large margin, Even compared with those methods that heavily rely on the OOD validation set, such as UOOD f and MAH our STEP approach is still better than them in most cases. However, good OOD validation set is expensive and nearly impossible to build in the real world. The number of OOD samples can be infinitely many, and fixed-size validation set cannot capture the complete OOD information. Therefore, introducing the validation set during training will reduce the model's generalization the real environment. We will verify this in detail in subsequent experiments.
Table 1: Performance comparison on various OOD benchmarks evaluated by common metrics. Methods with use extra OOD validation set. The best results are indicated in bold. Our approach outperforms other methods in most cases, even though they use an extra OOD validation set.
ID
OOD
Metrics
ODIN
MAH
UOOD
UooD
STEP
Dataset
Dutaset
Cifar10
TINe
81.00
6.30
87.67 + 2.47
90.46 + 9.74
99.07 + 0.48
99.99
0.00
TINr
59.10
86.88 0.87
84.67 9.41
92.63
3.42
95.61
0.36
AUROC
LSUNc
76.17 5.37
97.68 + 0.09
96.92 + 2.04
98.79 = 0.67
99.99
0.00
LSUNr
69.05 3.49
90.41 1: 1.00
80.87 24.45
97.81 1 0.94
99.07
0.20
10
TINc
61.65 6.71
71.15 = 2.20
98.34 + 1.57
98.84 = 0.83
99.99
0.01
TINr
54.46 + 0.74
73.94 + 1.79
84.80 + 8.87
95.31 + 0.93
93.51 + 1.17
LSUNc
46.99 = 4.99
93,91 + 3,41
97.49 + 1.48
99.31 = 0.62
99.99 = 0.00
LSUNr
52.06 1: 2.24
78.45 1: 1.11
97.61 1: 0.55
98.96 0.40
98. 28.20 0.56
Cifarl0
TINe
53.37 10.55
44.17 + 6.43
29.35 30,05
2.75 1.65
0.00 0.00
FPR at 95%TTP
TINr
89.76
1.45
58.57 3.09
31.72 11.50
19.61
9.50
17.63 1.10
LSUNc
64.06
9.12
7.73 0.46
6.59 1 3.22
3.56 + 1.93
0.00 1: 0.00
LSUNr
76.89 5.04
45.41 + 3.87
32.69 1 31.93
6.49 2.89
4,48 + 1.02
Cifur100
TINc
84.24 8.02
90.15 + 1.99
5.22 5.59
2.255
0,00 + 0.01
TINr
40.10
80.55 1.89
29.09 = 15.68
11.10 = 4.21
23.21 = 4.14
LSUNc
2.42 93.49
24.93 1: 21.75
6.24 3.80
1.93 2.43
0.00 1: 0.00
LSUNr
89.79 0.79
69.69 + 2.42
4.92 = 1.33
2.39 + 0.74
8.25 + 3.14
Cifar10
TINc
25,53 4.67
19.93 +: 2.63
11.59 11.35
2.54 1.27
0.12 1: 0.01
Detection Error
TINr
43.04 + 1.48
20.14 + 0.82
18.07 +: 5.55
11.71 + 4.56
10.77 0.52
LSUNc
29.57 3.822
6.28 + 0.25
4.20 2.12
2.58 + 1.32
0.11 + 0.01
LSUNr
16.23 + 0.95
18.40 = 15.68
4.99 1.91
4.66 + 0.57
Cifar100
TINc
40.95 +
32.58 + 1.64
3.67 3,62
2.76 1.00
0.32 1: 0.06
TINr
46.36 0.56
31.09 + 1.44
16.53 7.87
6,88 + 2.33
13.26 = 1.61
LSUNe
48.47
1.61
11.20
4.24 2.34
2.06 + 1.54
0.23 + 0.04
LSUNr
46.73 + 0.66
27.33 + 1.03
3.11 = 0.78
1,90 + 0.51
6.40 + 1.32
Cifar10
TINc
76.80 + 8.20
85.35 + 2.86
89.31 + 10.05
98.59 + 0.67
99.99 0.00
TINr
2.110
86.79 + 1.17
79.02 = 12.17
88.72 = 4.93
94.71 = 0.51
AUPR-In
LSUNe
72.16
1.660
96.70 1 0.21
94.78 1: 4.07
98.31 1 0,92
100.00 1 0.00
LSUNr
65.37 4.33
89.93 +: 1.23
79.41 + 19.89
96.86 + 1.27
99.02 - 0.20
00
TINc
58.29 + 3.01
71.18 + 2.69
97.35 + 2.04
98.24 + 1.50
99.99 0.01
TINr
52.96
0.59
70.95 + 2.20
77.32 + 9.81
91.67 = 1.29
91.91 + 1.34
LSUNe
47.41
2.86
2.86
92.26 2.17
95.45 1 2.32
99.09 1 0.88
99,99 1 0.00
LSUNr
50.47 1: 1.75
74.22 +: 1.14
95.53 1: 0.95
98.11 - 0.78
98.07 = 0.52
Cifar10
TINe
83.63 + 5.11
88.67 + 2.28
91.34 + 8.69
99.32 = 0.35
99,99 = 0.00
TINr
58.83 + 1.77
84.26 1 0.95
89.21 1 6.22
94.60 1 2.70
96.31 0.28
AUPR-OII Out
LSUNe
78.43 1 5.12
98.16 1 0.12 0.12
98.01 1 1.18
99.14 1 0.48
99,99 0.00
I.SUNr
70.51 + 3.97
88.84 + 1,20
84.45 21.48
98.41 + 0.70
99.14 0.19
Cifar100
TINe
62.88 + 7,90
2.21
98.77 + 1.23
99.08 = 0.51
99,99
0.01
TINr
55.94
1 0.71
71.57 1: 1.71
89.44 1 6.96
96.84 1 0.82
94.66
1.07
LSUNe
49.91 4.42
93.77 1: 5.30
98.33 1 0.99
99.39 1 0.48
99,99 0.00
L.UNNr
55.18 1.56
78.19 + 1.33
98.49 + 0.37
0.24 99.32
98.35 0.56

---

Generalization of OOD Detection. Our STEP approach and some previous studies (e.g.. UOOD MAH) ) could utilize OOD samples during the training phase. For example, our STEP performs contrastive learning on both ID) and OOD data, UOOD optimizes the discrepancy loss ID and OOD unlabeled data and selects the final model with an extra OOD validation set, MAH tunes their weighting parameters on an OOD validation set. Let known OOD samples denote the OOD samples that the algorithm used during the training phase, contrasting to unknown OOD samples. We want to explore whether the use of known OOD samples will reduce the performance of the model on unknown OOD samples. Therefore, we design a novel experiment for algorithms using OOD samples, in which the model is trained with ID dataset and known OOD samples while tested with unknown samples. As an example, we train the model on the ID data set (CIFAR-10) and OOD data set (TINr) and replace all OOD samples in the test set with OOD detection model with strong generalization should obtain consistent performance, no matter what OOD data CIFAR-10 with two different OOD data set pairs. From the results shown in Fig. we found that UOOD and MAH have severe performance degradation when detecting unknown OOD samples. This phenomenon is because the OOD validation set used by these methods introduces a severe bias their models. Furthermore, we also conducted experiments analyze the relationship between OOD detection performance and loss and verified the instability in the training process of the SOTA UOOD method. We put the extra experiments the supplementary material. ODIN's performance changes very randomly, which is also in line with expectations because it has only seen ID data during the training process. Our STEP approach gives a high and relatively close performance on both known and unknown OOD data sets, which proves the effectiveness and strong generalization of our approach. Further, we suggest that the experimental method proposed should be verified in all future OOD detection studies that use OOD samples in the training process.
OOD data set (TINc) when testing. An
used construct the testing set. We tested four OOD detection methods
AlIBut un
Ditaee
Ditani
AUPR-IK Wikkwwr
137%
AT30C
00%
6076
201
2014
27%
Mihilarchhll LO00S
700 vilittee EX000
ATRROC Kww.sTrrniannncoommmmmm OND Tattee
A119R.fe.m EantrsTittrom DCn Thraue
A1TFR Knvanrrrinvan Tambur
3orma3 SUNK
100%
Kavmat
1074
667
30%
Okm
000Y
Chev
Mith-de
Mith-ds
Figure 2: Performance of different methods on Known Unknown OOD data set evaluated by various metrics. The results shows that our STEP approach not only has very good OOD detection performance, but also can generalize to unknown OOD samples.
Ablation Study. As introduced in Section 2, our STEP approach contains four components in total: MAH, KNN, Unzipping, and Structure-Keep. Comprehensive ablation studies conducted to verify the effectiveness of each component. As shown in Tab. we sequentially add the components of STEP and verify the performance of each model on two OOD benchmarks. The first line in the table shows the results of directly distinguishing the minimum Mahalanobis distance from the target sample to each class center. Since necessary Σ cannot be accurately estimated, the detection performance is ideal. The second line proves that the geodetic distance can alleviate the inaccurate estimation problem to a certain extent, thereby improving the detection performance. The third line is the incomplete version our STEP approach to remove Structure-Keep. The result this line proves that the Structure-KKnd Keep technique is very important. Otherwise, the detection performance will be greatly reduced. The fourth line, our STEP approach, gives the best results. This proves that the four steps proposed in this article can only be integrated together to get the best results.
7

---

Table 2: Ablation Study of our STEP approach evaluated by AUROC. This table proves that every part of our approach indispensable.
Different parts of STEP
Data set pair
MAH KNN
Unzipping Sturture-Keep
CifarIO-TINr
Cilarl0-LSUNr
90.96 + 0.28
93,46 = 0.51
91.26 + 1.74
97.35 + 0.45
7
79.58 + 0.69
80.38 + 0.95
V
95.62 + 0.39
99.07 + 0.20
Robustness. In this paragraph, we verify the robustness of STEP to hyper-parameter K and the number of labeled data [Pr]. We test the performance of STEP on different OOD data sets for different choices of hyper-parameter K (the number of neighbors when KNN is built). Furthermore, we find that choosing smaller K helps improve the detection performance. Then we test how the amount ID labeled
large range from 2 to 18. Fig. (3a) shows that STEP is not sensitive the
data affects the performance of different methods. From the results shown in Fig, ()),, we find that
our STEP is very tolerant of the amount of ID labeled data. Even in the case of extremely insufficient ID labeled data, an acceptable performance still can be achieved by our STEP approach.
AUROC with Various
AUROC with Various Sizes of Labeled
100%
90%6
99%
TNN
AUROC
90%
IINe
LSUNG
50%
97%
46%
ODIN
30/
Mahnslucbis
967
UOOD*
Ous
95%
06/
100
150
200
250
K
Size of Labeled Data
(a) AUROC with various K on different OOD bench- marks.
(b) AUROC of different methods with various sizes of labeled data.
Figure 3: The robustness of STEP approach. (a), (b) show that STEP approach robust on K and size of labeled data, respectively.
4 Related Work
This work is mainly related to self-supervised learning, semi-supervised learning, positive-unlabeled learning, and OOD detection.
Self-supervised learning. Self-supervised learning a powerful framework to learn discriminative feature representations in an unsupervised fashion via artificially designed auxiliary tasks. Recently, contrastive learning [17 shows remarkable progress on it. Benefiting from the progress, some studies 42 [6] utilize the learned representations to cluster samples with unseen labels. STEP proposed this paper takes advantage of the powerful features derived from the use of contrastive learning. Any progress in comparative learning can be used by STEP to further improve OOD detection performance.
Semi-supervised learning. SSL 36 aims to leverage unlabeled data to improve the performance of the model when plenty of labeled data inconvenient and expensive access. Our paper is mainly related to deep SSL. The combining of SSL technology and DNNs has significantly improved classification accuracy. Many excellent studies, such as consistency regularization based methods entropy minimization based methods [I] and holistic methods 3 39 have been proposed recent years. There are also some studies Specifically, they aim to ensure the performance of SSL when unlabeled data contains OOD samples.
41 35
[3 50 that focus on improving the safeness of SSL..

---

However, these studies all consider the classification performance of the model for known categories under the semi-supervised setting and ignore the problem of overeonfidence in the OOD sample when testing. Efforts on this issue remain limited. Therefore, we propose the semi-supervised OOD detection setting and design STEP approach for it.
Positive-unlabeled learning. PU learning [1] is the setting where learner only has access to positive examples and unlabeled data. Studies in this direction can be mainly divided into three categories:
two-step techniques [32,. biased learning [33] and class prior incorporation 77]. Some recent studies expand this technique into the situation that includes anomalies and OOD samples. ADOA [51] considers the anomaly detection problem where we only can observe some labeled anomalies along
with unlabeled data. PUC [46] aims select data for network compression from massive unlabeled data that may contain OOD samples. However, these methods only consider the known distributions which pay little attention on the detection performance of unknown distribution.
OOD detection. OOD detection has been studied for a long history. The baseline 18 of this problem attempts to detect OOD samples depending on the predicted softmax class probability. Modified generative adversarial networks [26) are used to generate challenging OOD samples during the training stage, and the algorithm encourages the classifier to assign OOD samples uniform class probabilities. ODIN [3 applies the temperature scaling and input preprocessing to further strengthen the difference between ID samples and OOD samples. ELOC 4 uses the ensemble of K leave-out classifiers to detect OOD samples. There are some other studies that use energy-based models B4 12, hierarchical relations 23
24], . and so on. The current state-of-the-art method 49] for OOD detection utilizes the discrepancy between two classifiers to separate II) and OOD samples. Nevertheless, these studies either assume that there is an accurate classification model or assume that there is sufficient labeled data, which limits their application in the real world. There are also some unsupervised OOD detection studies [40] 19 2 38 utilizing the power of the contrastive learning framework. However, although these studies do not require labels, they still need large amount of ID data for training. Previous studies [3] have reported that collecting clean unlabeled data is also very difficult in the real world. Hence, we study a more general setting that is very common in real-world applications in this paper.
5 Conclusions
In this paper. we propose novel OOD detection setting, called semi-supervised OOD detection. In this setting, we aim to distinguish ID and OOD samples by using limited ID labeled data and large amounts of mixed unlabeled data. Due to the generality of this setting, it commonly occurs in real-world applications. In the case of only having limited ID labeled data, we find that the previous studies have suffered performance degeneration, mainly due the inaccurate estimation from the limited ID data. Focusing on this setting, we propose a novel STEP approach. Our main idea is to detect OOD samples in detection-specinf topological structures as the original feature space. Our STEP approach outperforms other methods by large margin in most cases and achieves remarkable detection performance on several benchmarks. Meanwhile, we also conduct comprehensive experiments to verify the robustness and generalization of our STEP approach. The limitation of our work is the lack of solid theoretical results. Broadly speaking, other OOD detection methods also have similar problems. We will put efforts into the theoretical understanding of OOD detection in future work.
space where maintain the same local
Broader Impact
In this work, we study OOD detection, which is a fundamental problem in deep learning. Specifically, we first proposed a novel OOD detection setting. In this setting, only limited ID labeled data and many mixed unlabeled data can be used for OOD detection. This is a novel and practical setting commonly appearing in real-worle applications because under this setting, we neither require large amount of labeled data nor clean unlabeled data. We propose the STEP approach to detect OOD samples in a detection-speciff space, greatly improving the performance of OOD detection. Our work will give instructions for those applications having difficulties collecting large quantities of pure ID labeled data while demanding detecting OOD samples to prevent potential dangers in real-world applications. At the same time, there is still much room for exploration in this setting, We hope our

---

work can inspire more discussions about OOD detection in real scenarios and drive more researchers build practical and robust OOD detection algorithms.
Meanwhile, we are aware that abuse of this technology can pose ethical issues. In particular, we note that people expect that real people rather than algorithmss make the judgments behind the system. Despite the risks of such AI research, developing and demonstrating such technologies is essential to understand the technology's practical and potentially troublesome applications. We hope that the responsible use of technology will stimulate discussion about these methods' practices and controls.
Acknowledgment
This research was supported by the NSFC (62176118, 61921006), and the Hikvision Cooperation Fund.
References
[1] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: survey, Machine Learning. 109(4):71--70, 2020.
[2] Liron Bergman and Yedid Hoshen, Classification- -haiiioon--aeedddd anomaly detection for general data. In Proceedings of the 8th International Conference on Learning Representations, pages 1-12, 2020.
[3] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: comi-superiised learning with distribution matching and augmentation anchoring. In Proceedings of the 8th International Conference on Learning Representations, pages 1-13, 2020.
[4] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1721-1730, 2015.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. Proceedings of the 37th International Conference on Machine Learning, pages 1597-1607, 2020.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Ying Li, Bing Liu, and Sunita Sarawagi, editors, Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 113-220, 2008.
[8] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-wond attacks deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1625-1634,
2018.
[9] Geli and Bing Liu. Breaking the closed world assumption in text classification. In Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 506-514, 2016.
[10] lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[11] Yves Grandvalet and Yoshua Bengio. Semi-super learning by entropy minimization. In Advances in Neural Information Processing Systems, pages 536, 2004.
[12] Will Grathwohl, Kuan-Chieh Wang, Joern-Hennik Jacobsen, David Duvenaud, Mohammad Norouzi. and Kevin Swersky. Your classifier is secretly an energy based model you should treat one. In Proceedings of the 8th International Conference on Learning Representations, pages 1-23, 2020.
[13] Lan Zhe Guo, Zhen Yu Zhang, Yuan Jiang, Yu-Feng Li, and Chi-Hua Zhou. Safe deep remi-supervised learning for unseen-class class unlabeled data. In Proceedings of the 37th International Conference on Machine Learning, pages 3897-3906, 2020.
10

---

[14] Lan-Zhe Guo. Zhi Zhou, and Yu-Feng Li. RECORD: resource constrained semi-supervised learning under distribution shift. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1636-1644, 2020.
[15] Lan- au-Zhe Guo, Zhi Zhou, Jie-Jing Shao, Qi Zhang, Feng Kuang, Gao-Le Li. Zhang-Xur Liu, Guobin Wu, Nan Ma, Qun Li, and Yu-Feng Li. Learning from imbalanced and incomplete supervision with its application to ride-sharing liability judgment. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 487-495, 2021.
[16] Kai Han, Sylvestre-Alvise Rebuffi, Sehastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automat- ically discovering and learning new visual categories with ranking statistics. In Proceedings of the 8th International Conference on Learning Representations, pages 1-13, 2020.
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
[18] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the 5th International Conference on Learning Representations, pages 1- 2017.
[19] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pages 15637-15648, 2019.
[20] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of-distribution image without learning from Vision and Pattern Recognition. pages 10951-10960, 2020.
data. Proceedings of the IEEE Conference on Computer
[21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
[22] Jeff Johnson, Mathhis Douze, and Herve Jegou. Billion-scale similarity search with gpus. IEFE Transac- tions on Big Data, pages 1-1, 2019.
[23] Josef Kittler and Cemre Zor Delta divergence: novel decision cognizant measure of classifier incongru- ence. IEEE Transactions on Cybernetics, 49(6):2331-25) 2223,,,233,,22 2018.
[24] Josef Kittler, Cemre Zor, Ioannis Kaloskampis, Yulia Hicks, and Wenwu Wang. Error sensitivity analysis of delta divergence-a novel measure for classifier incongruence detection. Pattern Recognition, 77:30 44, 2018.
[25] Alex Krizhevsky and Hinton Geoffrey. Learning multiple layers of features from tiny images. Tech Report, 2009.
[26] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training comfidence--alibrated classifiers for detecting out-of-distribation samples. In Proceedings of the 6th International Conference on Learning Representations, pages 1-16, 2018.
[27] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018.
[28] Shao- Yuan Li, Sheng-Jun Huang, and Songcan Chen. Crowdsourcing aggregation with deep bayesian learning. Science China Information Sciences, 64(3), 2021.
[29] Yu-Feng Li and De-Ming Liang. Safe semi-supervised learning: a brief introduction. Frontiers of Computer Science, 13(4)x669-676, 2019.
[30] Yu-Feng Li, Lan-Zhe Guo, and Zhi-Hua Zhou. Towards safe weakly supervised learning. IEEE Transac- tions on Pattern Analysis and Muchine Intelligence, 433443334-34 2021.
[31] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the 6th International Conference on Learning Representations, pages 2018.
[32] Bing Liu, Wee Sun Lee, Philip S. Yu, and Xiaoli Li. Partially supervised classification of text documents. In Proceedings of the 19th International Conference Machine Learning, pages 387-394, 2002.
11

---

[33] Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S. Yu. Building text classifiers using positive and unlabeled examples. In Proceedings of the 3rd IEEE International Conference Data Mining, pages 179-188, 2003.
[34] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-t--distribution detection. In Advances in Neural Information Processing Systems, pages 21464-21475, 2020.
[35] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pottern Analysis and Machine Intelligence, 41(8):1979-199) 2018
[36] Avital Oliver, Augustus Odena, Colin Rafel, Ekin Dogus Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep supervssde learning algorithms. In Advances in Neural Information Processing Systems, pages 9239-3250, 2018.
[37] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep supervisee learning algorithms. In Advances in Neural Information Processing Systems, pages 3235-3246, 2018.
[38] Vikash Sehwag, Mung Chiang. and Prateek Mittll, Ssd: A unified framework for self- supervisd outlier detection. In Proceedings of the 9th International Conference on Learning Representations, pages 1-17, 2021.
[39] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, pages 396-608, 2020.
[40] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, pages 11839 1199-11111 11852, 2020.
[41] Antti Tarvainen and Harri Valpola. Mcan teachers are hetter role models: Weight-avcraged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pages 1195-1204, 2017.
[42] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision, pages 268-200 288, 2020.
[43] Apoory Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. Out-o--distiibution detection using an ensemble of self supervised leave-out classifiers. In Proceedings the European Conference on Computer Vision, pages 550 564, 2018.
[44] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Mctaxas, and Chao Chen. A topological filter for learning with label noise. In Advances in Neural Information Processing Systems, pages 21382-21393, 2020.
[45] Miao Xu and I Lan-Zhe Guo. Learning from group supervision: the impact of supervision deficiency on lubti-label learning. Science China Information Sciences, 64(3), 2021.
[46] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. In Advances in Neural Information Processing Systems, pages 2561-2570 2019.
[47] Haiqin Yang, Shenghuo Zhu, Irwin King, and Michael Lyu. Can irrelevant data help semi-supervised learning. why and how? Proceedings of the 20th ACM International Conference on Information and Knowledge Management, pages 937-946, 2011.
[48] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image datasct using deep learning with humans in the loop. arXiv preprint 2015.
150iv
[49] Qing Yu and Kiyoharu Aizawa. Unsupervised out-s--diit detection by maximum classifier dis- crepancy. In Proceedings of the IEFE International Conference on Computer Vision, pages 9517-9525, 2019.
[50] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriuluum framework for open-set semi-supervised learning. In Proceedings of the European Conference on Computer Vision, pages 438 454, 2020.
12

---

[51] Ya-Lin Zhang, Longfei Li, Jun Zhou, Xiaolong Li. and Zhi-HHa Zhou. Anomaly detection with partially observed anomalies. In Proceedings the 27th International World Wide Web Conferences, pages 639-6466 2018.
[52] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44 53, 2018
13


Reviews:
[{"review_id": "rSJAw5ZK-U", "reviewer": "Reviewer_Jitz", "summary": "The paper introduces a semi-supervised OOD detection setting and proposes STEP as an approach for this situation. This presented methods combines a Mahalanobis distance with a semi-supervisedly trained backbone model using topological structure preserving optimized projections.\n", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The description of the methodology is not clear enough for me to see that it is sound and that the motivating principles are fulfilled.\nThe presented evaluation is insufficient to judge the efficacy of the presented method. The results are only shown for four OOD datasets, with two pairs of them being the same data only downscaled in different ways.\nAvailability of validation data from the tested out-distribution can not be assumed in an OOD detection setting, as is mentitoned in l.200. However, the evaluation (Fig. 2) for this most relevant setting of 'new OOD datasets' is extremely lacking: only two evaluations are shown, and for each the OOD dataset used for training only differs from the test dataset in in image rescaling methods. A sufficient evaluation would need to include several new OOD datasets, for example as in https://arxiv.org/abs/1812.04606 (Hendrycks et al.'s Outlier Exposure) Table 7.\nOverall, problems of language and structure make the paper very difficult to read and particularly the methodological section hard to assess. The results section is not comprehensive enough to appraise the practical benefits of the method. \n\n\nQuestions and points that remained unclear to me:\n - l. 96 How is the covariance matrix calculated? It is taken over all classes, and estimated on all ID samples, right? If so, how do missing labels make it less accurate?\n - Eq. 3: Should the first line be a sum instead of a quantifier over all i,j?\n - The discussion in 2.2 works with Mahalanobis distance only in the feature space (=input image space). Previous approaches [27] use latent representations of a classifier. Could this not be applied here with a classifier trained on the limited labeled data? In 2.3.2. it is still not clear to me where intermediate representations are actually used, since a backbone is mentioned for the calculation of Sigma-hat, but it is still a matrix in image space.\n - l. 142 & Eq. 7: What are the \"detection-specific space\" and N?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have included a very adequate Broader Impact section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "pH2cl9nOBKg", "reviewer": "Reviewer_U8Ns", "summary": "This paper studies semi-supervised learning with limited label data and  unlabeled data with OOD samples. The covariance in the Mahalanobis distance cannot be estimated in a robust way when only limited labeled data is available. The main idea of the paper is to find a projection  space that will preserve the local topology of the data points. More specifically, the projection matrix should maximize the distance between a given sample and all other  samples subject to the condition that the Mahalanobis distances between this point and points within its local neighborhood are preserved. ", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "If Mahalanobis distance is not a reliable measure due to limited data size then it is not clear why the proposed approach is trying to preserve this distance in the objective function. \n\nThe Mahalanobis distance in the original space becomes the Euclidean distance in the projected space if the projection matrix is selected as the lower triangular matrix of the precision matrix. OK but how is this going to help estimate a more robust precision matrix? In order for Cholesky to work the precision matrix has to be well-conditioned, which means for every class at least d+1 samples are needed.\n\nNot sure why the OOD datasets are selected from different datasets when a subset of the classes in CIFAR-10  or CIFAR-100 could have been used as OOD. Using scene dataset as OOD in CIFAR-10/100 makes the problem unrealistically easier.\n\nWriting of the paper needs improvement. There were many grammar issues and broken sentences. \n\n---------------\nUpdate:\n\nRegrettably, the author rebuttal does not address my main concerns about the paper. I am not convinced that the main idea of the paper that aims to leverage ill-conditioned Mahalanobis distance to learn useful structural information about the data so as to more effectively perform OOD detection will be very effective in OOD detection.  I stand with my original rating of \"clear reject\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None foreseen.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "oDdcHo3zx3d", "reviewer": "Reviewer_7FHe", "summary": "The paper attempts to extend the existing OOD detection problem to a real world scenario where the in-distribution data is only partially labeled (semi-supervised learning), unlike existing OOD problem formulation where they assume that in-distribution data is fully labeled. The proposed approach, termed as STEP in the paper, extends the Mahalanobis distance based OOD approach into the given semi-supervised OOD detection problem formulation. The proposed approach is evaluated on OOD benchmark with the most recent methods proposed for addressing traditional OOD formulation. Extensive experiment shows the effectiveness of proposed method for the proposed semi-supervised OOD detection problem as compared to existing methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper is very well written, has good structure, and is easy to follow. It does a good job in proposing the problem formulation and clearly explaining how the proposed OOD detection problem formulation defers from the typically used OOD detection problem formulation.\n\n\n+ The paper focuses on Mahanalobis distance based OOD detection. Sec.2.2 points out some of the difficulties in adapting existing Mahanalobis distance based OOD detection methods to the proposed semi-supervised OOD setting. The arguments made by the paper are reasonable and the formulation for semi-supervised OOD that follows also makes sense.\n\n\n+ Based on the objective formulated to overcome the estimation inaccuracies in computing covariance matrix under semi-supervised OOD formulation. To create a robust Mahanalobis distance based OOD detector, a novel training strategy is derived from the updated formulation which utilizes a projection matrix to overcome on the limited availability of labeled data.\n\n\n+ Experiments are conducted on standard benchmark datasets used for supervised OOD formulation. The comparison with most recent OOD methods is also provided and across multiple metrics, the proposed method is able to achieve better performance. Specifically, over the regular Mahanalobis method. Furthermore, ablation analysis shows different configuration of the proposed method providing key insights into the various aspects of the proposed idea.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Though the problem formulation is novel and clearly explained, there is still an inherent assumption over the data which is not given as much attention. There is an inherent assumption in the proposed semi-supervised OOD formulation that all the unlabeled in-distribution data belongs to only same K category as that of partially labeled data. However, if we were to consider that majority of training data is unlabeled in a true sense, there is no way of knowing what categories are present. If we consider this case, the proposed formulation should be modified to include known-unknown-distribution in unlabeled data as well, in addition to unknown-unknown-distribution used only during testing.\n\n\n- A possible way to change this formulation in my suggestion is borrowing few aspects of open-set semi supervised learning problem formulation as proposed here [1]. Though, it only deals with known-unknown-distribution in unlabeled data, in my opinion, it is more realistic scenario and stays true to the practical definition of unlabeled data.\n\n\n- The paper does an excellent job in comparing with existing most recent methods proposed to solve supervised OOD detection. However, the need of the proposed method is not properly established through experiments. To fully validate why we need to follow the proposed method for semi-supervised OOD, a baseline having SSL method as first step and OOD detector as second step should be created, SSL+OOD. For example, the model should be trained with a popular SSL method first like MixMatch [2] and later the model should utilize confident predictions to create Mahanalobis based [3] OOD detector. This provides a strong baseline for proposed problem formulation. Since, it the method compared in the paper were created with the supervised OOD detection in mind, directly comparing with them is not entirely fair to those methods. The improvement over suggested baseline would provide a more accurate comparison as it is trained by considering both SSL and OOD nature of the problem individually. In my opinion, the effectiveness of the method should come from the consideration of SSL and OOD nature jointly while developing the method.\n\n\n\n[1] Yu, Qing, et al. \"Multi-task curriculum framework for open-set semi-supervised learning.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n\n[2] Berthelot, David, et al. \"MixMatch: A Holistic Approach to Semi-Supervised Learning.\" Advances in Neural Information Processing Systems 32 (2019).\n\n\n[3] Lee, Kimin, et al. \"A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\" Advances in neural information processing systems 31 (2018).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DTfOzKQ_qKy", "reviewer": "Reviewer_TQrA", "summary": "This paper proposed semi-supervised OOD detection and a novel objective formulation that learns a new representation space in which OOD samples could be separated well.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: The setting is reasonable and the solution seems valid.\n\nCons:\n1.\tFrom my part, this work seems to heavily rely on pre-trained networks by contrastive learning. I am curious to see how is the method perform compared to the pure contrastive learning-based method, such as CSI [1] and SSD[2]. Unfortunately, Table 2 contains no information about this comparison.\n2.\tThe algorithm seems to have a O(n^2) complexity, I am excited to know what is the typical training time on benchmark datasets.\n3.\tIt looks like the hyperparameters are tuned on the test set in Figure 3 and the best result is presented in the main table. Besides, I wonder how is the weight determined for the two losses? Is the weight 1 optimal one?\n4.\tIt would be better to present the results under some other detection metrics in the experiment section, such as the Mahalanobis distance, energy score.\n5.\tI am wondering how is your method compared with those baselines based on likelihood ratios under deep generative models? Moreover, some popular baselines are important to compare with from my side, such as MSP, Gram matrices[3].\n6.\tIt is a bit unclear that how is the ensemble of representations from each backbone network’s layer used in experiments? Does each baseline have such a design?\n7.\tIt is interesting to look at the in-distribution classification accuracy of your proposed method on benchmark datasets.\n\nMinor points:\n1.\tMaybe it will give the proposed work a more comprehensive evaluation if some results on different network architectures are presented, say Wideresnet?\n2.\tIt is interesting to observe whether semi-supervised methods, such Mixmatch[4] perform well for OOD detection?\n\n[1]Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, pages 11839–11852, 2020\n\n[2]Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In Proceedings of the 9th International Conference on Learning Representations, pages 1–17,2021\n\n[3]Chandramouli Shama Sastry, Sageev Oore, Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices, ICML2020\n\n[4]David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel, MixMatch: A Holistic Approach to Semi-Supervised Learning, NeurIPS 2019\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attempts to extend the existing OOD detection problem to a real world scenario where the in-distribution data is only partially labeled (semi-supervised learning), unlike existing OOD problem formulation where they assume that in-distribution data is fully labeled. The proposed approach, termed as STEP in the paper, extends the Mahalanobis distance based OOD approach into the given semi-supervised OOD detection problem formulation. The proposed approach is evaluated on OOD benchmark with the most recent methods proposed for addressing traditional OOD formulation. Extensive experiment shows the effectiveness of proposed method for the proposed semi-supervised OOD detection problem as compared to existing methods.

However, there exists some limitations as follows. 

1) Theory: the theory part is not thoroughly evaluated. 

2) Experiments: 2.1) Insufficient OOD experiments: the authors only consider CIFAR-10/100, while most OOD papers also experiment with SVHN and TinyImageNet dataset. 2.2) Baselines: how is your method compared with those baselines based on likelihood ratios under deep generative models? Moreover, some popular baselines are important to compare with, such as MSP, Gram matrices.

In the end, although the idea and problem formulation presented in the paper has sufficient contribution to make to the field, I think that this paper may not be ready for publication at NeurIPS, but the next version must be a strong paper if above limitations can be well addressed.
Decision: Accept (Poster)