How Does Unlabeled Data Provably Help Out-of-Distribution Detection?
Authors: Xuefeng Du
Zhen Fang
Ilias Diakonikolas
Yixuan Li
Venue: ICLR 2024 poster (2024)
Keywords: out-of-distribution detection
learnability
Primary area: general machine learning (i.e., none of the above)

Abstract: Using unlabeled data to regularize the machine learning models has demonstrated promise for improving safety and reliability in detecting out-of-distribution (OOD) data. Harnessing the power of unlabeled in-the-wild data is non-trivial due to the heterogeneity of both in-distribution (ID) and OOD data. This lack of a clean set of OOD samples poses significant challenges in learning an optimal OOD classifier. Currently, there is a lack of research on formally understanding how unlabeled data helps OOD detection. This paper bridges the gap by introducing a new learning framework SAL (Separate And Learn) that offers both strong theoretical guarantees and empirical effectiveness. The framework separates candidate outliers from the unlabeled data and then trains an OOD classifier using the candidate outliers and the labeled ID data. Theoretically, we provide rigorous error bounds from the lens of separability and learnability, formally justifying the two components in our algorithm. Our theory shows that SAL can separate the candidate outliers with small error rates, which leads to a generalization guarantee for the learned OOD classifier. Empirically, SAL achieves state-of-the-art performance on common benchmarks, reinforcing our theoretical insights. Code is publicly available at https://github.com/deeplearning-wisc/sal.

Published as a conference paper at ICLR 2024
How DOES UNLABELED DATA PROVABLY HELP OUT-OF-DISTRIBUTION DETECTION?
Xuefeng D Zhen Fang Ilias Diakonikolas', Yixuan Li
Department of Computer Sciences, University of Wisconsin-Maddisn 2 AAustralian Artificial Intelligence Institute, University of Technology Sydney {xEdu, il ilias, sharonl1 ecs. wisc.edu, zhen. fang@uts edu. au
ABSTRACT
Using unlabeled data to regularize the machine learning models has demon- strated promise for improving safety and reliability in detecting out-of-distribution (OOD) data. Harnessing the power of unlabeled in-the-wild data is non-trivial due to the heterogeneity of both in-distribution (ID) and OOD data. This lack of clean set of OOD samples poses significant challenges in learning OOD classifier. Currently, there is a lack of research on formally understanding how unlabeled data helps OOD detection. This paper bridges the gap by introduc- ing a new learning framework SAL (Separate And Learn) that offers both strong theoretical guarantees and empirical effectiveness. The framework separates can- didate outliers from the unlabeled data and then trains OOD classifier using the candidate outliers and the labeled data. Theoretically, we provide rigorous error bounds from the lens of separability and learnability, formally justifying the two components in our algorithm. Our theory shows that SAL can separate the candidate outliers with small error rates, which leads to a generalization guarantee for the learned OOD classifier. Empirically, SAL achieves state-of-the-aar per- formance common benchmarks, reinforcing theoretical insights. Code is publicly available at https: /ggttub...///// com /eeppeerepeeeengeeeeeeeee wassswwsc/////ee sal.
optimal
INTRODUCTION
When deploying machine learning models in real-world environments, their safety and reliability are often challenged by the occurrence of categories and should not be predicted by the model. Concerningly, neural networks are brittle and lack the necessary awareness of OOD data in the wild (Nguyen et al., 2015). Identifying OOD inputs is a vital but fundamentally challenging problem-the the models are not explicitly exposed to the unknown distribution during training, and therefore cannot capture reliable boundary between in-distribution (ID) vs. OOD data. To circumvent the challenge, researchers have started to explore training with additional data, which can facilitate a conservative and safe decision boundary against OOD data. In particular, a recent work by Katz-Samuels et al. (2022) proposed to leverage unlabeled data in the wild to regularize model training, while learning to classify labeled data. Such unla- beled wild data offer the benefits of being freely collectible upon deploying any machine learning model in its operating environment, and allow capturing the true test time OOD distribution.
(OOD) data, which arise from unknown
Despite the promise, harnessing the power of unlabeled wild data is non-trivial due to the heteroge- neous mixture of ID and OOD data. This lack of a clean set of OOD training data poses significant challenges in designing effective OOD learning algorithms. Formally, the unlabeled data can be characterized by a Huber contamination model Pwikd := (1 - т Pm + TPoan where P.n and Pou the marginal distributions of the ID and OOD data. It is important note that the learner only observes samples drawn from such mixture distributions, without knowing the clear membership of whether being ID or OOD. Currently, a formalized understanding of the problem lacking for the field. This prompts the question underlying the present work:
"Equal contributions

---

Published as a conference paper at ICLR 2024
How does unlabeled wild data provably help OOD detection?
Algorithmic contribution. In this paper, we propose new learning framework SAL (Separate And Learn), that effectively exploits the unlabeled wild data for OOD detection. At a high level, our framework SAL builds two consecutive components: (1) filtering setpring-Seppraaee candidate outliers from the unlabeled data, and (2) classiiication-earr an OOD classifier with the candidate outliers, in conjunction with the labeled ID data. To separate the candidate outliers, our key idea is to perform singular value decomposition on a gradient matrix, defined over all the unlabeled data whose gradi- ents are computed based on classification model trained on the clean labeled ID data. In the SAL framework, unlabeled wild data are considered candidate outliers when their projection onto the top singular vector exceeds a given threshold. The filtering strategy for identifying candidate outliers is theoretically supported by Theorem 1. We show in Section 3 (Remark 1) that under proper con- ditions, with a high probability, there exist some specific directions (e.g., the top singular vector direction) where the mean magnitude of the gradients for the wild outlier data is larger than that of ID data. After obtaining the outliers from the wild data, we train an OOD classifier that optimizes the classification between the IID vs. candidate outlier data for OOD detection.
Theoretical significance. Importantly, we provide new theories from the lens of separability and learnability, formally justifying the two components in our algorithm. Our main Theorem I ana- lyzes the separability of outliers from unlabeled wild data using our filtering procedure, and gives a rigorous bound the error rate. Our theory has practical implications. For example, when the size of the labeled ID data and unlabeled data is sufficiently large, Theorems and 2 imply that the error rates of filtering outliers can be bounded by a small bias proportional to the optimal ID risk, which is small value close to zero in reality (Frei et al., 2022). Based on the eπтоr rate estimation, we give generalization error of the OOD classifier in Theorem 3, to quantify its learnability on the II) data and a noisy set of candidate outliers. Under proper conditions, the generalization error of the learned OOD classifier is upper bounded by the risk associated with the optimal OOD classifier.
Empirical validation. Empirically, we show that the generalization bound w.r.t. SAL (Theorem 3) indeed translates into strong empirical performance. SAL can be broadly applicable to non-convex models such modern neural networks. We extensively evaluate SAL common OOD detection tasks and establish
performance. For completeness, we compare SAL with two fam- ilies of methods: (1) trained with only I... and (2) trained with both I',. and an unlabeled dataset. On CIFAR- 100, compared to a strong baseline KNN+ (Sun et al., 2022) using only Pm. SAL outper- forms by 44.52% (FPR95)
average. While methods such as Outlier Exposure (Hendrycks et al., 2019) require a clean set of auxiliary unlabeled data, our results are achieved without imposing any such assumption on the unlabeled data and hence offer stronger flexibility. Compared to the most related baseline WOODS (Katz-Samuel et al., 2022), our framework can reduce the FPR95 from 7.80% to 1.88% on CIFAR-100, establishing near- perfeettcct results on this challenging benchmark.
2 PROBLEM SETUP
Formally, we describe the data setup, models and losses and learning goal.
K} be the label space for ID data. Given an unknown ID joint distribution Fxy defined over X' x y, the labeled ID data Sin ((x1, v1), (X,: yn]} are drawn independently and identically from IP Fxy. We also denote Pr, as the marginal distribution of Fxy on X, which is referred to as the ID distribution.
Labeled ID data and ID distribution. Let X' be the input space, and Y' =
Out-of-dist didtriitittbionnnn detection. Our framework concerns a common real-world scenario in which the algorithm is trained on the labeled ID data, but will then be deployed in environments containing OOD data from unknown class, i.e., y $ ), and therefore should not be predicted by the model. At test time, the goal is to decide whether test-time input is from ID or not (OOD).
Unlabeled wild data. A key challenge in OOD detection is the lack of labeled OOD data. In particular, the sample space for potential OOD data can be prohibitively large, making it expensive to collect labeled OOD data. In this paper, to model the realistic environment, we incorporate unlabeled wild data Swald { X,.} into learning framework. Wild data consists of both ID and OOD data, and can be collected freely upon deploying an existing model trained on Sin the Huber contamination model to characterize the Following Katz-Samuels et al. (2022),
2

---

Published as a conference paper at ICLR 2024
marginal distribution of the wild data
P Fwild == (1 (1-xx P +
(1)
where E (0, 1] and Pou is the OOD distribution defined over X. Note that the straightforward since no novelties occur.
is
Models and losses. We denote by hw X' + R predictor for ID classification with parameter w E W. where W is the parameter space. hw returns the soft classification output. We consider the loss function & R&K y IR the labeled IDD data. In addition, we denote the OOD classifier ge X' + IR with parameter 0 E θ where e is the parameter space. We use (g( (x), yb) to denote the binary loss function w.n.f ge and binary label yb Vb == {y+: y- where y+ R>o and E R& correspond to the ID class and the OOD class, respectively.
Learning goal. Our learning framework aims to build the OOD classifier go by leveraging data from both Stn and Swald- In evaluating our model, we are interested in the following measurements:
(1) (go A) == Ex- (1 (1))) > A}).
(2)
(2) (pR( A) == Ex-e, (1 > X}),
where À is threshold, typically chosen so that high fraction of data is correctly classified.
PROPOSED METHODOLOGY
In this section, we introduce a new learning framework SAL that performs OOD detection by lever- aging the unlabeled wild data. The framework offers substantial advantages over the counterpart ap- proaches that rely only on the data, and naturally suits many applications where machine learning models are deployed in the open world. SAL has two integral components: (1) seteriing-eprratee the candidate outlier data from the unlabeled wild data (Section 3.1), and (2) classification train a binary OOD classifier with the ID data and candidate outliers (Section 3.2). In Section 4, we provide theoretical guarantees for SAL, provably justifying the two components in our method.
3.1 SEPARATING CANDIDATE OUTLIERS FROM THE WILD DATA
To separate candidate outliers from the wild mixture Swild, our framework employs level-set esti- mation based on the gradient information. The gradients are estimated from a classification predictor hw trained on the ID data S We describe the procedure formally below.
Estimating the reference gradient from ID data. To begin with, SAL estimates the reference gradients by training classifier hw on the ID data Sin by empirical risk minimization (ERM):
W E arg min Rgin (bw), where Rg (hw)
Σ
(x, y:),
(3)
weW
W is the learned parameter and n is the size of ID training set Sin
The average gradient is
Σ Ve(h (x:),
where 4 acts reference gradient that allows measuring the deviation of any other points from it.
Separate candidate outliers from the unlabeled wild data. After training the classification predic- tor on the labeled ID data, we deploy the trained predictor hw in the wild, and naturally receives and OOD data. Key our framework, we perform filtering
data Swiu- mixture of unlabeled procedure on the wild data Swid. identifying candidate outliers based on a filtering score. To define filtering score, we represent each point Swild as gradient vector, relative to the reference gradient V. Specifically, we calculate the gradient matrix (after subtracting the reference gradient v) for the wild data as follows:
Ve(b gin (x1), A
G = Vi(h (Xm)) Dr.... V
(5)
3

---

Published as a conference paper at ICLR 2024
V Vi(hwgin (8), V
wild
wild
B
Swild
(a) Visualization of the projection
(b) Angle between the gradient and V
Figure 1: (a) Visualization of the gradient vectors, and their projection onto the top singular vector (in gray dashed line). The gradients of inliers from Swan (colored in orange) are close to the origin (reference gradient
V). In contrast, the gradients of outliers from Swild (colored in purple) are farther away. (b) The angle 0 between the gradient of set Swi and the singular vector Since the projected points (cross marks) the origin (sum over all the gradients Sway), v points to direction of OOD data in the wild with a small 0. This further translates into a high filtering score т. which is essentially the norm after projecting gradient vector onto v. As a result, filtering outliers by Sr {& E Swas T, T} will approximately return the purple OOD samples in the wild data.
searched maximize the distance from
where denotes the size of the wild data, and Us is the predicted label for wild sample x. For each data point X; in Swild, we then define our filtering score as follows:
T (Ve(h Sm (xx), Dz.) v.v)?
(6)
where () > the dot product operator and is the top singular vector of G. The top singular vector be regarded as the principal component of the matrix G in Eq. 5, which maximizes the total distance from the projected gradients (onto the direction of v) to the origin (sum over all points in Swita) (Hotelling, 1933). Specifically, is unit-norm vector and be computed as follows:
Σ Ve, -x.)
V E arg max
| 8.ES,as
Essentially, the filtering score TÜ in Eq. 6 measures the l norm of the projected vector. To help readers better understand design rationale, provide illustrative example of the gradient vectors and their projections in Figure (see caption for details). Theoretically. Remark below shows that the projection of the OOD gradient vector the top singular vector of the gradient matrix G is on average provably larger than that of the ID gradient vector, which rigorously justifies our idea of using the score T for separating the ID and OOD data.
Remark 1. Theorem 4 in Appendix D.I has shown that under proper assumptions, we have sufficient data and large-size model, then with the high probability:
the mean projected magnitude of OOD gradients in the direction of the top singular vector of G can be lower bounded by a positive constant C/T;
the mean projected magnitude of ID gradients in the direction of the top singular vector is upper bounded by small value close zero.
Finally, we regard Sr {x E Swild Ti T} the (potentially noisy) candidate outlier set, where T is the filtering threshold. The threshold can be chosen on the ID data Sin so that a high fraction (e.g., 95%) of ID) samples is below it. In Section 4, we will provide formal guarantees, rigorously justifying that the set Sy returns outliers with a large probability. We discuss and compare with al- ternative gradient-basee based scores (c.g., GradNorm (Huang al., 2021)) for filtering in Section 5.2. In Appendix K, we discuss the variants using multiple singular vectors, which yield similar results.
An illustrative example of algorithm effect. To the effectiveness of our filtering score, we test on two simulations in Figure 2 (a). These simulations are constructed with simplicity in mind, to facilitate understanding. Evaluations tion In particular, the wild data is a mixture of ID (multivariate Gaussian with three classes) and OOD. We consider two scenarios of OOD distribution, with ground truth colored in purple. Figure
complex high-dimensional data will be provided in Sec-
4

---

Published as a conference paper at ICLR 2024
Wild data scenario
an
oco
25
02
Wild data scenario 2
clec
BB
(a)
Data setup
(b) Filtered outliers green)
(c) Uncertainty
Figure 2: Example of SAL two different scenarios of the unlabeled wild data. (a) Setup of the ID/inlier Stiu and OOD/outlier data Swaid in the wild. The inliers are sampled from three multivariate Gaussians. We construct two different distributions of outliers (see details in Appendix Q). (b) The filtered outliers (in green) by SAL, where the rate of filtered outliers Sy containing inlier data is 8.4% and 6.4%, respectively. (c) The density distribution thus benefits the training of the OOD classifier leveraging the filtered outlier data for binary classification.
filtering score т. which separable for inlier outlier data wild and
(b) exemplifies the outliers (in green) identified using our proposed method, which largely aligns with the ground truth. The error rate of ST containing ID data is only 8.4% and 6.4% for the two scenarios considered. Moreover, the filtering score distribution displays clear separation between the ID vs. OOD parts, as evidenced in Figure 2 (c).
Remark 2. Our filtering process can be easily extended into K-class classification. In this case, one can maintain a class-conditioal reference gradient V one for each class k E [L. K). estimated on ID data belonging to class k, which captures the characteristics for each ID class. Similarly, the top singular vector computation can also be performed in a class-conditional manner, where we replace the gradient matrix with the class-conditional Gg. containing gradient vectors of wild samples being predicted as class k.
3.2 TRAINING THE OOD CLASSIFIER WITH THE CANDIDATE OUTLIERS
After obtaining the candidate outlier set ST from the wild data, we train an OOD classifier go that optimizes for the separability between the ID vs. candidate outlier data. In particular, training objective can be viewed as explicitly optimizing the level-set based on the model output (threshold 0), where the labeled ID data from S has positive values and vice versa.
(ge) Rsa, (go) R (ge) + Rs, (ge)
EXES (go 50 + ExEST ({go(X) > 0}.
To make the 0/1 loss tractable, we replace it with the binary sigmoid loss, smooth approximation ensure ID accuracy. Notably, the
of the 0/1 loss. We train go along with the ID risk in Eq. training enables strong generalization performance for test OOD samples drawn from PPan- We provide formal guarantees on the generalization bound in Theorem 3, as well as empirical support in Section 5. A pseudo algorithm of SAL is in Appendix (see Algorithm l).
4 THEORETICAL ANALYSIS
We now provide theory to support our proposed algorithm. Our main theorems justify the two com- ponents in our algorithm. As an overview, Theorem provides a provable bound on the error rates using our filtering procedure. Based on the estimations on eтror rates, Theorem 3 gives the gener- alization bound w.r.f the empirical OOD classifier ge, learned on ID data and noisy set of outliers. We specify several mild assumptions and necessary notations for our theorems in Appendix B. Due to space limitation, we omit unimportant constants and simplify the statements our theorems. We defer the full formal statements in Appendix C. All proofs can be found in Appendices D and E.
5

---

Published as a conference paper at ICLR 2024
4.1 ANALYSIS ON SEPARABILITY
Our main theorem quantifies the separability of the outliers in the wild by using the filtering proce- dure (c.f. Section 3.1). Let ER and E be the error rate of OOD data being regarded as ID and the error rate of ID data being regarded as OOD, i.c., ERR = I& E Swidd T, S T S wild and ERR, = x E S Ti > T where S and Swild denote the sets of inliers and outliers from the wild data Swadd Then ERR and ERR,, have the following generalization bounds.
Theorem 1. (Informal). Under mild conditions, if F'hh y) is 81-smooth w.E. Pwild has (7,, didiscreppyyy w.E.t I xx (c.f. Appendices B.2, B.3), and there is E (0, 1) s.t. (1 - n)²²² 83, R > O, then when n = St(d ((d/minn(n" A, (r 1) m (dd with the probability at least 0.9, for 0 < T 0.9M (M' the upper bound of score Ti).
881
Rs + 0 O
d
+ 0 TV (1 x)m (1
(
ERR S
(9)
T
max max(d, A7 / / TT21 - π )m
ERR S 8(()) +0(v x2n + +0(
(10)
where Ri is the optimal ID risk, i.e., R min [(x,)) xx 8 Ehh (1w((x):. (x). :)),
5(T)= max (0, - /21n3/(1 - T/M),, A 0.98n)2 8,3, Rig
(11)
d the dimension of the space W, and is the OOD class-prior probubiiiyy in the wild
Practical implications of Theorem 1. The above theorem states that under mild assumptions, the errors ER and ERR, upper bounded. For ERRm. if the following two regulatory conditions hold: 1) the sizes of the labeled ID) n and wild data m are sufficiently large; 2) the optimal ID risk RE is small, then the upper bound is tight. For ERR d(T) defined in Eq. 11 becomes the main error, if we have sufficient data. To further study the main error ((T) in Eq. 10, Theorem 2 shows that the error 8(T) could be close zero under practical conditions.
Theorem 2. (Informal). 1) A (1 c)xT for small error 0, then the main error 8(T) defined in Eq. 11 satisfies that
8(T) S T/M
(12)
2) 2.011 + 1.011 T., then there exists 1) E (0, 1) ensuring that A > 0 and A > æ hold, which implies that the main error 8(T) = 0.
Practical implications of Theorem 2. Theorem states that if the discrepancy Ç between two data distributions 1iild and In, is larger than some small values, the main error 5(T') could be close to zero. Therefore, by combining with the two regulatory conditions mentioned in Theorem 1, the error ERR could be close zero. Empirically, we verify the conditions of Theorem 2 in Appendix F, which can hold true easily in practice. In addition, given fixed optimal ID risk Rú and fixed sizes of the labeled ID n and wild data m, we observe that the bound of ERRg, will increase when goes from 0 to 1. In contrast, the bound of ERR is non-monotonic when increases, which will firstly decrease and then increase. The observations align well with empirical results in Appendix
Impact of using predicted labels for the wild data. Recall in Section 3.1 that the filtering step uses the predicted labels to estimate the gradient for wild data, which is unlabeled. To analyze the impact theoretically, we show in Appendix Assumption 2 that the loss incurred by using the predicted label smaller than the loss by using any label in the label space. This property included in Appendix Lemmas 5 and to constrain the filtering score in Appendix Theorem 5 and then filtering error in Theorem 1. In harder classification cases, the predicted label deviates more from the true label for the wild ID data, which leads to a looser bound for the filtering accuracy in Theorem 1.

---

Published as a conference paper at ICLR 2024
Empirically. calculate and compare the filtering accuracy and its OOD detection result CIFAR- 10 and CIFAR- TEXTURES (Cimpoi et al., 2014) as the wild OOD). SAL achieves a result of ERR, 0.018 and ERR 0.17 CIFAR- 10 (casier classification case). which outperforms the result of F. = 0.037 and F. = 0.30 on CIFAR- 100 (harder classification case), align- ing with reasoning above. The experimental details are provided in Appendix P. Analysis of using random labels for the wild data is provided in Appendix O.
4.2 ANALYSIS ON LEARNABILITY
Leveraging the filtered outliers Sr, SAL then trains an OOD classifier with the data from in- distribution Sth and data from ST OOD. In this section, we provide the generalization error bound for the learned OOD classifier to quantify its learnability. Specifically, we show that a small error guarantee in Theorem implies that we can get a tight generalization error bound.
(x), 26). i.e., (go()x)), Wb) s Theorem 3. (Informal). Lei L be the upper bound of (s(ga (x(ga(x),30)
L. Under conditions in Theorem 1, if we further require n = (dd/miin(= 22) m = Q((d + /2) т min(T, ,i))),
then with the probability at least 0.89, for any 0 < T < 0.9M' min / the OOD
classifier 8ö, learned by SAL satisfies
1-8(T) 8(T) + 9(1 - LB
3.5L
Pa (g ) 5 min RE., (ge)
R
BE0
TTL 8(T))T
(13)
+0 max{Vd, Vd min(Tr,
VH +0
max{VV vd', /'''''47 A7}
1
T² - m)m)
min A?T"
where DT. and are shown in Theorem 1, d' is the dimension of space θ T" -I(()))), and the risk RP..7... (go) corresponds to the empirical risk in Eq. 8 with loss E,, i.e.,
(e..P..((6,,) -Ex=F, (x( U+) Ex~P,.. (go
(14)
Insights. The above theorem presents the generalization error bound of the OOD classifier SØ, learned by using the filtered OOD data Sr. When have sufficient labeled ID data and wild data, then the risk of the OOD classifier gø, is close to the optimal risk, i.c., minøge (g, if the optimal ID risk Ri is small, and either one of the conditions in Theorem 2 is satisfied.
5 EXPERIMENTS
In this section, we verify the effectiveness of our algorithm on modern neural networks. We aim to show that the generalization bound of the OOD classifier (Theorem 3) indeed translates into strong empirical performance, establishing state-of-the-ar results (Section 5.2).
5.1 EXPERIMENTAL SETUP
Datasets. We follow exactly the same experimental setup WOODS Latz--amuuls et al., 2022), which introduced the problem of learning OOD detectors with wild data. This allows us to draw
fair comparisons. WOODS considered CIFAR-10 and CIFAR- 100 (Krizhevsky et al., 2009) ID datasets (P.).. For OOD test datasets (P (Pou)) we use a suite of natural image datasets including TEXTURES (Cimpoi al., 2014), SVHN (Netzer et al., 2011), 1), PLACES365 (Zhou al., 2017), LSUN-RESIZE & LSUN-C (Yu et al., 2015). To simulate the wild data (I" (P'wia). we mix subset of ID 0.1, which reflects the practical
data (as P.,.) with the outlier dataset (as P P.n.) under the default scenario that most data would remain ID. Take SVHN as an example, we use CIFAR+SVHN as the unlabeled wild data and test on SVII as OOD, We simulate this for OOD datasets and provide analysis of differing E {0.05, 0.1, ..., 1.0} in Appendix F. Note that we split CIFAR datasets into two halves: 25, 000 images as ID training data, and the remainder 25, 000 for creating the wild mixture data. We use the weights from the penultimate layer for gradient calculation, which was shown to be the most informative for OOD detection (Huang et al., 2021). Experimental details are provided in Appendix G.
7

---

Published as a conference paper at ICLR 2024
Table 1: OOD detection performance on CIFAR-100 as ID. All methods are trained on Wide ResNet-40-2 for 100 epochs. For each dataset, we create corresponding wild mixture distribution Pwiu = (1 xT r))PPnn + x
for training and test on the corresponding OOD dataset. Values are percentages averaged over 10 runs. Bold numbers highlight the best results. Table format credit to
et al. (2022).
OOD Datasets
Methods
SVIN
PLACESS65
LSUN-C
LSUN-RRS127:
TEXTURES
Averago
ACC
FIRRSS
AUROCC
FPR95
AUROC
FPRUS
AURRC:
FPR95
AUROC
FPRO5
AUROC
FPROS
AUROC
only
MSP
84.59
71.44
82.84
13.78
66.54
K3.19
82.42
75.38
83.29
73.34
79.94
25.55
75.96
ODIN
84.66
67.26
$7.88
71.63
55.55
K7.73
71.96
81.88
79.27
73.45
75.86
76 38
75.96
Mahnlanohis
57.52
86.01
58.83
67.87
91.18
69.60
21.23
96.00
59.39
90.57
59.63
82.03
75.96
Energy
85.82
73.99
80.56
75.64
35.32
93.53
79,47
79.2
79.41
76.28
72.12
79.69
75.96
KNN
63.71
78.
ReAct
74.33
74.32
91.19
75.96
DICE
88.35
72.58
81.61
75.07
26.77
91.74
80.21
78, 50
76.29
76.07
70.65
79.39
75.96
ASH
21.36
94.28
68.3 37
71.22
15.27
95 65
68. 18
85 42
40.57
92.29
42.81
87.77
75.96
CS1
64.70
84.97
$2.25
73.63
38.10
92.52
91.55
63. 42
74.70
92.06
70.26
81.44
69.90
KNN+
32:21
93.74
68.30
75.31
40.37
86.13
44.85
88.88
45.26
87,80
46.40
86.29
73.78
With I., und
1.57
99.63
93:43
99.29
27.89
18.99
Energy (wf OE)
1,47
99.68
54.67
86.09
2.52
99.84
2.68
99.50
37.26
91.25
19:72
95.19
73.46
WOODS
0.12
99.96
29.58
90.60
0.11
99.96
0.07
99.96
9.12
96.65
7.80
97:43
75.22
SAL
0.07
99.95
3.53
99.06
0.06
99.94
0.02
99.95
5.73
98.65
1,88
99.51
73.71
*0.02
$0.00
50.17
$0.06
50.21
$0,00
$0.03
50.34
$0.02
50.11
50.00
#0.78
Evaluation metrics. We report the following metrics: (1) the false positive rate (FPR95L) of OOD samples when the true positive rate of ID samples is 95%, (2) the area under the receiver operating characteristic curve (AUROCT), and (3) ID classification Accuracy (ID ACC1).
5.2 EMPIRICAL RESULTS
SAL achieves superior empirical performance. We present results in Table 1 on CIFAR- 100, where SAL outperforms the state-of-the-an method. Our comparison covers an extensive collection of competitive OOD detection methods, which can be divided into two categories: trained with and without the wild data. For methods using II) data Ph, only, we compare with methods such as MSP (Hendrycks & Gimpel, 2017). ODIN (Liang et al., 2018), Mahalanobis distance (Lee et al., 2018b), Energy score (Liu et al., 2020b), ReAct (Sun et al., 2021), DICE (Sun & Li, 2022), KNN distance (Sun et al., 2022), and ASH (Djurisic et al., 2023)-all of which use a model trained with cross-entropy loss. We also include the method based on contrastive loss, including CSI (Tack et al., 2020) and KNN+ (Sun et al., 2022). For methods using both ID and wild data, we compare with Outlier Exposure (OE) (Hendrycks al., 2019) and energy- renrryyyegulrrzzationn learning (Liu al., 2020b), which regularize the model by producing lower confidence or higher energy on the auxiliary outlier data. Closest to ours WOODS (Katz-Samuels et al., 2022), which leverages wild data for OOD learning with a constrained optimization approach. For fair comparison, all the methods in this group are trained using the same ID and in-the-wild data, under the same mixture ratio π = 0.1.
The results demonstrate that: (1) Methods trained with both ID and wild data perform much better than those trained with only ID data. For example, on PLACES365, SAL reduces the FPR95 by 64.77% compared with KNN+, which highlights the advantage of using in-the-wild data for model regularization. (2) SAL performs even better compared to the competitive methods using Puild- On CIFAR- 100, SAL achieves average FPR95 of 1.88%, which is a 5.92% improvement from WOODS. At the same time, SAL maintains a comparable ID accuracy. The slight discrepancy is due to that our method only observes 25,000 labeled ID samples, whereas baseline methods (without using wild data) utilize the entire CIFAR training data with 50,000 samples. (3) The strong empirical performance achieved by SAL directly justifies and echoes our theoretical result in Section where we showed the algorithm has a provably small generalization error. Overall, our algorithm enjoys both theoretical guarantees and empirical effectiveness.
Comparison with GradNorm as filtering score. Huang et al. (2021) proposed directly employing the vector norm of gradients, backpropagated from the KI. divergence between the softmax output and uniform probability distribution for OOD detection. Differently, our SAL derives the filtering score by performing singular value decomposition and using the norm of the projected gradient onto the top singular vector (c.f. Section 3.1). We compare SAL with variant in Table 2, where replace the filtering score in SAL with the GradNorm score and then train the OOD classifier. The result underperforms SAL, showcasing the effectiveness of our filtering score.
Additional ablations. Due to space limitations, we defer additional experiments in the Appendix, including (1) analyzing the effect of ratio (Appendix F), (2) results on CIFAR-10 (Appendix H),
8

---

Published as a conference paper at ICLR 2024
Table 2: Comparison with using GradNorm as the filtering score. We use CIFAR- 100 as ID. All methods are trained on Wide ResNet-40-2 for 100 epochs with π. = 0.1. Bold numbers are superior results.
OOD Datssets
Filter score
SVIN
PLAC1S365
LSUN-C
LSUN-RISIZE
TEXTURIS
Average
ID ACC
FPR95
AUROC
FPRO5
AUROC
FPR95
AUROC
FPR95
AUROC
FPR295
AUROC
FPRU5
AUROC
GradNomm
1.08
62.07
84.08
0.51
99.77
98.
50.39
83.39
23.84
93.12
73,89
0.07
99.95
3.53
99,06
0.06
99.94
5.73
98.65
99.51
73.71
(3) evaluation on unseen OOD datasets (Appendix D) (4) near OOD evaluations (Appendix J, and
(5) the effect of using multiple singular vectors for calculating the filtering score (Appendix K).
6 RELATED WORK
OOD detection has attracted surge of interest in recent years (Fort et al., 2021; Yang et al., 202 lb; Fang et al., 2022; Zhu et al., 2022; Ming et al., 2022a;c; Yang et al., 2022; Wang et al., 2022b; Galil et al., 2023; Djurisic et al., 2023; Tao et al., 2023; Zheng et al., 2023; Wang et al., 2022a; 2023b; Narasimhan et al., 2023; Yang et al., 2023; Uppaal et al., 2023; Zhu et al., 2023b;a; Bai et al., 2023; Ming & Li, 2023; Zhang et al., 2023; Gu et al., 2023; Ghosal et al., 2024). One line of work per- forms OOD detection by devising scoring functions, including confidence-basse
methods (Bendale & Boult, 2016; Hendrycks & Gimpel, 2017; Liang et al., 2018), energy- based score (Liu et al., 2020b; Wang et al., 2021; Wu al., 2023), cestance-based approaches (Lee et al., 2018b; Tack et al., 2020; Ren et al., 2021; Sehwag et al., 2021; Sun et al., 2022; Du et al., 2022a; Ming et al., 2023; Ren et al., 2023), gradient-based score (Huang al., 2021), and Bayesian approaches (Gal & Ghahramani, 2016; Lakshminarayanaa et al., 2017; Maddox et al., 2019; Malinin & Gales, 2019; Wen et al., 2020; Kristiadi et al., 2020). Another line of work addressed OOD detection by training- time regularization (Bevandić et al., 2018; Malinin & Gales, 2018; Geifman & EI- Vani,, 2019; Hein et al., 2019; Meinke & Hein, 2020; Jeong & Kim, 2020; Liu et al., 2020a; van Amersfoort et al., 2020; Yang et al., 2021a; Wei et al., 2022; Du et al., 2022b; 2023; Wang et al., 2023a). For example, the model is regularized to produce lower confidence (Lee et al., 2018a; Hendrycks et al., 2019) or higher energy (Liu et al., 2020b; Du et al., 2022e; Ming et al., 2022b) on the outlier data. Most regu- larization methods assume the availability of a clean set of auxiliary OOD data. Several works (Zhou et al., 2021; Katz-Samucl, et al., 2022; He et al., 2023) relaxed this assumption by leveraging the unlabeled wild data, but did not have an explicit mechanism for filtering the outliers. Compared to positive-unlabeled learning, which learns classifiers from positive and unlabeled data (Letouzey et al., 2000; Hsieh et al., 2015; Du Plessis et al., 2015; Niu et al., 2016; Gong et al., 2018; Chapel al., 2020; Garg al., 2021; Xu & Denil, 2021; Garg et al., 2022: Zhao et al., 2022: Acharya al., 2022), the key difference is that it only considers the task of distinguishing Pout and Iin. not the task of doing classification simultaneously. Moreover, we propose a new filtering score to separate outliers from the unlabeled data, which has bounded error guarantee.
Robust statisties has systematically studied the estimation in the presence of outliers since the pio- neering work of (Tukey, 1960). Popular methods include RANSAC (Fischler & Bolles, 1981), min- imum covariance determinant (Rousseeuw & Driessen, 1999), Huberizing the loss (Owen, 2007), removal based on k-nearest neighbors (Breunig et al., 2000). More recently, there are several works that scale up the robust estimation into high-dimensions (Awasthi et al., 2014; Kothari & Steurer, 2017: Steinhardt, 2017; Diakonikolas & Kane, 2019; Diakonikolas et al., 2019a; 2022a:b). Di- score for outlier removal but they focused on the error bound for the ID classifier. Instead, we provide new theoretical guarantees on outlier filtering (Theorem and Theorem 2) and the generalization bound of OOD detection (Theorem 3).
akonikolas et al. (2019b) designed a gradient-
CONCLUSION
In this paper, we propose a novel learning framework SAL that exploits the unlabeled wn-the-wild data for OOD detection. SAL first explicitly filters the candidate outliers from the wild data using a new filtering score and then trains binary OOD classifier leveraging the filtered outliers. Theoreti- cally, SAL answers the question of how does unlabeled wild data help OOD detection by analyzing the separability of the outliers in the wild and the learnability of the OOD classifier, which pro- vide provable error guarantees for the two integral components. Empirically, SAL achieves strong performance compared to competitive baselines, echoing our theoretical insights. A broad impact statement is included in Appendix V. We hope our work will inspire future research on OOD detec- tion with un unlabeled wild data.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank Yifei Ming and Yiyou Sun for their valuable suggestions on the draft. The authors would also like to thank ICLR anonymous reviewers for their helpful feedback. Du is supported by the Jane Street Graduate Research Fellowship. Li gratefully acknowledges the support from the AFOSR Young Investigator Program under award number FA9555023-1-0188. National Science Founda- tion (NSF) Award No. IIS-2237037 & IIS-2331669 Office of Naval Research under grant number 100014-23-1-2644 Philanthropic Fund from SFF, and faculty research awards/gifts from Google and Meta.
REFERENCES
Anish Acharya, Sujay Sanghavi, Li Jing, Bhargav Bhushanam, Dhruv Choudhary, Michael Rabbat, and Inderjit Dhillon. Positive unlabeled contrastive learning. arXiv preprint arXlv:2206.01206, 2022.
Pranjal Awasthi, Maria Florina Balcan, and Philip M Long. The power of localization for efficiently learning linear separators with noise. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pp. 4499-55.400 2014.
Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both out-of-distribttion generalization and detection. In International Conference on Machine Learning, 2023.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117445))30033-30077 2020.
Abhijit Bendale and Terrance Boult. Towards open set deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1563-1572, 2016.
Petra Bevandić, Ivan Krešo, Marin Oršié, and Siniša Šegvić. Discriminative out-o--distributiin detection for semantic segmentation. arXiv preprint arXiv: 1808.0:80 2018.
Markus Breunig, Hans-PPter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-
based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference Management of data, pp. 93-104, 2000.
Lactitia Chapel, Mokhtar Z Alaya, and Gilles Gasso. Partial optimal tranport with applications on positive-unlabeled learning. Advances in Neural Information Processing Systems, 33:2903-2913, 2020.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Ilias Diakonikolas and Daniel M Kane. Recent advances in algorithmic high-dimensional robust statistics. arXiv preprint arXiv:1911.05900 2019.
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li. Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 48(2):742 864, 2019a.
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stew- art. Sever: A robust meta-algorithm for stochastic optimization. International Conference on Machine Learning, pp. 1596-1606, 2019b.
Ilias Diakonikolas, Daniel Kane, Jasper Lee, and Ankit Pensia. Outlier-robust sparse mean esti- mation for heavy-tailed distributions. Advances Neural Information Processing Systems, 35: 5164-5177, 2022a
Ilias Diakonikolas, Daniel M Kane, Ankit Pensia, and Thanasis Pittas. Streaming algorithms for high-dimg nigh-dimensional robust statistics. In International Conference on Machine Learning, pp. 5061- 5117, 2022b.
10

---

Published as a conference paper at ICLR 2024
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In International Conference on Learning Representa- tions, 2023.
Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detect- ing out-of-distribution objects. In Advances in Neural Information Processing Systems, 2022a.
Xuefeng Du, Xin Wang. Gabriel Gozum, and Yixuan Li. Unknown-awart object detection: Learning what you don't know from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022b.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't ton't know by virtual outlier synthesis. In Proceedings of the International Conference on Learming Representations, 2022c.
Xuefeng Du, Yiyou Sun, Xiaojin Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. In Advances in Neural Information Processing Systems, 2023.
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. International conference on machine learning, pp. 1386-1394, 1394, 2015.
Zhen Fang. Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribttion detection learnable? Advances in Neurul Information Processing Systems, 2022.
Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory, pp. 2668-2703, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the International Conference Machine Learning, pp. 1050-1059, 2016.
Ido Galil, Mohammed Dabbah, and Ran EI- Yaniv..v. A framework for benchmarking class-out-of- distribution detection and its application imagenet. In International Conference on Learning Representations, 2023.
Saurabh Garg, Yifan Wu, Alexander J Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and pu learning: modern approach. Advances Neural Information Processing Systems, 34:8532-8544,
2021. 2021.
Saurabh Garg, Sivaraman Balakrishnan, and Zachary Lipton. Domain adaptation under open set
label shift. Advances in Neural Information Processing Systems, 35:22531-22546, 2022.
Yonatan Geifman and Ran EI- Yaniv. Selectivenet: A deep neural network with an integrated reject option. In Proceedings of the International Conference on Machine Learning, pp. 2151-2159, 2019.
Soumya Suvra Ghosal, Yiyou Sun, and Yixuan Li. How to overcome durse-of-iimeessionaltt for ood detection? In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Tieliang Gong, Guangtao Wang, Jieping Ye, Zongben Xu, and Ming Lin. Margin based pu learning- In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Anqi Liu, Yixuan Li, Tong Sun, and Ani Nenkova. A critical analysis of out-of-distribution detection for document understanding. In EMNLP P- Findings, 2023.
11

---

Published as a conference paper at ICLR 2024
Rundong He, Rongxue Li, Zhongyi Han, Xihong Yang, and Yilong Yin. Topological structure learning for weakly supervised ousto--distriuution detection. In Proceedings of the 31st ACM International Conference Multimedia, pp. 4858 23584 2023.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 41- 50, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribuu examples in neural networks. Proceedings of the International Conference on Learning Repre- sentations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations, 2019.
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Cho-Jui Hsieh, Nagarajan Natarajan, and Inderjit Dhillon. Pu learning for matrix completion. In International conference machine learning, pp. 2445-2453, 2015.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
Taewon Jeong and Heeyoung Kim. Ood-maml: Meta-learning for few-shot out-of-distribution de- tection and classification. Advances in Neural Information Processing Systems, 33:3907-3916, 2020.
Julian Katz-Samuels Julia Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In International Conference on Machine Learning, 2022.
Pravesh Kothari and David Steurer. Outlier-robust moment-estimation via -f--ores arXiv preprint arXiv: 2017.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just bit, fixes overconfidence in relu networks. In International conference on machine learning, pp. 5436- 5446, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pp. 4413- 2017.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-caiibratee classifiers for detecting dut-o--dist samples. In Proceedings of the International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems, 31, 2018b.
Yunwen Lei and Yiming Ying. Sharper generalization bounds for learning with gradient-dominated objective functions. In International Conference Learning Representations, 2021.
Fabien Letouzey, François Denis, and Rémi Gilleron. Learning from positive and unlabeled exam- ples. In International Conference on Algorithmic Learning Theory, pp. 7 71-85. Springer, 2000.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of distributti image detection in neural networks. In Proceedings of the International Conference on Learning Representations, 2018
12

---

Published as a conference paper at ICLR 2024
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498- -512,, 2020a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribttion detec- tion. Advances in Neural Information Processing Systems, 33:21464-21477 2020b.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32:13153-13164 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in Neural Information Processing Systems, 31, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved un- certainty and adversarial robustness. In Advances in Neural Information Processing Systems, 2019.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In Proceedings of the International Conference on Learning Representations, 2020.
Yifei Ming and Yixuan Li. How does fine-tuning impact out-of-distribution detection for vision- language models? International Journal of Computer Vision, 2023.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022a.
Yifei Ming, Ying Fan, and Yixuan Li. POEM: In Proceedings of the International Conference on Machine Learning, pp. 15650-15665, 2022b.
detection with posterior sampling.
Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of-distributiin detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022c.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for dut-of-distribution detection? In Proceedings of the International Conference Learning Representations, 2023.
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, and Sanjiv Kumar. Learn- ing to reject meets ood detection: Are all abstentions created equal? arXiv preprint 2rXIv:2301.12386 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High fidence predictions for unrecognizable images. In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 427-436, 2015.
Gang Niu, Marthinus Christoffel du Plessis, Tomoya Sakai, Yao Ma, and Masashi Sugiyama. The- oretical comparisons of unlablee- learning against positive-negative learning. Advances in neural information processing systems, 29, 2016.
Art B Owen. A robust hybrid of lasso and ridge regression. Contemporary Mathematics, 443(7): 59-72, 2007.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. A simple fix to mahalanobis distance for improving near-ood detection. CoRR, abs/2106.09022, 2021.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. distriitttii detection and selective generation for conditional language models. In International Conference Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2024
Peter Rousseeuw and Katrien Van Driessen. A fast algorithm for the minimum covariance deter- minant estimator. Technometrics, 41(3):212-25 223, 1999.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. ISBN 1107057132.
Jacob Steinhardt. Does robustness imply tractability? a lower bound for planted clique semi- random model. arXiv preprint arX1v:1704.05120, 2017. 2017.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribttion detection. In Proceedings of European Conference on Computer Vision, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activa- tions. In Advances in Neural Information Processing Systems, volume 34, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributte detection with deep nearest neighbors. In Proceedings of the International Conference on Muchine Learning, pp. 20827- 20840, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in Neural Information Processing Systems, 2020.
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In Pro- ceedings of the International Conference on Learning Representations, 2023.
John Wilder Tukey. A survey of sampling from contaminated distributions. Contributions to prob- ability and statistics, pp. 448-485, 1960.
Rheeya Uppaal, Junjie Hu, and Yixuan Li. Is fine-tuning needed? pre-trained language models are near perfect for out-of-domain detection. In Annual Meeting of the Association for Computational Linguistics, 2023.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. Proceedings of the International Conference on Machine Learming, pp. 9690-9700, 2020.
Roman Vershynin. High-dimensional probability. 2018.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't 't know? Proceedings of the Advances Neural Information Processing Systems, 2021.
Qizhou Wang. Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for -ustr--disti detection. Advances in Neural Information Processing Sys- tems, 35:15555--15555 2022a.
Qizhou Wang, Zhen Fang. Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning augment distributions for out-of-distribution detection. Advances in Neural Information Processing Systems, 2023a.
Qizhou Wang. Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye HAO, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In International Conference Learning Representations, 2023b.
Yu Wang. Jingjing Zou, Jingyang Lin, Qing Ling, Yingwei Pan, Ting Yao, and Tao Mei. Out-of- distribution detection via conditional kernel independence model. In Advances in Neural Infor- mation Processing Systems, 2022b.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng. Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In Proceedings of the International Conference Machine Learning, pp. 23631-23644, 2022.
14

---

Published as a conference paper at ICLR 2024
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
Qitian Wu, Yiting Chen, Chenxiao Yang, and Junchi Yan. Energy-based out-of-distribnn detection for graph neural networks. In International Conference on Learning Representations, 2023.
Danfei Xu and Misha Denil. Positive-umlabeled reward learning. In Conference on Robot Learning. pp. 205-219, 2021.
Jingkang Yang. Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In Proceedings of the International Conference on Computer Vision, pp. 8281-8289, 2021a.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 2021b.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of- distribution detection. Advances in Neural Information Processing Systems, 35:32598-32611, 2022.
Puning Yang, Jian Liang, Jie Cao, and Ran He. Auto: Adaptive outlier optimization for online test-time ood detection. arXiv preprint S.ti::2303.1226677 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: 506. 03366,,36,, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference, 2016.
Jingyang Zhang, Jingkang Yang. Pengyun Wang, Haoqi Wang. Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306. 09301, 2023.
Yunrui Zhao, Qianqian Xu, Yangbangyan Jiang, Peisong Wen, and Qingming Huang. Dist-pu: Positive-unlabelee learning from a label distribution perspective. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14461-14470, 2022.
Haotian Zheng, Qizhou Wang, Zhen Fang, Xiaobo Xia, Feng Liu, Tongliang Liu, and Bo Han. Out-of-distribution detection learning with unreliable out-of-distribution sources. arXiv preprint arXiv:2311.03236 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4066):1452-1464 2017.
Zhi Zhou, Lan-Zhe Guo, Zhanzhan Cheng, Yu-Feng Li, and Shiliang Pu. STEP: Out-of-distribbtioo detection in the presence of limited in-distribution labeled data. In Advances in Neural Informa- tion Processing Systems, 2021.
Jianing Zhu, Hengzhuang Li, Jiangchao Yao, Tongliang Liu, Jianliang Xu, and Bo Han. Un- leashing mask: Explore the intrinsic out-of-distribution detection capability. arXiv preprint arXiv:2306.037115 2023a.
Jianing Zhu, Geng Yu, Jiangehao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for cuttoo--istriuttion detection via informative extrapolation. arXiv preprint XrXiv:2310 100 10...393.......... 2023b.
Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue', Xiang Tian, bolun zheng, and Yaowu Chen. Boosting distr---tttiiitionn detection with typical features. In Advances in Neural Information Processing Systems, 2022.
15

---

Published as a conference paper at ICLR 2024
How Does Unlabeled Data Provably Help Out-of-Distribution Detection? (Appendix)
A ALGORITHM OF SAL
We summarize our algorithm in implementation follows.
Algorithm SAL: Separate And Learn
Input: In-distribution data S {x,, y w))j-n Unlabeled wild data Swild (& K-way classification model hw and OOD classifier ga. Parameter spaces W and Ô. Learning rate Ir for ge.
Output: Learned OOD classifier SØ,"
Filtering stage
1) Perform ERM: Wya E argmin werii RsRsa (w).
2) Calculate the reference gradient as (x) Ve(hw (x;), 91).
3) Calculate gradient on Swild as Ve(hw (x:), VR.) and calculate the gradient matrix G.
4) Calculate the top singular vector of G and the score Ti = (Ve(hw (xx), Tx,) - V, v)
5) Get the candidate outliers ST = {x; E Swild; 2 T).
# Training Stage
for epoch in epochs do
6) Sample batches of data 8 Br from ID and candidate outliers ,
7) Calculate the binary classification loss Rgo BB (ga).
8) Update the parameter by 0r Ir V Rg (go)-
end
B NOTATIONS, DEFINITIONS, ASSUMPTIONS AND IMPORTANT CONSTANTS
Here summarize the important notations and constants in Tables and 4, restate necessary defi- nitions and assumptions Sections B.2 and B.3.
B. NOTATIONS
Please Table 3 for detailed notations.
B.2 DEFINITIONS
Definition 1 (8-smooth). We say a loss function E(hw (x). v) (defined over X' x 2) is B-smooth, if for any X and
[V)(h(()))) 9) T((hw(()): 4) 2)1l2 & 3l[w - w'll
Definition (Gradient-based Distribution Diserepancy). Given distributions and Q defined over X, the Gradient-based Distribution Discrepancy w.r.t. predictor hw and loss & is
dP, Q) |/ h) Ro(w,, 1))l2z
(15)
where h is classifier which returns the closest one-hot vector of hw. Ry Re h) Exwel(hw: h) and Ro(h,, h)) = Ex h).
Definition 3 (1r, discrrepan,,, We say wild distribution Pwita has (y, () disscrepancy w.r.t. ID joint distribution Pmm i > minwew Rixy (hw), and for any parameter w E W satisfying that Rexy (hw) I y should meet the following condition
de (Pin, Pwita) > S.
where Raxy (hw) T. (x.xx))>Px) E(b (x),
16

---

Published as a conference paper at ICLR 2024
Table Main notations and their descriptions.
Notation
Description
Spaces
X, y W.C θ
the input space and the label space.
the hypothesis spaces
Distributions
P Piild, P Pou
data distribution for wild data, labeled ID data and OOD data the joint data distribution for data.
IP Pxy
Data and Models
w, x, v
weight/input/hee top-1 right singular vector of G
V.r
the average gradients on labeled ID data, uncertainty score label for ID classification and binary label for OOD detection
and yb
Vx
Predicted one-hot label for input
hw and go
predictor on labeled in-distribution and binary predictor for OOD detection inliers and outliers in the wild dataset.
wild Sout
Stn Swild
labeled ID data and unlabeled wild data size of Stn size of Swild
n, m
T
the filtering threshold
ST
wild data whose uncertainty score higher than threshold T
Distances
and
the radius of the hypothesis spaces W and 6, respectively Eg norm
1|2
Loss, Risk and Predictor
(,,-), Eb(:s))
ID loss function, binary loss function the empirical risk w.r.t. predictor hw over data S the risk w.r.t. predictor hw over joint distribution Pxy
Rs(hw) (hw)
Rexy (hw)
Rem.Feu (ge)
the risk defined in Eq. 14 the error rates of regarding ID as OOD and OOD as ID
ERR... ERR....
In Section F, we empirically calculate the values of the distribution discrepancy between the ID joint distribution Pxy and the wild distribution Piita-
B.3 ASSUMPTIONS
Assumption 1.
The parameter space W C B(wo, 11) C Idd (lz ball of radius vı around wo):
The parameter space O B(Q.: r2) Rd (lg ball of radius ra around 0,);
(x), y) 2 0 and (hw (x),y) y) is 31-smooth; lo(go(x) vo) and (go (x), yb) is 32-smhoth;
sup(x,))xxx)) VC(h,, (x), 3) bj (up(x,)))XxX) Vi(g,, (x), V6)lla by:
sup(x (hwo (x). = By, su (ggo (x), yb) = B2.
For neural networks with smooth activation functions and softmax output function, can check that the norm of the second derivative of the loss functions (cross-entropy loss and sigmoid loss) is bounded given the bounded parameter space, which implies that the 3-smoothness of the loss functions can hold true. Therefore, our assumptions are reasonable in practice.
Remark 2.
Assumption 2. E(h(x), fx) S mingey ((h(x), y). where The returns the closest one-hot label of the predictor h's output
17

---

Published as a conference paper at ICLR 2024
Remark 3. The assumption means the loss incurred by using the predicted labels given by the classifier itself is smaller or equal to the loss incurred by using any label in the label space. If y Vx. the assumption is satisfied obviously. If y o Ux. then we provide two examples illustrate the validity of the assumption. For example, (1) if the loss & is the cross entropy loss, let K = 2, h(x) [h1, h22 (classification output after softmax) and hn > h2 Therefore, we have Vx 0. Suppose y L, we can get (h((x), Txx = =log((h1) < ((h(x), 4) = - log(h2). (2) If is the hinge loss for binary classification, thus we have K 1, let h(x) h1 and thus 9x Suppose y 1, we can get (x)(x), Fx) max(0, + h1) < max(0, 1 - h1) E(h(x),
B.4 CONSTANTS IN THEORY
Table 4: Constants in theory.
Constants
Description
M 8ir + bi + B1
the upper bound of loss (hw (x), y). see Proposition 1
M' -((3r + b,)
the upper bound of filtering score T
M B
a constant for simplified representation the upper bound of loss the dimensions of parameter spaces W and O, respectively the optimal ID risk, i.c., R millew E(x,))) e (hw (x), y)
T= 3ar + bar2 + B2
y6), see Proposition
d. d'
R
B(T)
the main error in Eq. 10 the discrepancy between Pin and IP (iild the ratio of OOD distribution in willd
C MAIN THEOREMS
In this section, we provide a detailed and formal version of our main theorems with a complete description of the constant terms and other additional details that are omitted in the main paper.
Theorem 1. If Assumptions and 2 hold, Pwild has (7, -)-discrepancy w.r.f. TP Pxy, and there exists n E 10 S.L. A (1 22 831 R 0, then for
M M(r + 1)d
M2d
M M(1 + 1)d
+
(7 R 22):
m = 21
T
17252
with the probability at least 9/10, for any bound of filtering score TÜ, ie.. Ti S M').
T < M (here M' 2(grr= + b1)2 is the upper
881 R +
M
d
M
d
ERR <
0
+
0
(16)
T
T
T
(1 - π mm
ERR out S S((T) + 0
M
d
I-T/M
T2n
(17)
0 max MM /d, A
1-T/M"
22 T)m
where R is the optimal ID risk, iee. R minwew E ((x,))-Px (bw((x), 1)-
8(T) = max -
A = 22 831 R
(1 (1-T/M")
(18)
M - 3r1 + biri + B1, M = MB1.
and d the dimension of the parameter space W, here 31, r1, By are given in Assumption 1.
18

---

Published as a conference paper at ICLR 2024
Theorem 2. 1) 1f 2 (1 e)xt for small enror 0, then the main error 8(T) defined in Eq.
11 satisfies that
8(T) - 1 T/M
2) 2 28, 1.0117 then there exists v E (0, 1) ensuring that A > 0 and A > hold, which implies that the main error 8(T) O.
Theorem 3. Given the same conditions in Theorem 1, if we further require that
M min AZjs m = 72 min A3
M2d + A
n n=
then with the probability at least 89/100, for any <T < 0.9M' min (1, A//=}, the OOD classi- fier Bø, learned by the proposed algorithm satisfies the following risk estimation
3.5L
8(T) + 9 L3
RR Fa (go,)
S. inf (go)) OEG
+ 1- ((T)
R
M(1 8(T))T
+0 L max m Vd' min n A
V o
m (M vdl A7
(19)
minfTT T
T² 7)m
where Ri A7. M, M,, M and d are shown in Theorem 1, d' is the dimension of space O
I. B2r ber2 + B2, T' = TT/(1 + +T);
and the risk P. (go) is defined as follows:
R (Bō Rx~F, (go(()))))4 + (o
Theorem 4, Given the same conditions in Theorem 1, with the probability at least 9/10, 10,
/ 3 + O(B
d
Bx Ti 583, +0
m1- )
0.98n?²²
83, Rin
Ex Seat T5
é é(n,, m),
π furthermore, the realizability assumption for ID distribution holds Shalev-Shwarts & Ben-David, 2014; Fang et al., 2022), then
Ti < 0 V 9 + 0(3 V (1- ④ m)
d
Eg Six Sill
0.98n
E mil
E
e' (n,
π
where
e'nn, m) < 0 ( -
3
and R D7 M and d are shown in Theorem 1.
19

---

Published as a conference paper at ICLR 2024
D PROOFS OF MAIN THEOREMS
D.1 PROOF OF THEOREM 1
Step With the probability at least - -6 > 0,
St Ti 5831
Mr (3 + bi)d
Mri + bi)d
+4.3
Cv
+ +CV (1 (1- v (6/6//
7
2 2l08(6/0)
21og(6/0)
3M
M
n
(1 (1-z)) x.m m /mlo0(6/0)/2
This can be proven by Lemma 7 and following inequality
S and
S Ss
Ve(h (X:), liwgia (xx)) - E (x S Ve(h (x)), Bb) 1
Step 2. It is easy to check that
Sild
Sout Swita
Ex Ti
St To +
S T2.
Swilaa
Step 3. Let
(rr + +G
Mni + bi)d
c),m) 1
n
(1 (1- (1-T)m V
2 log(6/0)
2 2log(6/5)
+3M
M
n
(1- (1-T)m /mlog(6/8))
Under the condition in Theorem 5, with the probability at least
100 3 > 0,
97
Ex T3
m
28
S miil
83, R
|smial m
e(n, m)]
S Sout
100
m
25
m
8,, Rm
c(n, m)l
wild
100
V log 6/0
98
83, - -)(,
22 π 16g(6/8)
100
In this proof, we set
/log
98
A(n, =
88, R. - c(n, m)].
72 /2m + x log((6/0
100
Note that A" = 0.98m)222 - -,31 Ri then
1 1 - A E(nn m) -eem))+++ + m)))
A(n, m)
where <(m) = Vlog 6/5/ ( 22m
2m + x /lo((6/0)).
Step 4. Under the condition in Theorem 5, with the probability at least 97 - 36 0,
100
x
Sout
T; S T)
min (1, A(n, m))
<
(20)
|
1 1 I-TM
wild
20

---

Published as a conference paper at ICLR 2024
and
{xx E S wild 1 > TH
8,2 Rig c(n, m)
(21)
wilal
T
We prove this step: let Z be the uniform random variable with wwild as its support and Z =
T (2(3rr b2)²), then by the Markov inequality, we have
{xg wild 7. T)
+ 61)))) 2 Ann, m)) - (2 b1)
wiid
(2 + b2))
(22)
Let Z be the uniform random variable with Wiid as its support and Z(i) = Ts. then by the Markov inequality, we have
{xx
Sin
:7 T)
B
831 R + c(n, m)
P(Z()) 7) <
(23)
Sin
Step 5. If 1 47/(1 - c/M",, then with the probability at least
3 0.
100
{
wild T4 T
+ M' (n, m)
(24)
wiill
M' T
and
{x: E Sin T > TH
831 Rijn
En(n, m)
(25)
S
T
where e' e'(n, m) = E(n,, m)/ π 4 ((n))(n,, m).
Step 6. If we set = 3/100, then it is easy to see that
<(m) S π2 Tm
e(n,m) s 0 V - M V (1- T)m
é' ((n, s 0 3M
/ Va, + (I 22 m
π
Step By results in Steps 4, and 6, We complete this proof.
21

---

Published as a conference paper at ICLR 2024
D.2 PROOF OF THEOREM 2
The first result is trivial. Hence, we omit it. We mainly focus on the second result in this theorem. In this proof, then
83 + 0.997 88 + 883 + +T)
Note that it is easy check that
< 2.011 / R3 + 1.011TT R + 1.011 1..111 + π.
Therefore,
9 v0.98
18 0.99T
88,3 +œ,
which implies that D7 > n. Note that
- (1-n)< 2 v0.98 (0.98/881117
83, +
83, R2 + 0.99T)
183112
which implies that > 0. We have completed this proof.
D.3 PROOF OF THEOREM 3
Let
0" E arg min F.r (0). 0-e
Then by Lemma and Lemma 14, we obtain that with the high probability
Ri,, (gø,)
Re (go.)
=Rr, P.. (go,)
Rsa,,, (0,
s, (go,) Rs
+Bg Sr (ga-)
RR (ga-)
SRr. P... (gă))
Rg ST (go,)
+Bg Sp (ga-)
Re (ga-)
$2 sup R (Bo)
R (go)
00e
sup (R (go) R (go)) oee
sup OE
(R (go) - Rggm (go)
3.5L
((T) + 9(1 T) LB1
T(1 5(T))T t
8(T)
max [3, . Vd, vd' + T)
min n. ATT
L 13 M MVO, vd, (1+
min k, A/TT
22
T
D.4 PROOF OF THEOREM 4
The result is induced by the Steps 1, 3 and 6 in Proof of Theorem 1 (see section D.1).
22

---

Published as a conference paper at ICLR 2024
E NECESSARY LEMMAS, PROPOSITIONS AND THEOREMS
E.1 BOUNDEDNESS
Proposition 1. If Asssupttin holds,
(x), S Bir +4 = /M12
sup sup
wEW (x.))xxx
sup sup a (x.))XX))
VV((go 1b)l2 S 3222 + b2-
sup sup WEW (x
f(hw (x), - Bir +brr + B1 = M,
sup sup OEB (x.)))XXX)
(g0((x), vb) S 82r2 + b2r2 + B2 = L.
Proof. One prove this Mean Value Theorem of Integrals easily.
Proposition 2. If Assumption holds, for any W,
[[ (x). v) 11 < 28, (x),
Proof. The details of the self-bounding property can be found in Appendix of Lei & Ying (2021).
Proposition If Assumption holds, for any labeled data S and distribution P,
V |1 < 2 Rs(h)ss))))
Vw E W,
VR= (hw )1 Ro VW W.
Proof. Jensen's 's inequality implies that Rs (hw) and R (hw) are Smosmo. Then Proposition implies the results.
23

---

Published as a conference paper at ICLR 2024
E.2 CONVERGENCE
Lemma 1 (Uniform Convergence-l). If Assumplion 1 holds, then for any distribution P, with the probability least - 0, for any E W,
2log + -CN
Mr (811 bi)d
RRs(w) Re M.
n
n
where 1S), M Birī biri B1, d is the dimension W, and C uniform constant.
Proof of Lemma I. Let
X B(x,)) (x), [
Then it clear that
Es-pn Xnw = 0.
By Proposition 2.6.1 and Lemma 2.6.8 in Vershynin (2018),
cn vn
N Nn ll, S
[1((hw(x)) (x). y) f(hw ())l)))))((xx)))))
where ||| is the sub-gaussian norm and co is a uniform constant. Therefore, by Dudley's entropy
integral (Vershynin, 2018), we have
bp Vn
too
ES-r sup X S weW
Vlog((FF 1 Je,
where by a uniform constant, F= = (((hwix,)) number under the 0 norm. Note that
w E W), and NFF 22 is the covering
S bu +20
ESnPa sup Xnn wEW
Vlog.N(F,e,Loo0) Lo
Vn J0
bn M
Vn (IogNNFF.C,Lo)
bn M
S/' (F, )x)
Vn
Then, we use the McDiarmid's Inequality, then with the probability at least e- > 0, for any w E W.
bu Vn
M ('
21
X
Me, L )e M
Similarly, we can also prove that with the probability at least 1 - c +t > 0, for any w E W,
by
M L''' VlogN(FF Me, M
2t
Vni
Therefore, with the probability at least 2e- 0. for any W,
bo M /'' VlogN(F Me, Lo M
2t
|m
Vn
Note that (hh (x), 8) is (3 + b1) -1))-iipcchit w.rt. variables w under the || 1l2 norm. Then
+ +b),,l 112) s (1 2r1 + b1)
N(F, Vc, 1
Me
which implies that
|0 v TJ,(N(F, Me, Lo sva (/ V llog(1 2r (3ir + bi).
)de
Me
<vā /' / 12 + b,)
2r 2r +b,) M
de 2/
Me
We have completed this proof.
24

---

Published as a conference paper at ICLR 2024
Lemma 2 (Uniform Convergence-Il) II). If Assumption 1 holds, then for any distribution P, with the probability at least - 8 0,
1og(22
M(r1 1)d
(Rs(hw) (Rs(hw))l2 - BV
+ CV
n
n
where n = (S), d is the dimension of W, and C a uniform constant.
Proof of Lemma 2. Denote {(v, (x), (x), 3) = (V)(h(()).))) v) by the loss function over param-
eter space W x {v = E R.). Let b is the upper bound of ε(v, (x), Using the same techniques used Lemma I, we can prove that with the probability at least w E W and any unit vector v RR
for any
210g(2/0)
b(ri + 1)
(Rs((hw) - (Rr(hw), v) S <b/
+ CV
n
which implies that
2 21og(2/0)
b(ri 1)3,d
(hw - (Rr(hw)ll 111 1 <b/
+ CV
Note that Proposition implies that Proposition 2 implies that We have completed this proof.
631 S M.
b S M
Lemma 3. Let Shee
Swild be the samples drawn from in. With the probability at least 1-6 6 >> 0.
(2/(2/0)
Sth Waall//SSun - (1 T) -
2
which implies that
log(2/5){Swol
||S (1 - SSiut <
Proof of Lemma 3. Let X; be the random variable corresponding to the case whether i-th data in wild data drawn from Pin. i.e., X, = 1, if i-th data is drawn from Pin: otherwise, X; 0. Applying Hoeffding's inequality, we can get that with the probability at least 1 - 8 > O.
1og(2/8)
|| (tall/// //wal (1 -²]
(1-=)]= s V 2|Swatll
(26)
25

---

Published as a conference paper at ICLR 2024
E.3 NECESSARY LEMMAS AND THEOREMS FOR THEOREM
Lemma 4. With the probability at least 1-5 > O, the ERM oplimizer Wi is the minwew Rp(hw) 0(1/ (i)risk point, i.e.,
log (o/()
Rs (hws)
S
min Ro (hw)) + M wEW
2n
where n = -S].
Proof of Lemma 4. Let w E arg min weW Rp Applying Hoeffding's inequality, we obtain that with the probability least 1 - 8 >> 0,
log(1/6)
Rs(hws)
min Rp(ww) S Rs(hww) Ri (hww) S MV WEW
2n
Lemma 5. If Assumptionss 1 and 2 hold, then for any data S ~ p and S' ~ p" with the proba- bility at least 1-6 8 O,
Mri (Biri bbjd
Mri(Biri bi)d
Rs (hws) S min Rp(hw) + CV WEW
+ +CV
n
n
2 2log(6/8)
2log(6/0)
+ VM
+ M
n'
Mr (31m + b.dd + 2M
2 (og(6//))
Ry(bws In) Rr(wws) min Rr (hw) + OV wEW
n
n
where C. a uniform constant, and
R3 (hww h)) = E. (xx.))pp(hwe. (x), n(x)).
Proof of Lemma 5. Let Ws' E arg min Rs (hw) and w E arg min Rr (Bphh-)). By Lemma 1 and Hoeffding Inequality, obtain that with the probability at least 1 - 0,
Rs (bws) R
<R (wws) - Rp(bwa) + Re Re Rs(wws) + Rs(w")
- Re(w")
Mr (Bir1 bi)d
Mr (rrr bj)d + 2M
2log(6/8) + MM
2 (og(6//))
sc
+ CV
n
n
n
n'
Re (Rwws)) Rpphww SRe (hws)) Rs(hws) + RR(w") Rp(w")
Mr + b,jd
21og(6/88
<C1
+ -2M
Lemma 6. If Assumptionss 1 and 2 hold, then for any data S ~ p and S' ~ p"', with the proba-
bility at least 1 28 > O,
E [x Ve(h
Rs (h 11 < 8B, min Re (hw))
wEW
Mri (8ir + br)d
Mri(Sin + b
+
C
n²
2 2log(6/0)
2 2lo((6/0)
+3M
+ M
n'
where C. is uniform constant.
26

---

Published as a conference paper at ICLR 2024
Proof of Lemma 6. By Propositions 2, 3 and Lemmas 4 and 5, with the probability at least 1-28 28
0,
(( Ve (x), hx(x)xxx))))))))))) - V RRs(ww, 112
<2B ().))/)) Ve(hws (x), h(x)))l1 2 RR (bws) )1
<481 (lsr:hww:/ h)) + Rs (hws))
- 4,31 (Rs (hws) ) + Rs (Rww)))
Mri (Birz + bi)d
Mri (3rm + bn )d
<4, (2 min wEW
(h + CV
+ CV
n
2 2lo((6/5)
(lg(6/8))
+ 3MV
M
n
Lemma 7. Let Shn C Swild be samples drawn from Pn- If Assumptions and 2 hold, then for any data Swild ~ Tpm wild and S ~ pn with the probability at least 3
Ex Vi(Wws (x), h(xx)) ( )
881
min Rlwhw) WEW
Mr (3im + bi
Mri(Bir + bj)d
+48 Ov
+
V
x x)m v (og(6/8)//
2 llgg6//8) M
2 2log(6/0)
+ 3M
(1 - nm - / log(g/////// 6/00/2
where C a uniform constant.
Proof of Lemma 7, Lemma and Lemma 6 imply this result.
Lemma 8. If Assumptions and hold, then for any data S Ln with the probability at least 1 8 0,
1og(1/8)
[s -(x)) (hws )1( $ 83, min Re + 4M, WEW
2n
Proof of Lemma 8. With the probability at least - 8> 0,
V VRs (x), fi(x)) Rs hw 112
V Rs (bws (x), li(x)) ||| V I&s (bws) ll2
V/2 28, Rs(hws (x), h(x)) + v 2/3g R.s (hws)
log( (og(1/5)
/2 28 min Re (bw) M V
wEW
2n
27

---

Published as a conference paper at ICLR 2024
Theorem 5. If Assumptions I and 2 hold and there exists r (0, 1) such that A (1 -n)? 831 min Re, (bw) > 0, when
M (r1 + 1)d M2d
M + M ( + 1)d
m 0
AT²
(r R2)
172
with the probability at least 97/100,
Ex
100
Proof of Theorem 5. Claim 1. With the probability at least 1
20 > 0, for any E W,
(2/6)
2 (2/5)
dw (Fim Pwnta) d'w (S%, Swild) - BV
+ BV
M(ri + 1)d
M(ri + 1)d
CV
+ +CV
where B = M, S is the feature part of S and C' is a uniform constant.
We prove this Claim: by Lemma is notable that with the probability least 1 1- 26 0,
& (Pin in: Pwit)) d (S) (Siss
V Rp, (hw, h) Re (hw, h) |1 VRsy (hw, h) RSs (hw hi]l2
V Re. (hw, h)
Rp (hw, h) V VRsg (hw, h) + V VRsua (hw: 1i]l12
V Re. (hw, h)
V Rg (hw, h)
VR (hw. h)
VRs (hw: 1))
2 220((2/8)
2 222((2/5) B
B(ri 1)3
B(n 1)8rd
<BV
+ CC
+ +C1
n
m
n
m
22g(
2 log llog2/00 (2/8) B
M(ri 1)d
M(ri + 1)d
SBV
+ C
+ +C
m
n
171
Claim 2. When
Vn = 2 M Mvd+ M /loo(66/0)
(27)
min Rr. (hw)
with the probability at least -48 0,
2log(2/8)
log 12g(22/8
Ex To 2 (s- BB
B
n
m
-CV M(ri + l)d
M(r + 1)d
log (1/8)
C
2/ 233
(in
Rr(hw) My
wEW
2n
We prove this Claim: let v* be the top-1 right singular vector computed in our algorithm, and
V. E arg max (E(r..))~ Ve(bw (xx), vc)
ER.ES.n Ve(hwgg (Kt,) h(xx))), v).
28

---

Published as a conference paper at ICLR 2024
Then with the probability at least 46 0,
Sunt To
Swild ((VC(hw sm (Rx), h(x;)) (x),,,))) S Ve(hw (x)), v*) ))
Ex, Swild ((Vé(bi si (x,), li(x))) (x SS Ve gi (x)), [x))) +)) 2
(5x Ve(w gi (xx), Vj) Suu Ve(ww su (x:): h(x;)(())) v)
(1x Ve(hw sin (x)), 43)) Exa V (6(ww (R:), h(xx)) l
(dw sin (Sxx Swina)
[|B(2),00) /-u Ve(hw (xi). - (xxx S Ve(hw (xx). 1l2)
(5 (-BB BV 2log(2/0)
-B 2log(2/8)
M + M 1)d
log log(1/0),
CV
2/
23r
(min Rp M WEW
m
2n
In above inequality, have used the results in Claim I, Assumption 2, Lemma 5 and Lemma 8.
Claim 3. Given = 1/100, then when
n = 0 M + M(1 1)d
M + M(1 1)d
m = 2
ArZ
r)²²2
the following inequality holds:
(c BV
2 1og(2/))
BV
2 2log(2/6)
CV.
M(r 1)d
CV
M(1 + 1)d
n
m
n
m
1og(1/0)
98
-2 2/1 (min Rr(hw) + MV wEW
2n
100
We prove this Claim: when
64
/4vlog
log 10
n
2
A
it is easy to check that
log log(1/8)
(1 (1- 2 2/22 28
min Rp (p(h)) + M wEW
2n
Additionally, when
200 log 200B² r2²c²
20 M(1 + 1)d
n >
201222
it is easy to check that
7 100
2 log (200)
M(r l)d
> B
ON
n
Because
200 log 2
200 M 1)d 64 64.1ogg(00)) (10)31 M
M + +M(ri + =1dd..
max
S s0(("
20
An²
conclude that when
n = Ω M + M(1 + 1)d
An?
29

---

Published as a conference paper at ICLR 2024
(18()
2 (2g0)200)
M(ri + 1)d
99
17 2 23 min (p + M wEW
)-BV
+CV
100 7?.. (28)
2n
When
200 log 200B rf5²
20 + 1jd
2np
we have
2 log (200)
M(1 1)d
Bv
CV
100
m
Therefore, if
M + M(1 + 11dd
m 92(
n/²22
we have
7
2 (200)
M(r + l)d
Combining inequalities 27, 28 and 29, complete this proof. CV
100 BV
(29)
m
E.4
NECESSARY LEMMAS FOR THEOREM 3
Let
R5, (go) = Ex (go())) y-), S (go) = E Ex- (x) V-),
Ht = Ex Subb (go (x), 8+), R (ge) = (x
and
RT (ga)
Let
Squ {8, E
S
Cout
wild
X < T},
S = {x, E S
wild
x; T),
Soul
{X; E wild
x; T)
S { E wild : S 7).
Then
ST S s
Sour U S
wild
Let
1 1- m-mi(11,77/m
B.Mvd
A(n,m) =
1 1- T//M'
1- T/M' m2n
M/M V1- -
+0
1 T-T/M'
x)m
8(n, m)
8/1 Ri
0
MV T
VH 0
1
T
Lemma 9, Under the conditions of Theorem 1, with the probability at least 9/10,
[Srl < |
+ |Spur
< 8(m, m) Sii + +SSuur
[Sr]
S
(nn, m)] Sour
S
(n, m)) Swadd
30

---

Published as a conference paper at ICLR 2024
Proof of Lemma 9. is conclusion of Theorem 1.
Lemma 10. Under the conditions of Theorem 1, with the probability at least 9/10, - 8(n, m)) wild
1
A (n, m)
(ō,n, m)) 5 wital wild
wild
Sr
S
[1
(n,, m) Silld wild
Proof of Lemma 10. This can be conclude by Lemma 9 directly.
Lemma 11. Under the conditions of Theorem 1, with the probability least 9/10, ( --
L)(n,m)
Lö(n,m)
1 π
Rg, (go) gis (go)
+0( m'mm
[] ((n,m)
LA(n, m)
Big (go)
RS (go) S
LL(n,
A(n, m)]
Proof of Lemma 11. It is clear that
S
S |ST
Rs, (ge) R gun (ge)
gur go
su (go)
[ST
Soutt
|
R Son (go)
Re (go)
Sour
woid
LA (ni m)
L8 (n, m
wild
1 (nn, m)]
[ A (nn, m))
Soul wild
LA (n,m) m)
L6 (n. m)
1 1-
1
+ +0(
1
(nn, m)
A (n, m)]
π
S
Sin
R Son (go)
R,s (go)
R Son (go)
R (go)
Sr
[
Sout
S
+
n R5u (go)
+
R
(go)
Sout
Sout
wild
LA (n, m)
LA (n,
(nn, m)]
Lemma 12. Let A(T) = 11 -8T)). Under the conditions of Theorem for any nf > 0, when
Mad
dm + A7 x)
n= ? 12 1/M1-)//)//
m = Ω
7/2 2 T) (
with the probability at least 9/10.
< LA(n, m) + (1- A')((T)
L8(n, m)
( (-- ग + +0(
1
Rg, (ge) - -R (ge) will
(1 AT)
Tmm
1.2L
Rig (ga) - -R (ga) < 1 5(T))
5(T) + LŌ(T)
+0( LBM + T)
[ / + +0
I(3 A +T)
1
min( A/TT
min( AT]T
T2 7)
31

---

Published as a conference paper at ICLR 2024
Proof of Lemma 12. This can be concluded by Lemma 11 and by the fact that (1- n1) (n, m) directly.
Lemma 13. Ler (T) = -(()). Under the conditions of Theorem when
Mdd
2 + A 12211 - min nT, App
Ω
mi, ATp3
Ω
with the probability at least 9/10, for any < T < 0.9M' min(1, A
1.2L
9L TT) R
Rs, (ge)
-Rgan (go) -
0(T)
kill
1 - ((T)
TT(1 - -8(T))
L3,Mvd(1+7
[ + +0 L(BIMVd+ 40)
+ 0
min( ATT
min(T, A
7 - (1--T)m Tmm
1.2L
R Rga (ge)
Rsg (ga)
8(T) + LŌ(T)
1 ((T)
L3,Mvd(1+T +
min( A (=))
( + o 11)0
47
+
min( A
T mm
Proof of Lemma 13. Using Lemma 13 with 17 8/9, obtain that
Rs, (go)
R guu (go)
T.2
( T)
LE(N)
IE(n)
Le(m)
R +
+
8(T)
TT(1 8(T))
ATT)
TT(T)
A(T)
RL T2 *2A(TTT
+ Lā(m)
Lā(m)
Lε(n)
1
0
O
AT((T) 0
m
*²A(T)
TA(T)
where
e(n) = 0 3IMVd
<(m) (
31MVV+ -I
T/M
V n2n
1 1-T/M-
22
E(n) ( 31 B.
E(m) 0
31 MM
1
T
T
n
Using the condition that 0 < T <
0.9M' min (1, A//T], /.. we have
T ++1
T/M' 50
A(T)
min (1, ///
Then, we obtain that
1.2L
8(T) + 91 - T)
Rey (go) Rosm (go) < 1- (-(T))
TT(1 ((T)) P
0
LB L +T)
min( ATTT V 0
L(BIMV (1) +T) min(T, ATJT
72 z)m
Using the similar strategy, we can obtain that
1.2L.
R Rig. (go) R5, (go) S 1 - (T) 8(T) + LS(T)
0 L L3, + T)
V=) + +0 + L(BI 42) (T))
minf 4.TT
min( A
m
32

---

Published as a conference paper at ICLR 2024
Lemma 14. Under the conditions of Theorem 1, when
M
Mad + A T2 π) min m ,., 2332
[T, ANj2
with the probability at least 0.895, for any
T < 0.9M' min 11, /}
1.2L
9L (1 -
Rs, (go) RP.. (go) - 1-ō(T) 8(T)
TT(1 8(T)) Ri
+0 L V d(1 TT)
min T, 17TT V /1))
L mmxx ( Vd, Vd, +((++
1
min [T, 4/TT
T2 27 m7
1.2L
RT (go) Rs, (go) v 1 6(T)
8(T) + L((T)
+ L3, T)
! o
L mmxx (3: V vT, ( +
1
min AUTT
min (1, A
n T² π m
Proof. By Lemmas and 3, under the condition of this lemma, we can obtain that with the high probability,
d'
R (go)
Sun (go) S O
.m
Then by Lemma 13, we can prove this lemma.
F EMPIRICAL VERIFICATION ON THE MAIN THEOREMS
Verification on the regulatory conditions. In Table 5, we provide empirical verification on whether the distribution discrepancy ( satisfies the necessary regulatory condition in Theorem 2, i.e., < 2 2.011 / (8, R. 1.011 R. We use CIFAR- 100 as ID and TEXTURES as the wild OOD data.
Since R the optimal ID risk, i.e., R = min (x.x)) (h (x), y). it can be a small value close in over-parametrized neural networks (Frei al., 2022: Bartlett et al., 2020). There- fore, omit the value of 2.011 / /83, 1 R The empirical result shows that easily satisfy the regulatory condition in Theorem 2, which means our bound is useful in practice.
Table 5: Discrepancy value Ç with different ratios n.
0.05
0.1
0.2
0.5
0.7
0.9
1.0
C
0.91
1.09
1.43
2.49
3.16
3,86
4.18
1
0.23
0.32
0.45
0.71
0.84
0.96
1.01
Verification on the filtering errors and OOD detection results with varying т. In Table 6, we empirically verify the value of ERR and ERR,, in Theorem and the corresponding OOD detection results with various mixing ratios ग. We use CIFAR-100 as ID and TEXTURES as the wild OOD data. The result aligns well with observation of the bounds presented in Section 4.1 of the main paper.
G ADDITIONAL EXPERIMENTAL DETAILS
Dataset details. For Table 1, following WOODS (Katz-Samuch et al., 2022), we split the data follows: We use 70% of the OOD datasets (including TEXTURES, PLACES365, LSUN-RESIZE
33

---

Published as a conference paper at ICLR 2024
Table The values ERRm.. ERR and OOD detection results with various mixing ratios n.
0.05
0.1
0.2
0.5
0.7
0.9
1.0
ERR
0.37
0.30
0.22
0.20
0.23
0.26
0.29
ERR
0.031
0.037
0.045
0.047
0.047
0.048
0.048
FPR95
5.77
5.73
5.71
5.64
5.79
5.88
5.92
and LSUN-C) for the OOD data in the wild. We use the remaining samples for testing- resting-time OOD detection. For SVHN, we use the training set for the OOD data in the wild and use the test set for evaluation.
Training details. Following WOODS (Katz-Samuels et al., 2022), we use Wide ResNet (Zagoruyko & Komodakis, 2016) with 40 layers and widen factor of 2 for the classification model hw. We train the ID classifier hw using stochastic gradient descent with a momentum of 0.9, weight decay of 0.0005, and an initial learning rate of 0.1. We train for 100 epochs using cosine learning rate decay, a batch size of 128, and a dropout rate of 0.3. For the OOD classifier go. we load the pre-trained ID classifier of hw and add an additional linear layer which takes in the penultinate-layer features for binary classification. We set the initial learning rate to 0.001 and fine-tune for 100 epochs by Eq. 8. We add the binary classification loss to the Il) classification loss and set the loss weight for binary classification to 10. The other details are kept the same as training hw-
H ADDITIONAL RESULTS ON CIFAR- 10
In Table 7, we compare our SAL with baselines with the ID data to be CIFAR-10, where the strong performance of SAL still holds.
Table 7: OOD detection performance on CIFAR-10 as ID. All methods are trained on Wide ResNet-40-2 for 100 epochs with . = 0.1. For each dataset, we create corresponding wild mixture distribution Pwai ==
(1 - Pm + Pour for training and test on the corresponding OOD datasct. Values are percentages averaged over 10 runs. Bold numbers highlight the best results. Table format credit (Katz-Samucls al., 2022).
OOD Dalanels
Methods
SYEN
PLACES365
LSUX-C
LSUN- GUN-RESIZE
TENTURES
Average
ID
FPRR55
AUROC
FTR95
FPR95
AUROC
With only
MSP
48.49
91.89
39.48
88.30
30.50
95.65
$2.15
91.37
59.28
88.50
50.04
91.12
94. 94.84
ODIN
33.35
91.96
57.40
84.49
15.52
97.04
26.62
94.57
49.12
84.97
36.40
90.61
94,
Mahalanobis
12.89
97.62
68.57
84.61
39.22
93.23
15.00
97.33
93.34
Energy
35.59
8.26
94.24
24.53
95.96
94.71
50.90
ReAct
40.76
89.57
41.44
90.44
14.38
97.21
33.63
93.58
53.63
86.59
36.77
91 .48
94.84
DICE
35.44
89.65
46.83
$6.69
6.32
98.68
23. 23.93
93.56
53.62
82.20
34,23
90.16
94,84
ASII
6.51
98.65
48.45
88.34
0.50
99.73
98.
21.34
95.09
17.05
96.15
94.81
2.99
2.95
9.65
With and Full
OB
0.85
99.82
23,47
94.62
1.84
99.65
0.33
99.93
10:42
98.01
7.38
98.41
94.07
Energy (w/ OE)
4.95
98.92
17.25
95.84
1,93
98.83
13.43
8.52
97.95
94.81
WOODS
97.00
99.99
99.14
0.02
0.07
99.78
(Ours)
$0.00
20.00
#0.03
90.00
$0.01
$0.00
$0.00
+0.00
$0.02
$0.01
=0.01
=0.00
$0.57
I ADDITIONAL RESULTS ON UNSEEN OOD DATASETS
In Table 8, we evaluate SAL on unseen OOD datasets, which are different from the OOD data we use in the wild. Here we consistently use 300K RANDOM IMAGES as the unlabeled wild dataset and CIFAR- 10 as labeled in-distribution data. We use the different OOD datasets (TEXTURES, PLACES365, RESIZE, SVIIN and LSUN-C) for evaluation. When evaluating on 300K RAN- DOM IMAGES, we use 99% of the 300K RANDOM IMAGES dataset (Hendrycks et al., 2019) as the wild OOD data and the remaining 1 % of the dataset for evaluation. и is set to 0.1. We observe that SAL can perform competitively on unseen datasets as well, compared to the most relevant baseline WOODS.
34

---

Published as a conference paper at ICLR 2024
Table 8: Evaluation on unseen OOD datasets. use CIFAR-10 as and 300K RANDOM IMAGES as wild data, All methods are trained on Wide ResNet-40-2 for 50 epochs. Bold numbers highlight the best results.
Methods
SVHN
PLACES365
LSUN-C
LSUN RESIZE
TEXTURES
300K RAND. IMG.
ID ACC
EPR95
AUROC
FPR95
AUROC
FFR95
AUROC
EPR95
AUROC
FPR95
AUROC
FVR95
AUROC
DF WOODS
13.18
97.34
30.54
93.31
5.87
98.86
14.32
97.44
25.60
94.35
30.69
92.80
94.21
23.74
94.72
24.59
94.54
19.14
94.74
4.94
14.76
96.25
2.73
98.23
98.15
11,60
97.21
10.20
97.23
Following (He et al., 2023), we use the CIFAR-100 100 as ID, Tiny ImageNet-crop (TINe)/Tiny ImageNet-resize (TINr) dataset as the OOD in the wild dataset and TIN:/TINc the test OOD. The comparison with basclines is shown below, where the strong performance of SAL still holds.
Table 9: Additional results on unseen OOD datasets with CIFAR-100 as ID. Bold numbers are superior results.
OOD Datasets
Methods
TINR
TINC
FPR95
AUROC
FPR95
AUROC
STEP
72.31
74.59
48.68
91.14
TSL
57.52
82.29
29.48
94,62
SAL (Ours)
43.11
89.17
19.30
96.29
ADDITIONAL RESULTS ON NEAR OOD DETECTION
In this section, we investigate the performance of SAL on near OOD detection, which is more chal- lenging OOD detection scenario where the OOD data has a closer distribution the in-distribution. Specifically, we use the CIFAR-10 10 as the in-distribution data and CTFAR-100 training set as the OOD data in the wild. During test time, we use the test set of CIFAR-100 100 as the OOD for evalua- tion. With mixing ratio of 0.1, SAL achieves an FPR95 of 24.51% and AUROC of 95.55% compared to 38. 92% (FPR95) and 93.27% (AUROC) of WOODS.
In addition, study near OOD detection in different data setting, i.e., the first 50 classes of CIFAR- 100 as ID and the last 50 classes as OOD. The comparison with the most competitive base- line WOODS is reported follows.
Table 10:
Near OOD detection with the first 50 classes of CIFAR- 100 ID and the last 50 classes OOD. Bold numbers are superior results.
OOD datasct
Methods
CIFAR-50
FPR95
AUROC
ID ACC
WOODS
41.28
89.74
74.17
SAL
29.71
93.13
73.86
K ADDITIONAL RESULTS ON USING MULTIPLE SINGULAR VECTORS
In this section, we ablate on the effect of using singular vectors to calculate the filtering score (Eq. 6). Specifically, we calculate the scores by projecting the gradient Vi(hw,g (x:). V for the wild data X; to each of the singular vectors. The final filtering score is the average over the scores. The result is summarized in Table 1I. We observe that using the top I singular vector for projection achieves the best performance. As revealed in Eq. 7, the top singular vector v maximizes the total distance from the projected gradients (onto the direction of v) to the origin (sum over all points in Swil),, where outliers lie approximately close to and thus leads to a better separability between the ID and OOD in the wild.
35

---

Published as a conference paper at ICLR 2024
Table 11: The effect of number of singular vectors used filtering score. Models are trained on Wide
ResNet-40-2 for 100 epochs with π 0.1. We use TEXTURES as the wild OOD data and CIFAR 100 as the ID.
Number singular vectors c
FPR95
AUROC
5.73
98.65
6,28
98.42
6.93
98.43
4
7.07
98.37
7.43
98.27
6
7.78
98.22
L ADDITIONAL RESULTS ON CLASS-AGNOSTIC SVD
In this section, evaluate our SAL by using class-agnostic SVD opposed class-conditional SVD as described in Section 3.1 of the main paper. Specifically, we maintain a class-conditional reference gradient VA. one for each class
[1, K] estimated on ID samples belonging to class k. Different from calculating the singular vectors based on gradient matrix with G2 (containing gradient vectors of wild samples being predicted as class k), we formulate single gradient matrix G where each row is the vector Vf(hw (x;): F&.. V G=:- for X; E Swild- The result is shown in
Table 12, which shows similar performance compared with using cansiconniiional SVD.
Table 12: The effect of using class-agnostic SVD. Models are trained on Wide ResNet-40-2 for 100 epochs with 0.1. CIFAR 100 is the in-distribution data. Bold numbers superior results.
OOD Datasts
Methods
SVHN
PLACES165
LSUN-C
LSUN RESIZE
TEXTURES
Average
ACC
I9995
AUROC
FPR95
AUROC
1P995
AUROC
11995
AUROC
TPR95
AUROC
[PR95
AUROC
SAL (Class agnostic SVD)
0.12
99.43
3,27
99.21
0.04
99.92
0.03
99.27
5.18
98.77
1.73
99.32
73.51
SAL
IU07
99.95
3.53
99.06
0.36
99.94
0J12
99.95
5.73
98.65
188
99.51
73.71
M ADDITIONAL RESULTS ON POST-HOC FILTERING SCORE
We investigate the importance of training the binary classifier with the filtered candidate outliers for OOD detection in Tables 13 and 14. Specifically, we calculate our filtering score directly for the test ID and OOD data on the model trained on the labeled ID set S only. The results are shown in the row ASAL (Post-hoe)" in Tables 13 and 14. Without explicit knowledge of the OOD data, the OOD detection performance degrades significantly compared to training additional binary classifier (a 15.73% drop on FPR95 for SVHN with CIFAR- 10 as ID). However, the post-hoc filtering score can still outperform most of the baselines that use Pin only (c.f. Table 1), showcasing its effectiveness.
Table 13: OOD detection results of using post-hoc filtering score on CIFAR- 10 ID. SAL trained on Wide ResNet-40-2 epochs with 0.1. Bold numbers are superior results.
OOD Datasets
Methods
SVIN
LSUN-C
LsuN-RosizH
Averae
ID ACC
FPROS
AUROC
FPRO5
AUROC
FPR05
AUROC:
FPR95
AUROC
FPRO5
AUROC
FPR05
AUROC
SAI. (Poer-heo))
15.75
93.00
23.18
96.35
6.28
96.72
15.59
89.83
23.63
87.72
16.89
90.74
94.44
99.98
0.07
99.99
99.99
0.90
99.74
99.78
93.65
Table 14: OOD detection results of using post-hoe filtering score on CIFAR-100 100 as ID. SAL trained on Wide ResNet-40-2 for 100 epochs with π 0.1. Bold numbers are superior results.
Datssets
Methods
SVIN
PLAnex365
LSUx-C
LSUN- Rusiz:::
TEXTURUS
Average
ID ACC
FPR95
AUROC:
FPROS
AUROC
FPR95
AUROC
FPR95
AUROC:
FPR05
AUROC
FPRU5
AUROC
(post-hoc)
39.75
35.94
84.53
23.22
36.38
85.45
73.71
36

---

Published as a conference paper at ICLR 2024
N ADDITIONAL RESULTS ON LEVERAGING THE CANDIDATE ID DATA
In this section, we investigate the effect of incorporating the filtered wild data which has a score smaller than the threshold (candidate ID data) for training the binary classifier go. Specifically, the candidate ID data and the labeled ID data are used jointly to train the binary classifier. The comparison with SAL on CIFAR- 100 is shown as follows:
Table 15: OOD detection results of selecting candidate ID data for training on CIFAR- 100 ID. SAL is traindd Wide ResNet-40-2 for 100 epochs with 0. Bold numbers are superior results.
0OD Datseets
Methods
SWIN
Pi ACFES5365
L.SUN-C
LSUN-RESIZE
TEXTURES
Average
FPRM5
AUROC
AUROC
FPRU5
RPR95
AUROC
FPRO5
FPRUS
AUROC
Candidate ID data SAL (Qurs)
L.23
99.87
2.62
99.18
0.04
99.95
0.02
99.91
4.71
98.97
1.72
99.58
73.83
0.07
99.95
3.55
99.06
0.06
99.94
0.02
99.95
5.73
98.65
1.88
99.51
75:71
The result of selecting candidate ID data (and combine with labeled ID data) shows slightly better performance, which echoes our theory that the generalization bound the OOD detector will be better if we have more IID training data (Theorem 3).
O ANALYSIS ON USING RANDOM LABELS
We present the OOD detection result replacing the predicted labels with the random labels for the wild data follows. The other experimental details are kept the same SAL.
Table 16: OOD detection results of using random labels for the wild data CIFAR- 100 as ID. SAL trained on Wide ResNet-40-2 for 100 epochs with π 0.1. Bold numbers are superior results.
Mishods
SVHN
PLACES365
LSUN-C
LSUN-REStEE
TEXTURES
Avenge
ID ACC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
w/ RRRddmm labels
39.36
89.31
77 77.98
78.31
47.46
88.90
67.28
80.23
$4.86
86
57 39
$4.73
73.68
As we can observe, using the random labels leads to worse OOD detection performance because the gradient of the wild data be wrong. In our theoretical analysis (Theorem 5), we have proved that using the predicted label can lead to good separation of the wild ID and OOD data. However, the analysis using random labels might hold since violates the assumption (Definitions and 3) that the expected gradient of ID data should be different from that of wild data.
P DETAILS OF THE ILLUSTRATIVE EXPERIMENTS ON THE IMPACT OF PREDICTED LABELS
For calculating the filtering accuracy, SAL trained on Wide ResNet-40-2 for 100 epochs with 0.1 on two separate ID datasets. The other training details are kept the same Section 5.1 and Appendix G.
Q DETAILS OF FIGURE 2
For Figure 2 in the main paper, we generate the in-distribution data from three multivariate Gaus- sian distributions, forming three classes. The mean vectors are set to [-2, 0j. [2, 0] and [0, 2v3),
respectively. The covariance matrix for all three classes is set to 0.25
0.25 For each class,
generate 1, 000 samples.
For wild scenario 1, we generate the outlier data in the wild by sampling 100, 000 data points from multivariate Gaussian ((0, 1 where is 2 2 identity matrix, and only keep the 1, 000 data points that have the largest distance to the mean vector [0, 3/1 For wild scenario 2,
37

---

Published as a conference paper at ICLR 2024
generate the outlier data in the wild by sampling 1, 000 data points from multivariate Gaussian M{110, 2 vg 0.25 I). For the in-distribution data in the wild, we sample 3, 000 data points per class from the same three multivariate Gaussian distributions as mentioned before.
R SOFTWARE AND HARDWARE
We all experiments with Python 3.8.5 and PyTorch 1.13.1, using NVIDIA GeForce RTX 2080Ti
GPUs.
RESULTS WITH VARYING MIXING RATIOS
We provide additional results of SAL with varying и, i.e., 0.05, 0.2, 0.5, 0.9, and contrast with the baselines, which are shown below (CIFAR-100 as the in-distribution dataset). We found that the advantage of SAL still holds.
Table 17: OOD detection results with multiple mixing ratios ा with CIFAR- 100 as ID. SAL trained on Wide ResNet-40-2 for 100 epochs. Bold numbers are superior results.
OOD Dataeet
Methods
SVEN
PLACES365
LSUN-C
LSUN-RESIZE
TEXTURES
ID ACC
FPR95
AUROC
FPR95
AUROC
FPR9S
AUROC
FPR9S
AUROC
FPROS
AUROC
0.05
OE 2.78
98.84
80.22
6.73
98.37
2.06
99.19
32.86
90.88
71.98
Energy w/ OE
2.02
99.17
56,18
83.33
4.32
98.42
3.96
99.29
40.41
89.80
73.45
WOODS
0.26
99.89
32.71
90.01
99.77
0.79
99.10
12.26
94.48
74.15
SAL (Ours)
0.17
99.90
6.21
96.87
0.94
99.79
0.84
99.37
5.77
97.12
73.99
0.2
OE
2.59
98.90
55.68
84,36
4.91
99.02
1.97
99.37
25.62
93.65
73.72
Energy w/ OE
1.79
99.25
47.28
86.78
4.18
99.00
3.15
99.35
36.80
91.48
73.91
WOODS
0.22
99.82
29.78
91.28
0.52
99.79
0.89
99.56
10.06
95.23
73.49
SAL (Ours)
0.08
99.92
2.80
99.31
0.05
99.94
0.02
99.97
5.71
98.71
73.86
0.5
OF
2.86
99.05
40.21
8R.75
4.13
99.05
1.25
99.38
22.86
94.63
73.38
Energy OE
2.71
99.34
34.82
90.05
3.27
99.18
2.54
99.23
30.16
94.76
72.76
WOODS
0.17
99.80
21.87
93.73
0.48
99.61
1.24
99.54
9.95
95.97
73.91
SAL (Ours)
0.02
99.98
1.27
99.62
0.04
99.96
0.01
99.99
5.64
99.16
73.77
17
0.9
OB Energy w/ OF WOODS
19.78
96.29
1.64
99.75
12.74
72.02
0.97
99.64
17.52
96.53
1.36
99.73
0.94
99.59
14.01
95.73
73.62
11:34
95.83
6.72
98.73
73.86
SAL (Ours)
0.03
99.99
2.79
99.89
0.05
99.99
0.01
99.99
5.88
99.53
74.01
T COMPARISON WITH WEAKLY SUPERVISED OOD DETECTION BASELINES
We have additionally compared with the two related works (TSL (He al., 2023) and STEP (Zhou et al., 2021)). To ensure fair comparison, we strictly follow the experimental setting in TSI. and rerun SAL under the identical setup. The comparison CIFAR- 100 shown follows.
Table 18: Comparison with relevant baselines on CIFAR- 100. Bold numbers are superior results.
OOD Datasets
Methods
LSUN-C
LSUN REN-RR
FPR95
AUROC
FPR95
AUROC
STEP
0.00
99,99
9.81
97.87
TSL
100.00
1.76
99,57
SAL (Ours)
0.00
99.99
0.58
99.95
38

---

Published as a conference paper at ICLR 2024
U ADDITIONAL RESULTS ON DIFFERENT BACKBONES
We have additionally tried ResNet-18 and ResNet-34 as the network architecture- are among the most used in OOD detection literature. The comparison with the baselines on CIFAR- 100 is shown in the following tables, where SAL outperforms all the baselines across different architectures. These additional results support the effectiveness of our approach.
OOD detection performance on CIFAR- 100 epochs. For each dataset, we create corresponding wild mixture distribution Iwaa = (1 x)lPin + xPPou for training and test on the corresponding OOD datasct. Bold numbers highlight the best results.
Table 19:
All methods are trained on ResNet-18
OOD Dutasets
Methods
SVHN
PLACES365
LSUN-C
RES--R
TEXTURES
Average
ACC
FPR95
AUROC
FPR95
AUROC
EPRO5
AUROC
FPR35
AUROC
FPR95
AUROC
FPR05
AUROC
40.94
Mahalanobis
22.44
95.67
92.06
61.39
68.90
86
23.07
94.20
62.39
79.39
53.89
83.39
78.67
Energy
81.74
84.56
$2.23
76.88
34.78
93.93
73.57
82.99
85.87
74,94
71.64
82,02
78.67
KNN
83.62
72.76
82.09
80.03
665
84-82
71,05
81. 24
76.58
77.90
75.92
79.35
78.67
70.81
$4.65
49.08
75.64
KNN+
43:21
90.21
84.62
74,21
30.12
82.48
76,92
$0.81
63.21
84,91
63.61
82.52
77.03
With Pa and 2-d1
OF
3.29
97.93
62.90
80.23
7.07
95.93
4.06
97.98
33.27
90.05
22.12
92.42
74.89
SAT.
2,29
97.96
96.66
97.81
97:10
8.28
5.13
Table 20: OOD detection performance CIFAR-100 ID. All methods trained ResNet-34 for 100 epochs. For each dataset, we create corresponding wild mixture distribution Paldd = (1 x)P + x Fouu for training and test on the corresponding OOD dataset. Bold numbers highlight the best results.
Datasets
Methods
SVHN
PLACES365
LSUN-C
LSUN-RESIZ
TEXTURES
Average
ID ACC
FPR95
AUROC
FPR95
AUROC
FPRUS
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
With F. only
78
84.38
14.21
75.28
74.51
86.51
75.27
70.
82.16
79.11
Mahulsnohis
87.09
73.89
Energy
66.91
85.25
81.41
76.37
59.77
86 69
66.52
$4.49
79.01
79.96
70.73
82.55
79.04
KNN
81.12
73.65
79.62
78.21
63.29
85.
73.92
79:77
73.29
80.35
74.25
79 79.51
79.04
ReAct
82.85
70.12
81.75
76.25
80.70
83.03
47
83. 28
74. 60
81.61
77. 46
78 36
79.04
DICE
83.55
72.49
85.05
15.92
73.59
75.20
90.90
79.30
77.83
33.53
76.15
79.04
64.53
77.58
Wids Ps and P-8aa
OB
2.11
98.23
60.12
83.22
6.08
96.34
3.94
98, 13
30.00
92.27
20.15
93.64
75.72
Energy (w/ OE)
1,94
95.03
68 68.84
85.94
7.66
92.04
6.85
97.63
40.82
93.07
25.22
92.74
78.75
WOODS
2.08
97.33
88.93
97.34
93.30
V BROADER IMPACT
Our project aims to improve the reliability and safety of modern machine learning models. From the theoretical perspective, our analysis can facilitate and deepen the understanding the effect of unlabeled wild data for OOD detection. In Appendix F, we properly verify the necessary condi- tions and the value of our error bound using real-world datasets. Hence, we believe our theoretical framework has a broad utility and significance.
From the practical side, our study can lead to direct benefits and societal impacts, particularly when the wild data is abundant in the models' operating environment, such in safety-critich applications ie., autonomous driving and healthcare data analysis. Our study does not involve any human sub- jects violation of legal compliance. We do not anticipate any potentially harmful consequences our work. Through our study and releasing our code, we hope to raise stronger research and societal awareness towards the problem of exploring unlabeled wild data for out-of -fittiribittiionn detection in real-world settings.
39


Reviews:
[{"review_id": "hnjDxVo5iO", "reviewer": "Reviewer_BtG8", "summary": "This paper presents a novel framework for Out-Of-Distribution (OOD) detection, named SAL, which aims to improve machine learning models through regularization using unlabeled data. SAL comprises two main components: (1) Filtering–distinguishing potential outliers from the general dataset, and (2) Classification utilizing the identified candidate outliers to train an OOD classifier. The paper includes pertinent theoretical proofs to substantiate the proposed method, and experimental results are provided to demonstrate its effectiveness.", "questions": "1 Could you design experiments to further confirm and answer issues in weaknesses?\n2 In table 1, how can this discrepancy be accounted for in datasets or scenarios that perform well in other methods but have degraded performance (ID ACC) in SAL?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1 SAL’s methodology is structured around two distinct phases—screening and classification—which can be independently optimized, offering enhanced flexibility.\n2 Utilizing a Large Volume of Unlabeled Data: SAL effectively leverages substantial amounts of unlabeled data to extract valuable information, thereby bolstering its detection capabilities.\n3 Theoretical Support: Beyond its impressive empirical performance, SAL is underpinned by robust theoretical foundations.", "weaknesses": "1 In scenarios where the actual OOD data markedly diverges from the outliers present in the unlabeled dataset, there arises a question regarding the preservation of SAL’s performance.\n 2 The efficacy of SAL is significantly influenced by the quality of the unlabeled data employed, indicating a substantial dependence on data integrity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQbxpbruWt", "reviewer": "Reviewer_XJRS", "summary": "This paper introduces a novel setting for OOD detection, termed as ”wild OOD detection,” building upon the foundation established by the preceding work ”Training OOD Detectors in their Natural Habitats.” A novel methodology, denoted as SAL, is presented, encapsulating at wo-stage process comprising filtering and classification components. Empirical evaluations have demonstrated that SAL achieves SOTA performance, boasting substantial improvements over existing methods. Additionally, theoretical underpinnings are provided to bolster the credibility and effectiveness of SAL.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A theory has been established to investigate aspects of separability and learnability. This contribution is both novel and significant. \n\n2. Experimental evaluations conducted on standard benchmarks demonstrate that SAL achieves SOTA performance. \n\n3. A novel method grounded in theory has been developed to advance safe machine learning practices. Theory serves as a crucial driver in this endeavour. I am very happy to see the novel work on provable OOD detection.", "weaknesses": "1. Could you provide explanations or conduct experiments to elucidate the factors contributing to the decreased ID accuracy depicted in Table 1? \n\n2. Why is pi set to 0.1 in most experiments? Could you conduct additional experiments to investigate whether pi remains robust across a range of values? Furthermore, does the performance of pi align with theoretical predictions? \n\n3. It appears that the top singular vector is crucial for SAL. Have you conducted any experiments to demonstrate the performance when considering the top 2, top 3, ..., top k singular vectors?\n\n4. What would occur if you were to use the gradient norm in place of the top singular vector? Could you elucidate the rationale behind opting for the top singular vector instead of the norm?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPpZnPrDRp", "reviewer": "Reviewer_dSEG", "summary": "Leveraging unlabeled data has shown potential in enhancing the safety and reliability of machine learning models for out-of-distribution (OOD) detection, despite the challenges posed by the heterogeneity of both in-distribution (ID) and OOD data. This paper introduces SAL (Separate And Learn), a novel learning framework that addresses the existing gap in understanding how unlabeled data aids OOD detection by providing strong theoretical guarantees and empirical effectiveness. SAL operates by isolating potential outliers from the unlabeled data, training an OOD classifier with these outliers and labeled ID data, and achieving state-of-the-art performance on standard benchmarks, thereby validating the theoretical framework and its components.", "questions": "1. Near OOD Scenario: It is unclear how effective the method presented in this article would be in a near OOD scenario, such as treating the first 50 classes of CIFAR-100 as ID and the last 50 classes as OOD. This specific situation could pose challenges since near OOD samples may share similarities with the ID data, potentially affecting the method's performance.\n2. Generalization Performance Across Different Backbones: The article does not provide information on how well the method generalizes when different backbone architectures are used. The performance stability of the method when transitioning between various model architectures is a critical aspect to consider for its widespread applicability.\n3. Generalization Performance on unseen OOD data: There is no discussion on the method's effectiveness when the test OOD data and the OOD data in the unlabeled set are not identically distributed. Understanding how the method handles such disparities is crucial for evaluating its robustness in real-world scenarios.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript presents some theoretical analyses as well as a number of intriguing illustrations.\n2. The experimental results look promising, with comparisons made against numerous baseline methods.", "weaknesses": "1. The manuscript lacks crucial baselines, such as [1] and [2], which are essential for a comprehensive evaluation, and fails to provide an analysis or comparison with them.\n2. The essential topic of the article is weakly supervised out-of-distribution detection, although it is described from different perspectives.\n\n[1] Zhou, Zhi, et al. \"Step: Out-of-distribution detection in the presence of limited in-distribution labeled data.\" Advances in Neural Information Processing Systems 34 (2021): 29168-29180.\n\n[2] He, Rundong, et al. \"Topological structure learning for weakly-supervised out-of-distribution detection.\" arXiv preprint arXiv:2209.07837 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2KOOKp43sz", "reviewer": "Reviewer_ZN1X", "summary": "This study addresses wild Out-Of Distribution (OOD) detection, benefiting from the availability of more unlabeled data to enhance OOD data identification. The paper proposes the utilization of the top singular value as a criterion to differentiate between In-Distribution (ID) and OOD samples. The authors, grounded in novel theoretical insights, posit that ID samples should exhibit larger top singular values compared to OOD samples. Both experimental results and theoretical analyses corroborate the effectiveness of the proposed method, Separate And Learn (SAL). Overall, this work stands out for its solid foundation.", "questions": "Please address the issues outlined in the weaknesses. This work is solid and good. So addressing all of my concerns comprehensively will lead me to reconsider and potentially raise my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Using gradients to make distinctions is a common and intuitive approach that makes sense to me. How ever, the method of distinguishing based on the top singular value is both intriguing and non-obvious. This innovative discovery holds significant importance for OOD detection.\n\n- The availability of extra unlabeled data undoubtedly enables wild OOD detection to outperform traditional OOD detection, which does not utilize unlabeled data. However, what I find astonishing is that the performance of wild OOD detection surpasses even that of outlier exposure, showcasing its remarkable effectiveness.\n\n- The method is supported by a theoretical examination of its generalization bounds, ensuring a solid foundation for reliable ML.", "weaknesses": "- One of my major concerns with the paper lies in the discrepancy assumption stated in Theorem 1. To validate this, additional experiments with diverse datasets are imperative to ascertain whether the assumption is consistently met in practical scenarios. While I acknowledge that assumptions are essential for theoretical foundations, and the experiments in Appendix F provide a good start, they are insufficient for a comprehensive evaluation. I recommend extending the empirical analysis to additional datasets (e.g., CIFAR-10 as ID and another data as OOD), with the goal of thoroughly investigating the generality of the assumption and its ability to underpin SAL robustly.\n\n- In the majority of the experiments conducted, the value of pi is set to 0.1. To ensure a comprehensive evaluation, it is crucial to conduct additional experiments with varied values of pi.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)