Can We Ignore Labels in Out of Distribution Detection?
Authors: Hong Yang
Qi Yu
Travis Desell
Venue: ICLR 2025 Poster (2025)
Keywords: Out of Distribution
Uncertainty
Self Supervised
Unsupervised
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Labels are necessary in out of distribution detection

Abstract: Out-of-distribution (OOD) detection methods have recently become more prominent, serving as a core element in safety-critical autonomous systems. One major purpose of OOD detection is to reject invalid inputs that could lead to unpredictable errors and compromise safety. Due to the cost of labeled data, recent works have investigated the feasibility of self-supervised learning (SSL) OOD detection, unlabled OOD detection, and zero shot OOD detection. In this work, we identify a set of conditions for a theoretical guarantee of failure in unlabeled OOD detection algorithms from an information-theoretic perspective. These conditions are present in all OOD tasks dealing with real world data: I) we provide theoretical proof of unlabeled OOD detection failure when there exists zero mutual information between the learning objective and the in-distribution labels, a.k.a. ‘label blindness’, II) we define a new OOD task – Adjacent OOD detection – that tests for label blindness and accounts for a previously ignored safety gap in all OOD detection benchmarks, and III) we perform experiments demonstrating that existing unlabeled OOD methods fail under conditions suggested by our label blindness theory and analyze the implications for future research in unlabeled OOD methods.

Published as a conference paper at ICLR 2025
CAN WE IGNORE LABELS IN OUT-OF-DISTRIBUTION DETECTION?
Hong Yang, Qi Yu, Travis Desell Rochester Institute of Technology 1 Lomb Memorial Dr, Rochester, NY 14623, USA (hy3134, 34, jdvse, yu)eri..ee..
ABSTRACT
Out-of-distribution (OOD) detection methods have recently become more promi- nent, serving as a core element in safety-critices autonomous systems. One major purpose of OOD detection is reject invalid inputs that could lead to unpre- dictable errors and compromise safety. Due to the cost labeled data, recent works have investigated the feasibility of self-supervised learning (SSL) OOD de- tection, unlabeled OOD detection, and zero shot OOD detection. In this work, we identify set of conditions for a theoretical guarantee of fallure unlabeled OOD perspective. These conditions
detection algorithms from an information-theone are present all OOD tasks dealing with real-world data: I) we provide theo- retical proof of unlabeled OOD detection failure when there exists zero mutual information between the learning objective and the in-distribution labels, a.k.a. 'label blindness', II) define new OOD task Adjacent OOD detection that tests for label blindness and accounts for a previously ignored safety gap all OOD detection benchmarks, and III) perform experiments demonstrating that existing unlabeled OOD methods fail under conditions suggested by our la- bel blindness theory and analyze the implications for future research in unlabeled OOD methods.
INTRODUCTION
Safety-critical applications of deep neural networks have recently become an important area of in- vestigation the domain of artificial intelligence, ranging from autonomous driving (Ramanagopal et al., 2018) to biometric authentication (Wang & Deng 2021) to medical diagnosis (Bakator & Radosav] 2018). In the setting of safety-critica systems, it is no longer possible to rely on the closed-world assumption (Krizhevsky et al. 2012). where test data is drawn i.i.d. from the same distribution as the training data, known as the in-distribution (ID). These models will be deployed in an open-world scenario (Drummond & Shearer (006), where test samples can be nut-of-distribution (OOD) and therefore should be handled with caution. OOD detection seeks to identify inputs con- taining label that never present in the training distribution. The motivation for OOD detection simple: we do not want safety-criticl systems to act on an invalid prediction, where the predicted label cannot be correct because the label was never present in training.
There is significant interest in unlabeled OOD detection due to various factors. A method that does rely on labels can save significant costs in labeling data, as proposed by (Sehwag et al. 1021). is also be possible to skip training on the in distribution data if such a model is generalizable, as proposed by (Wang al., 2023). Self supervised and unlabeled learning methods can also scale to much larger datasets and it important for these models to be robust to OOD data. Recent work in unlabeled OOD detection methods, including (Sehwag et al. 2021] Tack et al., 2020 Liu et al. 2023 Guille-Escuret et al.) 2024 Wang et al.] (003)), promise to improve safety using only unlabeled data. These methods can achieve even greater performance than a simple supervised baseline (Hendrycks & Gimpel 2016), suggesting that one could replace supervised training with self-supervised learning (SSL) for safety critical OOD detection task. This family of SSL OOD methods differ from traditional supervised OOD methods, including (Fort et al. 2021), by the of only unlabeled data. The importance of labels is an active area of research in OOD detection (Du et al. 2024alb) bb)

---

Published as a conference paper at ICLR 2025
When view SSL from an citormmtiionthhorrteee perspective, the selection of features depends solely on the SSL objective and not on the labels. This, however, provides no guarantee that any features relevant for label prediction will be retained. Figure provides provides an example of how SSL
features can be less effective for identifying label. Our theory importantly shows that, when the label-relevant features are independent of the features relevant for the SSL. algorithm's successful operation, OOD detection is guaranteed to fail due to what we call 'label blindness' and that this label blindness occurs regardless of how one selects the ID dataset from the population of all data. Our experiments also suggest that Zero Shot OOD methods [Wang et al., 2023; Esmaeilpour et al.) 2022) may also suffer from this issue. We show that unsupervised OOD detection methods behave in the same way as SSL in the context of information theory.
However, one can unintentionally avoid label blindness problem via the selection of the OOD dataset when constructing OOD benchmarks. Existing methods gen- erally consider ID and OOD data from dif- ferent datasets, e.g., (Fort al. 2021), (Sehwag et al., 2021). and (Hendrycks al. (019). In these benchmarks, there is no significant overlap between the ID) and OOD imput data, allowing OOD detection algorithms to succeed on features indepen- dent of the label. To address this issue and to test for label blindness, we introduce the Adjacent OOD detection task to evalu- OOD detection al-
Figure An example failure case by visualizing the heatmaps of the gradient of unlabeled SimCLR trained Resnet (Chen et al. 2020) using the GradCAM method (Selvaraju al. 2017). The OOD detection task is to detect OOD facial expressions. In this case, the OOD detection method fails as justified by our the- oretical work, where the representations do not exhibit strong gradient in regions commonly associated with facial expressions (i.e., eyebrows, mouth, etc.).
ate the performance gorithms when there is significant overlap between the OOD input data and ID input data. We also prove that is impossible guarantee that a real world system will never encounter OOD input data that sig- nificantly overlaps ID input data.
This work aims to answer the following question: can we ignore labels when engaging in OOD detection? Through numerous experiments and theoretical proofs, show that it is safe to ignore labels when performing OOD detection. This is contrary to the increasing recent efforts that propose new self supervised, unsupervised, and other unlabeled OOD detection methods. This work's key contributions include:
The Label Blindness Theorem. We theoretically prove that any SSL or Unsupervised Learning algorithm will fail when its information required for the surrogate task is independent of the information required for predicting labels. Through this proof, we conclude that there cannot be a generally applicable SSL or Unsupervised learning OOD detection algorithm as there will always exist independent labels due to the no free generalization theorem, see theorem 3.5) Adjacent OOD detection benchmarks. We introduce the concept of bootstrapping without placement of the ID labels to create the Adjacent OOD detection task. To the authors' knowledge, this OOD detection task is novel to and absent from research in OOD detection. This task evalu- ates OOD detection when there is significant overlap OOD data and ID. We also theoretically prove that overlapping OOD and ID data possible in every real world dataset.
Impact on existing and future OOD methods. We demonstrate that existing SSL and Unsuper- vised Learning OOD methods fail under the conditions suggested by our theory and that existing benchmarks do not capture such failures. We also evaluate zero shot OOD detection methods, which fail in a similar manner to SSL. and Unsupervised Learning OOD methods. We make recommendations on the development and testing of future OOD methods.
2 PRELIMINARIES
2.1 LABELED AND UNLABELED
DETECTION
The task of distr---ittiitttion detection is to identify semantic shift in the data (Yang et al. 2021). This is determining when predicted label could match the true label y & V .ns where W.
2

---

Published as a conference paper at ICLR 2025
represents the set of in-distribution training labels. In this case, would consider the semantic space of the sample and the training distribution to be different, representing a semantic shift. We can express the probability that sample is out-of-distribution via P & /m) One bascline method to calculate P(y & Win a)) is to take - MSP( where MSP is the maximum softmax probability from classifier for particular datapoint.
Furthermore, we are only concerned with labels that can be generated using only æ, via function f which depends solely æ and no other information. f may represent human labelers that generate y. If we consider Yau as the set of all possible labels that can be generated from f( E Xau), subset of Xill considered as Xtraining may not contain all labels in Yaul For real world datasets, is possible that in S Yall.
We can also approach the problem of OOD detection without the use of labels. One can train model on ID data using surrogate task for the purposes of computing metric. For example, (Se-) hwag al. (02T) trains resnet with SimCLR and computes the Mahalanobis distance between the training representations and the test sample representations to compute the OOD score. Altema- tively, one could utilize a pretrained model with broad knowledge to compute a metric to use as the OOD score, such as in (Wang et al. 2023).
2.2 SELF-SUPERVISED AND UNSUPERVISED LEARNING
This section covers representation learning and its implications for SSL and unsupervised learning. there is no mutual information between two random variables, neither can be used to reduce un- certainty about the other (Shannon I948). In both self-supervised and unsupervised OOD detection, there is no mutual information between the intermediate representations and the OOD detection task, the OOD detection system cannot reduce uncertainty with respect to the OOD detection task using the intermediate representations.
Representation learning can be formulated finding distribution p(z[x) that maps the observa- tions from æ E X to z E Z, while capturing relevant information for some primary task. When y represents some primary task, we consider only z that is sufficiently discriminative for accom- plishing the task y. For simplicity, we consider y as a classification label, but can represent any objective or task. Federici et al. (2020) show that this sufficiency is met when the information relevant for predicting is unchanged when encoding x + z.
Definition 2.1. Sufficiency: representation z of x is sufficient for y if and only if I y 2) = 0.
Since there exists the sufficient statistic x = z, we must consider the minimal sufficient statistic which conveys only relevant information for predicting y. An SSL algorithm seeks learn the minimal sufficient statistic via the information bottleneck framework (Shwartz-Ziv & LeCun (023))
Definition 2.2. Minimal Sufficient Statistic. sufficient statistic z is minimal if, for any other sufficient statistic s. there exists a function f such that z f()).
Information bottleneck optimization can be expressed as the minimization of the representation's complexity via [x; z) and maximizing its utility [(z; y). This results in the information theoretic loss function below, where 33 is a trade-off between complexity and utility (Shwartz-Ziv & LeCun 2023]. In practice, learning without y requires surrogate task y., e.g., (Chen al. a 2020), with the loss defined as:
L 1= 31(zzy).
should be noted that the primary task y may be equal to the SSL task ys- In such a case, com- pression towards the minimal sufficient statistic still occurs. This is important because unsupervised methods for deep neural networks (DNNs) will use a surrogate task y. to train the DNN's 's weights. Thus, if we assign the primary task for unsupervised learning method to be equal to its surrogate task, it will behave identically to SSL from the perspective of information theory.
When x has higher information content than y. there exists information in x that is not relevant for predicting y. This can be better understood by dividing 1 (x; z) into two terms (Federici et al. 2020) as follows:
(x;z) = [(x; 2 ly)
( predictive information
supertlucus information
3

---

Published as a conference paper at ICLR 2025
However, superfiuous information is not affected by the labels of primary task, only by and ys- Using information theory, we can show that any SSL OOD detection algorithm will fail when the surrogate task ys is independent of the labels in the in-distribution dataset. This applies to unsuper- vised OOD detection algorithms that also use surrogate task.
GUARANTEED OOD DETECTION FAILURE
This section introduces the concept of Label Blindness, with one key supporting theorem and one key supporting lemma. The full proofs for these theoretical results are provided in Appendix [ Note that Rx represents the support of random variable x such that Rx = {æ P(x) > 0}. For
clarity, we refer to cases where / [ X2) 0 as Strict Label Blindness and discuss Approximate
Label Blindness / (x11; x2) R the end of section 3.1
3.1 LABEL BLINDNESS THEOREM (STRICT LABEL BLINDNESS)
We identify guarantee of OOD detection failure for any information bottleneck- based optimization process if the unlabeled learning objective is independent from labels used to determine the ID set, described by Corollary 3.3 This corollary is derived from two concepts: strict label blindness in minimal sufficient statistic and the independence of filtered distributions. We first consider the minimal sufficient statistic and how it leads to strict label blindness; see Theorem 3.1
Theorem 3.1. Strict Label Blindness in the Minimal Sufficient Statistic.
Let come from a distribution. x is composed of two independent variables x1 and X2- Let Y1 be surrogate task such that H(y
O. Let be any sufficient representation of x for y1 that satisfies the sufficiency definition 2.1 and minimizes the loss function C [(x]Xg; - 31 (1). The possible that minimizes L and is sufficient must meet the condition 1 [x22 z)) 0.
Detailed proof in Appendix D.4
Intuitively, the minimal sufficient representation cannot encode any information independent of the surrogate learning objective, otherwise it would not be minimal. This means that the representation will be blind to any label built upon the independent information.
However, Theorem 3.1 is not sufficient to guarantee OOD failure. This is because the selection of the ID training set could change the learned representation z, possibly improving OOD detection performance by increasing mutual information, / (x2; z) > O. We formally disprove this possibility through Lemma 3.2
Lemma 3.2. Independence of Filtered Distributions.
Let come from distribution. is composed of two independent variables X1 and x2. For X22 where Rag, C Rwy there exists no x such that Hxx1(x) < Hxxx).
Detailed proof in Appendix D.5
Lemma 3.2 states that filtering on label generated on one of two independent variables cannot provide information about the other. This applies the selection of data from the population, if the selection criteria is independent of the learning objective. This means that the strict label blindness properties predicted by Theorem 3.1 will apply to ID training data. These two concepts bring to main result strict label blindness in filtered distributions; see Corollary 3.3
Corollary 3.3. Strict Label Blindness in Filtered Distributions. Let x come from a distribution. x is composed of two independent variables X1 and X2. Let Y1 be surrogate task such generated by y1 fi(x1) Hyyl(x)) O. Let y2 be label such that H (y2)X2) = 0 and y/2 = f2 Let Ynn be as subset of labels Yn C. Ryy. Let x' be a subset of where Rx Rx n {ar IR f2((zz) Vin) such that x' is composed of independent variables x and x, and yí = Si (1())) The sufficient representation z learned by minimizing C= (xxxxxx;;z) (1( y') must have 1(x2; 0 and 1(y2;z) 0.
Detailed proof in Appendix D.6
This means that, when we select the ID training data, if the selection criteria and labels are indepen- dent of the surrogate learning objective, then we can guarantee failure in OOD detection due to the absence of any information in the learned representation z. For simplicity, we refer to this concept, supported by Corollary 3.3, as the problem of Striet Label Blindness.
4

---

Published as a conference paper at ICLR 2025
In summary, when the surrogate learning task can be achieved without learning about features rele- vant for the label, it will not learn any features relevant for the label. Figure Tis visualized example of this. If the SSL or unsupervised learning method fails to learn any label-relevant features, then any OOD detection algorithm built from those representations cannot differentiate hetween the la- bels selected as ID and those not selected ID. This guarantees failure in OOD detection because no label information passes through the information bottleneck.
Implications of Strict Label Blindness in Real World Situations We can utilize Fano Inequality to extend our understanding of strict label blindness to consider situations of where the variables are not fully independent. The lower bound for prediction error defined by the entropy of the target label y less the mutual information between the input and target label, shown in Theorem 3.4 Under strict label blindness, when / (x; y) = O, the lower bound for error is at its maximum. When I yx: & O, the lower bound for error is large enough to be unreliable. We refer to this condition as Approximate Label Blindness and we conduct experiments to evaluate this condition. Unless specified as strict, label blindness refers to the approximate case.
Theorem 3.4. Fano's 's Inequality (See (Roberl 1952)).
Let y be a discrete random variable representing the true label with y possible values and car- dinality of |3]| and x be a random variable used to predict y. Let be the occurrence of an er- ror such that y * ÿ where ý fxx). Let H₃ represent the binary entropy function such that H₂(e) = - P(e) log P(c) - (1 P(e)) log(1 - P(e)). The lower bound for P(e) increases with lower y(
Ho(e) + P(e) [))( - 2 (ly) ((x:)).
3.2 DISTINCTIONS FROM THE STATE-OF-THE-ART
Previous work by (Federici et al., 2020) introduced the No Free Generalization Theorem, which contains some similar concepts. states that a compressed representation of x cannot contain information for all possible labels of but does not guarantee OOD detection failure.
Theorem 3.5. No Free Generalization (See (Federici ef al. (020)).
Let x, z and y be random variables with joint distribution p(x, y. Let z' be a representation of x that satisfies [(x; z) > I(x; z²), then it is always possible to find label y for which z' is not predictive for y while z is.
Our theoretical work shows the exact conditions that guarantee OOD failure based on the mutual in- formation between the labels and the loss function, when performing SSL or unsupervised learning.
Recent work by (Du al. 2024b) investigates how labels can improve the performance of OOD algorithms. Their work does not describe any guarantee of failure OOD detection in the absence of labels, but does support the idea that labels are important for OOD detection.
3.3 THEORETICAL IMPLICATIONS
Our work applies deep neural networks (DNNs) trained without labels for the purpose of OOD detection. The key assumption of information bottleneck compression is generally applicable to DNNs (Shwartz-Ziv & Tishby 2017). Regardless of other assumptions, such as the multi-view as- sumption, an information bottleneck DNN trained without labels will still compress data irrelevant to its loss objective, even that data relevant for its intended task. It does not matter what task the training process was originally designed for because the unlabeled training process ultimately generates/adheres to its own learning objective. For any learning objective, there will exist an inde- pendent feature unless compression is not possible, as in yx: = I(x: x). If there is compression, then there exists labels for which OOD detection failure is guaranteed.
Our work predicts guarantee of failure only when consider the OOD set of all non ID data. In our own experiments and in work by (Schwag et al.] 202T, Hendrycks et al., 2019 Liu et al. 2023), purely self-supervised and unsupervised OOD methods can perform well against common benchmark OOD sets. This suggests that the choice of ID set and OOD set pairs can unintentionally hide label blindness failure. Alternatively, we can also construct test to identify if the OOD de- tection algorithm suffers from label blindness. To construct such a test, we rely on the insight from Corollary 3.3 and the use of a simple statistical method.
5

---

Published as a conference paper at ICLR 2025
4 BENCHMARKING FOR LABEL BLINDNESS FAILURE
4.1 BOOTSTRAPPING AND THE ADIACENT OOD BENCHMARK
One logical consequence of Corollary 3.3 is that one cannot avoid failure due to label blindness by selecting different labels for one's s ID set, so long as the label selection is independent from the bluelearning objective. To test any OOD detection algorithm for label blindness failure, this simply entails selecting different labels for one's ID set. To construct this benchmark, labels to be considered as ID and other labels to be consider OOD. This is similar to bootstrapping, but without replacement. If an OOD detection algorithm is "approximately label blind', its average OOD detection performance across the samples should be poor. We refer to this as the Adjacent OOD Detection Benchmark.
randomly sample
4.2 WIIY ADIACENT OOD IS SAFETY-CRITICAL TO ALMOST ALL REAL WORLD SYSTEMS
The Adjacent OOD detection benchmark evaluates the performance of OOD detection algorithms when there may be significant overlap between the data and OOD data. This condition applies to all systems where it is impossible guarantee that there will be significant overlap in the feature space between ID and OOD data. This is true for almost all real world systems and is theoretically proven below.
Theorem 4.1. Unavoidable Risk of Overlapping OOD Data
Let x come from a distribution. Let f be some labeling function to generate labels y such that y f(x). where there are at least two unique labels Ry| > 1. Let Xin be a random subset of where Rx.. & Rx and [xx..l < cc. Let Yin be labels generated from yin = fz The probubility that randomly selected x contains y not present in Ry.. is always greater than
Detailed proof in Appendix D.7
In theory, this risk can be reduced to acceptable level by adding more data to the training dataset. However, this reduction in risk requires the assumption that the collected data is randomly sampled. This is almost never true for real world datasets and often the opposite is true, where the nature of sampling can significantly increase this risk.
One risk factor present in every real world dataset is the dataset creation date. By creating the dataset at any specific point in time, the dataset cannot be randomly sampled with respect to time because it is impossible to collect data from the future. For example, if where create dataset of diseases today, it would not contain any future diseases. In this example, the probability that the training dataset is incomplete is 100%, which guarantees that there will be OOD data that significantly overlaps with ID data. For most real world systems, the only safe assumption is that there may OOD data that overlaps with ID data and is necessary plan accordingly.
The failure predicted by the label blindness theory is easiest to detect in the adjacent OOD situation. Where there likelihood of adjacent data, Theorem predicts OOD detection failure. Where there is no adjacent data, features independent of the label can still be used to distinguish between II) data and non adjacent OOD data, as shown in various experiments this paper and others (Sehwag
ct al. 2021 Hendrycks et al. 2019 Liu et al. 2023).
4.3 COMPARING ADJACENT, NEAR, AND FAR OOD BENCHMARKS
Many unlabeled OOD methods generally perform quite well on far and near OOD tasks. Far OOD is often defined by ID and OOD sets with different semantic labels and styles (Fang et al., 2022). One such far OOD benchmark is MNIST as ID data and CIFAR 10 as OOD data. Near OOD contains ID and OOD sets with similar semantic labels and styles (Fang et al. 2022). These tasks tend to be more difficult for existing OOD detection methods than far OOD detection tasks. One such near OOD benchmark is CIFARIO as ID and CIFARIO0 100 as OOD. However, the overlap in the near OOD detection benchmarks is significantly less than the adjacent OOD detection benchmark, which evaluates the maximum possible feature overlap. For example, an Adjacent OOD benchmark on the ICML Facial Expressions dataset may contain the same face with different expressions, resulting in significant feature overlap. These existing benchmarks do not provide sufficient safety guarantees in applications where there may be significant overlap between IDD and OOD data.

---

Published as a conference paper at ICLR 2025
4.4 IMPLICATIONS FOR OOD FROM UNLABELED DATA
While methods that utilize only unlabeled data, such as (Sehwag et al. 2021 Liu et al. 2023 Guille- Escuret et al., 2024), show promising results both near and far OOD tasks, their performance in the adjacent OOD detection tasks depends on the mutual information between the learned represen- tation and the ID labels. Our theoretical work suggests that such methods will perform poorly, if the surrogate task is independent of the labels.
The adjacent OOD detection benchmark can also evaluate the performance of zero shot OOD detec- tion methods. While our theoretical work does not extend to pretraining due to the use of labels, it also still important to consider the performance when OOD data overlaps ID data.
5 EXPERIMENTAL RESULTS
We conduct the following experiments to verify the existence of label blindness in unlabeled O0I) detection methods. All hyperparameters and configurations were the best performing from their respective original paper implementations, unless noted otherwise. Experiments are re- peated 3 times. Note that code for fully replicating experiments of this work can be found at https: /ttp:////tttbb.t
5.1 EXPERIMENTAL SETUP
Supervised Baseline. We use Maximum Softmax Probability (MSP) (Hendrycks & Gimpel 2016) as our baseline supervised method for comparison. We augment the training data using ran- dom rotation, horizontal flip, random crop. gray scale, and color jitter. Images are resized to 64 x 64. We train using stochastic gradient descent with momentum and a cosine annealing learning sched- ule. We train for 10 warm up epochs followed by 150 regular epochs, selecting the weights with the highest validation accuracy. We use a standard ResNet50 architecture.
Self-supervised Baselines. We use two SSL methods to evaluate how representations are learned, SimCLR (Chen et al. 2020) and Rotation Loss (RotLoss) (Hendrycks et al.) 2019). Images are resized to 64 x 64 for both cases. For SimCLR, we augment the training data using random rotation, horizontal flip, random crop, gray scale, and color jitter. For Rotation Loss, crop and horizontal flip. We train using stochastic gradient descent with momentum (and a cosine annealing learning schedule) and employ standard ResNet50 architecture and train for 10 warm up epochs followed by 500 regular epochs, selecting the weights with the best-learned representations. We use KNN classifier determine the best representations during validation the end of each epoch.
only random
To evaluate OOD performance, we use two methods generate the OOD score of each sample, SSD (Sehwag et al. 2022) and KNN, similar to (Sun et al. 2022). SSD considers the OOD score as the Mahalanobis distance of the sample from the center of all in-distribution training data samples. The KNN method considers the OOD score as the Euclidean distance from the N'th nearest neighbor of the test sample to all in-distribution training samples. Both methods are distance based OOD detection and are commonly used with representation learning. We use the same representation mentioned in the previous paragraph.
Unsupervised Baseline. To consider how an unsupervised OOD detection method functions, we evaluate the diffusion impainting OOD detection method proposed by (Liu et al.] 2023) using code provided in their paper's ' linked repository. We utilize the training configuration that generated the paper's main results, which involved an alternating checkerboard mask 8 x 8, an LPIPS distance metric to calculate the OOD score, and 10 reconstructions per image. We modify only the input image size to be 64 x 64 for all datasets and additional experiments to evaluate performance on their alternative MSE distance metric. This method is representative of other generative methods, such Xiao et al. (2020).
Zero-shot Baseline.
To consider how well zero shot learning algorithms perform, evaluate the CLIPN model presented by (Wang et al.) 2023). We utilize their pretrained weights provided their paper's repository and perform zero shot OOD detection on our adjacent OOD detection bench- mark. We evaluate CLIPNs performance using 3 of their paper's algorithms, Maximum Softmax Probability, Compete to Win (CTW), and Agree to Differ (ATD).
7

---

Published as a conference paper at ICLR 2025
5.2 ADIACENT OOD DATASETS
To create the Adjacent OOD detection task, we randomly split 25% of all classes into the OOD set and retain 75% as the ID set. We also repeat our experiments three times with different seeds to account for different splits of the ID and OOD set. Only ICML Facial expressions has major class imbalance for one of its seven classes. See Appendix for examples of the datasets.
The ICML Facial Expressions dataset (Erhan al. 2013) contains seven facial expressions split across 28, 709 faces in the train set and 7, 178 in the test set. The expressions include anger, disgust, fear, happiness, sadness, surprise, and neutral. Self-supervised algorithms may not learn relevant features for distinguishing expressions and instead learn features relevant for distinguishing faces.
The Stanford Cars dataset (Krause al. 2013) contains 16, 185 images taken from 196 classes of cars. The data is split into 8, 144 training images and 8, 041 testing images, with each class being split roughly 50-50. Classes 2012 Tesla Model S or 2012 BMW M3 coupe. This creates a particularly challenging Adjacent OOD task because of the reliance on more subtle features differentiate cars.
typically very fine-grained, at the level of Make, Model, Year, e.g.,
The Food 101 dataset by (Bossard et al. 2014) consists of 101 food categories and 101, 000 images. There are 250 manually reviewed test images and 750 training images for each class. Note that training images were not cleaned to the same standard as the test images and will contain some mislabeled samples. We believe that this should not significantly detract from the Adjacent OOD nature of the dataset.
5.3 EXPERIMENTAL. RESULTS
Experimental results for Adjacent OOD are presented in Table I is apparent that the baseline supervised method performs better than most unlabeled methodss on the Adjacent OOD detection task, In cases where the unlabeled methods exhibits performance as good as random guessing, is likely that the learned representation contains little information about the semantic label. This is contrary to the reported performance improvements presented in unlabeled OOD papers (Sehwag et al. 1 Hendrycks et al. 2019 Liu et al. 2023), as our experimental results suggest unlabeled OOD is significantly worse than a simple MSP baseline.
It is important to note that the zero shot CLIPN method performs well when the label text's usage in pretraining is similar to the label text's usage in the ID data. In the case of the Cars dataset, the pretraining dataset CC3M (Sharma al.. 2018) contains many images captioned with the make and model of various cars, resulting in good performance. The Food dataset also sees similar label usage in the pretraining set. However, the Faces dataset's labels are not aligned. For example, there are multiple images associated with the emotion angry that do not contain human face, such as an image of a angry fist. When there is little or no mutual information between the pretraining data and the ID labels, zero shot methods will perform poorly in OOD detection tasks. Examples of pretraining data are provided in the appendix G
In appendix F.I we show adjacent OOD results for CIFAR10 and CIFAR100. We observe decent OOD performance on the unlabeled SimCLR compared the labeled supervised MSP. This likely because the SimCLR algorithm is better at learning the relevant features in these datasets and that the classes are more visually dissimilar, resulting in less overlap of OOD and ID data. appendix F22 we show strong results far OOD performance for SimCLR based OOD detection, which confirms findings in papers that test unlabled OOD methods against a far OOD detection benchmark, (Sehwag
et al. 2021 Tack al. 2020 Liu et al. 2023 Guille- Guille-Escuret et al. 2024 Wang et al. (023).
6 RELATED WORK
Detection. Yang et al. 2021 defines OOD detection as the detection of a semantic shift. A semantic shift is a shift in the label space, where the label for a sample does not ex- ist in the training set. OOD detection is crucial in applications where failure is very costly and/or the probability of OOD inputs are very high, as in autonomous driving (Huang et al. 2020). Following (Hendrycks& & Gimpel 2016), there have been many advances in supervised OOD detection. Some
of these improved methods include those based on the ODIN score (Liang et al. 2017). Mahalanobis
distance (Lee al. 2018), energy score (Liu et al. 2020), minimum other score (Huang & Li 2021) and deep Adjacent-neighhenn through the use of label information during training.
(Sun et al. (022). These methods differ from self-supervised methods
8

---

Published as a conference paper at ICLR 2025
Table I: Results from experiments across various datasets and methods. Unlabeled methods perform poorly in adjacent OOD detection. CLIPN performance is due to labels present in the pretraining
dataset and is discussed in section 5.3
Higher AUROC and lower FPR is better.
Faces
Cars
Food
Method
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
Supervised MSP
70.8+0.3
88.2+0.2
69.2+0.9
88.8+0.8
78.8+1.2
81.1+1.6
SimCLR KNN
52.0+4.2
95,0+1.3
52.5+0.4
94.0=0.5
61.1+2.8
91.6#1.6
SimCLR SSD
55.0+4.5
95.1+2.0
52.7=0.7
93.7+1.1
64.440.8
89.3+0.5
RotLoss KNN
46.1
95.8+0.4
51.1=0.6
94.8+0.7
49.7+3.8
94.9=0.9
RotLoss SSD
46.6#3.0
95.7+0.5
50.7+1.9
95.0+1.2
50.7+3.6
94.9=0.9
Diffusion LPIPS
54.7#4.6
94.2+3.7
53.8+1.8
93,9+1.2
52.942.2
94,4=0.6
Diffusson MSE
55.3+2.2
9..2+1.4
51.6#1.6
94.4#0.5
52.5+3.4
94.2+0.6
CLIPN CTW
47.0=1.4
97.340.3
65.0=5.1
69.429.4
70.942.9
69.1=7.0
CLIPN ATD
44.2+1.4
97.5+0.2
81.1+4.3
56.6+10.4
84,9+0.2
53,9#4,5
CLIPN MSP
58.7+4.4
95.9+1.4
76.5+1.4
75.440.6
80.5+1.6
74.0=1.4
Self-Supervieed and Unsupervised Learning. Unsupervised learning involves finding underly- ing patterns within unlabeled data. Diffusion models (Ho al.) 2020) and generative adversarial networks (GAN) (Karras 2017) can be considered as unsupervised learning. Most DNN unsuper- vised learning methods define some task based learning objective, such as the reconstruction task via autoencoders (Baldi 2012) the discrimination between real and fake data in GANs (Creswell cct al. 2018).
SSL can be considered a variation of unsupervised learning that focuses learning representation Z from input X, with respect to task Y, such that ((; Y) is maximized and / [X; Z) is minimized. be considered SSL with respect to their learning objectives. Notably, (Shwartz-Ziv Ziv & LeCun (023) provides a unified information- theoretic view of SSL.. Recent advances in SSL methodology include joint embeddings based on SimCLR (Chen et al. 2020) and SimSiam (Chen He 2021). 1) as well as those based on generative
The unsupervised methods referenced in previous paragraph
models (He 2022).
Unlabeled Diitri-Diitiittiinn Detection. Unlabeled OOD detection methods include self su- pervised and unsupervised methods. Self-supervised OOD detection methods can vary significantly in their definition of the term self-supervised. Methods that train on OOD information (Mohseni the trained OOD sets. Methods
et al. (020)) inherently biased towards better performance accessing in-distribution labels (Vyas et al., 2018) are not self-supervised. We consider the self- supervised OOD detection any OOD method that does not access in-distribution labels out-of-distributie set. With this definition, all self-supervised OOD methods must contain an SSL objective and some way determine the OOD score based on model output, which is often times metric based on the model's 's learned representation(s). Many unsupervised OOD detection methods
the
fall under this definition as well, such as (Daxberger & Hernández-Lobato 2019: Liu et al. 2023
Xiao et al. 2020).
Methods based on representation and scoring were presented the following key efforts. (Hendrycks et al. 2019) combines rotation loss and rotation score whereas (Sehwag et al. (021) utilizes SimCLR (Chen et al., 2020) and the Mahalanobis distance. (Khalid et 2022) combines adversarial contrastive learning with singular value decomposition while (Zhang et al. 2021), in con- trast, combines flow-based generated models with the Kolmogorov- -Sminov distance. The method of (Xiao et al. 2020 is based on integration of variational auto encoders with likelihood regret.
Contrastive representation learning methods have been used to improve the robustness of supervised OOD detectors. (Sun et al. 2022) uses supervised contrastive learning (Khosla al. 2020) to improve the performance of a KNN-based OOD detector. Most recently, work by et al. 2024) utilizes maximum mean discrepancy combined with contrastive SSL..

---

Published as a conference paper at ICLR 2025
Zero shot OOD detection methods are more recent development in unlabeled OOD detection. These methods utilize the CLIP model presented in (Radford et al. 2021 with two notable and recent publications (Esmacilpour et al. 2022) and [Wang et al. 2023)
7 DISCUSSION
7.1 IMPACT OF LABEL BLINDNESS ON FUTURE RESEARCH
A consequence of the label blindness theorem is that there cannot exist single unlabeled OOD detection algorithm for all unlabeled data. However, unlabeled learning methods, such as SimCLR, vital for improving OOD detection. The model of (Sun et al. (022) learns representations using a supervised version of SimCLR, similar to (Khosla et al. 2020). The combination of multi- view information bottleneck with supervised classes produces more robust representation of the in-distribution data than using only a supervised loss. Recent work by (Du et al. 2024a) provides a strong theoretical basis for why unlabeled data can improve OOD detection performance.
Future work should focus on overcoming approximate label blindness through few or one shot meth- ods that can better incorporate label information. Such methods could incorporate a tiny amount of labeled data to avoid the complete independence condition described in Theorem More work needs to be done to determine how much label information sufficient for the adjacent OOD detec- tion detection benchmark.
7.2 IMPACT OF LABEL BLINDNESS ON REAL WORLD PROBLEMS
One can still enjoy the benefits of unlabeled OOD methods when the consequences label blindness acceptable. For example, if the objective was to detect any disease at all, then detecting novel disease as in distribution would not problematic. Unlabeled OOD can also be used in cases where the risk of adjacent OOD) data is acceptably low. The risk defined by Theorem 4.1 is less relevant when randomness in the collection ID data can be ensured. For example, an ID dataset of World War 2 aircraft would not be biased by the collection date and the risk of overlapping OOD data can be reduced to effectively zero.
Unlabeled OOD detection can also work well when the learned features are relevant for the OOD setting. In the case of adjacent OOD detection, an unlabeled method should perform well if the learning objective is closely related to the ID labels. Alternatively, unlabeled OOD detection can be in used cases where one expects only near and far OOD data.
CONCLUSION
In this work we provide an answer to the question, can we ignore labels for OOD detection? Our theoretical work shows that the answer no, unless the unlabeled method happens capture the relevant features and does not need to work for different sets of labels. Due to the lack of existing benchmarks that capture the theoretically expected failure, we introduce novel type of OOD task, Adjacent OOD detection. This task addresses the critical safety gap caused by significant overlap of ID and OOD data. We show that the Adjacent OOD task accurately captures the failure in unlabeled OOD detection that is hypothesized by our theory. We hope our work will help support more robust research into OOD detection and improve the safety of Al applications.
ACKNOWLEDGMENTS
This material is based upon work supported by the United States National Science Foundation under grant #2225354.
The authors acknowledge Research Computing at the Rochester Institute of Technology for provid- ing computational resources and support that have contributed to the research results reported in this publication (RIT) 2024).
REFERENCES
Mihalj Bakator and Dragica Radosav. Deep learning and medical diagnosis: A review of literature.
Multimodal Technologies and Interaction, 2(3):47, 2018.
2018.
10

---

Published as a conference paper at ICLR 2025
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 37-49. JMLR Workshop and Conference Proceedings, 2012.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 -aa--n discriminative com- ponents with random forests. In European Conference on Computer Vision, 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35 (1):53-65, 2018.
Erik Daxberger and José Miguel Hemándezz-oobto. Bayesian variational autoencoders for unsu- pervised out-of-distributio detection. arXiv preprint arXiv: 2019.
Nick Drummond and Rob Shearer. The open world assumption. In eSI Workshop: The Closed World of Databases meets the Open World of the Semantic Web, volume 15, pp. 2006.
Xuefeng Du, Zhen Fang, Ilias Diakonikolas, and Yixuan Li. How does unlabeled data provably help cattof-distribution detection? arXiv preprint arXiv:2402.03502, 2024a.
Xuefeng Du, Yiyou Sun, and Yixuan Li. When and how does in-distribution label help out-of- distribution detection? arXiv preprint arXiv:2405./8635, 2024b.
Dumitru Erhan, lan Goodfellow, Will Cukierski, and Yoshua Bengio. Challenges in representation learning: Facial expression recognition challenge, 2013. URL
Sepideh Esmacilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot ditoo--distribution detec- tion based on the pre-trained model clip. In Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 6568-6576, 2022.
Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribttion detection learnable? Advances in Neural Information Processing Systems,
2022.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07011, 2020.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of -uitrf-disttibution detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
Charles Guille-Escuret. Pau Rodriguez, David Vazquez, Ioannis Mitliagkas, and Joao Monteiro. Cadet: Fully self-supervised out-of-distribution detection with contrastive learning. Advances in Neural Information Processing Systems, 36, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rxiv. 2016.
Dan Hendrycks, Mantas Mazcika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.

---

Published as a conference paper at ICLR 2025
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distributio detection for large semantic the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
space. In Proceedings the pp. 8710-8719, 2021.
Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp. Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100 2020.
Tero Karras. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 17X1:11110.11990 2017.
Umar Khalid, Ashkan Esmaeili, Nazmul Karim, and Nazanin Rahnavard. Rodd: A self-supervised approach for robust out-of-distribution detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 163-170. IEEE, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) Sydney, Australia, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of -uttof-distribution image detection in neural networks. arXiv preprint arXiv:1706.02699, 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of -fiisrribitttiionnn detec- tion. Advances in neural information processing systems, 33:21464 211775,4-21475, 2020.
Zhenzhen Liu, Jin Peng Zhou, Yufan Wang, and Kilian Q Weinberger. Unsupervised out-of- distribution detection with diffusion inpainting. In International Conference on Machine Learn- ing. pp. 22528-225386 PMLR, 2023.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 5216-5223, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Manikandasriram Srinivasan Ramanagopal, Cyrus Anderson, Ram Vasudevan, and Matthew Failing to learn: Autonomously identifying perception failures for self- driving IEEE Robotics and Automation Letters, 3(4):3860-3867, 2018.
RIT.
Research computing services, 2024.
URL https://www.ZZ wtt.:tttetdd.//////////
reseerchhcoppttinn ing//
M Robert. Fano. class notes for mit course 6.574: Transmission of information. MIT. Cambridge, MA, 8:33, 1952.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2 /2051, 2021.
12

---

Published as a conference paper at ICLR 2025
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-44 423,, 1948.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Longg Pupers), pp- 2018
Ravid Shwartz-Ziv and Yann LeCun. To compress not compress compress-self--eeee self- supervisd learning and information theory: A review, arXiv preprint 23Xiv:334.003355 2023.
Ravid Shwartz-Ziv and Naftali Tishby, Opening the black box of deep neural networks via informa- tion, arXiv preprint arXiv: 1703.07003.000 1000000 2017.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference Machine Learning, pp. 20827-20840 PMLR, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 13.11839-11852, 2020.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore Willke. Outtoo-diistriuutin detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 550-564, 2018.
Hualiang Wang, Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1802-1812, 2023.
Mei Wang and Weihong Deng. Deep face recognition: A survey. Neurocomputing, 429:215-244, 2021.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An distr--dit detection score for variational auto-encoder. Advances in neural information processing systems, 33:20685- 20696, 2020.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2 2021.
Lily Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-of-distribution detection with deep generative models. In International Conference on Machine Learning, pp. 12427-12436. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Appendix
A IMPACT STATEMENT
This paper presents work whose goal is to advance the field of machine learning. In particular, this paper seeks improve the theoretical understanding of safery in machine learning based systems. We hope to positively impact society by improving the work of future researchers and practitioners building safer AI systems.
B PROPERTIES OF MUTUAL INFORMATION AND ENTROPY
In this section we enumerate some of the properties of mutual information that are used to prove the theorems reported in this work. For any random variables w, x, y and z
(Pt) Positivity:
I(x:y) 2 0, (x:: z) 2 0
(P2) Chain rule:
(xxy:)== 1y;:)) + 1 1(x;z
(P3) Chain rule (Multivariate Mutual Information):
(xx;y;2) 1(y; ( x)
(Pa) Positivity of discrete entropy: For discrete x
H(x) 2 0, Hx y) 2
(Ps) Entropy and Mutual Information
H(x) H(x y) T(x;y)
(P&) Conditioning a variable cannot increase its entropy
H(ylz) < H(y)
(Pr) A variable knows about itself as much any other variable can
Ixx;x) 2 ((x;))
(Ps) Symmetry of Mutual Information
I(x:y) x
(P&) Entropy and Conditional Mutual Information (This is simply Ps conditioned on z)
I(x: y|z) = (x|z)) (xlyzz)
(P1o) Functions of Independent Variables Remain Independent
7(x;y)= = T(((x):))
14

---

Published as a conference paper at ICLR 2025
C
THEOREMS AND PROOFS OF PREVIOUS WORK
This section contains the supporting theorems and proofs provided by previous work (Federici et al., 2020).
When random variable z is defined to be a representation of another random variable we state that z is conditionally independent from any other variable in the system once x is observed. This does not imply that must be deterministic function of x. but that the source of stochasticity for independent of the other random variables. As result whenever z is representation x
I zz a xb) = O,
for any variable (or groups of variables) and b in the system. This condition is accounts for the randomness experienced in training neural networks and the error expected from human labelers. This condition applies to this and the following sections.
C.1 SUFFICIENCY
Proposition C.1. Let and be random variables from joint distribution p(x,y). Let be representation of x, then z is sufficient for y if and only if [(x:y) = T(y:2)
Hypothesis:
(H1) is a representation of : Ily; x) =
Thesis:
(T)) 1(xy) 2) =0 1(x:y) 1(y;z)
Proof.
I(x;y 1z) (Pt) I(x:y) [(x;y;z)
(Pa) I(x:y) I(y:z) + I(y: 2 x)
(H) I(x;y) I(y:z)
Since both y(x;)) and 7(y:z) are non-negative (Pi), [xxy 2) =0 0 1(y; = ((x;y)
D MAIN THEOREMS AND PROOFS
We ignore cases where the determined variable has entropy of Generally, Hy]
H (y) > 0. Also, we only consider cases where the random variables have more than zero entropy.
Note that Rx represents the support of random variable such that Rx (x RR P(æ) 0}.
D.1 LOWER BOUND OF MUTUAL INFORMATION FOR SUFFICIENCY
Lemma D.I. Let x and y random variables with joint distribution p(x, y). Let be a represen- tation of x that is sufficient, as per definition 2.1 Then [xx; z) 2 1(z;y) and I [(x; z) 2 l(x;y).
Hypothesis:
(H1) is a representation of T(y; x) =0
(H2) z a sufficient representation of x: ylz)) =
Thesis:
(T:) Vz.I(x;z) 2 I(z:y), 2 I(x:y)
Proof. By Construction
15

---

Published as a conference paper at ICLR 2025
[(xylz)) (12)
[zy; x) [(;;x
[(x;y) + I(x: z/y) - I(2; x)
(PropB1) I(z:y) + I(x:2ly) I(2;x)
[(z; = I(z;y) + 1(x:
(Pi)
I(z;x) 2 y(z
Note that y(z;y) y) for all sufficient representations, per proposition
C.1
This supports our intuition that the information in the representation consists of relevant information [(z; y) and irrelevant information [(x; zly). By definition of sufficiency, there must be enough information for (z; y) in (x; which is to say that the size of the encoding cannot be smaller than the minimum size to encode all of yx;
D.2 CONDITIONAL MUTUAL INFORMATION OF NOISE
Lemma D.2. Let and be independent random variables and be function of x with joint distribution p(x, y,2). 2). The conditional mutual information [(x; z is always equal to the mutual information [x; z). As in the information content is unchanged when adding noise.
Hypothesis:
(H)) Independence of x and y =(x;y) = 0
(H2) z is fully determined by x: H(zix) =
Thesis:
(Ti) 1(x: 1xx;
Proof. By Construction.
(C1) Demonstrates that Hz)xxy)
(P4)
(P))
2 H(zlxy) H(z)x)
(Ha)
H(z|xy) 3 0
(C2) Demonstrates that ((zzy) = 0
(Ha) I({(x)))
I(z;y)
(Pio) ((x;y)
(Hi)
Thus
16

---

Published as a conference paper at ICLR 2025
I(x: (P)
H(zly) H(z(xy)
H(lly)
(P) ((z) ((zyy)
(Ca H(z) -0
(8a) H(x) H()xx)
(B) ((x;2)
This supports the intuition that if one added random noise channel it will not change the mutual information.
D.3 FACTORIZATION OF BOTTLENECK LOSS
Lemma D.3. Let x be random variable with label y such that H Hyy)x) = and is a sufficient representation of x for y. The loss function C = I(x; (2; is equivalent to L = H()) (I,yy)) with B as some constant.
Hypothesis:
(H1) z is fully determined by x H(((x)
Thesis:
(Ti) 1(x: 3I(z;)) = H(z) (1(z))
Proof. By Construction.
I(x;2) - (I( (Ps) H(z) H(zlx) 3I(2,y)
(H)) H(z) 31(z.y)
Due to the relationship between and z, create intuitive factorization of the bottleneck loss function. Effectively, we want to maximize y(z:)) while minimizing the information content of
D.4 STRICT LABEL BLINDNESS IN THE MINIMAL SUFFICIENT STATISTIC
Theorem D.4. Let come from distribution. is composed of two independent variables x1 and
X2. Let yı be a surrogate task such that H x1) ) = 0. Let z be any sufficient representation of x for yı that satisfies the sufficiency definition 2.1 and minimizes the loss function = 1(x1x2;; 31 (z; The possible z that minimizes C and is sufficient must meet the condition 1 [xx;; z) = 0.
Summary This proof uses the derivative of the loss function establish the possible set of local
minima that satisfies C. For any possible minima of C, the representation z must contain information of only x1- H(z
only X2- H (z|x2) We show that possible set of all local minima must satisfy H (z) = by showing that the other two cases must always have greater L. This proves the Theorem that the learned representation cannot contain information about X2.
0) or both X1, X,,X2 3,,22>> H xx,x)) 0.
(H1) fully determined by H(2xx) =0
17

---

Published as a conference paper at ICLR 2025
(H2) is a representation of I(y; xx) = 0
(H3) z a sufficient representation of y(x: =
(H4) x is composed of two independent variables X1,X2 x (xx1xx)) = 0
(Hs) y is fully determined by x1: Hy|xx)
Thesis:
(Ti) Vx.I 2) =0
Proof. By Construction
(C1) demonstrates that L H(z) (I( via factoring (xx1x2;2) Alternatively, Theorem D.3 creates the same result.
(Tz) [(x1;)) + 1(x2;2|x1)
[(x1, X2;
(fb) H(z) (z|xx)) + 2(x2;2/x1)
(Ra) H(z) Hz(xx) ) H((xx1) - H(22x1x2)
(8)) H(z) Hz)xx)) + H(2|xi) -0
C (z)) (T(zyy)
(C2) Demonstrates that ((z;y) = 1 (x: y) as per Theorem
(C3) Demonstrates that ((2;y) is a constant across all sufficient representations because Theorem applies.
(CA) Demonstrates that for all possible z satisfying (Hs), their loss be compared using only
C, (z) for comparing across z
dc (Ct) (2) dz
(1
dz
dz
(Ca)
H
dz
(Cs) Demonstrates that the value of (z) at all possible that minimizes L is the same. Even for different minimal& z, they must have the same (z) to minimal. When comparing possible minimal solutions to C, H Hz)) is constant across all minimal solutions.
(C&) Demonstrates that any z that satisfics sufficiency must satisfy (z;x) 2 ((z and [(z;)) 2
yx;;) as per Theorem D.1
(C+) Demonstrates that minima(s) exists only where (z) [(; y) and H(z)x) = Note that
(z)) ((x;y) ((; is the most compact representation size that is sufficient.
(Cn) 2 T(z;y)
T(z:x)
(P5)
H(z) H (zlx) 2 ((z;y)
Vz|Co Ha I(z;x) > [(z; () 3x''z' = f(z) [(2x)) > 1(z';x) A Co A H3
From (C7) there exists only 3 types of minimas, separated by their dependence on the variables X1, X2. As per (H1), any z must follow one of the types.
1. Dependent only on X1: (z|H(2/X1)) 0 - 7(x;;2) = 0
2. Dependent only X2: Vz (z/xx2) 7(x2;2) > 0
18

---

Published as a conference paper at ICLR 2025
3. Dependent on both X1X2: Vz (z), A (z//x,) > A H(z [(x2; z) >
From here we will show that all type and type 3 minimas always fail (H43) or have greater C than any type 1 minima.
Type X1: Vz (z(x)) = =0 [(x2; z) 0
(Cs) Demonstrates that there exists (z) = I (z: I (x1: z) and it is set of minimas satisfying
(CT). This also establishes an upper bound for solutions L due (C₃). Therefore, any solution for type type 2, and type must satisfy y2: I (xx1;2) to be sufficient and (2; = [xx1;2) be minimal.
(Ca) I(z:y)
1 I(z;x)
(11
T 7(x1, X2;
(P%)
_ (xx2 z) + (10))+((x100 ((x1:2)(x2)
(Type1)
0 0+ [[x1;
(Theorun
D22 S
T(x1;2)
(P))
S (1z)) (z
3 {x)1(x1;z) l(zzy) = [x; (xxy)
(Cg) Demonstrates that there exists H(z') < (z) that satisfies sufficiency if z satisfies (Cs) and is also f(z; x2) = O.
Cs [(x1;2) T(x:y)
H H(z) [(xiiz) =(x);z)
(C2) 7(x1;=)) [(x1;2) T(y:2) T(x:y)
Type 2 X2: (2)x() = + 1(x2;2) > 0
(C1o) Demonstrates that no type minima exist, simply because it would contain no information regarding X1, thus failing to satisfy (Ha). This is because z cannot contain any information about X1, otherwise we would not satisfy (z 0. If the representation contains information about y, then it is not sufficient.
Hz) = fxx2)
0 (144) [(x1; X2)
(P1o) 1(f(x)):x2)
(H%) ((y:x2)
(Pio) T(y; [ (xx2)
0 = ((yz)
Type 3 x;; (z|x x2) 0 (z > 0 H 0 - I(x;;z)
(Cn) Demonstrates that any z that could be minimal must also satisfy (Cs) for sufficiency. Note that (Cs) implies that any 7(x;;z) > T(x;; y) not minimal.
19

---

Published as a conference paper at ICLR 2025
(Ce)
{(zy) S x(z
(H4)
1xx1x2;2)
(P))
T(z;y) - (xx12)) + 1(x2;2/xx)
(Cx) + [(x12)) = I(z;y)
(C12) Demonstrates that any z' where (z2; > [(z; x2) and (z; x2) that maintains (zz') = (z) results in solutions that are not sufficient required by (Hz) because know that the size of the representation must at least 1(x; y) as defined in (C₂)
Cs + H (z) constant across all minima
Cs + H (z) 11 (z') for z' to be minimal
Cs + [(x1; z) = [(x:y)
I(xg + H (z/xx1) =
Vz' (xx'1(x);22)) 0: (xx'(x)) H(z)x)) (1))x1)
H(z((x)) (z) + H(z') H(z'l ((1) < H(z)
H(zxx)
(Pt) [(x12)) < [(x;z))
(C6) [xx z²) < yxx;y)
(Cis) Demonstrates that combining (C11) and (C12). there is type solution that has equal to the minimal type solution that also maintains sufficiency (Ha) and (Cs). This confirms the definition of entropy, in that encoding more independent information requires more bits nats.
This means that only a type solution can be both minimal and sufficient, which proves the thesis.
compare the losses of all sufficient solutions with L Hz).. Of those sufficient solutions, the one that minimizes L is the one with the smallest H (z). The minimal sufficient representation is that captures only all of y(x and nothing else. Thus the minimal cannot have T(x2; 2) > because such z would encode information outside of T(x1;y).
To summarize this proof,
D.5 INDEPENDENCE OF FILTERED DISTRIBUTIONS
Lemma D.5. Let x come from a distribution. x is composed of two independent variables x1 and X2. For X2 where Ray Rayg there exists X2 such that H (x1[x2) (xx)).
Summary This proof uses the chain rule of mutual information to show that contradiction arises x could filter in non random way.
Hypothesis:
(H1) X² is fully determined by x2 11 (x2) = 0 where Ray C Raz
(H2) Independence of and y2 1(x1;x2)
Thesis:
(Ti) (x)x < H(xi)
Proof. by contradiction H(xi (x2) < (x1)
(C1) Demonstrates xxx2;'x)= x(x);xb))
20

---

Published as a conference paper at ICLR 2025
(P4) H (xx)) - (x2
[(x2; xx)
(Hi) H(xg)
(P)) H(x)) (x22(xx)
(Pa) x(x2xx6)
(C2) Demonstrates ((x2:x1(x22) =0
(P)) [xxxxxxx1) -apxx)))(((x)))) (xx2;X2)
(C1
[
[x2xx22) 1xx2ixx) x²y
(14)
X12; I(x2;x1|X0)
S - (xx1)2x1) - I(x2;x)) xS)
(P))
2 0
0
(C₂) Demonstrates x() via non independence implied by -Ti
Contradiction arises when we consider symmetric applications of the chain rule to 7 ((x1;xxx))
T(x1;x(x2) (Pa)
((x1;x2) [(x1;x2|xx)
(Cs)
7(x1;x/x2)
0
1(x1:x2()
xx)
Since (xx: cannot be both zero and greater than zero, -T creates a contradiction, which supports Ti.
easy to confuse this with the existence of non independent subset An B, where A, B are independent events. However, this example violates (H1), since we cannot determine C using only A or only B.
D.6 STRICT LABEL BLINDNESS FILTERED DISTRIBUTIONS GUARANTEED OOD FAILURE
Corollary D.6. Let x come from a distribution. is composed of two independent variables x and x2. Let y1 be surrogate task such generated by y1 = fi(x1) (y)1xx)) 0. Let y2 be a label such that H (y2)x2) = O and y2 = f2(22). Let Vin be as subset of labels Yin C Ryy Let x' be subset of x where Rx Rxx n {x R. fz(Ø2) E Yin) such that x' is composed of independent variables x and x², and y4 = (x( The sufficient representation z learned by minimizing L = [(x], xzzz) 3I(z,y') must have I(x;; and I(y2; 0.
Summary
This proof combines Theorem D.5 and Theorem D.4
Hypothesis:
(H1)& fully determined by H(z[x)
21

---

Published as a conference paper at ICLR 2025
(H2) is representation of Ily; xx) =
(H3) z a sufficient representation of y(x: =
(H4) x is composed of two independent variables X1,X2 x (x1;x2) =0
(Hs) y is fully determined by x1: H( 00
(Ha) x' is subset of x filtered by Vin Rxx Rx 0 fær E ER f2(222 E Vin)
Thesis:
(T:) (xx;(x = [xx2:
Proof. By Construction.
(C1) Demonstrates that xxx1;x)) = 0 due to Lemma D.5
Using (Pto), we know that independent functions stay independent and thus x2) 0, x2) = O. From Theorem D.4 we know that encoding an variable independent of the tar- get y results in higher loss, therefore (x2;z) 0 and / (x2;; 2) 0 since both independent of x1.
By combining Lemma D.5 and Theorem D.4 we know that any surrogate learning objective inde- pendent of a downstream objective (say classifying labels) results in a representation containing no information for the downstream objective. If contains no information for one objective, it contains no information for derivitives of that objective (eg. no label information means no OOD detection information).
D.7 UNAVOIDABLE RISK OF OVERLAPPING OUT OF DISTRIBUTION DATA
Theorem D.7. Let x come from distribution. Let f be some labeling function to generate labels y such that y f(x), where there are at least two unique labels Ry > Let Xin. be random subset of x where Rx... & Rx and [Rx,.l < cc. Let yin be labels generated from Yin = f(ri))) The probability that randomly selected contains y not present in Ry.. is always greater than O.
Hypothesis:
(H1) x comes from any distribution
(H2) is label generated from function y = (ar) such that RRyl >
(H3) X; random subset of x where Rxx. S Rx and Rxxil < 0o and yin f (xx.).
Thesis
(Ti) Vx. & Ry,.. 0)
Proof. by contradiction (-7i) P(f(x) & Ry...)=0)
(C1) Demonstrates that Ry.. Ry because there must exist sample such that f(x)
Ry...
(C2) Demonstrates that P(f(x) yn)) 0, where yn Ry
Contradiction arises when we consider that it is possible to sample the same label yn for any finite number of repetitions, per (C2). This would create label yn Thus, there always exists Ry.. ç Ry which contradicts (C1).
of any finite size consisting only of the
More realistically, Ry.. consist of all elements of Ry except and still guarantee P(f(x) & Ry > 0).
22

---

Published as a conference paper at ICLR 2025
E
SAMPLE IMAGES FROM DATASETS USED IN THE EXPERIMENTS
Space intentionally left blank.
23

---

Published as a conference paper at ICLR 2025
Figure 2: From left to right, sample images from the datasets ICML Facial Expression, Stanford Cars, and Food 101. These datasets contain classes that are visually similar, in contrast to CIFAR 10, which includes classes such as airplane and dog that are not visually similar.
The above figure shows sample images from the datasets used in the experiments.
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 ADIACENT OOD ON CIFARIO AND CIFARI00
See table for results on CIFARIO and CIFARIO0 adjacent OOD benchmarks. SSL methods per- form better than the Faces, Cars, and Food dataset.
CIFARTO
CIFARTOO
Method
AUROC
FPR95
AUROC
FPR95
Supervised MSP
85.345.9
73.0#9.1
78.3+0.9
80,9+1.2
SimCLR KNN
77.648.0
75.6#6.0
68.8+1.7
88.8+1.9
SimCLR SSD
77.648.0
69.1
70,2+1.2
88.2+2.4
RotLoss KNN
71.4+9.1
83.3+8.3
48. 182.2
94.2+1.5
RotLoss SSD
71.1+6.7
82.6+9.0
47.9+2.3
96. 10.55
Table 2: Adjacent OOD Performance on CIFARIO and CIFAR100.
F.2 FAR OOD DETECTION PERFORMANCE
A SimCLR KNN based SSL OOD detection method performs extremely well on far OOD tasks.
See 3
Data vs
AUROC
FPR95
OOD CIFARIO
ICML Faces
99.740.1
0. 1
Stanford Cars
10..1
8.19
Food 101
99.8+0.1
0.0+0.0
OOD CIFARIO0
ICML Faces
99.7+0.1
0.2+0.1
Stanford Cars
99.2+0.2
1.5+0.1
Food 101
99.4+0.1
1.440.1
Table 3: Using CIFARIO and CIFAR as OOD sets, we see that far OOD detection performance for the SimCLR KNN method is very good. Unlabeled OOD detection methods tend to perform very well in far OOD tasks, see (Sehwag et al. 2021 Tack et al. 2020 Liu et al. 2023 Guille-Escuret
al. 2024 Wang et al. 2023
24

---

Published as a conference paper at ICLR 2025
G REVIEWING CLIPN PRETRAINING DATA
We consider sample images from CC3M that contain the label from their respective benchmark. We observe figure 3 that images containing the word angry often do not contain a human face. This in contrast to the labels for the Cars and Food dataset, where the pretraining data is very similar the benchmarking data, see figure and
sck
shut
$ stZEN
AD9191160
WWWACEESIOOAS..O 170903583
Figure Comparing Images from the CC3M Dataset with captions containing the word angry. These are drastically different from the images in ICML face dataset. Images captioned with other facial expressions also tend to lack a human face.
Figure 4: Comparing Images from the CC3M Dataset with captions containing the word BMW Series, Dodge Ram, and Honda Odyssey, left right. These quite similar to the images in Cars dataset
Figure 5: Comparing Images from the CC3M Dataset with captions containing the word Caeser Salad, Donut, and Pizza, left to right. These are are quite similar to the images in Food 101 dataset
25


Reviews:
[{"review_id": "XZKtEf93vB", "reviewer": "Reviewer_EZPK", "summary": "The paper addresses challenges in out-of-distribution (OOD) detection, focusing on \"label blindness,\" where OOD algorithms fail due to a lack of labeled data. It proves that unlabeled OOD detection inherently fails when label information is not linked to the learning objective, proposing an Adjacent OOD benchmark to capture these failures. Experimental results reveal that existing unlabeled methods struggle with overlapping IID and OOD data.", "questions": "- Could the benchmark be vulnerable to artificially low variance during testing due to potential biases in data selection or label overlap?\n- It seems that the notion of of \"label blindness\" is central to the theory of this paper. How do you address practical situations where correlations between features and labels might enable OOD detection without explicit labels, and would this differ from the guarantees provided by you theoretical assumptions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- theoretical foundation proving that unlabeled OOD detection methods are inherently limited by \"label blindness\" -- a fundamental flaw in current self-supervised and unsupervised OOD approaches.\n- novel OOD detection benchmark to test OOD methods under conditions of high overlap between in- and out-of-distribution data, filling a critical gap in safety evaluation.\n- Extensive experiments that demonstrate current unlabeled OOD methods often fail in real-world scenarios with overlapping ID and OOD data, highlighting the need for label-aware approaches.\n- Offers clear guidance for future OOD research by identifying key challenges and proposing benchmarks that better capture the limitations of current methods.", "weaknesses": "- Limited applicability of theoretical claims -- while the paper introduces a \"Label Blindness Theorem,\" rely heavily on specific assumptions about mutual information and independence (e.g., Theorem 3.1 and Corollary 3.3), which may limit the generalizability of the \"label blindness\" concept to real-world applications where these assumptions don't hold.\n- the practical applicability of this theory is not fully explored and assume idealized conditions (Section 3.1) which might not reflect the complexity of real-world OOD scenarios.\n- Relies heavily on specific datasets like ICML Facial Expressions, Stanford Cars, and Food-101, which limits generalizability, as these may not capture the diversity of OOD detection in other modalities such as language or video or audio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ILqLO4TgPl", "reviewer": "Reviewer_kvex", "summary": "This paper mainly focuses on the Out-of-distribution (OOD) detection problem in self-supervised learning settings, which investigates the conditions for the theoretical guarantee of failure in unlabeled OOD detection from the information-theoretic perspective. In summary, this work first identifies one scenario in which any self-supervised learning or unsupervised learning algorithm will fail when there exists zero mutual information between the learning objective and in-distribution labels and then introduces a new detection task, namely adjacent OOD detection, which is theoretically proved to possibly exist in real-world dataset. Experiments with several existing self-supervised and unsupervised learning OOD methods are conducted to support the major claims of this work.", "questions": "1. It seems to be an unknown reference at the end of the first key contributions on Page 2.\n2. Although the adjacent ood is discussed to be different from those previously defined near ood or far ood scenarios, what is the difference between adjacent ood with anonymous outlier detection if there is significant feature overlap at both settings?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper focuses on a new setting, i.e., self-supervised learning OOD, and investigates the feasibility of OOD detection from a theoretical perspective.\n2. This paper theoretically proved the existence of failure conditions for unlabeled OOD detection algorithms, namely, label blindness, providing insights from the information-theoretic perspective.\n3. This paper also introduces a new task, i.e., adjacent OOD detection, which is theoretically demonstrated to be possible in the real-world dataset.", "weaknesses": "1. The writing and presentation can be further improved, since there are numerous typos and unclear notation or concept definitions. To list some, there is no clear introduction to the general paradigm of unlabeled OOD detection algorithms, and the preliminary about OOD detection is unclear in how to detect the OOD sample beyond using $P(y\\notin Y_{in}|x)$.\n2. The motivation and practical significance for considering self-supervised OOD detection is not clear. Given the current presentation and introduction content, the reviewer can hardly understand why we should conduct unlabeled OOD detection. If we have the pre-trained model on the ID data, even if we adopt the self-supervised learning in pertaining, can we have the pseudo label for those unlabeled data for OOD detection? It could be better to illustrate the overall research setting using some figure or framework before the existing Figure 1 for a better explanation.\n3. It seems there are no explicit assumptions for the theoretical results and most theoretical results are based on the sufficiency definition in definition 2.1 and the minimal sufficient statistic in definition 2.2. It is questionable whether the theoretical insights reflect the real-world challenge or not, as there is limited empirical justification for those intermediate theoretical results based on mutual information. It could be better if the presentation incorporated more quantitative results regarding the mutual information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsdyUyyJyB", "reviewer": "Reviewer_yEDC", "summary": "This paper provides a theory to explain the failure of unlabeled OOD detection. The authors also define a new OOD task to test for label blindness and conduct experiments to verify the correctness of their theorem.", "questions": "Please address my comments in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The paper is clear and easy to understand.\n\n[2] The theory clearly explains the potential problem of unlabeled OOD detection.\n\n[3] The theory is verified by the empirical studies, raising real concerns about unlabeled OOD detection.", "weaknesses": "[1] While the main theory is sound in this paper, it is intuitive that the unlabeled OOD methods will fail if there is zero information between the learning objective and the label: For example, if we use linear regression to fit random noise, it is supposed to learn nothing. If the learning objective of the unlabeled OOD detection learns every feature but still has zero information with the label, then my understanding is that labeled OOD methods will also fail. Could the authors provide some comments on how the theory should be adapted in such a case?\n\n[2] While Theorem 4.1 provides some justification on the effectiveness of the proposed Adjacent OOD task, there is a gap between Theorem 4.1 and the main theory: even if P(a randomly selected x contains y not present in $R_{y_{in}}$)>0, this is not exactly connected with the mutual information between the learning objective and the label. I would suggest the authors provide some more evidence, either theory or just simple simulation, to provide a more rigorous connection between the main theory and conclusion of Theorem 4.1.\n\n[3] It is not clear how the theory can provide practical guidance on designing better learning objectives of unlabeled OOD. The paper provides the Adjacent OOD task, but this is different from suggesting a way of improving existing design: we know they fails, but we don't know how to improve them. Could the authors provide some comments on this?\n\n[4] Please highlight the technical contribution when deriving the theorems. Since as elaborated in Weakness [1], the main theory is intuitive, it is not clear how difficult it is to prove it.\n\n[5] Writing: please revise the wording in this paper to be more objective. For example, the word \"you\" or \"your\" appear 6 times in total.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)