Pathologies of Out-of-Distribution Detection

Authors: Yucen Lily Li, Daohan Lu, Polina Kirichenko, Shikai Qiu, Tim G. J. Rudner, C. Bayan Bruss, Andrew Gordon Wilson
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Out-of-distribution detection, robustness

Abstract:
There is a proliferation of out-of-distribution (OOD) detection methods in deep learning which aim to detect distribution shifts and improve model safety. These methods often rely on supervised learning to train models with in-distribution data and then use the models’ predictive uncertainty or features to identify OOD points. In this paper, we critically re-examine this popular family of OOD detection procedures, revealing deep-seated pathologies. In contrast to prior work, we argue that these procedures are fundamentally answering the wrong question for OOD detection, with no easy fix. Uncertainty-based methods incorrectly conflate high uncertainty with being OOD, and feature-based methods incorrectly conflate far feature-space distance with being OOD. Moreover, there is no reason
to expect a classifier trained only on in-distribution classes to be able to identify OOD points; for example, we should not necessarily expect a cat-dog classifier to be uncertain about the label of an airplane, which may share features with a cat that help distinguish cats from dogs, despite generally appearing nothing alike. We show how these pathologies manifest as irreducible errors in OOD detection and identify common settings where these methods are ineffective. Additionally, interventions to improve OOD detection such as feature-logit hybrid methods, scaling of model and data size, Bayesian (epistemic) uncertainty representation, and outlier exposure also fail to address the fundamental misspecification.

Decision: reject
Reviewer scores: 3, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730067493184

thread_type:
official_review

<thread_object>
creator: reviewer_s8wv

<content>
summary:
This paper critically re-examines the popular family of OOD detection procedures, which rely on supervised learning to train models with in-distribution data and then use the models’ predictive uncertainty or features to identify OOD points. The analysis reveals deep-seated pathologies. It argues that these procedures are fundamentally answering the wrong question for OOD detection, with no easy fix. Uncertainty-based methods incorrectly conflate high uncertainty with being OOD, and feature-based methods incorrectly conflate far feature-space distance with being OOD. Moreover, there is no reason to expect a classifier trained only on in-distribution classes to be able to identify OOD points. It shows how these pathologies manifest as irreducible errors in OOD detection and identifies common settings where these methods are ineffective. Additionally, it shows that interventions to improve OOD detection such as feature-logit hybrid methods, scaling of model and data size, Bayesian uncertainty representation, and outlier exposure also fail to address the fundamental misspecification.

strengths:
1. This paper introduces a fresh perspective by highlighting that many popular OOD detection methods – whether supervised or generative – are not effectively addressing the core question: Is this unlabeled point from a different distribution? 

2. This paper provides concrete examples illustrating how widely-used OOD detection techniques fundamentally miss the mark for true OOD detection. The paper further demonstrates that both feature-based and logit-based methods inherently suffer from irreducible errors, limiting their effectiveness in this context.

3. This paper is well-structured, with clear explanations, making it easy to follow.

weaknesses:
1. While the paper argues that popular OOD detection methods do not directly address the question of whether an unlabeled point is from a different distribution, these methods have demonstrated strong performance across established benchmarks. It remains unclear whether adopting the approach suggested in the paper would lead to meaningful improvements in performance, raising questions about the practical utility of the findings.

2. The experiments are relatively simple and may not sufficiently support the claims. For instance, the study relies on a small-scale experiment using a LeNet-5 model trained to classify automobiles and trucks from CIFAR-10 to argue that OOD examples often exhibit low uncertainty. However, it is unclear whether these findings generalize to more complex architectures, such as ResNet, or to larger, real-world datasets.

3. While the paper provides a critical analysis of existing OOD detection methods, it does not propose new approaches to address the identified limitations or offer concrete suggestions for overcoming the challenges it highlights. This limits the practical contribution and leaves open questions about how to improve OOD detection in light of the paper’s critiques.

questions:
1. Given that existing OOD detection methods perform well on benchmarks, how do the authors envision a method that directly addresses the question “Is this unlabeled point from a different distribution?” improving current performance? Could the authors provide further theoretical insights or experimental scenarios to clarify when and why their proposed perspective might offer an advantage?

2. Have the authors considered evaluating their findings with more advanced architectures, such as ResNet, and on larger or more diverse datasets beyond CIFAR-10? Doing so could help demonstrate the generalizability of the claims. 

3. While the paper offers a valuable critique of existing methods, have the authors considered proposing potential approaches or heuristics to address the identified limitations? Even if speculative, some guidance or future directions could make the paper’s contributions more actionable for researchers looking to build on the work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629571998

thread_type:
official_review

<thread_object>
creator: reviewer_7rbf

<content>
summary:
Authors claim that current OOD detection methods, often based on supervised models and predictive uncertainty, are fundamentally misguided, as they attempt to answer the wrong questions about whether a sample truly belongs to the training distribution.

Currently, common OOD detection methods involve training a supervised model on in-distribution data and subsequently using the model’s predictive uncertainty or features to detect OOD instances. These approaches are typically divided into two main streams
(1) Feature-based methods and (2) Logit-based methods.

However, authors argue that these methods inherently answer the wrong question because they do not measure distribution membership but rather whether a sample leads to unexpected model representations. First, feature-based methods fail when features of OOD and in-distribution data overlap or are indistinguishable, leading to irreducible errors in OOD detection. Second, logit-based methods with high label uncertainty does not reliably indicate OOD samples, as in-distribution samples with high label ambiguity may appear as OOD. This conflation leads to missed detections or incorrect OOD classifications.

Also, alternative attempts to improve OOD detection through scaling, hybrid models, or outlier exposure fall short, as these interventions still rely on the flawed assumptions underlying feature- and logit-based detection.
(1) Scaling model size: Increasing model and dataset size does not fundamentally address the limitations in feature separation for OOD detection.
(2) Hybrid methods: Combining features and logits may show slight improvements but does not resolve the underlying misalignment with true OOD detection which adressed above.
(3) Outlier Exposure: Training with outlier samples to simulate OOD data may improve detection but reduces generalization performance on covariate shifts, highlighting a trade-off between detection and generalization.
(4) Methods based on epistemic (Bayesian) uncertainty, which should theoretically improve with increased data, fail as they conflate uncertainty about the model’s knowledge with uncertainty over whether a sample is OOD. 
(5) Generative models, which compute likelihoods of training data, often fail in OOD detection due to their inability to differentiate low-likelihood in-distribution data from high-likelihood OOD data.

strengths:
This paper tried to adress its critical examination of the conceptual limitations of widely used OOD detection methods. By challenging the assumptions underlying feature- and logit-based OOD techniques, the authors reveal fundamental pathologies—such as the conflation of high uncertainty with out-of-distribution status. This critique goes beyond incremental improvements and instead questions the foundational approach, which could reshape the field’s direction. Overall the paper is well written in clear presentations with a topic should be high interest in the field.

weaknesses:
Although, I agree that there is a fundamental pathologies in OOD detection which mentioned in the strength of the paper, I still have some concerns regarding the paper.

1. OOD features that are indistinguishable from ID features may due to the small model representation space. Larger models with various classes may learn detailed representation that can distinguish.

2. Recent OOD methods have devided definition of OOD score and uncertainty. It might seem simmilar, but OOD scores are for detecting whether the input is OOD or ID, not serving as the label is certain or not. This refers there can be a sample with low label uncertainty with high OOD scores (and this is the reason that most SOTA ood detection methods outperform MSP in tables).

questions:
Interesting paper with nice contributions. You may want to address the weaknesses I listed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692473792

thread_type:
official_review

<thread_object>
creator: reviewer_nhyc

<content>
summary:
The paper argues that OOD detection is fundamentally misspecified. It provides a review of
existing OOD techniques and conducts mini-experiments to demonstrate that each suffers
from fundamental flaws arising from the way OOD detection is framed.

strengths:
A substantive assessment of the strengths of the paper, touching on each of the following
dimensions: originality, quality, clarity, and significance. We
encourage reviewers to be broad in their definitions of originality and significance. For
example, originality may arise from a new definition or problem
formulation, creative combinations of existing ideas, application to a new domain, or
removing limitations from prior results. You can incorporate Markdown
and Latex into your review. See https://openreview.net/faq (https://openreview.net/faq).

The paper is well presented, easy to follow, and makes a well argued case. The authors
conduct mini-experiments to demonstrate pathologies across a wide range of OOD methods
and intend to release code publicly to reproduce examples. The topic is relevant to the
community and raises awareness of the need to clarify the framing and purpose of OOD
detection.

weaknesses:
A substantive assessment of the weaknesses of the paper. Focus on constructive and
actionable insights on how the work could improve towards its stated
goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks
novelty, provide references and an explanation as evidence; if you
believe experiments are insu&quot;cient, explain why and exactly what is missing, etc.

While the paper is well presented and provides a compelling argument, it&#39;s not clear what
the new research contribution is, and as such I cannot recommend it for acceptance in the
main research track at ICLR. I&#39;d encourage submitting as a review article to a journal or as a
position paper to a workshop, but without a significant research contribution I can&#39;t
recommend it for the main research track.
Additional related work:
* Fahim Tajwar, Ananya Kumar, Sang Michael Xie, Percy Liang, &quot;No True State-of-the-Art?
OOD Detection Methods are Inconsistent across Datasets&quot; 2021
https://arxiv.org/abs/2109.05554
* Damien Teney, Yong Lin, Seong Joon Oh, Ehsan Abbasnejad &quot;ID and OOD Performance
Are Sometimes Inversely Correlated on Real-world Datasets&quot; NeurIPS 2023
https://arxiv.org/pdf/2209.00613
Minor:
* Figure 1 doesn&#39;t seem to be referenced in the main text
* Line 89: If taking arg max of a function with respect to K, then I&#39;d expect K to appear
somewhere in the function. I believe the arg max should be with respect to index i ∈ {1, .., K}
and the index shown in the function as either y_i or y = i.
* line 99: features or features -&gt; features of features?
* Line 344: OO -&gt; OOD

questions:
Are the authors claiming a new finding or novel contribution in relation to any specific
experiment, or are these primarily intended to demonstrate known limitations as evidence to
support the paper&#39;s main argument?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704324846

thread_type:
official_review

<thread_object>
creator: reviewer_k5mb

<content>
summary:
The paper presents a critical analysis on prior approaches for out-of-distribution (OOD) detection. The central argument of the paper is that prior approaches cannot detect certain categories of OOD samples, due to limited information regarding OOD samples at test time. The paper demonstrates failure modes for common categories of OOD detection methods such as feature-based, logit-based, and uncertainty-based methods, with concrete empirical evidence and illustrations.

strengths:
1. The paper is overall well written with clear figures and illustrations.
2. The failure modes for different categories of OOD detection methods are clearly demonstrated with examples.

weaknesses:
1. **Limited New Insights and Lack of Technical Depth**: The paper falls short in presenting new insights and lacks significant technical depth. For example:

- Section 4.1: The paper states that “no feature-based method can correctly detect these OOD inputs that have indistinguishable features from ID” (L217) and demonstrates this claim with examples. However, this observation is already well-established in the literature and is almost self-evident given the definition of feature-based OOD detection. Instead of demonstrating the existence of OOD samples that share features with ID (which is an expected finding), a deeper analysis of the root causes—such as the training method, model architecture, or the impact of pre-training—would provide more value.
- Section 4.2: The claim that “OOD examples often have low uncertainty” is supported by an experiment using only LeNet-5 to classify automobiles and trucks from CIFAR-10. The limited scope of this experiment is insufficient to support a generalized conclusion that OOD examples often exhibit low uncertainty.
- Section 5.6: The failure mode of generative models is demonstrated solely with a simplified two-class Gaussian example and lacks accompanying experiments. The relevance of this simple demonstration to recent advancements in generative models remains unclear.

2. **Lack of Scientific Rigor**: The paper lacks scientific rigor in most of its experimental sections. Each section follows a similar format where a failure mode is claimed (often an obvious one) and then demonstrated with a basic experiment (e.g., “To demonstrate feature overlap, we train a ResNet-18 on a subset of CIFAR-10 classes,” L247). These conclusions are drawn from experiments involving only a single model and dataset, leaving open questions about the generalizability of the findings across different training and evaluation settings.


Despite these issues, I acknowledge the educational value the paper provides by illustrating failure cases across different categories of OOD detection methods. I recommend that the authors consider submitting this work to a blog post track, where it could serve as a useful resource for educational purposes.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731604943153

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank all of the reviewers for their feedback. We present a general response to the reviewers here, and we also address individual reviewers in separate posts below. 

OOD detection has been a vital component of AI safety and robustness, and many methods have been developed to address this problem. In our work, we are the first to highlight the fundamental pathologies of entire families of popular approaches to OOD detection, such as methods which utilize the features or logits of supervised models. We demonstrate that these approaches do not directly address OOD detection but instead answer entirely different questions, and we provide concrete failure modes across a diverse set of models and datasets. We do not propose yet another method which claims to improve OOD performance while following fundamentally pathological paradigms; instead, our research contribution lies in our novel demonstrations of the irresolvable failure modes when using supervised models trained on in-distribution data to detect out-of-distribution data.

Because there are many works which criticize specific OOD detection procedures, it may seem, superficially, as if our work lacks novelty. However, many SoTA OOD detection methods focus on adaptations within these existing families, often implying the opposing stance that improved methodology can address these innate limitations. For example, Sun et al [1] attribute the problem with feature-based approaches to the distributional parameterization and thus propose a feature-based approach. Wei et al [2] attribute the problem with logit-based approaches to model overconfidence and thus propose a novel logit-based approach. Wang et al [3] state that the performances of logit-based and feature-based methods “are limited by the singleness of their information source” and thus propose a novel hybrid approach. In contrast, we argue that even these novel methods, or any other logit-based or feature-based methods, still have fundamental limitations that arise from the misspecification of supervised training. No prior work has explored the inherent pathologies of OOD detection methods which cannot be mitigated by improved methodology.

We would also like to clarify that we have extensive experiments over a wide variety of models and datasets, and our results are not limited to toy examples. In fact, our paper includes empirical results for hundreds of combinations of model architectures, training procedures, and ID and OOD datasets.
- To demonstrate the pathologies of feature-based methods (section 4.1), we provide experiment results using four different models: ResNet-18, ResNet-50, ViT-S/16, and ViT-B/16. We use ImageNet-1k as our ID data, and we test on three different OOD datasets: ImageNet-OOD, Textures, and iNaturalist. We also use a ResNet-18 trained on CIFAR-10 for visualization purposes, since smaller feature dimensions and fewer classes lead to clearer 2D plots; furthermore, we also visualize failure modes for ResNet-50 on ImageNet-1k.
- To understand how these feature-based pathologies scale with model and dataset size (Section 5.1), we benchmark 12 different models of varying sizes and training methods, as listed in Appendix B.3.
- For pathologies of logit-based methods (Section 4.2), we show results for ResNet-18, ResNet-50, ViT-S/16, ViT-B/16, trained on ImageNet, and ViT-G/14 trained with DINOv2. We also use a LeNet-5 trained on CIFAR-10 for visualization.
- To understand the scaling behaviors of logit-based methods, we evaluate 54 models from timm and torchvision, with 9 different architectures and 6 different pre-training setups, listed in Appendix B.2.

While these experimental setups were included in the original paper, we have revised the text to highlight the diverse set of models and datasets benchmarked. 

Even though it has become popular to use out-of-the-box supervised predictive models for OOD detection, we make the case that this family of approaches is fundamentally misspecified for this purpose. Due to the profound significance of this undertaking and the broad relevance of our work, we believe our paper provides an urgent and needed contribution to the community. We hope these points, and our responses, can be accounted for in the final assessment, and that reviewers may be open minded about revising their understanding.

[1] Sun et al, Out-of-Distribution Detection with Deep Nearest Neighbors\
[2] Wei et al, Mitigating Neural Network Overconfidence with Logit Normalization\
[3] Wang et al, ViM: Out-Of-Distribution with Virtual-logit Matching\
</content>
</thread_object>

</thread>
<thread>
creation date:
1732432079968

thread_type:
official_comment

<thread_object>
creator: area_chair_nfhh

<content>
comment:
Dear Reviewers,

The public discussion phase is ending soon, and active participation is highly appreciated and recommended. Thanks for your efforts and contributions.

Best regards,

Your Area Chair
</content>
</thread_object>

</thread>
<thread>
creation date:
1733109213507

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response Update

comment:
Numerous OOD detection methods have emerged in recent years, each identifying specific weaknesses of feature- or logit-based approaches and proposing tailored workarounds that still rely on these representations. In contrast, our work takes a fundamentally different perspective: instead of pursuing methodological advances designed to address minor and narrowly focused limitations, we critically examine the foundational limits of *any* feature-based approach. We explicitly quantify these limitations across an extensive range of models and datasets, and we identify that there are clear upper bounds for any feature or logit-based methods. Furthermore, our paper provides evidence that scaling up these models is not sufficient to solve these pathologies, and fundamentally different methods are required. Our paper provides a comprehensive understanding of the pathologies of OOD detection based on supervised models, offering valuable insights beyond the critiques of individual methods.

We closely examine the impacts of training details such as model architecture, pre-training method, OOD dataset, and more, systematically exploring how these factors influence the limitations of OOD detection methods. We show, for example, how extensive pre-training can soften some of the misspecifications we observe.

However, fully addressing all of the identified limitations significantly exceeds the scope of a typical research paper, which often identifies a single minor limitation that is narrow in scope as motivation for a methodological remedy (e.g., max-softmax -> max-logit). By contrast, resolving the pathologies we identify would require a fundamental paradigm shift within the field of OOD detection, similar in scope to resolving adversarial robustness or other well-established fields. Furthermore, the current state of OOD detection reflects years of contributions from the community, making it unrealistic to expect a single section of our paper to offer a resolution to these deeply-seated far-reaching challenges. Indeed, part of the point of the paper is to help constructively change the trajectory of this research area.

Although the contributions of a paper which proposes novel methodology are straightforward to identify, there is also significant value in contributions which advance our understanding of existing methods and prompt further research. There have been many successful works published at top venues which focus on exploration [1, 2, 3, 4, 5, 6]. For instance, in “How Good is the Bayes Posterior in Deep Neural Networks Really?” (ICML 2020, 373 citations) [7], the authors state in the conclusion that “Our work has raised the question of cold posteriors but we did not fully resolve nor fix the cause for the cold posterior phenomenon.” In “Understanding deep learning requires rethinking generalization” (ICLR 2016 best paper) [8], the authors conclude that “we have yet to discover a precise formal measure under which these enormous models are simple.” Had methodological innovation been required of these papers, they all would have been rejected. Our work follows a similar trajectory, where we focus on explicitly identifying the behaviors and pathologies of existing methods. 

[1] Wilson et al, “The marginal value of adaptive gradient methods in machine learning” (NeurIPS 2017, 1278 citations)\
[2] Recht et al, “Do ImageNet Classifiers Generalize to ImageNet?” (ICML 2019, 1889 citations)\
[3] Lotfi et al, “Bayesian Model Selection, the Marginal Likelihood, and Generalization” (ICML 2022, Best Paper)\
[4] Bae et al, “If Influence Functions are the Answer, Then What is the Question?” (NeurIPS 2022)\
[5] Ilyas et al, “Adversarial Examples Are Not Bugs, They Are Features” (NeurIPS 2019, 2114 citations)\
[6] Rylan et al. "Are emergent abilities of large language models a mirage?." (NeurIPS 2023, Outstanding Paper). \
[7] Wenzel et al, “How Good is the Bayes Posterior in Deep Neural Networks Really?” (ICML 2020, 373 citations)\
[8] Zhang et al, “Understanding deep learning requires rethinking generalization” (ICLR 2016, Best Paper)

Our work fundamentally challenges the premise underlying many OOD detection methods through extensive conceptual and empirical demonstrations. We believe that it is timely and will be of great interest to the community, given the plethora of new OOD methods. We focus on a fundamental conceptual understanding of deeply-seated limitations, rather than methodologies intended to address minor and more narrowly focused limitations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tries to investigate the issues of existing OOD detection methods. The starting point of this article is quite high, and the authors claim that their findings are important and make a foundational contribution. However, one reviewer raised severe concerns about overclaim and the lack of scientific rigor, including inconsistent choice of datasets, no systematic control of variables, and arbitrary dataset choices without justification. Other reviewers also felt the research contribution of this paper is not clear. AC oversaw the whole review process. From my reading of this paper, I believe the critical analysis and identified failure mode of OOD detection methods are valuable and inspiring for researchers. However, many concerns raised by the reviewers are not minor and reasonable, which means this paper can not be published on ICLR in its current version.