Your Classifier Can Be Secretly a Likelihood-Based OOD Detector
Authors: Jirayu Burapacheep
Yixuan Li
Venue: Accepted by TMLR ()

Abstract: The ability to detect out-of-distribution (OOD) inputs is critical to guarantee the reliability of classification models deployed in an open environment. A fundamental challenge in OOD detection is that a discriminative classifier is typically trained to estimate the posterior probability $p(y|\mathbf{z})$ for class $y$ given an input $\mathbf{z}$, but lacks the explicit likelihood estimation of $p(\mathbf{z})$ ideally needed for OOD detection. While numerous OOD scoring functions have been proposed for classification models, these estimate scores are often heuristic-driven and cannot be rigorously interpreted as likelihood. To bridge the gap, we propose Intrinsic Likelihood (INK), which offers rigorous likelihood interpretation to modern discriminative-based classifiers. Specifically, our proposed INK score operates on the constrained latent embeddings of a discriminative classifier, which are modeled as a mixture of hyperspherical embeddings with constant norm. We draw a novel connection between the hyperspherical distribution and the intrinsic likelihood, which can be effectively optimized in modern neural networks. Extensive experiments on the OpenOOD benchmark empirically demonstrate that INK establishes a new state-of-the-art in a variety of OOD detection setups, including both far-OOD and near-OOD.

Published in Transactions on Machine Learning Research (12/2024)
Your Classifier Can Be Secretly a Likelihood-Based OOD Detector
Jirayu Burapacheep Deportment of Computer Science Stanford University
jineyu@stanford.eei edu
Yixuan Li
sharonki &cs. wiise. edu
Department Computer Sciences
University of Wisconsin Madison
Reviewed on OpenReview: https: // openreview. net/ formm? id= FMAAJPWBMS
Abstract
The ability to detect autto--distributtin (OOD) inputs is critical to guarantee the reliability of classification models deployed in an open enviromment. A fundamental challenge in OOD detection is that discriminative classifier is typically trained to estimate the posterior probability p(ylz) for class y given an input z, but lacks the explicit likelihood estimation of p(z) ideally needed for OOD detection. While numerous OOD scoring functions have been proposed for classification models, these estimate scores are often heuristic-driven and cannot be rigorously interpreted as likelihood. To bridge the gap, we propose Intrinsic Likelihood (INK), which offers rigorous likelihood interpretation to modern discriminative. based classifiers. Specifically, our proposed INK score operates on the constrained latent embeddings of discriminative classifier, which are modeled as mixture of hyperspherical embeddings with constant norm. We draw a novel connection between the hyperspherical distribution and the intrinsic likelihood, which can be effectively optimized in modern neural networks. Extensive experiments on the OpenOOD benchmark empirically demonstrate that INK establishes a new -ath--ffthhe-rt in a variety of OOD detoction setups, including both far-OOD and near-OOD.
1 Introduction
The problem out-of-listribution (OOD) detection has garnered significant atlention reliable machine learning. OOD detection refers the task of identifying test samples that do not belong to the same distribution as the training data, which can occur due to various factors such the emergence of unseen classes in the model's operating environment. The ability to detect OOD samples is crucial for ensuring the reliability and safety of machine learning models (Amodei et al., 2016), especially in applications where the consequences of misidentifying OOD samples can be severe. To facilitate the separation between in- distribution (ID) and OOD data, a rich line of works (Yang et al., 2021b) have developed OOD indicator functions, which, at the core, compute scalar score for each input signifying the degree of OOD-ness. The design of these scoring functions has emerged as critical cornerstone achieving effective OOD dctection.
By definition, OOD data inherently diverges from ID data by means of their data density distributions, rendering likelihood log- likelioodo ideal scoring function for detection. However, fundamental dilemma in OOD detection is that a discriminative classifier is typically trained to estimate the posterior probability plylz) for class given an input z, but lacks the explicit. likelihood estimation of p(z) needed for OOD detection. While many OOD scoring functions have been proposed for classification models, they are either heuristic-driven and cannot be rigorously interpreted as log-likclihood, or impose strong assumption on the density function that can fail hold in test time (Lee et al., 2018b). Against this backdrop, the key research question of this paper thus centers

---

Published in Transactions on Machine Learning Research (12/2024)
Truck (ID)
Dog (ID)
OOD
Penguin (OOD)
Penguin (OOD)
Encoder Network
Cat (ID)
ID
Intrinsic Likelihood
Figure I: Overview of intrinsic likelihood framework for OOD delection. The neural network is trained on the in-distribution (ID) data, which lies on the unit hypersphere in the latent space. The intrinsic likelihood score is theorctically cquivalent the oog-liielhhod log)(z), which suits OOD detection.
RQ: How to design an OOD scoring function that can offer rigorous likelihood interpretation to modern discriminative- based classifiers without resorting to seporate generative models?
To bridge the gap, we introduce a novel framework INtrinsic liKelihood (INK) for classifier- OOD detection, which can be theoretically reasoned from log- lieliikl perspective. The key insight is that need to incorporate explicit probabilistic modcling of the data distribution p(z) directly into optimizing the posterior p(ylz), which is effectively achieved by our framework. Specifically, our proposed INK score operates on the constrained latent embeddings of discriminative classifier, which are modeled as mixture of hyperspherical embeddings with constant norm (see Figure 1). Under our probabilistic model, each class has a mean vector in the hypersphere, where the class-conditional density function decays from the center exponentially by the similarity between an embedding to the center. The proposed INK score is formalized by drawing novel connection between the hyperspherical distribution and the density function under our probabilistic model. Theoretically, we show that the INK score is equivalent to the log-likelihood score log p(z) (up to some constant difference), and thus can act rigorously Though simple and clegant in hindsight, establishing this connection was non-trivial. We lay out the full theoretical derivation in Section 3.1, along with how to effectively optimize for the intrinsic likelihood in the context of discriminative neural networks in Section 3.2.
density-based score for OOD detection.
We show that our theoretical guarantee indeed translates into competitive empirical performance (Section 4). We extensively evaluate our method on the latest OpenOOD benchmarks (Zhang al., 2023a), containing CIFAR and ImageNet-Ik when compared to the state-of-the-art method ASH (Djurisic et al,, 2023), which employs heuristic-driven activation shaping approach and lacks a rigorous likelihood interpretation. Moreover, we show that our method is effective under both far-OOD and near-O0D OOD scenarios and generalizes to different model architectures, including ViT (Dosovitskiy et al., 2021). The overall ovaluation demonstrates notable strengths of our approach in both empirical performance and theoretical underpinnings.
ID datasets. Our method exhibits significant performance improvements
2 Preliminaries
Let X' and y {l...., C}} represent the input space and ID label space, respectively. The joint ID distribution, represented as Px,y, is joint distribution defined over X' x X.. During testing time, there some unknown OOD joint distributions D D defined over X x ye, where y is the complementary set of V. We also denote p(x) as the density of the ID marginal distribution Px,. According to Fang et al. (2022), OOD detection can be formally defined as follows:
2

---

Published in Transactions on Machine Learning Research (12/2024)
Definition 2.1 (OOD Detection). Given labeled ID data Dm (xx1))))... (XN, v yN)), which is drawn from Px,t independent and identically distrôbuted, the aim of OOD delection is to learn predictor g by using Dia such that for any test data x: 1) if x is drown from Px, then can classify x into correct ID classes, and 2) if x is drown from Pxo. then g can detect x as OOD data.
Definition 2.2 (Likelihood-basee OOD Detector). Given ID data density function po(.) estimated by a model fo and pre-defined threshold X, then for any data x E X,
gx) ID, if po(x) 2 X; otherwise, g(x) OOD.
(1)
The performance of OOD detection thus heavily relies on the estimation of data density pe(x). However, in supereised learning, the classifiers directly estimate the posterior probability po(ylx) instead of the likelihood px(x), which makes likelihood based OOD detection non-trivial. Our focus on discriminative models differs from unsupervised OOD detection using gencrative-baset models, which do not offer classification capability. For completeness, we review these studies extensively in the related work (Section 5).
3 Methodology
In this section, we introduce our proposed framework, INtrinsic liKelihood (INK), for OOD detection. We first provide an overview of the key challenge and motivation. Then, we introduce the notion of intrinsic likelihood and thcoretically justify its fcasibility as an indicator function of OOD in Section 3.1. Finally, in Section 3.2, we discuss how lo optimize neural networks to achieve the desired intrinsic likelihood for OOD detection.
A classifier is typically trained via maximum likelihood estimation (MLE) on the training dataset Din {(x, 1) in order maximize the predictive probability for the corresponding class:
argmaxg II
Ply y 12),,
(2)
where is the index of the sample, z; is the latent representation of x,, and N is the size of the training set. By the Bayes' rule, the posterior probability can be rewritten as the following under uniform prior:
ply 2) p(z v)p(v) p(z)
p(z y) Σイ= p(z =)
Thus, one can define S(z) = log E y = as a measure of how well given embedding matches the distribution of ID samples, which is proportional to the log- lig-liieldhood logp(z). Despite the connection, the challenge lies in how to concretely define the class-conditional latent distribution p(z y), as well as how to efficienlly oplimize neural networks to achieve such lalent distribution while training a classifier. In what follows, we address these two challenges.
3.1 Intrinsic Likelihood
To address the challenge of defining class-conditional latent distributions, we model the representations by the von Miscs-Fisher (yMF) distribution in the hypersphere. A hypersphere is topological space that is equivalent to standard (d 1)-sphere, which represents the set of points in d-dimensional Euclidean space that are equidistant from the center (see Figure 1). The unit hypersphere is a special sphere with a unit radius and is denoted as 5d-1 = {z E Rd ||z|| 1} in the (d (d-11--h 1-1--dimensional space. The formal definition of vMF distribution is given below:
Definition 3.1 (von Miscs-Fisher Distribution (Fisher, 1953)). For a unit vector z E Rd in class c, the conditional probability densily functtion of the vMF distribution defined as
p(z y = c) Za(ss) exp(111 z),
(4)
3

---

Published in Transactions on Machine Learning Research (12/2024)
where Me E Rd denotes the mean direction of the class c, R 2 0 denotes the concentration of the distribution around Mer and Za(K) denotes the normalization factor.
Benefits of the probabilistic model. We model the latent. representations as vMF distribution because it is a simple and expressive probability distribution in directional statistics. Compared with multivariate Gaussian distribution, vMF distribution avoids estimating large covariance matrices for high-dimensional data that is shown to be costly and unstable literature (Chen al., 2017). Mcanwhile, choosing the vMF distribution allows example, fixed-norm vectors are known to improve training stability in modern machine learning, where dot products are ubiquitous and are beneficial for learning good representation space. For this reason, hyperspherical representations have been popularly adopted in recent contrastive learning literature (Chen et al., 2020a; Khosla et al., 2020).
have the features live the unit hypersphere, which leads to several benefits. For
Under this probabilistic model, an embedding z is assigned to the class e with the following probability
Za ( z)
ply =c c z: [K. (1,
Ei=1 Zd(k) (x)()x)] explu T 2/T) expl 2/T)
(5)
where r = 1/K denotes a temperature parameter.
Now, we are ready draw a novel connection between the hyperspherical distribution and the intrinsic likelihood, which is formalized below.
Definition 3.2 (Intrinsic Likelihood Score). We define the intrinsic likelihood function over Rd in terms of the log partition function (or denominator in Eq. 5):
S(z; (1 T -lgg Σ exp(u]22TT)
(6)
Theorem 3.1 shows that our scoring function in Eq. 6 can be functionally equivalent to the log-likelihood score, for OOD detection purposes.
Theorem 3.1. Under uniform class prior, the intrinsic likelihood score S(z) is a logarithmicc function of the density p(z), with a constant difference. The two measurements return the same level set for OOD detection.
Proof. The likelihood p(z) is the probability of observing a given embedding z under the vMF distribution. p(z) can be calculated as a summation of class-conditional likelihood p(z y = =), weighted by the class prior:
plz) = Σp(e p(z 1y j)p(y = 5)
j=1
Za(k)
Σ exp(1 2/7),
where the right-hand side is obtained by plugging in Eq. and class prior 1/C. By connecting the intrinsic likelihood S(z) in Eq, and the density function p(z) in Eq. 7, we can show the following equivalence between the two:
C
S(z: (T, 1 -T.
=T. log likelihood of
log
log Za(L/T)
const, independent
Note that the second term is independent of z, and can be treated a constant. Thus, S(z) functionally equivalent to the logelikelihood score, for OOD detection.
4

---

Published in Transactions on Machine Learning Research (12/2024)
Intrinsic likelihood for OOD detection. Given the above equivalence between INK score and log- likelihood, we propose the following OOD detector:
8x 5x(x) 1(S(z) 2 X},
(9)
where samples with higher scores are classified as ID, and vice versa. The threshold X is chosen based on the ID score at a certain percentile (e.g., 95%).
Generalization to class-imbalanced datasets. In Theorem 3.1, we adopt the commonly assumed scenario prevalent in OOD detection literature, where datasets are class-balances with a uniform prior distribution over classes, i.e., pyy = c) = 1/C. However, we acknowledge that this assumption may not always hold in real-world scenarios.
To further demonstrate the generality and effectiveness of our method, we extend it to account for non-uniform class priors. We perform standard maximum likelihood estimation on the training dataset ((x,,))))=
N
argmax, x=1
II p(y= = 2 {N. (,,
(10)
where i is the index of the sample, j is the index of the class, and N is the size of the training set. The posterior probability is given by:
ply = 1z {N, (1,,)) ply exp(( 2/T)
(11)
j) 2/T)
which accounts for the class prior probability, allowing for non-uniform class distributions. We define the generalized intrinsic likelihood as:
C S(2 (T,1g)E)))))) = log > ply = j) exp(p 2/T), 1=1
(12)
which maintains functional equivalence to the log-likelihood for OOD detection, even when the class priors are not uniform.
3.2 Optimizing Intrinsic Likelihood in Deep Neural Networks
So far, we have established that intrinsic likelihood derived under the vMF distribution desirable for OOD detection. Now, we discuss how to optimize neural networks for intrinsic likelihood.
Maximum likelihood estimation. We aim to train the neural network, such that the hyperspherical embedding conforms to a mixture of vMF distributions defined in Eq. 5. Specifically, we consider a deep neural network h: X + Rd which encodes an input x E X' to a normalized embedding z == h(x). To optimize for the vMF distribution, one can directly perform standard maximum likelihood estimation on the training dataset ((x, :
N argmax, II = 2i (N. (N,
(13)
where is the index of the sample, is the index of class, and N is the size of the training set. In effect, this loss encourages each ID sample to have a high probability assigned to the correct class in the mixtures of the vMF distributions. While hyperspherical learning algorithuns have been studied (Meltes et al., 2019; Du et al., 2022a; Ming et al., 2023), none of the works explored its principled connection to intrinsic likelihood for test-time OOD detection, which our distinct focus. We discuss the differences further Section 3.3. Morcover, we show new insights below that minimizing this loss function effectively shapes the intrinsic likelihood to aid OOD detection.
5

---

Published in Transactions on Machine Learning Research (12/2024)
How does the loss function shape intrinsic likelihood? We show that the learning objective induce a higher intrinsic likelihood for ID data. By taking negative-log likelihood, the objective function in Eq. 13 is equivalent to minimizing the following loss:
N N
L
ply = yi z;: (T, (T,14,511))
(14)
1=1
We can rewrite the loss by expanding Eq.
1
exp( 2i/T)
C
log
N
2j=1 ( 2:/T)
C
1
1
s(z Vi) + log Σ exp(- (xp(-s((s.3)/t)
(15)
N
f=1
where o(z.,)) = m zę- During optimization, the loss reduces s(z,, y) for the correct class while increasing it for other classes. This can be justified by analyzing the gradient of the loss function with respect to the model parameters 0, for embedding and its associated class label y:
ac(z, v; 00
0s(2, y)
(16)
00
1 C Ōs(z.j)
exp(-s(z,j))
00
Σe= exp(-s(z,c))
(z (z v)
Ōs(z.j)
00 (1 p(Y = 11122 12)
p(Y = 11
(17)
00
jzy
for
The sample-wise intrinsic likelihood S(z) of ID data is S(z) = log Ee exp(-s(z, c)/T), which dominated by the term s(, y) with ground truth label. Hence, the training overall induces a higher intrinsic likelihood for ID data. By further connecting to our Theorem 3.1, this higher intrinsic likelihood can directly translate into high (1) log p(z), for training data distribution.
3.3 Differences w.r.t. Existing Approaches
Table 1 summarizes the key distinctions among prevalent hyperspherical methodologies, specifically in terms of their training-time loss functions and test-time scoring functions. Both SSD+ (Sehwag et al., 2021) and KNN+ (Sun et al., 2022b) employ the SupCon (Khosla et al., 2020) loss during training, which does not explicitly model the latent representations as vMF distributions. Instead of promoting iattance-to-rrottyye similarity, SupCon promotes instance-to-innstanee similarity among positive pairs. SupCon's 's loss formulation thus does not directly correspond to the vMF distribution. Geometrically speaking, our framework dircctly operates on the vMF distribution, key property that enables log- lig-likeldhood interpretation.
Our approach also differs significantly from CIDER (Ming et al., 2023) and SIREN (Du et al., 2022a) in terms of OOD scoring function, which employ either KNN distance or maximum conditional probability max p(y) z). In contrast, our method establishes the novel connection between the hyperspherical representations and intrinsic likelihood for OOD detection, which enjoys rigorous theoretical interpretation from a density perspective. We show empirically in Soction 4 that our proposed OOD score achieves competitive performance on different OOD detection benchmarks, and much more computationally efficient by climinating the need for time-consuming KNN searches.

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Summary of key distinction among different hyperspherical approaches.
Training- time loss function
Test-time scoring function
Likelihood interpretation
SSD+ KNN+ CIDER SIREN
SupCon
Mahalanobis (parametric)
X
SupCon
KNN nion-aarmmetric)
vMF
KNN (non poraaaamtttrcc)
X
vMF
KNNN max p(y'z) 2))
X
INK (Ours)
vMF
Intrinsic likelihood (parametric)
4 Experiments
4.1 Setup
Benchmarks. Our evaluation encompasses both far-OOD and near-OOD detection performance, following the same setup the latest OpenOOD benchmark' (Zhang et al., 2023a). The benchmark includes CIFAR (Krizhevsky, 2009) and ImageNet-lk (Deng et al., 2009) as ID datasets. Due to space constraints, primarily focus on the most challenging ImageNett benchmark in the main paper, and report the full results for CIFAR datasets in the Appendix E. For the ImageNet benchmark, iNaturalist. (Van Horn et al., 2018), Textures (Cimpoi et al., 2014), and Openlmage-O (Wang et al., 2022) are employed as far-OOD datasets. In addition, we also evaluate on the latest NINCO (Bitterwolf al., 2023) as the near-OOD dataset, which circumvents the issue of contaminated classes found in other near-OOD datasets (Vaze et al., 2022). We adhere to the same data split used in OpenOOD (Zhang et al., 2023a), which involves the removal of images with semantic overlap.
Evaluation metrics and implementation details. We utilize the following metrics, which are widely
used to assess the efficacy of OOD detection methods: (1) false positive rate (FPR@95) of OOD samples
when the true positive rate of TD) samples set at 95%, (2) area under the receiver operating characteristic curve (AUROC), and (3) ID classification accuracy (ID ACC). Due the space limit, we provide the implementation details in the Appendix B and ID accuracy in Appendix D. For INK score, we use test-time
temperature Test 0.05 based on validation (see Appendix C). We further show the effect of the temperature in our ablation study (Section 4.3).
4.2 Main Results
Intrinsic likelihood score establishes state-of-the-art performance. We extensively compare the performance of our approach with state-of-the-art methods. The compared methods are categorized as cither hypersppherccal-baeedd or non-llyperspherical--beeee All the non-hyyerspheericl-baeeee methods, including MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018b), Energy (Liu et al., 2020), ViM (Wang et al., 2022), ReAct (Sun et al., 2021), DICE (Sun & Li, 2022), SHE (Zhang et al., 2023b), and ASH (Djurisic et al., 2023) use the softmax cross-entropy (CE) loss for model training. Hyperspherical-bann based methods, on the other hand, include SSD+ (Sehwag et al., 2021), KNN+ (Sun et al., 2022b), and CIDER (Ming et al., 2023). It is worth noting that we have adopted the recommended configurations proposed by prior works. Further details of each bascline method are included in Appendix B.
As shown in Table 2, our method INK displays competitive OOD detection performance compared to existing ones on the far-OOD group of the ImageNet benchmark. Here we highlight two observations: (1) INK also outperforms the current stale-of-the-arl cross-entropy loss method ASH (Djurisic et al., 2023), which employs a heuristic-driven activation shaping approach and lacks a rigorous likelihood interpretation. In contrast, INK is derived from a more principled likelibood-based perspective while being empirically strong. (2) INK score ourperfomms the current state-of-the-art lyperspherical-bbsed methods CIDER by 8.97% and KNN+ by 6.60% in FPR299.. Compared to CIDER, our method enjoys a significant performance boost by leveraging the likelihood interpretation within hyperspherical space, and a drastic reduction in computational cost (more in Section 4.3).
com hctm://aaagaaagggo//0eennnn
7

---

Published in Transactions on Machine Learning Research (12/2024)
Table 2: OOD detection performance for ImageNet (ID) with ResNet-50, using OpenOOD benchmark (Zhang et al., 2023a). INK achieves competitive performance compared state-of- the-art. methods. Statistical significance is estimated on random independent runs.
Far-OD Datasets
Near-OD Dataset
Method
Naturalist
Textures
OpenInape-O
Average
NINCO
FPR
AUROC
FPR.
AUROC 1
FPR.
AUROC
FPR
AUROC:
FPR
AUROC
Methods using cross-entropy loss
MSP
43.34
88.94
60.88
80.13
50.13
86.00
51.45
86.25
56.87
79.95
ODIN
36.07
90.63
49.44
89.06
46.55
86.58
44.02
58.76
68.13
T7.77
Mahalanobis
73.80
63.67
42.77
69:27
72:15
51.96
62.91
61.63
78.83
62.38
Energy
31.34
92.51
45.76
86.26
38
88.96
38.39
00.58
VIM
30.67
92.65
10.19
86,52
32 82.82
98..78
21.66
89.98
62 22.28
78.63
ReAct
16.73
96.34
29:63
91.87
32.57
93.41
26.31
93.87
56.91
$1.73
DICE
33.18
91.17
44.25
88.24
47.83
86.77
41.82
68.72
66.97
76.01
SHE
31.13
89.56
35.30
90,50
51.98
83.33
41.44
87.80
69.75
76.48
ASHI
14.10
97.06
15.27
93.26
29.20
19.52
52.90
Methods using hyperspherical
representation
SSD+
39.82
88.41
71.90
81.86
59 (15
81.65
56.92
$1.97
61.74
80.61
KNN-
18.29
94.90
19:90
93.56
27.40
89.05
21.87
92.50
53.35
$4.15
CIDER
20.79
94.39
21.86
92.54
30.06
80.07
24.24
92,00
57.42
62.36
INK (ouns)
10.0450.67
97.49==008
7.52#2.05
98.6240.61
28.26=
94.24=0.85 28
15.2740027
96.78460.8
48. 48.1541.00
86.4240.13
Table 3: Average and standard deviation of inference time (per sample) for each ID dataset. The INK score is computationally more efficient compared to the KNN search.
Score compute time (usL)
Score
CIFAR-10
C1FAR-100
ImageNet-1k 1k (a 1%)
ImageNet-1k (a 100%)
KNN+ or CIDER
3.5619 + 0.2646 0.0076 + 0.0003
5.4886 + 0.1854
3.6157 + 0.1186
336.1780 + 4.3871
INK (ours)
0.0157 + 0.0007
0.2745 + 0.0145
0.2745 + 0.0145
Near-OOD detection. We also evaluate the performance of our method in the more challenging near-OOD scenario. As shown in Table 2, INK demonstrates competitive performance, outperforming CIDER by 9.27% in FPR@95. This improvement underscores the versatility of the INK score, proving its effectiveness in near- near-OOD OOD detection scenarios where ID samples. Moreover, our method shows an enhanced performance over ASH in near-OOD detection, with an improvement of 4.84% in FPR&95. This advancement also signifies progress in bridging the performance gap between hyperspherical- based methods and coss-entropy loss methods.
samples bear a close semantic resemblance to the
CIFAR benchmark. We evaluate the performance of our method on the CIFAR benchmark. Consistent with the results observed in the ImageNet benchmark, the INK score displays competitive performance in handling both far-OOD and ncar-OOD groups, compared to the current state-of-the-art mothods. Due to space constraints, further details are included in Appendix E.
4.3 Additional Abalations and Discussions
Computational cost. The current state-of-the-art hyperspherical representation methods, such as KNN+ and CIDER, rely on a non- poramaratttric KNN score, which requires a nearest neighbor search in the embedding space. Although recent studies on approximated nearest neighbor search have shown promise (Aumiller et al., 2020), the computation required grows with the size of the embedding pool N. In contrast, INK only incurs complexity O(C), where the number of classes C < N is independent of ID sample size. Thus, our method can significantly reduce the computation overhead. To demonstrate this, we conduct experiment to compare the average compute time between the KNN score and the INK score. In this experiment, we assess the performance of the model on CIFAR-10, CIFAR-100, and ImageNet-lk benchmarks using default values of & in Sun et al. (2022b). In particular, we measure the time taken to calculate the score per sample, excluding the time to extract the sample embedding from the neural network. KNN-based methods utilize the Faiss library (Johnson et al., 2019), which optimized for efficient nearest-neighhor search. The size of the embedding pool is 50,000 for both CIFAR-10 and CIFAR-100, 12K for ImageNet (a 1%), and 1.2M for
8

---

Published in Transactions on Machine Learning Research (12/2024)
ImageNett (a 100%), where denotes the fraction of training data used for the nearest neighbor search. Our results, shown in Table 3, demonstrate that INK is significantly faster than the KNN-based methods, making it computationally desirable for OOD detection tasks.
Relationship with energy score. We discuss the relationship of INK with prior work by Liu et al. Liu et al. (2020), which attempted to interpret the energy score derived from multi-class classifier from likelihood-basee view. The key distinction between the two methods lies in the geometry of the
60
Method
Energy
Ours
FPR@9 (%)
40
embedding space, from which the score is derived. Specifically, an energy score defined as the negative log of the denominator in the softmax function:
30
20
E(x; T llgg Σ exx((f(xx//T),
10
0
where (x) denotes the logit corresponding to the derived from the Euclidean space logits fx(x) can be wn- constrained, and does not necessarily reflect the log- likelihood (see Theorem A. and proof in Appendix). In contrast, our framework explicitly models the la- tent representations in the discriminative classifier as constrained hyperspherical space, which enables a rigorous interpretation from a log-likelihood perspective (c.f. Theorem 3.1). In formulation (Eq. 5), the classifier's logit (x) is equivalent to μ4, z, where is the penultinate layer embedding of input x and H, can be interpreted the weight vector connecting the embedding and output logit. Different from Liu et al. (2020), the embedding z and the weight connection 15 are explicitly constrained together to form meaningful probabilistic distribution during our optimization.
iNaturalist
Textures
Openlmage-0 Datasets
NINCO
class j. Unfortunately, the energy
Comparison of Energy vs. INK on Ima- geNet (ID) with ResNet-50, using OpenOOD bench- mark Zhang et al. (2023a). INK consistently outper- forms Energy far-OOD and near-OOD datasets.
Figure 2:
Figure illustrates a comparison the performance on the ImageNet benchmark. Our method reduces the average FPR@95 by 23.12%, which validates that the proposed intrinsic likelihood score is sig- nificantly more effective for OOD detection. The performance improvement can also be reasoned the- oretically, as we show in Theorem 3.1. Overall, our method enjoys both strong empirical performance and theoretical justification. Additionally, for more detailed understanding, we have included a visualiza- tion of ID-OOD score distributions in Figure 4. We illustrate comparison between the distributions of energy and INK scores for ImageNet (ID) vs. Tex- tures (OOD). The noticeable separability of using INK score enhances the performance OOD detec- tion.
46
Average FPR@955 (%)
44
40
0.0
0.2
0.4
0.6
0.8
1.0
Test temperature
Figure 3: Ablation on test-time temperatures. The results are averaged across the far-OOD test sets and 3 random training runs, based on ResNet-34 trained on CIFAR-100.
Ablation study test-time temperature
We conduct ablation study on the test-time temperature parameter to investigate its impact OOD detection performance, as shown in Figure 3. The curve highlights that the optimal value of 7 roughly matches the training-time temperature. This also matches our theory, where INK under the same temperature T in testing allows recovering the log-likelihood log pzz) (up to some constant) learned training time.
Ablation study on different model architecture. To explore the impact of model architecture on the performance, additionally evaluate the Vision Transformer (ViT-B-16) (Dosovitskiy et al., 2021). As

---

Published in Transactions on Machine Learning Research (12/2024)
Energy
Ours
Density
20
40
0.4
0.6
0.8
1.0
Score
Score
ID
OOD
Figure 4: Density plots illustrating the distribution of energy score (left) and INK (right). Lighter and darker blue shades represent the distributions of scores for the ID and OOD datasets, respectively,
shown in Table 7 in Appendix, our INK score maintains competitive performance with the VII model. This underscores the adaptability of our method across various model architecturos. Further details are included in Appendix F.
Ablation study on class-imbalance scenarios. We further verify this class-imbalanced setting empirically by eliminating half of the training examples from the first 5 classes in CIFAR-10 and the first 10 classes in CIFAR-100. Table 4 compares the FPR between ASH and the generalized intrinsic likelihood, which shows that our method significantly outperforms the state-of-the-art in both far-OOD and near-OOD detection tasks.
Table 4: Comparison of OOD Performance with Imbalanced Class Prior. Generalized INK achieves competitive performance compared to state-of-the-art method ASH across different datasets.
CIFAR-10 Imbalanced (ResNet-18)
CIFAR-100 Imbalanced (ResNet-34)
Dataset
ASH
Ours
Dataset
ASH
Ours
Far-OOD
Far-OOD
MNIST
25.60
23.97
MNIST
48.29
37.72
SVHN
52.73
9.12
SVHN
33.19
10.47
Texture
42.00
18.69
Texture
55.31
34.29
Places365
40.28
26.12
Places365
67.81
62.47
far- Average
40.15
19.48
Far-OOD Average
51.15
36.24
Near-OOD
Near-OOD
CIPAR-100
53.44
40.33
CIFAR-10
74.39
76.36
TIN
48.38
28.47
TIN
65.74
53.60
Near-OOD Average
50.91
34.40
Near-OOD Average
70.07
64.98
5 Related Works
OOD detection for classification models. OOD detection an essential component for reliable machine learning systems that operate in the open world. Recent studies in OOD detection focus post-hoe strategies
10

---

Published in Transactions on Machine Learning Research (12/2024)
and train-time regularization (Yang et al., 2021b). In particular, post-hoc strategies focus on deriving test-time OOD scoring functions, including confidence-based scores (Hendrycks & Gimpel, 2017; Liang et al., 2018), distance-basee based score (Lee et al., 2018b; Tack et al., 2020; Sehwag et al., 2021; Sastry & Oore, 2020; Ren et al., 2021; Sun et al., 2022b; Du et al., 2022a; Ming ct al., 2022; Ren et al., 2023), energy-based score (Liu et al., 2020; Wang et al., 2021; Morteza & Li, 2022; Zhang el al., 2023b; Jiang el al.; 2023; Lafon et al., 2023; Peng et al., 2024), gradient-baan based score (Huang et al., 2021; Behpour et al., 2023; Chen et al., 2023; Liu et al., 2023a), and multimodal score (Ming et al., 2022; Miyai et al., 2023). Recent high-performing methods focus on suppressing activations of pre-trained models (Sun et al., 2021; Sun & Li, 2022; Djurisic et al., 2023; Xu et al., 2023b). On the other hand, train-time regularization approaches (Bevandié et al., 2018 8;; Malinin & Gales, 2018; Lee et al., 2018a; Geifman & (l-yaniv, 2019; Hendrycks et al., 2019; Hein et al., 2019; Meinke & Hein, 2020; Van Amersfoort et al., 2020; Yang et. al., 2021a; Du et al., 2022b; Wei et al., 2022; Katz-Samuell et al., 2022; Du et al., 2022a; Huang et al., 2022; Colombo et al., 2022; Yu et al., 2023; Tao et al., 2023; Ming ct al., 2023; Wang et al., 2023; Xu al., 2023a) design an algorithm to encourage the model to produce desired properties. Different from prior works, we regularize a model to produce hyperspherical representations aligned with the vMF distribution and propose intrinsic likelihood a test-time scoring function, which has not been explored in the past. Moreover, our method offers theoretical justification based on the likelihood view, which is lacking in previous approaches.
bisel OOD detection. Deep generative models have been explored for unsupervised OOD detection (Nalisnick et al., 2019; Ren et al., 2019; Abati et al., 2019; Choi et al., 2019; Serrà et al., 2020; Xiao et al., 2020; Wang ct al., 2020; Schirmeesser et al., 2020; Sastry & Oore, 2020; Kirichenko et al., 2020; Li et al., 2022; Cai & Li, 2023; Liu al., 2023b). While these approaches directly estimate the likelihood of ID data, the unsupervised nature prohibits elassification ability. Moreover, issues have been highlighted that deep generative models can frequently assign spuriously higher likelihoods to OOD than ID samples (Ren et al., 2019), and the performance can often lag behind the discriminative approaches (Yang et al., 2021b). In contrast, our method circumvents this issue by virtue of its supervised learning objective, leveraging ground truth semantic labels for each ID data point. Consequently, the latent embeddings are shaped and guided by image semantics without necessarily overfitting on background elements.
Hyperspherical representation. Hyperspherical representation under vMF distribution has been exten- sively used in various machine learning applications, such as supervised classification (Kobayashi, 2021; Scott et al., 2021; Govindarajan et al., 2023), face verification (Hasnat et al., 2017; Conti et al., 2022), generative modeling (Davidson et al., 2018), segmentation (Hwang et al., 2019; Sun et al., 2022a), and clustering (Gopal & Yang, 2014). In addition, some rescarchers have utilized the vMF distribution for anomaly detection by employing generative models and using it as the prior for zero-shot learning (Chen et al., 2020b) and document analysis (Zhuang et al., 2017). Recent studies devise vMS-based learning for OOD detection, which offers a direct and clear geometrical interpretation of hyperspherical embeddings. Specifically, Du et al. (2022a) explores modeling reprosentations into vMF distribution for object-level OOD detection, while CIDER (Ming el al., 2023) shows that additional regularization for large angular distances among different class prototypes is crucial for achieving strong OOD detection performance. Our work adopts -aseb learning approach for OOD detection and builds upon these recent studies by regularizing a model to produce representations that align with the vMF distribution and proposing a new intrinsic likelihood scoring function for test-time OOD detection.
6 Conclusion
In this paper, we introduce intrinsic likelihood for detecting out-of-distributio samples. Our approach is based on density estimation in the hyperspherical space, which enjoys a sound interpretation from a log-likelihood perspective. Through extensive experiments on common OOD detection benchmarks, we demonstrate the state-of-the-art performance of our approach for both far-OOD and near-OOD detection. Morcover, our method outperforms bNN-b approaches in terms of computational efficiency. Overall, our proposed intrinsic likelihood score provides a promising solution for both effective and efficient OOD detection.
11

---

Published in Transactions on Machine Learning Research (12/2024)
7 Broader Impacts
OOD detection can be used to improve the reliability of machine learning models, which can have significant positive impact on sociely. In situations where saving fealures a concern, the use of the KNN score may not be feasible as it requires access to a certain amount of labeled data. In contrast, our INK score does not require access to a pool ID data. This makes it a valuable tool in various applications and industries where sensitive information needs be protected. It important note that the usefulness and effectiveness of OOD detection may vary across different applications and domains, and careful evaluation and validation are necessary before its deployment.
Acknowledgment
We gratefully acknowledge the support from the AFOSR Young Investigator Program under award number -A9550-23-1-0184 National Science Foundation (NSF) Award No. IIS-2237037 & IIS-233160) Office of Naval Research under grant number X00004-23-1-2643, Philanthropic Fund from SFF, and faculty research awards/gifta. from Google and Meta.
References
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent Space Antoregression for Novelty Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision and Pallern Recognátion, 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 1606066666 2016.
Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull. Ann-benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems, 87, 2020.
Sima Behpour, Thang Doan, Xin Li, Wenbin He, Liang Gou, and Liu Ren. Gradorth: A simple yet efficient. out-of-distribution detection with orthogonal projection of gradients. In Advonces én Neural Information Processing Systems, 2023.
Petra Bevandić, Ivan Krešo, Marin Oršić, and Siniša Segvić. Discriminative out-of-distribution detection for semantic segmentation. ar preprint arXiv:1808.0700 2018.
Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out-of-distributiin detection evaluation. In International Conference on Machine Learning, 2023.
Mu Cai and Yixuan Li. Out-of-distribation detection via frequency-regullriied generative models. Proceedings of IEEE/CVE Winter Conference on Appliculions of Compuler Vision, 2023.
Jinggang Chen, Junjie Li, Xiaoyang Qu, Jianzong Wang, Jiguang Wan, and Jing Xiao. GAIA: Delving into gradient-based attribution abnormality for out-of-distribution detection. In Thirlg-seventh Conference on Neural Information Processing Systems, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020a.
Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. A boundary based out-of-distribne classifier for generalized zero-shot learning. In European Conference on Computer Vision, 2020b.
Xixian Chen, Michael R Lyu, and Irwin King. Toward efficient and accurate covariance matrix estimation compressed data. In International Conference on Machine Learning, pp. 767-776. PMLR, 2017.
Hyunsun Choi, Eric Jang, and Alexander A. Alemi. Waic, but why? generative ensembles for robust anomaly detection, 2019.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In TEEE Conference Computer Vision and Pallern Recognition, 2014.
Pierre Colombo, Eduardo Dadalto Camara Gomes, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. Beyond mahalanobis distance for textual OOD detection. In Advances Neural Information Processing Systems, 2022.
Jean-Rémy Conti, Nathan Noiry, Stephan Clemencon, Vincent Despiegel, and Stéphane Gentric. Mitigating gender bias in face recognition using the von mises-Fisher mixture model. In Proceedings of the 89th International Conference on Machine Learning, 2022.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak, Hyperspherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the webj. Signal Processing Magazine, IEEE, 29:141- 1421,1111111111 11 2012. doi: 10. D0.109/AISP.202..211477
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Xuefeng Du, Gabriel Gozun, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distribution objects. In Advances in Neural Information Processing Systems, 2022a.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. Proceedings of the International Conference Learning Represenlations, 2022b.
Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is In Advances in Neural Information Processing Systems, 2022.
detection learnable?
Ronald Aylmer Fisher. Dispersion on sphere. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 217((130):295 305, 1953.
Yonatan Geifman and Ran Yaniv.... SelectiveNet: A deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Siddharth Gopal and Yiming Yang. Von mises fisher clustering models. In Proceedings of the 31sl International Conference on Machine Learning, 2014.
Hariprasath Govindarajan, Per Sidén, Jacob Roll, and Fredrik Lindsten. DINO as von hises-fisher mixture model. In The Eleventh International Conference on Learning Representations, 2023.
Md. Abul Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, and Liming Chen. von mises-fisiee fisher mixture model-based deep learning: Application face verification. arXiv preprint arXiv:1706. 04264, 2017.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In 2019 IEEE/CV Conference on Computer Vision and Pottern Recognition (CVPR), 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and distr--distribution examples in neural networks. Proceedings of International Conference on Learning Representations, 2017.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Regresenlations, 2019.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In Advances in Newral Information Processing Systems, 2021.
Wenjian Huang, Hao Wang, Jiahao Xia, Chengyan Wang, and Jianguo Zhang. Density-driven regularization for out-of-distribution detection. In Advonces in Newral Information Processing Systems, 2022.
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang- Chich Chen. Segsort: Segmentation by discriminative sorting segments. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting out-of- distribution data through in-distribution class prior. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-55) 547, 2019.
Julian Katz-Samuuls, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. Tn Proceedings of the 39th International Conference on Machine Learning, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of- distribution data. In Advances in Neural Information Processing Systems, 2020.
Takumi Kobayashi. T-vmf similarity for regularizing intra-class feature distribution. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Paltern Recograliio,, 2021.
A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Computer Science Department, University of Toronto, 2009.
Mare Lafon, Elias Ramzi, Clément Rambour, and Nicolas Thome. Hybrid energy based model the feature space for out-of-distribution detection. In International Conference on Machine Learning. 2023.
Kimin Lee, Honglak Lec, Kibok Lee, and Jinwoo Shin. Training confidenco-celibrraeed classifiers for detecting out-of-distributiin samples. In International Conference on Learning Represenlations, 2018a
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. Tn Advances in Neural Information Processing Systems, 2018b.
Junnan Li, Caiming Xiong, and Steven Hoi. Mopro: Webly supervised learning with momentum prototypes. In International Conference on Learning Representations, 2020.
Yewen Li, Chaojie Wang, Xiaobo Xia, Tongliang Liu, and Bo An. Out-of-distribation detection with an adaptive likelihood ratio on informative hicrarchical vac. Aduances in Neural Information Processing Systems, 35:7383-7733 2022.
Shiyu Liang, Yixuan Li, and R. Srikant. Euhancing the reliability of out-of-distributiio image detection in neural networks. In International Conference on Learning Representations, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.
Yibing Liu, Chris Xing Tian, Haoliang Li, Lei Ma, and Shiqi Wang. Neuron activation coverage: Rethinking out-of-distribution detection and generalization. arXiv preprint 022899,, 2023a.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Zhenzhen Liu, Jin Peng Zhou, Yufan Wang, and Kilian Q Weinberger. Unsupervised out-of-distribution detection with diffusson inpainting. In International Conference on Machine Learning, 2023b.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, 2018.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. Proceedings of International Conference on Learning Representations, 2020.
Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in Newral Information Processing Systems, 32, 2019.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with uusion-langual representations. In Advances in Newral Information Processing Systems, 2022.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How exploit hyperspherical embeddings for disto--dittiibution detection? In The Eleventh International Conference on Learning Representations, 2023.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution detection via prompt learning. Sherth-Seventh Conference on Neural Information Processing Systems, 2023.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. Proceedings of the AAAI Conference on Artificial Intelligence, 36, 2022.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011.
Bo Peng, Yadan Luo, Yonggang Zhang, Yixuan Li, and Zhen Fang. Conjnorm: Tractable density estimation for 2024.
detection. In The Twelfth International Conference on Learning Representations,
Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Takshminarayanan. Likelihood ratios for out-of-distribution dotection. In Advances in Neural Information Processing Systems, 2019.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanann A simple fix to mahalanobis distance for improving near-ood detection, 2021.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribntion detection and selective generation for conditional language models. In The Fleventh International Conference on Learning Representations, 2023.
Chandramouli Shama Sastry and Sageev Oore. Detecting ontttf-distribntionn examples with Gram matricos. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 2020.
Robin Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hicrarchies of distributions and features. In Advances in Neural Information Processing Systems, 2020.
T. R. Scott, A. C. Gallagher, and M. C. Mozer. von miscs fisher loss: An exploration of embedding geometries for supervised learning. In 2021 IREE/CVF International Conference on Computer Vision (TCCV), 2021.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Vikash Sehwag, Mung Chiang, and Prateck Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Represenbations, 2021.
Joan Serrå, David Álvarez, Viceng Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-baat based generative models. In International Conference on Learning Represenlations, 2020.
Yihong Sun, Adam Kortylewski, and Alan Yuillc. Amodal segmentation through out-of-task and out- of-distribution generalization with a bayesian model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In Europeon Conference on Computer Vision, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detoction with rectified activations. In Advances in Newral Information Processing Systems, 2021.
Yiyou Sun, Yifci Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributie detection with deep nearest neighbors. In 2022 International Conference on Machine Learning, 2022b.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 2020.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8769-8778, 2018. doi: 10. CVVP//228... 00914.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed- set classifier is all you need. In International Conference on Learning Representations, 2022. URL https://openrevier net/ pest/forummiiee "d-rumidd=bhLPPJy9992
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distributtin with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, 2022.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Ti. Can multi-label classification networks know what they don't know? In Advances in Neural Information Processing Systems, 2021.
Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment. distributions for out-of-distribution detection. In Thirty-sevendh Conference Neural Information Processing Systems, 2023.
Ziyu Wang, Bin Dai, David Wipf, and Jun Zhu. Further analysis of outlier detection with deep generative models. Advances in Newral Information Processing Systems, 2020.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural work overconfidence with logit normalization. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 2022.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of of--istribuuton detection score for variational auto-encoder. In Advances in Newral Information. Processing Systems, 2020.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post-hoc out-of- distribution detection enhancement, 2023a.
Mingyu Xu, Zheng Lian, Bin Liu, and Jianhua Tao. VRA: Variational rectified activation for out-of-diee distriibbtioon detection. In Thirty-seventh Conference Neural Information Processing Systems, 2023b.
Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In Proceedings of the IEEE International Conference on Computer Vision, 2021a.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprinl arXiv:2110. 11334, 2021b.
Shuyang Yu, Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Turning the curse of heterogeneity in federated learning into a blessing for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2023.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xucfeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprinl arXin:2306. Xii:2306009301, 2023a.
Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, xiaoguang Liu, Shi Han, and Dongmei Zhang. Out-of-distributta detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference Learning Representations, 2023b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding, 2016.
Honglei Zhuang, Chi Wang, Fangbo Tao, Lance Kaplan, and Jiawei Han. Identifying semantically deviating outlier documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
17

---

Published in Transactions on Machine Learning Research (12/2024)
A Derivation
Theorem A.1 (Misalignment between energy score and likelihood). The energy score defined in Equalioo 4.3 does not necessarily reflect the negalive log probabilily:
E(x: * -lgg (x).
To prove this, one can rewrite the posterior probability pa = c/x) = xo(xxy=e)/ro()) Note that one can always rescale the logits (x(x) by function log ((x), without changing the posterior probability: (x))(x)). this case, the negative log of the denominator in softmax (i.e., the energy score) is loolpe x (x)(,)),, which no longer truthfully reflecting the log p(x).
(y = c(x) = px y =
B Experimental Details
Software and hardware. We conduct experiments on NVIDIA RTX A6000 GPUs (48GB VRAM), We use Ubuntu 22.04.2 LTS as the operating system and install the NVIDIA CUDA Toolkit version 11.6 and cuDNN 8.9. All experiments are implemented in Python 3.8 using the PyTorch 1.8.1 framework.
Training cross-entropy models. For methods using cross-entropy loss, such as MSP (Hendrycks Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018b), Energy (Liu et al., 2020), ViM (Wang et al., 2022); ReAct (Sun et al., 2021), DICE (Sun & Li, 2022), SHE (Zhang et al., 2023b), and ASH (Djurisic et al., 2023), the models are trained using stochastic gradient descent with momentum 0.9 and weight docay 10-1. The initial learning rale is 0.1 and decays by factor of 10 at epochs 100, 150, and 180. We train the models for 200 epochs on CIFAR. For the ImageNet benchmark, we adopt Torchvision's 's pre-trained ResNet-50 model with ImageNet-1k 1k weights.
Training contrastive models. For methods using SupCon loss (Khosla et al., 2020), such as KNN+ (Khosla et al., 2020) and SSD+ (Sehwag et al., 2021), we adopt the same training scheme as in (Ming et al., 2023). For CIFAR, we use stochastic gradient descent with a momentum of 0.9 and weight decay of 10-4. The initial learning rate to 0.5 and follows cosine annealing schedule. We train the models for 500 epochs. The training-time time temperature is set to be 0.1. For the ImageNet model, we use the checkpoint provided in Sun et al. (2022b).
For methods using vMF loss, we train on CIFAR using stochastic gradient descent with momentum 0.9 and weight decay 10-1 The initial learning rate is 0.5 and follows a cosine annealing schedule. We use a batch size of 512 and train the model for 500 epochs. The training-tine temperature the www.ntialllooiinvveereeeeee for CIFAR-100. For ImageNet-lk, fine-tune pre-trained ResNet-50 model in Khosla et al. (2020) for 100 epochs, with an initial learning rate of 0.01 and cosine anncaling schedule.
set to be 0.1. We adopt (EMA) for the prototype update Li et al. (2020), with momentum of 0.5
Evaluation configurations.
We outline the configurations for methods that require appropriate hyperparameter selection, which has alrcady been extensively studied in the literature.
MSP uses the maximum softmax probability obtained from the logits. The method does not require any specific configuration.
ODIN uses temperature scaling to calibrate the softmax score. We set the temperature to 1,000.
Mahalanobis utilizes class conditional Gaussian distributions based on the low- and upper-level features obtained from a model. These distributions are then used calculate the Mahalanobis distance.
Energy derives the energy score, which includes the temperature parameter. We set the temperature to the default, value of = 1.
18

---

Published in Transactions on Machine Learning Research (12/2024)
ViM generates an additional logit from the residual of the feature against the principal space. We set the dimension of principal space to D = 256 for ResNet-18 and ResNet-34 and D = 1024 for ResNet-50.
ReAct improves the energy score by rectifying activations at an upper limit, which is set based the p-th percentile of the activations estimated on the in-distribution (ID) data. We set to the default value of p = 90.
DICE utilizes logit sparsification enhance the vanilla energy score, which is set based on the p-th percentile of the unit contributions estimated on the ID data. We set the sparsity parameter p 0.7.
SHE stores the mean direction of the penultimate layer features from correctly classified training samples. During inference, the Hopfield energy score is calculated as the dot product between the sample embedding and the class mean of the predicted class.
KNN+ and CIDER utilize the KNN score, which requires selecting a number of nearest neighbors k. Following the settings Ming et al. (2023), we set to 300 and 1,000 for CIFAR-100 and the ImageNet benchmark, respectively.
ASH proposes three heuristic-based approaches for activation shaping: Pruning, Binarizing, and Scaling. In line with the OpenOOD framework, we adopt the Binarizing method (ASH-B), specifically with parameter setting of p = 90.
C Validation Method for Selecting Test-time Temperature
To select the test-time temperature for our proposed INK score, we follow the validation method outlined in Hendrycks et al. (2019). We generate validation distribution by corrupting in-distribution data with speckle noise, creating speckle-noised anomalies that simulate out-of-distribution data. After that, compute the performance of INK at different test-time temperatures the validation set and select the one that achieves the highest AUROC.
D ID Classification Accuracy
Table 5 presents the in-distribution classification accuracy for each training dataset. We evaluate the classification accuracy by performing linear probing on normalized features, following the approach in Khosla et al. (2020). Using vMF loss shows competitive ID classification accuracy compared to standard cross-cntropy loss, indicating it does not compromise the model's capability to distinguish samples between in-distribution classes.
Table 5: ID classification accuracy CIFAR-100 and ImageNet (%).
ID classification accuracy t
Method
CIFAR-100
ImageNet-Ik
Cross-entropy
75.03
76.15
vMF
74.28
76.55
E Results on CIFAR Benchmark
In this section, we present additional results and analysis on the CIFAR-100 benchmarks. Specifically, we utilize CIFAR-10 and TIN as the near-OOD datasets, while MNIST (Deng, 2012), SVHN (Netzer el al., 2011), Textures (Cimpoi et al., 2014), and Places (Zhou et al., 2016) serve as our far-OOD datasets. As shown in Table 6, INK displays competitive performance compared to existing stato-of-tho-art methods. INK achieves average FPR@95 values of 44.51% and 61.83% for the far-OOD and near- -ODD groups. As shown in Table 6, INK shows strong performance the far-OOD benchmark (average FPR 44.51%) %), outperforming
19

---

Published in Transactions on Machine Learning Research (12/2024)
state-of-the-art methods for large distribution shifts. Performance near-OOD, however, is comparable across methods, likely due to CIFAR's lower resolution compared to ImageNet-1K. Compared to KNN+ and CIDER, our method achieves competitive performance while reducing the inference time by more than 300 times (Section 4.3).
Table 6: OOD detection performance for CIFAR-100 (ID) with ResNet-34. INK achieves competitive performance compared to state-of-the-art methods.
PanOOD Datasnts Teslues
Near-OD Darsent
Mithod
AINIST
SVEX
Placs
Averoge
CIFAR-13
TIN
Avcrage
AUROC1
Methods
MBSP
72:36
10.33
13.97
55.05
61:11
73.16
07.82
56.97
18.51
17.00
ODIN
69
62.22
61.94
19.05
70.58
14.30
03.73
75.00
091.12
13.31
12.74
77.64
04.07
99.63
18.70
79.54
51.44
07.81
$4.00
81.73
12.78
$7.40
83.37
10.54
07.90
71.58
88.04
04.30
73.53
03.20
$8.04
08.80
Trwegy
57
77.17
19:95
881.0
70.78
12-30
55.46
76.31
58.13
17:30
61.11
77.51
55.25
85.33
58.51
79.39
VIM
19.25
71.30
38.35
08.07
15:12
18.30
63.41
75.13
31.45
$1.50
12.03
73.31
56.71
80.01
$3.86
76.71
BeAct
193.77
78.35
08.81
81.72
39:38
18.24
$4.03
78.08
B7.06
23-22
11223
77:28
58.21
82.25
18.21
71.23
DIUE
36.50
69.61
62:86
73.52
00:27
05
54.54
50.27
01.53
77.40
58.01
79.36
18.07
78.35
02:00
70.40
78
72.07
06.90
63.04
61:13
73.47
03.31
08.00
76.17
73.03
23.33
05.32
30.75
00.27
hyperspheiical
KNN++
27.31
47:41
53.79
09.35
50.30
79.70
380
25.15
65:37
57.93
77.42
77.07
56.11
08.31
06.15
CIDER
35.
59-14
52.17
74.59
03.91
78.30
Ouzs
08.00
24,80
23.41
98.93
33.50
00.75
08.24
79.77
44.53
$2.30
72.07
74.30
00.90
84.39
61.83
79.27
F Results Vision Transformer architecture
This section details the performance evaluation of different methods for out-of-distribution (OOD) detection using the Vision Transformer (VIT) architecture, focusing the ImageNet-1k dataset. Our approach involves fine-tuning pre-trained ViT-B-16 model over 100 epochs. The initial learning rate is set 0.01, and we employ a cosine annealing schedule for optimization. The performance of various methods is compared in Table 7. INK demonstrates competitive performance in both far-OOD and near-OOD groups, with an FPR@95 rate of 38.64% and 48.80%. This result highlights the robustness and versatility of our proposed method when applied to diverse architectural models. As note, while INK achieves competitive results, it does not consistently outperform all methods.
G Visualization Analysis for Large-scale Dataset
Figure 5 presents the UMAP visualization of the learned embeddings derived from a subset of ImageNet-Il 1k classes and larger-scale out-of-distribution (OOD) datasets. These visualizations indeed reveal compact representations, where each sample appears to be effectively drawn towards its corresponding class prototype. A notable separation between in-distribution (ID) and OOD classes is observed.
Table 7: OOD detection performance for ImageNet (ID) with ViT-B-16. INK achieves competitive performance compared state-of-the-art methods.
Far-OOD Datasets
Near-OOD Datasct
Method
iNacuralist
Textures
Opeulnage-O
Average
NINCO
FPR
AUROC
FPR 1 AUROCT FPR
AUROCT
FPR
AUROCT
FPR
AUROCT
Methods using cross-entropy loss
MSP
42.42
58.19
36.44
85.06
56.11
51.87
51.99
85.04
77.35
78.11
ODIN
79.55
85.62
77.16
85.95
76.72
92.62
65.14
Mahalasobis
20.86
96.01
38.00
89.41
30.35
92.38
29.64
92.60
48.76
86.52
Enezgy
83.58
79.30
83.65
81.17
88.79
76.48
85.34
78.65
94.16
66.022
VIM
17.59
95.72
10.11
ReAes
48.22
86. 11
$5.87
86.66
57.68
$4.29
53.26
85.69
78.50
75.43
DICE
47.92
82.51
54.79
82.21
52.57
82.23
51.76
82.32
$1.09
71.67
SHE
22.17
93.37
25.65
ASH
97.02
50.63
98.49
48.53
94,80
55.52
96.10
51.56
95.40
52.52
Methods using hyperspherical representation
CIDER
93.27
INK (ours)
28.15
92.90
50.57
85.16
36. 80
89.28 28
38.61
89.11
48.86
81.55
20

---

Published in Transactions on Machine Learning Research (12/2024)
ImageNet (ID) and INaturalist (OOD)
ImageNet (ID) and Textures (OOD)
ImageNet (ID) and Openlmage-O
Labels
Labels
Labels
Goese
Crane
Tiger
Tiger
Tiger
Zobra
Zebra
Zebra
Castle
Castle
Coséle
Geogle
Beagle
Beagle
Miroohhone
Microplore
Nicrophone
Boe
Bee
Boe
Flastic
Peodic
bag
Figure UMAP visualization of a subset of ImageNect classes and OOD datascts. The class prototypes are designated by a star symbol *, while the OOD embeddings are distinguished by pink color.
H Limitation
A key strength of our method its grounded interpretation from likelihood perspective. This approach, however, necessitates training the model under the vMF distribution. While our method achieves accuracy comparable to stale-of-the-art models in certain scenarios, it's important to note that it is less common than models trained using traditional cross- cross-entropy loss. This may limit its immediate applicability in environments where standard cross-entropy loss models are the norm. However, we believe that the trustworthiness of the models is worth the effort training them a certain way.
21


Reviews:
[{"review_id": "VNtsslQ6GB", "reviewer": "Reviewer_w67q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nThe paper \n1. is well-written and well-presented.\n2. provides theoretical insights.\n3. provides extensive experiments to compare the proposed method with the state-of-the-art approaches.\n4. honestly discusses the limitations of INK.\n\n**Weaknesses**:\n1. The model needs to be trained under the vMF distribution, which can limit the usage of INK in practice.\n2. According to the results in Table 6, it seems that unlike ImageNet-1k, INK is less efficient on CIFAR-100 compared to the state-of-the-art methods. Similar observation holds for ViT (Table 7).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel framework called INtrinsic liKelihood (INK) for out-of-distribution (OOD) detection in classification tasks. Through extensive experiments, the authors show that INK provides competitive performance compared to the state-of-the-art methods on different datasets including ImageNet-1K and CIFAR-100 as well as multiple architectures such as ResNet and Vision Transformer (ViT).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The paper argues that the problems with previous works are: (I) \"heuristic-driven and cannot be rigorously interpreted as log-likelihood\", and (II) \"impose strong assumptions on the density function that can fail to hold in practice\". \nThe authors should explain why inability to interpret a method as log-likelihood is necessary a limitation for a given method. Moreover, they should clarify whether INK suffers from problem (II), i.e. strong assumptions on the density function.\n\n2. The authors should explain why their method on CIFAR-100 and ViT is less efficient compared to ImageNet-1K and ResNets. Does it relate to the number of classes of the dataset, the resolution of the images, the convolutional layer vs. linear layer, the normalization layer (BatchNorm in ResNets and LayerNorm in ViTs), and etc?\n\n3. Based on (2), the authors should revise some claims in the paper such as \"As shown in Table 7, our INK score maintains competitive performance with the ViT model\". To me, INK underperforms the existing methods in many cases for the ViT architecture and CIFAR-100.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e1YKbnIBnz", "reviewer": "Reviewer_UJfx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Inspired by vMF distribution in the hypersphere, the authors provide Intrinsic Likelihood Score for OOD detection with theoretical analysis.\n- They discuss the optimization issue of their method in DNN and class-Imbalanced datasets in practice.\n- The extensive experiments impress me, and the proposed method achieves state-of-the-art performance across various OOD scenarios, which is promising.\n\nWeaknesses\n- The assumption in Theorem 3.1 appears quite strong, as it requires a uniform class prior. In practice, datasets are often imbalanced, so relaxing this assumption would render the derivation for Theorem 3.1 inapplicable, making the theorem’s applicability somewhat restricted. \n- While the authors discuss \"Class-Imbalanced Datasets\" in Section 5, they show that their metric \"maintains functional equivalence to the log-likelihood for OOD detection.\" So if your method is equivalent in principle to the previous method, why is your method better in practice?\n- I'm not sure why the proposed method performs better than existing methods. Does it requires significant hyper-parameter tuning?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work addresses a key issue in OOD detection by introducing INK, a novel OOD scoring approach that provides a rigorous likelihood-based interpretation for discriminative classifiers. INK operates on constrained latent embeddings modeled as a mixture of hyperspherical embeddings with constant norms, bridging hyperspherical distributions with intrinsic likelihood. Experiments show that INK achieves state-of-the-art performance across various OOD scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n- Please provide more evidence and support for the uniform class prior assumption in Theorem 3.1, especially in the context of OOD detection. The authors can also address my concern by discussing why the proposed method is equivalent in principle to the previous method but achieves better performance in practice.\n- Include a more thorough breakdown of why specific methods perform as they do, perhaps through additional ablation studies or statistical tests. More detailed discussions of the results can provide stronger evidence and help justify the effectiveness of the proposed method.\n\nClarity:\n- What is the motivation of introducing \"log\" in Definition 3.2 (Intrinsic Likelihood Score)? Mathematically speaking, if there is no \"log\", the derivation of (8) changes from \"+\" to \"×\"?\n- The authors may combine Section 5 with some discussion in Section 3.1? From my perspective of view, the content in Section 5 can be broken apart and incorporated in previous sections.\n\nTypos:\n- bold $\\theta$ in equation (14)?", "broader_impact_concerns": "I have no concern since there is enough discussion in the \"Broader Impacts\" section in the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oFLkNzrWef", "reviewer": "Reviewer_nD6V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The theoretical analysis uses a clever technique to do OOD detection that leverages the way in which we already train ML models.\n2. Evaluation is extensive and clearly shows via many comparisons that **INK** beats previous work in both performance and efficiency at a fixed true positive rate.\n3. The authors thoroughly explore other features of **INK** such as its relationship with existing work, its computational cost, sensitivity to hyperparameters, etc.\n\n**Weaknesses**\n\n1. Evaluation methodology for the proposed method is somewhat unclear. Although the authors *do* describe how to optimize neural networks under their framework in Section 3.2, the evaluation that follows doesn’t make it clear how this is done, or whether this is comparable to standard practices.\n2. In the setting where one wants to detect OOD examples, false negatives can be costly. Given that the method is better than prior work, analysis at lower FNR (higher TPR) would be interesting (e.g. 0.1% FNR). How does the FPR change as the fixed TPR increases?\n3. **Minor Comment**: The theoretical results depend on the assumption that the embeddings lie on the unit hypersphere, which is an assumption used in prior work. This enables the authors to find a nice result, but has practical constraints on the implementation of **INK**.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method, **INK**, to check whether a given sample at inference time is out of distribution (OOD). By leveraging the way in which classifiers are implicitly trained to maximize the likelihood of embeddings matching in distribution (ID) data, the authors create a new, efficiently computable score for OOD detection. They evaluate **INK** on several benchmarks and against existing OOD detection methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explicitly describe how the attack is implemented. For example:\n\n    * Are the models in the evaluation trained in a standard way?\n\n    * In the implementation of **INK** is $\\mu_c$ computed by taking the empirical mean of the embedding pool? \n\n\n2. Describe the practical limitations of working with the assumption of embeddings being distributed over the unit hypersphere.\n\n3. If possible, explore whether the method can be (at least heuristically) adapted to standard classification or contrastive learning settings.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a novel OOD detection method, called INtrinsic liKelihood (INK). This is inspired by the fact that classifiers are implicitly trained to maximize the likelihood of embeddings matching in distribution (ID) data. Extensive empirical results verify the effectiveness of the proposed detection method. 

**Decision** All the reviewers appreciate the technical and experimental contributions. They all recommended the acceptance, I quickly went through the paper and agreed with their recommendations.
Decision: Accept (poster)