Learning to Shape In-distribution Feature Space for Out-of-distribution Detection
Authors: Yonggang Zhang
Jie Lu
Bo Peng
Zhen Fang
Yiu-ming Cheung
Venue: NeurIPS 2024 poster (2024)
Keywords: Out-of-distribution Detection
Primary area: safety_in_machine_learning

Abstract: Out-of-distribution (OOD) detection is critical for deploying machine learning models in the open world. To design scoring functions that discern OOD data from the in-distribution (ID) cases from a pre-trained discriminative model, existing methods tend to make rigorous distributional assumptions either explicitly or implicitly due to the lack of knowledge about the learned feature space in advance. 
The mismatch between the learned and assumed distributions motivates us to raise a fundamental yet under-explored question: \textit{Is it possible to deterministically model the feature distribution while pre-training a discriminative model?}
This paper gives an affirmative answer to this question by presenting a Distributional Representation Learning (\texttt{DRL}) framework for OOD detection. In particular, \texttt{DRL} explicitly enforces the underlying feature space to conform to a pre-defined mixture distribution, together with an online approximation of normalization constants to enable end-to-end training. Furthermore, we formulate \texttt{DRL} into a provably convergent Expectation-Maximization algorithm to avoid trivial solutions and rearrange the sequential sampling to guide the training consistency. Extensive evaluations across mainstream OOD detection benchmarks empirically manifest the superiority of the proposed \texttt{DRL} over its advanced counterparts.

Learning to Shape In-distribution Feature Space for Out-of-distribution Detection
Yonggang Zhang', .Ji Lu Bo Peng", Zhen Fang", Yiu-ming Cheung
'oong Kong Baptist University
2 Australian Artificial Intelligence Institute, University Technolog Sydney
Abstract
outoof-distribution (OOD) detection critical for deploying machine learning models in the open world. To design scoring functions that discern OOD data from the in-distribution (ID) cases from pre-trained discriminative model, existing methods tend to make rigorous distributional assumptions either explicitly implicitly due to the lack of knowledge about the learned feature space in advance. The mismatch between the learned and assumed distributions motivates raise fundamental yet under-exploree question: Is it possible to deterministically model the feature distribution while pre-training discriminative model? This paper gives an affirmative answer to this question by presenting a Distributional Representation Learning (DRL) framework for OOD detection. In particular, DRL explicitly enforces the underlying feature space to conform to pre-defined mixture distribution, together with online approximation of normalization constants to enable end-to-end training. Furthermore, we formulate DRL into provably convergent inpectation-Maxiiizzan
algorithm to avoid trivial solutions and rearrange the sequential sampling to guide the training consistency. Extensive evaluations across mainstream OOD detection benchmarks empirically manifest the superiority of the proposed DRL over its advanced counterparts.
Introduction
Despite the significant progress in machine learning that has facilitated broad spectrum of classifi- models often operate under closed-world scenario, where test data stems
cation tasks 45 from the same distribution the training data. However, real-world applications often entail scenarios which deployed models may encounter unseen classes of samples during training, giving rise to what is known oui-of-distribution (OOD) data. These OOD instances can potentially undermine a model's stability and, in certain cases, inflict severe damage on its performance. Accordingly, reliable discriminative model should not only correctly classify known In- In-Distribution (ID) samples but also flag any OOD input as "unknown". This directly motivates OOD detection [3] [6] [4],
which makes significant differences in ensuring the safety of decision-critical applications, c.g.. autonomous driving 26 medical diagnosis 85 and cyber-security [11]
Up to now, a plethora of OOD detection algorithms have been developed recently by leveraging post-hoc analysis on the pre-trained model. The seminal work (2 leverages the maximum softmax probability (MSP), also known as the softmax confidence score, for OOD detection, which is built upon the hypothesis that OOD data should trigger relatively lower softmax confidence than that of ID data. ODIN (35] extends MSP by using temperature scaling and input perturbation to amplify the ID/OOD separability. Approaches of this category, albeit intuitive, are challenged by the observation that deep neural networks are prone produce over-confident predictions, i.c. abnormally high softmax confidences, even though the inputs are far away from the training data [50]. As a result,
*Correspondence Yiu-ming Cheung (ymc@ @ome@comm.hie comp.hkbu.edu..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

advanced methods turn to design alternative OOD scoring functions by resorting to the stored ID
patterns in gradients [25], intermediate features 2 34 62
36
57
76 77 and logits [2] 38 67
78]. However, most of these methods suffer from the lack of inherent connections and theoretical understandings
[8] regarding the detectability of OOD data.
Given that OOD data, by definition, inherently diverges from ID data by means of their data density distributions, some works [3 48 54 are motivated to focus on constructing scoring functions that can effectively replicate the behaviour of the ID density function, which opens the door to density- based OOD detection. Despite the empirical success, the power of density- based OOD detection has yet to be fully unleashed even with the helm of feature-shaping shaping [1] 59 60 72 79, 80 84 and neuron pruning [ 61). This is because current methods struggle with making a strong distributional assumption of the underlying feature space either explicitly or implicitly due to the lack of knowledge about the learned feature space in advance. The urgent need to cast off the dilemma that those pre-defined distributions fail to necessarily hold in practice 62 directly promotes the following important yet explroex question:
Is it possible to deterministically shape the ID feature distribution while pre-training a discriminative model?
Methodological Contribution. In this paper, we propose a novel learning framework called Dis- tributional Representation Learning (DRL) that bridges the gap between network pre-training and density-based scoring strategy. At high level, we explicitly enforce the underlying feature space to conform to pre-defined distribution, inspired by A1 In this way. OOD detection can be naturally approached in an assumption -ree manner, hence providing stronger flexibility and generality. When implementing our idea, it could always be non-trivia to parameterize the ID data distribution since the computation of normalization constants tends to be costly and even intractable [1)]. Towards this dilemma, instead of introducing impractical constraints to make the normalization constants input-independent or known, we propose an online approximation of normalization constants to enable end-to-end training.
Theoretical Contribution, Importantly, we provide a theoretical framework that formulates DRL as an Expectation- Expectation-Maxxiizaalll Maximizaaionn (EM) algorithm 47. In particular, by introducing latent variables as ID classes, we obtain the Bayes-optimal inference of the posterior distribution of the latent variables given the observed data in the E-step. In the M-step, we propose to maximize the evidence lower bound (ELBO) of the log conditional likelihood with respect all parameters so that the optimization process can not only benefit from the classification process but also avoid leading to trivial solutions. We prove that the ELBO is bounded, and the EM algorithm contributes to the convergence of ELBO. Moreover, we rearrange the sequential sampling by constraining half of each mini-batch coinciding with the previous iteration, which deals with the inconsistency issue caused by the integration of the EM algorithm into the batch-based training routine.
Empirical Contribution. We extensively evaluate DRL on mainstream OOD detection benchmarks and establish titt--ofthheassd performance compared with three families of methods: (1) pre- trained with cross-entropy, (2) pre-trained with contrastive learning, (3) pre- trained with cross-entropy and fine-tuned with training- time regularizations. For example, on CIFAR-10. 10, DRL achieves 11.58% FPR95 on average, significantly outperforming PALM 440 by 3.38%. Further, for completeness, we extend the evaluation of our method to more strict settings, including (1) large-scale OOD detection, (2) hard OOD detection, and (3) unsupervised OOD detection.
2 Preliminary
Notations. We write vectors as bold-faced lowercase characters. Considering K-way classification as a case study, use X' and y {1,..., K} indicate the input space and ID label space, respectively. The joint ID distribution, represented as Pxir, is a joint distribution defined over X' x y. During testing time, there are some unknown OOD joint distributions Px.Yo defined over X' x ye, where y is the complementary set of y. We also denote p(x) as the density of the ID OOD detection can be formally defined as follows: marginal distribution Px. According 11 16
Problem (OOD Detection) Given a labeled ID dataset D ((x1, 91 31), NxN, yN which is drawn from Px, independent and identically distributed, the aim of OOD detection is to learn
2

---

predictor g(-) by using D such that for any test data x: 1) if is drawn from Px, then g can classify X into correct ID classes, and 2) if drawn from Pxo. then g can detect OOD data.
OOD Scoring, Existing methods [25] 62 34 67
tend to adopt a post-hoc strategy to detect OOD samples, L.., given well-trainee discriminative model fe X + R using D, and scoring function S, then x is detected as ID data if and only if S(x: fo) 2 X, for some given threshold A:
g(x) = ID, if S(x; Jo) 2 X; otherwise, g (x) = OOD.
(1)
natural view for the motivation of the post-hoc strategy to use a level for ID density p(x) to discern ID and OOD data. To be specific, the main objective is to construct an efficient scoring function that can effectively replicate the behaviour of the ID density function p(x) such that S(x; fo) & p(x). From this perspective, let po (x) be the ID density function estimated by the discrimination model fo. we can rewrite Eq. ① as follow:
g (x) ID, if po (x) 2 >; otherwise, gxx) OOD.
According to prior works, the design principle for pe(x) can be either logit- bast-b or feature-basse based.
Logit-based OOD Methods derive (x) by formulating the discriminative model fo as an energy- based model 18 32 where collection of energy values are turned into probability density po (x)
by implicitly resorting to the Hibbs-Boltzman distribution, i.e.
(x = exp (x)T/ x exp Fo(2)/T) Fo(z) = -T log Σ exp (Ta/T),
K
(3)
Z
where Z (-x((=Eo( dx is input-independent normalization constant and [T1, [T1,..., Tk] E RKK denotes the logit vector produced by classification layer with the interme- diate feature IR as the input. > O denotes a temperature hyperparameter and, when - 0, the negative energy score 37 - F-(()) will degenerate into the MaxL.ogit score 21 78 namely,
K
lim (z(z) = lim T log Σ exp (rw/T)= max Tk- kEy
T-O
Feature-based OOD Methods derive po (x) by explicitly assuming the intermediate feature space Z learned by the discriminative model fe to follow a pre-defined distribution. Therein, a representative example [48] is Gaussian mixture distribution under uniform ID-class prior (i.e., po 1/K). Formally, let (M rLI) denotes the k-th ID class-conditional Gaussian distribution with a tied covariance matrix 72, then we have
K exp
(z (4) (TE) (z 1c)
K
(x(
exp
2 (2 (TE) mn)
K
(4)
Clearly, Eq. (4) is equivalent to the GEM score intra-class intensity and 7 -> 0 degenerates Eq. into the maximum Mahalanobis distance 33 (up to constant), i.e.,
34 when t 1. Reducing gradually reinforces the
exp
(z Mx) (TZ (rE)-
(z m4)
lim 7 log (x) = lim log T-0
K
k=1
lim log T00
exp
(2 m4x T (z 1x)/T
lim 7 log K V (22))//T2) +00
2
k=1
max key
- -(z
kx) I (z pk)
lim log K KV (2m)
constant VXEX
Methodology
3.1 Motivation
We hereby present the motivation behind our work. light of the aforementioned discussion, can be found that existing OOD detection methods tend to make distributional assumptions pe (x)
3

---

either explicitly or implicitly with the aim of density estimation due to the lacking true knowledge of (x)).. However, these distributional assumptions would not be held in many practical scenarios, leading to the mismatch between distributions of the learned and assumed features.
To illustrate this for the logit-based methods, forge mathematical connection between the classification objective and ID density function pe
exp (ry/T)
K
log
TTyT log
exp (Ta/T) = Ty/T log (x),
(5)
E exp exp (Tu/T)
where the constant Eq. 3 Maximizing the classification objective in Eq. 5 is to maximize the logit 7& while minimizing the estimated density function pø (x).. This implies that training discriminative model drives the ID marginal distribution away from the
case. This cannot support the idea that the ID marginal distribution can necessarily align with the Gaussian distribution in the feature- ones. Furthermore, it has been argued by [77] that the learned latent features fail the Henze-Zirkie multivariate normality test (24). which challenges the rationality of Gaussian- based distributional assumptions. We notice the possibility of non non-parametric density estimation based on nearest-neighbor distance [17] 62 in the latent feature space. Albeit assumption- free, this practice hampers the scalability as it takes ((N)) memory to store the latent feature of ID training samples and (NNd) computation for neighborhood discovery. Most recently, [54] empirically shows that parametric density estimation can significantly benefit OOD detection more than the non- con-paraettric counterpart if more reasonable distributional assumptions are introduced.
However, the bottleneck OOD detection still lies in the consistency between the assumed distribu- tion and the unknown ground truth. In view of this, this paper proposes to deterministically shape the learned feature distribution while training a discriminative model so that we can relax the widely adopted distributional assumptions.
3.2 Data distribution modeling
To preserve inter-class structures, this work, following common practice K 46 explicitly models
the ID) marginal distribution Px, as weighted aggregation of ID class-conditioned distributions, ie.,
K
log (x = log po(x(k) pk))..
(6)
k=
Consistent with probabilistic theory, po (x(x))) can be formulated into the following general form:
Pe (x{]k = h(z, k)
((k) - /
h(z, k) dz.
(kk
Z
where (z, k) represents a non-negative density function defined over the lower-dimensional latent feature space in which high-frequency and imperceptible details are abstracted away [55]. Note that our method is generic to the choice while this paper, inspired by [69]. focuses on an exemplar
based on the Euclidean distance, i.e., 1ız, k) (- 14 /2T), that is closely connected
to universally optimal point configurations 4 g. Motivated by [13] 46). further introduce &
normalization over the latent feature space to keep ||z||| = 1. Without loss of generalization, let
Kuj = /2/ || and E
|| /T, we then have:
exp
-(1 - 1a z
11 11) /
exp(cx
Po (x
(8)
J exp ((1 - 2. 1mx
de'
T T(p), Ex)
where N (ptg, (4) = JAEz exp
(cusii 2) dz.
3.3 Online approximation of normalization constant π (figg <k)
Regarding the computation of (x(k)) in Eq. 88) it is imperative to accurately calculate the nor- malization constant (f²,, <x) that seems to be intractable due to the integral over the latent feature space. When connecting Eq. (8) with the well-known von Mises Fisher (vMF) distribution, one

---

can easily check that the normalization constant π (p(+) denotes the modified Bessel function of the first kind and order p. Unfortunately. Ip(.) is complicated function. To cast off this dilemma, the most straightforward idea, as 13 46) involves fixing the & norm of Na to be a constant value, e.g., ||11112 = 1, vk. This will result in
= / +
(4x)) where
Ek = 1/TT, Vk and therefore allows one to bypass the computation of the normalization constant T (iug, Ea) as T T(pk, ex) turns to be a class-independent constant term during optimization. Despite the simplicity, it crucial to acknowledge that this practice implicitly implies the rigorous assumption that all classes ought have similar level of concentration, and would likely reduce the flexibility of the mixture model in Eq. (6) Indeed, this inflexibility in the latent feature space tends to deteriorate the learning of the feature mapping function. In this paper, we proceed from different perspective and propose to use an online approximation thereof. To be precise, with a sufficiently large p. (.(.) can be approximated using the following uniform expansion [39, i.e.
1p(pg) ~ (24p) + Q2)1/4
exp (pw)
(5(s) == p v (1+ 1222 + log
q 1 + (1 + 92)
(9)
92)
where ~ denotes a Poincaré asymptotic expansion with regard to p, == (1+ q2)/22 and polynomials (s). Empirically, we find that p 127, which corresponds to the latent feature dimension d 256, and evaluating only the first term in the sum, i.e. (s(s) = 1, are sufficiently enough to contribute to satisfactory approximation while keeping simplicity. Please refer to Appendix A for more details.
3.4 DRL as Expectation-Maximizaan
In practice, is hard to directly optimize the log- bog-liielhhood function of trivial solutions in which the latent feature space either scales up towards infinity or collapses to the origin point. To avoid collapse while making the classification process maximally beneficial to distribution modeling, let us start from the following evidence lower bound (ELBO) of log (x):
64 68 due to the existence
log px)
Σ log - KL [g(/xx) || pe po (4)] + KL (x<k)) ((o
(10)
2 Σ (k) KL l| EL
where [-] [-] denotes the Kullback-Leibbe (KI.) divergence. We derive this ELBO in Appendix B. To make the inequality hold with equality so that the ELBO reaches its maximum value log p(x). we require KL (g(k) x) | [k[)] = 0. By replacing (k((x) with pa (k|x), we can formulate the (EM) framework.
maximization of the ELBO into the
E-step. With the fixed 0, at the iteration t, this step aims to estimate 4++1 (k|x) to make qt+1 (k:xx)
Pe, (k)x) so that we can have ELBO gc+1. x; 0:) = pe, (x). Here, we estimate pe, (k x) with the soft labels produced by the discrimination model at the iteration f, i.e. f parameterized by 0.. To
this end, by applying Bayes' theorem to po, (x), we approximate (k((((()) as:
(k|x) = po kb (k) = ΣK explexii 2) (ft,) ex)
expleyf T 2)/ (ity
(11)
ΣK Po (x (c)p((c)
M-step. With the sub-optimal qc-1 (k|x) = pe, (k|x) after E-step, we turn to maximize the ELBO. Given that po (k) = 1/K and qt-1 (A|x) is fixed during optimization, the KL term in ELBO x; 0) can be reduced to a constant form KL (g(k ll [o(] = log K + H [x(][))], which results in:
K
0 0+11 = arg max G.
Σ
(k(xx) log (x(k)
(12)
Convergence. At the E-step of the iteration 6, we estimate qt+1 (k|x) ensure ELBO x; 0,) log pe, (k))). At the M-step after the E-stp,, we have obtained 0.41 with a fixed q++1 (k|x) such that ELBOOgq+1+ x; 0.41) 2 ELBO(q(41, x: 0t). Accordingly, we arrive the following sequence:
log pe (x) > ELBO(g.41. gt 0(41) Z ELBO x; 0.) = log (x).
(13)
5

---

Since log Po..: (k(x) 2 log Po, (k:x)), one can guarantee that ELBO(g, x; can converge to a certain value with the proposed EM framework.
0) is upper- upper-boundee bounded and
Implementation. When integrating the EM framework into the batch-based training routine, we note that the optimization of (LB x: 0) would suffer from the inconsistency issue. This is because the prediction pe (k|x) only gets updated when x is fed as the input. In other words, the prediction pe (k|x) is only updated once per training epoch. However, the discriminative model fe keeps updated throughout mini-batches in the training epoch. To address this problem, we rearrange the sequential sampling by constraining half of each mini- mini--btcc coinciding with the previous mini-batch. Meanwhile, the rest of the half will coincide with the next mini-batch can obtain on-the-fly soft labels generated by the discriminative model fo in the previous mini-batch. For a clear notation, let B. = {Bo B be the m-th mini-batch sampled from D in the current training epoch, we then have Buext Bprr and B 0.
In this way. half of the current
3.5 Overall training objective
Our method enables end-to-end training, where the overall training objective R(:; is to maximize linear combination of the classification objective (t. :; and the ELBO in Eq. (10), i.e.
R(Bm: 0) (x,.)) Rxts y; 0) + (x.)) ELBO x; 0),
(14)
where the hyperparameter B > O modulates the relative importance of two losses. Based proposed distribution modeling in Eq. (6). the classification for a sample x can take place with a Bayes-based rule instead of a parametric softmax layer. As such, the the classification objective Reta(x (x, y; 0) naturally turns to be:
exp(En 2) (i) Ex)
Rets (x, 0; = log
(15)
Dk=1 exp(Exi* 2)/T(ppx:
Remark. While recent works 13 40 46 come with Bayes-based classification objective as well by modeling the latent feature space as mixture of vMF distributions. However, we emphasize that optimizing the classification objective alone does not necessarily drive the extracted training features towards the pre-defined distribution. For example, a latent feature can be far away from the corresponding class prototype H, while still being correctly classified as long as Z is relatively closer to H,, than to other class prototypes. We, accordingly, mitigate this problem with the ELBO in Eq. (IO) to explicitly measure the extent to which a training sample fits the assumed distribution. Besides, these methods are limited by assuming the concentration parameter 6Y to be class-uniform for a simplified computation of Eq. (15) even though [13 considers optimizing G along with the discriminative model. By contrast, method learns Ca directly through the prototype magnitudes while proposing an online approximation of the normalization constant T (ūt, e&) to enable end-to-end training.
Related work
ciitr--Distribution Detection has attracted surge of interest in recent years, which is motivated by the empirical observation 50 that neural networks tend to be over-confident in OOD data (52] One
line of work performs OOD detection by devising post-hoc scoring functions, including confidence-
based methods [21 38 78) mmsed methods 67], distance-basse approaches [4 62
36 57 76 77), gradient-based approaches (25], and Bayesian approaches 29 44). Another line of work addresses OOD detection by fine-tuning a pre-trained discrimination model with training-time regularizations that help the model learn ID/OOD discrepancy following the guideline of outlier exposure 23 or negative prompts [53] For instance, the discriminative model is regularized to
produce lower confidence [33 43 smaller feature magnitudes [37 or higher energy [10] for outlier points. More recently, some works have considered a practical scenario where the auxiliary outliers can be arbitrarily different from the real OOD data, therefore, distributionally augmenting the observed OOD data. Besides, the given OOD samples tend to include unlabelled ID counterparts (27]. In view of this, WOOD (27 formulates learning with noisy OOD samples as a constrained optimization problem while SAL [12 separates candidate outliers from the unlabeled data and then trains a binary classifier using the candidate outliers and the labeled ID data. Most regularization methods, unfortunately, assume the availability of auxiliary OOD data, not to mention resource-intensive re-training processes, while our method maintains the same training scheme as standard cross-entropy loss without requiring additional OOD data in training.

---

Table 1: OOD detection results on the CIFAR-10 benchmark with ResNet-18. indicatess larger
values are better and vice versa. The best results in the last two columns are shown in bold.
Method
SVHN
Places365
LSUN
ISUN
Texture
Average
FPRUS]
AUROC1
FPR95)
AUROC
FPROEL
AUROCT
FPR95)
AUROCT
FPR95]
AUROC1
FPR95
AUROC
MSP
59.66
91.25
62.46
88.64
51,93
92.73
$4.57
92.12
66.45
88.50
59.01
90.65
95.55
86.57
94.65
Energy
54.41
91.22
37.22
92.70
10.19
98.05
27.52
95.59
55.23
89.37
36.91
93.39
92.32
96.51
ASH
28.94
94.84
27
91.31
9.06
98.34
21.61
95.95
35.02
93.63
27.29
94,81
Maha
9.24
97.80
83.50
69.56
67.73
73.61
98.63
23.21
92.91
37.94
86.50
KNN
27.97
95.48
87.84
89.93
18.50
96.84
24.68
95.52
26.74
94, 96.96
29.15
94.55
CONT
18.71
96.48
53.44
89.18
22.20
95.95
22.64
95.87
25,80
95.11
28.56
94,52
Vim
24.95
95.36
63.04
86.57
7.26
98.53
33.17
94.65
56.40
86,21
36.96
92.26
VOS
15.69
96.37
37.95
91.78
27.64
93.82
30.42
94.87
32.68
93.68
28 88
94,10
37.38
94.69
38.31
93.04
10.63
97.93
98.01
SSD+
2.47
99.51
22.05
95.57
10.56
97.83
28.44
95.67
9.27
98.35
14.56
97.38
KNN+
96.21
CIDER
2.89
99.72
23.05
94,09
5.45
99.01
20.21
96.64
12.33
96.85
12.95
97.26
PALM
0.34
99.91
28.81
94,80
1.11
99.65
34.07
95.17
10.48
98.29
14.96
97.57
DRL
7.91
98.82
19:17
95.65
12.87
99.09
11.92
98.12
4.92
97.48
11.58
97.83
Representation Learning for OOD Detection is emerging topic to enhance the performance of post-hoc distance-based OOD detectors. In particular, CSI [63] investigates the type data
augmentations that are particularly beneficial for OOD detection while other works [58] 70] verify
the effectiveness of applying the off-the-shelf multi-view contrastive losses such as SimCLR (5
and SupCon (28) for OOD detection. CIDER [46] proposes prototypical contrastive learning
framework for OOD detection by promoting stronger ID-OOD separability than SupCon loss, where regularization strategy is to ensure that all samples are compactly located around their corresponding class prototype. PALM [40 extends CIDER by introducing mixture of prototypes to represent each class and performing prototype-level contrastive learning to enhance intra-class compactness and inter-class discrimination. This paper significantly differs from PALM and CIDER in the following two aspects: 1) we avoid normalizing the class prototypes protect the generalization ability of the data distributional modeling, and 2) we learn a feature space that best describes the pre-defined distribution rather than features that preserve inter- and intra- class structures.
5 Experiments
Software and Hardware. We perform all experiments NVIDIA A100 GPU using Pytorch.
Baseline Methods. We compare DRL with representative methods, including MSP ODIN 35
Energy [37]. ReAct (37). ASH [II], Mahalanobis (Maha) (34), KNN (62). CONJ [54]
Vim [66].
VOS 14), CSI (63) SSD+ (58), KNN 337, CIDER [46), and PALM [40]. It worth noting that we have adopted the recommended configurations proposed by prior works, while concurrently standardizing the backbone architecture to ensure equitable comparisons.
Evaluation Metrics. The detection performance is evaluated via three widely used metrics: 1) the false positive rate of OOD data measured when the true positive rate of ID data reaches 95% (FPR95): 2) the area under the receiver operating characteristic curve (AUROC) is computed to quantify the probability of the ID case receiving higher score than the OOD case. The reported results of DRL are averaged over 5 independent runs.
5.1 Main results on CIFAR benchmarks
Following the setup in 46 40 we consider CIFAR-I0 30 and CIFAR-100 30
ID datasets and train ResNet-15 18 20 and ResNet-34 34 (20 on them respectively. We train the model using stochastic gradient descent with momentum 0.9, and weight decay 10 44 for 500 epochs. The initial learning rate is 0.5 with cosine scheduling and the batch size is 512. There are six datasets for OOD detection with
regard to CIFAR benchmarks: SVHN 449, LSUN 75 iSUN [3], Places [82), and Textures (6). At inference time, all images are of size 32x32. Tables and presented the performance of different methods, where our method significantly outperforms existing methods. Specifically, compared with the mostly advanced PALM, DRL reveals 3.38% average improvements w.r.t. FPR95 CIFAR-1O.
7

---

Table OOD detection results on the CIFAR-IO 100 benchmark with ResNet-34. indicates larger
values are better and vice versa. The best results in the last two columns are shown in bold.
Method
SVHN
Places365
LSUN
ISUN
Texture
Average
FPR95]
AUROC1
FPR95)
AUROC:
FPROEL
AUROCT
FPRS5)
AUROCT
FPR95]
AUROC1
FPR95]
AUROC
MSP
78.89
79.80
$4.38
74.21
83.47
75.28
84.61
74.51
86.51
72.53
83.12
75.27
75.23
78.70
79.1
Energy
66.91
85.25
81.41
76.37
59.77
86.69
66.52
84.49
79.01
79.96
70.72
82.55
83.55
79.99
ASH
52.96
90.19
72.62
76.38
75.18
76.52
55.55
81.86
56.17
86.75
62.50
83.53
Maha
87.09
80.62
84.63
73.89
84,15
79.43
83.18
78.83
61.72
84.87
80.15
79.53
KNN
46.25
90.39
82.08
75.44
60.85
85.61
71.56
86.28
62.39
83.95
64.63
84.33
CONT
46,19
90.44
$0.81
75.83
60.45
85,90
64.62
87.77
62.13
83.97
62.84
84,74
Vim
73,42
84.62
85.34
69.34
86,96
69.74
85.35
73.16
74.56
76,23
81.13
74.62
VOS
43.24
82,80
76.85
78.63
73.61
84.69
69.65
86.32
57.57
87.31
64.18
83,95
92.65
76.27
SSD+
31.19
94.19
77.74
79.90
79.39
85.18
S0.85
84.08
66.63
86,18
67.16
85.91
CIDER
23.09
95.16
79.63
73.43
16.16
96.33
71.68
82.98
43.87
90.42
46.89
87.67
PALM
3.29
99.23
64,66
84.72
9.86
98.01
28.71
94.64
33.56
92.49
28,02
93.82
DRL
20.15
94.07
76.64
77.55
16.97
94.63
32:57
92.33
31.97
92.09
35 66
90.13
5.2
Extensions
Hard OOD Detection. We consider hard OOD scenarios, in which the OOD data semantically similar to those of the ID cases. With the CIFAR- 100 as the ID dataset for training ResNet-34 34. we evaluate our method 4 hard OOD datasets, namely, LSUN-Fix [7],, ImageNet-Fix [9], ImageNet- Resize (9). and CIFAR-10. 10. We select a set of strong baselines that are competent in hard OOD detection with results in Table It can be found that method beat the state-of-the-ar across
the considered datasets, even for the challenging CIFAR-100 versus CIFAR-10 setting.
Table 3: Evaluation on hard OOD detection tasks. 1 indicates larger values are better and vice versa. The best result in each column is shown in bold.
Method
LSUN-Fix
ImtgeNet-Fix
ImsseeeetResize
CIFAR-10
Average
FPR95]
AUROC*
FPR95
AUROC
FPRO5
AUROC1
FPR9S
AUROCI
FPR95
AUROCT
SSD+
83.36
76.63
76.73
79.78
83.67
81.09
85.16
73.70
82.23
77.80
KNN+
84.96
75.37
75.52
79.95
68.49
84.91
84.12
75.91
78.27
79.04
CIDER
90.94
70.31
78.83
77.53
56.89
87.62
84.87
73.30
77.88
77.19
PALM
77.15
77.24
66.19
82.51
27.02
95.03
87.25
72.28
64.40
81.76
DRL
68.64
78.75
59.92
87.19
40.85
91.11
74.12
80.93
60.88
84.50
Unsupervised OOD Detection. To verify the reliance of method the availability of ground- truth labels, we consider unsupervised OOD scenarios. Following [40], we take unlabelled CIFAR- 100 the ID dataset train ResNet-34 from scratch. Due the lack ground-trutt labels, resort to maintaining a momentum teacher, inspired by DINO [83] to produce soft pseudo-labels as surrogate. For fair comparison,
the same data augmentation techniques with 40)
Table 4: Evaluation unsupervised OOD detection tasks. f indicates larger values are better and vice versa. The best results in the last two columns are shown in bold.
Method
SVHN
Places365
LSUN
ISUN
Texture
Average
FPRUSI
AUROCT
FPR95:
AUROC*
FPR9SL
AUROCT
FPRU5L
AUROCT
FPR95|
AUROC1
FPR95
AUROC
KNN
72.97
15.49
76.75
SSD
60.13
86.40
79.05
73.68
61.94
84.47
84,37
75.58
71.91
83.35
71.48
80,70
CSI
86.23
87.79
92.13
PALM
13.86
97.53
85.63
69.46
21.28
95.95
53.43
89.06
42.62
88.33
43.37
88.07
25.11
94.50
79.06
74.10
36.88
90.88
30.66
93.96
31.59
91.58
40.66
89.00
Large-scale 0OD Detection. We conduct experiments on the ImageNet benchmark, demonstrating
the scalability of our method. Specifically, we inherit the setup from 440 46 where the ID dataset
is ImageNet-100 (9), and OOD datasets include iNaturalist [65]. SUN 71
Places365 [83]. and

---

Textures 171. Following the setting in the previous works 40 46), we fine-tune the last residual block of ResNet-50 (0 pre-trained on ImageNet- 1K [9] for 20 epochs with the learning rate 10-3 while freezing the rest of the parameters. At inference time, all images are resized to x24. In Figure] | we reported the performances of four OOD test datasets respectively. It can be seen that our method reaches mtateoof-the-ae results on average across four OOD datasets.
50
98
KNN+
PALM
CIDER
DRL
FPR95 (%)
AUROC (%)
88
0
86
Places
Textures
NNaturalist
Textures
INatoralist
Average
Figure OOD detection results on the ImageNet-100 benchmark with ResNet-50.
5.3
Ablation study
In this section, we conduct an ablation study to validate our motivation and design. In Figure 2a we compare the effect between fz-noraalzied and unnormalized class prototypes. While fz-normalzied class prototypes enable DRL to avoid estimating the normalization constant, can observe that unnormalized class prototypes come with a significant performance increase detecting OOD samples. We suspect that unnormalized class prototypes come with all classes are allowed to have a different level of concentration from each other. Besides, recalling that we have rearranged sequential sampling to optimize the ELBO of DRL in consistent manner, Figure [2b empirically examines the effectiveness this practice by demonstrating that the OOD detection performance considerably drops without the rearranging of sequential sampling.
reliable assumption that
50
IregeMet-TER
wwageffer--Ve 180 20)
0.2
Ialtarallett I000)
8.8
Nuturallet: (000)
EPR95 (%6)
40
Dereity
30
0.1
9.4
20
10
4.0
Places
SUN
iNaturalist
Textures
(b)
Figure 2: Ablation Study on the proposed DRL: (a) distribution of feature embeddings learned with unnormalized (left) and fz-normalzied (right) class prototypes; (b) the benefits of the proposed sequential sampling rearrange to OOD detection performance.
Conclusion
In this paper, we propose a novel learning framework DRL that mitigates the gap between pre-trained discriminative models and density-based post-hoc OOD detection. Methodologically, DRL focuses on answering an important yet under-exx under-explored question of whether it is possible to deterministically shape the ID feature distribution while pre- rre-training diseriminative model. Theoretically. DRL is formulated as an Expectation- Expectation-Maxxiizzatiii Maximization algorithm, where we design bounded ELBO and rearrange the sequential sampling for consistent optimization. Empirically, DRL achieves consistently strong performance of OOD detection compared to competitive baselines on multiple benchmarks, which implies the superiority of our proposed DRL. We hope our work can inspire future research on shaping ID feature space for density-based post-hoc OOD detection.

---

7 Acknowledgement
This work was supported in part by the NSFC/ Research Grants Council (RGC) Joint Research Scheme under the grant N_ HKBU214/21, the General Research Fund of RGC under the grants 12201321, 12202622, 12201323, and the RGC Senior Research Fellow Scheme under the grant: SRFS2332-2502.
References
[1] Yong Hyun Ahn, Syyonn-Mooo Park, and Seong Tae Kim. Line: cuttt--distribution detection by leveraging important neurons. In CVPR, 2023,
[2] Mouin Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi. Neco: Ncural collapse based out-of-distribution detection. In ICLR, 2024.
[3] Christopher M Bishop. Pattern recognition and machine learning. Springer Google Schola, 2:645-678,
2006.
[4] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifuable sets, volume 4.
Springer, 2019.
[5] Ting Chen, Simon Kornblith. Mohammad Norouzi. and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020.
[6] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[7] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing
textures in wild. In CVPR, 2014.
[8] Henry Cohn and Abhinav Kumar. Universally optimal distribution of points on spheres. Journal the American Mathematical Society, 20x1):99-148. 2007.
[9] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenct: large-scale hicrarchical image database. In CVPR, 2009,
[10] Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophohia. NeurlPS, 2018
[11] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for nut-of-distribution detection. In ICLR, 2023.
[12] Xuefeng Du, Zhen Fang. Ilias Diakonikolas, and Yixuan Li. How does wild data provably help ood detection? In ICLR, 2024.
[13] Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li, Siren: Shaping representations for detecting out-of-distribbuton objects. In NeurlPS, 2022,
[14] Xuefeng Du Zhaoning Wang, Mu Cai, and Sharon Li. Towards unknown-aware learning with virtual outlier synthesis. In ICLR, 2022.
[15] Zhen Fang. Yixuan Li, Feng Liu, Bo Han, and Jie Lu. On the learnability of out- of--iitribttion detection. Journal of Machine Learning Research, 25, 2024.
[16] Zhen Fang. Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurlPS. 2022.
[17] Soumya Suvra Ghosal, Yiyou Sun, and Yixuan Li. How to overcome curse-of- diw.ooiiiiiiiiiiitiiiyyy.. for out-of- distribution detection? In AAAI, 2024.
[18] Will Grathwohl, Kuan-Chiel Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat like one. In ICLR, 2020.
[19] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016.
[21] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling nut-of-distribution detection for callworldd settings. In ICML, 2022.
[22] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples
in neural networks. In ICLR, 2022.
[23] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure.
In ICLR, 2019.
[24] Norbert Henze and Bemd Zirkler. A class of invariant consistent tests for multivariate normality. Commu- nications in statistics-Theory and Methods, 19(10):355-3617, 1990.
[25] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS. pages 677-689. 2021.
[26] Xiaowei Huang, Daniel Krocning, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu,
and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100270, 2020.
[27] Julian Katz- Samuels. Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In ICML, 2022.
10

---

[28]
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot,
Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
[29]
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being baycsian, even just bit, fixes overeonfi-
dence in relu networks. In ICMI., 2020.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Hao Lang, Yinhe Zheng. Yixuan Li, Jian Sun, Fei Huang, and Yongbin Li. A survey on out-of-distriiution detection in nip. Transactions on Machine Learning Research, 2023.
[32] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based
learning. Predicting Structured Data, 1(0), 2006.
[33] Kimin I.c, Honglak Lee, Kihok Lce, and Jinwoo Shin. Training confidence-calibratee classifiers for detecting out-of-distribution samples. In ICI.R, 2018.
[34] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for delecting out-of-distribution samples and adversarial attacks. In NewrlPS, 2018.
[35] Shiyu Liang. Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
[36] Litian Liu and Yao Qin. Detecting out-of-distribtioo through the lens of neural collapse. In ICLR, 2024, Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -ittr--distrbbtiinn detection. In NewrIPS, 2020.
[37]
[38] Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-based out-of- distribution detection. In CVPR, 2023.
[39] Daniel W Lozier. Nist digital library of mathematical functions. Annals of Mathematics and Artificial Intelligence, 38: 109 2003.
[40] Haodong Lu, Dong Gong, Shuo Wang. Jason Xue, Lina Yao, and Kristen Moore. Learning with mixture of prototypes for out-of-distribution detection. In ICLR, 2024.
[41] Jie Lu, Junyu Xuan, Guangquan Zhang, and Xiangfeng Luo. Structural property-aware multilayer network embedding for latent factor analysis. Pattern Recognition, 76:228-241, 2018.
[42] Jie Lu, Hua Zuo, and Guangquan Zhang. Fuzzy multiple-souree transfer learning. IEEE Transactions on Fuzzy Systems, 28(12):3418 333333333331 2019.
[43] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In NeurIPS, 2018. Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. In NewrIPS, 2019.
[44]
[45] Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. JEFE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513- 1-31133-33332252 2022.
[46] Yifci Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for nut-of-distributio detection? In ICLR, 2023.
[47] Todd K Moon. The anpectationmmaximizatiie algorithm. IEEE Signal Processing Magazine, 13(6):77-00,
1996.
[48] Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In AAAI, 2022.
[49] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[50] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are casily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
[51] Andre T Nguyen, Fred Lu, Gary Lopez Munoz, Edward Raff, Charles Nicholas, and James Holt. Out of distribution data detection using dropout bayesian neural networks. In AAAI, volume 36, pages 7877-7885 2022.
[52] Jun Nie. Yadan Luo, Shanshan Ye. Yonggang Zhang, Xinmei Tian, and Zhen Fang. Out-of-distribtiion detection with virtual outlier smoothing. International Journal of Computer Vision, pages 1-18, 2024.
[53] Jun Nie, Yonggang Zhang. Zhen Fang. Tongliang Liu, Bo Han, and Xinmei Tian. -fitt--diittiittio detection with negative prompts. In ICLR, 2024.
[54] Bo Peng, Yadan I Luo, Yonggang Zhang. Yixuan i., and Zhen Fang, Conjnorm: Tractable density estimation for rut-of-distribttio detection. In ICLR, 2024.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffussonn models. In CVPR, 2022.
[56] Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of- of-tiitiibttion detection: Solutions and future challenges. Transactions on Machine Learning Research, 2022.
[57]
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices.
In ICML. 2020.
[58]
Vikash Sehwag, Mung Chiang, and Prateek Mittll, Ssd: A unified framework for self-supervised outlier detection. In ICLR, 2021.
[59]
Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection.
In NeruPPS, 2022.
[60] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribtiion detection with rectified activations. In
NewrIPS, 2021.
[61] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, 2022.
11

---

[62] Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
[63]
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. In NeurlPS, 2020.
[64]
Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image
clustering. ICLR. 2020.
[65]
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie, The inaturalist species classification and detection dataset. CVPR, 2018. Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In CVPR, 2022.
[66]
[67]
Haoran Wang. Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? In NeurlPS, 2021.
[68]
Qizhou Wang. Bo Han, Tongliang Liu, Gang Niu, Jian Yang. and Chen Gong. Tackling instance-dependent label noise via a universal probabilistic model. In AAAI, 2021.
[69]
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML. 2020.
[70] Jim Winkens. Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia Mac MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, al. Contrastive training for improved out-of- -ff---iiiiiittionn detection. In arXiv, 2020.
[71] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, 2010.
[72] Mingyu Xu, Zheng Lian, Bin Liu, and Jianhua Tao. Vra: Variational rectified activation for out-of- distribution detection. In NerulPS, 2023.
[73] Pingmei Xu, Krista A Ehinger. Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao.
Turkergaze: Crowdsourcing saliency with webcam based eye tracking. In arXiv, 2015.
[74] Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. International Journal of Computer Vision. pages 1-28, 2024.
[75] Fisher Yu. Ari Seff. Yinda Zhang. Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construc- tion of large-scale image dataset using deep learning with humans in the loop. In arXiv, 2015.
[76] Yeonguk Yu, Sungho Shin, Seongju Lee, Changhyun Jun, and Kyoobin Lee. Block selection method for using feature norm in out-of-distribution detection. In CVPR, 2023.
[77] Jinsong Zhang. Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang. Shi Han, Dongmei Zhang, et al. -uitof-diitributtin detection based on in-distribution data patterns memorization with modern hopfield energy, In ICLR, 2022.
[78]
Zihan Zhang and Xiang Xiang. Decoupling maxlogit for
detection. In CVPR, 2023, Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. Towards optimal feature-shaping methods for out-of-distriiuttoo detection. In ICLR, 2024.
[79]
[80]
Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng. and Stephen Gould, Towards optimal feature-shaping methods for out-of-distribation detection. In JCLR, 2024.
[81] Zhong Qiu Zhao, Peng Zheng. Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review. IEEF Transactions on Neural Networks and Learning Systems, 20(11):311233222 2019. [82] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence.
40(6):1452 1452-1464, 2017.
[83] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence,
4056)1145211464 2017.
[84] Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang. Hui Xue, Xiang Tian, Yaowu Chen,
ct al. Boosting ofiit--diittibutiion detection with typical features. In NeruPPS, 2022.
[85] David Zimmerer, Peter M Full, Fabian Isensee, Paul Jäger, Tim Adler, Jens Petersen, Gregor Köhler, Tobias Ross, Annika Reinke, Antanas Kascenas, el al. Mood 2020: A public benchmark for out-of-distribution detection and localization on medical images. IEEE Transactions on Medical Imaging, 4111073772--2738 2022.
12

---

A Approximating vMF normalization constant
To approach the vMF probability density function for the random d-dimensional unit vector and concentration parameter c, one needs to get the normalization constant by calculating /2/2-1/ (2/)4/2a/2211()) where (-)-) denotes the modified Bessel function of the first kind and order p. However, (p(:) complicated function. To cast off this dilemma, as explained by Eq. (9) in the main paper, we propose approximate (p(-) by resorting to an asymptotic expansion. We compare the values of log (-) obtained by our proposed approximation with the values of log (+(+) obtained by the scipy implementation where define approximation error d(e) follows:
δ(e) = log [p(e) log Ip(e)
(16)
can be seen from Figure 3 that the approximation error is small compared to the actual function
values when p 127. This implies that the latent feature dimension d 256 sufficiently large to get a good approximation of the normalization constant.
200.0
0.0
200.0
127
0.0
40000
127
0.002
4000
0.0
50.0
100.0
150.0
200.0
250.0
0.0
50.0
100.0
150.0
200.0
250.0
Figure 3: Left: The actual value of our proposed approximation of log (c): Right: The approxima- tion error 8(c) in computing (-)-) between our proposed and the scipy implementation.
B Training stability on CIFAR-100
To verify that our method consistently provides strong performance, train with independent seeds for CIFAR-100 and report the average and standard deviation of FPR and AUROC in Table
Table Ablation on stability. OOD detection performance of our method on CIFAR- 100. Results averaged over 5 independent runs.
Method
SVHN
Places365
LSUN
iSUN
Texture
FPR95
AUROCT
FPR95
AUROC1
FPRO5
AUROC1
FPR95L
AUROC
FPR95L
AUROCI
Mean
10.15
98.07
66.64
81.55
6.97
98.63
22.57
96.33
21.97
96.09
Std.
1.52
0.11
3.06
2.41
2.28
1.46
3.25
0.87
2.24
1.94
C
Limitations
This paper only explores one type of realization of the non-negative density function. It will be exciting to explore more possibilities for the realization and parameterization.
D Broader impacts
Our project aims to improve the reliability and safety of modern machine learning models. Our study can lead to direct benefits and societal impacts, particularly for safety-critical applications such as autonomous driving. Our study does not involve any human subjects or violation of legal compliance. We do not anticipate any potentially harmful consequences to raise stronger research and socictal awareness towards the problem out-tf-distrrbuttinn detection in real-world settings.
work. Through study, hope
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly illustrate the focused problems and state our contribution method- ologically, theoretically and empirically in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our method in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
the paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide implementation details of our method in the Experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We will release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwticccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implementation details of our method the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided the standard deviation of our method in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the number and type of used GPU in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///////ip..eeeee http://urrripcccppppeee http:///.cc.o.l.ttttttcccccueee.
Answer: [Yes]
Justification: I have read the ethics review guidelines before conducting research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the societal impact of our method in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not involve the use of any pretrained language models, image generators, or scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we have cited the original paper that produced the code package or dataset and the used datasets in this paper are properly licensed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi www.swtthhhce...c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
18

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The datasets and models used in this paper are open-source.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The datasets are not with crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The datasets are not with crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "jgicgYbdgw", "reviewer": "Reviewer_s5Ce", "summary": "The paper introduces an innovative learning framework, Distributional Representation Learning (DRL), designed to bridge the gap between network pre-training and density-based scoring strategies. DRL is formulated as a provably convergent Expectation-Maximization algorithm. Key contributions include the introduction of unnormalized class prototypes, which enhance the flexibility of the mixture model, and an online approximation of normalization constants, enabling end-to-end training. This framework represents a significant advancement in the field, offering both theoretical and practical improvements over existing methods.", "questions": "- The paper initially criticizes existing methods for imposing strict distributional assumptions due to limited prior knowledge of the learned feature space. However, it then introduces a pre-defined mixture distribution for the underlying feature space. Can you clarify how this approach aligns with the initial criticism and motivation of your method?\n- How does DRL's performance compare with other density-based OOD detection methods? Can you provide competitive performance metrics and analysis to evaluate DRL's relative effectiveness?\n- What is the impact of the total number of classification classes (k) on computation and memory requirements? A detailed analysis would help in understanding the scalability and resource demands of your model.", "limitations": "- The paper primarily explores a single realization of the non-negative density function and imposes a pre-defined mixture distribution on the underlying feature space, which seems to contradict its initial motivation.Further exploration of different realizations could provide a more robust evaluation of the proposed approach.\n\n- The paper includes experiments for large-scale OOD detection primarily on the ImageNet-100 dataset. If extended to the larger ImageNet-1k dataset, there are concerns that inference could become significantly slower and memory usage substantially higher. Addressing these potential scalability issues would be important for practical applications on large-scale datasets.\n\n- There are various writing and expression errors throughout the paper. Improving the clarity and precision of the writing would enhance the overall readability and comprehension of the work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The theoretical framework presented formulates DRL as an Expectation-Maximization (EM) algorithm, offering a robust foundation for the learning process. Recognizing the difficulty in directly optimizing the log-likelihood function, the authors adeptly shift focus to the optimization of $\\text{ELBO}(q, x; \\theta)$. To address potential inconsistency issues in this optimization, they introduce a sequential sampling rearrangement technique, significantly enhancing OOD detection performance. Additionally, One of the key strengths of the paper lies in its innovative approach to handling normalization constants. Rather than imposing impractical constraints to make these constants\ninput-independent or known, the authors propose an online approximation method, enabling seamless end-to-end training. The approach substantially enhances out-of-distribution (OOD) detection performance on CIFAR-10 and CIFAR-100 datasets.", "weaknesses": "- **Contradiction in Assumptions:** While the paper notes that existing methods often impose strict distributional assumptions due to the lack of prior knowledge about the learned feature space, it still enforces the underlying feature space to conform to a pre-defined mixture distribution. This approach appears to contradict its initial motivation.\n- **Influence of Total Number of Classes (k):** The paper does not address the impact of the total number of classes (k) on computation and memory requirements. Including a detailed analysis of this aspect would provide a clearer understanding of the model's scalability and resource demands.\n- **Performance Comparison with Other Density-Based Methods:** The paper lacks a clear comparison of DRL's performance with other density-based OOD detection methods.\n- **Theoretical Proof for Sequential Sampling:** The paper lacks sufficient theoretical proof for the rearrangement of sequential sampling to ensure consistent optimization. Providing a rigorous theoretical foundation would strengthen the claims made in this regard.\n- **Discussion on Hyperparameter $\\beta$:** The paper does not include a discussion on the hyperparameter $\\beta$. Including a comprehensive analysis of this hyperparameter, its impact on the model, and guidelines for its selection would be beneficial.\n- **Equation Reference Correction:** In line 120, the reference to equation (7) should be corrected to equation (5).\n- **Equation Mistakes:** In lines 176 and 183, there are errors in the equations that need to be addressed and corrected for accuracy.\n- **Lack of Description for Table 5:** In line 288, under the section \"Unsupervised OoD Detection,\" there is an insufficient description of Table 5. A more detailed explanation is required to clarify the contents and significance of the table.\n- **Symbol Description:** In line 138, there is no clear description of the symbols in the formula. Providing detailed definitions and explanations of all parameters involved is essential for comprehensibility and reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing. We have provided detailed responses below, hoping your concerns can be adequately addressed.\n\n**Q.1.** Contradiction in Assumptions\n\n**A.1.** We apologize for the misunderstanding and will provide explicit explanations highlighting that our method aligns with the motivation. \n- As shown in section 2, existing feature-based and logit-based methods explicitly or implicitly assume $p_{\\theta}(x)$ as the Gibbs-Boltzmann distribution and Gaussian mixture distribution, respectively. However, there is no theoretical guarantee to ensure these distributional assumptions hold under any discriminative models $f_{\\theta}$ that are trained by minimizing the classification objective. This inconsistency would degrade OOD detection performance.\n- Given that $p_{\\theta}(x)$ is typically unknown after a discriminative model $f_{\\theta}$ is trained by minimizing the cross entropy, our method proposes to deterministically shape $p_{\\theta}(x)$ to conform to the known distribution defined via Eq. (7). This avoids the above unalignment issue. Namely, this leads to our motivation: is it possible to deterministically shape the ID feature distribution while pre-training a discriminative model? In this way, the defined distribution can be safely used for OOD detection since the involved distributional assumption can hold after training where Eq. (10) is minimized.\n\n**Q.2.** Influence of class number (K)\n\n**A.2.** Thanks for your kind suggestions. Our method's memory and time complexity linearly scales with K so that our method has the same scalability nature as the CE-based training. \n\n- The parameters involved in our method are located in 1) backbone and 2) prototypes $\\mu\\_1,...,\\mu\\_K$. Since the backbone is orthogonal to our technical design, we omit the memory complexity of the backbone. Therefore, the memory complexity of our method is $O(K)$. \n- Computing Eq. (14) for each x requires to compute $p\\_{\\theta}(x|k)$ for each $k$. By omitting the time complexity of the backbone and dot product, the time complexity of our method is $O(K)$. \n- We conduct ImageNet-100 for large-scale OOD, aiming to keep consistent with prior works [a,b].\n\n[a] Learning with Mixture of Prototypes for Out-of-Distribution Detection\n\n[b] How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection\n\n**Q.3.** Comparison with Density-Based Methods\n\n**A.3.** While this paper has considered popular post-hoc density-based methods, including Energy, Maha, and CONJ, for comparison, as per your valuable advice, we compare to more baselines (Maxlogit [c] and GEM [d]) and report the averaged results across 5 OOD datasets used in this paper as follows: \n|| FPR95 | AUROC|\n| :-| :-|:-| \n| [d]|47.38|91.03|\n| [c] |33.55|90.97|\n|Ours|11.58|97.83|\n\nWe compare it to the training-based density-based method [e], where an extra flow model is introduced on top of the pre-trained model. Here, following [e], we evaluate our method on CIFAR-10 and report the averaged results across 6 OOD datasets (5 used in this paper+LSUN-Resize)\n| | FPR95 | AUROC|\n| :-| :- | :-| \n| [e]|16.26|97.19|\n| Ours|13.19| 97.46|\n\n[c] Scaling Out-of-Distribution Detection for Real-World Settings\n\n[d] Provable Guarantees for Understanding Out-of-distribution Detection\n\n[e] FlowCon: Out-of-Distribution Detection using Flow-Based Contrastive Learning\n\n**Q.4.** Justification of Sequential Sampling\n\n**A.4.**  We will add the following explanations.\n+ The proposed DRL is formulated into a provably convergent EM framework. The inconsistency issue that occurs in the batch-based training scheme originates from the nature of the EM framework rather than our technical design: during batch-based training scheme,  $p_{\\theta}(k|x)$ is only updated once per epoch while $f_{\\theta}$ keeps updated throughout batches (iterations). The EM framework is widely used in deep learning literature  [f,g,h,i,j,k,l,m], where inconsistency is an open problem.\n+  Our key idea is to update the model in the current iteration and take data from the upcoming iterations into consideration. This motivates sequential sampling to have $B\\_m=(B\\_m^{pre},B\\_m^{next})$, where $B^{next}\\_{m-1} = B^{pre}\\_{m}$. In this way, by optimizing classification objective over $B\\_m^{next}$ encourage $p\\_{\\theta_{m}}(k|x)$ to be close to the ground truth, the estimated $q\\_{m+1}(k|x) = p\\_{\\theta_{m}}(k|x)$ can be reliable for $\\mathcal{B}\\_m^{next}$.\n \n[f] Knowledge Condensation Distillation\n\n[g] MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering\n\n[h] Prototypical Contrastive Learning of Unsupervised Representations\n\n[i] Joint Unsupervised Learning of Deep Representations and Image Clusters\n\n[j] Deep Clustering for Unsupervised Learning of Visual Features\n\n[k] Learning representation for clustering via prototype scattering and positive sampling\n\n[l] Stable Cluster Discrimination for Deep Clustering\n\n[m] Unsupervised Visual Representation Learning by Online Constrained K-Means\n\n**Q.4.** Ablation on $\\beta$\n\n**A.4.** As per your advice, we conduct the ablation study on $\\beta$ (with CIAR-10 as ID dataset) and show in the following:\n|$\\beta$ |0.08|0.1|0.2|0.4|\n| :-| :-| :-| :-| :-|\n|Averaged FPR95|12.19|11.85|11.58|12.64|\n\n**Q.5** Description for Table 5\n\n**A.5.** In Table 5, we extend our method to the unsupervised setting. Following DINO [n], we keep a momentum teacher to produce soft labels as the target. As with DINO, we use the centering operation to avoid collapse.\n\n[n] Emerging properties in self-supervised vision transformers\n\n**Q.6** Equation Mistakes\n\n**A.6.** Thanks for your comments. The revised paper has fixed the typos and mistakes you mentioned.\n\n**Q.7** Symbol Description in line 138\n\n**A.7.** Thanks for pointing out the typo. We have fixed this in our revised paper and show the correct definition of $\\Phi(k)$ in Eq. (7)\n$\\Phi(k) = \\int_{\\mathbf{z} \\in \\mathcal{Z}} h(\\mathbf{z}, k) d\\mathbf{z}$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5BY0QoaqoF", "reviewer": "Reviewer_7foW", "summary": "The paper proposes an in-distribution (ID) modeling approach, termed distributional representation learning (DRL), which enhances the convergence of ID latent feature learning. The authors include theoretical proof to corroborate the proposed approach. They have conducted a few experiments on standard OOD benchmarks and explored different OOD cases.", "questions": "1. To perform online approximation, I would like to know if the batch size is an important parameter.\n2. The description of $g_\\theta$ in equation 7 (line 138) seems missing.\n3. In Table 4, the proposed methods can hardly be comparable to the PALM in the ImageNet-Resize OOD detection results. Are there any reasons?\n4. In Table 5, unsupervised settings seem to yield more failure cases, such as SVHN and LSUN datasets for the proposed method compared to PALM. Can the authors explain a bit about the results?\n5. Why the authors did not include CIDER in Table 5?", "limitations": "The scope of OOD detection is widespread but the authors only consider simple and standard datasets. The experiments might introduce limitations to the study.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The theoretical contribution enhances the idea of representation learning in the latent space. This theoretical proof helps explain related previous works that have proposed several ID modeling techniques. \n2. The proposed method includes practical implementation techniques in the methods corresponding to the proposed theory.\n3. The presentation is generally clear and easy-to-understand to readers.", "weaknesses": "1. Although the proposed data distribution modeling method is claimed to mitigate the distribution assumptions in the previous works, the model convergence still relies on prior works' assumptions such as vMF. Even though the techniques with underlying assumptions do not directly influence the geometry of the latent space, the proposed method is still not viable without these techniques. Other optimization strategies without the assumptions might be worth considering. \n2. The improvements in the standard benchmarks are not significant and convincing. A few previous works' results are not consistent with the number in the papers. For example, using CIDER on the LSUN dataset obtains 30.24 FPR while 16.16 FPR was reported in the original paper for the CIFAR-100 ID data experiment.  The average FPR for Imagenet-100 is reported to be 25.9 in the original paper and GitHub repository which is much lower than the proposed method with 30.31 FPR in Table 3.\n3. The ablation study and analysis of the proposed method can be improved. Figure 1(a) compares the effects of l2-normalization which has been widely known in most previous feature-based OOD detection methods. For Figure 1(b) observing the difference of sequence sampling, additional convergence analysis should be provided, such as loss trajectory or convergence time.\n4. As described in line 189, the ELBO convergence issue might be a concern. Without sufficient evidence, it would be hard to justify the proposed method can completely avoid this kind of issue. As the proposed method is still a sampling strategy, the uncertainty of occasional unconverge situation might still occur.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our work. Below are detailed responses to your valuable comments.\n\n**Q.1.** Although the proposed method is claimed to mitigate the assumptions in previous works, the model convergence still relies on prior works' assumptions (vMF). Even though the assumptions do not influence the geometry of the latent space, the proposed method is still not viable without these techniques.\n\n**A.1.** We apologize for the misunderstanding. We will add the following explanation to highlight our motivation.\n\n- Previous work assumes $p_{\\theta}(x)$ (denoted as p_x in the left) to be the vMF distribution without theoretical guarantee. The mismatch between the assumed and learned distributions will degrade detection performance.\n\n-  Our method, with the derived ELBO (Eq. 10), enjoys the theoretical guarantee that the learned p_x conforms to vMF distribution. Here, introducing vMF is to compute the normalization constant in a closed form.\n\n- Our method can shape the learned p_x to fit various distributions, i.e., it is viable without the help of vMF. For example, if we, following [a], define $h(z,k)$ to be a Bregman divergence [b], the resulting p_x turns to be an exponential family distribution, which is left as our future work.\n\n[a] ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection\n\n[b] Clustering with Bregman Divergences\n\n**Q.2.** The improvements in the standard benchmarks are not significant and convincing.\n\n**A.2.** We kindly note that the reported results of most compared baselines in all tables of this paper originate from PALM [c]. We are sorry for not explicitly illustrating this to make the reviewer confused. \n\n- While the reimplemented CIDER in PALM performs worse than the original CIDER on LSUN with CIFAR-100 as ID data, it is advantageous for the former to have better-averaged results across 5 OOD datasets than the latter. This is because OOD data tends to be from various domains rather than a specific one. \n\nFollowing PLAM, the experiments on ImageNet-100 are conducted on **ResNet-50**. However, the CIDER results reported on the GitHub repository are based on **ResNet-34** and thus can not be used for comparison. Thus, we **never** undergrad existing works.\n\n[c] Learning with Mixture of Prototypes for Out-of-Distribution Detection\n\n**Q.3.** The ablation study and analysis can be improved. Additional convergence analysis should be provided.\n\n**A.3.** Thanks for your constructive advice. We have added the loss trajectory to the revision, as shown in the PDF file.\n\n**Q.4.** It would be hard to justify the proposed DRL can completely avoid this issue. As it is still a sampling strategy, the uncertainty of occasional unconverge situations might still occur\n\n**A.4.**  We will add the following explanations.\n\n+ The proposed DRL is formulated into a provably convergent EM framework. The inconsistency issue that occurs in the batch-based training scheme originates from the nature of the EM framework rather than our methodological design: during batch-based training scheme,  $p_{\\theta}(k|x)$ is only updated once per epoch while $f_{\\theta}$ keeps updated throughout batches (iterations). The EM framework has been widely used in deep learning literature [a,b,c,d,e,f,g,h], where inconsistency is an open problem.\n\n+  Our key idea is to update the model in the current iteration and take data from the upcoming iterations into consideration. This motivates sequential sampling to have $B\\_m=(B\\_m^{pre},B\\_m^{next})$, where $B^{next}\\_{m-1} = B^{pre}\\_{m}$. In this way, by optimizing classification objective over $B\\_m^{next}$ encourage $p\\_{\\theta_{m}}(k|x)$ to be close to the ground truth, the estimated $q\\_{m+1}(k|x) = p\\_{\\theta_{m}}(k|x)$ can be reliable for $\\mathcal{B}\\_m^{next}$.\n\n+  Our key contributions show that one can deterministically shape p_x to conform to the known distribution defined via Eq. (7) to avoid the unalignment between the learned and assumed distributions. We also admit that the inconsistency issue is a fundamental topic, but addressing it with a theoretical guarantee is still an open problem in the literature and out of the scope of our work.\n\n[a] Knowledge Condensation Distillation\n\n[b] MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering\n\n[c] Prototypical Contrastive Learning of Unsupervised Representations\n\n[d] Joint Unsupervised Learning of Deep Representations and Image Clusters\n\n[e] Deep Clustering for Unsupervised Learning of Visual Features\n\n[f] Learning representation for clustering via prototype scattering and positive sampling\n\n[g] Stable Cluster Discrimination for Deep Clustering\n\n[h] Unsupervised Visual Representation Learning by Online Constrained K-Means\n\n**Q.5.** To perform online approximation, I would like to know if the batch size is an important parameter\n\n**A.5.** According to the definition in Line 159, the online approximation in Eq. (9) does not leverage data statistics within batches, i.e., it is independent of the batch size.\n\n**Q.6.** The description in eq 7 (line 138) is missing\n\n**A.6.** Thanks for pointing out the problem caused by typos. We have fixed this mistake:\n$p_{\\theta}(x|k)=\\frac{h(z,k)}{\\Phi(k)},\\quad \\Phi(k) = {\\int_{z \\in \\mathcal{Z}} h(z, k) \\, d z}$, where $\\mathcal{Z}$ is the latent feature space.\n\n**Q.7.** In Tables 4 and 5, the proposed methods can hardly be comparable to PALM\n\n**A.7.** We suspect this is because PLAM introduces more than one prototype for each ID class to learn better features regarding discriminating the aforementioned OOD dataset from the ID dataset. We kindly note that our method achieves a better average performance than PALM.\n\n**Q.8.** Why is CIDER not in Table 5?\n\n**A.8.** Unlike PALM proposed for unsupervised and supervised settings, CIDER is proposed for supervised settings. Thus, it is unclear how to extend CIDER to the setting. Thus, we exclude CIDER from Table 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RDIUkmhabo", "reviewer": "Reviewer_5Lwp", "summary": "The paper provides a novel approach called DRL to help optimize post-hoc OOD detection methods by explicitly shaping the ID space during pre-training. In particular, DRL is defined through an Expectation-Maximization algorithm with alongside a structured mini-batch setting. The resulting DRL is shown to have strong empirical performance across multiple OOD detection methods and benchmarks. Additionally, the authors also provide a rigorous theoretical setup for justifying the DRL optimization schema.", "questions": "The reviewer has some confusion regarding the necessity of the sequential sampling alterations that the authors noted. In particular, the reviewer would like to get some more clarification regarding the inconsistency issues noted in Section 3.4.", "limitations": "The authors have adequately addressed limitations to the methodology as well as any negative societal impact. Additionally, the reviewer does not foresee any potential negative impact resulting from this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n- The paper provides a novel method for dealing with the problem of ID space shaping for OOD detection which is often overlooked in OOD detection.\n- DRL shows strong empirical performance across multiple high-resolution benchmarks and traditional CIFAR benchmarks.\n- The paper also provides a rigorous set of theoretical analyses to help better understand the underlying DRL method.", "weaknesses": "Weakness:\n- A small concern of the reviewer is the formatting and structure of the paper. In particular, given the dense nature of the work, the reviewer would highly encourage some additional context to help introduce the reviewer to the DRL method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your dedicated time and effort in reviewing our work. In response to the valuable comments provided, we have provided detailed responses to each point raised, hoping that our responses can adequately address your concerns. Please find our responses below.\n\n**Q.1.** More clarification regarding the inconsistency issues\n\n**A.1.** We kindly note that the inconsistency stems from that, under the batch-based training scheme, the model used in the E-step of the $m$-th iteration is trained without the data used in the M-step of the same iteration. In particular, the prediction $p_{\\boldsymbol{\\theta}}(k|\\mathbf{x})$ is only updated once per training epoch while the discriminative model $f_{\\boldsymbol{\\theta}}$ keeps updated throughout mini-batches (iterations) in the training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kdu6l1RYQN", "reviewer": "Reviewer_WmsK", "summary": "This paper looks to perform OOD detection via distributional representation learning rather than assume some pre-specified distributional form for the ID data. Their motivations are that there can exist inconsistencies between the assumptions on the ID distribution from prior work and the actual unknown ground truth distribution.\n\nThey consider learning a feature space to approximate a mixture distribution over the ID data. The authors propose an online approach to approximate the normalization constant over the mixture distribution.\n\nThe authors also formulate this approach via a provably convergent EM algorithm to improve training stability. Specifically, they use latent variables as in-distribution classes, using Bayesian inference to derive the posterior distribution over latents given observed data (E-step). Then, the authors maximize the ELBO (which they also prove is bounded, as during each iteration of the algorithm, this value increases).\n\nThe authors empirically demonstrate the improvement of this approach over alternatives on average across multiple datasets when training on CIFAR10 and CIFAR100. They further show strong performance in hard OOD detection settings, where the goal is to detect data from CIFAR10 (and other datasets) when training on CIFAR100.", "questions": "How quickly does this method converge? Does the EM algorithm require a comparable number of steps or computational complexity to the other considered approaches?", "limitations": "Yes, limitations are adequately addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors show strong empirical results in average, compared to the considered baselines. \n\nTheir approach also circumvents the need for pre-specified distributional forms for the ID data, which (as the authors note) can be an issue in practice.", "weaknesses": "No apparent weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's dedicated time and effort in evaluating our work. In response to the insightful comments provided, we have provided detailed responses to each point raised, hoping that our responses can adequately resolve your concerns. Please find our responses below.\n\n**Q.1.** How quickly does this method converge?\n\n**A.1.** Thanks for your insightful idea. As can be seen from the loss trajectory shown in the uploaded PDF file, training with the same epochs as prior works [a,b] is sufficient for our method to converge. We will highlight the nature of our method following your valuable comments.\n\n[a] Learning with Mixture of Prototypes for Out-of-Distribution Detection, ICLR 2024\n\n[b] How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection, ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We visualize our loss curve in the uploaded PDF file.

Meta-review: The paper introduces a new framework called Distributional Representation Learning (DRL) for OOD detection by explicitly enforcing an underlying pre-defined mixture distribution via expectation-maximization.

The reviewers appreciated the innovative approach to OOD detection via in-distribution modeling, the practicality of the proposed procedure along with its strong empirical performance, and the theoretical backing for the procedure. The main reviewer concerns were sufficiently addressed in the author response.
Decision: Accept (poster)