ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection
Authors: Bo Peng
Yadan Luo
Yonggang Zhang
Yixuan Li
Zhen Fang
Venue: ICLR 2024 poster (2024)
Keywords: Density-based OOD
Importance Sampling
Tractable density estimation
Primary area: societal considerations including fairness, safety, privacy

Abstract: Post-hoc out-of-distribution (OOD) detection has garnered intensive attention in reliable machine learning. Many efforts have been dedicated to deriving score functions based on logits, distances, or rigorous data distribution assumptions to identify low-scoring OOD samples. Nevertheless, these estimate scores may fail to accurately reflect the true data density or impose impractical constraints. To provide a unified perspective on density-based score design, we propose a novel theoretical framework grounded in Bregman divergence, which extends distribution considerations to encompass an exponential family of distributions. Leveraging the conjugation constraint revealed in our theorem, we introduce a \textsc{ConjNorm} method, reframing density function design as a search for the optimal norm coefficient $p$ against the given dataset. In light of the computational challenges of normalization, we devise an unbiased and analytically tractable estimator of the partition function using the Monte Carlo-based importance sampling technique. Extensive experiments across OOD detection benchmarks empirically demonstrate that our proposed \textsc{ConjNorm} has established a new state-of-the-art in a variety of OOD detection setups, outperforming the current best method by up to 13.25\% and 28.19\% (FPR95) on CIFAR-100 and ImageNet-1K, respectively.

Published as a conference paper at ICLR 2024
CONJNORM: TRACTABLE DENSITY ESTIMATION FOR OUT-OF-DISTRIBUTION DETECTION
Bo Peng'; Yadan Luo?; Yonggang Zhang², Yixuan Li", Zhen Fang" University of Technology Sydney, Australia The University of Queensland, Australia? Hong Kong Baptist University, Hong Kong University of AinboonsinMMaiioon USA*
no.peng-778a udstuentt uts edu. au, y y.luu.ug. edu. au
csygzhang comp. hkbu. edu. edu,hk, sharonl fcs wisc. edu zhen. fang@uts.. .erg@uuts.edu.au edu.au
ABSTRACT
Post-hoc out-of-distribution (OOD) detection has garnered intensive attention reliable machine learning. Many efforts have been dedicated to deriving score functions based on logits, distances, or rigorous data distribution assumptions to identify low-scoring OOD samples. Nevertheless, these estimate scores may fail to accurately reflect the true data density or impose impractical constraints. To provide unified perspective density-basee score design, propose novel theoretical framework grounded in Bregman divergence, which extends distribu- tion considerations to encompass an exponential family of distributions. Lever- aging the conjugation constraint revealed in our theorem, we introduce CON- JNORM method, reframing density function design as search for the optimal norm coefficient against the given dataset. In light of the computational chal- lenges of normalization, we devise an unbiased and analytically tractable estima- tor of the partition function using the Monte Carlo-based importance sampling technique. Extensive experiments across OOD detection benchmarks empirically demonstrate that proposed CONINORM has established a new state-of-the-ar in a variety of OOD detection setups, outperforming the current best method by up to 13.25% and 28. 19% (FPR95) on CIFAR- 100 and ImageNet- IK, respectively.
INTRODUCTION
Despite the significant progress in machine learning that has facilitated a broad spectrum of classi- fication tasks [Gaikwad et al. 2010: Huang al., 2014; Zhao et al.) 2019 Shantaiya et al.) 2013 Krizhevsky et al. 2012, Masana et al. (022). models often operate under closed-world scenario, where test data stems from the same distribution the training data. However, real-world applica- tions often entail scenarios which deployed models may encounter unseen classes of samples dur- ing training. giving rise to what is known as -itt---diittibution (OOD) data. These OOD instances have the potential to undermine a model's stability and, in certain cases, inflict severe damage upon its performance. To identify and safely remove these OOD data in decision-critical tasks (Chen) et al. 20222] Zimmerer et al.) 2022), OOD detection techniques have been proposed. To facilitate easy separation of in-distribution (ID) and OOD data, mainstream OOD approaches either lever-
age post-hoc analysis or model re- re-training (Ming et al.) 2022a Wei et al. 2022 Chen et al.
2021
[Huang& & Li [021]] Du et al., 2022] Katz-Samuels al.
[002 Wang et al. 2023 Lee et al.
2017
by using density-basse -based (Morteza & Li 2022), output-based (Liu et al. 2020). distance-based
[Lee
al. 2018) and reconstruction- based strategies (Zhou 2022).
Following previous works (Liu et al. 2020 Liang et al.] [017] Hendrycks & Gimpel| 2016 Sun
itt al. [202 Ahn et al., 22023] Djurisic et al. 2022 Lee et al. 2018), we focus on the post-hoc 00D
detection strategy, which offers more practical advantages than learning-based OOD approaches
"Equal Contribution
1 "Corrsponndence Zhen Fang fanen.fangee @ut..cdu.auu)

---

Published as a conference paper at ICLR 2024
without requiring resource-intensive re-training processes. Our key research question of this ap- proach centers on how to derive a proper scoring functions to indicate the ID-ness of the input for effectively discerning OOD samples during testing. By definition, OOD data inherently diverges from ID data by means of their data density distributions, rendering estimated density an ideal metric for discrimination. Nevertheless, it is non-trivial to parameterize the unknown II) data distribution for density estimation since the computation of normalization constants tends to be costly and even intractable (Gutmann & Hyvärinen 2012a). While recent attempts have been made by modeling ID data as some specific prior distributions, Le., the Gibbs -osltzmmnnn distribution in (Liu et al. 2020) and the mixture Gaussian distribution (Morteza & Li 2022), to factitiously make normalization constants sample-independent or known, this practice imposes strong distributional assumptions on the underlying feature space. Furthermore, it offers no theoretical guarantee that those pre-defined
distributions necessarily hold in practice.
In this paper, we introduce an innovative Bregman divergence- (Banerjee et al. 2005) theo-
retical framework aimed at providing a unified perspective for designing density functions within an expansive exponential family of distributions (Amari, 2016). This framework not only bridges the gap between existing post-hoc OOD approaches (Liu et al., 2020; Morteza & Li, 2022) but also highlights a valuable conjugation constraint for tailoring density functions to given datasets. Without loss of generality, we focus on the conjugate pair of I, and lg norms and propose the CON- INORM method. This approach reframes the density function design as a search for the optimal norm coefficient within a narrow range. To facilitate tractable estimation of the partition function for normalization, we compare two existing estimation baselines and put forward a Monte Carlo- based importance sampling technique, which yields an unbiased and analytically tractable estimator.
PRELIMINARIES
Let X' and y {1..... K} represent the input space and ID label space, respectively. The joint ID distribution, represented as Pxir. is a joint distribution defined over X' x V. During testing time, there are some unknown OOD joint distributions Dx defined over X y where ye is the complementary set of V. We also denote (x() as the density of the ID marginal distribution Px,- According to (Fang et al. 2022). OOD detection can be formally defined as follows:
Problem 1 (OOD Detection). Given labelled ID data Din {(x1, yI)...., (N), which is drawn from Px,r, independent and identically distributed, the aim of OOD detection is to learn a predictor y by using Din such that for any test data x: 1) if x is drawn from Dx, then g can classify X into correct ID classes, and 2) if x is drawn from Dxo: then can detect x as OOD data.
Post-hoc Detection Strategy. Many representative OOD detection methods (Liu et al. 2020 Liang Hendrycks & Gimpell 2016 Sun et al. 2021 Ahn et al. 2023 Djurisic et al. [202 Lee al. 2018) follow a post-hoe strategy, i.e., given a well-trained model to using Din. and scoring function S, then x is detected as ID data if and only if (x; fe) 2 X, for some given threshold A:
et al.
2017
g (x) = ID, if S(x; fo) > X; otherwise, g (x) OOD.
(I)
Following the representative work (Morteza & Li, 2022), a natural view for the motivation of the post-hoc strategy is to use level set for ID density pr (x) to discern ID and OOD data. Its main objective is to construct an efficient scoring function S, that can effectively replicate the behavior of the ID density function, Pi Pı(x), i.e., S(x; fe) x px()). Therefore, using the density-based frame- work, (Morteza& & Li. 2022) rewrites the post-hoe strategy as follows: given ID data density function (e()) estimated by well-trained model fo and a pre-defined threshold X, then for any data x E X.
g (x) = ID, (x 2 X; otherwise, g (x) = OOD.
In this work, we mainly utilize the density- framework to design our theory and algorithm.
Density Estimation Modeling. The performance of density-based OOD detection heavily relies on the alignment between the estimated data density (x(x) and the true density pr (x). Considering a commonly used assumption in OOD detection, i.e., the uniform class prior on ID classes (Jiang et al., 2023), pe x) can be expressed as the aggregate of the ID clss-conditioned distributions pe (x(k);
K
K
(x =
Σ po (x(k) Do (k) α
Σ Po (x k=
(3)
2

---

Published as a conference paper at ICLR 2024
main objective is to estimate the class distribution of ID data, in order to effectively construct the data density po for discriminating between ID and OOD data.
Based on Eq.
Without loss of generality, employ latent features extracted from deep models surrogate for the original high-ding high-dimensional raw data x. This is because z is deterministic within the post-hoc framework. Consistent with probabilistic theory, express (z
(z) It) in the following general form:
go (2,
po (z)k)
(4)
(b))
where ge(z. k) represents a non-nega non-negative density function, and (k) f go k)) dz denotes the partition function for normalization. According to prior works, the design principle for (z z. k) can be divided into categories: logit-bused, distance-based and density-bused methods.
Logit-basee based OOD methods (Li et al. 2020 Hendrycks et al. 2019) resort to derive (z k) from
logit outputs. a representative work, energy-based method (l et al. (020) explicitly acknowl- edges (z, k) by fitting the Gibbs Boltzmann distribution, i.e., go k) exp(fe /T') where fä is the kth coordinate fa and is a temperature parameter. This directly results an energy-based scoring function E(z) = log =k=1 go(z. z. k). However, it can be easily checked from Eq. 4 that Ezz)
log p(z) holds if and only if (k)) method has demonstrated empirical effectiveness. it is essential to recognize that this condition, i... ((k) constant, Ak E D, may not always hold in practical scenarios. Differently, Hendrycks & Gimpel 2016 proposes the maximum softmax score (MSP) to estimate OOD uncertainty:
constant, Vk E V. While the energy-based
go (2, k)
MSP(z)=
max Po(klz) max Jem m.m..,.,, K
* po (z).
(5)
=k=1 (0 k)
where (2, k) = exp(f%). However, as shown in Eq. 5 there exists misalignment between MSP and the true data density, making ultimately MSP suboptimal solution to OOD detection.
Distance-based OOD methods (Lee et al.
2017) target on deriving (z(z,N) k) by assessing the proximity of the input to the k-th prototype
GEM
Than Darsity
0.20
GEN
Thu Denaty
0.15
Hkk The selection of appropriate similarity met-
rics is crucial in capturing the intrinsic geomet- ric data relationships. One of the most repre- sentative metrics used is the maximum Maha- lanobis distance (Lee et al. 2017), which for- mally defined as,
0.10
0.10
0.05
0.05
0.00
0.00
10
10
Figure I: Illustration of the alignment of GEM score and true density of Gaussian (Left) and Gamma (Right) distributions.
Maha(z) (z) = max -(z us) Eー- pa)
k=
max log (o 1:) of po (z).
=
k=
The distance metric can be considered as the density function ge(2, in Eq. 4 This interpretation allows us to bypass the estimation of the partition function and leads to a significant observation: the distance measures are not directly proportional to the true data density.
Density-based OOD methods have rarely been studied compared to the previous two groups, pri- marily because of the complexities involved in estimating (k).. Recently, method called GEM (Morteza & Li (022) has been proposed, with the assumption that the class-conditional density conforms Gaussian distribution: let go k) = exp( { mx)) Σ- 1k))).
K GEM(Z)= Σ
exp( (z )x)
) T--1 1--1(z 12)))
K (z
K Σ k=1
Do (2)k) = (z)2)
+
K
A=1
where Σ E Rdxd is the covariance matrix. Note that o(k) / (2 in this case. This Gaus- sian assumption, while simplifying the estimation of (lk). enables the direct utilization of the Ma- halanobis distance go kz. However, it is crucial to acknowledge that this methodology may impose constraints on its ability to generalize effectively across a wide range testing scenarios due to the strict Gaussian assumption it relies upon.
3

---

Published as a conference paper at ICLR 2024
Discussion. Empirical examination of a toy dataset presented below reveals where the GEM's Gaussian assumption may prove inadequate, as demonstrated in Fig [ For the purpose of visual- ization, we begin by considering simple scenario in which the imput distribution is a mixture of two-dimensional Gaussians with means //2 = 2/11 = 8 and variances σ1 = 02 = respectively. While the GEM well align with the true data density function, the alignment of GEM scores with the true data density is noticeably compromised when the p(x) is changed to a mixture of Gamma and Gaussian distributions (as shown on the right). In order to ensure an accurate estimation of the ID class- condiconniiional density, two fundamental questions arise:
Can we develop a unified framework that guides the design of go k)?
Within this framework, how can we obtain a tractable estimate for (Ik) without presuming any particular prior distribution of po (z)k))²
In the following section, we propose a novel theoretical framework to answer the above questions.
METHODOLOGY
In this section, we first present the main Bregman Divergence-basee theoretical framework of OOD detection in Sec. 3.1 This framework unifies density function formulation and connects with prior OOD techniques, leveraging the expansive exponential distribution family. Motivated by theory, we introduce a novel approach called CORM to determine the desired ge through an exhaus- tive search for the best norm coefficient p. To enable tractable density estimation, we explore two partition function estimation baselines and propose our importance sampling in Sec. 3.2
3.1 BREGMAN DIVERGENCE-GUIDED DESIGN OF go(
In formulating our theoretical framework, is imperative adopt a universal distribution family to model the ID class-conditioned distributions po (x(k) without constraining ourselves to any par- ticular choice. In this work, we consider the broad Exponential Family of Distributions (Brown) 1986). The family encompasses wide range of probability distributions frequently employed in prior OOD investigations, such as Gaussian, precise, the exponential family of distribution can be formally defined follows:
Boobs-Bollzzann, and gamma distributions. To be
Definition 1 (Exponential Family of Distribution (Brown 1986)). A regular exponential family Po (z(lk is a family of probability distributions with density function with the parameters qs:
Po (z)k) = exp(z The - -(())))))))))))))))))))) - (2)),,
(6)
where (-) the so-called cumulant function and is a convex function of Legendre type.
By employing different cumulant functions v-()) and parameters mi one can create diverse class- conditioned distributions pe. (z)k). Nevertheless, it has been argued by AAzuury & Warmuth (2001); Chowdhury et al. (2023) that directly learning 77 to fit the ID data is computationally costly and
even intractable. To mitigate this challenge, a corresponding dual theorem (referred to as Theorem T has been developed. This theorem asserts that any regular exponential family distribution can be presented through a uniquely determined Bregman divergence (Bregman [967), as defined below:
Definition 2 (Bregman Divergence (Bregman 1967)). Let vp-)) be a differentiable, strictly convex function of the Legendre type, the Bregman divergence defined as:
dz [z. z') = ppz) - p(z') (z z'z T Vip(z').
where Vip(z') represents the gradient vector of (()) evaluated at z'.
The choices of the convex function op in Bregman divergence result in diverse distance metries. For instance, 1) When (zz) = || 2 the resulting de corresponds to the squared Euclidean distance; 2) When szx) is the negative entropy function, de represents the KL divergence; and 3) When (zz) be expressed as a quadratic form, de represents the Mahalanobis distance. Next, Theorem bridges the Bregman divergence and the exponential family of distributions.
Theorem (Forster& & Warmuth (2002)). Suppose that vb(-) and (p.)) are conjugate Legendre func- tions. Let po [z] be a member of the exponential family conditioned on the k-th ID class with cu- mulant function yp and parameters n (k- 1,..., K), dip be the Bregman divergence, then po (z)k)
4

---

Published as a conference paper at ICLR 2024
can be represented follows: po (z)kk = exp( d
(z. (..M(( (
9 (o(z)), where p(m))) is the is function uniquely deter-
pectation parameter corresponding to 77s (Banerjee et al. 2005), (.(.) mined by (p)), and agnostic pu(mss)
Remark, As a direct implication of Theorem unified theoretical principle emerges for the design of go (z, k) for OOD detection, owing to the conjugate relationship between db and y. In essence, when seeking an appropriate v for given dataset, the optimal design of go go(2, k)) should inherently adhere to the requirements of the corresponding Bregman divergence: let yo(+) +6 (+). then
gz k) exp(-dę (l):))))
Given that g. (z) is agnostic to the choice of exclude this term from consideration by treating it as a constant in our analysis, which gives us a systematic approach to answering the question &
CONJNORM. Given the expansive function space for the selection of the convex function ei our fo- cus is on simplifying the search process by utilizing the Ip norm as tù, denoted as CONINORM, where <m)))) = _ 17 li. Therefore, the task of selecting an appropriate & is equivalent to identifying a suitable p from the range of (1, +oo) for the given dataset. The Ip norm offers several advantagcous properties, including convexity and simplicity in its conjugate pair. Firstly, the Ip norm is convex for all 2 1. ensuring the presence of a global minimum during optimization. Secondly, the lp norm has well-defined and simple conjugate pair, namely the l norm, where g represents the conjugate exponent of p such that 1/p+ 1/4 1. This simplicity in the conjugate pair enhances computational tractability and facilitates the determination of yp "0"
P 1 | where q
(z)) (2)
(9)
p 1 To this end, the desired Bregman divergence dp can be determined as
2
dz(2, p(n))) = 2 || + 2 ( )I - (z, (
(10)
Hence, our final ID density can be estimated by combining Eq. 6 and Theorem
Po (z)
1 go(z,k) k)
Σ
1
((N))))
(11)
K
((k)
K
Te (z', ((n))))
km
In the context of our CONINORM framework, where we treat p as a hyperparameter, the process of searching for the optimal pore and identifying the most suitable density function de for a given dataset becomes straightforward. We present experimental results that explore the effects of varying p as illustrated in Fig.
3.2 ESTIMATION OF PARTITION FUNCTION ((-)
To an estimate of Po in Eq. 11 it is imperative to accurately approximate the partition function (k). The most straightforward approach to address this challenge involves fitting distinct kernel density functions, each corresponding to a different class. By employing this method, the density function go can be effectively normalized:
FollowingGGutmann Mnih & Kavukcuoglu (2013), we assume the pre-trained neural network is perfectly expressively such that the unnormalized density function de is self- eeff-normalized i.e., +(k) constant, vk E ) In this way, there is no need to explicitly compute the partition function (A)), which is given by
Baselines 1: Self-Normallzatiion (SN). Following Gutmann & Hyvärinen 2012b): et al. (018);
Po (z) Σ expl(dd(zz ((m)))
(12)
Baselines 2: Normalization via Kernel Density Estimation. Kernel density estimation (KDE) (Chen] 2017: Kim & Scout 2012) is a statistical method that is commonly used for probability density estimation. This approach is inherently non-parametric, providing flexibility in the choice of kernel functions (e.g., linear, Gaussian, exponential). Mathematically, the KDE-based estimation of partition function & (k) can be formulated as follows: let Dj be the ID training data with label k
(kDE(k)= h go (z, k)
1
Σ
KC( 90 k) go (zz, kb)
(13)
h

---

Published as a conference paper at ICLR 2024
80
Exponentia
Exponcndal
FPR95 (%)
NDE- UDE-Unear
KOE-Geuteann
KIDE KDE-Gaaaaan
KDE-Uiner
FPR85
20
INaturalist
Place
Texture
INaturaiist
Sun
Texture
OOD Dataset
OOD Dataset
Figure 2: Evaluations of different partition function estimation baselines on ImageNet: Left: Mo- bileNetV2 and Right: ResNet50.
with h 0 as the bandwidth that determines the smoothing of the resulting density function.
Experimental comparisons w.r.t. different baselines on the OOD detection benchmarks are summa- rized in Fig As we demonstrate later, we propose to leverage the means of importance sampling for theoretically unbiased estimation instead, which provides stronger flexibilityy and generality.
Ours: Importance Sampling- -Based Approximation. To enhance the flexibility OOD detection, we consider Monte Carlo method and construct simple and analytically tractable estimator to theoretically unbiasedly approximate them by means of importance sampling (IS) (Lu al. 20155] Tokdar & Kass 2010; Ben Alaya et al., 2023). Specifically, let Po (z) be tractable distribution that has been properly normalized such that Po (z) dz 1, we draw data
censity-based
= y2) (z": } mate (k)) by
from the ID training data following the distribution po (z), and esti-
1 71
go (0 k) Do (zz)
(is
(14)
For simplicity, we set Po uniform distribution the training ID data Din and N a with as the sampling ratio. In practice, we find that = 10% is sufficient to get decent performance. Besides, desirable property of importance sampling is that the estimator P (A; S) is theoretically unbiased (Liu et al. (2155). i.e., Es-p, [Pis (k; S)] = ((k). IS provides us with a simple but effective approach answering the question
EXPERIMENTS
4.1 EXPERIMENTS SETUP
Baseline Methods. We compare our method
Table I: OOD detection on CIFAR benchmarks. We average the results across OOD datasets. indicates larger values are better and vice versa.
with representative methods, including MSP
(Hendrycks & Gimpel 2016), ODIN Liang
ct al. 2017). Energy (Liu et al.] 2020
ASH
The best result in each column is shown in bold.
Djjurisic et al. (022), DICE (Sun & Li
2022
ReAct (Sun et al.) 2021), Mahalanobis
(Maha)
[ee al. 2018). GEM (Morteza & Li 2022).
Method
CIFAR-10
CIFAR-100
FPR95]
AUROCT
FPR95
AUROCT
KNN (un et al. 2022) and SHE (Zhang et al.
2022
It worth noting that have adopted the recommended configurations proposed by prior works, while concurrently standardizing the backbone architecture to ensure equitable comparisons.
MSP
48.73
92.46
80.13
74,36
ODIN
24.57
93.71
58.14
84.49
Energy
26.55
94.57
68.45
81.19
DICE
20.83
95.24
87.23
ReAct
26.45
94.67
62.27
84.47
ASH
15.05
96.61
41.40
90.02
Maha
31.42
89.15
55.37
82.75
Evaluation Metrics. The detection per- formance is evaluated via two threshold- independent metrics: the false positive rate of OOD data is measured when the true positive rate of ID data reaches 95% (FPR95); and the area under the receiver operating characteristic curve (AUROC) is computed to quantify the prob-
GEM
29.56
92.14
49.31
86.45
17.43
96.74
41.52
88.74
SHE
23.26
94.40
54.66
82.60
Ours
13.92
97.15
28.27
92.50
Our++ASH
12.14
97.64
25.66
93.21
ability of the ID case receiving higher score compared the OOD case. Reported performance

---

Published as a conference paper at ICLR 2024
Table OOD detection results on the ImageNet benchmark with MobilcNet-V2. 1 indicates larger values are better and vice versa. The best result in each column is shown in bold.
Method
INaturalist
SUN
Places
Textures
Average
FPR95]
AUROCT
FPR95L
AUROC
FPR95]
AUROCT
FPR95]
AUROC1
FPR95]
AUROCT
MSP
64,29
85.32
77.02
77.10
79.23
76.27
73.51
77.30
73.51
79.00
ODIN
55.39
87.62
54.07
85.88
57.36
84.71
49.96
85.03
54.20
85.81
Energy
59.50
88.91
62.65
84.50
69.37
81.19
58.05
85.03
84,91
ReAct
42.40
91.53
47.69
88.16
51.56
86.64
38.42
91.53
45.02
89.60
DICE
43.09
90.83
38.69
90.46
53.11
85.81
32,80
91.30
41.92
89.60
ASH
39.10
91.94
43.62
90.02
58.84
84.73
13.12
97.10
38.67
90.95
Maha
62.11
81.00
47.82
86.33
$2.09
83.63
92.38
33.06
63.60
71.01
GEM
65.77
45.53
87.45
$2.85
43.49
86.22
80.25
KNN
46.78
85.96
40.18
86.28
62,46
82.96
31.79
90.82
45.30
86.51
SHE
47.61
89.24
42.38
89.22
56.62
83.79
29.33
92.98
43,98
88.81
Ours
29.06
93.89
46.74
87.10
62.07
81.41
10,30
97.53
37.04
89.98
OuisAASH
24.08
94.36
30.19
92.63
46.26
87.57
12.70
97.20
28.31
92.94
results for our method are averaged over independent runs for robustness. Due to the space limit, provide the implementation details in the Appendix.
4.2 MAIN RESULTS
Evaluation on CIFAR Benchmarks. Following the setup in Sun & Li (2022), we consider CIFAR- 10 and CIFAR- (Krihevssyy et al.) 2009) as ID data and train DenseNet- 101 (Huang et al. 2017) on them respectively using the cross-entropy loss. The feature dimension of the penultimate layer 342. For both CIFAR-10 and CIFAR- 100, the model is trained for 100 epochs, with batch size 64, weight decay le-4, and Nesterov momentum 0.9. The start learning rate is 0. and decays by factor of 10 at 50th, 75th, and 90th epochs. There are six datasets for OOD detection with regard to CIFAR benchmarks: SVHN (Netzer et al., 2011). LSUN-Cro Crop YYu et al. 2015), ASUN-Resiz YYu al. 2015), iSUN (Xu et al. 2015)). Places (Zhou et al. 2017). and Textures (Cimoii et al. (014)). At test time, all images are of size 32x32. Table presented the performance of our approach and existing competitive baselines, where the proposed approach significantly outperforms existing methods. Specifically, comparing with the standard post-hoc methods, our method reveals 3.51% and 0.41% average improvements w.r.t. FPR95 and AUROC on the CIFAR-10 dataset, and 13.25% and 3.76% of the average improvements on the CIFAR- 100 dataset. For advanced works that consider post-hoc enhancement, e.g., ASH and DICE, method still significantly performs better on both datasets
Evaluation on ImageNet Benchmark. We conduct experiments on the ImageNet benchmark, demonstrating the scalability of our method. Specifically, we inherit the exact setup from (Djurisic et al. 2022), where the ID dataset ImageNet-1k (Krizhevsky et al.) 2012), and OOD datasets in- clude NNttrralist (Xiao et al.) 2010b), SUN (Xiao |l.] 2010a)). Places365 (Zhou al. 2017). and Textures (Cimpoi et al.) 2014). We use the pre-trained MobileNetV2 (Sandler et al. 2018)) models for ImageNet-1k provided by Pytorch (Paszke al., 2019). At test time, all images are resized to x24 224. In Table 2 we reported the performances of four OOD test datasets respectively. It can be seen that our method reaches tfut-oof-the-ar with 21.51% FPR95 and 95.48% AUROC on aver- age across four OOD datasets. Besides, we notice that ASH can further considerably improve our method by 9.27% and 2.06% w.r.t. FPR95 and AUROC respectively. We suspect that removing a large portion of activations at late layer helps to improve the representative ability of features.
4.3 ABLATION STUDY
Extracted Features z. This paper follows the convention in feature-based OOD detectors (Sun et al. 2022, Zhang et al. 2022 Djurisic et al. (022),, where features from the penultimate layer are utilized to estimate uncertain scores for OOD detection. Fig provides an experimenta evaluation on the choice of working placement. It can be seen that feature from the deeper layer contributes to better OOD detection performance than shallower ones. This is likely due to the penultimate layer preserves more information than shallower layers.
Sampling Ratio a. In Fig we analyze the effect of the sampling ratio a on CIFAR-100 and ImageNet- 1k datasets. We vary the random sampling ratio within (1%, 5%, 10%, 50%. 100%). We note several interesting observations: (1) The optimal OOD detection (measured by FPR95) remains similar under different random sampling ratios a especially when > 10%, which demon-

---

Published as a conference paper at ICLR 2024
1.SUN (000)
CIFA-11 II))
1000
CIFAR-10 10 (ID)
ISUN (OOD)
CIFAR-10 (ID)
LSUN (OOD)
Densky
Densey
800
Density.
.500
600
400
300
-25
20
15
-10
-5
25
15
-10
-5
-40 -30
-20
-10
(a)
(b)
(c)
Figure 3: Ablation study using feature extractions from (a) the first, (b) the second, and (c) the last
dense block of the DenseNet on the CIFAR-I0.
FPRe5 (%)
32
FPRSS (%)
FPR05 (%)
45
40
30
40
30
36
29
98
30
10
50
100
15
10
50
100
144 1.6 1.8 2.0 22 2.4 Norm Coeffidint
1.8 1.8 2.0 2.2 2 26 288 3.0 Norm Coefficient
Sampling Ratio (%)
Sampling Ratio (56)
(a) CIFAR 100
(b) ImageNet-IK
(c) CIFAR-I0
(d) CIFAR-10
Figure 4: Ablation study w.r.t varing sampling ratio in red; and the norm coefficient p in blue.
strates the robustness of our method to the sampling ratio. (2) our method still achieves competitive performance on benchmarks even when sampling 1% total number of training data.
Parameter Sensitivity of 1p- We conduct a comparative assessment of OOD performance while varying the ip norm coefficient p. which directly governs the density function De on CIFAR beneh- marks. The results, as depicted in Fig 4 reveal consistent trend across both datasets. Notably, the FPR95 scores exhibit clear minimum within the range of (2, 3), suggesting that our proposed CONINORM approach can efficiently identify the optimal normalization without the computational overhead. is worth highlighting that when p = 2, sigmifying the Bregman divergence (O, degenerates into the squared Euclidean distance (corresponding to Gaussian densities), the OOD performance does not attain its peak. This observation underscores the limitation Gaussian as- sumptions and underscores the generality and effectiveness of our CONINORM.
Sensitivity of with fixed ip norm. We also conduct a comparative assessment of OOD perfor- mance by varying the value of while fixing the I, norm. The experimental results CIFAR-100 under two cases where
2.5 and p 3.0. Note that the performance of our method tends to be more appealing when g satisfies the conjugate condition that q = p/(p 1). exceeding the case where 9 = 2.0 by nearly 10% on FPR95. This empirically echoes Theorem
4.4 EXTENSION TO MORE PROTOCOLS
In this section, we assess the versatility of the proposed CONINORM approach in (1) Hard OOD detection and (2) Long-tailed OOD settings. For more extensions, please refer to the Appendix.
4.4.1 HARD OOD DETECTION
We consider hard OOD scenarios Tack al. 2020), of which the OOD data semantically similar to that the ID cases. With the CIFAR 100 as the ID dataset for training ResNet-50. we evaluate our
Table Evaluation on hard OOD detection tasks. t indicates larger values are better and vice versa.
The best result in each column is shown in bold.
Method
LSUN-Fix
ImageNet-Fix
ImageNNe-RRsize
CIFAR-10
Average
FPR951
AUROC
FPR95
AUROC1
FPR95
AUROCA
FPR95]
AUROC1
FPR95
AUROC!
MSP
90.43
63.97
88.46
67.32
86.38
71.24
89.67
66.47
88.73
67.25
ODIN
91.28
66.53
82.98
72.89
72.71
82.19
88.27
71.30
83.81
73.23
Energy
91.35
66.52
83.02
72.88
72.45
82.22
88.17
71.29
83.75
73.23
ReAct
93.70
64.52
83.36
73.47
62.85
85.79
89.09
69.87
82.25
73.41
Maha
90.54
56.43
83.24
61.84
75.83
64.71
90.27
54.36
84.97
59.33
KNN
91.70
69.70
80.58
76.46
68.90
85.98
75.57
81.12
76.93
SHE
93.52
63.56
85.62
70.75
81.54
76.97
89.32
71.52
87.50
70.70
Ours
85.80
72.48
76.14
78.77
65.38
86.29
84.87
75.88
78.05
78.35
8

---

Published as conference paper at ICLR 2024
FPR95 (%)
FPR95 (%)
35
40
30
25
35
5/3
2.0
1.4
1.5
2.0
Value of
Value of
Figure 5: Comparisons of varying q when p is fixed at 2.5 (Left) and 3.0 (Right) on CIFAR-
Table 4: Evaluation on long-tailed OOD detection tasks. 1 indicates larger values are better and vice
versa. The best result in each column is shown in bold.
Method
SVIN
LSUN
iSUN
Testure
Places365
Average
HPRSSS
AUROCT
HPR55]
AUROCT
HFR95]
HFR55]
HFR55
AURUC"
AUROCT
MSP
97.82
$6.45
82.48
73.54
97.61
$4.95
95.51
54.53
92.49
60.08
93.18
59.91
ODIN
98.70
48.32
64,80
83.70
97.47
52.41
95.
49.27
91.56
58.49
89.70
58.44
Energy
98.81
47.03
97.37
95.82
91.73
57.09
Mahu
74.01
83.67
77.88
72.63
49.42
85.52
63.10
81.28
94.37
53.00
71.76
75.22
82.35
73.54
97.51
95.73
92.65
93.20
59.72
ODIN-RW
98.82
52.94
83.79
69.28
96.10
50.64
96.95
45.14
93.36
$1.5
93.80
53.90
En
98.86
+9.07
77.30
77.32
96.25
50.91
96.86
45.27
93.03
54,02
92.46
55.32
KNN
Ours
40.16
91.00
45.72
87.64
41.89
90.42
40.50
86.80
91.74
58.44
$2.00
82.86
method on 4 hard OOD datasets, namely, LSUN-Fix Yuu et al. 2015), ImageNet-Fix (Krizhevsky et al., 2012), ImageNet-Rest Resize (Krizhevsky et al.) 2012), and CIFAR-10. The model is trained for 200 epochs, with batch size 128, weight decay 5e-4 and Nesterov momentum 0.9. The start learning rate is 0.1 and decays by factor of 5 60th, 12th, 160th epochs. We select set of strong baselines that are competent hard OOD detection, and the experiments are summarized in Table 3 It can be that method beat the state-of-the-ar across the considered datasets, challenging CIFAR-100 versus CIFAR-10 setting. The reason is that our lp norm-induced density function can better capture the ID data distribution.
for the
4.4.2 LONG-TAILED OOD DETECTION
We consider long-tailed tailed OOD scenarios (Wang et al., 2022 Bai et al. 2023), of which the ID training data exhibits imbalanced class distribution. We the long-tailed versions of CIFAR datasets with the setting in Cao et al. 2019); Zhong et al. (2021). It is by controlling the degrees of data imbalance with imbalanced factor 3 =NNaax/
min- where N and Nuin are the numbers of training samples belonging to the most and the least frequent classes. Following (Zhong et al. 202T Zhou al. 2020). we pre-train the ResNet-32 (He et al. (016) network with 5 50 on CIFAR- 100 for 200 epochs with batch size 128, weight decay 2e-4 and Nesterov momentum 0.9. The start learning rate is 0.1 and decays by factor of at the 160-th, 180-th epochs. The performance of methods and baselines (RP) and Reweighting (RW) in (Jiang et al. 2023) to modify previous OOD scoring functions. The performance gain in Table/emmiircaa empirically demonstrates that using uniform ID class distribution does not make our method incompatible with the model that is pre-trained with class-imbalane imbalanced data.
shown in Table 4 where we introduce the strategies of Replacing
CONCLUSION
In this paper, we present a theoretical framework for studying density-based OOD detection. By establishing connections between the exponential family of distributions and Bregman divergence, we provide unified principle for designing scoring functions. Given the expansive function space for selecting Bregman divergence, we propose pair of conjugate functions to simplify the search process. To address the challenging problem of the partition function, we introduce computation- ally tractable and theoretically unbiased estimator through importance sampling. Empirically, our method outperforms numerous prior methods by significant margin several standard benchmark datasets using various protocols. Since we only consider a pair of conjugate functions in finding Bregman divergence and evaluate method on Convolutional neural networks. In the future, interesting to delve deeper into the design of Bregman divergence and incorporate large-scale pre-trained Vision-Languan Models (VLMss..

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work is partially supported by the Australian Research Council (CE200100025 and DE240100105). Li gratefully acknowledges the support of the AFOSR Young Investigator Pro- gram under award number FA9550-23- 1-0--4, National Science Foundation (NSF) Award No. IIS- 2237037 & IIS-2331669, and Office of Naval Research under grant number N00014-23-1-2643.
ETHIC STATEMENT
This paper does not raise any ethical concerns. This study does not involve any human subjects practices to data set releases, potentially harmful insights, methodologies and applications, poten- tial conflicts of interest and sponsorship, discriminatioo/Diis//aainn concerns, privacy and security legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To make all experiments reproducible, we have listed all detailed hyper-parametee We upload source codes and instructions in the supplementary materials.
REFERENCES
Yong Hyun Ahn, Gyeong-Moon Park, and Seong Tae Kim. Line: Out-of-distribution detection by leveraging important neurons. arXiv preprint arXiv:2303. 13995, 2023.
Michael S Albergo and Eric Canden-Eijnden Building normalizing flows with stochastic inter- polants. arXiv preprint arXiv:2209.1557/ 2022.
Shun-ichi Amari Exponential families and mixture families of probability distributions. In Infor- mation Geometry and Its Applications, pp. 31-49. Springer, 2016.
Katy Azoury and Manfred K Warmuth. Relative loss bounds for on-line density estimation with the exponential family of distributions. Machine learning, 43:211-246, 2001.
Jianhong Bai, Zuozhu Liu, Hualiang Wang, Jin Hao, Yang Feng, Huanpeng Chu, and Haoji Hu. On the effectiveness of out-of-distribution data in self-supervised long-tail learning. arXiv preprint StHi:22306.09334. 2023.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling of images using a gener- alized normalization transformation. arXiv preprint arXiv:1511.06281, 2015.
Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Cluster- ing with bregman divergences. Journal of machine learning research, 6(10), 2005.
Mohamed Ben Alaya, Kaouther Hajji, and Ahmed Kebaier. Adaptive importance sampling for multilevel monte carlo euler method. Stockastics, 95(2):303-327, 2023.
Lev M Bregman. The relaxation method of finding the common point of convex sets and its applica- tion to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200-217, 1967.
Lawrence D Brown. Fundamentals of statistical exponential families: with applications in statistical decision theory. Ims, 1986.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with diteblddistrbuttio--wwar margin loss. Advances in neural information processing systems, 32, 2019.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, Septem- ber 13-17, 2021, Proceedings, Part III 21, pp. 430-445. Springer, 2021.
10

---

Published as a conference paper at ICLR 2024
Long Chen, Yuchen Li, Chao Huang. Bai Li, Yang Xing. Daxin Tian, Li Li, Zhongxu Hu, Xiaox- iang Na, Zixuan Li, et al. Milestones autonomous driving and intelligent vehicles: Survey of surveys. IEEE Transactions Intelligent Vehicles, 8(2):104610566 2022.
Yen-Chi Chen. A tutorial on kernel density estimation and recent advances. Biostatistics Epi-
demiology, 1(1):161-187, 2017.
Sayak Ray Chowdhury, Patrick Saux, Odalric Maillard, and Aditya Gopalan. Bregman deviations of generic exponential families, In The Thirty Sixth Annual Conference on Learning Theory, Pp- 394 449. PMLR, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 2016.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for dut-of-distribution detection. arXiv preprint arXiv:2209.09855 2022.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by
virtual outlier synthesis. arXiv preprint 4rXiv:2202.01197, 2022.
Zhen Fang. Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurIPS, 2022.
Jürgen Forster and Manfred K Warmuth. Relative expected instantaneous loss bounds. Journal of Computer and System Sciences, 64(1):76-102, 2002.
Santosh K Gaikwad, Bharti W Gawali, and Pravin Yannawar. A review on speech recognition technique. International Journal of Computer Applications, 1063):16-24, 2010.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International conference on machine learning, pp. 881-889. PMLR, 2015.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statistics. Journal of machine learning research, 13 (2), 2012a.
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13 (2), 2012b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of di----iibiiiitiion examples in neural networks. arXiv preprint arXiv: 1e.0.0000.22300 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv: 1911:9911..1113.1 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
11

---

Published as a conference paper at ICLR 2024
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribun detection for large semantic space. 2021 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. pp. 8706-8715, 2021.
Xuedong Huang, James Baker, and Raj Reddy. historical perspective of speech recognition.
Communications of the ACM, (7(1):94-100) 2014.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting out-of-distribution data through in-distribution class prior. 2023.
Julian Katz- Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In International Conference on Machine Learning, pp. 10848-10865. PMLR, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
JooSeuk Kim and Clayton D Scott. Robust kernel density estimation. The Journal of Machine Learning Research, 13(1):2529- 2565, 2012.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrateee classifiers for detecting out-of-distributio samples, arXiv preprint 1rXiv:1711.09325, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outto--distributiin samples and adversarial attacks. Advances neural information processing systems, 31, 2018.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffu- sion model is secretly zero-shot classifier. arXiv preprint arXiv:2303.16200 2023.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.026600 2017.
Qiang Liu, Jian Peng, Alexander Ihler, and John Fisher III. Estimating the partition function by dis- criminance sampling. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pp. 514-522, 2015.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -ut-o--distribution detec- tion. Advances in neural information processing systems,
2020.
Mare Masana, Xialei Liu, Bartlomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image clas- sification. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: In International Conference on Machine Learning, pp. 15650-15660 15665. PMLR, 2022a.
detection with posterior sampling.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for duttof--iittibuttio detection? arXiv preprint arXivv2203.044550 2022b.
Andriy Mnih and Koray Kavukeuoglu. Learning word embeddings efficiently with noise-contrastive estimation. Advances in neural information processing systems, 26, 2013.
12

---

Published as a conference paper at ICLR 2024
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detec- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7831-7840, 2022.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probubility, 29(2):429-443. 1997.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, 2011.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684 10695, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Sanjivani Shantaiya, Keshri Verma, and Kamal Mehta. A survey on approaches of object detection. International Journal of Computer Applications, 65(18), 2013.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribttion detection. In European Conference on Computer Vision, pp. 691-708. Springer, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: dut-tf-distribution detection with rectified activa- tions. Advances Neural Information Processing Systems, 34:14441511 2021.
Yiyou Sun, Yifci Ming, Xiaojin Zhu, and Yixuan Li. Outoof-distriuttion detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. Advances in neural information processing systems, 33:11839-1188 11822, 2020.
Surya T Tokdar and Robert E Kass. Importance sampling: areview. Wiley Interdicciplliarr Reviews: Computational Statistics, 2(1))54-60, 2010.
Alexandre B. Tsybakov. Introduction to nonparametric estimation. 2008.
Benigno Uria, Mare-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184- 7220, 2016.
Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J Smola, and Zhangyang Wang- Partial and asymmetric contrastive learning for ouiti--disttibutioon detection in long-tailed recog- nition. In International Conference on Machine Learning, pp.
PMLR, 2022.
Qizhou Wang, Junjie Ye, Feng Liu. Quanyu Dai, Mareus Kalander, Tongliang Liu. Jianye Hao, and Bo Han. -fttt--diitiibuttin detection with implicit outlier transformation. arXiv preprint arXiv:2303.000 050033. 2023.
13

---

Published as a conference paper at ICLR 2024
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural net- work overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631-236444 PMLR, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to 200. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010a.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to In 2010 JEEE computer society conference computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010b.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsoureing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506:
2015.
Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, Shi Han, Dongmei Zhang, al. Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference on Learning Representations, 2022.
Lily Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures nut-of-distribution detection with deep generative models. In International Conference on Machine Learning, pp- 12427-12436. PMLR, 2021.
Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review. IEEE transactions on neural networks and learning systems, 30(1 33(11):332 1):3212-3233 2019.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jía. Improving calibration for long-tailed recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16489-16498, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6))1452-11466 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: caateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 9719-9728, 2020.
Yibo Zhou. Rethinking reconstruction autoencoder-basse
detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7379-7387, 2022.
David Zimmerer, Peter M Full, Fabian Isensee, Paul Jäger, Tim Adler, Jens Petersen, Gregor Köhler, Tobias Ross, Annika Reinke, Antanas Kascenas, et al. Mood 2020: A public benchmark for out- of-distribution detection and localization medical images. IEEE Transactions Medical Imaging, 41(10):2728-2) 2738, 2022.
14

---

Published as a conference paper at ICLR 2024
Table 5: OOD detection on CIFARI00 benchmarks, We average the results across 6 OOD datasets. t indicates larger values are better and vice versa. The best result in each column is shown in bold.
Metrics
MSP
ODIN
Energy
ReAct
DICE
Maha
GEM
KNN
ASH
Ours
FPR95]
79.29
77.47
76.66
69.95
73.06
71.75
65.21
62.22
65,29
54.47
AUROCT
79.9
74.36
80.14
81.99
79.86
63.14
77.81
84.22
81.36
87.26
A APPENDIX
A.1 LIMITATIONS
The limitation of this work lies in manually searching a good value of p to determine Bregman divergence. We do not test method on large-scale models.
A.2 OOD DATASET
For experiments where CIFAR benchmarks are the ID data, we adopt SVHN (ettzrr et al. 201 LSUN-Cruv YYu et al. 2015). LSUN-Resizz (Yu et al. 2015), iSUN (Xu et al. 2015), Places (Zhou et al. 2017). and Textures (Cimpoi et al. 2014) as the OOD datasets. For experiments where ImageNet-1K 1K is the ID data, we adopt iNaturalist (Xiao al. (010b), SUN (Xiao al.! (2010a),
Places365 (Zhou et al. 2017). and Textures (Cimpoi et al. (0114) and the OOD dataset.
A.3 IMPLEMENTATION DETAILS.
Similar to DICE (Sun & Li 2022). adopt Tiny-ImageNet-222 (Le & Yang 2015) the auxiliary OOD data with the searching space of p as (1.3). We remove those data whose labels coincide with ID cases. We set p = 2.2 for experiments in CIFAR-10, p 2.5 for experiments in CIFAR-100, p 1.5 and 1.8 for experiments in ImageNet- on ResNer50 and MobileNetv2 respectively,
A.4 RESULTS WITH DIFFERENT BACKBONES
In the main paper, have shown that method is competitive DenseNet and MobileNet. In this section, we show in Table 5 and Table 6 that the strong performance of our method holds on ResNet50 (He al. 2016). All the numbers reported are averaged over OOD test datasets described in Section 4.2 For ImageNet-Ik, We use the pre-trained models provided by Pytorch. At test time, all images are resized 224x224. For CIFAR- the model trained for 200 epochs, with batch size 128, weight decay 5e-4 and Nesterov momentum 0.9. The start learning rate is 0.1 and decays by a factor of at 60th, 120th and 160th epochs. At test time, all images are of size 32x32.
A.5 OOD DETECTION WITH CONTRASTIVE REPRESENTATIONS
We explore the compatibility of our method with contrastive representations. Closely following the training protocol in Ming et al.] (202b)): Khosla et al.] (2020), we pre-train ResNet-34 (He et al.) 2016) on CIFAR- 100 with the SupCon (Khosla et al.J 2020) and CIDER (Ming et al. (022b) losses respectively. We train the model using stochastic gradient descent for 500 epochs with batch size 512, Nesterov momentum 0.9, and weight decay le-4. The initial learning rate is 0.5 with cosine scheduling. Table 7 demonstrates, under both SupCon and CIDER settings, ours consistently outper- forms the Maha, SHE and KNN scores by a large margin, highlighting our method's 's effectiveness.
A.6 MORE RESULTS ON LONG-TAILED OOD DETECTION
we further test the compatibility of our method to the model that is pre-trained with clas-imballanee data. In this section, we use the long-tailed versions of CIFAR-10
Continuing from Section
datasets with the setting in Cao et al. (2019): Zhong et al. (2021) with an imbalanced factor 3 = 50. Following (Zhong et al., 2021 Zhou et al. 2020). we pre-train the ResNet-32 (He et al., 2016) network with 3 50 CIFAR-10 10 for 200 epochs with batch size 128. weight decay 2 and Nesterov momentum 0.9. The start learning rate 0.1 and decays by a factor of 5 at the 160-th,
180-th epochs. The performance of our methods and baselines shown in Table
15

---

Published as a conference paper at ICLR 2024
Table 6: OOD detection results on the ImageNet benchmark with ResNet-50. values are better and vice versa. The best result in each column is shown in bold.
indicates larger
Method
iNaturalist
SUN
Places
Textures
Average
FPR95
AUROCT
FPR95)
AUROC1
FPR95)
AUROC1
FPR95)
AUROCT
FPR95]
AUROC1
MSP
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
ODIN
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85.62
56.48
85.41
Energy
55.72
$9.26
82.86
53.72
85.99
58.41
86.17
ReAct
20.38
96.22
24.20
94.20
33.85
91.58
47.30
89.80
31.43
DICF
25.63
94.49
35.15
90.83
46.49
87.48
31.72
90.30
34.75
90.77
Maha
97.00
52.65
98.50
42.41
98.40
41.79
55,80
85.01
87.43
55.47
GEM
51.67
81.66
68.87
73.78
79.52
67.34
35,73
86.54
58.95
77.33
KNN
59.77
85.89
68.88
80.08
78.15
74_10
10.90
97.42
54.68
84.37
SHE
45.35
90.15
45.09
87.93
54.19
84.69
34.22
90.18
44.71
88.24
Ours
9.62
97.97
37.75
90.13
48.99
86.60
9.61
97.74
26.49
93.11
Table OOD detection results on CIFAR-100 under contrastive learning. t indicates larger values better and vice versa. The best result in each column is shown in bold.
Method
SVIIN
LSUN
ISUN
Texture
Places365
Average
FPROS,
AURUCT
AUROLT
HPRU5
AURUCT
EPR95
AUROCT
AUROCT
SupCon+Malaa
14.47
97.31
67.81
79.33
80.71
50.35
79.79
95.93
54.17
75.96
SupCea+SHE
28.35
94.34
72.53
78.81
99.18
19.31
72.16
68.07
$3.17
75.41
71.08
67.19
KNN
38.68
92.61
43.16
91.43
67.89
85.04
58.46
86:65
75.18
77.22
56.67
86.59
CupCon+4Curs
28.74
15,43
97.28
74.97
CaDER+Maba
18.52
96.36
88.86
71.93
79.55
79.76
54.31
77.89
95.47
52.93
67.34
75.77
CIDER ISH
28.16
94.09
69.06
82,60
99.69
15.51
75.57
62.84
87.21
72.41
71.94
65.51
CIDER+KNN
CIDER
15.87
96.55
6.04
98.76
$2.45
88.32
31.97
93.31
72.31
75.94
35.73
90.58
A.7 DETAILED CIFAR RESULTS
Table 10 and Table 11 supplement Table in the main text, as they display the full results on each of the 6 OOD datasets for DenseNet trained on CIFAR-IO and CIFAR-100 respectively. Table I2 supplement Table as it displays the full results on each of the 6 OOD datasets for ResNet50 trained on CIFAR-100 100 respectively
A.8 DISCUSSION ON DEEP GENERATIVE MODELS FOR DENSITY ESTIMATION
Since density estimation plays key role in our method, our work is related to deep generative models that achieve empirically promising results based on neural networks. Generally, there are two families of DGMs for density estimation: 1) autoregressive models (Germain al.) [2015] Uria et al.) 2016 Papamakkris et al. (017)) that decompose the density into the product of conditional densities based probability chain rule where Each conditional probability is modeled by parametric density (e.g., Gaussian or mixture of Gaussian) whose parameters are learned by neural networks, and 2) normalizing flows (Rezende & Mohamed, 2015 Ballé et al. 2015 Dinh al. 2016 Grover ct al. 2018 Albergo & Vanden-Eing Vanden-Eijnden] (2022) that represent input as an invertible transformation of latent variable with known density with the invertible transformation
composition of series of simple functions. While using DGMs for density estimation seems to be a valid and intuitive option for bensity-based OOD detection, this requires training DGM from scratch and therefore violates the principle of post-hoc OOD detection, i.c., only pre-trained models at hand are expected to be used detect OOD data from streaming data at the interference stage. Besides, Zhang et al. (2021) finds that DGMS tend to assign higher probabilities or densities to OOD images than images from the training distribution. We also explore the possibility of integrating pre-trained Diffusion models (Peebles & Xie 2023 Rombach et al. 2022) into zero-shot cass-conditioned density estimation based on (1) in Li al. (2023). Unfortunately. the computation is intractable due to the integral. Although authors in Li al. (2023) a simplified ELBO for approximation, there is theoretical guarantee that the ELBO can well align with the data density not to mention the computational- inefficient inference of diffusion models. We will leave this challenge future work.
16

---

Published as a conference paper at ICLR 2024
Table Evaluation on long-tailed OOD detection tasks on CIFAR-IO. 1 indicates larger values are
better and vice versa. The best result in each column is shown in bold.
Method
SVHN
LSUN
ISUN
Testure
Places365
Average
FPROS]
AUROOCT
FPROS)
AUROCT
FPRUS]
AUROO:
AUROCT
AUROC!
AUROC1
MSP
90.12
78.00
83.62
86.38
53.29
90.00
17.23
77.24
87.42
17.88
78.34
82.012
ODIN
Energy
99.74
75.83
77.12
42.15
88.25 18.29
79.13
71.71
83.78
62.56
76.13
67.10
Maha
80.80
81.86
98.07
67.41
75.34
83. 17
74.66
80.83
92.51
68.22
84.28
76,70
90.00
ODIN-RW
88.64
70.74
$2.37
84.93
63.97
84.37
82.62
68.93
74,70
79.47
72.46
77.69
Energy-RW
94.10
67.02
54.60
87.21
52.19
89.10
79.65
71.87
75.90
80.42
71.29
79.12
KNN
77.29
86,80
77.90
78.91
50.98
93.54
76.56
88.73
57.70
77.31
68.09
Ours
28.89
93.65
55.36
85.55
48.08
88.78
54.45
85.29
78.77
79.05
53.11
86.46
A.9
INTRACTABLE LEARNING OF THE EXPONENTIAL FAMILY NATURAL PARAMETER
Given the fact that J 6o (z)k) dz = 1, we then have:
f exp {z n ((nx) - (z(z)} dz
(15)
Eq. 15 means that, for any known (-) and g-bb(-), one can learn the natural parameter by solving the following:
exp exp
exp {z 7h - (z) dz
(16)
Since the right side of Eq. [6]includes the integral over latent feature space that is high-dimensional, learning the natural parameter of an Exp. Family is said be intractable.
A.10 THEORETICAL JUSTIFICATION
Let B denotes the Borel o-algebra Z and P(Z) denotes the set of all probability measures (Z, B), We recall the following definitions:
Definition 3 (Total Variation). Let Pi, P2 P22). The total variation(TV) is defined by:
8(P,, P2) sup (P((A) P2(A))
(17)
AgB
We use the following characterization of TV (See Müller (1997) Theorem 5.4): Lemma 1. Let P,, P2 P(Z) and let F denotes the unit ball in L (Z), ie..
= (f E L (Z) S then have the following characterization for the TV distance,
(18)
6(P1, sup f EZEP,f(z))
(19)
fef
Next, let us recall the definition of Kullback Bicilback-Leibee((LL) divergence, Definition (KL Divergence). Let P,,P2 P(2) be two probability measures with density func- tions pı and p2 respectively. The KL divergence is defined by
/
(z(z) In P1(z)
KL(P,
dz (2(z)
(20)
J2E2
whenever the above integral defined.
Next, recall the following standard lemma that computes KL divergence between exponential family distributions.
Lemma (Relation between Bregman Divergences and KL Divergence (Banerjee et al. 2005)). Let Pr, P2 E P(Z) conform to exponential family distributions with the corresponding density functions Pr and p2 are parameterized by and dp respectively, then we have the following:
KL(P; (P2) (p(mm)) p(T12))
(21)
where d. .) is the so-called Bregman Divergence.
17

---

Published as a conference paper at ICLR 2024
Next, recall the following inequality that bounds the TV by KL divergence. (see Tsybakov 2008), Lemma 2.5 and Lemma 2.6])
Lemma (Pinsker inequality). Let P2, E P(2) then we have the following:
P2) S V
KL(P, P2)
(22)
Theorem 2. We have the following bound: E ExEP, Lio (z) - EEEP lão (z) S
(23)
where : K E V 2d (1l(ss) (e(m)))))
Theorem bounds the measure D in terms of Bregman divergence between (177)) and (ouu). can be observed that will converge to a - This indicates that the performance of our method can be guaranteed by sufficiently discriminative feature space where the averaged Bergman divergence between ID-elass means and OOD data mean sufficiently large. This theory is empirically justified by our results in Section A.5 where CIDER are more beneficial to our method than SupCon with the former learning more powerful feature representations than the latter.
Proof. First, notice that po (z) E [0, 1]. Vz E Z, Therefore, by Lemma I have, EzEP, 1Po (z)] ExEP (Po (z)] S 6(Pi,, Pomt)
(24)
Next, recall that Pmn (z)) 1 Ex= Pmn (z)k), let Pk denotes the probability distribution corre- sponding to Pm (-Ik) and by triangle inequality and the definition of total variation we obtain
R
K
1
8(Pin, Pont) 5
pk in Pont) = sup AEB
pl Pour Pour(A)
(25)
K
K
k=1
k=
sup AEB
p (A) -Poul (A))
(26)
K
k=
K
5(Pti Pou))
(27)
K
Finally, by Lemma
and Lemma
3
we have:
1 2d (M(())))) (o(Tou)))
ö 8Ph Pout) S (KL(Pi||P)
(28)
Putting all together, obtain
1
EREP, p0 (2)) ExEP. 100 (z))
(29)
K
and the proof is complete.
A.11 CONTRIBUTION SUMMARY
The contributions of our method are summarised as follows:
It is always non-trivial to generalize from specific distributiiondiitannee to a broader dis- tribution/distance family since this will trigger an important question to the optimal design the underlying distribution (.). To answer this question, we explore the conjugate re- lationship as guideline for the design. Compared with other hand-crafted choices, proposed I, norm general and well-defined, offering simplicity in determining its conju- gate pair. By searching the optimal value of p for each dataset, we flexibly model ID data data-driven manner instead of blindly adopting a narrow Gaussian distributional assumption in prior work, i.e., GEM (Mortzza & Li 2022) and Maha (Lee al. 2018).
18

---

Published as a conference paper at ICLR 2024
Table 9: Additional results of long-tailed OOD detection on Cifar-100, where we consider two baselines: (a) the ID training data is with class imbalance and (b) the ID training data is with class balance. f indicates larger values are better and vice versa.
Baseline
Maha
GEM
KNN
Ours
FPR95]
AUROCT
FPR95}
AUROCT
FPR95
AUROCT
FPR95
AUROCT
(a)
71.76
75.22
66.82
76.97
58.11
81.75
52.00
82.86
(b)
67.39
77.16
56.67
82.93
60.11
79.22
48.46
84.02
Our proposed framework reveals the core components in density estimation for OOD de- tection, which was overlooked by most heuristic-based OOD papers. this way, The framework not only inherits prior work including GEM (Morteza & Li, (2022) and Maha (Lee et 2018) but also motivates further work to explore more effective designing prin- ciples of density functions for OOD detection.
We demonstrate the superior performance of our method on several OOD detection benchmarks (CIFARI0/100 and ImageNet-IKe different model architectures (DenseNet, ResNet, and MobileNet), and different pre-training protocols (standard classification, long- tailed classification and contrastive learning).
A.12 LIST OF ASSUMPTIONS
The assumptions made in our method are given follows:
1. The ID class prior is uniform, i.e., Po (k) K
2. (.) const and (-(-) -11l
We note that 1) Assumption is made in many post-hoc OOD detection methods either explicitly or implicitly (Jiang et al., 2023). Experiments in Section 4.4.2 show that our method still outperforms in long-tailed scenarios with Assumption 1, and 2) Assumption 2 helps to reduce the complexity of the exponential family distribution. While is possible to parameterize the exponential family distribution in more complicated manner, our proposed simple version suffices perform well.
A.13 A CLOSER LOOK AT EXPERIMENTS ON LONG-TAILED OOD DETECTION
As shown in Table all methods that involve the use of ID training data suffer from decrease in their averaged OOD) detection performance when the ID training data with class imbalance, Note that we keep using the network are-trained on the long-taiie tailed version of CIFAR- 100 for fair compar- ison. Even so, our method consistently outperforms in both scenarios, which implies the robustness of our method. We suspect the reason is that the flexibility of the norm coefficient provides with the chance to find a compromised distribution from the exponential family.
19

---

Published as a conference paper at ICLR 2024
Table 10: Detailed results on six common OOD benchmark datasels: Textures, SVHN, Places366, LSUN-Cropp LSUN Resize and iSUN. For each dataset, we use the same DenseNet pretrained on CIFAR- 100.
AUROCT
74.36
84.49
81.19
84.47
87.23
90.02
82.73
82.60
88.75
92.50
93.21
Average
FPR95
80.13
58. 14
68.45
62.27
49.72
41.40
55.37
54.66
41.52
28.27
25.66
AUROCT
74.31
76.84
77.72
74.04
77.26
71.62
61.39
77.90
67.19
73,99
74.51
Places365
FPR95.
82.55
82.32
79.20
82.65
79.58
85.86
92.66
74.94
88.72
80.29
72.27
AUROCT
71.48
71.48
71.03
78.95
76.42
92.35
79.39
76.49
93.44
94.77
94.73
Texture
FPR95L
84.79
82.34
84.15
77.78
65.04
34.02
62.39
60.05
24.73
20.41
40
100. 1 indicates larger valuees are better and vice versa.
ISUN AUROCT
70.17
83 84
78.95
86.55
90.08
91.30
93.21
76.42
39.41
96.36
97.41
FPR95,
85.99
67.05
74.54
65.27
48.72
46.67
31.38
72.36
39.41
18.77
18.27
AUROCT
69.18
84.22
80.14
87.88
91.04
90.12
94.20
74.14
90.65
96.11
96.93
LSUN-R
FPR951
85.24
65.22
70.65
60.08
49.40
51.33
23.07
77.63
46.95
20.34
21.34
AUROCT
85.60
97.93
97.43
94.92
99.74
98.94
86.30
99.66
92.85
96.97
97 85
LSUN-C
FPR95
60.49
10.54
14.72
25.55
0.93
5.52
68. 90
1.01
31.62
15.94
12.15
AUROCT
75.40
92.65
81.85
81.41
88.84
95.76
95.67
91.02
96.40
97.05
97.83
SVHN
FPR95
81.70
41.35
87.46
83.81
54.65
25.02
22.44
41.98
17.67
5.37
10.53
Ours+ASH
Method
MSP
ODIN
Energy
React
DICE
ASH
Maha
SHE
KNN
Ours
20

---

Published as a conference paper at ICLR 2024
Detailed results on six common OOD benchmark datasels: Textures, SVHN, Places LSUN-Crop. LSUN Resize and iSUN. For each dataset, we use the same DenseNet pretrained on CIFAR- 10. indicates larger values are better and vice versa.
AUROCT
92.46
93.71
94.57
95.24
94.67
96.61
89.15
94.40
96.74
97.15
97.65
Average
FPR95L
48.73
24.57
26.55
20.83
26.45
15.05
31.42
23.26
17.43
13.92
12.14
AUROCT
88.57
88.55
91.64
89.13
91.03
88.34
63.15
92.39
92.06
91.82
91.83
Places365
FPR951
63.02
52.85 2..55
39.40
48.59
43.31
48.45
85.14
38.32
40,00
42.37
47.72
AUROCT
88.15
82.38 38
86.43
88. 88.18 18
92.47
95.09
92.15
83.52
95.09
96.72
96.83
Texture
FPR95L
64.15
57.50
56.12
41.90
43.58
24.34
21.51
51.10
27.09
17.06
17.22
AUROCT
94.52
90
98.07
97.72
98.90
97.25
98.05
97.65
98.65
.9
99.
ISUN
FPR95,
42.3
3.98
10.07
4.36
12.72
5.17
9.78
10.49
13.77
6.42
2.77
AUROCT
94.51
99.02
98.12
99.20
97.87
98.92
97.09
98.23
97.35
98.92
99.58
LSUN-R
FPR95
42.10
3.09
9.28
3.91
11.46
4.96
9.03
15.24
5.18
1.65
AUROCT
95.54
99.15
99.92
98.84
99.73
66
99.85
99.76
99.56
LSUN-C
98
98
FPR95
33.57
4.70
3.81
0.26
5.96
0.90
56.55
0.58
0.81
7.13
1.87
AUROCT
93.48
94.57
93.99
95.90
93.87
98.65
98.31
94.48
98.52
98.95
98.70
SVHN
FPR951
47.24
25.29
40.61
25.99
41.64
6.51
6.42
30.02
7.64
5.37
6.61
Ours+ASH
Table 11:
Method
MSP
ODIN
Energy
DICE
React
ASH
Maha
SHE
KNN
Ours
21

---

Published as a conference paper at ICLR 2024
Detailed results on six common OOD benchmark datasets: Textures, SVHN, Places365 LSUN-Crop LSUN LSUN-Resize Resize, and iSUN. For each ID dataset, we use the same DenseNet pretrained on CIFAR- 100. 1 indicates larger values are better and vice versa.
AUROCT
79.29
94.57
79.86
81.99
81.36
63.14
84.22
87.26
79.
Average
FPR95L
79.29
77.47
26.55
73.06
69.95
65.29
71.75
62.22
54.47
AUROC1
69.27
69.78
91.64
65.97
70.96
66.83
61.31
70.98
75.77
Places365
FPR95,
87.01
87.30
39.40
91.65
84.99
84.47
82.65
81.04
76.77
AUROCT
73.10
74.81
86.43
75 73.84
81.60
84.47
69.04 .4.04
85.65
88
Texture
FPR95
87.50
87.61
56.12
81.13
68.60
61.13
53.40
54.15
42.41
AUROC1
79.93
83.04
98.07
81.16
85.30
76.27
65.45
86.56
86.97
ISUN
FPR95
78.36
76.01
10.07
81.76
66.36
79.35
71.69
65 65.49
67.51
AUROCT
81.99
84.61
98.12
81.75
84.65
77.03
65.38
87.76
87.54
LSUN-R
FPR95L
75.09
72.78
9.28
79.00
67.93
77.01
74.26
65.13
68.25
AUROCT
58.71
90.50
99.15
95.95
90.98
95.37
58.71
84.50
91.10
LSUN-C
FPR95,
91.36
77.68
3.81
18.07
40.08
26.21
91.36
60.87
40.53
AUROCT
58.98
77.68
93.99
78.47
78.45
88.20
58.98
89.80 89.84
93.26
SVHN
FPR95,
63.14
88.46
40.61
86.75
91.73
59.22
57.16
46.63
31.35
Table 12:
Method
MSP
ODIN
Energy
DICE
React
ASH
Maha
KNN
Ours
22


Reviews:
[{"review_id": "GwWPsoTrau", "reviewer": "Reviewer_EBSh", "summary": "This paper studies the problem of density estimation for density-based out-of-distribution detection. The authors firstly point out that existing logit, distance, and density based OOD methods can be either inconsistent or too restrictive. Then the authors utilize the property of exponential family and its relation to Bregman divergence to induce a modeling of the density function using conjugate norms. They further combine different method for the estimation of partition functions and finally experimentally validate their methods.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The assumption of exponential family is mild and the authors utilize the property of conjugate functions to derive a concise formulation of density functions.\n2. The estimation of partition functions can be combined with different methods, which indicates the flexibility of the proposed method. \n3. The experimental results on widely-used benchmark datasets validate the usefulness of the proposed method.", "weaknesses": "The theoretical results of this paper are based on the exponential family of distribution. I think this is an explicit assumption on the prior distribution, which contradicts your answer to question ♠. An analysis on the potential extension of your results can be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wBvQzwbsua", "reviewer": "Reviewer_WvTp", "summary": "The authors present a data density estimation method targeted towards out of distribution detection. The authors parameterize Bregman Divergences, which in turn are shown to parameterize Exponential Families up to an approximable normalization constant.\n\nThe authors conduct extensive evaluations on numerous OOD tasks and include ablations, showing improvements over benchmarks on many tasks.\n\nIn general the mathematical presentation is careful, results are contextualized, and the reader learns both about OOD in general and about the specific presented method, picking up some tricks on parameterizing densities along the way.", "questions": "1)\n\nCould you please clarify this phrase? I re-read it a few times and just did not understand its meaning\n\n\"Without loss of generality, we employ latent features z extracted from deep models as a surrogate for the original high-dimensional raw data x. This is because z is deterministic within the post-hoc framework.\"\n\n2)\n\nPlease answer my main question in Weaknesses, on why no discuss of the role in DGMs for flexible density estimation in OOD detection.\n\n3) small comment:\nplease tell the reader more about why learning the natural parameter of an Exp. Family intractable.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors conduct extensive evaluations on numerous OOD tasks and include ablations, showing improvements over benchmarks on many tasks.\n\nIn general the mathematical presentation is careful, results are contextualized, and the reader learns both about OOD in general and about the specific presented method, picking up some tricks on parameterizing densities along the way.", "weaknesses": "Nothing particularly bad and things seem correct and well reported and well explored. \n\nMostly, the weakness would just be the lack of discussion of Deep Generative Models. The paper seems to present density estimation as the main challenge of OOD detection. For some of the considered benchmarks such as GEM, the main criticism is the specific distribution assumptions, hence the exploration in this work across the Exponential Family.\n\nOn the other hand, Deep generative models (DGMs) are a flexible approach for modeling data distributions without making distributional assumptions. Not all DGMs give the user a computable density, but there are some that do such as Normalizing Flows and more recently methods like Flow Matching, Stochastic Interpolants, and others to name a few. More generally some models give you un-normalized log densities, which also seem to be okay for this work considering that this work is willing to estimate certain normalization constants.\n\nIt's totally okay to explore a non-DGM-based method in this work, but I think it would strongly benefit from some contextualization and an attempt to answer this question in at least one way: \n\nFor high dimensional data such as images, why should someone not pick generic deep generative models that admit densities (or un-normalized densities, or maybe log density lower bounds) and why instead should someone stick with search within the exponential family (for which you give good methods, algorithms, etc, and for which you get good results)?\n\nThere is some older work on role of DGMs in OOD detection (such as https://arxiv.org/abs/2107.06908) but I think a LOT of progress has been made in image DGMs since then (like DDPM https://arxiv.org/abs/2006.11239, Interpolants, https://arxiv.org/abs/2209.15571, Diffusions in latent space, https://arxiv.org/abs/2212.09748, etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOARPSnnPb", "reviewer": "Reviewer_LYsh", "summary": "The paper studies the task of out-of-distribution (OOD) data detection for supervised learning tasks. The proposed method is essentially along the lines of density level set thresholding, and the density estimates leverage a particular type of exponential family (uniform) mixtures originating from a Bregman-divergence-related framework. After choosing specific parameters for computational tractability and performance, the authors compared their method with multiple baselines and prior works to demonstrate its effectiveness.", "questions": "Following the comments on the weaknesses, it would be helpful if the authors could\n- Provide theoretical justification (guarantees) for the proposed method.\n- Explain and summarize the technical novelty in addition to prior works.\n- List out all the assumptions made leading to the final method.\n\nIn addition, it would be good if the authors could make the notations more distinguishable, e.g., addressing the overuse of $\\hat{p}_\\theta(\\star)$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper reads well and introduces background and prior works properly. In addition, the authors leverage concrete examples and visualizations, such as plots and tables, to help the reader get the most critical points. All these benefit the readability and clarity of the submission. \n\nRegarding novelty and originality, the proposed method differs from existing approaches algorithmically and originates from a more general framework. It is good that the authors spend efforts building theoretical justifications and showing underlying motivations. \n\nThe experiment results (seem to) suggest that the new method has advantages over the prior works and often outperforms its predecessors, at least within the experimental setting of the authors. Sometimes, there are seven to ten competitors and two to five benchmark datasets, such as ImageNet and CIFAR-10. From my perspective, this is the most substantial contribution added by the paper, showing the method's practicality.", "weaknesses": "One area for improvement is that while the method comes from a theoretical framework, I need to find explicit theoretical guarantees and technical claims to justify its effectiveness. So, the lack of theoretical justification is a weakness worth addressing, potentially deriving some for simple cases like the Gaussian one or explaining why the algorithm tends to perform well for small p values. \n\nThe second weakness is that the prior work (Morteza & Li, 2022) already proposed a method based on Gaussian assumptions and Mahalanobis distance. The extension in this paper, at least logically, is relatively straightforward, i.e., from Gaussian to Exponential Family and from Mahalaobis distance to Bregman-divergence (an extension). Maybe it's worth adding a section summarizing the paper's technical novelty. \n\nAnother weakness is that while the framework's formulation seems general, multiple assumptions come along the way. For example, the authors assumed a uniform prior and set $\\psi$ to the $l_p$ norm. Of course, these might be necessary for a computationally tractable approach and could be acceptable in their current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "piU6rTDyjj", "reviewer": "Reviewer_MqR8", "summary": "This paper proposes a new scoring method for post-hoc out-of-distribution (OOD) detection, by considering the OOD detection problem as a density estimation over the exponential family. Using the connections between the exponential family of distributions and the Bregman divergence, the original density estimation problem over the exponential family is converted into the problem of finding the Bregman divergence. Then, to reduce the search space for selecting Bregman divergence, the authors propose a pair of conjugate functions and reframe the original problem into the problem of finding the optimal norm coefficient $p$ against the given dataset. The partition function is estimated using the Mont Carlo-based importance sampling technique. Experimental results demonstrate the efficacy of the proposed score, with varying $p$ depending on the dataset (p=2.2 for CIFAR-10, 2.5 for CIFAR-100, 1.5 and 1.8 for ImageNet-1k on ResNet50 and MobileNetv2).", "questions": "- Can the authors provide any reasonable method to choose $p$ given the training dataset and the corresponding features given NN architecture without the hyperparameter search?\n- The partition function is estimated using the importance sampling-based approximation. For long-tailed OOD detection when the ID training data exhibits an imbalanced class distribution, I guess the accuracy of the importance sampling-based estimation may decrease given a limited number of tail-class predictions. Given that, can the authors elaborate why their method still outperforms in the long-tailed scenarios?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Proposed a unified scoring method for post-hoc OOD detection using a general exponential family.\n- Converted the original search problem over the expansive function space of Bregman divergence into a simple problem of selecting optimal norm coefficient.\n- Demonstrated the effectiveness of the proposed score on CIFAR-10/100, ImageNet-1k, and scenarios including hard OOD detection and long-tailed OOD detection.", "weaknesses": "- The main search problem of optimal coefficient $p$ for OOD scoring is remained as a hyperparameter search, which may constrain the practicality of the proposed score. Furthermore, the OOD detection performance (FPR95) is quite sensitive to the value of $p$ (Figure 4), which implies that the performance of the proposed score highly depends on the hyperparameter $p$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)