Outlier Synthesis via Hamiltonian Monte Carlo for Out-of-Distribution Detection
Authors: Hengzhuang Li
Teng Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Trustworthy Machine Learning
Out-of-Distribution Detection
Outlier Detection
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Out-of-distribution (OOD) detection is crucial for developing trustworthy and reliable machine learning systems. Recent advances in training with auxiliary OOD data demonstrate efficacy in enhancing detection capabilities. Nonetheless, these methods heavily rely on acquiring a large pool of high-quality natural outliers. Some prior methods try to alleviate this problem by synthesizing virtual outliers but suffer from either poor quality or high cost due to the monotonous sampling strategy and the heavy-parameterized generative models. In this paper, we overcome all these problems by proposing the Hamiltonian Monte Carlo Outlier Synthesis (HamOS) framework, which views the synthesis process as sampling from Markov chains. Based solely on the in-distribution data, the Markov chains can extensively traverse the feature space and generate diverse and representative outliers, hence exposing the model to miscellaneous potential OOD scenarios. The Hamiltonian Monte Carlo with sampling acceptance rate almost close to 1 also makes our framework enjoy great efficiency. By empirically competing with SOTA baselines on both standard and large-scale benchmarks, we verify the efficacy and efficiency of our proposed HamOS.

Published as a conference paper at ICLR 2025
OUTLIER SYNTHESIS VIA HAMILTONIAN MONTE CARLO FOR OUT-OF-DISTRIBUTION DETECTION
Hengzhuang Li, Teng Zhang
School Computer Science and Technology.
Huazhong University of Science and Technology, Wuhan, China (hengzhuangli, lengzhang/@hust. wew.eeeuuuuuu.urrr.moeoeoe edu.cn
ABSTRACT
Out-of-distribution (OOD) detection is crucial developing trustworthy and re- liable machine learning systems. Recent advances in training with auxiliary OOD data demonstrate efficacy in enhancing detection capabilities. Nonetheless, these methods heavily rely acquiring large pool of high-quality natural outliers. Some prior methods try to alleviate this problem by synthesizing virtual out- liers but suffer from either poor quality or high cost due to the monotonous pling strategy and the Sravy-parameterized generative models. In this paper, we overcome all these problems by proposing the Hamiltonian Monte Carlo Ouilier Synthesis (HamOS) framework, which views the synthesis process as sampling from Markov chains. Based solely on the in-distribution data, the Markov chains can extensively traverse the feature space and generate diverse and representa- tive outliers, hence exposing the model to miscellaneous potential OOD scenar- ios. The Hamiltonian Monte Carlo with sampling acceptance rate almost close to also makes our framework enjoy great efficiency. By empirically compet- ing with SOTA baselines both standard and large-scale benchmarks, ver- ify the efficacy and efficiency of our proposed HamOS, Our code is available at: https:/ / /g thu..uu...... com
INTRODUCTION
Despite the impressive achievements of deep neural networks across various practical applications, they unable to make dependable predictions when faced with owt-of-distribution (OOD) samples (Nguyen et al., 2015; Bendale & Boult, 2016; Hendrycks & Gimpel, 2017), as the input data may not be
70
CSI
Scale
CIFAR- FAAR-100PP FPR95 4
ASH
65
KNN+
MSP
Energy
ODIN
60-
drawn from in-distribution (ID) (Bommasani al., 2021). To guarantee the reliability of machine learning systems, models
Relation
VOS
KNN
55
PALM
should recognize data from unknown classes instead of mak-
SSD+
ing overconfident predictions, known as OOD detection. It has
NPOS
50
CIDER
Ours
garnered significant attention within the Safety AI community,
HamOS
Baselines
ilous situations, especially vital for safety-critical applications like autonomous driving (Heidecker et al., 202 la:b) and medi- cal image analysis (Ulmer et al., 2020; Linmans et al., 2020).
as erroneous predictions of OOD samples might result in per-
45
20
10
50
CIFAR-10
FPR95
Figure 1: OOD detection perfor- mance CIFAR and ImageNet. The size of the dots indicates AU- ROC w.r.t. ImageNet-
Prior research can be categorized into two paradigms: post-hoc methods and regularization-buee based methods. The former focuses on designing robust scoring function to differentiate between ID and OOD samples (Hendrycks Gimpel, 2017; Liang et al., 2018; Liu al., 2020; Mahalanobis, 1936; Sun et al., 2022), while the latter incorporates supple- mentary regularization during the training phase, with or without auxiliary OOD data, to enhance the model's discriminative ability (Hendrycks et al., 2019; Wei al., 2022: Zhang et al., 2023a; Zhu et al., 2023b; Ming et al., 2023a), typically achieving better performance than post-hoc methods.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Outlier exposure (OE) approaches (Hendrycks et al., 2019; Zhang al., 2023a; Park et al., 2023), which employ meticulously gathered supplemental OOD data for training, can markedly enhance detection efficacy. Among them, certain studies (Chen et al., 2021; Ming et al., 2022; Jiang et al., 2024a) investigate sophisticated sampling strategies utilizing an extensive pool of auxiliary OOD data, while other studies (Zhang et al., 2023a; Wang et al., 2023; Zhu et al., 2023b; Zheng et al., 2023) suggest augmenting auxiliary outliers using subset of natural outliers as anchors. However, these methods may be constrained as they require rigorously collected natural outlier data, which is infeasible for many domain-spectie applications where high-quality outliers are expensive.
To address the issues encountered by OE methods, some studies (Du et al., 2022b; Tao al., 2023; Du et al., 2023; Pourreza et al., 2021) propose to synthesize virtual outliers without assuming the availability of natural OOD data, resulting in a more flexible OOD-aware training scheme. Due to advancements in generative models (Goodfellow et al., 2020; Esser et al., 2021; Ho et al., 2020), cer- tain studies (Lee et al., 2018a; Greic et al., 2021; Kong & Ramanan, 2021; Du et al., 2023) generate outliers in pixel space, while others (Du et al., 2022b; Tao et al., 2023) primarily employ Gaussian sampling to generate outliers in feature space. The latter approach provides a more straightforward method for incorporating OOD supervision signals into training, avoiding the extensive computa- tion of generative models. Despite attaining considerable performance, current studies either impose stringent assumptions on the IDs or exclusively sample from sub-regions near decision boundaries, resulting in synthesized outliers that lack diversity and representativeness which is demonstrated to be crucial for learning with OOD outliers (Kong & Ramanan, 2021; Chen et al., 2021; Ming al., 2022: Zhu et al., 2023b; Jiang et al., 2024a). Ensuring ID accuracy in the original work is also essential for regularization- based approaches. From the above analysis, a pivotal question arises:
How can we efficiently synthesize diverse and representative outliers based solely on the ID data?
Faced with the challenges mentioned above, we propose an innovative Hamiltonian Monte Carlo Outlier Synthesis (HamOS) framework for OOD detection, presenting a paradigm shift in outlier synthesis by explicit sampling through Markov chains with ID feature embeddings. HamOS samples diverse and representative virtual outliers in the latent feature space, which act crucial surrogate outlier supervision signals for learning essential unknown semantic information absent in the ID data. Unlike sampling through Gaussian distribution, we formulate the synthesis process as Markov chains, wherein a new outlier is generated based on the preceding one (see Fig- ure 3). We characterize the OOD likelihood in hyperspherical space by a novel OOD-ness estimation to assess the probability level of sample from unknown classes. An iterative sequence of outliers can be generated using Hamiltonian Monte Carlo (HMC) (Duane et al., 1987), efficient sampling algorithm of score-based Markov Chain Monte Carlo (MCMC) (Metropo-
VAR+0.02
VAR-0.04
VAR=0.11
0.0-
OOD Score
-0.5
-..0
1.5-
-2.0
vós
NPOS
Hamos
Synthesis Methods
Figure 2: OOD scores of virtual outliers synthesized with different methods.
lis et al., 1953; Neal, 1993), integrated with the OOD-ness estimation. The sequence of outliers exhibits varying degrees of OOD characteristics, which can yield distinct supervision signals for training. Figure shows the OOD scores of generated virtual outliers, where HamOS generates more diverse outliers, which occupy broader range of OOD scores with more significant variance.
HamOS maintains and updates the ID feature embeddings during training as ID prior knowledge. Initialized with midpoints between ID clusters, HamOS chooses the data points with higher OOD- ness density values against the ID feature embeddings. Specifically. to sample a batch of outliers for training, we establish Markov chains between close ID clusters and gather the outliers in each Markov chain. To generate the next outlier feature in a Markov chain based on a pair of close II) clusters, we estimate the OOD-ness for candidate data using the averaged KNN distance (Gu et al., 2019; Zhao & Lai, 2022) from the pair of ID clusters. The candidate is chosen it exhibits higher OOD-ness value. Additionally, to prevent erroneous outlier synthesis (i.e., points with higher OOD- ness values lie within ID clusters), we put a hard margin barrier that keeps the generated outliers away from ID clusters. The barrier is calculated via kernel density estimation with the von Mises- Fisher kernel. The generation pipeline ensures that we can get a mini-batel of outliers spanning a substantial area in the feature space while possessing diverse levels of OOD-ness. We then train the
2

---

Published as a conference paper at ICLR 2025
model with OOD discernment loss and ID contrastive loss to learn proper hyperspherical space. As for the inference-tinee OOD detection, we can adopt distance- based scoring functions to calculate detection scores. HamOS is computationally tractable framework compatible with numerous HMC variants, ID contrastive losses, and scoring functions.
We conduct extensive empirical analysis demonstrate the state-of-the-art (SOTA) performance of HamOS. On the standard OOD detection benchmark, HamOS significantly surpasses the compet- itive baselines that either synthesize virtual outliers (Du et al., 2022b; Tao al., 2023) regularize models (Tack et al., 2020; Sehwag et al., 2021; Sun et al., 2022; Ming et al.. 2023b; Lu et al., 2024) by a considerable margin. Specifically, HamOS enhances the OOD detection performance indicated by FPR95 on CIFAR-10 10 and CIFAR- 100 (Krizhevsky & Hinton, 2009) by 27.1 27.17% and 5. 5.96% re- spectively, evaluated with five standard OOD test datasets. Experiments on ImageNet-1k 1K (Deng et al., 2009) further demonstrate the flexibility for HamOS to scale to large scenarios. In Figure 1, HamOS demonstates superior performance on CIFAR-10/100 and ImageNet-IK, Comprehensive ablation studies elucidate the intrinsic mechanism of HamOS, showcasing its superiority against other baselines. Our primary contribution is delineated as follows:
We are the first investigate the new framework for outlier synthesis via Markov chains instead of sampling from Gaussian distribution. We hope this paper inspires more works to explore advanced algorithms to explicitly sample outliers from feature space.
We propose the general Hamiltonian Monte Carlo Outlier Synthesis (HamOS) framework that generates diverse and representative virtual outliers to enhance ID-OOD separation in the latent space for better OOD detection performance.
We conduct extensive experiments on standard and large-scale benchmarks to demonstrate that HamOS establishs SOTA performance compared with baselines. Comprehensive ab- lation conducted to reveal the intrinsic mechanism and superiority of HamOS.
2 PRELIMINARIES
This paper studies OOD detection in multi-class classification setting, where X Rdd is the input space and y 21 ...., C} is the label set. Let DID {(x;, p:) be the ID training dataset, drawn i.i.d. from the distribution PiPy on X' V, and piP denote the marginal distribution on X'
2.......
2.1 OOD DRTECTION
OOD detection can be formulated binary classification task, i.e., for any input X, the model should determine whether is from P!D or Poon, where P denotes some unknown distribution whose label set has overlap with V. The prediction for OOD detection is indicator function 1{5(x) I 3} where S(x) is some scoring function and | is a threshold to ensure a high true positive rate of IDD data (e.g., 95%). Given input æ,, is identified OOD data S(x) S B.
2.2 HAMILTONIAN MONTE CARLO
Inspired by the Hamiltonian mechanics (Hamilton, 1833). Hamiltonian Monte Carlo (HMC) (Duane et al., 1987) is proposed as an efficient Markov Chain Monte Carlo (MCMC) method. Given the tar- get distribution density P(=), HMC views as position and introduces an auxiliary momentum q, then each state is described tion of the Hamilton's 's Equation. After the Markov chain has burned in, the position sequence {z,] samples from the target distribution. Compared to the vanilla MCMC method (e.g., Metropolis algorithm), HMC can generate a succession of samples with less dependence meanwhile keep the rejection probability arbitrarily small, by making the evolution traverse long distance in the state space with fine-tuned small step size. These two propertics make it suitable for generating diverse outliers for eegularization-baee OOD detection. Please refer to Appendix C. for more details.
tuple (z, q) and new proposal (z", q²)) is generated via the evolu-
3

---

Published as a conference paper at ICLR 2025
ID Datasel
Fealure Embedding
Classification Branch
Hamiltonian Monte Carlo
Logtt
Softras
Protecye
CR
FC
Sample
Nidpoont
Hy Hyperspherical Embedding
Virtael Outieer
Backhone
Porjection Head
Marine Chain
Sampling Region
Restricted Region
Contrastive
OOD Discomment
Figure 3: Depiction of the HamOS training framework. We design a dual-head framework uti- lizing backbone for feature extraction. (1) The FC head preserves the initial ID classification efficacy; (2) The projection head transforms the feature embedding into a reduced-dimensional hy- perspherical space, where we explicitly generate outliers through Hamiltonian Monte Carlo utilizing our innovative OOD-ness estimation. The spherical space is shaped by ID contrastive loss and OOD discermment loss to enhance differentiation between ID and OOD data.
METHOD: HAMILTONIAN MONTE CARLO OUTLIER SYNTHESIS
This section details how HamOS synthesizes outliers using solely the ID training data and how outliers are utilized to regularize the decision boundaries for better ID-OOD differentiation.
3.1 OVERVIEW
As illustrated in Figure 3, our framework HamOS consists of three components: 1) feature em- bedding f X' + H that maps input æ into some higher dimensional feature space; 2) fully connected (FC) layer: f(x) + y for the original ID classification task; 3) a latent embedding: where MLP(.)) is multiple layer perceptron and norm(-) is the & normalization so that the output embedding is on the unit hypersphere. HMC also generates out- liers this hypersphere. Different from previous works (Du et al., 2022b; Tao et al., 2023) which simply add Gaussian noise to boundary samples, our method utilizes Markov chains explore significantly larger area in the hyperspherical space. By setting the potential energy the estimated OOD-ness density, the samples from the Markov chains HMC are diverse and representative, ensuring the trained model with great power distinguish between ID and OOD data.
f(x) (oom(MMP((FF..
3.2 SYNTHESIZING OUTLIERS VIA HAMILTONIAN MONTE CARLO
Estimating OOD-ness via the distance to the k-th nearest neighbor. Although we have only ac- cess to the ID data, we can measure the OOD-ness ness for any sample, i.e., a quantitative characterizing the likelihood that a sample is OOD rather than ID (Dang et al., 2015; Bergman et al., 2020; Sun al., 2022), by leveraging its distance to the ID data. Specifically, for any embedding z lying on the unit hypersphere and the ID data embedding sets Zi, Zc. one for each class, the OOD-ness for z w.r.t. Z, defined as the Euclidean distance:
pOOD (z; 2e) = -|==1-000 22(x)ll),
where Ze(k) is the k-th nearest neighbor in Z, for As the feature embeddings are projected to the unit hypersphere, our primary goal is to synthesize outliers lying between ID clusters, which are critical for shaping the ID boundaries and exposing the model critical potential OOD scenarios. Consequently, for each pair of ID classes Z,, and Z,, we measure the OOD-ness for by
pOOD (z; 2.) + pooD (z; 2,)
(2OD Z,: 2,)
and run HMC with the potential energy as
OOOD (z; Zw, 2.) = - log pOOD (z: Zu, 2x) = -log Σ
pOOD (2; Z,) + constant.
(2)
4

---

Published as a conference paper at ICLR 2025
1.7
155
-0.4
Dimension
147
1.0
OOD Score
-0.6
0.88
500T
-0.5
-0.8
1A7
1de
-1.2
-1.6-
0.05
0.2
2.00
-17
HamOS
NPOS
041
0.11
0.34
0.15
Dimension 0
Synthesis Round
ID Probability
(a) Depiction of OOD-ness Density
(h) Round-wise Analysis
(c) Synthesized Outliers
Figure 4: HamOS synthesizes outliers of varying levels of OOD-ness: (a) OOD-ness density with illustrated synthesis process; (b) the OOD distribution of the virtual outliers at each synthesis round with different step sizes; (c) the probabilities and OOD scores of the synthesized outliers.
Figure 4(a) depicts the geometry of the OOD-ness estimation, showing that the OOD-nest density low between ID clusters and high in the peripheral region. More analysis the design of the OOD-ness density estimation is provided Appendix G.
Synthesizing outliers by OOD density estimation via HMC. HMC incorporates auxiliary mo- mentum q sampled from Gaussian distribution for describing the state as (z, q). The Hamilto- nian (Duane al., 1987) is formulated as [2, q) g (z) | /2 where the potential energy U (2) is defined in Eqn. (2), thus the virtual outliers collected along the Markov chains obey the marginal distribution pooD
Given the potential energy, the Leapfrog discretization is typically adopted as the numerical ap- proximation of the evolution of Hamilton's Equation (Hamilton, 1833). For HamOS, the target distribution lies on the unit hypersphere, thus the Spherical HMC (Lan et al., 2013) is applied and the Leapfrog update on the unit sphere turns to:
ạ (2+1/22) q
_ [I.
E
(z (z(0) T) 210 200 GOOD (2)00, 20))
q
(c+1/2)
(f+ 0(0 cos (1lg (2+122) 112c)
| l|2 sin (c+1 1|2))
(3)
((++1/2) + (0 (6+1/2) sin (11g ((11/2) 112) ) + q (c+1/2) cos (11 ((112)+))))
((+1) = (0+1))) - C (Ia 1e+ (2(+1)) uun ooo (2(+1)))
where l is the Leapfrog timestep, is the step size. Eqn. (3) ensures candidates lie on the unit sphere. According to the Hamilton's Equation (Hamilton, 1833), the energy function (:00(()) must be differentiable ensure the solvability, with the gradient of the energy function calculated as:
v (000 (z; Zu, 2x)) = exp (-U000 (z; Zu, 2x))
Zu(k)
+
=b(k)) || )
(4)
Zu(iill2
The primary goal of synthesizing outliers via HMC is to generate outliers that represent diverse levels of OOD-ness, as well as spanning an extensive region in hyperspherical space, without neces- sitating the convergence to the target distribution. Given a pair of ID clusters Z,, and Z,.. the midpoint of the two clusters b = Iuula employed as the initial state a Markov chain due to its relatively low OOD-ness density, where th and μ4 are the prototypes of the two ID clusters. Fig- ure 4(a) shows the midpoint has meager OOD-ness density value. We provide round- wise analysis in Figure 4(b) with different step size where outliers with varied levels of OOD-ness are generated each round, exhibiting a broad range of OOD characteristics, also illustrated in Figure
Rejecting erroneous outliers located within ID clusters. HMC performs the Metropolis- Hastings (Metropolis et al., 1953; Hastings, 1970) acceptance step to ensure that the newly gener- ated sample possesses a reduced energy value. However, the design of potential energy of OOD-ness neglects the ID likelihood, potentially resulting in false outliers conflated with ID embeddings. Con- sequently, we introduce hard margin in the acceptance step in HMC. We utilize the kernel density estimation (KDE) (Rosenblatt, 1956) with von Mliss-Fisher kernel (Mardia & Jupp. 2009; Fisher, 1953) to approximate the ID probability, detailed in Appendix C.2. The hard margin 5 is applied to
5

---

Published as a conference paper at ICLR 2025
Algorithm Hamiltonian Monte Carlo Outlier Synthesis (HamOS)
Input: ID training embeddings: 2, number of ID classes: C, synthesis rounds: R, Leapfrog steps: L, step size: 6, number of adjacent ID clusters: Nadj, k-th nearest neighbor: k, hard margin: ò
Output: a batch of virtual outliers 2
1: for class 1, C do
2: Calculate the initial points bej TR,+p, where μ4; is the centroid of Z,, and
Metu
3: 1 adj indexes the closest IID clusters from 2. measured by cosine similarity.
4: Set the current position
bej.
5: Calculate the hard margin threshold t_ as Eqn. (5).
6:
for synthesis round r = 1. ... R do
7:
Initialize z(4) at current position z.
8:
Sample a new momemtum g( ~ (O)
Set +- 20) (x)) 1
9:
10:
Calculate H (z(1), )) = 0OOD (1) + k(q
11:
for (= 1, L do
12:
((+11/2) q(() TT - 210 (T) 2e U0OD())))
13:
((+1) = =0 cos (1 (++1/2) (2c) + 11 क sin (1g
((ti/a)
14:
((+1/22) + -2 22( |1 (1+1/2)| (1 (8- (6-1/2) 22c) + (t+11 cos (lgg (e+112)) (2c).
15:
g(++l) = =q
(++1/2)) 1/2) - 2 [[I - ((++)) ((+1)) V 200 U00D (2 (z(11))
end for
17:
Calculate H(z(4+3) , ((++))) = (000(2( + K ((g
for the proposed state.
Calculate the acceptance probability a exp - H(z(L+1), ( ((++)) + H22 (1) (()))
18:
19:
Accept (L++)) as current position z according to and - log max, Pib ( (z(1+11) )) > t...
20:
end for
21:
Gather the sequential positions in the R rounds as ZoOD.
end for
the log-likelihood of the initial point b to form a lower threshold, defined as follows:
L_ log max p!D (bu,v) 6.
(5)
where the estimated probability Pf is defined in Eqn. (11). The hard margin will block the new proposed point if has much higher ID probability than the initial point. Figure 4(c) shows that HamOS generates outliers with diverse OOD scores and low ID-likelihood.
We sample multiple Markov chains to form a mini-batch outliers, parallelly generating outliers starting from midpoints w.r.t. the nearest Natj ID clusters for each ID class. The outlier batch encom- passes a broad range potential OOD scenarios, covering a massive area in hyperspherical space. We adopt the Spherical HMC (Lan et al., 2013) for sampling the unit sphere. The pseudo-code is provided in Algorithm 1, with the whole training pipeline displayed in Algorithm 2 in Appendix D.
3.3 TRAINING WITH SYNTHESIZED OUTLIERS
The model is regularized jointly with ID training data and synthesized virtual outliers to produce em- beddings with enhanced ID-OOD discernibility. The virtual outliers reside between ID clusters where the model is not optimized. Analogous to maximum likelihood estimation (MLE), our goal is to minimize the likelihood of being assigned to any ID classes for the synthesized outlier data:
(CIFAB-10)
(SVHEN)
Virctal
Epuch
2.60
Epoch
11.79
Deasit
Epoch
Epoch
1o
8.13
9..30
Eporh 18
60.34
Epods 26
20
5.67
arg min I plu (Kjs 13)
OOD Score
Figure/ 5: OOD performance improved
where μ4 is the ID prototype class j updated in exponeniaa-moving-avrngg (EMA) manner, z; is
continuously with synthesized outliers.

---

Published as a conference paper at ICLR 2025
Table 1: Main results with CIFAR-10/1ON as ID dataset (%). Comparison with competitive OOD
detection baselines. Results are averaged across multiple OOD test datasets with multiple runs.
Methods
CITAR 10
CIEAR 100
PPR95
AUROCT
AUPRT
ID-ACCT
FPR951
AUROCT
AUPRI
ID-ACCt
Post hoc Menhods
1o.as
91.101o.m
81.70.5..2 15.02
10.17
59.78
11.25
66.86
76.65 .
ODIN
56 .8.0
85.7024.17
70.08=11..44
95. 17 17.0.11
63.19e2.51
78.01=1.2
65, 15.20=2..0
76.69
EBO
41. 41.854:0.00
91.7921.54
79.70
95. 17
60.8621.85
78.3221.31
78.32
06.7321
76 69
KNN
22.86
02.9810..r
88 88.7410.77
95. 17.101
56.9812.00
81.01
70.601a.0
76.69_o
ASHI
54.22
16.37
12.33130..0
85. 15.1010.14
66.8410.57
77.14.1.12
62.24
76.20.o.85
Scale
63.18
71.74-16.24
63. 23.00-22.5a
15.15-
69.27
77.25-11.0
11.42
76.69-0.24
Relation
26.28-1.05
92.31+0045
86.75
95. 95.17=0.11
59.61122.4
79.69+1.05
68. 08.76
76.69-0
Regutavization Aossed Mathods
CSI
21.22
93.7732...3
89. 89-74 os
02.03
69.34
73.46.00.77
61.37.0 75
61.75...11
SSD+
18. 1.20
94.85+00.57
90.88.00.85
B33 83.95=0..5
54.083+1.02
801.6400.00
89.73ee1.0
75.633=0.99
KNN+
19.68=1.86
91.41+0.0
90. 90.1640.00
83.79=0.55
61.25+0.5t
78.24+0.05
06.6120.58
72.18=0.as
VOS
42.37
91.4225.8
79. 79.16
9G 06.055...0 os
58.5521
A0
08.33.2161
74.7120.07
CIDER
16.28_0.os
95.701n.an
92. 02.3362..0m
93.98
49.64.1.. 1.50
81.77 an
13.2221.19
75. 76.001.0.4
NPOS
14:39 10.07
96.61 1o.
93.35_...7
83.95 10.13
11.41
81 81.022..m
72.4911.54
74.53 Lo.02
PALM
32. 32.25:44114
90.54:1.46
84.44 84.44:22.1
B33 833.9332..0
13...77
79. 29.95
70.21
74.67 so.as
HanOOStour))
10. 10.48
97.11++.20
94.94=0..s
#4.67
46.68+1.44
83.64-0.64
75.52-1.80
76.12
the synthesized embedding of the outlier batch of size M. Specificly. the objective function is equivalent to minimizing the following OOD discernment loss in log-likelihood form:
1
A
1
exp (z (3/T)
LOOD-dise = M
log
C
Σ exp (2] p1/T)
1=1
where - = 1/x j is the temperature. Since highly distinguishable ID representations are crucial for OOD detection (Tack et al., 2020; Schwag et al., 2021; Ming et al., 2023b; Tao et al., 2023), we leverage the contrastive learning objective for shaping compact ID boundaries, Compatible with multiple contrastive losses (Chen et al., 2020; Khosla et al., 2020; Ming et al., 2023b) (shown in Table 12), HamOS adopts CIDER loss (Ming et al., 2023b) defaultly. We train with Cross-entropy loss on the classification head simultaneously. Our training objective is formalized as follows:
Lamoss LCF + LID.con + XaCoon-iie:
where A is the trade-off co-efficient to balance LOOD-dise and the other two terms. In Figure 5, we display the score distributions along the training phase. showing that the synthesized outliers span wide range of OOD scores which help broaden the gap between ID and OOD data.
3.4 INFERENCE-TINE OOD DETECTION
We adopt the KNN distance scoring function as (Sun et al., 2022) for OOD detection during time:: (z(=";k)
1[-1|=* (k)) l2 3}. where Z {z1, =2, , =N is the ID embedding set, (R) denotes the k-th nearest neighbor in Z w.r.t. 2², and ( is the indicator func- tion. The threshold /9 is chosen to correctly classify large part of ID data as ID (i.e., TPR=95%).
4 EXPERIMENTS AND ANALYSIS
This section consists of an extensive empirical analysis of the proposed HamOS. First, we provide the experimental settings in detail (in Section 4.1); second, we report the performance compari- son of HamOS with series of state-of-the-an basclines (in Section 4.2); third, present further discussions and ablation studies to clucidate the intrinsic mechanism of HamOS (in Section 4.3).
4.1 SETUP
We display the critical parts of the experimental settings for the main performance comparison. For more details of experiment setups, please refer to Appendix D.2.
Datasets. Following the common practice for benchmarking the OOD detection (Zhang et al., 2023b), we use CIFAR-10) CIFAR- 100 (Krizhevsky & Hinton, 2009) and ImageNet-1k (Deng et al., 2009) as ID datasets, and adopt series of datasets as OOD testing data. For CIFAR ID datasets, we use MNIST (Deng, 2012), SVHN (Netzer et al., 2011), Textures (Cimpoi et al., 2014), P1aces365 (Zhou et al., 2017), and LSUN (Yu et al., 2015) as OOD testing data; for ImageNet- -1k,, we use iNatualist (Van Horn et al., 2018), Textures (Cimpoi et al., 2014), SUN (Xiao al., 2010), and P1 665 (López-Cifuentes al., 2020) OOD testing data.
7

---

Published as a conference paper at ICLR 2025
Table 2: Results with ImageNet- K as ID dataset (%). Comparison with regularization- based OOD detection baselines. Results are averaged across multiple runs.
Marbwds
ID-ACC
D7R49,
AUTRECI
AE/PT
119445
AUBOOC
119895
AUROOC
[1900]]
AL/ER
[1/W99)
ALTROCT)
54.32
77:35
00:00
14
$5.15
57.30
03.01
22.00
50.51
67-03
00.50 0.
01.23
61.00 11.88
$1.60
07.50 10:83
67-26
KNN-
76-36
00.60
35
$1-77
E6.58
03.53
71 (//0))
83:13
06. 57
90 90
0/:03
08. 03.30
53.58
00.3 10
26.94
CIDER
18.
$1.31
10:15
$3.00
13: 28
87 1/-92
15:33
42
20: (1)
03:03
09.81
$6.65
04.00
73.13
PALM
18.04
55,85
06.50
25,
11.31
18.38
01.01
11.00
14.32
43.00
00, 211
00.02
50. 50
14.33
14.30
HanOS/vani
54.06
85.81
14.50
38.56
20.01
09.02
$3.90
02.11
15.05
41.84
94
44.09
85.00
98.22
76.30
Training scheme. We assume the existence of pretrained models and constrict the training phase to the fine-tuning scenario. which is more feasible and practical in real-world cases. We adopt pretrained ResNet-33 (He et al., 2016) for CIFAR datasets and ResNet-50 (He et al., 2016) for ImageNet- IK dataset and use stochastic gradient descent (SGD) (Bottou, 2010) to fine-tune for 20 epochs using the ID training datasets solely, with drop out rate, momentum, and weight decay set to 0, 0.9, and 1.0 x 10-1 respectively. The learning rate is initialized as 0.01 and decayed to 0 using cosine annealing (Loshchilov & Hutter, 2017). We synthesize a batch of outliers at each iteration.
OOD detection baselines. We evaluate the performance of the proposed HamOS against other com- peting baselines across two research lines: (1) for post-hoc methods, we adopt MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), EBO (Liu et al., 2020), KNN (Sun al., 2022), ASH (Djurisic et al., 2023), Scale (Xu et al., 2024), and Relation (Kim et al., 2023); (2) for regularization-based methods, we adopt CSI (Tack al., 2020), SSD+ (Sehwag et al., 2021), KNN+ (Sun et al., 2022), VOS (Du et al., 2022b), CIDER (Ming et al., 2023b), NPOS (Tao et al., 2023), and PALM (Lu et al., 2024). The post-hoc methods are evaluated on pretrained models, whereas the regularization-ben based methods are evaluated after fine-tuning the pretrained models.
Evaluation metries. We employ the following three prevalent metrics to assess the performance of OOD detection: (1) the Area Under the Receiver Operating Characteristic curve (AUROC) (Davis & Goadrich, 2006; Fawcett, 2006): (2) the Area Under the Precision-
curve (AUPR) (Manning & Schutze, 1999); and (3) the False Positive Rate (FPR) at 95% True Positive Rate (TPR), abbreviated as FPR95 (Liang et al., 2018). The AUROC quantifies the diagnostic efficacy of level-set estimation; the AUPR is especially suited for imbalanced scenarios; and FPR95 is a sensitive metric at high recall levels. We also report the classification accuracy on the ID datasets (ID-ACC) to evaluate the model's performance on the original classification task. The mean results across multiple runs with standard deviation are reported to demonstrate the stability of HamOS.
4.2
MAIN RESULTS
HamOS outperforms competitive baselines. In Table 1, we report the averaged results using five OOD testing datasets for the baselines and the proposed HamOS. The results show that the proposed HamOS can remarkably improve the OOD detection capability of the pretrained model. First, for the contrastive training-based based methods (i.e., CSI, SSD+, KNN+, CIDER, and PALM), HamOS can additionally simulate representative supervision signals of OOD data that lies between ID classes, hence facilitating learning more proper representation passes the CIDER by 35.63% and 5. 5.96% in FPR95, respectively, on CIFAR-10 and CIFAR- 100. Second, as for the synthesis-based methods (i.e., VOS and NPOS), the proposed HamOS generates outliers according to OOD-ness estimation rather than applying Gaussian noise to ID embeddings in low-likelihood regions, resulting in more diverse and representative outliers. Specifically, HamOS surpasses NPOS by 2..77% and 9.20% in FPR95 on CIFAR- 10 and CIFAR- 100 respectively. In general, the regularization based methods can outperform the post-hoc host-ho methods by introducing more specific training, as the OOD metrics show, but at the expense of performance on the original classi- fication task (i.e., lower ID-ACC). Nonetheless, HamOS achieves high OOD detection performance while preserving competitive ID classification performance. The detailed results for each OOD test dataset are in Table 5 in Appendix F. We also present the results on hard OOD benchmarks (Zhang et al., 2023b) in Table 6, in Appendix F to validate the efficacy of HamOS under hard scenarios.
the unit sphere. Specifically, HamOS sur-
HamOS is superior under the large-scale scene. We also assess the efficacy of HamOS on the large-scale ImageNet- IK dataset. Table 2 presents the results compared with regularization-art baselines. As the results suggest, our proposed HamOS outstrips all the regularization- base- lines on average, improving the FPR95 by 2.98% compared with VOS. We also provide evaluation
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation results on compactness and separation (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
Method
ID-OOD Separationt
Inter-class Dispersion
Intre-class Compactness]
MNIST
SVHN
Textures
Places365
LSUN
Averuged
SSD+
18. 20.79
22 68
19.75 48
19. 19.44ao.1 to
22. 75.0.0
20. 30.62=0.25
53. 700000
15. 0..c
CIDER
56. 56.0721164
68 08.33320..
68. 00 47
54.77a0. as
37 37.982a.4
59: 0..44..
96.51 14
34. 34.66
NPOS
66 64.87
85, 85.7011.1
65.42
61.7010.41
72.8810.05
70.51
90.57
31 92:
PALM
30.09 -1.0
56. -
19.92e1.09
55. 55.06+0.40
31.37=0.17
53. 533.12=0.1
93. 93.72=022
40. 01.88--0.3
HamOSroors)
72.78.44.00 m
22 42
71. 71.50.4*
78.13_2 wm
83.8724.01
20.70..mmm m.
86.492445
34.68gg..
60
EIPAR-10
CIAAB-10
Mahalanobis KNN
30
Texeline
Namos
50
11.08
45.55
21:23
FPR95
40
FPROS
FPR95
300
15
L656
15:49
16:00
10:90
G0-
10-
16:2
HNC
MALA BMEIC
41
SUDA
CIOSSA
RPRS
PALM
HsmoS
C8
SIoCLS
SupCon
CIDER
Sampling Algorithms
ID Contrastive Loss
(a) Different sampling algorithms
(b) Different distance metrics
(c) Different contrastive losses
Figure 6: HamOS is general framework compatible with different sampling algorithms, scor- ing functions, and ID contrastive losses: (a) HamOS can consistently achieve superior perfor- mance with variants of HMC with Random Walk as the baseline;(b) HamOS outstrips the baselines with both Mahalanobis and KNN distance as scoring functions; (c) HamOS displays consistent en- hancement equipped with different ID contrastive losses.
on post-hoc methods in Appendix F. HamOS demonstrates enhanced performance when integrated with diverse post-hoc approaches (e.g., attaining 25.55 %) in FPR95 with Scale scoring function), shown in Table 11 in Appendix F; the results show that the enhancement is consistent with various post-hoc methods, suggesting that HamOS can improve the intrinsic capability of the pretrained model for OOD detection. HamOS maintains the ID classification efficacy as well at the same time.
HamOS helps to learn distinguishable representations. Next, we quantitatively analyze the hy- perspherical embedding quality for ID-OOD separability and the differentiation of ID) clusters. We leverage the cosine similarity to reflect the ID-0OD separation, ID inter-class dispersion, and ID intru-class compactness, defined in Eqn. (14) in Appendix C.4. The ID-OOD Separation and the inter-class Dispersion are better with higher values, while the intra-class compactness is better with smaller values. Table 3 presents the results in angular degree form for better readability. The results show that HamOS leads to higher ID-OOD separation, reflecting superior OOD detection perfor- mance compared to the baselines. HamOS displays an averaged 11 11.62% improvement compared with NPOS. As for the inter-class dispersion and intra-class compactness, though HamOS doesn't 't achieve distinguishable values, it can achieve competitive ID classification accuracy by maintaining the original classification head, as reported in Table indicating that the inter-class dispersion and intra-class compactness are not straightforward metrics for evaluating the ID-OOD differentiation. For example, CIDER outstrips SSD+ by large margin on the inter-class dispersion (in Table 3) while attaining approximately equivalent ID accuracy as SSD+ (in Table 1).
4.3 FURTHER DISCUSSIONS AND ABLATION
HamOS demonstrates versatility with various sampling algorithms. We substitute the original HMC algorithm in HamOS with other variants to validate the proposed framework's versatility. Fig- ure 6(a) shows that the proposed framework HamOS is consistently effective with various sampling algorithms compared with the Random Walk algorithm density estimation as the pivotal component. The full averaged results are in Table 9, where we also report the time required to synthesize mini-batch of outliers in milliseconds, reflecting the time efficiency of different HMC variants. Please refer to Appendix C.3 for details of variants of HMC.
baseline, highlighting the OOD-ness
HamOS is robust to different scoring functions. To further demonstrate the robustness of the proposed HamOS, we ablate with different scoring functions. First, we compare HamOS with regularization-bass methods using KNN distance (Sun et al., 2022) and Mahalanobis distance (Lee

---

Published as a conference paper at ICLR 2025
ID ICEPAR (SVEN) NPOS Gutlen
(SVIL)) Tamos Outliers
VOS Dutliens
Figure 7: t-SNE (van der Maaten & Hinton, 2008) Visualization: (a)VOS; (b)NPOS; HamOS.
al., 2018b). The results on CIFAR-100 in Figure 6(b) show that our proposed HamOS achieves competitive performance with the two distance metrics. The full averaged results are in Table 10. Second, we evaluate HamOS with multiple scoring functions. Since HamOS retains the classifi- cation head, it is compatible with other post-hoc detection methods. We report the results when equipped with other scoring functions Table 11 in Appendix F. The results demonstrate that HamOS enhances the intrinsic detection capability regardless of the scoring function employed.
HamOS demonstrates consistent improvements using different ID contrastive losses. Recall that proposed HamOS synthesizes virtual outliers utilizing solely the IDD feature embeddings, which are regularized with ID contrastive losses. We validate the efficacy of HamOS additionally using two advanced ID contrastive training losses, i.e., SimCLR (Chen et al., 2020; Winkens al., 2020) and SupCon (Khosla et al., 2020), with Cross-entropy loss (Tao et al., 2023) as the baseline. In Figure 6(c), we report the FPR95 on CIFAR-10. shows that our proposed HamOS can consis- tently enhance the detection capability compared to training singly with the ID contrastive loss. The full averaged results are in Table 12 Appendix F with additional experiment details.
HamOS displays exceptional efficiency. Though the HMC typically incurs more computational cost than simpler Gaussian sampling. HamOS still keeps the time cost comparable to VOS and NPOS and even surpasses NPOS on ImageNet- K, resulting from the high quality of sampling and highly efficient implementation. Refer to Appendix F.1 for detailed analysis.
Ablation studies loss weight Xa, k value for calculating OOD ness density, hard margin 6, Leapfrog steps L, step size e, number of nearest clusters Nudj, and synthesis rounds We provide thorough ablation studies to elucidate the effect of various factors in HamOS framework in Appendix F.2. The ablation demonstrates the robustness of HamOS for some hyperparameters and the tuning preference for others. Detailed implementation of HamOS is provided in Appendix D.
4.4 ADDITIONAL VISUALIZATIONS
We utilize t-SNE (van der Maaten & Hinton, 2008) to intuitively compare the synthesized outliers by VOS (Du et al., 2022b), NPOS (Tao et al., 2023), and our HamOS in Figure 7. The ID (i.e., CIFAR- 10) and OOD (i.c., SVHN) features are obtained by the projection head with l normalization. HamOS can synthesize outliers spanning a broad range of areas the feature space between ID) clusters, hence exposing the model to miscellaneous potential OOD scenarios. On the other hand, voS might generate erroneous outliers that reside in ID clusters; NPOS might generate gathered outliers that lack diversity. We provide additional discussion with visualizations in Appendix G.
CONCLUSION
In this paper, we propose a novel framework, HamOS, for synthesizing virtual outliers via Hamil- tonian Monte Carlo. Diverse and representative outliers are sampled from Markov chains which traverse broad area in the hyperspherical space, facilitating representation learning by various OOD supervision signals between ID classes. HamOS adopts Y-head architccture that optimizes CE loss for ID classification, and ID contrastive loss and OOD discernment loss for better ID-OOD separability the unit sphere. Extensive empirical analysis demonstrates the superiority of HamOS against competitive baselines on both standard and large-scale benchmarks. We conduct sufficient ablation studies reveal the mechanism of HamOS under various circumstances. We hope this pa- per provides an inspiring perspective for future research on synthesizing outliers for OOD detection.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Major Project (2022ZD01 (4803)))148003 the Natural Science Foundation of Wuhan (2023010201020222),, and the Major Program (JD) of Hubei Province (2023BAA024).
REFERENCES
Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, pp. 1572,157,, 2016.
Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. In Proceedings of
the the 33rd IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Julian Besag. Comments on "representations of knowledge in complex systems" by u. grenander and mi miller. Journal of the Royal Statistical Society Series B, 566591-552):44 1994.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of the 19th International Conference on Computational Statistics, pp. 177-186. Springer, 2010.
Anthony Chen, Huanrui Yang. Yulu Gan, Denis A Gudovskiy, Zhen Dong, Haofan Wang, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Split-ensemble: Efficient ood-aware ensemble via task and model splitting. In Proceedings of the 41st International Conference on Machine Learning, 2024a.
Jiankang Chen, Tong Zhang, Wei-Shi Zheng, and Ruixuan Wang. Tagfog: Textual anchor guidance and fake outlier generation for visual out-of-distribution detection. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, 2024b.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Proceedings of the 24th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, pp. 430- 445, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffre Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607, 2020.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the Computer Vision and Pattern Recognition, 2014.
Taurus T Dang, Henry YT Ngan, and Wei Liu. Distance-based k-nearest neighbors outlier detection method large-scale traffic data. In Proceedings of the 2015 IEEE International Conference on Digital Signal Processing, pp. 507-510. IEEE, 2015.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and curves. In Proceedings of the 23rd International Conference on Machine Learning, pp. 233-240,, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, Icee, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6): 142, 2012.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribuute detection. Proceedings of the 11th International Conference on Learning Representations, 2023.

---

Published as a conference paper at ICLR 2025
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. Towards unknown-aware learning with virtual outlier synthesis. In Proceedings of the 10th International Conference on Learning Rep- resentations, 2022a.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. Proceedings of the 10th International Conference on Learning Representations, 2022b.
Xuefeng Du, Yiyou Sun, Jerry Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with
diffusion models. Advances in Neural Information Processing Systems 36, 2023.
Simon Duane, A.D. Kennedy, Brian J. Pendleton, and Duncan Roweth. Hybrid monte carlo. Physics Letters B, 195(2):21--22,, 1987.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the 34th IEEE Conference on Computer Vision and Pattern Recog- nition Workshops, pp. 12873-12883, 2021.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625 630, 2024.
Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27(8):6618874 2006.
Ronald Aylmer Fisher. Dispersion on sphere. Proceedings of the Royal Society of London, Series
A. Mathematical and Physical Sciences, 217(11300:295-305, 1953.
Mark Girolami and Ben Calderhead. Riemann Manifold Langevin and Hamiltonian Monte Carlo Methods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 73(2):123- 214, 03 2011.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-11)) 2020.
Matej Greie, Petra Bevandić, and Sinisa Segvic. Dense open-set recognition with synthetic outliers generated by real nvp. In Proceedings of 16th International Conference Computer Vision Theory and Applications, pp. 133-143, 2021.
Tobias Groot and Matias Valdenegro Toro. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. Proceedings the 4th Workshop on Trustworthy Natural Language Processing, 2024.
Xiaoyi Gu, Leman Akoglu, and Alessandro Rinaldo. Statistical analysis of nearest neighbor methods for anomaly detection. In Advances in Neural Information Processing Systems 32, 2019.
William Rowan Hamilton. On General Method of Expressing the Paths of Light, and of the Planets, by the Coefficients of a Characteristic Function. PD Hardy, 1833.
W. Keith Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57(1):97-109, 1970.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Florian Heidecker, Jasmin Breitenstein, Kevin Rösch, Jonas Löhdefink, Maarten Bieshaar, Christoph Stiller, Tim Fingscheidt, and Bernhard Sick. An application-driven conceptualization of corner cases for perception in highly automated driving. In Proceedings the 32nd IEEE Intelligent Vehicles Symposium, pp. 644-651. IEEE, 2021a.
Florian Heidecker, Abdul Hannan, Maarten Bieshaar, and Bernhard Sick. Towards comeer detection by modeling the uncertainty of instance segmentation networks. In Proceedings of the 25th International Conference on Pattern Recognition Workshops, pp. 361-374, 2021b.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. In Proceedings of the Sth International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In Proceedings of the 7th International Conference on Learning Reprexentations, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mosta- jabi, Jacob Steinhardt, and Dawn Song. Scaling distr--iistiiuution detection for real-world set- tings. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 8759-8773, 2022.
Mitch Hill, Erik Nijkamp, Jonathan Mitchell, Bo Pang, and Song-Chun Zhu. Learning probabilistic models from generator latent spaces with hat ebm. In Advances in Neural Information Processing Systems 35, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33, 2020.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. In Advances in Neural Information Processing Systems 34, 2021.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-709, 2005.
Wenyu Jiang, Hao Cheng, MingCai Chen, Chongjun Wang, and Hongxin Wei. DOS: Diverse outlier sampling for out-o--distribution detection. In Proceedings of the 12th International Conference on Learning Reprexentations, 2024a.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided OOD detection with pretrained vision-language models. In Proceedings of the 12th International Conference on Learning Representations, 2024b.
Hamidreza Kamkari, Brendan Leigh Ross, Jesse C. Cresswell, Anthony L. Caterini, Rahul G. Kr- ishnan, and Gabriel Loaiza- Ganem. A geometric explanation of the likelihood ood detection paradox. In Proceedings of the 41st International Conference Machine Learning, 2024.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems 33, 2020.
Jang- Hyyn Kim, Sangdoo Yun, and Hyun Oh Song. Neural relation graph: A unified framework for identifying label noise and outlier data. Advances in Neural Information Processing Systems 36, 2023.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Pro- ceedings of the International Conference on Computer Vision, pp. 813-822, 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report O, University of Toronto, 2009.
Shiwei Lan, Bo Zhou, and Babak Shahbaba. Spherical hamiltonian monte carlo for constrained target distributions. In Proceedings of the 30th International Conference on Machine Learning, volume 32, pp. 629-637, 2013.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibratee classifiers for detecting -ut-o--distiibutton samples. Proceedings of the 6th International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of didtr---itttibttion samples and adversarial attacks. In Advances in Neural Information Process- ing Systems 31, 20.
13

---

Published as a conference paper at ICLR 2025
Sculbi Lee, Jihyo Kim, and Sangheum Hwang. Reflexive guidance: Improving oodd in vision-language models via self-guided image-adaptive concept generation. arXiv preprint arXiv:2410.14975. 2024.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of--istributiin image detection in neural networks. In Proceedings of the 6th International Conference on Learn- ing Representations, 2018.
Jasper Linmans, Jeroen van der Laak, and Geert Litjens. Efficient out- out-o--distribution detection in digital pathology using multi-head convolutional neural networks. Medical Imaging with Deep Learning, pp. 465-478. PMLR, 2020.
Litian Liu and Yao Qin. Fast decision boundary based of the s1 International Conference on Machine Learning, 2024.
detector. In Proceedings
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-basee out-of-distribution detec- tion. In Advances in Neural Information Processing Systems 33, 2020.
Yibing Liu, Chris XING TIAN, Haoliang Li, Lei Ma, and Shiqi Wang. Neuron activation cov- erage: Rethinking out-of-distribution detection and generalization. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Alejandro López-Cifuentes, Marcos Escudero- Escudero-Vinolo, Jesús Bescós, and Álvaro Garcfa-Martin Semantic-aware scene recognition. Pattern Recognition, 102:107256, 2020.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Pro- ceedings of the 5th International Conference on Learning Representations, 2017.
Haodong Lu, Dong Gong, Shuo Wang, Jason Xue, Lina Yao, and Kristen Moore. Learning with mixture of prototypes for out-of-distribution detection. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Prasanta Chandra Mahalanobis. On the generalised distance statistics. In Proceedings of the 12th National Institute Science of India, pp. 49-55, 1936.
Christopher Manning and Hinrich Schutze. Foundations of statistical natural language processing. MIT press, 1999.
Kanti Mardia and Peter Jupp. Directional statistics. John Wiley and Sons, 2009.
Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1092 1953.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: dustr--dist detection with posterior sampling. In Proceedings of the 39th International Conference on Machine Learning, pp. 15650 15665, 2022.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How exploit hyperspherica embeddings for out-of-distribution detection? In Proceedings of the 1 11th International Conference on Learning Representations, 2023a.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How exploit hyperspherical embeddings for out-of-distributtio detection? In Proceedings of the Ish International Conference on Learning Representations, 2023b.
Hossein Mirzaei. Mohammad Jafari, Hamid Reza Dehbashi, Ali Ansari, Sepehr Ghobadi, Masoud Hadi, Arshia Soltani Moakhar, Mohammad Azizmalayeri, Mahdich Soleymani Baghshah, and Mohammad Hossein Rohban. RODEO: Robust outlier detection via exposing adaptive out-of- distribution samples. In Proceedings of the 4 st International Conference on Machine Learning, 2024.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of ma- chine learning models. In Proceedings of the 37th International Conference on Machine Learning, volume 1 119, pp. 6950-6960 2020.
14

---

Published as a conference paper at ICLR 2025
Atsuyuki Miyai, Jingkang Yang. Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. arXiv preprint arXiv: 2403. 20331, 2024.
Felix Moller, Diego Botache, Denis Huseljic, Florian Heidecker, Maarten Bieshaar, and Bernhard Sick. Out-of-distribution detection and generation using soft brownian offset sampling and au- toencoders. In Proceedings of the 34th IEEE Conference on Computer Vision and Patlern Recog- nition Workshops, pp. 46-55,200 2021.
Radford M Neal. Probabilistic inference using markov chain monte carlo methods. Department of Computer Science, University of Toronto Toronto, ON, Canada, 1993.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In Advances in Neural In- formation Processing Systems 24 workshop on deep learning and unsupervised feature learning, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition, pp. 427-436, 2015.
Sangha Park, Jisoo Mok, Dahuin Jung, Sachyung Lee, and Sungroh Yoon. On the powerfulness of textual outlier exposure for visual ood detection. In Advances in Newral Information Processing Systems 36, 2023.
Omead Pooladzandi, Pasha Khosravi, Erik Nijkamp, and Baharan Mirzasoleiman. Generating high fidelity synthetic data via coreset selection and entropic regularization. In Advances in Neural Information Processing Systems 36 workshop on synthetic data for empowering ML research, 2023.
Masoud Pourreza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi, and Mohammad Sabokrou. G2d: Generate to detect anomaly. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 2002-2011, 2021.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates of Density Function. The Annals of Mathematical Statistics, 27(3):832 - 837, 1956.
Vikash Sehwag, Mung Chiang, and Prateck Mittal. Ssd: A unified framework for self-supervised outlier detection. In Proceedings of the 9th International Conference Learning Representa- tions, 2021.
Liang Shi, Boyu Jiang, and Feng Guo. Sevlm: a vision-language model for driving safety critical event understanding. arXiv preprint arXiv: 2410.00982, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribee detection. In Proceedings of the 17th European Conference on Computer Vision, pp. 691-708 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: tions. In Advances in Neural Informatinn Processing Systems 34, 2021.
detection with rectified activa-
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of--istribuut detection with deep nearest neighbors. In Proceedings of the 39th International Conference Machine Learning, volume 162, pp. 20827-20840, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems 33, 2020.
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parament outlier synthesis. In Proceed- ings of the 11th International Conference on Learning Representations, 2023.
15

---

Published as a conference paper at ICLR 2025
Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. Advances in Neural Information Processing Systems 33 workshop the machine learning for health, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):557- 2008.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition, pp-
7769,8778, 2018.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: citoof-distribution with virtual- logit matching. In Proceedings of the 35th IEEE Conference on Computer Vision and Pattern Recognition, pp. 4911-499 1-4920, 2022.
Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-disstibuttio detection with implicit outlier transformation. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng. Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In Proceedings of the 39th International Con- ference on Machine Leurning, pp. 23631-23644 2022.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learming, pp. 681-688, 2011.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, A. Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved out-of-distribution detection, arXiv preprint arXiv:2007.05566 2020.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485-3445 3492, 2010.
Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post-hoc out-of-distribution detection enhancement. In Proceedings of the 12th International Conference Learning Representations, 2024.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: 100..5000.00 03365, 2015.
Boxuan Zhang, Jianing Zhu, Zengmao Wang. Tongliang Liu, Bo Du, and Bo Han. What the input is expanded in ood detection? In Advances in Neural Information Processing Systems 37, 2024.
Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards -uito--distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5531-5540, 2023a.
Jingyang Zhang, Jingkang Yang. Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. In Advances in Neural Information Processing Systems 36 workshop the distribution shifts, 2023b.
Puning Zhao and Lifeng Lai. Analysis of knn density estimation. IEEE Transactions on Information Theory, 68(12):7971-79) 2022.
Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. Towards optimal feature- -haupng methods for distr--iistiibttinn detection. In Proceedings of the 12th International Conference Learning Representations, 2024.
16

---

Published as a conference paper at ICLR 2025
Haotian Zheng, Qizhou Wang, Zhen Fang, Xiaobo Xia, Feng Liu, Tongliang Liu, and Bo Han. -f detection learning with unreliable out-of-distribution sources. Advances in Neural Information Processing Systems 36, 2023.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. Journal of Vision, 17(10):99-3377 2017.
Armando Zhu, Jiabei Liu, Li Keqin, Shuying Dai, Bo Hong, Peng Zhao, and Changsong Wei. Ex- ploiting diffusion prior for out-of-distribus detection. Irish Interdisciplinary Journal of Science and Research, 08: 17 1-85 2024.
Jianing Zhu, Hengzhuang Li, Jiangchao Yao, Tongliang Liu, Jianliang Xu, and Bo Han. Unleashing mask: explore the intrinsic out-of-distribution detection capability. Proceedings of the 40th International Conference Machine Learning, 2023a.
Jianing Zhu, Geng Yu, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for dustrf-distribution detection via informative extrapolation. In Advances in Neural Information Processing Systems 36, 2023b.
17

---

Appendix
Contents
Reproducibility Statement
19
Related Work
19
B.1 OOD Detection
19
B.2
Score-based Markov Chain Monte Carlo Sampling
20
C
Additional Preliminaries
20
C.I Training with Auxiliary OOD Data
20
C.2 Modeling the ID Hyperspherical Distribution
................................
20
C.3 Hamiltonian Monte Carlo
C.4 Hyperspherical Evaluation Metric
22
D Detailed Implementation of HamOS
22
D.1 Realization of HamOS
22
D.2
Detailed Experiment Setups
23
Baselines and Hyperparameters
23
E.1
Post-hoe Baselines Regularization--asse Baselines
23
E.2
24
Additional Experiment Results
25
F.1
Effiacy Analysis of HamOS Additional Ablation Studies
25
F.2
31
Discussions
34
H Broader Impact
37
18

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY STATEMENT
Some important aspects are summarized as follows to facilitate reproducing the results in this paper:
Datasets. All datatsets adopted by this paper are publicly available, as is introduced in Section 1. For OOD detection evaluation, we ensure there is no overlap between OOD test data and ID training data. The data processing follows the common practice.
Assumption. The main experiments are set under the fine-tuning scenario where a well- trained model the original classification task is available, and the ID training dataset is also available for subsequent fine-tuning, We don't 't make any assumption of the existence of natural auxiliary outliers for fine-tuning, which is more general and practical scheme.
Experiment environment. All experiments in this paper are conducted for multiple runs on a single NVIDIA Tesla V100 Tensor Core with 32GB memory using Python version 3. 3.10.9. The deep learning environment is established using PyTTrch version 1.13.1 and Trochvision version O. 14.1 with CUDA 12.2 in the Ubuntu 18.04.6 system.
B RELATED WORK
B.1 OOD DETECTION
OOD detection is essential for constructing trustworthy machine learn- ing systems in open-world scenarios (Nguyen et al., 2015). One straightforward line is to devise powerful scoring functions a post-hoe fashion. As baseline methods, OpenMax (Bendale & Boult, 2016) introduces
Post-hoc OOD Detection
model layer for detecting unseen classes, and MSP (Hendrycks & Gimpel, 2017) leverages maximum softmax probability. The following works either focus on de- signing distinguishing scoring functions or trying to exploit feature-embeddding (Sun et al., 2021; Djurisic et al., 2023; Xu et al., 2024; Zhao et al., 2024; Liu & Qin, 2024), logits (Hendrycks et al., 2022: Wang et al., 2022), or probability outputs (Liang al., 2018: Liu et al., 2020; Sun & Li, 2022). Additionally, Mahalanobis distance-based confidence score (Lee et al., 2018b) utilizes class conditional Gaussian distributions differentiate OODs from IDs; GradNorm (Huang et al., 2021) utilizes information from gradient space; ANNNbased score (Sun et al., 2022) calculates euclidean distance from ID priors; ASH (Djurisic al., 2023) and Scale (Xu et al., 2024) explore the rectifi- cation of activation for better detection; Relation (Kim et al., 2023) inspects the structure of feature graph to detect OOD sample; NAC (Liu al., 2024) models the neuron activation coverage for discrimination. Nonetheless, post-hoc methods only achieve suboptimal performance with fixed overconfident models compared with regularization based methods.
Regularized OOD Detection Another viable approach for OOD detection is to incorporate reg- ularization techniques into the training process, with or without auxiliary OOD supervision signals. Contrastive training schemes (Winkens al., 2020; Tack et al., 2020; Sehwag al., 2021; Sun et al., 2022; Ming et al., 2023b; Lu et al., 2024) create distinguishable ID feature embeddings. Log- itNorm (Wei al., 2022) learns with constant normalization; UM(Zhu al., 2023a) inspects the middle training stage for better detection ability; ISH (Xu et al., 2024) integrates scaling tech- nique into the training phase; Split-Ensemble (Chen et al., 2024a) splits the classification task into sub-ones within a single model. Natural OOD supervision forces the model to produce uniform dis- tributions (Hendrycks et al., 2019; Zhang et al., 2023a) or higher EBO scores (Liu al., 2020) for outliers. Recent works propose advanced sampling strategies for auxiliary OOD data, among which ATOM (Chen al., 2021) greedily mines auxiliary OOD data, POEM (Ming al., 2022) adopts Thompson sampling to explore and exploit the auxiliary OOD data, while DOS (Jiang et al., 2024a) selects informative outliers from clusters.
Generative OOD Detection Without available natural OOD data, some studies propose to syn- thesize virtual outliers (Lee et al., 2018a; Moller et al., 2021: Pourreza et al., 2021; Du et al., 2023; Zheng et al., 2023) with generative networks (e.g., GooGGooffelo et al., 2020) or dif- fusion model(Ho et al., 2020)) or from feature space (Du et al., 2022a: Tao et al., 2023). There are methods that focus OOD detection on energy-based model, such Hat EBM (Hill et al., 2022) which incorporates the generator into its forward pass, thus enabling explicit modeling of the gen- erator's outputs. DOE (Wang al., 2023) transforms the original OOD samples into worse ones;
19

---

Published as a conference paper at ICLR 2025
DivOE (Zhu et al., 2023b) leverages information extrapolation for synthesizing more informative outliers; TOE (Park et al., 2023) generates textual outliers for CLIP-based OOD detection; Tag- Fog (Chen et al., 2024b) utilizes Jigsaw technique generate fake outliers; RODEO (Mirzaei et al., 2024) adversarially synthesizes hard outliers; NF-DT (Kamkari et al., 2024) generates outliers con- sidering both low-likelihood and local intrinsic dimension. However, these works suffer from limi- tations for requiring real outliers as anchors or high computational costs with seavy-parameterized generative models. Synthesizing outliers based solely on ID data has been underexplored hitherto, with existing works primarily focusing on applying Gaussian noise to get new outliers. This study investigates sampling outliers from Markov chains via Hamiltonian Monte Carlo for diversified, high-quality virtual OOD samples without the assumptions of the availability of natural outliers. Extensive empirical analysis demonstrates the efficiency and efficacy of the proposed framework.
B.2 SCORE-BASED MARKOV CHAIN MONTE CARLO SAMPLING
Markov Chain Monte Carlo (Metropolis et al., 1953) employs transition kernel to produce a sample that has the target distribution, including score- score-based based Markov Chain Monte Carlo, which leverages the gradient generate new points explicitly. Hamiltonian Monte Carlo (HMC) (Duane al., 1987) is an efficient algorithm that samples from high dimensional space with a high acceptance ratio. The Metropolis-adjusted Langevin algorithm (MALA) (Besag. 1994) is a variant of HMC with the Leapfrog step as 1. HMC can be extended to be Riemann Manifold HMC (RMHMC) (Girolami & Calderhead, 201 1) using a dynamic covariance matrix for sampling the momentum from Gaussian distribution. This paper utilizes the Spherical HMC (Lan et al., 2013) to synthesize outliers on the unit sphere and ablate on variants of HMC to demonstrate the generality of the proposed HamOS.
C ADDITIONAL PRELIMINARIES
This section provides more preliminaries for our outlier synthesis framework.
C.I TRAINING WITH AUXILIARY OOD DATA
To regularize the model for better detection capability, research line (Hendrycks et al., 2019; Chen et al., 2021; Ming et al., 2022: Zhang et al., 2023a) devise a paradigm that utilizes an auxiliary OOD dataset during training. among which outlier exposure (OE) (Hendrycks et al., 2019) is representative method. The learning objective can be generally formalized as follows:
min E Epp, [ECE(2 y; 0)] [Con 0)],
where À is the trade-off hyperparameter, Les is the Cross- Etoss-entropy loss, and Loe is the regularization with auxiliary outliers ã. Loe can be defined as Kullback-Leibhe divergence between softmax out- puts and the uniform distribution. Though OE is a powerful research line in OOD detection, heavy reliance on natural OOD data limits its application. Thus, generating virtual outliers is proposed to mitigate this challenge (Du et al., 2022b; Tao et al., 2023; Du et al., 2023). However, how to sample diverse and representative outliers distinctive from ID remains a key challenge.
C.2 MODELING THE ID HYPERSPHERICAL DISTRIBUTION
Benefiting from the advanced representation learning (Ming et al., 2023b; Lu et al., 2024), we for- mulate the IDs in sampling space with the hyperspherical model which projects an ID input as z onto the unit sphere (|| = 1) in a d-dimensional space. The projected embedding z can then be effectively modeled by the von Mises-Fisher (yMF) distribution (Mardia & Jupp, 2009; Fisher, 1953). The probability density kernel for a normalized vector 2 is formulated as:
4/2-1
(s) = Za(k) exp (s14 T 2) = (2x) (R) exp (ri4² 2),
(10)
where ju denotes the mean vector of the vMF kernel, K 2 0 represents the concentration parameter, (K) is the modified Bessel function of the first kind at order v, and Za(k) is the normalization factor. We approximate the condicooniiional probability density function with kernel density esti- mation (KDE) (Rosenblatt, 1956) produce more precise estimation of the ID probability density
20

---

Published as a conference paper at ICLR 2025
function. Specifically, let 2c { z2, ........ 2n denotes the normalized embedding set of training data from class c. The class class-conditional probability density function of x is then approximated as:
Σ pa(=: 2,,
pe(2; Ze: K)
(11)
where pa(z; =f,K) k) is delineated Eqn. (10), and is the number of samples from class The final ID probability density function is given by softmax as follows:
Zc. Zc,
PiP Ei De Zi,
(12)
The estimated probability density function with the vMF kernel offers accurate modeling of the unit sphere in such high-dimension space, which is employed to prevent erroneous outlier generation.
C.3 HAMILTONIAN MONTE CARLO
Given the target distribution density P(z) x exp(-U(z)) where (z) is interpreted as the potential energy in position and the momentum q sampled from the standard normal distribution. N(00, I), the Hamiltonian function is the total energy
H(z - U()) =l1al2
and the Hamilton's Equation is given by
dz;
0H 0g.
dqi df 0z,
0H
0U
= gc.
dt
0z:
where is the fictional time. To simulate the evolution, in each round, HMC takes the current (z, q) as initial state and performs L steps Leapfrog discretization with step size e,
dz (0) = z(0) + 29(0),
2(0 + c/2) = =(0) + dt
dq
dU dz (z(n -1/2)),, n I.....L
q(ne) q(nn 1)e) + dt
((n 1/2)c) q((n 1)c)
z(nn + 1/2)e) z(nn 1/2)e) + dz (ne) 2((n 1/2)c) eg(n)), I...., b-1 c x(L) x(LL 1/2)c) 2 gq(L)).
The final state (z(Le), q( (e)) the proposal, which is accepted and set the initial state of the next round with probability min 1, exp( (=)))), (()))H((((e)) ((llc))g((LL))) (Lc))) Since Hamilton's 's Equation implies the conservation of energy, i.e., H(z, q) constant evolution, the acceptance rate of the Metropolis test for HMC should be exactly 1. However, due to the small error caused by Leapfrog discretization, occasionally the value of H(2,0 4) may decrease which results in some rejection of proposals.
Additionally, we conduct the ablation study on the score-based Markov Chain Monte Carlo algo- rithms to demonstrate the generality of our HamOS framework. We utilize the random walk al- gorithm with Mecropolis-Hasting acceptance step (Metropolis et al., 1953; Hastings, 1970) as the baseline. We also employ the Riemann Manifold Hamiltonian Monte Carlo (RMHMC) (Girolami & Calderhead, 2011), 1), which effective if the target distribution concentrated on a diw-dimensional manifold. To accelerate computation, we adopt the simplified version of RMHMC where the dy- namic covariance matrix for sampling momentum qi is calculated as:
2i Z,-1),
(13)
where J (set to 2) is the number of previous states considered. When setting the Leapfrog step to 1, HMC reduces to the Metropolis-Adjuseee Langevin Algorithm (MALA) (Besag, 1994), also known as Langevin Monte Carlo (LMC). Manifold MALA (dubbed mMALA) (Girolami & Calderhead, 2011) can also be derived from RMHMC. Therefore, we experiment with the following MCMC algorithms for sampling outliers: Random Walk, HMC (default), MALA, mMALA, and RMHMC.
21

---

Published as a conference paper at ICLR 2025
C.4 HYPERSPHERICAL EVALUATION METRIC
We define the ID-OOD separation, ID inter-class dispersion. and ID intra-class compactness to analyze the quality of hyperspherical embeddings for representation learning methods, following CIDER (Ming et al.. 2023b). The ID-OOD separation cOsseperation measures the angular distance between ID and OOD; a larger value indicates that it is easier to distinguish OOD from ID. The inter-class dispersion coSDispersion reflects how sparse the IDs distribute on the unit sphere. The intra- class compactness cOSCompactness demonstrates the degree to which ID samples are clustered to its prototype. Straightforwardly, a model with better detection capability should produce embeddings with large cosSeperation value. The three metrics are defined follows respectively.
Pr: DOUD
(DOOD test
max T
Sepperrration
DOOD
coiDisspession (1L)
Σ
p
(14)
C
j=1
COSCompaciness
(DID (4)
DDI test
2 1,1 = }}
1=1
where Me is one of prototypes that have the largest cosine similarity with the OOD test sample, DOOD is the OOD test data, DiD is the ID test data, and 1 is the indicator function.
D DETAILED IMPLEMENTATION OF HAMOS
D.1 REALIZATION OF HAMOS
In this section, we provide the detailed implementation of our proposed HamOS. The outlier syn- thesis process is described in Algorithm 1, and the training pipeline of HamOS is described in Algorithm 2. The model is assumed to be well-trained on the ID data.
Our primary goal is to synthesize diverse and representative outliers and integrate the synthesized virtual outliers into the training phase with ID training data to regularize the model for better ID- OOD differentiation. For outlier synthesis, we utilize the Hamiltonian Monte Carlo with our OOD- ness density estimation in Eqn. (2) to sample sequences of outliers that span a wide range of OOD- ness. For regularization training, we optimize Cross-entropy loss and ID contrastive loss with ID data and OOD discernment loss with synthesized OOD data.
For outlier synthesis, maintain class-conditional dynamic buffer of ID feature embeddings {2,). for calculating OOD-ness density. Specifically, we update the fixed-size array ID when taking a new batch of ID data, eliminating the oldest ones. The k-tth nearest neighbor embed- dings in ID clusters are then retrieved to calculate OOD-ness density. The initial points of Markov chains are obtained by the centroids of ID cluster pairs, i.c., the normalized mean vector given two ID clusters. The initial points are treated as a mini-batch, each deriving a Markov chain parallelly through matrix operations. The number of synthesis rounds is fixed (e.g., default be 5). For rejecting erroneous outliers, the ID embedding buffer is utilized to estimate the ID probability for a candidate, as in Eqn. (5). The accepted candidates are treated as positions for the next round of synthesis, while the rejected candidates are discarded with their preceding points as the positions of the next round of synthesis. The samples in each Markov chain are gathered to form mini-batch of outliers for each training iteration.
ID classes
For regularization training, we follow the common practice of contrastive training (Tack et al., 2020; Chen et al., 2020; Ming et al., 2023b; Lu et al., 2024) to apply dual augmentation to get two per- spectives of ID data. The augmented ID data is then used to update the ID embedding buffer and calculate the CE loss. The ID embeddings manner, which are used to calculate the ID contrastive with the ID embeddings as in Eqn. (22) or the OOD-discernment loss with the virtual outliers in Eqn. (7). The loss is calculated as Eqn. (8).
also used to update the ID prototypes in the EMA
22

---

Published as a conference paper at ICLR 2025
Algorithm 2 Training Pipeline of HamOS
Input: model: { MLP(.); 0J fine-tuning epochs: T', ID training data: D train learning rate: 7/ OOD-discemment loss coefficient: Aa;
Output: fine-tuned model ((T);
1: for epoch 1, I do
2:
for ID data mini-batch 1TDD from Diai do
Conduct dual augmentation for ID sample (æ), 1/) ~ B.D to get (i, @ d vew) Project the ID data to hyperspherical space 2, = norm (M @ (.j))) 5: Update ID embedding buffer with Zz.
3:
4:
6:
Calculate Cross- Pooss-Entropy loss LcB with f(z, @ at)) and y @ y. Calculate the ID contrastive loss LID-con with 2j (and y).
7:
8:
Sample mini-batch of virtual outliers 2OoD in Algorithm 1.
9:
Calculate the OOD discernment loss LOOD-dise-
10:
Calculate the final loss Liaams LCr + LID.con + A&LoOD
11: Update the model weights: 0 <- V VoLammsss.
12:
end for
13:
end for
Table 4: Training Configurations of HamOS.
Factor
CIFAR-10
CIFAR- 100
ImageNet-
Training epochs
20
20
20
Trainable part
whole
whole
layer-4+MMP
Learning rale
0.01
0.01
0,0001
Momentum
0.9
0.9
0.9
Batch
128
128
256
Feature dimension
128
128
256
Weight decay
1.0 x 10-4
1.0 x 10-4
1.0
10
LR schedule
Cos. Anneal
Cos. Anneal
Cos. Anneal
Prototype update factor
0.95
0.95
0.95
Buffer size of ID data
1000
1000
100
Bandwidth k
2.0
2.0
2.0
OOD-discermment weight As for KNN distance
0.1
0.1
0.1
200
200
100
Hard margin Leapfrog step L
0.1
0.1
0.1
3
3
Step size
0.1
0.1
Number of adjacent ID clusters Nodj
4
4
4
Synthesis rounds R
5
5
5
D.2 DETAILED EXPERIMENT SETUPS
For fine-tuning a pretrained model, we set the training epochs to 20 following previous works (Ming et al., 2023b; Tao et al., 2023), with the size of mini-batch set to 128 for CIFAR-10/100 and 256 for ImK For ImageNet-IK, we freeze the first three layers of the pretrained model fol- lowing Ming et al. (2023b) and use the cosine annealing as the learning rate schedule beginning at 0.0001. For memory efficiency, we set the size of the cans--oondiiioal ID buffer to 1000 for CIFAR- 10/100 and 100 for ImageNet- IK. We summarize the default training configurations of HamOS in Table 4. For experiments that trains the model from scratch, the setups are detailed in Appendix F.I.
E BASELINES AND HYPERPARAMETERS
In this section, we briefly introduce the baselines adopted compared with HamOS, as well as their specific hyperparameters adopted by this paper.
E POST-HOC BASELINES
Maximum Softmax Probability (MSP) (Hendrycks & Gimpel. 2017). MSP leverages the max-
imum softmax probability distinguish OOD input from ID samples. The scoring function is
23

---

Published as a conference paper at ICLR 2025
defined as follows, SMSP ax; 0) max P(y = c/a;; 0) = max softmaxxx((rrr
(15)
where 0 is the weights of the well-trained model and is one of the classes. A larger MSP score indicates a sample is more likely to be ID data, reflecting the model's 's confidence in the sample.
ODIN (Liang et al., 2018). ODIN proposes scale the softmax outputs with temperature factor and conduct tiny perturbations to the input data. The ODIN scoring function is defined as follows,
SODIN(#: 0) max P(y c| @;; 0) max softmax (F (i;;; 0)/T),
(16)
where T is the temporature factor, and ø is the perturbed input samples controled by e. The r and e set to 1.0 x 10s and 1.4 x 10-1 defaultly as suggested in Liang et al. (2018).
EBO (Liu et al., 2020). The EBO scoring function measures the information contained by the logits of the model, which is adopted to discriminate between ID and OOD samples. The EBO scoring function is defined as follows,
SEHO [x: 0) -r log Σ exp (F(a;
(17)
where 7 is the temperature factor which set to 1.0 defaultly in this paper. OOD samples tend to have higher EBO scores, and ID samples typically have lower scores.
K-Nearest Neighbor (KNN) (Sun et al., 2022). This scoring function finds the k-th nearest neighbor for input sample given the training dataset embeddings. The KNN scoring function is defined as follows,
SKNN f) = || ((()) 12,
(18)
where f is the well-trained feature extractor that maps an input to the feature space, ||:122 L-2 norm, and ((k) the k-th nearest neighbor in the feature space. The ls is set to 50 default in this paper.
ASH (Djurisic al., 2023). ASH efficient post-hoc method that prunes and scales the model's activation. The modified representation is then put forward to the rest of the modules for ID classi- fication or OOD detection. In this paper, we use the ASH-S version and set the prune percentile to 65% as in Djurisic et al. (2023) and use the EBO scoring function for detection.
Scale (Xu et al., 2024). Scale scaling of the activation based on the mechanism analysis of ASH. Scale can further increase the EBO score separation between ID and OOD data. We adopt the 85% prune rate as suggested in Xu et al. (2024) and use the EBO scoring function for Scale.
subsequent work following ASH, which conducts solely the
Relation (Kim et al., 2023). Relation utilizes the relational structure graph and identifies the OOD data via the similarity kernel values. Specifically, the Relation scoring function is defined as follows,
Spelation (z: f) Eies k(x, x²))
(19)
where & is the similarity kernel function defined in Kim et al. (2023), and x is the training data.
F.2 REGUILARZZATOON-BASSR BASELINES
CSI (Tack et al., 2020). CSI initiates a new contrastive training scheme for OOD detection, which contrasts the sample with distributionally-shiffe augmentation of itself and trains the model with the contrastive training objective. The CSI scoring function is defined as,
Sesi(æ; max cos (f f())) [](
(20)
where cos is the cosine similarity, and xm is a sample from the training dataset with the maximum cosine similarity with the input data.
SSD+ (Sehwag et al., 2021). SSD+ utilizes the supervised contrastive learning objective (i.c., Sup- Con (Khosla et al., 2020)) get proper representation and utilizes the Mahalanobis distance-based
24

---

Published as a conference paper at ICLR 2025
scoring function (Lee al., 2018b) for detecting OOD data in feature space. The Mahalanobis distance scoring function is defined as,
Mahalamobis (z; f) = max - _ (f(x -c)))
(21)
where Mee is the mean of multivariate Gaussian distribution of class e and Σ is the tied covariance of the C' class-conditional Gaussian distributions.
KNN+ (Sun et al., 2022). KNN+ typically adopts the same training pipeline as SSD+ and leverages the KNN scoring function for detecting the OOD samples, defined as Eqn. (18).
VOS (Du et al., 2022b). VOS opens a promising direction that trains the model for OOD detection with virtual outliers. It models the feature space by mixtures of Gaussian distribution and synthesizes virtual outliers using clas-condiitinal information.
CIDER (Ming et al., 2023b). CIDER proposes to project the feature embeddings onto the hyper- spherical space and optimize the dispersion loss (termed as Edisp) and compactness loss (termed as Coomp) for proper ID representation. The losses are defined respectively follows,
c
daisp
=
C
log
1 + i} exp (M, p3/T),
1=1
1=1
(22)
exp ( Mc(1)/T)
Commp
log
ΣS= exp (ž, pn/T))
where μe is the prototype of class e and is the temperature factor. The final loss objective of CIDER is formulated as LCIDER daisp + Acheomp. where X. is set to 0.5 defautly.
NPOS (Tao et al., 2023). NPOS proposes a non-parametric framework for synthesizing virtual out- liers by selecting the far ID samples as anchors for sampling outliers from Gaussian distribution. The synthesized outliers are then used to optimize the binary training objective. The hyperparameters set as suggested in Tao al. (2023).
PALM (Lu et al., 2024). PALM a subsequent method following CIDER, which proposes to learn with a mixture of prototypes for each class instead of forcing the ID data to be close to only one prototype. The training configuration is set as in Lu et al. (2024).
F ADDITIONAL EXPERIMENT RESULTS
In this section, we offer additional experiment results further to validate the efficacy and efficiency of the proposed HamOS.
F.1 EFFIACY ANALYSIS OF HAMOS
Full comparison results on CIFAR-10 and CIFAR-100. We provide the main OOD performance comparison results in Table 5. As the results show, our proposed HamOS can outstrip the base- lines on most OOD test datasets and achieve significant averaged improvement, demonstrating the effectiveness of HamOS on multiple OOD scenarios.
Additional results on hard OOD benchmarks. To further validate the efficacy of HamOS, we provide additional performance evaluation on the hard OOD benchmark (Zhang et al., 2023b). In Table 6, we treat CIFAR-100 and Tiny-ImageNet (Le & Yang, 2015) as hard OOD test datasets for CIFAR-10. Table 7, we treat CIFAR-10 and Tiny-ImageNer as hard OOD test datasets for CIFAR-100. The hard OOD benchmark provides more OOD scenario where the ID and OOD datasets have similar semantics. The results display that our proposed HamOS can outstrip all the baselines with CIFAR-10 10 the ID dataset, while none of the methods achieve absolute superiority with CIFAR-100 as the ID dataset, indicating the difficulty of this benchmark.
ID-OOD separation with CIFAR-100 ID dataset. We provide the compactness and separation analysis with CIFAR- 100 as the ID dataset to guantitatively analyze the quality of learned repre- sentations. Shown in Table 8, HamOS can help the model to produce representations with larger
25

---

Published as a conference paper at ICLR 2025
Table 5: Full results of the main comparison (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
Mithols
FPRO51
AUROCT
AUPRR
FPRO51
AUROCT
AUPRT
AUROCT
AUPRI
Poet Nowe Menheuds
MSP ODIN
52.51em
05 13:0022
73.49
25.52am.r
47.862
27. 27.533x2
96 00.0882
as
US 15.80
BBO KNN
2T.37
94.72
75,50
64.76 L15.06
88.15
12:
97.10 97.101..12
98.24
20.6516.54
00.822...2 0.20
82.0211.55
23. 23.00 2.46
51.25 00.00
87 87.02:1.40
94.9415.3
95, 05.22,00 o
ASH
40 200.0024:00
12+ x
30 17 m
94.50 44
02.41
Relation
20.09
00.27
13.95.1.75
24.900
86 28.44
21.04:1.:3
94.20s1.99
94.06
Regustar
Eosntie Nuscal Methusts
CSI
21.50
00 27
15, 76
92
10.82a1
97. 17.34cc.aa
07.67 17.67-n
SSD+
02 48
$1.1115.1
12 12.89
D4 48
1001.a.c
97 27.46.5.58
UT.77
KNN-
24.50.5.33
92.3 92.54.1.00
50.51
14 14..2
06 06..3:3-440
25 79
18:231mm
96.05:::30
96.71
26.4724055 am
08.61-2.0
76.24 76.24 200
46, 2.
89.27 c9.44
18. 18
96.25
18.01asuas
10
NPOS
06 0663222100
09
07 50
58.
37
us Sl.
PALM
37.6126-11
84, 16.08.1
(8. Glae
S4.5 $4.1134
90.56
83. 23.28
10 ce
57 20.20
U7 28.5
CIFAR-10
Honn(S/ours))
10.63:0.91
07.14:0.99
01.37
3.67 10.50
59.01
98.13 0.59
4.00
08. 08.42:...a4
98.83
Testur
Placcs
Ascxagol
Mechisds
MSP
10.08
90.04.0.4
22.29
44 2..00
99.21.09.4
18
12.17.6.t
91.10at.71
81 11.70_5 es
81
71.08 an
BBO
50.81-12.50
80.
89 20.00.
50.
80.79:1229
85 80.
41 41.85aua
91.794..a
79.70 10.70
KNN
22.85
03.30 20
945 1a
28. 70
81 S1g11
22 22886g1
92 ai.
88 16.74.m
ASH
68.02220 x0
25.06
55.28
T4.52
81,6527 as
50.88-:3 1:40
64.22am5 os
ST ST.97am.0 as
72. 23.88
Scale
19.50.
36.33
78.24 13.54
10.35
68 28.188
77.74 30.4....4
63 20.53
25.02:2.92
Luo
Naseea Methods
24.9612.41
12
$4.02
89.72
78 78.80122.4
50.735
H9.74
SSD+
20, 20.14g1 Ma.
98 20.40
25 80g
92.94
94 00
94.85 64.885a..0
90 18.88
KNN-
09.78
56. 15
25. 12
95. 26:27:mm
91 98 28
94.41
D0 00-A0
VOS
60:94 23.00
88. 112
85.77 15.35
50.4 46 La.
88.75.1.74
63.96 13.57
42.37 21.33
91.4211.39
70, 10.16113..6
CIDER
19.07 13.24
10.3
23.
10.16
16:24 Loccs
08.70.1..7
85
PAL.M
45.04
88.
92.35 31
33. 51
91 67
90
52-25 22.22.
00.54.
84 S4.4
OomOSsourr)
10.80.0.59
95.8 95.82:0.3
97 66:00..
18 19
95. 18:0
88 7..
10. 10:48.0.70
97. 07.11:0.26
94 94.0
MNIST
SVHN
L.SUN
Pooshoo
Mechosly
MSP
58.7713.70
75.51 2.20
62 12 16.75
20.56..3.
B1 68 18.57
56 1.
79.0821..7
81 $1.32 o
ODEN
35.58
$1.47.22
18
20.97
71.42223m
30
71.70ga 14
$2.51 a-
23.47
75.80-21
14.8
83.
50.834
KNN
80
58
58 15.32
1722
80.31
ASH
76.0423.15
71. 21.54.0.7
90.881133
G0 750 20.700110.11
80.01
65 65.68
56.47 56.57
78.71.
80.70
Scale
78.9044.0:
72.41.21
53.11asms
02 dlaace
80.
80.75_..0
G4 64.3
60.91...
50.47a...77
80001
Relation
31.4723.00
70.15 13
57.70
61 11gm mg
70.52g4 tn
B3 83:30g co
400
70.88.21.43
77.300 oo
70. 70.122
68 08.01-1.0
50: 4..4
00.66-... 1s
86 41 Lit
85, 42
77, 77:2823-49 07
1..09ammm 19
TTan
28 31
22:2211 64
KNN-
66:08-110
$1.70
50.77:
82 18
77.
VOS
54.94:3..5
40.91.4.0
12 08
07.5641.44
60
82.04 82.0615.10
02.00.1.41
02. 142334..4
CIDER
69. 00.3112.30
08:06-200
80 11.11224
30 11 0.50
01 83100.43
86 28 0.59
83. $3.55
65.67 11.00
60.27
70. m0
77288g-- 49
07
20: 10g o
CIEAR
PALM
38.40
07.50-4
30.38
as
86.12
24
39.50
91. 21.75.x.r
90. 19.
Testure
Places
Averaged
Prosston Medhandy
MSP
62.8822-15 05
76.20
S1.6921
56 S8.
78.7723
58 08.82a0
D9.7Han
77.25 3M
06.66
ODEN
70.61.a..s
77.88
$4.081..4
08. (5.07.a 12.5
76.39.9.4
51 18.98 1219
78.00
63-20-22
ERD
49.77
78.91-10
$4.11
02 1144
78. 78.42:.a/4
56
008001157
78-524:.1
06.73
ASH
7L.104
07
$3.02
70.79
16.19
1x
m
(22
Scale
70.20:3.42
76.36...4
82 12.6210.8
74.5813.77
76.00
47 07
09 09.27
17.20
61.42
Relation
55.0913.24
80.71.a.s
S6. 17.33
63. 63.44110.0
78.55 78.86.99.0
55 45
50.0015:4
70.00011.04
08.70.111
Requlartiation
CSI
18.94
65.07 L.8
72. 12.76
72 14
69.71 10.32
47 1877
70.4611..37
61.37 to
SSD+
83
58 40
81 67.77a0 20
02 a0
09.73
VOU
67.70.4.77
78: 78:57
70.05
76.54.3
m
21
58.55
$1.50a5:2
68:30-1
CIDER
48 43
83 83.54.9.0
54 54.9721.0
80 80.000:3.0
83 21
49 19
51.77at.5
73.22-1
NPOS
48.8245.41
88.85...1
90 16.6...
56.00 cass
79.19-3.57
58.21 28.21.1.4
51 11as
51.12.t.a
72.40
70.84
201-41-200
$5.81
77.85.3..7
70.35....
70.21-13
HamOSSoorn)
49.21g1191
82:70ga
50.02g033
81.4ggm
64:00.m
46.08.1 44
83.642044
75.0221.0
ID-OOD separation values, significantly benefiting the ID-OOD differentiation. The inter-class dispersion and intra-class compactness don't 't directly correlate with the ID-ACC or the OOD perfor- mance. Although HamOS doesn't 't exhibit the highest inter-class dispersion or the lowest intra-class compactness, it indeed surpasses all the methods compared in Table 3, 8 by both OOD metrics and ID-ACC.
Sampling with variants of HMC. We provide the full averaged results of HamOS with variants of HMC algorithms in Table 9. We treat the Random Walk with Metropolis-Hasting
as the baseline. The results show that our proposed outlier synthesis framework is compatible with various sampling algorithms. More advanced sampling algorithms can be adopted for specific scenarios.
Using different distance metrics as scoring functions. We offer the full averaged results of using Mahalanobis (i.e., defined as Eqn. (21)) and KNN (i.e., defined Eqn. (18)) distance metrics in Ta-
26

---

Published as a conference paper at ICLR 2025
Table 6: Results on hard OOD benchmarks with CIFAR-10 as ID data(%). Comparison with com- petitive OOD detection baselines. (averaged by multiple runs)
Methods
CIFAR-100
Thnymigghhet
Averaged AUROCC
FPROS)
AUROCT
AUPRT
FPRAS
AUROCT
AUIPRT
FPR95]
AUPR
Phstetoo
Mestoott
MSP
65.799a6.0
45.17
80.15...
89.32
50. 60.48
88.17.0.75
ODIN
$1.41
77.49
72.54
80.14.5..1
77.79
79.27
78.81
7..36.
EBO
68.87 saus0
86.17s1.53
82.46
89.000-1.41
87.47 a.40
63.89
87.00,1.57
64.97
KNN
37.22 22
89.80
40.50
29. 29.75
91.78-0.2
93.52.0..4
33. 162.49
90.82
992 02.011:0
ASH
78.55
79.25
73.97
72.30
81.91-7856
79.00 19.09
75.42-15.11
80.60 60.60
76.52
Scale
85. 85.48
17
66. 66.55-
82. 13.
72, 22.94
78 23.03
71.50 11.50=15
69 29
Relation
40.40 85
88, 99.9990..
60.13gg.44
33 28
90 00.80
92 65
89. 89.94ga
90. 90.88
Metloods
CSI
36.77
88. 10.47
$9.51 60.73
29 2.70
90.82 0.50
53,00
30.15
89. 89.52.a.55
91. 01.200
SSD+
36.52210.6
89. 20.5
90.30
28. 43.3.a
91.79-00.21
93, a3.d
32. 28.77
90.67
91 s0.00
KNN+
35. 00.000...00
90.00 90.02.0..5
91,05
26.57
92.34.n..s
94. a5.00
30.79
91.18
92. a2..5
VOS
69 50
65. 65.74:22.4
60 86
$8.50
26.55
65. 21 -5.21_115.4 at
87. 27.16x2.00 39
10
CIDER
6:
00.47a0.1
23 92822a
05.17
94, 20.r
28. 12230.0
01. 01.822an.0 os
27.933nn.12
NPOS
90. 73
25. 73ga-s
03-41gn-m
94. 82
00
02.07 22.07aa an
15gn.am
PALM
35.78ag ax
81.91 de
83.41g1-4
46 54.
86.21111.0
10.00:1.:5
51.1 15
1..0 70
HamOS(oorrs
29.57 29.57-99.6
21.36ggn.4 40
92: 1ggas
20.72g1.a as
94.37g0
95.58g1.0
14...7
92 80 17
Table 7: Results on hard OOD benchmarks with CIFAR-100 as ID data(%) ()). Comparison with competitive OOD detection baselines. (averaged by multiple runs)
Methods
CIFAR-10
ThnytaageNer
Averaged AUROCT
FPR95
AUROCT
AUPRT
FPRUSL
AUROCT
AUPRT
FPR95
AUPR
Past-hoe
Mestoule
MSP ODIN
62 2..3
17.06
77.7
52. 7..
80.81 -0.0
78.934a..0
18
09.10
77.4740.74
06.25
78.700-0.s
83.00.0.0
67.18
77.3440.30
23.499...4
64.18
17.82...53
o..86.0.s
67.42.1.10
81.012-0.16
86.12.n.0
60.80
79.42.0.20
12.8
78.801044
14.27
49 49.622.0.2
$3.33
88.38.0. 10.1
60:57 $.30
10.0.3
0..44
70.00
15.04:1.06
73.8412.7
71.51:330
70.88
$1,30-11.00
71.00
75.96
77 1.30
10.68
72.14
79 79.81:285
73.55-1.06
27.50
77.15
74.01:1.4
Rolation
15.26
76.71
72.4320.4e
82.7 62.74
07.00
63. AM.1122.. 3.
79.72px.21
80.01...21
Mettooss
CSI
14.13
69
69 80.55
G7 a755
73.51
80,96
70.89
71.30.0.2
TB. .0.59
SSD+
86. 16.41.00.5
7T.
78, 78.5010.43
51.19 51.19.a.0
82.48
87.9 87.94an22
18 10.33
80. 80.24:
82.33
KNN+
$0.2 s0.28
10.08
69 89.58
57. 57.78a1.00
80.644-026
85.92
19.00 19.00311443
70.36....1
T7.95
VOS
72. 87
10.61
T2. 73.73
70.02=2.2
78.000-mm
82.50.0.0
71.24
15. 15.8
77 60.60....
CIDER
69. A3aa
26.59
35.31an0"
51.82anaz
81.95_2117
87 37.8
60 12.6233.0
79. 79.27 2m
81. 4..77 57..7
NPOS
77. 27.27
78. 78.96.0.44
40.66.1.40
55 65.49amm
80.55_mmm
86. 96.10aan
66. 18.38811.1
77. 77.27
78. 42
PALM
96.06am
48. 48.4720.8
45.68
87 25.25
64.13gmmm
69 a0
91 6449
56. 56.30
57 77
Ham(SSoers)
73.57
76.82g1:1
23
51.07gam
82.69g209
88. 88.04.0m
62. 22132211
79.75g0 20
80. 18.78
Table 8: Ablation results on compactness and separation with CIFAR- as ID dataset(%) ()). Com- parison with competitive OOD detection baselines. Results are averaged across multiple runs.
Method
OOD Separation?
Inter class Seporttion
Intra class Compuctness
MNIST
SVIN
Textures
Places 365
LSUN
Averaged
SSD
28 30ann
38, 82
36 92
17.27am0
34. 76.75as
34 34.83
00.005
56. BB unas
CIDUR
42 26
59 61
$2. $2.4
50.5 50.58aa
51 1..56an
S 51.30.0.1
50.37
73.
NPOS
43. 6041.00
60 a7..5
52.01 lao
50 00.05an..0 on
51 55....0
51. 51.91-e.a3
50.41 couus
73.00 78.00
PALM
34 12.00
41.800.2.c0
29.20-..47
28.71.11.
37. 17.000
38-44-c.4
84.444.0.45
711 711.24-0.23
HumOStears)
60 20.681.g.m
53.077
62.30_mmos
53.1 38.14gmmm w>
53.1 53.12gn
00.30ggmm
73. 70gmn
Table 9: Ablation results on different sampling algorithms (%, time reported in millisecends). Com- parison with competitive OOD detection baselines. Results are averaged across multiple runs.
Sam, Algo.
CIEAR- 10
CIFAR 100
FPR95
AUROCT
AUPRI
ID-ACCT
Timei
FPR95)
AUROCT
AUPRI
ID-ACC1
Time!
Random Walk
18. 00
95. 08
91 35.35
52 68
0.96
50.06 50.06.22:
81. 42.22
72 72.69
75. 75.35
3.24
IMC
10 48
97. 1
94, 24
94 64.87201
34.71
46 68
83 64
75. 03.52_mm 3
76. 12
287 48
MALA
11 11
96. 74g
94 20
94 51
9.06
47 47.8321
63. 10
74.664
76 78.78an
29.80
SMALA
10. 10.89
97. a0.00
94 54.531000
94 54.00.0.35
15:32
47.27
64.5610
745770
78. 78.90
44.96
RMBMC
11.21
97. 97.07amas
94.77a031
94.02 94.0220as
47.03
40.8020ss
63.02200s
74.5011
76. 76.16.0.0
381.65
ble 10. The results show that our proposed HamOS is effective with different distance-basel scoring functions. Since HamOS maintains the original classification head, which makes it compatible with other host-hoc methods, we further conduct ablation studies the scoring functions in Table 1 1.. HamOS consistently improves when equipped with different post-hoe methods, distinguishing its superiority in enhancing the model's intrinsic detection capability.
Integrating with various ID contrastive objectives. Recall that HamOS only requires differen- tiable ID representations, which makes it compatible with multiple ID contrastive losses. We provide
27

---

Published as a conference paper at ICLR 2025
Table 10: Ablation results on different distance metric (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CIFAR-I0
CIFAR-100 -O0
Distance Metric
Methods
FPR95]
AUROCT
AUPRT
FPR95
AUROCT
AUPRT
SSD+
18.49+1.20
94.85+0.57
90.88-0.85
54.03=1102
60.664.0.60
69. 09.73=1.00
CIDER
16,4720
95.3820.4
92.22-0.17
28.34_2.24
81.99 81.9921.14
73.67
Mahalanobis
NPOS
13.10#1.0
96.31-0.52
93.77+0.51
49.85-2211
$1. $1.02+0.76
72.52+1.56
PALM
32.25+4.11
90.51=1.46
81.44-2.14 14
55. 13.0079
79. 19.95+1.20
70.21 2112=11.5
HamOSSouur))
+3.29
95.56+0.48
93.37=0.39
46 46.29::0.15
83.23
75.22=0.30
KNN+
19.68
94.4110.00
90.46_o.0s
61.25gn.s1
78. 78.24.10.s
66.6419.85
CIDER
16.28+4 6s
95. 45.7620.a7
92.36 ac
19.64121.00
81. 81.77=0.05
73. 73.22=1.18
KNN
NPOS
14.39 .7
96.61 10.06
93.35
51 .1 11.58
81 $1.02200.s8
72.49
PALM
10.03
841.5511.72
78.28
72.51 22.44
74.08
60.0111.88
Ham OStours
10.4810.76
97.11
94.94 20.84
46. 68 44
83. 83.642n.0 64
75.5221.30
the full averaged results of using different ID contrastive objectives in Table 12. Cross-entropy loss is adopted as the baseline. We calculate the ID probabilities as in Eqn. (12). The results show that the synthesized outliers in HamOS serve as valuable regularization for better ID-OOD differentiality.
HamOS outstrips baselines when training from scratch. We also apply the proposed HamOS to train from scratch scenarios. We train the initial model for 100 and 500 epochs on CIFAR-IO and CIFAR-100 datasets, with learning rate initially set to 0.1 and 0.5, respectively. The warming-up epochs are 20 and 50, respectively, for training 100 epochs and training 500 epochs, during which the model is only trained on ID data with Cross-entropy loss. For synthesis-based methods (i.e., VOS, NPOS, and our HamOS), the synthesis process begins at the 50th and 250th epoch, respectively. The results of training from scratch for 100 epochs are in Table 13, showing that HamOS is still effective when training from scratch. Note that the results in Table 13 are generally worse than those in Table 1, which can be reasonable since the fine-tuning is based on the well-trained models. The results of training from scratch for 500 epochs are in Table 14, demonstrating that HamOS can still outperform the baselines in this scenario.
Computational Efficiency of HamOS. We report the synthesis time of a mini- mini-batch for VOS (Du al.. 2022b). NPOS (Tao al., 2023) and our HamOS Table 15 on CIFAR-10, CIFAR-100, and ImageNet- 1K respectively. The results show that HamOS only introduces polynomial extra time cost while synthesizing outliers of much higher quality than the baselines. The computational efficiency results from two factors:
High quality sampling. Since HamOS can generate diverse and representative outliers, necessitates much fewer outliers to achieve superior performance than VOS and NPOS. In default configurations, HamOS only generates 20 OOD samples per class, whereas VOS samples 200 outliers and NPOS samples 400 outliers per class from Gaussian distribution. The low requirement of virtual outliers enables HamOS to perform efficiently, particularly in large-scale scenarios.
High efficient implementation. The Markov chains are generated parallelly across all classes using matrix operations. Additionally, the advanced similarity search technique also boosts the calculation of k-NN scores. This can be more straightforwardly verified by consulting the code in the supplementary material.
Sensitivity of the initial midpoints. As our primary goal is to synthesize diverse and representa- tive outliers with the estimation of the target unknown distribution produced by the N-NN distance metric, we iteratively sample data points from low OOD-ness region to high OOD-ness region. To this end, we choose the midpoints of ID cluster pairs with relatively low l--N distances from ID clusters, while remaining confusing and pivotal for the two ID clusters.
To examine the sensitivity of HamOS to the choice of midpoints, we apply Gaussian noise to the initial midpoints at different levels small perturbations and display the results in Table 2 below. The scale of Gaussian noise controlled by the standard deviation a. The results in Table 16 show that the performance of HamOS continuously deteriorates the amount of Gaussian noise to the midpoints increases, implying that the choice of midpoints between ID clusters is essential. Intuitively, perturbations on the midpoints may compel the initial states into areas that are of high
28

---

Published as a conference paper at ICLR 2025
Table 11: Ablation on different scoring functions (%). Comparison with competitive OOD detection baselines, Results are averaged across multiple runs.
ID datasets
Scoring Function
FPR95]
AUROCT
AUPR
MSP
16.17
91.1010.71
81.7015.6
MSP+HamOS
18.63
20.41
90.1 90.1220.92
ODIN
58.
18.04
85.7014.17
70.08 11.84
ODIN+HamOS
26.2011.64
94.62 11.11
88.1010. 10.600
EBO
41.85=13.78
11.7991.54
79.70
EBO+HamOS
17.13
96.35 10,22
01.7711.45
CIFAR-10
KNN
22.86 11.12
92.98
10.42
88.74 L..
KNN+HamOS
10.48+0.76
97.11 +0226
94.944+0.s0
ASH
54.22
87.37 16.00
72.33 116.40
ASH-HamOS
33.21 10.32
92.83
0..04
81.35
Scale
63. 13.188.23.6
77.74.11..2
63. 13.003120.2
Scale+HamOS
23.09
95.59 10.59
86.95
Relation
26. 26.28+1.68
92.31 +0.43
86. 86.75+a.u
Relation+HamOS
13.07
96 96.85+1.11
93. 93.4540.90
MSP
59.78+2.16
77 77.25+1.22
66 06.8641.56
MSP+HamOS
52.97
80.46.0..8
71.7810.44
ODIN
63. 23..9922.51
78.01
65..20
ODIN+HamOS
59.06.10.2
79.99
68. 68.5511.44
EBO
6..86
78.32 11:31
66.73 1.35
EBO=HamOS
49.6 19.68.1. 10.90
83.55
74.52
CIFAR-100
KNN
56.96
81.0111.19
70.60 29
KNN+HamOS
46.68 46.6811.44 11.44
83.6410.e6
75.52 75.5211.30 1.300
ASH
66. 06.8440.87
77.1441.12
62.24
ASH+HamOS
60. 60.3111.79
77.19
66.65 66.6310.55
Scale
69.27
77.25
61.42
Scale+HamOS
53.60 53.60+1.40
84.1640.11
72.57 72.57
Relation
59.64
79.69
68.76 1.78
Relation+HamOS
46.88.0.45
84.69
76.65+1..9
MSP
57.91
82.63+0.24
93.08+00.28
MSP+HamOS
56.49.11
83.09
93. 1440.91
ODIN
52. 12.112:1. 1.39
86.3440.44
94.45-1...7
HDmN+
48.71 10.35
87.86 87.8610.10
94.8710.77
EBO
47.71+1.65
86.7720..8
94.50
EBO+HamOS
46.1510.54
87.21
94.59
ImageNet-IK
KNN
46.60
1:0.09 83.99
93.08
KNN+HamOS
44.59.1042
88.59 10.08
95. 24
ASH
26.75
93.37 11.00
96.46 Lo.81
ASH-HamOS
25.78
94.07 +0.94
97. 07.04 1.14
Scale
34.0611.60
91.53 10.36
95. 95.89
Scale+HamOS
25.55
94.01 1..30
96. 96.67.10. 0..9
Relation
47. o7
86. 86.98 0..07
94.00+1.23
Relmtion+HamOS
49.08
35.81+0.80
94 02.22
O0D-ness, undermining the diversity of outliers. The perturbed midpoints may also reside in the ID clusters, bringing down the acceptance rate.
Visual verification of the efficacy of HamOS. We further verify the effectiveness of HamOS in Figure 8, showing that outliers synthesized by HamOS can help broaden the gap between ID and OOD distributions. The distribution of synthesized outliers occupies wide range of OOD scores, indicating its diversity and representativeness. We additionally provide the score distributions of ID and OOD samples along the training on CIFAR- 100 in Figure 9 as supplement to Figure 5. Figure shows that the outliers of various OOD scores are helpful regularizing the model for better detection performance.
29

---

Published as a conference paper at ICLR 2025
Table 12: Ablation results on different ID contrastive training objectives (%).
Comparison with
competitive OOD detection baselines. Results averaged across multiple runs.
CIFAR-10
CIFAR-100
Method
FPR95]
AUROC1
AUPR1
ID-ACC1
FPR95.,
AUROCT
AUPRT
ID-ACCT
28.
91.09
$6.08
94.26
66.42
75.34
61.22
76.64
CE+HamOS
21.21
93.20
88.28
94.04
57.28
78.64
64.68
74.46
SimCLR
22.
92.74
88.19
93.66
58.37
76.14
62.05
72.01
SinCLR+HamOS
16.56
93.99
88.57
93.37
53.76
81.45
67.72
72.79
SupCon
19.68
94.41
90.46
93.77
61.25
78.24
66.64
72.18
SupCon+HamOS
15.49
95.74
93.46
93.72
55.94
81.76
67.61
72.69
CIDER
16.28
96.76
92.36
93.98
49.64
81.77
73.22
75.09
CIDER+HamoOS
10.48
97.11
94.94
94.67
46.68
83.64
75.52
76.12
Table Training from scratch for 100 epochs with CIFAR- 10/100 as ID dataset (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple OOD test datasets with multiple runs.
CITAR-I0
FPR9SL
AUROCT
AUPRT
ID-ACCT
FPR9S]
AUROCT
AUPRT
ID-ACCT
CSI
19.64.1.44
94.98
90.59.0.0s
B161
61.42.0.0
64.80
50.93
50.30:
SSD+
20.19-0.74
93.96+...1
89.55
83.88400.2
56.866+2..o
79.69
28.533-2.4
713. 735844-0.s5
KNN+
22. 22.74=0..1
93.17+00.11
88. 18
83.76-0.22
60.8h+2.00
76.07=1.30
65. 65.15
73.39++1.5
VOS
27 27.06
93.8922.32
85 95.1..
95 05.3222000
61.2T+2.82
77 17.802a..0
60.6721
77.57
CIDER
18. 18.70.9.14
95.5110072
90 90-42a1.04
98.37
64.361..
80.27
70.31
74. 74.43
NPOS
28 28.421444
92.7010.74
85.4811:71
83.33
77.58
73
64.9511.0
74.71
16.79.1.29
96.02.o.az
92.10.0.ip
91-43.nn.77
57.541g-a2
75 75.32
11.00
HamOS(ours)
12.14:022
96 96.67.10.07
94.45go.3
94.9920.10
50.1221.1
80.912206
11
75 75-4720.00
Table 14: Training from scratch for 500 epochs with CIFAR-10/100 as ID dataset (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple OOD test datasets with multiple runs.
Methods
CIFAR-I0
CIFAR-100
FPR95L
AUROC1
AUPR1
ID-ACC1
FPR95]
AUROCT
AUPRI
ID-ACCT
CSI
12.15....7
96.83
94.53.0.o7
15.17.n0.1
67.68
72.77.1.44
61.05
65.48:.011
SSD+
21.27
92.78+0.co
89.32 82.0420
B33 63.660
63.72+0.55
73 73.099c..a
61 61.60eem77
71. 71.40-=1.0
KNN+
31.2140.01
90 90.23+1.m
83 83.34+2.11
88. 88.77=2.51
65.81+2.02
75. 75.11+0.05
62, 62.73=11.0
69. 80.-8-11.4
VOS
32.5321.v
91.98-2.54
85.5921.20
98. 98.62
52.31
82. 10.1.11
72. 72.8242.44
75.32=2.06
11.50.0.10
07.244..00
04.44.10.0
48.71
79 79.93.1..7
73.39
75.76_.
NPOS
19.61=1.10
93,201+0.50
88 (6.60622.75
85.333en.04
72.89
71.85+2.22 28
58. 655
71, 21.2626..2
PALM
15. 15.5226.17
77. 77.47+2.77
93.74.a..2
93.17 17:1720.11
15.56+2.54
86. 10.40.0
77.62 77.62=1.15
70. 70.36
HamOS(ours)
10.57:2.17
97. 97.7330.90
94.4121.14
93.83...0
40.81
89.3120.6
79 79.69120.0
75.46
Table 15: Synthesis time for generating mini-batch of outliers (seconds).
Method
CIFAR-10
CIFAR-100
ImageNet-IK
VOs NPOS
0.021
0.233
3.371
0.027
0.254
4.695
HamOS(ours)
0.035
0.287
3.625
Table 16: Perturbation the initial midpoints (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CIFAR-10
CIFAR-100
FPR95]
AUROC1
AUPR
ID-ACCT
FPR95]
AUROCT
AUPR
ID-ACC1
0.0 (default)
10.48
97.11
94.94
94.67
46.68
83.64
75.52
76.12
0.001
10.53
96.96
94.38
94.48
47.18
83.63
74.87
75.64
0.01
10.82
96.50
94.89
94.34
47.64
82.67
74.42
75.73
0.05
11.08
96.49
94.04
94.41
47.43
82.55
74.02
75.63
0.1
12.61
96.16
93.41
94.72
47.64
82.90
74.62
76.06
0.2
11.83
96.69
94.12
94.33
48.11
82.74
74.19
75.76
30

---

Published as a conference paper at ICLR 2025
3.-
ID (CIFAR 100)
(CIBAR 100)
DOD (SVIEN)
OOD (SVHNN)
OOD (SVIIN)
2.-
OOD (SVIIN)
Vartual Outtier
Virtull Outlier
Virtual Outlit
Virtual Outlie
Denasit
Density
2.0
Deasiit
2.0
8.6
1.0
1.0
0.3
0.3
-1.5 OOD Score
-1.5
0.0
-3.5
-3.6
0.0
0.0
3.0
-15
-1.3
-15
0.0
OOD Score
OOD Score
OOD Score
(a) Before Training
(b) After Training
(c) Before Training
(d) After Training
Figure 8: Visual verification of the efficacy of HamOS on CIFAR-10 and CIFAR-100: (a) be- fore training with HamOS outliers;(b) after training with HamOS outliers:()) before training with HamOS outliers;(d) after training with HamOS outliers.
(CIFAR-100)
OOD (SVHN)
Virutal Outliers
55.10
Epoch
51.12
Spoch
Density
Eloch
Epoch
10
89.71
37.51
Epoch
15
31.64
137 37.33
Spoch 50
20
30.38
-1.6
-0.8
0.0
OOD Score
FPR95
Figure 9: The trend of OOD score distributions of ID, OOD, and virtual samples along the training on CIFAR-100.
Table 17: Ablation results loss weight As (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CIFAR-10
CIFAR: 100
Ad
FPR95
AUROCT
AUPRI
ID-ACCT
FPR95L
AUROCT
AUPRT
ID-ACC1
0.01
12.92
96.61 10.34
94.67 io.23
48. 12
82.51 1o.m
14.40
75.8810.7
0.05
11. 11.89::.45
96.71-1.19
94 12.22:0041
94.59
47.17
82 83
74.72
75.91
0.I(default)
10. =0000
97. 97.11+0.22
94.94 64.994o..s
94.67+0.13
46 48.68+1144
83. 83.611.0.0
75.52-1.5
76. 76.12+0.14
0.3
1o.72
96.51_o...
91. 91.21
94.38 10.05
48.37 20.0
71.691..11
75.71
0.5
17.18:0.20
95.31 20.30
92.28 92:28
94.32:0.22
48.56
82.9450.1e
73.8140.2
75.42
F.2 ADDITIONAL ABLATION STUDIES
Ablation loss weight for Loon In HamOS, the virtual outliers used derive the OOD discernment loss LoOD-dise- which is combined to the CE loss and ID contrastive loss with a weight Ag- In Table 17, study the efficacy of weight λa {0.01, 0.05, 0. 0.3, 0.5). On the one hand, too small a value (c.g., 0.01) of da introduces mild supervision on the outliers, which leads to degraded OOD performance.
the other hand, Xa of high value (e.g., 0.5) may distort the hyperspherical space and thus undermine detection capability.
Ablation on the i value in OOD-ness estimation. Recall that we estimate the OOD ness density by averaging the KNN distance within a pair of ID clusters in Eqn. (2). In Table 18, we analyze the efficacy of & with different values. The results show that l value that too small can lead to degraded OOD performance while the performance is stable when A is bigger.
Ablation on hard margin 8. Recall that we apply hard margin ó based on the initial points to prevent erroneous outliers that have high ID probability in Eqn. (5). We ablate the hard margin in Table 19 with verying values (e.g.. 8 = {0.0, 0.05, 0.1, 0.15, 0.2}). The results show that with hard margin that is too small, erroneous outliers will be generated, which can introduce spurious OOD supervision signals, while a large hard margin will increase the rejection ratio and result in less diverse and representative outliers.
31

---

Published as a conference paper at ICLR 2025
Table 18: Ablation results on value for OOD-ness estimation (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CITAR 10
CIFAR 100
FPR95L
AUROCT
AUPRI
ID-ACCT
FPR95]
AUROCT
AUPR
ID-ACC1
50
13.63
96 a5
93. 33.1s
94.46-0.a7
49.47
81. 81.70po.zz
22.92
76.09
100
40.
96 36.60.+0.38
91.41=1.7
94 as
46 46.98-1.70
82.97+0
74.86+
76. 18.044
200ldefault)
20.78
20
91.94
91.67-0.15
16 68
83. 33.64=0.e 64
75.52+130
12
500
11.00g0.02
96. 96.50g0.72
98.82g1m
93.9320.48
17.
58.88g
76.3120.00
76.78g0.45
Table 19: Ablation results on hard margin (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CIFAR-10
CIFAR-100
8
FPR95.
AUROCT
AUPRI
ID:ACC!
FPR951
AUROC1
AUPPR
ID-ACC1
0.0
12.48
96.60
90.651044
94. 14.06
47.99
82. 82.4320.a
14.08
75.82 10.19
0.05
11.90
96.56go.8s
03.50.1.49
94.61
49.
81. 81.9910.31
73 73.861o.7
76.06go.00
(ddeaaul))
10.4810.70
97.13 1o.2s
10.94.0.a6
94.51 10.15
61
63.64_n..6
75.5211.30
14.
0.15
11.48
96.72_u.40
93.941..74
94.57 20.0s
48. 18...8
82.19
74.02
75.79
0.2
12.45 12.45++.50
96.36.0.79
90..62-0.o1
47.34.0022
83.40+0.7t
74.67+:228
76.03
Table 20: Ablation results on Leapfrog steps I. (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CIFAR-ID
CIFAR
FPR95
AUROCT
AUPRT
ID DDACC1
FPR95
AUROCT
AUPR
ID ID-ACC1
11
96.91
10.00 11
94.50.0.45
83 79
10
74.66 48
7G.7822.4
11.7611.65
96.74_6.47
94. 10
94.51 20.5
48. 87
82. 82.201o.5s
73.63 70
75. 75.07
3(default)
10.18=0.re
97. 97.11+0.20
94 949000000
94 91.6770.11
16 68
83. 61
75. 52.52=1
76. 12
11.1420s4
96.53_6.00
14g0
94. 94.50
48.40.1.00
82. 82.47.0.am
73.97
75 75.64g0
11.55=0.16
96.16-0.00
94.36+0.75
91.33-0.0
48.3 18.31+0.16
82.0 22.088e0.00
74.38+1.12
76. 76.23::0.6
Table 21: Ablation results on step size c (%). Comparison with competitive OOD detection base- lines. Results averaged across multiple runs.
CIFAR-I0
CIFAR- 100
FPR95
AUROCT
AUPRI
ID-ACCI
FPR95
AUROC1
AUPRI
ID-ACC1
0.01
12. 27.2911.55
96.42_0.10
93-87
94.31 10
94.31
48.9220.17
82.29
73. 73.801os7
76.03
0.05
12.4510.42
96.56_a.41
94.021aas
94.33
48.27 10.45
82.361n.t
74.07
76.061..01
(dd(faall))
10. 10
97. 97
M4.94.0.80
67.6772.11
68 44
83 33.64.0.64
75. 52.52 30
76. 12
0.3
11. 11.54.0-47
96.54_o.61
93. 03.9819.1
94.66_0.m
46.8021167
83.4510-a
75-3610as
76.9310.11
0.5
12.04
96.49_o.50
90-87
18.10_0.4t
47.1310.2
83.1410.s
74.81
75.9610.m
Ablation on Leapfrog steps L. Leapfrog Integrator is technique that simulates the true continue solution to Hamilton's 's Equation with discrete approximation. Leapfrog conducts L updates to the momentum q and position z. We provide ablation different Leapfrog steps in Table 20. The results show that HamOS is insensitive to the Leapfrog steps L. achieving superior OOD detection performance with varied Leapfrog step values.
Ablation on step size The step size of HMC controls the rate of convergence. Large step size encourages fast convergence the high OOD-ness regions while small step size leads to low con- vergence, which results in less diverse outliers, with results in Table 21 proving this attribute. A moderate step size facilitates generating diverse outliers that stretch out a wide range of OOD-ness density.
We provide visualizations the synthesized outliers with different Leapfrog steps (1,2,3, 4,5) and step sizes (0.01, 0.05. 0.1, 0.3, 0.5} on CIFAR-10 and CIFAR-100 in Figure 12 and Fig- ure 13. The visualizations intuitively show that the synthesized outliers in each round will be less diverse with smaller step size (i.e., 0.01) and may converge fastly to high OOD-ness density with larger step size (i.e., = 0.5). Besides, a minor Leapfrog step can result in outliers with small variance (i.e., the span of distributions is tiny), and enormous Leapfrog step may lead to diverse outliers. As the visualization suggests, moderate values should be set
but unstable distributions for Leaffrog step J. and step size e.
32

---

Published as a conference paper at ICLR 2025
Table 22: Ablation results on number of nearest ID elusters Naj (%). Comparison with competitive OOD detection basclines. Results averaged across multiple runs.
Noti
CITAR 10
CIFAR 100
FPRO5_
AUROCT
AUPRT
ID-ACCT
FPR951
AUROC1
AUPR
ID-ACC1
10 10.6G2o.04
97. 07.15
94 20
94. 24
47-42s1.7s
82 02.58go.ae
74. 71.91
76. 16
2/2
10. 10.86_0..5
96 31
94 23
94. 47
47.37
63. 11
70.2220.3s
76.00
3/4ddefault)
11.08
97 07.001o.13
24.20.000
94.61
46
83. 10.64
75 11.32
76.12
10defaulty10
10.4
97. 27.10
47.38g0.30
83.58_n.22
75.48
76.01
5/20
10.85 10.85_1.07
96 26.8620090
94.24 94.2410.9
94.5010310
47.02g1:30
83, 83.48_o.8s
73.40g0.56
73.9620.53
Table 23: Ablation results on synthesis rounds R (%). Comparison with competitive OOD detection baselines. Results are averaged across multiple runs.
CIFAR-ID
CIFAR-
[PR95]
AUROCT
AUPR
ID IDACCT
FPR95.
AUROCT
AUPR
ID ID-ACC1
13. 2.00
95.51
27
94 20.0020.02
19.577.0.0
20
73.56
75.87=0.r
12.16
96.74_6.46
94.00
04.68
48. 5...333.
82.35 22.351...0
73.67
73.6721:41
75.64.0.12
10.61=0.41
96.93-6.09
94.65=0044
91 01.0000..0
17. 17.51+0.07
82. 12:0.2
74.48+6.17
75.90 75.91+0.8e
10.86.2011
96.99 46
94.60go
94.72:0000
17.2021.55
82. 82.71.0.2
74.92:6.71
76 76.21:0.10
S(default)
10.48=0.7e
97.11=0.20
94.94-0.a6
91.67-0.15
46.00 18.68-1144
83. 83.64:00.4
75.52-1.80
78.12+0.14
ID CEPAR- 200) (SVIIN)
CIPARR 100) IEVIRNO
IP ICIBAR 1001 CEVINOO Guiliers
SPOS
Figure 10: t-SNE Visualization on CIFAR-100: (a)VOS; (b)NPOS; HamOs.
Ablation on number of nearest neighbors Nadj- To sample more representative outliers, we only generate outliers between though ID clusters close each other. Specifically, we choose the Najj closest ID clusters w.r.t. to an II) cluster and calculate the corresponding midpoints as the start points of Markov chains. The results in Table 22 indicate that HamOS is not sensitive to the number of closest adjacent ID clusters considered to generate outliers. Note that set Nadj {1, 2, 5} for CIFAR-10 10 and Najj (1,2, 2, 4, 10, 20) for CIFAR-100, 100, considering the number ID classes.
Ablation synthesis rounds R. As formularize the outlier synthesis process Markov chains, the length of Markov chains also impacts the quality of synthesized outliers. We ablate on the synthesis rounds R, reported in Table 23. The results show that too short of Markov chains (i.e., R = 1) can't help achieve superior OOD performance, while greater R (i.e., > 2) demonstrate efficacy, indicating the efficiency of HamOS for generating outliers.
Ablation on Feature dimension. In high-dimensional spaces, the Euclidean distance metric may be less discriminative, probably impairing the estimation of OOD-ness. The computational cost will also increase as the feature dimension scales. To assess the performance of HamOS with differ- ent feature dimensions, we conduct empirical analysis at dimensions 64, 256, and 512, alongside 4 representative baselines in Table 24. The results show that HamOS consistently achieves supe- rior performance almost settings, demonstrating dimensions. As the feature dimension scales up, HamOS only displays tiny performance degrada- tion. Some baselines can be drastically influenced by the feature dimension, such as SSD+ which begins to collapse at 512 dimension CIFAR-10 and PALM begins to collapse at 256 dimension on CIFAR- 10 and 512 dimension on CIFAR- 100. Surprisingly, PALM achieves high performance at 256 dimension CIFAR- 100, indicating that it requires careful tuning the dimension bring out the efficacy. Note that the dimension of the penultimate layer of the backbone is 512, so we scale up 512 edge case.
stable performance across various feature
33

---

Published as a conference paper at ICLR 2025
Table 24: Ablation results on feature dimension (%). Comparison with competitive OOD detection baselines, Results are averaged across multiple runs.
Dimension
Methods
CIFAR-I0
CIFAR- 100
FPR95
AUROC1
AUPRI
ID-ACCI
FPR95
AUROC
AUPR
ID-ACCf
SSD+
18.39
94.85
90.76
93.93
54.66
80.58
69.83
75.88
CIDER
15.53
96.02
92.67
93.52
48.36
82.38
74.54
74.74
64
NPOS
17.64
95.60
92.16
94.00
67.74
17.01
62.78
74.48
PALM
44.36
$4,02
76.43
93.87
66.95
78.89
63.55
74.45
HamOS(ours)
SSD+
18.49
94.85
90.88
933.95
54.03
80.64
69.73
75.63
128
CIDER
16.28
95.76
92.36
93.98
49.64
81.77
73.22
75.09
(Default)
NPOS
14.39
96.61
93.35
93.95
51.41
S1.02
72.49
74.53
PALM
32.25
90.54
84.44
93
55.13
79.95
70.21
74.67
HamOStours)
10.48
97.11
94.94
94.67
46.68
83.66
75.52
76.12
SSD+
16.05
95.74
92.36
941 66
50.97
82.71
71.80
75.51
CIDER
15.50
96.37
93.10
933.89
47.75
83.49
75.24
75.12
256
NPOS
13.04
97.00
93.83
94.00
70.65
76.10
60..4
74.52
PALM
92.74
68.18
42.41
10.00
46.81
83.62
75.48
73.09
HamOStours)
10.81
97.03
94.18
94.66
17.26
83.61
75.36
75.28
SSD+
85.59
66.83
16.64
93.46
51.70
81.67
70.58
74.97
CIDER
13.38
96.69
93.30
93:72
49.04
81.70
73.46
75.60
512
NPOS
15.22
96.27
92.79
94.33
69.09
77.14
61.01
74.94
PALM
96.71
54.06
34.86
10.00
95.97
42.63
31.44
1.00
HamOStours)
11.51
96.95
94.61
94.50
48.93
82.19
74.35
75.99
G DISCUSSIONS
HamOS vs. Outlier Exposure. Recall that our proposed HamOS generates virtual outliers solely based on the ID training data, while outlier exposure methods usually employ a large pool of natural outliers for regularization. For example, the Tmmy-ImageNet (Le & Yang. 2015) is commonly used the auxiliary dataset with CIFAR10/10 outlier dataset that can be even bigger than the ID dataset excessively undermines the feasibility of OE methods. Besides, the quality of the outliers is crucial for specific ID datasets, which should be diverse and informative, as demonstrated in previous works (Chen et al., 2021; Ming et al., 2022; Zhu al., 2023b; Jiang al., 2024a). Superior to OE methods, HamOS framework can synthesize diverse and representative outliers without any requirement for additional datasets, enjoying significant flexibility and efficiency.
ID datasets. The requirement for auxiliary natural
HamOS vs. Contrastive-based Methods. While the contrastive-basse methods (Winkens et al., 2020: Tack et al., 2020: Sehwag al., 2021: Sun et al., 2022: Ming al., 2023b; Lu et al., 2024) mainly focus on learning differentiable ID embeddings, our HamOS can additionally introduce OOD supervision signals from the areas between ID clusters where numerous potentials OOD scenarios are explored through Markov chains. The empirical analysis in Table 12 demonstrates the efficacy of HamOS against training with contrastive losses singularly, as well highlights the generality of HamOS, which is compatible with more advanced learning objectives for shaping the feature space.
Lying the core of our HamOS framework, the HMC sampling algorithm distinguishes our method from previous synthesis-based methods (Du et al., 2022b; Tao et al., 2023) by sampling from Markov chains, which underexplored for outlier syn- thesis. The synthesized outliers expand vast area in the feature space, occupying various levels of OOD-ness, thanks to the property of HMC that can generate trajectories from low-density regions to high-density regions. In addition to Figure 7, We provide additional t-SNE visualization of the synthesized outliers with CIFAR-100 as an ID dataset in Figure 10. While VOS tends to generate erroneous outliers that lie within ID clusters and NPOS may generate gathered outliers that lack di- versity, the outliers generated by our HamOS stretch through a vast area in the feature space without intruding on the ID clusters, which significantly enhances the model's 's detection capability. Here analyze the advantages of HMC compared to traditional Gaussian sampling:
HamOS vs. Synthesis-based Methods.
HMC samples diverse and representative outliers. As well-studied insight, the quality of outliers substantially affects the model's detection performance (Hendrycks et al., 2022), with diverse and representative outliers being demanded (Zhu et al., 2023b; Jiang et al., 2024a). HMC traverse large area in the latent space from low OOD-ness region
34

---

Published as a conference paper at ICLR 2025
1.93
1.93
4
1,78
4
1.78
00D-nes D-ness Density
2
2
1.33
1.33
0
1.10
1.18
1.04
1.04
-
ID Class 0
0.89
-2
0.89
ID Class
-4
Midpoint
0.74
-4
0.74
Virtual Outlier
0.41
0.44
-4
2
4
2
2
4
(a) HMC process
(b) Gradient field
Figure 11: Geometry analysis of the OOD-ness density estimation: (a) the sampling process of HMC with the OOD-ness density;(b) the gradient field of the OOD-nes: density.
to high OOD-ness region, thus generating diverse outliers representing different levels of OOD characteristics. However, Gaussian sampling, limited by its variance, always gener- ates samples concentrated to the mean point, prohibiting its ability to explore the hyper- spherical space.
HMC samples more informative outliers compared to Gaussian sampling. With our crafted OOD-ness estimation, new outliers are generated with the gradient guidance, which explic- itly points out the direction of the critical region that contains unknown OOD characteris- ties. The Gaussian sampling, however, stretches out to every direction evenly, resulting in generating outliers that may be less informative. The mix of outliers evenly distributed in every direction can hardly convey critical OOD characteristies to help the model learn differentiable representations.
HamOS vs. Traditional Score-based Sampling. HamOS displays numerous advantages compared traditional score-based sampling algorithms, like score matching (Hyvärinen, 2005), Langevine dynamics (Welling& & Teh, 2011), and diffusion models (Song et al., 2021), demonstrating the effi- cacy and efficiency of the framework. The advantages are outlined below:
HMC exhibits high sampling rate with low computational burden. Conventional score- based sampling may introduce extra learnable parameters which can increase the compu- tational load. HMC can leverage the informative gradients of the potential energy function to guide the sampling process with Leapfrog integration for exploring a large area in the latent space, handling the complex target distribution in high dimensions. The direct guid- ance from the gradients results high acceptance rate of nearly 1.0, thereby keeping the computational burden practical. Parallel matrix operations can enhance the efficiency of the HMC sampling process.
The estimation of the target unknown distribution enables the direct synthesis of OOD samples in the latent space. The objective of traditional score-based sampling is typically to converge to the target distribution guided by the training data, e.g., diffusion mod- els are supposed to generate legitimate data, prohibiting the ability generate samples from unknown distributions. HamOS, however, aims to collect diverse samples along the Markov chains, not requiring iterative sampling to converge. While other sampling algo- rithms struggle with the complex unknown distribution, such as Gaussian sampling that evenly explores all directions, HMC leverages explicit guidance find OOD characteris- tics that can be directly employed to optimize the model for better detection performance.
HamOS significantly exploits the information of ID priors for outlier synthesis by the class- conditional sampling between ID clusters. Traditional score-based sampling incorporates label information with the conditioned score model, which acts as a black box. In HamOS, with the OOD-ness estimation, outliers are explicitly sampled using the ID priors. Specifi- cally, in the high-dimensional spherical space, midpoints ID clusters serve as straightfor- ward starting points of the Markov chains, guarantecing the correctness of OOD synthesis. As the outliers are synthesized purely based on the ID data, HamOS has great generality to any dataset.
35

---

Published as a conference paper at ICLR 2025
On the design of OOD-ness density estimation. We want to estimate the OOD-ness (or the likeli- hood to be OOD), indicating the probability of a sample from unknown distributions, based solely singular ID clus-
on the ID data. We start by defining the OOD-ness of sample corresponding ter in Equ (1). As we want to generate critical outliers between ID clusters, we further average the OOD>-ness of a sample from pair of ID clusters. This definition of OOD-ness suffices to calculate the potential energy, the gradient of which is employed to direct the sampling of Markov chains among the ID cluster pairings. Multiple Markov chains are generated within different ID cluster pairs, ultimately collected as a diverse mini- batch of outliers. The OOD-ness is measured solely by the k-NN distance without the assumption of any external factors. The micro-operation of HMC generates new samples with higher OOD-ness in each Markov chain, macroly leading to diverse set of outliers that have varying degrees of OOD characteristics. We conduct further analysis of the OOD-ness density's 's geometry in Figure 11 11. As shown in Figure G. the Markov chains traverse from regions of low OOD-ness density to regions of high OOD-ness density, collecting a broad spectrum of outliers with diverse OOD characteristics. To delve deeper into the mechanism of the OOD-ness density, we plot the gradient field in Figure G. where the directions of arrows represent the direction of the density gradients and the scale of arrows represent the normalization of the density gradients. The gradient field further clarifies that points with high OOD-ness density tend to be proposed in the Markov chains. However, we impose no restrictions to the design of the OOD-ness density es- timation, despite that KNN-based density already displays superior efficacy. More estimations of OOD-ness desmity can be explored: for example, the reciprocal of the ID probability density shares a similar geometry of our OOD-ness density. The flexibility of the OOD-ness density estimation design further highlights the generality of our proposed HamOS.
Why does HamOS incorporate Cross-entropy loss? Recall that the training objective of HamOS consists of three terms: the Cross-entropy loss, the ID contrastive loss, and the OOD discermment loss. The OOD detection requires the framework to handle the dual-task scheme, where the original classification task is managed by the classification head trained with CE loss, and the OOD detec- tion task is dealt with by the projection head that maps the features onto the unit hypersphere for better ID-OOD diffcrentiation, to which end we adopt the general framework of OE method (i.c., introduced in Appendix C. Our empirical results show that HamOS maintains IID accuracy signifi- cantly. In Table and Table 2, in addition to achieving superior detection performance, HamOS also displays better ID classification performance than almost all regularization-basee methods. HamOS manages to preserve the ID prediction ability by two components:
The cross-entropy loss is adopted along with the fine-tuning phase to balance the opti- mization on the contrastive loss and the OOD discermment loss in Equation 8. This joint training scheme reaches good equilibrium between the original classification task and OOD detection.
The original classification head is preserved, which not only maintains the ID accuracy but also makes the HamOS framework compatible with many other post-hoc detection methods that manipulate features, logits, or softmax scores. As the classification and the detection are separated, they can rarely impair each other.
On the quality of synthesized OOD samples. The primary focus of this paper is to boost the OOD detection performance through outlier synthesis. We formalize the novel framework HamOS which models the synthesis process as Markov chains and generates virtual outliers with the guidance of the innovative potential energy function. With this in mind, the quality of the generated OOD samples is directly reflected by the detection performance and partially highlighted by the OOD scores in Figure 2, 4(b), 4(c). High variance of the OOD scores indicates diverse OOD characteristics that the generated samples have.
In previous works (7hu et al., 2023b; Jiang et al., 2024a), task-specific metrics are adopted to vali- date the quality of outliers. The quality of synthesized samples may also affect ID performance. For examplc, in Pooladzandi et al. (2023), a subset of synthesized data is selected by CRAIG (Mirza- soleiman et al., 2020) and samples with low entropy are eliminated by an entropic filter, making the quality of the synthesized samples high enough to improve ID-ACC. There may be techniques that can be designed to boost the classification performance well the detection performance.
Application of HamOS on other data modalities. As HamOS only assumes a pretrained network that transforms the original high-dimensional raw data to the representation format that is more suit-
36

---

Published as a conference paper at ICLR 2025
able for various tasks, such as classification, clustering, and generation. In NLP, numerous tasks involve learning the dense vector of embeddings, c.g., BERT and GPT produce contextual word embeddings for classification autoregressive generation. Sentence-level or document-level repre- sentation learning is also practical in tasks like semantic search or document classification. There- fore, the HamOS framework can be adopted in many NL.P tasks that require dense representation learning. In other modalities, such as graph-structured data or point-cloud data, HamOS can also be adopted since the data should be transformed into dense vector representation for various tasks, such as molecular structure modeling, protein design, and climate prediction. The superior performance of HamOS on the image data has been confirmed in this paper; nonetheless, its efficacy on various data modalities requires further investigation in future studies.
H BROADER IMPACT
Cross-pollination opportunities. The synthesis process of HamOS has many components analo- gous score-based generative models, presenting significant prospects for cross-pollination. Here we outline some potentials for future research:
The gradient score be produced by a hierarchical model instead of the k-NN metric. The fundamental process of generative models is to turn noise point into a legitimate point in an iterative manner, in which score model is adopted to predict the gradient without sampling directly from the target distribution. Analogously, we can develop score model, that may integrate the information from ID priors to produce gradients as better guidance for the outlier synthesis in each iteration. But this may introduce additional parameters, increasing the computational cost.
Both HamOS and diffusion models adopt an iterative synthesis process which can be con- trolled by scheduling strategies. It is feasible to apply a schedule that can adjust dynami- cally based on the progress of outlier synthesis, which may potentially improve the quality of outliers by regulating the gradient scale. There are numerous schedules choose from, such as linear, cosine, or any custom configuration. The adaptive scheduling strategy may also improve the versatility and adaptability of HamOS, accommodating diverse datasets, tasks, and data modalities.
Additional techniques of generative models, such as classifier-free guidance, offer diverse research directions for HamOS. The classifier-free guidance technique is primarily adopted introduce guidance information directly into the diffusion process without training an additional classifier. In HamOS, classifier-free guidance may help in lightweightly dis- tilling external OOD information to the synthesis process of OOD samples, leading to a better balance between diversity and representativeness, thus elevating the quality of OOD samples.
Conversely, the HamOS framework may serve as a beneficial prior work for improving the generative models. The non-parametric synthesis process may inspire more innovation in exploring and exploiting the latent space, leading to a high-qualit, efficient diffusion process. As HamOS facilitates learning a proper representation that is ID-OOD differentiable, there is potential to improve the fidelity of generation by shaping the latent space for diffusion models.
controllable, and
Impact on foundation models. Despite the substantial performance of foundation models using many data modalities, such as GPT. CLIP, and diffusion models, their trustworthiness and reliability remain underexplored. With the wide adoption of foundation models, the safe deployment and appli- cation of them are becoming more critical. The uncertainty estimation of the foundation models has attracted great attention, exploring how make the outputs of the models more trustworthy (Jiang et al., 2024b; Zhu et al., 2024; Lee et al., 2024; Zhang et al., 2024).
OOD detection is becoming more essential as the foundation models are employed in increasing safety-critical applications, such as healthcare, finance, autonomous driving, and cybersecurity. The failure of self-uncertainty estimation may lead dangerous outputs or risky decisions (Shi et al., 2024), as foundation models' overconfidence is not yet overcome (Groot & Valdenegro Toro, 2024). OOD detection acts as an indicator of when the L.Mss/VLMs should be trusted, with erroneous
37

---

Published as a conference paper at ICLR 2025
outputs, that don't align with training data, being detected before being displayed to users, such as hallucination detection (Farquhar et al., 2024) and unsolvable problem detection (Miyai et al., 2024).
HamOS, general framework. offers samples, hence improving the robustness and helping the model produce more proper representa- tions. Recall that HamOS assumes pretrained backbone network that projects the input data to the latent space, which aligns with typical representation learning schemes, endowing the nature of compatibility the training of foundation models. Empirical analysis on small-to-large scale benchmarks has been done to validate the generality of the HamOS system. As vision-based tasks are widely deployed in real-world applications, such autonomous driving and medical image di- agnosis, it is more common to encounter OOD scenarios, henceforth we assess the efficacy and efficiency of HamOS on image tasks. This doesn't restrict the application of HamOS in other do- mains, such as NLP and graphs, which also adopt the representation learning paradigm.
perspective to augment the training data with negative
38

---

Published as a conference paper at ICLR 2025
-0.4
OOD Score
-0.6
-0.8
-1.0
-16
-10
Synthesis Round
Leapfrog step
-0.4
OOD Score
-066
-0.8
-122
0.01
0.05
-14
0.5
-1.81
Synthesis Round
(b) Leapfrog step = 2
-0.4
OOD Score
-0.6
0.8
-1.0
-1.2
-14
-16
Synthesis Round
Leapfrog step 3
-0.4
OOD Score
-0.6
-0.8
-1.0
-1.2
0.05
-14
-16
Synthesis Round
(d) Leapfrog step = 4
OOD Score
-0.6
-0.8
-1.0
-1.2
Synthesis Round
(e) Leapfrog step =
Figure 12: Round-wise analysis CIFAR-10 with different Leapfrog steps and step sizes: the outliers are synthesized with different Leapfrog steps (i.e., 2,, 3, 4, 5)) and different step sizes (i.e.. (0.01. 0.05, 0.1, 0.3, 0.5)).
39

---

Published as a conference paper at ICLR 2025
-0.2
-044
OOD Score
-0.6
-1.0
0.01
0.05
0.1
-14
-16
Synthesis Round
(a) Leapfrog step 1
-0.2
-0.4
OOD Score
-0.6
-1.0
0.01
-12
00
Synthesis Round
(b) Leapfrog step = 2
-0.2
-0.4
OOD Score
-0.6
-0.8
-1.0
0.01
-12
0.05
-14
Synthesis Round
(c) Leapfrog step 3
-0.2
-0.4
OOD Score
-0.6
0.8
-1.0
-1.2
-14
Synthesis Round
(d) Leapfrog step
-0.2
-0.4
OOD Score
-0.6
-0.8
-1.0
-1.4
-1.6
Synthesis Round
Leapfrog step
Figure 13: Round-wise analysis CIFAR-100 with different Leapfrog steps and step sizes: the outliers are synthesized with different Leapfrog steps (i.c., {1,2,3, 4, 5)) and different step sizes (i.e., (0.01, 0.05, 0.1, 0.3, 0.5)).
40


Reviews:
[{"review_id": "w7HgQO3qli", "reviewer": "Reviewer_hv7F", "summary": "Many existing OOD detection methods rely on a large, high-quality set of natural outliers. To address limitations with synthetic outliers, this paper introduces the Hamiltonian Monte Carlo Outlier Synthesis (HamOS) framework, which uses Markov chains to generate diverse and representative outliers based solely on in-distribution data. HamOS achieves efficient and high-quality synthesis, showing superior performance over state-of-the-art baselines on standard and large-scale benchmarks.", "questions": "- How does the proposed method compare to NPOS and VOS in terms of time cost? \n\n- How stable are the experimental results (e.g., what is the standard deviation)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation behind this method is clear and compelling, focusing on synthesizing effective virtual OOD data to enhance the robustness of machine learning models. \n- Unlike traditional approaches that rely heavily on large, high-quality pools of natural outliers, this method leverages the technical innovation of Markov chains to generate diverse and representative outliers from only in-distribution data. \n- This framework not only provides high-quality synthetic outliers efficiently but also demonstrates superior performance across multiple datasets, validating its effectiveness and efficiency compared to state-of-the-art methods in OOD detection.", "weaknesses": "- There are several detailed errors in the article that affect the reader's experience. For example, the caption of Figure 1 mentions ImageNet.\n- The manuscript's explanations are difficult to follow, especially in the method section regarding how OOD samples are synthesized and how HMC is utilized. This is particularly challenging in the part involving Equation 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c4uurDBinS", "reviewer": "Reviewer_5Vcs", "summary": "The paper proposes HamOS (Hamiltonian Monte Carlo Outlier Synthesis), for out-of-distribution (OOD) detection that uses Hamiltonian Monte Carlo sampling to generate synthetic outliers from in-distribution data. The key innovation is using HMC to sample diverse outliers in a learned hyperspherical feature space, guided by an OOD-ness density estimation based on k-nearest neighbors. The framework combines classification, contrastive learning, and OOD discernment losses to learn representations that better separate ID and OOD data.", "questions": "- How does the performance of HamOS scale with feature dimension? Hyperspherical geometry might become problematic at higher dims.\n- How sensitive is the method to choosing initial midpoints between the clusters?\n- Could you provide more theoretical insight into what hard margin to choose and a principled way to select values for different datasets?\n- What makes HMC-based sampling more effective than simpler Gaussian sampling approaches?\n- See, above \"weaknesses\". I'd be willing to revise score upwards, provided this is addressed.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This provides a novel application of HMC for outlier synthesis, and provides an elegant integration of hyperspherical embeddings with HMC sampling. Novel synthesis framework avoids the need for access to real outlier data. \n- The benchmarks studied are comprehensive, written up in a well-structured experimental section. SoTA performance is achieved.\n- The code in the supplementary section is cleanly integrated, kudos!\n- Visualizations are intuitive and informative, well presented!", "weaknesses": "The paper presents Hamiltonian Monte Carlo (HMC) for sampling outliers in latent space, sharing similarities with score-based generative models. It defines an OOD-ness energy function: $U_{OOD}(z; Z_u, Z_v) = -\\log P_{OOD}(z; Z_u, Z_v) = -\\log \\sum_{i=u,v} P_{OOD}(z; Z_i) + \\text{constant}$, which is analogous to score models' $\\nabla_x \\log p(x)$ and HamOS's $\\nabla_z U_{OOD}(z)$\n\nThe HMC sampling procedure follows $q^{(ℓ+1/2)} = q^{(ℓ)} - \\frac{\\epsilon}{2}(I_d - z^{(ℓ)}(z^{(ℓ)})^\\top)\\nabla_{z^{(ℓ)}} U_{OOD}(z^{(ℓ)})$, similar to Langevin dynamics $x_{t+1} = x_t + \\frac{\\epsilon}{2}\\nabla_x \\log p(x_t) + \\sqrt{\\epsilon}z_t$. \n\nThe main differentiator - HamOS's objective function $L_{HamOS} = L_{CE} + L_{ID-con} + \\lambda_d L_{OOD-disc}$ combines classification, contrastive learning, and OOD detection, differentiating it from traditional diffusion models by sampling between ID clusters to find OOD regions rather than sampling from noise to data distribution.\n\nTo improve the paper, the authors should add a section comparing their work to score-based sampling, properly cite relevant prior work on gradient-based sampling in latent spaces, and explain their advantages over traditional score-based approaches. These advantages include computational efficiency, direct optimization for OOD detection, and class-conditional sampling between clusters. The paper could also discuss potential cross-pollination opportunities, such as incorporating score network architectures for OOD-ness estimation, adopting noise scheduling strategies from diffusion models, and utilizing classifier-free guidance techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UW64WDHQbK", "reviewer": "Reviewer_HEF3", "summary": "This paper used Spherical HMC to generate OOD data and then from that figure out which data is ID and which data is OOD.", "questions": "I know HMC can be a bit computationally heavy, can the authors talk about this a bit? It is not immediately obvious to me if one should care about computational load in this setting, but it would be good to discuss runtimes or theoretical cost vs other methods non the less. \n\nDo the authors think something like Spherical RMHMC would do a better job or is Spherical HMC (SpHMC) good enough? \n\nI am wondering if the authors have any idea about the quality of the generated samples? Can they be used to boost the ID or OOD classification accuracy of the underlying models? I ask this because there was a paper that used MCMC to sample from an EBM and then train on those samples. They found this reduced performance unless the images were of high quality. It would be interesting if the samples generated via SpHMC are of high enough quality and diverse enough to actually directly boost acc performance of the model. \n\n[1 ]Pooladzandi, Omead, et al. \"Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization.\" arXiv preprint arXiv:2302.00138 (2023). https://arxiv.org/abs/2302.00138", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Uses Spherical HMC to better sample from the energy basin to sample OOD points. \n\nPrincipled and theoretically sound way to generate OOD samples and consistently outperforms other methods on the compared tasks and datasets.", "weaknesses": "In Fig 7 TNSE HMC is working a lot better to sample the OOD regions compared to NPOS, but in Figure 10 NPOS actually seems to be performing just as well or maybe even better as Spherical HMC. \n\nOne thing that is not clear to me is how we can understand the quality/utility difference between the OOD points sampled between the two methods other than the effect of outlier detection, but this might be out of the scope of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lMEwUct0lz", "reviewer": "Reviewer_by3L", "summary": "The paper proposes a novel framework called Hamiltonian Monte Carlo Outlier Synthesis (HamOS) for out-of-distribution (OOD) detection. The primary goal of HamOS is to address the limitations of current outlier synthesis approaches, which often rely on either extensive collections of natural OOD data or generative models that are computationally demanding.\n\nHamOS uses Hamiltonian Monte Carlo (HMC) to generate diverse and representative virtual outliers directly from the feature space of in-distribution (ID) data. Unlike previous methods that use Gaussian sampling or generative models, HamOS utilizes Markov chains to sample new virtual outliers by traversing a broad area of the feature space. The process ensures that the synthesized outliers are well-distributed and possess varying degrees of OOD characteristics, which helps in improving OOD detection capabilities.\n\nThe framework is evaluated on standard benchmarks like CIFAR-10, CIFAR-100, and ImageNet-1K, and shows significant improvements in OOD detection metrics compared to state-of-the-art baselines. The proposed HamOS achieves a balance between diversity and representativeness of synthesized outliers.", "questions": "1. Could you provide additional context that clearly distinguishes this work from existing generative or sampling-based methods for OOD detection? Specifically, how does your approach go beyond being a new method and qualify as a new paradigm?\n\n2. In the era of large language models (LLMs) and multimodal foundational models, what role do you envision for OOD detection, especially for image data? How does your proposed method relate to or complement the capabilities of LLMs?\n\n3. Can the proposed HamOS be adapted to other domains, such as text (e.g., natural language processing) or scientific discovery? If feasible, could you elaborate on the generalization of HamOS to these other types of data? Providing a discussion or a hypothetical example would help establish the broader applicability of your work.\n\n4. How does HMC sampling compare to other sampling methods, such as Gaussian sampling or Random Walk MCMC, in the context of OOD detection? Can you provide either a theoretical or empirical comparison to justify the choice of HMC?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Using HMC for generating diverse virtual outliers is a creative adaptation of a well-known sampling technique in statistics and machine learning. Unlike the conventional generative or Gaussian-based sampling methods, this approach allows the model to extensively traverse the latent feature space, thus creating diverse and representative OOD samples. This unique utilization of Markov Chain Monte Carlo (MCMC) methods for OOD detection offers an innovative approach that is distinct from prior generative and noise-based sampling strategies.\n\n2. The concept of OOD-ness estimation to guide the sampling process is also a good addition. By focusing on hyperspherical embeddings and using distance metrics to determine the likelihood of being OOD, the paper contributes a perspective on how to quantify and synthesize OOD characteristics in feature space.", "weaknesses": "1. The paper's claims regarding the novelty of using Markov chains for outlier synthesis are overstated. Specifically, the statement in the first contribution on page 1 claims that outlier synthesis using Markov chains is a \"new paradigm\" for OOD detection, which is misleading. Markov chains are a well-established approach in sampling, and the application to outlier synthesis does not represent an entirely new paradigm, but rather a new technique within an existing category.\n\n2. In the current era, large language models (LLMs) and multimodal foundational models have demonstrated remarkable performance in numerous domains, including computer vision, text understanding, and scientific knowledge discovery. The paper only evaluates HamOS on natural image datasets (e.g., CIFAR-10, CIFAR-100, ImageNet-1K), which raises concerns about its significance and practical applicability in other fields.\n\n3. The concept of \"OOD-ness\" is central to the proposed approach, but its definition remains unclear. Specifically, it is not evident how OOD-ness is formally quantified or measured, which complicates understanding how diverse outliers are synthesized. There is a lack of clarity regarding whether OOD-ness is defined solely based on distance metrics (such as k-nearest neighbors) or if other factors are considered.\n\n4. The advantages of Hamiltonian Monte Carlo (HMC) over other sampling methods, such as Gaussian sampling or Random Walk MCMC, are not sufficiently discussed. The reader is left wondering why HMC is particularly suitable for OOD detection and how it performs better than other, potentially simpler, methods.\n\n5. In Table 3, HamOS shows low inter-class dispersion, which is concerning as it suggests poor separation between different ID classes in the feature space. This could potentially affect the performance of the method on in-distribution (ID) classification, as class separation is critical for robust classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)