The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection
Authors: Qingyang Zhang
Qiuxuan Feng
Joey Tianyi Zhou
Yatao Bian
Qinghua Hu
Changqing Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Out-of-Distribution Detection; Uncertainty estimation;
Primary area: safety_in_machine_learning
TL;DR: This paper provides both analysis and solution to overcome the dilemma between OOD detection and generalization.

Abstract: Out-of-distribution (OOD) detection is essential for model trustworthiness which aims to sensitively identity semantic OOD samples and robustly generalize for covariate-shifted OOD samples. However, we discover that the superior OOD detection performance of state-of-the-art methods is achieved by secretly sacrificing the OOD generalization ability. The classification accuracy frequently collapses catastrophically when even slight noise is encountered. Such a phenomenon violates the motivation of trustworthiness and significantly limits the model's deployment in the real world. What is the hidden reason behind such a limitation? In this work, we theoretically demystify the "\textit{sensitive-robust}" dilemma that lies in previous OOD detection methods. Consequently, a theory-inspired algorithm is induced to overcome such a dilemma. By decoupling the uncertainty learning objective from a Bayesian perspective, the conflict between OOD detection and OOD generalization is naturally harmonized and a dual-optimized performance could be expected. Empirical studies show that our method achieves superior performance on commonly used benchmarks. To our best knowledge, this work is the first principled OOD detection method that achieves state-of-the-art OOD detection performance without sacrificing OOD generalization ability. Our code is available at https://github.com/QingyangZhang/DUL.

The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection
Qingyang Zhang'; Qiuxuan F Tianyi Zhou', Yatao Bian',
Qinghua Hu', Changqing Zhang"t College of Intelligence and Computing, Tianjin University! A*STAR", Tencent AI Lab²
Abstract
outtof--distiibution (OOD) detection is essential for model trustworthiness which aims sensitively identify semantic OOD samples and robustly generalize for covariate-shiffes OOD samples. However, we discover that the superior OOD detection performance of state-of-the-art methods is achieved by secretly sacrificing the OOD generalization ability. Specifically, the classification accuracy of these models could deteriorate dramatically when they encounter even minor noise. This phenomenon contradicts the goal of model trustworthiness and severely restricts their applicability in real-world scenarios. What is the hidden reason behind such limitation? In this work. theoretically demystify the "sensitive-robuss dilemma that lies in many existing OOD detection methods. Consequently, a theory-inspired algorithm is induced to overcome such dilemma. By decoupling the uncertainty learning objective from a Bayesian perspective, the conflict between OOD detection and OOD generalization is naturally harmonized and dual-optimal performance could be expected. Empirical studies show that our method achieves superior performance on standard benchmarks. To our best knowledge, this work is the first principled OOD detection method that achieves state-of-the-art OOD detection performance without compromising OOD generalization ability. Our code is available at https://gituu..cmm
Introduction
Endowing machine learning models with -fttt--distribution (OOD) detection and OOD generalization ability are both essential for their deployment in the open world [1]
3] We borrow an example of autonomous driving from [4] to demonstrate the motivation of these two tasks. Given a machine learning model trained on in-distribution (ID) data (top image in Fig. (a)), OOD detection aims to sensitively perceive uncertainty arising upon outliers that do not belong to any known classes of training data [5] (bottom right image in Fig. (a)). While OOD generalization expects machine learning models to be robust in the presence of unexpected noise or corruption, e.g., rainy or snowy weather (bottom left image Fig. (a)). this paper, we reveal that many previous methods pursue OOD detection performance at secret cost of sacrificing OOD generalization ability. To make things worse, we observe that some SOTA OOD detection methods may result catastrophic collapse in classification performance (~15% accuracy degradation) when encountering even slight noise. One pioneering work [4] makes a trade-off between OOD detection and OOD generalization, but the relationship between these two tasks is still largely unexplored. The learning objectives of these two tasks are seemingly conflicting at first glance. OOD detection encourages sensitive uncertainty awareness (highly uncertain prediction) unseen data, while generalization expects the prediction to be confident and robust under unforeseeable distributional shifts. Previous work in OOD detection
Work done during intersship Tencent AT Lab.
Correspondence to Changqing Zhang <zhangchangging @tju.edu.cn>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

OOD generalizatio (Ervor)
Entropy Bayesian
EBM
Trade-off Area
SCONE
ID
XPOEM
WOODS
x DUL (Ours)
MSP
Covariato-shifted 0OD
Semantic OOD (OOD detection)
OOD detection (EPR95)
(OOD generulization
(a) Three types of data arise in the open world
(b) Dilemma of current SOTA methods
Figure 1: (a): Models trained on in-distribution (ID) data inevitably encounter distributional shifts during their deployment. OOD generalization expects the model to correctly classify covariate-shifted data that undergoes noise or corruption due to environmental issues. OOD) detection aims to identify samples that do not belong to any known classes for trustworthiness consideration. (b): Limitations of current advanced OOD detection methods. We consider 8 representative OOD detection methods including the baseline method MSP [6] (without any OOD detection regularization), Entropy [7]. EBM g Bayesian [9], SOTA OOD detection methods WOODS [T), POEM M. recent advanced SCONE which aims seek for a good trade-off and the proposed DUL. All these methods exhibit a degraded generalization ability compared to baseline method MSP and lie in a trade-oft off area except our DUL. The goal of this paper is to understand and mitigate this phenomenon.
research area 4 characterizes the relationship between OOD detection and OOD generalization as a trade-off and thus striking for balanced performance. However, this trade-off significantly limits the employment of current state-of-the-arr OOD detection methods. Naturally, one might require the model be aware of the OOD input for ensuring safety, but certainly does not expect to sacrifice the generalization ability, not to mention that the catastrophically collapsed classification performance under noise corruption.
In this work, we first uncover the potential reason behind this limitation by characterizing the generalization error lower bound of previous OOD detection methods, which is referred to sensitive- robust dilemma. To overcome the dilemma, we devise a novel Decoupled Uncertainty Learning (DUL) framework for dual-optimal performance. The decoupled uncertainties separately responsible for characterizing semantic OOD (detection) and covariate-shifted OOD (generalization). Thanks to the decoupled uncertainty learning objective, dual-optimal OOD detection and OOD generalization performance could be expected. Our emphasis lies on a particular category of OOD detection methods in the classification task, including max softmax probability (MSP) based model (6) energy-based model (EBM) [8] and Bayesian methods [9]. This selection offers two-fold advantages. First, MSP EBM and Bayesian detectors encompass major OOD detection advances in classification task (5).
Second, numerous OOD detection works in diverse learning tasks (classification, object detection [2] time-series prediction and image segmentation) are all roughly related classification 113. The contributions of this paper are summarized as follows:
This paper reveals that existing SOTA OOD detection methods may suffer from catastrophic
degradation in terms of OOD generalization. That is, their superior OOD detection ability is achieved by (secretly) sacrificing OOD generalization ability. We theoretically demystify the sensitive-robust dilemma in learning objectives as the main reason behind such a limitation.
In contrast to previous works that characterize OOD detection and generalization as conflic- tive learning tasks and thus implying inevitable trade-off, propose novel learning framework termed Decoupled Uncertainty Learning (DUL) to successfully break through the limitation beyond simple trade-off. Our DUL substantially harmonizes the conflict between OOD detection and OOD generalization, which achieves the best OOD detection performance without sacrificing the OOD generalization ability.
We conduct extensive experiments on standard benchmarks to validate our findings. Our DUL achieves dual-optimal OOD detection and OOD generalization performance. To our best knowledge, DUL is the first method that gains state-of-the-art OOD detection performance without sacrificing OOD generalization ability.

---

2 Related works
OOD detection aims to indicate whether the input arises from unknown classes that are not present in training data, which is essential for model trustworthiness. In the classification task, the majority of advanced OOD detection methods include MSP detectors which characterize samples with lower
max softmax probability as OOD 66 14 15 [6] EBM detectors identify high energy samples as OOD and frequently establish better performance than MSP detectors 8
4) and various other types OOD detection methods such distance-based detectors 18 non con-paraeetric KNN- detectors [19] which also show promises. According to the training paradigm, OOD detection methods can be split into auxiliary OOD-free and auxiliary OOD-required methods. Auxiliary OOD-free methods directly use the model pre-trained on ID data only for OOD detection. Another line of methods assumes that some OOD data is accessible during training and incorporates auxiliary outlier datasets (collected from websites or other datasets) for further enhancing OOD detection performance. By exposing the model to some semantic OOD during training. auxiliary OOD-required methods frequently outperform auxiliary OOD-free methods on commonly-used benchmarks 20 S].
OOD generalization expects the model to be robust under unforeseeable noise or corruption 21 22 23
Basically, OOD generalization expects invariant and confident prediction on OOD data. Examples include classic domain adaption (DA) methods which encourage the model's behavior to be invariant across different distributions [21] 26 27]. Besides, test-time adaption (TTA) directly
24 25
encourages confident predictions on OOD data by minimizing predictive entropy (28 29 30).
However, as we will show later, confident prediction and invariance are seemingly conflictive to OOD detection purpose and further imply an unavoidable trade-off. The most related work to our paper is SCONE [4) which strikes to keep a balance between OOD detection and generalization performance. We argue that such trade-off is not necessary and the conflict can be elegantly eliminated.
Uncertainty estimation in Bayesian framework. In the Bayesian framework, predictive uncertainty be regarded an indicator of whether the input sample prone be OOD. Since OOD) samples are unseen during training and thus should be of higher uncertainty than ID. The overall predictive uncertainty of a classification model can be decomposed into three factors according to their source, including data (aleatoric) uncertainty (AU), distributional uncertainty (DU), and model (epistemic) uncertainty (EU) 9 AU measures the natural complexity of the data (e.g., class overlap, label noise) and EU results from the difficulty of estimating the model parameters with finite training data. DU arises due to the mismatch between the distributions of test and training data. A line of classic measurement can be used to capture various types of uncertainty including entropy, mutual information, and differential entropy 9
3 Preliminaries
We consider K-class classification task with classifier f : X + R parameterized by 0, where A' is the input space and = 2,, ... K} denotes the target space. The model output fe(x) is considered as logits. The k-thh element in logits is denoted as fz( indicates the confidence of predicting x to class k. The predicted distribution Fxx) is obtained by normalizing ff fx) with the softmax function. We first formalize all possible distributions that the model might encounter.
In-distribution PID which denotes the distribution of labeled training data.
Covariate-s shifted OOD XY which is relevant to OOD generalization. PCOV is of the same label space with ID. However, its marginal distribution Px encounters shifts due to unexpected noise or corruption.
Semantic OOD x is the distribution of data that do not belong to any known class. Its label space has no overlap with the known ID label space, i.e., )" ny =
In the following paper, we omit the subscript for simplicity. The goal of OOD detection is to build detector G: - [I, OUT] to decide whether an input æ is semantic OOD data or not through thresholding function G deduced from classifier f
IN G₂(x)) = OUT
{OUT
91(z) S7
(1)
g5(x)
>7
3

---

where is the threshold. 9, is an OOD scoring function deduced from f. which is expected to assign higher value OOD than ID. For example, MSP detectors, 9y (z) maxx F(x) where F F(z) is the predicted softmax probability (negative max softmax probability). In EBM detectors, 9f is realized by the energy function Exx; f) := log Σ efi(()) and the semantic imput of OOD should be of high energy 8 Since is difficult to foresee PSEM one will encounter, board line of OOD detection works 17
12 32 [3] regularize the model on some auxiliary OOD data PSEM train during training (e-g.. data from the web or other datasets), and expect the model can learn useful heuristic to handle unknown test-time OOD PSEM, The learning objective is shown as follows
test
min ((z.y)) (Lce))--in(Co((((()))) v)] + PiSEM (Creg (+)];
where Lce is the standard cross entropy loss for the original classification task. Freg is the OOD detection regularization term depending on the detector used, which generally encourages a high entropy between F(x) and the uniform distribution for MSP detector 77. In EBM detectors [. freg is realized as a margin ranking loss to explicitly encourage large energy gap between ID and semantic OOD. In this paper,
uncertainty train For example, Lreg is set to
interested in this setting for the following reasons:
1) In contrast labeled data in supervised learning literature, auxiliary OOD data can be unlabeled and easy-to-collected in practice 11.
2) Most SOTA methods involve auxiliary outliers 20 for superior performance.
3) Even under some strict assumptions that PSEM unavailable, recent works utilize GAN (15], diffusion model
334 or sampling strategy 12 to generate "virtual" outliers for training.
Thus we believe this setting is promising and the cost of auxiliary outliers is minor given the importance of ensuring model trustworthiness. At dest-time, the model is evaluated in terms
ID accuracy (ID-Ace () which measures the model's 's performance p!D.
OOD accuracy (OOD-Acc 1) measures the OOD generalization ability on pcov,
False positive rate at 95% true positive rate (FPR95L) := E (L( = IN)) mea- sures the OOD detection ability, where is chosen when true positive rate (TPR) is 95%. I is the indicator function. In OOD detection, ID samples considered negative.
is worth noting that in the standard OOD detection setting have any overlapped classes or samples with training-time auxiliary OOD data PSEM Let y/SEM and USEM be the label space of PSEM and train respectively, we have y train 0. Otherwise, OOD detection would be a trivial problem.
the test OOD data should not
Sensitive-robust Dilemma of Out-of-distributiin Detection
In this section, we detail the limitation of current OOD detection methods: their OOD detection performance is achieved at the cost of generalization ability. This limitation implies the potential risk of SOTA OOD detection methods and underscores the urgent need for better solution. Firstly, we re-examine representative OOD detection methods of six different types, including 1) baseline model MSP that trained without any OOD detection regularization (6, 2) cetrrooyrregulaiizatii (Entropy) that encourages high predictive entropy on OOD 77 which is devised for MSP detectors, 3) engrgy-regularization for EBM detectors that enforces the output with high energy score for OOD input 8 4) Bayesian uncertainty learning that encourages high overall uncertainty on OOD 19, 5) state-of-the-art OOD detection methods WOODS [10] and POEM 6) the most related SCONE that seeks for a trade-off between OOD detection and generalization performance.
Limitation of current OOD detection methods. In Fig. (b), we investigate current OOD detection methods in terms of OOD classification error and FPR95. The expected classifier should yield both low OOD classification error and FPR95. As it is observed, despite the superior OOD detection performance, all above methods significantly underperform the baseline MSP in terms of OOD generalization. By contrast, our method (DUL) successfully overcomes the limitation.
Theoretical justification. Toward understanding the limitation, we provide theoretical analysis for two types of most popular OOD detection methods, i.c., MSP and EBM detectors. Our analysis

---

identifies the sennsitive-robus" dilemma as the main reason behind such limitation. The roadmap of our analysis is: (1) inspired by transfer learning theory, we first reveal that OOD detection regularization applied on semantic OOD may also affect the behavior of model covariate-shifted OOD; (2) then we demonstrate why MSP detectors suffer from poor generalization by characterizing its generalization error bound; (3) we further identify that EBM methods [8] suffer from similar
drawback when incorporating with gradient- basee optimization. First of all, we recap the definition of disparity discrepancy in transfer learning theory [35] 27
Definition (Disparity with Total Variation Distance). Given two hypotheses f E F and distribution P. we define the Disparity with Total Variation Distance between them as
disp ff) (F(
where Fy,, F are the class distributions predicted by f', respectively. (VII)))) is the total variation distance, i.e., TV (F))))))) = 3xx=11Ff,, Fyl.
Definition (Disparity Discrepancy with Total Variation Distance, DD with TVD). Given a hypothe- sis space F and two distributions P, Q, the Disparity Discrepancy with Total Variation Distance (DD
with TVD) is defined as
dx(P.Q)= = sup (dispp(f', dispo(f'.f). f',E
(4)
Disparity discrepancy (DD) measures the "distance" between two distributions P, P.Q which considers the hypothesis space. DD is one of the most fundamental conceptions in transfer learning theory which constrains the behavior of hypothesis in should not be dissimilar substantially on different distributions P' and Q. If the DD between semantic OOD and awwariae-shifted OOD is limited, one can suppose that OOD detection regularization applied to semantic OOD samples will also influence covariate-shifted OOD. Thus encouraging high uncertainty on semantic
the model's behavior OOD may also result in highly uncertain prediction on covariate-shine shifted OOD, which is potentially harmful to generalization ability. We first formalize this intuition for MSP detectors.
Theorem 1 (Sensitive-robuu) robust dilemma). Let pCov, test be the covariate-shifted OOD and seman- tic OOD distribution. GError pcov ( denotes standard cross entropy loss taking expectation on peov, i.e, generalization error. Then we have
1 - log K]
GError (f) 2 C
E
PSSM teat
Creg(1 (f)
dr dy
PSEM
8K
2x
OOD goneralization
OOD detection loss
(5) where Lreg is the OOD detection loss devised for MSP detectors defined in [7], i.e. cross-entropy between predicted distribution Fxx) and uniform distribution. d>(pCOV PSEM) is DD with TVD that measures the dissimilarity of covariate-shifted OOD and semantic OOD, C and K. are both some constants depending on hypothesis space F, pCov and PSEM
test
The proof is deferred in Appendix A Theorem demonstrates that for MSP detectors, the OOD detection objective confficts with OOD generalization. The model's generalization error lower bound negatively correlated with OOD detection loss that the model tries to minimize. Thus given a limited d,., pursuing low OOD detection loss on test will also inevitably result in highly uncertain prediction on pcov It is worth noting that such an interpretative theorem is applicable for all MSP-based OOD detectors no matter whether the model involves PSEM during training not. Since the inherent motivation of OOD detection methods lies in minimizing the OOD detection loss in PSEM, regardless of the training strategies used.
Why a limited d (PCCV PSEM) is practical? In Theorem 1 (PCOV PSEM) measures the dissimilarity between pcov and trst It seems that this lower bound will be very small and trivial when d pcov pSEA) is large enough. However, since the semantic OOD samples can be any samples that do not belong to ID classes, one can suppose that semantic OOD samples are extremely diverse and some are of high similarity with ID and covariate-shifted OOD
37 Detecting these
"ID-like" OOD samples is inherently the core challenge of OOD detection [ 38). Thus, is reasonable to assume limited d= pCoV PSEM). ). We provide more discussions in the Appendix D.2
D.2
Encompassed by 336 special case, our definition is realized using TVD for theoretical convenience.
5

---

As presented before, the key limitation of MSP detectors is that they enforce high-entropy prediction on semantic OOD, We proceed reveal that EBM detectors suffer from similar issues due the natural property of gradieet-based optimization. For EBM detectors, feeg is defined by g is
Lrog PSEM mmax(mouT []))),)) piD pin [max(0, E(x) mIN)?
(6)
which constrains the energy score E(2; f)= -log Σ ef((x) of ID sample æ' to be lower than that of OOD sample E. mIN, MOUT manually selected margins. Although such regularization does not indicate high entropy prediction at first glance, unfortunately, we demonstrate that EBM detectors also tend to uncertain prediction when equipped with gradient-based optimization. Here focus on Gradient Descent (GD) as a showcase. In each training epoch t. model fo is updated with GD as 0241 02 v/V Teeg, where v) the learning rate. For any sample ã drawn from PSEM, the gradient VoLreg can be written as
:=
Vi (
e "1 E())),
higher for larger fa(
where moUT E E()) > 0 (otherwise the gradient is zero) and fā( is the predicted logits on the k-th class. For sample when class à has larger predicted logit, contributes more the overall gradient VoLreg and thus could obtain more optimization efforts during backpropagation. Eventually, one can infer that when an EBM detector about to converge, tends to high-entropy prediction on train accordingly. Incorporating this into the established Theorem | this is likely to harm the generalization ability of pcov, Empirical evidence can support this supposition (see Table 14 in Appendix Therefore both MSP and EBM detectors face the "sensitive- dilemma.
Decoupled Uncertainty Learning
We demonstrate how to handle the dilemma between OOD detection and generalization by decoupled uncertainty learning in the Bayesian framework. Unlike the most related work 4 which aims to seek good trade-off, our method successfully gets out of the aforementioned sensitive-robust dilemma.
Uncertainty Estimation in Bayesian Framework. We first revisit the theoretical properties of different types of uncertainty in Bayesian framework. Non-Bayesian classifiers consider the model's output ((x as logits, which is then normalized with softmax to directly model predictive categoricals p())z)) While in Bayesian framework [9]. fxx is considered as parameters of Dirichlet distribution p(u(z) x) firstly, which is used to model the prior of predictive categoricals p(o(zz) by
[(ao) I ((ak)
K
p(u[z) Dir(pla)
II 1 a= f(x),
(8)
k=1
where Dir(ula) is Dirichlet distribution and is the concentration parameters of Dirichlet. The sum of all O E a (noted as ag) is so called the strength of the Dirichlet distribution, i.e., 0g > 0, og En Og- After obtaining prior p[u|2r (z. f), the final predicted posterior p(ğ (z:) over class labels is given by calculating the mean of the Dirichlet prior
data uncertainty
/
p p(ilar 0)
p(il)) plula, dyc.
(9)
overall uncortainty
distributional uncortainty
From Bayesian perspective, given deterministic parameters 0, the overall uncertainty of final prediction p()]z)) can be decomposed into two factors, including data (aleatoric) uncertainty (AU) and distributional uncertainty (DU). DU lies in p(p]a, 8) which defined as uncertainty due the mismatch between the distributions of test and train data. AU is described by p(ilp)) which captures natural complexity of the data (e.g., class-overlap) 9. By definition, OOD detection is primarily associated with DU which is only a part of the overall uncertainty. While generalization is related to
the overall uncertainty of p(r) ar as we mentioned in related works (both AU and DU). One essential property of DU is that it can be high even if the expected categorical p(ğ)a 2 0) expresses low overall

---

uncertainty. Such a property is well suited to achieve OOD detection and generalization jointly since high DU no longer necessarily indicates high overall uncertainty.
Decoupled Uncertainty I Learning. While the aforementioned Bayesian framework enjoys theoretical
potentiality, its learning object [9] lacks consideration of OOD generalization. Similar to other OOD> detection methods, it also directly enforces high overall uncertainty on OOD
min Epi (L(p(y) (p(( (=)))(()) + Epsex (p) Dir( 1)), truin
(10)
where p(ylar),p(())) are the ground-truth distribution and predicted distribution on IDD The model's prediction on OOD is enforced to be close to a rather flat Dirichlet distribution. It is worth noting that Dir(ula- 1) means all classes are equiprobable, and the entropy of the final prediction is maximized. As shown in Fig. (b), the vanilla Bayesian method [Р] also suffers from degraded OOD generalization performance. To this end, we propose Decoupled Uncertainty Learning (DUL), a novel OOD detection regularization method that explicitly encourages high DU on OOD samples without affecting the overall uncertainty. Similarly to previous OOD detection methods [8. our DUL is also devised in finetune manner for effectiveness. Given a classifier fe, well pre-trate pre-trained on pID, the goal of DUL lies in enhancing its OOD detection performance without sacrificing any generalization ability. Specifically, we finetune the model by encouraging higher DU but non-increased overall uncertainty on PSEM The learning objective DUL is
train
min E PID LCE (1) + Ea max (ho + mOUT) - b)]l troin
0
ID classification
high distributional uncertainty (detection)
(1I)
s.t.
H (p(oaa)) =
H (Po(o)t)
pSEM
train
BOn increeseedd overall uncertainty (gencralization)
where (-) is the entropy. p)) and po (g]āa are the predicted distribution on semantic OOD data á after and before finetuning. The first term is the original ID classification loss. The second term is OOD detection loss, which encourages high DU on outlier ă. moUT and T > are hyperparameters. ha. h are DU on ă before and after finetuning. Here we measure DU with the differential entropy (h(p(u) ž. B]] = Jgx-1 - p(u[ (nn(z))((( ()(()))))(((())))))()()))))) (d) S is a K-simplex). We refer interested readers the Appendix D.I mathematical details. The third term constraining on (p(})²) avoids increment of overall uncertainty during finetuning and thus the generalization ability can be retained. Considering the difficulty of constrained optimization, we convert Eq. I into an unconstrained form and get our final minimizing objective
E Epm (f E
pSEM
{A |mmx((0, (ho + mouT) - hIl- + erain
(L
}.
classification
high distributional uncertainty
unchanged overall uncertainty
(12)
where is hyperparameter. In contrast to previous Bayesian method [9, DUL only encourages high DU rather than overall uncertainty on OOD and explicitly discourages high entropy in the final prediction. The implementation details are in Appendix D.I
6 Experiment
We conduct experiments to validate our analysis and the superiority of DUL. The questions to be verified are Q1 Motivation. To what extent does OOD detection conflict with OOD generalization in previous methods? Q2 Effectiveness. Does DUL achieve better OOD detection and generalization performance compared to its counterparts? Q3 Interpretability. Does the proposed method well decouple uncertainty as expected?
6.1 Experimental Setup
Our settings follow the common practice g I1 20 5 in OOD detection. Here we present a brief
description and more details about datasets, metrics, and implementation are in Appendix B.I and B.2
Datasets. ID datasets PID, We train the model on different ID datasets including CIFAR- 10, CIFAR- 100 and ImageNet-200 (a subset of ImageNet-1K IK (39 with 200 classes). Auxiliary 0OD
datasets PSEM In CIFAR experiments, we use ImageNet-RCC as PSEM ImageNet-RC: a down- train
train sampled variant of the original ImageNet-Iee 1K which is widely adopted in previous OOD detection
7

---

Table 1: OOD detection and generalization performance comparison. Substantial (2 0.5) improve- ment and degradation compared to the baseline MSP [6] (training without any OOD detection regularization) are highlighted in blue or red, respectively. The best and second best results are in bold or underlined. DUL is the only method that achieves SOTA OOD detection performance (mostly the best or second best) without sacrificing generalization i.e., the value of the entire row is either blue or black. Full results with standard deviation and diverse types of corruption in Appendix
pIo /PGEM
Method
Model generalization
OOD detection
ID-Ace
OOD-Acc
AUROCT
AUPR
MSP
96.11
87.35
41.96
89.28
68.00
CIFAR-IO None
EBM (pretrain)
96.11
87.35
$2.45
89.34
75.22
Maxlogits
96.11
87.35
32.90
89.26
74.47
Mahalanobis
96.11
87.35
32.53
93.93
74.96
Entropy
96.04
72.37
6.63
98.72
91.00)
EBM (Linetune)
96.10
79.03
3.61
98.39
94.88
CIFAR-107
POEM
94.32
78.89
3.32
98.99
99.38
ImageNet-RC RC
DPN
95.69
4.28
98.53
WOODS
96.01
80.14
7.12
98.45
92.46
SCONE
95.96
78.50
7.02
98.45
92.46
DUL (ours)
96. 0..00
88.01 200.20
5.89
98.47100002
92.444
DUL' (ours)
06.01
87.53=0.40
3,99
98,28=40 01
98.40
Entropy
95.94
80.51
11.60
97.93
92.16
EBM (finctune)
95. 35
83.67
19.36
87.51
83.63
CIFAR-I0/
POEM
95.44
83.17
24.34
86.83
94.25
TIN-597
DPN
94.39
79.23
17.27
94.92
87.6T
WOODS
95.57
84.68
2.58
98,29
93.08
SCONE
95.19
84,688
8,02
98.21
933.08
DUL (ours)
96.06 00
87
6.87
98.21 $0.01
91. +:022
DUL' (ours)
95.9 25.00
88.10=0.07
10.3g
07.67720.00
98,59
MSP
80.99
53.95
74.63
80,19
42.59
CIFAR-100/
EBM (pretrain)
80.99
55.95
67.42
82.67
49.35
None
Maxlogits
80.90
55.95
60.32
82.30
47.60
Mahalanobis
80.99
55.95
61.51
85.97
56.10
Entropy
80.21
45.48
22,
95.33
82.34
EBM (Linetune)
80.53
48.14
13:47
96.78
87.84
CIFAR-100/
POEM
78.15
42,18
9.89
47.79
98.40
ImageNat-RC
DPN
75.90
50.14
18.36
95.42
74.45
WOODS
80.69
5:1.38
38.15
92.01
71.79
SCONE
80.80
56.73
47.60
89.61
65.29
DUL (ours)
81.30
M6 D6.22=#3:2
12. =2049
96.24
86.72
DUL (ours)
$1.23 00
55.41 =10.54
11.12
95.46*-0.56
96.449--00.3
Entropy
80.15
46.25
93.50
79.81
EBM (finetune)
79.94
50.00
26.87
91.68
80.08
CIFAR-108
POEM
78.68
52,55
$2.71
91.30
94.65
TIN-597
DPN
78.44
47.67
24.99
93.55
81.63
WOODS
79.26
53.13
36. 76.71
92.15
73.12
SCONE
79.63
52,70
35. 60
92.47
73.58
DUL (ours)
80.85
56, 16:1:33
23.32
94.48
80.82
DUT. (ours)
80.50 06
56.22 1.05
22.75 10.76
90.88=00.0
96.33
ImageNet-200/
MSP
85.15
74.84
58.23
86.98
82.27
EBM (pretrain)
85.15
74.84
51.94
88.18
81.75
None
Maxlogits
85.15
74.84
51.62
88,30
81.71
Entropy
84.92
74.75
03.62
80.005
86.02
EBM (finetune)
84.14
73.31
59.73
87.54
82.81
ImageNet- 200 /
DPN
81.87
74.40
63.84
87.18
80.69
ImageNet 800
WOODS
$1.99
74.98
51.71
88.30
81.80
SCONE
84.93
74.91
52.52
88.19
84.50
DUL (ours)
85.65
75.59
49.14 20.18
89.27
85.62
works || 17 We also conduct experiments on the recent TIN-597 20 as an alternative.
When ImageNet-200 is ID, the remaining 800 classes termed ImageNet-800 are considered as
PSEM
train OOD detection test sets PSEM are suite of diverse datasets introduced by commonly
used benchmark 5 In CIFAR experiments, we use SVHN 40). Places365 [41), Textures [42].
LSUN-R, LSUN-C [33 and iSUN [44] as PSEM When pID is ImageNet-200, PPEM consists of iNaturlist 45 Open- Image 44).. NINCO 47 and SSB-Hard [48]. It is worth noting that in
standard OOD detection settings, there should be no overlapped classes between PID PSEM and PSEM otherwise OOD detection is trivial problem. o OOD generalization test sets pooV the original ID test set corrupted with additive Gaussian noise of AO 5), following [4. Besides, we also conduct experiments on CIFARTO-C. IFFAR100-C and ImageNet-C which involve 15 diverse types of different noise corruption (c.g., snow, rain, frost, fog...) in Appendix
train
8

---

Metrics. For OOD detection performance evaluation, we report the average FPR95, AUROC and AUPR to be consistent with [T.. OOD generalization ability is compared in terms of classification accuracy (OOD-Acc). Besides, we also report classification accuracy on II) test sets (ID)-Ac)).
Compared methods. We compare DUL with board line of OOD detection methods, including auxiliary OOD required and auxiliary OOD free methods. o Auxiliary OOD-free methods do not require PIEM during training, including MSP 6 Maxlogits (49], pretrained EBM 88] and
Mahalaobis [8] Auxiliary OOD-required methods explicitly regularize the model on train including entropy- cetropy-rggularzzation (Entropy) 77), finetuned EBM 88, DPN of Bayesian framework [9]
POEM 11 and WOODS [0]. We also compare our DUL to recent advanced SCONE 4 which aims to keep a balance between OOD detection and generalization.
6.2 Experimental Results
Dilemma between OOD detection and generalization (Q1). We validate the dilemma mentioned before in Fig. [ As shown in Tab. []] though many advanced methods establish superior OOD detection performance, their OOD generalization degrades lot. For example, recent SOTA POEM achieves nearly perfect OOD detection performance on CIFARIO 10 when ImageNet-RC serves as train with 3.32% false positive error rate (FPR95). However, its OOD-Acc drops a lot (about 10%) compared to baseline MSP This phenomenon is also observed in other advanced methods. To further detail this phenomenon, finetuned EBM and show the performance on both OOD detection and generalization. As shown in Table 3 when the regularization strength increases, OOD detection performance improves (lower FPR.), while the OOD generalization performance degrades (higher error rate).
reduce the weight of OOD detection regularization terms in Entropy and
OOD detection and generalization ability (Q2). As shown in Tab. [ establishs strong overall performance in terms of both OOD detection and generalization. We highlight a few essential obser- vations: 1) Compared to auxiliary OOD free methods, DUL establishes substantial improvement due to additional regularization on auxiliary outliers. 2) Compared to auxiliary OOD required methods, our method achieves superior OOD detection performance without sacrificing general- ization ability. Meanwhile, previous OOD detection methods commonly exhibit severely degraded classification accuracy, with many cases increasing by more than 10% eπror rate. 3) Comparison to the most related work SCONE [4]. Despite recent advanced SCONE simultancously considering both two targets, we observe that it can be hard to find good trade-off. In contrast, dual-optimal OOD detection and generalization performance is achieved by our DUL. Noted that DUL is the only method that achieves state of- the-art detection performance (mostly the best or second best) without degraded generalization ability (no red values in the entire row). The sensitive-robuss dilemma is no longer observed in our method. These observations justify our expectation of DUL. 4) Combining with existing methods. Besides, to further demonstrate the effectiveness of the proposed DUL, we also add the unchanged overall uncertainty term in Eq.12 12 to the original Entropy and finetuned EBM. The results in Table 2 show that DUL regularization can also benefit EBM. However, combining Entropy with our regularization can not improve the accuracy substantially. This is not surprising, since the target of Entropy (high entropy prediction) and our DUL (non-increased entropy) directly conflict according to Theorem 1. 5) Comparison to methods with an extra OOD detect branch. Different from aforementioned methods, a line of recent OOD detectors [50 51 52 17 employ extra output branches aside from the classification logits (with shared backbone for feature extraction). For these OOD detectors, our theoretical analysis is not directly applicable and further analysis from a feature learning perspective may be needed in future work. However, the proposed DUL is devised in finetune manner. Compared to OOD detectors with extra output branches that requires re-training the classifier from scratch, DUL can be applied to any pre- trained model (c.g.. from torchvision, huggingface), with modest computation overhead.
Visualization of estimated uncertainty (Q3). To evaluate the uncertainty estimation, we visualize the distribution of ID (CIFAR-10) 10) and OOD (SVHN) samples in terms of uncertainty. As we can see in Fig. (b), our DUL establishes a distinguishable (distributional) uncertainty gap between test-time ID and OOD data, which indicates a good sensitiveness for OOD detection. By contrast, the baseline method MSP (Fig 2 (a)) can not effectively discriminate ID and OOD. Besides, we visualize the predictive entropy (overall uncertainty) on covariate-shifted OOD (CIFAR-10 10 with Gaussian noise) in Fig. 2 (c), our DUL yields much lower entropy compared to other methods. Besides, we visualize
the data uncertainty on semantic OOD test data (Textures) when CIFAR-10 10 is ID in Fig. 6.2 The

---

investigated methods are 1) pretrained model training on ID dataset only, 2) finetuned model with OOD detection regularization (ablating the last term Eq. and 3) finetuned model with the full DUL method described by Eq. As shown in Fig. 6.2 to keep the overall uncertainty and enlarge the distributional uncertainty (for OOD detection), the data uncertainty must be reduced. We use 1q from [9] to calculate data uncertainty. The distributional uncertainty is shifted by subtracting that on ID dataset. These results meet our expectation.
(eemancic OOD)
OOD)
Predictive Entropy 12
Ennopy
CIFARIO (ID)
CI (TD)
Damily
Density
-55 Negutive probability
-c.8
-80
Differential eatropy
Noise
(a) MSP
[b) DIIL
(c) Entropy on covariate-hhitee OOD
Figure Visualization of different types of uncertainty estimated by DUL.
Data uncertainty
Distributional uncertainty
Total Uncertainty
Dersity 8
Danaity
0..4
Dersity
0.2
1.2
1.4
1.6
1.8
2.0
2.2
-2
0
2
2
Pretrain
Finetune w/o DUL
Finetune w. DUL
Figure 3: Visualization of uncertainty on semantic OOD test dataset when CIFAR-10 is ID dataset. Without DUL (orange), all three types of uncertainty will increase altogether. In contrast, DUL (green) increases the DU but decreases the AU, which further lead unchanged overall uncertainty.
Table 2: Additional results when equip DUL existing methods i.c., Entropy and finetuned
Table 3: We tune the weight of OOD detection regularization term for EBM well Entropy and report the FPR (OOD detection metric) and error rate (Err, OOD generalization metric). The experimental settings are the same with Table 2.
train is EBM. ID dataset is CIFAR-10. CIFAR-10 IetgeNet-RC. test is the original testset corrupted by Gaussian noise (0,
Model pencralization
Method
I1>-Acc
00D-AAe
FPR
AUC
Entropy
96.04
72.57
6.63
98.72
Eutropy OOD-Err
EBM OOD-Em
EBM (fnetune)
96.10
FPR L
POEM
94.32
78.80
3.32
0
9.55
35.15
9.55
20.57
BBM w. DUL
95.19
87.45
6.17
98.28
10
13.58
8.36
1 10
9.16
14.69
Entropy DUL
96.10
87.41
29.56
95.92
10-3
15.48
6.37
1 10-3
10.32
13.54
DUL DUL" 96.04
96.02
87.53
88:01
5.99
98.47
08.28
x 10-1
x
17.97
18.53
5.71
5.00
1
1 10-1
24.38
16.43
6.11
8.15
Conclusion
This paper provides both theoretical and empirical analysis towards understanding the dilemma between OOD detection and generalization. We demonstrate that the superior OOD detection performance of current advances are achieved at the cost of generalization ability. The theory-inspired algorithm successfully removes the conflict between previous OOD detection and generalization methods. For SOTA OOD detection performance, our implementation assumes that auxiliary outliers are available during training. This limitation is noteworthy for our DUL as well as the most existing SOTA OOD detection methods. We argue that this added cost is minor and reasonable given the significance of ensuring model trustworthiness in apen-enniionmentts. Reducing the dependency on auxiliary OOD data can be an interesting research direction for the future exploration.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (62376193 and 61925602). The authors thank NeurIPS anonymous peer reviewers for their helpful suggestions.
10

---

References
[1] Chunjong Park, Anas Awadalla, Tadayoshi Kohno, and Shwetak Patel. Reliable and trustworthy machine learning for health using dataset shift detection. Advances in Neural Information Processing Systems, 34:3043-3056, 2021.
[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: /606. 06565, 2016.
[3] Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint 21X1::2188.13624, 2021.
[4] Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both out-of-distribun generalization and detection. In International Conference on Machine Learning. 2023.
[5] Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprint 2.0.::23006093001 2023.
[6] Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and -iitr--distribution examples in neural networks. International Conference on Learning Representations, 2017.
[7] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. International Conference on Learning Representations, 2019.
[8] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based duiti--diitiibution detection. Advances neural information processing systems, 2020.
(9] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 2018
[10] Julian Samzela Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In International Conference on Machine Learning, 2022.
[11] Yifei Ming. Ying Fan, and Yixuan Li. Poem: cittof-distribution detection with posterior sampling. International Conference on Machine Learning, 2022.
[12] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't 't know by virtual outlier synthesis. International Conference Learning Representations, 2022.
[13] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey, arXiv preprint 10Xiv22100.11333 2021.
[14] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection neural networks. International Conference on Learning Repre- sentations, 2018.
[15] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrateee classifiers for detecting out-of -iiitr--ddtiittiionn samples. International Conference on Learning Representations, 2018.
[16] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 2019.
[17] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases, 2021.
[18] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting ofiis--iitiiiitti samples and adversarial attacks. Advances in neural information processing systems, 2018.

---

[19] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. -ittt---iitiiittionnn detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[20] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, al. Openood: Benchmarking generalized out-of- distribution detection. Advances in Neural Information Processing Systems, 35:32598-3261 2022.
[21] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv:/907.02890 2019.
[22] Zhaolin Hu and L. Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, 2013.
[23] Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distributionally robust optimization. International Conference on Learning Representations, 2021.
[24] Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distribu- tional robustness with principled adversarial training. International Conference on Learning Representations, 2018.
[25] Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, Lanqing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopu- lation shift via uncertainty-awar mixup. Advances in Neural Information Processing Systems, 2022.
[26] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International conference machine learning, pages 7523-7532. PMLR, 2019.
[27] Yuchen Zhang. Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference Machine Learning. 2019.
[28] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations, 2021.
[29] Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38644 2022.
[30] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:230..12400, 2023.
[31] Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? Advances in neural information processing systems, 2017.
[32] Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning augment distributions for out-of-distribution detection. Advances in Neural Information Processing Systems, 36, 2024.
[33] Hyunjun Choi, Hawook Jeong, and Jin Young Choi. Balanced energy regularization loss for out-of-distribution detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 15691-15700, 2023.
[34] Xuefeng Du, Yiyou Sun, Jerry Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[35] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, 2015.
[36] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. Advances in neural information processing systems, 21, 2008.
12

---

[37] William Yang, Byron Zhang. and Olga Russakovsky. Imagenet-ood: Deciphering modern out-of-distribution detection algorithms. International Conference on Learning Representations. 2024.
[38] Yichen Bai, Zongbo Han, Changqing Zhang, Bing Cao, Xiaoheng Jiang, and Qinghua Hu. Id- like prompt learning for few-shot out-of-distribution detection. arXiv preprini arXiv:2311.15245 2023.
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211 1-222, 2015.
[40] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning. Granada, Spain, 2011.
[41] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2017.
[42] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
[43] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506: 03365, 2015.
[44] Junting Pan and Xavier Giró-i Nieto. End-to-end convolutional network for saliency prediction. arXiv preprint arXiv: 150..01422 2015,
[45] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[46] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7): 12867):1956-199 1981, 2020.
[47] Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In out? fixing imagenet out-of- distribution detection evaluation. In International Conference on Machine Learning, 2023.
[48] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. The semantic shift benchmark. In ICML 2022 Shift Happens Workshop, 2022.
[49] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real- world settings. International conference on machine learning, 2022.
[50] Wenjun Miao, Guansong Pang, Xiao Bai, Tianqi Li, and Jin Zheng. Out-off-istribuu detection in long-tailed recognition with calibrated outlier class learning. Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 4216-4224, 2024.
[51] Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of-distribution detection: Many methods based on ood training data estimate a combination of the same core quantities. In International Conference Machine Learning. pages 2041-2074. PMLR, 2022.
[52] Kai Liu, Zhihang Fu, Chao Chen, Sheng Jin, Ze Chen, Mingyuan Tao, Rongxin Jiang, and Jieping Ye. Category-extensible dustt--dis detection via hierarchical context descrip- tions. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[53] Clément L Canonne. A short note on an inequality between kl and tv. arXiv preprint arXiv:2202.07198, 2022.
[54] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4921 -9930, 2022.
[55] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need? International Conference Learning Representations, 2022.
[56] Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. International Conference on Learning Representations. 2019.
[57] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Conference, 2016.
[58] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. 2016.
[59] Yarin Gal et al. Uncertainty in deep learning. 2016.
14

---

Appendices
Proofs
15
B Experimental Details
17
B.1 Datasets details
17
B.2 Implementation details
18
C Additional Results
20
C.I Uncertainty estimation.
20
C.2 Time-consuming comparison.
20
C,3 Full results with standard deviation.
20
C.4 Results of different types of corruption.
C.5 OOD detection results on individual datasets.
21
C.6 Empirical evidence
24
Discussions
24
D.I Math derivation
24
D.2
Discussion about Disparity Discrepancy
24
D.3 Social Impact
25
A Proofs
First, we recap the definitions of Disparity with Total Variation Distance and Disparity Discrepancy. Definition (Disparity with Total Variation Distance). Given two hypotheses f" F and distribution P, we define the Disparity with Total Variation Distance between them as
disp f) = (F
(13)
where F,, Fy are the class distributions predicted by f', respectively. TVI-)l-) is the total variation distance (TVD), ie., (F(l = E
&
Definition 4 (Disparity Discrepancy with Total Variation Distance. DD with TVD), Given hypothe- sis space F and two distributions P, Q, the Disparity Discrepancy with Total Variation Distance (DD with TVD) is defined as
d=(P.Q)= = sup (dispp( (is FJJFF
(14)
Since TVD is distance measurement of two distribution. It yields the triangle equality. That is, for any distribution Px support on and hypotheses f', f2 and f E we have
disp disp < E.-PATVVVE (1 E._, (VC (15)
disppy
To prove Theorem 1, we need the following lemmas.
Lemma 1. For any F, we have
Epcov (V(F S EPSEM (V + dy(ppov) pSEM
+ A
(16)
where constant independent of U the K K-classes uniform distribution. pCov the Dovariate-shffedd OOD distribution. test is the semantic OOD distribution.
15

---

Proof. Let f be the hypothesis which jointly minimizes the total variance distance between the predicted distribution Fj with uniform distribution U/ taking expectation on pcov and PSEM which is to say
argmin ( pcov (TV JEF
B. PEEM (F)))()
(17)
Set À Re poov
(TV(P)((<)]())) then by the triangle equality we + E yo
tout
have
Epcov (V())
< poor (f. + E Pcov (Po,)IV(F)
- EpSer (F)) - E PSEM TV (FyllU) + disppcov (f.f* + Epcov TV (Fy-1-1
< Epser [V(r)IUU + EPSEN (U)-
disp pars f*) disp f*) + EppooTV((F
< E
TV (Pr](U) + + (Pes pcov) + A.
test
Seat
(18)
n
Intuitively speaking, Lemma demonstrates that if the classifier express high overall uncertainty on PSEM (i.e., the predicted distribution F, is close to uniform distribution), it will also tend to high uncertain prediction on pcov given a limited d (PSEM, pcov).
Lemma 2. [Inequality between KL and TV] For any K-cluss distribution P and Q on (1, K} and K > 0, the following inequality holds
KL(PIIQ) - K 2 TV(P
(19)
2k
Proof. The proof can be found in [53] page 8.
Lemma 3. Denote the OOD detection loss used for MSP detectors as frog, then we have
F } (Crog log K)
E
PSEM TV(F S EpSEM V Leat
2
(20)
where [VI-)l-) is total variance distance (TVD). U denotes uniform distribution support on Y 11,2...K), Creg is defined in [7] is the cross-entropy between predicted distribution Fy(x) and uniform distribution U.
Lemma 2 means that minimizing the OOD detection loss will constrains the predicted distribution to be close to uniform distribution, which is a intuitive and straightforward result.
Proof. In K-classes classification task, for any sample drawn from PSEM have
Greg (f)) KL(U H(U).
(21)
Applying Pinsker's Inequality, the following inequality holds
Lreg(((2)) = KL(F)IIU) + H(U) > 2TV H(U).
(22)
Noted that H(U) = log K, we can re-write above inequality
(F)(())) S / (Con (J())) - log K).
(23)
Then, by taking expectation PSEM we can get the result.
Now are ready to present the proof of Theorem 1.
16

---

Proof. By the definition, the generalization error can be written as
GError (ʃ) : (x. PRO LCE(J())),))
(24)
(z.g)) xx
+ H (Ptrue))
where Perue is the target distribution given input i (i.e., the true class distribution) and LCB(-) denotes the cross-entropy loss.
Applying Lemma 2, for any x we have
(PV K
K (LePrrrellP
(25)
2x
By the sub-additivity of TVD, we have
LL (P [x)) 2
(TV [fj)) +
2x
(26)
[P [
Taking expectation PCOV, have
pgrr (f) =E (z.g)) KL(P Fy(x)) + H (Prree)
1
(2.))~ [TV (Vw.r/oooVVVVVV..... -V)
K
+ Eggoo H (Prrue)
2x
4
(27)
Applying Lemma and Lemma 2,
GError (f) = -E Epgoy K KL[P [F]] + E peoo H ((P (Ptrue)
E
peov x3
(V true 10) (V(F(())IIU) + REggy ((Prrue)
2x
2
poov Xy
(Vttrruelll
E Epcoy TV(F,IIU) + + Epoy H (Ptrae)
2k
2k
1
2
poov (Ptrralll
E. Pere (V
2k
2k
1
X
ds (P,Y
PSEM)
Epgor H (Ptrue)
2k
2x
4
1
E Eerrm / / (Creg log K)
E. peov TV (V(rrrue (U)
2k
1
d (COO
PSEM)
4
+ Eppoy H H(Parue))
2x
(28)
Given the fact that XX Ptrue both fixed, ((Prrue) and TV (V(rrruel(U) UT) constants for each x. Finally, we get
1 d=(Pooo PSEM),
GError pooy (f) 2 C
- pSEm V / (Free log K)
(29)
2x
2x
where C= 2ne (Epgov (V X 8) Pog ((Prrue) is constant.
B Experimental Details
B.1 Datasets details
ID datasets PID ID datasets chosen following common practice in OOD detection. We use CIFAR-10, CIFAR- 100 and ImageNet-200 as pín ImageNet-200 is a subset of the original ImageNet- 1K introduced by [20, 5].
Auxiliary OOD datasets PSEM For CIFAR experiments, we use ImageNet-RC and TIN-597 as auxiliary datasets. ImageNet-RC is down-sampled variant of the ImageNet-IK, which consists
train"
17

---

of 1000 classes and 1,281,167 images. We also conduct experiments on TIN-597 as an alternative for ImageNet-RC. TIN-597 is introduced by recent work [5]. The resolutions of ID and auxiliary samples are both 64 x 64. For ImageNet experiments, we use subset of ImageNet-11 1K consisting of 200 classes as ID datasets. The remaining images belong to other 800 classes are utilized as auxiliary datasets. The resolutions of ID and auxiliary images are both 224 x 224.
OOD detection test datasets PSEM, In CIFAR experiments, following standard practice [8], we use SVHN [40], Textures [42], Places365 [41], iSUN [44], LSUN-C and LSUN-R [43] to evaluate the OOD detection performance, The SVHN test set comprises 26.032 color images of house numbers. o Textures (Describable Textures Dataset, DTD) consists of 5,640 images depicting natural textures. Places365 dataset consists scenie images of 365 different categories. Each class consists of 900 images. o The iSUN dataset is subset of the SUN database with 8,925 images. The Large-scale Scene Understanding dataset (LSSUN) comprises a testing set with 10,000 images of 10 different scenes. LSUN offers two datasets, LSUN-C and LSUN-R. In LSUN-C, the original high-resolution images are randomly cropped into 32 32. Meanwhile, in LSUN-R, the images are resized to 32 x 32. In ImageNet experiments, we follow the settings of [5], where OpenImage-O [54], SSB-hard [55], Textures [42], iNaturalist [45] and NINCO [47] are selected as OOD detection test datasets. OpenImage-O contains 17632 manually filtered images and is 7.8 x larger than the ImageNet dataset. o SSB-hard is selected from ImageNet-2 IK. It consists of 49K images and 980 categories. Textures (Describable Textures Dataset, DTD) consists of 5.640 images depicting natural textures. o iNaturalist consists of 859000 images from over 5000 different species of plants and animals. NINCO consists with total of 5879 samples of 64 classes which are non-overlapped with ImageNet-IK.
OOD generalization test datasets pcov, Following previous work [4], we corrupt the original test data with Gaussian noise of zero mean and variance of in the main paper. In appendix, we conduct additional experiments involving CIFARIO-C, CIFAR 100-C and ImageNet-C [56] with 15 diverse types of noise.
B.2 Implementation details
B.2.1 CIFAR experiments.
We use WideResNNet44-100 [57] as the backbone network, which comprises 40 layers. The widen factor is set to 10. We use SGD optimizer to train all methods with dropout strategy. The dropout rate 0.3. The momentum set to 0.9 and weight decay is set to 0.0005.
Pretraining details. The pretrained model is obtained by training WideResNet-40-10 for 200 epochs with an initial learning rate of 0. We decay the learning rate by a factor of 0.2 the 60-th, 120-th, and 160-th epochs. Batch size is set to 128.
Finetuning details. o For Entropy and EBM, we finetune the pretrained model for 20 epochs with an initial learning rate of 0.001, utilizing a cosine annealing strategy to adjust the learning rate. Following the official implementation. the weight of OOD detection regularization term is set to 0.5 and 0.1 for Entropy and EBM (finetune) respectively. The hyperparameters MID and mOOD in EBM regularization learning are set to -25 and -7 respectively. The ID batch size is 128 and the OOD batch size is set to 256. o For SCONE, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing cosine annealing strategy to adjust the learning rate. The batch size is 32, the OOD batch size is 64. The margin of the OOD detection boundary is set to 1. To be aligned with most previous works in OOD detection and generalization, we assume PCOV is unavailable during finetuning. o For WOODS, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing& a cosine annealing strategy to adjust the learning rate. The ID batch size is 32, the OOD batch size is 64. Other hyperparameters settings are consistent with SCONE, o For DUL, ag is set to 12. While finetuning on CIFARIO, the MID and MOOD are set to 10 and 30 respectively. The weight X, are set to 0.3 and 2. We train for 20 epochs with an initial learning rate of 0.00005, utilizing a cosine annealing strategy to adjust the learning rate. While finetuning on the miD and MOOD are set to 10 and 30 respectively. The weights X, are set to 0.05 and 2 respectively. We finetune for 20 epochs with an initial learning rate of 0.00005, utilizing a cosine annealing strategy to adjust the learning rate. While finetuning on CIFAR 1000 (00/mmaeeeet-RCC we set hn- = 0. The min and MOoD are set to -430 and -370 respectively. We train for 30 epochs with an initial learning rate of 0,0001, utilizing cosine annealing strategy to adjust the learning rate.
18

---

The weights x, are set to 0.1 and respectively. For CIFAR-I00//aaeeN 100////0///eeeRR..... we set 7 = 2 and otherwise T = 1. For DUL', we use Thompson sampling strategy [11] for OOD informativeness mining. The sampling hyperparameters are consistent with that of POEM.
Training from scratch details. For POEM, we train from scratch for 200 epochs with initial learning rate of 0.1, and decay the learning rate by factor of 0.2 at the 60-th, 120-th, and 160-th epochs following [57]. The ID and OOD batch size are set to 128 and 256 respectively. Following the official implementation, the pool of outliers consists of randomly selected 400,000 0,0000 samples from auxiliary datasets, and only 50,000 samples (same size as the ID training set) are selected for training based on the boundary score. o For DPN, we train for 200 epochs with an initial learning rate of 0.1, and decay the learning rate by a factor of 0.2 the 60-th. 120-th, and 160-th epochs. The Dirichlet parameters are calculated by performing ReLU plus one on the model's 's outputs, i.c., = ReI.U((())) + og is set to 15 and 12 respectively when training on CIFARIO and CIFARTO0. The auxiliary datasets are ImageNet and TIN-597. The ID) and OOD batch size are set to 128 and 256 respectively. When training on CIFAR10O/TIN: 0.05. In other cases, À is set to 0.5.
the OOD regularization weight À set to
B.2.2 ImageNet experiments.
We use ResNet18 [58] as the backbone network. We use SGD optimizer to train all the models. The momentum is set 0.9.
Pretraining details. The pretrained model is obtained by training ResNet18 for 100 epochs with an initial learning rate of 0.1, utilizing cosine annealing strategy adjust the learning rate. The weight decay is set to 0.0001. Batch size is set to 64.
Finetuning details. For Energy regularized learning, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.001, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is detection regularization term are set to 0. Specifically, the mii and mOoD in energy regularization method are set to -25 and -7 respectively. The ID batch size is 64 and the OOD batch size is set to 128. o For Entropy, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.001, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0001. The ID and OOD batch size are set to 64 and 128 respectively. Following the official implementation, the weights of OOD detection regularization term are set to 0.5.o For SCONE, we finetune the pretrained model for 10 epochs with an initial learning rate of 0. 0.0002, utilizing a cosine annealing strategy adjust the learning rate. The weight decay is set to 0.0005. The batch size is 32, the OOD batch size is 64. The margin of the OOD detection boundary is set to 1. To be aligned with most previous works in OOD detection and generalization, we assume PCOV unavailable during finetuning, o For WOODS, we finetune the pretrained model for 10 epochs with an initial learning rate of 0.0002, utilizing cosine annealing strategy to adjust the learning rate. The batch size is 32, the OOD batch size is 64. Other hyperparameters of WOODS are consistent with SCONE. e For our DUL, the Dirichlet parameters a are calculated by performing ReLU and exp operation on the model's 's outputs, i.e., a exp (ReLU((())) measure the distributional uncertainty by the strength of Dirichlet distribution. X, y are set to 0.1 and 4 respectively. We set 7 = 1 in large-scale ImageNet experiments.
0.0001. Following the official implementation, the weights of OOD
For numerical stability on large scale benchmark, we
Training from scratch details. o For DPN, we train ResNet18 for 100 epochs with an initial learning rate of 0.1, utilizing a cosine annealing strategy to adjust the learning rate. The weight decay is set to 0.0001. Batch size is set to 64. The Dirichlet parameters a are calculated by performing ReLU plus one on the model's outputs, i.e., = ReLU(f(x)) + 1. The ID classification loss is set to KL-divergence between predicted class distribution under Dirichlet prior and target distribution because of the inconvenience of directly setting og. The target distribution is obtained by label smoothing strategy with parameter of 0.01 [9]. The weight of regularization term applied on OOD auxiliary samples is L.
19

---

Algorithm 1: Pscudo Code of Decoupled Uncertainty Learning (DUL)
Input ID data piD, auxiliary outliers PSEM classifier fe, pretrained on PID
train
Output: finetuned classifier fø
Initialize = Oo:
for each iteration do
Obtain ID sample (x, from piD and auxiliary outlier 7 from PSEM.
Update model parameters 0 by minimizing objective defined in Eq. 12;
end
C
Additional Results
C.1
Uncertainty estimation.
We add Gaussian noise with zero mean and varying variance e on CIFAR-10 and investigate the estimated distributional uncertainty and overall uncertainty. Distributional uncertainty is measured by differential entropy. It clear that with DUL regularization, the prediction yields a low overall uncertainty and high distributional uncertainty covariate-shifted data. We conduct experiments CIFAR- MAAR-10/maageee-RRR and CIFAR-107TIN-559 tabular results are shown Tab. 4.
Table 4: Mean value of estimated uncertainty on CIFAR-IO-C with varying severity of Gaussian noise with zero mean and variance of c.
PX/PX"
Uncertainty
DUL
0.0
2.0
4.0
6.0
8.0
10.0
-21.33
-18.94
-15.62
-13.71
-12.91
-12.85
Distributional uncortainty
CIFAR-10 ImageNct-RC
-21.23
-19.42
-16.96
-15.47
-14.85
-14.58
0.04
0.47
1.31
1.93
2.20
2.28
Total uncertainty
0.03
0.17
0.48
0.77
0.98
1.14
-20.44
-20.14
-18.29
-16.23
-14.71
-13.72
Distributional uncertainty
CIFAR-10 TIN-597
-21.48
-20.88
-19221
-17.70
-16.57
-15.78
0.06
0.15
0.51
1.06
1.55
1.92
Total uncertainly
0.04
0.08
0.18
0.34
0.51
0.68
C.2 Time-consuming comparison.
We compare the time-cost of proposed DUL other raining-requiiet OOD detection methods in Tab. We run all the experiments on one single NVIDIA Geforce RTX-3090 GPU. Compared with other OOD detection methods in finetune manner, DUL does not introduce noticeably extra cost of computation.
Table 5: Average execution times (s) per epoch of training required OOD detection methods. Compare other OOD detection methods, DUL does not introduce noticeable computational cost.
Method
CIFAR- CIFAR-I0/InageN 10/ImageNet-Re
CIFAR- 10/TIN-997
CIFAR- 10W/mageNNttRC CIFAR-IOW 100/TIN-597
EBM (finetune)
354,87
229.12
355.70
112.47
Entropy
355.98
140.88
1108.43
120.85
DPN
842.05
75.62
841.69
80.98
POEM
615.51
483.04
825,87
500.01
WOODS
808.77
291.37
906.67
282.50
SCONE
911.83
183.57
1169.33
160.00
DUL
329.58
101.08
925.62
100.43
DUL*
598.68
597.82
544.26
595.25
C.3 Full results with standard deviation.
Full results with standard deviation are presented this section. In CIFAR experiments, we report the mean and standard deviation in 5 random runs. In ImageNet experiments, report the mean and
20

---

standard deviation in 3 random runs to be consist with [5]. CIFAR experimental results are shown in Tab. 6. Large-scale ImageNet results are shown Tab, 7.
Table 6: OOD detection and generalization performance comparison with standard variance. Marginal improvement and degradation (2 0.5) compare to the baseline method MSP are highlighted in blue or red respectively. The best and second best results are in bold or underlined, DUL is the only method achieves state-of-art OOD detection performance (mostly the best second best) without trade-offs on generalization i.c., the value of entire row is either blue or black.
pID / train
Method
ID/OD generalization
detection AUROC1
ID-Acc.
OOD-Ace.
FPR
AUPR
MSP
96.11 96.1120.09
87.35=0.3
41.9643385
89,28=1.12
68. 68.00=2.11
CIFAR-I0 Only
EBM (pretrain)
96. 96.11=0.09
87.35=0.08
32.45
89.34#1221
75 75.22#266
Maxlogits
96.1120.09
87.3520.06
32.90
89.26*1/91
74.4752235
Mahalanobis
96.11 16.01
87.35=0.06
32.5849001
93.98=2.08
74.96#7.47
Entropy
96. 06.04-0.14
72.5773..7
8.632200
75.014
14.00210000
EBM (pretruin)
0.23
79.03
3.61 4:0
98. 98.39
$5.85%5 +5.8800.01
POEM
94,32=034
78.89=22.8 as
3.32
98.99 2011
20.12
CIFAR-10
DPN
95.6920.17
85.52#0.51
4,28
98.53+0.17
94, $4.99=0.000
Net-RN
WOODS
96.01
80.14
7.12
98.4020031
92.92
SCONE
95.96
78. 78.80 78.80=11.7
7.02
98.45
92.46*0055
DUL (ours)
96. 96.02=0.07
88.01
88.01#10.44 110.04
0.89#0.30
98. 98.1750:12
92. 02.41=1:34
DUL (ours)
96. 36.0420.05
87. 27.5322077
5. 5.99 25
98, 08,28
98 28.4402.000
Entropy
95.94 25.91
80.51 80,51#0000
11.60
97. 97.93400:5
92. 92.16
EBM (pretrain)
95.38 20115
93.6721-44
1..92
87.51 1:353
23.63321.75
POEM
95.44=0.18
83.17#1.50
24.34 4:248
86,83
94.25 11.22.20.0
CIFAR-10
DPN
94.3920.58
79.23
17.27
94.9242066
87.67#0.3
TIN-597
WOODS
95.57
83.12
7.58
98.29
93.3920030
SCONE
95. 95.19 19.1970777
84.68
8. 8.02#0.92
98, 08.22
93.08 $3.08800.00
DUL (ours)
96.06 00.0
87,93
6.87 t0.82
98.2 18021
91.25 91.29#1:15
DUL (ours)
95.94
88.10 10.27
10.34
97.67450.09
98.59 20021
MSP
80. 00.99
55, $3.95
74.63
80,19
42, 59.2797
CIFAR-100
EBM (pretrain)
80.99=0.16
55.95
67.42
82
49 35.35 3-.0000
Maxlogits
80.9920.16
55.95
69.37
Only
Mahulanobis
80.90 =0.18
55.95#1.88
61.51 61.51 43302
85,97#11.2
36 36.10
Entropy
80.2 80.21 20.00
45.48 20.78
29 1:152
05.33
82.31 22.31111.11
EBM (finctune)
80.53=0.22
48.1 48.14#00.3
13,47 4:4.4
26.78
87.84
POEM
78.15
42. 12.18425 189
9.89
97.79
98.40
CIFAR-100
DPN
78, 78.9
50.1 50.14
18.36
95.42440.17
74.45
ImageNet-RC
WOODS
80.00 =0.69
54.38 54.38*44442
38. 38.15 1:.99
92.01
71.79*7995
SCONE
80. =0.30
56.73359.06
47. 44.607
89.61 80.61
65 29.2992...0
DUL (ours)
81,30 0000
56, 56.27
12.49 Aovaaa
05.24
86, 72
DUL (ours)
81 22:22
50.07
11.255:1.00
95.33
415 20.42
Entropy
80.1520-17
46.25 4M
26.88
93.50 26
79. 19.81 21133
EBM (finetunc)
79.91
50.00=0.50
26.8771113
91.68+0.44 45
80.05
80.08#0.00 (08-0.77
POEM
78.68=0.13
52.5g=1.06
32.71 4:0.06
01.30
94. #5.65500.49
CIFAR-100
DPN
78. 78.4420222
47.672000
25.02
93.55
81.6 $1.63*1122
TIN-597
WOODS
79.26
53. 58.13 132229977
36.71 36.71 A7702
92, 42-155
73 73.4244.72
SCONE
79.5
52, 7..005
35.60
92.4772..19
73. 23..88
DUL (ours)
=0:8
56. 19
23.3241:11
94,48200.0
80 82
DUL (ours)
80.5020.25
1.
20.85
90.88=0.28
96.33 #0.00
C.4 Results of different types of corruption.
We conduct additional experiments CIFARIO-C, CIFAR100-C and ImageNet-C with 15 dif- ferent types of corruption. The results validate that the proposed method can improve the overall performance under different types of corruption.
C.5 OOD detection results on individual datasets.
We provide OOD detection results of DUL on each individual OOD detection test dataset in Tab. and Tab. 9, based on the checkpoint with random seed 1.
21

---

Table 7: OOD detection and generalization performance comparison with standard variance. Substan- tially improvement and degradation (2 0.5) compare to baseline method w.r.t. MSP are highlighted in blue or red respectively. The best and second best results are in bold or underlined. Similar with CIFAR experiments, DUL establishes strong OOD detection performance (always the best second best) without degraded generalization i.e., the entire row is either blue or black.
plD /PEEM
Method
ID/OD Generalization
OOD Detection
ID-Ace. f
OOD-Ace. 1
FPR
AUROC1
AUPR 1
MSP
85.15 15
74.8420.47
58.234h1.54
86.98=0244
82.2730.32
FBM (pretrain)
85.15 15.15
74.84
51.9440.82
88. 88.18=0.11
84.75.00.06
Maxlogits
85 15.55 15..33
74.84*0.47
51 62.0..0
88.30
84.71
ImageNet-200
Entrupy
84.92
74.76-
53.62
89.05
85,02
ImageNet-800
EBM (finetune)
84.14 84.14#0.11
73.31
59. 43.055
87,54 87,54=0.03
82.81 82.81 100.1
DPN
84.87.0.30
74.40--0..0
63. 14.004
87, 87.18-0.16
80.69-40.35
WOODS
84.99
74.98 2o.4n
51.7142.84
88,30
84.80 40.98
SCONE
84.9340.71
74.91
52.5222..4
88, 88.19=0.41
84.50
DUL (ours)
85.65 85:65
75.59
49.14 20:15
89.27
85.62 20.00
Table 8: OOD detection results of DUL on each individual OOD detection test dataset.
plD ppsis
LSUN-crop AUROCT
Places365
LSUN-resize
iSUN
Texture AUROCT
SVHN
Method
FPR.
AUROC1
FPR]
AUROCT
AUROC1
FPR
AUROCT
CIFAR-10
DUL
6.75
98.75
15.55
96.34
0.00
99.63
0.00
99.57
3.20
98.89
8.35
98.24
ImageNat-RC
DUL
12.79
98.08
14.99
96.11
0.00
99.49
0.00
99.45
1.72
98.79
4.84
98.47
CIFAR-10
DUL
0.90
99.47
22.40
95.21
0.10
99.52
0.30
99.45
8.00
97.94
5,90
98.52
TIN-597
DULJ
3.48
91.48
99.78
9.69
CIFAR-100
DUI
44.65
83.73
21.05
79.
0.00
99.63
0.00
99.61
3.05
98.53
5.00
15
ImageNet-RC
DUL
47.75
81.29
14.92
95.08
0.00
0.00
99.46
1.70
98.70
4.44
98.02
CIFAR-I00
DUL
69
3,80
TIN-597
DUL
6.25
98.55
78.95
64.09
0.02
99 84
0.00
99.83
39.79
85.02
5.61
98.51
Table OOD detection results of DUL on each OOD detection test dataset.
pID Praii
Method
Openlmage-O O
SSB hard
Textures
iNaturalist
NINCO
FPR
AUROCT
FPR.
AUROCT
EPRL
AUROCT
FPR,
AL ALROCT
FPR
AUROCT
ImageNet-220095 200/800
DUL.
49.68
91.31
72.40
80.60
30.76
92.98
33.21
94.76
59.92
86.61
Table 10: Classification error rate comparison CIFARIO-C. ID dataset is CIFARIO.
Noise Shor
Blu Metion
Weather
Digina
Mathod
Clansa
Impul.
Defis:
Cibes
Zeom
Snew
Frret
Fag
Rrih.
Cantr:
Fiaetic
Pieel
JPFG
Avg.
MEP
53.35
30,30
44.07
15.61
42.35
19.33
19.53
14.30
18.02
9.51
5,09
15.19
13:45
22.71
18.61
23.64
HBM (peetmin)
19.35
18.03
9.51
13:45
Manlogiie
19.35
18.03
11:45
Mahtalanobis
39.58
42.35
18.02
13:45
Faimpry EBM
31.72
11.10
14,41
18.76
26.04
POEM WOODS SCONE
ImageNet
23.16
22.64
14.30
13.57
22.33
30.40
62.36
49.09
60.77
15.56
74.37
19.25
20.16
18.79
25.41
9.84
5.45
16.76
14.16
24.07
20.45
20.00
63.20
50.01
61.96
15.61
77.16
18.65
20.24
19.77
26.43
9.82
5.54
16.85
14.09
24.46
20.64
20.64
DUUL (Oans)
53.46
40.11
4371
15.02
43.77
19.49
19.80
14.27
18.07
9.82
5.56
18:40
13.56
22.65
18.69
21:71
Entrgy
65.68
$2.08
56.40
15.66
$1.96
22.73
24.43
18.57
24.64
10.56
5.51
16.35
15.88
25.19
41.25
30,00
EBM (inemune)
58.95
$4.39
52.91
13.30
67.25
23.99
24.33
17.00
12.15
12.15
6.07
1977
17.11
26.85
31.51
28.59
42.55
47.90
27.19
30:41
20.46
26.0
14.55
7.32
18.33
27,40
75.23
TEN-597
$1.06
30.35
32.15
11:39
31.2
87
59.83
17.34
27.17
13.47
16.17
DUT
24.06
22

---

Table 11: Classification error rate comparison on CIFAR100-C. ID dataset is CIFARI00.
Nexse Shoe
Blhr
Weather
Digitl
Mathod
passt
Gama.
Impul.
Defoc:
Glass
Merion
Zoom
Snew
Frost
Her.
Centrr
Flastic
Piool
JPMB3
Arg.
MSP
76.95
67.70
74.51
56.08
75.49
40.69
41.39
40.52
46.65
31.44
22.59
40.29
35.31
42.63
44.03
45,00
H0M (peetrann)
Masdogris
Novz
76.99
96.9%
67:10
m0
14.51
14.51
16.08
56.08
75.40
78.40
42.09
42.09
41:35 199
41:39
+0..2
40.53
48.05
05
31.44
11 14
22.39
09.59
60:25
60:20
15.31
15.31
43.55
42.85
44.95
48.9%
48.00
48.00
Mahtalanobis
76.95
67.70
74.51
36,08
78.49
40.09
41.39
40.52
46.65
31.44
22.59
40.29
35.31
42.83
44.93
48.03
Enirgy EBM (inciune)
38.19
45.50
39.77
37.19
$2.39
73:51
90.55
47.29
39.15
37.65
53.39
POEM
ImageNia-IICC
WOODS
77.47
40.92
41.93
47.05
23.30
36.21
46.16
48.75
SCONE DUL (Ous)
67.30
40.91
41.59
00.22
31.35
23.09
40.10
42.42
45.42
47:922
77.23
67.95
75.13
55.83
78.32
39.74
40.64
39.76
13
30.96
22.45
39.17
34.81
43.03
44.64
47.73
Entrayy EBM (linclun))
85.34
75.44
79.77
38.02
$2.74
42.23
44.17
44.52
53.35
32.80
23.67
38,00
37.85
44.55
62.91
52.22
$1.34
72.78
78.14
$7.55
50.00
42.94
43.64
43.70
51.11
33.52
23.92
40.19
38.13
45.05
13.27
32.33
DPN
81.81
72.87
79.09
39.61
$1.25
44.66
46.21
45.42
$2.80
34.28
25.64
40,59
39.15
47.66
76.40
53.83
POUM
TIN-092
78.85
20.33
08.08
7795
41.01
44:21
01.00
51.23
14
25.17
63:00
17
17.28
45:81
$1.55
51.84
91.14
72.39
38.50
79.00
43.23
44.99
49:21
40.33
39.01
41.19
47.80
50.17
SCONE
72.30
77.19
37.91
78.70
43.99
42.81
39.26
48.34
49-95
40.79
Table 12: Classification error rate comparison on ImageNet-C. Here we test compared methods on subset of the original ImageNet-C consisting of 200 classes. ID dataset is ImageNet-200.
Noise Shot
Blur Motion
Weather Frost
Digital
Method
PrEM
Impul
Defice
Glass
Coutr.
Piacl
JPBG
Avg.
MSP EBM (pretnin) Maxlopits
70.7
715
56.01
55.5
67.0
56.5
$3.3
46.3
50.3
51,9
$7.9
None
52.2
52.2
70.7
70.7
715
715
56.0
56.0
58.5
55.5
53.5
$2.5
54.1
54.1
67.3
67.3
67.0
67.0
63.5
63.5
56.5
56.5
53.3
53.3
46.3
46.3
50.3
50.3
$1.9
51,9
57.9
57.9
Estrogy EBM (linctune
51.3
71.2
71.7
54.4
54.6
51.8
53.2
66.9
66.2
62.7
55.8
51.4
45.1
49.7
51.1
57.3
$2.9
12.2
72.8
56.0
56.2
53.5
54.1
67.8
67.4
64.0
57.0
52.2
46.5
51.4
52.9
59.5
DEN WOODS SCONE
ImngeNes-9100
31.4
$1.9
69.2
69.4
69.6
70.0
55.2
56.5
54.8
55.0
51.9
$2.9
$2.9
53.1
66.2
65.5
65.3
66.0
62.5
62.3
55.6
55.5
54.5
52.6
43.5
46.0
49.7
49.7
51.4
51.3
$7.2
$7.0
51.6
69.4
70.0
55.4
55.0
52.1
55.1
66.3
66.0
62.5
55.7
53.0
45.8
49.9
51.4
57.]
DUL (Ouns)
31.1
69.1
70.5
55.1
54.5
51.8
524
66.2
65.9
62.5
55.7
53.0
45.4
49.4
50,9
36.9
Table 13: Comprehensive comparison involving 15 different types of corruption from commonly-used domain adaption benchmark [52]. Substantial (2 0.5) improvement and degradation compared to the baseline MSP [6] are highlighted in blue or red respectively. DUL is the only method that achieves SOTA OOD detection performance without sacrificing generalization i.e., the value of the entire row is almost black or blue. The best or second best results are highlighted in bold or underlined. MD is the shorthand of Mahalanobis.
OOD generalization (Faror rate J)
ODD detection
Mothod
Irain
teain Gousss Shot Impul. Defoc, Glass
Motion Zoom
Snow
Frost
Fog
Brit, Contr. Elast,
Piael IPEG
Avg.
[FRR
AUC1
MSP
77.0
67.7
40.7
40.5
40.3
35,3
48.0
89.3
EBM
None
77.0
67.7
74.5
36.1
78.5
40.7
41.4
40.5
46.7
31.4
22.6
40.3
35.3
42.8
44.9
4%.0
32.5
89.3
Maxingit
77.0
67.7
74.5
36.1
78.5
40.7
41.4
40.5
46.7
31.4
22.6
40.3
35,3
42.8
44.9
48.0
32.9
89.3
MD
67.7
78.5
40.3
93.9
Eatropy
81.7
72.9
87:0
10:22
24.0
45.
539
35.1
24
50.8
31.2
47.6
45.8
52.44
6.66
V8.7
EBM (FT)
Image-ee.RCC
82.3
73.5
90.6
39.2
90.7
41.4
44.4
47.3
$4.9
31,3
21.7
39.2
37.7
51.5
49.3
53.1
3.6
98.1
DPN
77.6
68.6
83.6
39.7
86.8
43.3
44.9
47.0
53.7
36.4
26
43.5
37.6
47.5
44.5
52.0
4,3
98.5
POEM
83.7
85.0
52.5
43.4
42.5
99.0
WOODS
CIFAR-
77.5
68.5
77.5
36.4
80.6
40.9
41.9
41.3
47.1
31.4
23.3
39.7
36.2
42.9
46.2
48.8
7.1
98.5
SCONE
76.4
67.3
75.3
36.2
77.0
$0.9
41.6
40.2
45.5
31.4
23
40.1
35.9
42.4
45.4
47.9
7.0
98.5
DUL (Ours)
71.2
68.0
75.1
35.8
78.5
39..
39.3
31.0
22.4
39.2
34.8
43.0
47.7
5.9
98.5
Entropy
83.31
75.4
79.3
38.0
44.5
35.0
37.9
TT.6
97.9
HBM (FT)
81.2
72.38
73.
57.6
80.0
42.9
43.6
43.7
51.1
33.5
23.9
40.2
38.1
45.1
73.3
52.3
19.4
87.5
DPN
TIN- 507
81.8
72.9
79.1
39.6
81.3
44.9
46.2
45.4
52.4
34.3
25.6
40.9
39.2
47.7
76.4
53.8
17.3
94.9
POEM
78.9
70.3
43.4
$2.6
WOODS
81.1
72.4
76.5
306
79.0
4..
44.6
42.4
49.2
33.0
24.3
40.2
39.0
41.2
47.8
50.22
7.6
98.3
SCONE
80.9
72.3
77.2
37.59
78.7
42.1
13.4
12.8
19.5
32.2
24.0
39.3
38.0
11.8
48.3
50.0
8.0
98.2
DUL (Ours)
77.0
67.7
74.2
36.2
78.4
40.3
41.3
40.2
46.3
31.4
22.9
40.8
35.3
42.7
45.1
4%.0
6.9
98.2
MSP
None
52.2
70.7
71.5
56.0
55.5
$2.6
54.1
67.3
67.0
63.5
56.5
$3.3
46.3
50.3
$1.9
57.9
58.2
82.3
ERM
52.2
70.7
71.5
56.0
55.5
$2.6
54.1
67.3
63.5
56.5
53.3
46.3
50.3
51.9
57.9
32.5
89.3
Maslogits
ImoopNNr-200 2000
52.2
70.7
71.5
56.0
55.5
52.6
54.1
67.3
67.0
63.5
56.5
53.3
46.3
50.3
51.9
57.9
51.9
88.2
Eatropy
ImageNeu 800
31.3
71.22
71.7
54.4
$4.66
51.8
55.
1669
66.2
62.7
35.8
51.4
45.1
40.7
51.1
37.1
51..6
19.1
EBM (FT)
52.9
72.2
72.8
56.0
56.2
53.5
54.1
67.8
67.4
61.0
57.0
52.2
46.5
51.4
52.9
58.5
59.7
87.5
DPN
52.4
46.0
57.2
87.2
WOODS
51.4
69:4
70.0
55.2
54.8
51.9
52.9
662
66.0
62.5
55.6
52.6
45.6
19.7
51.3
57.0
$1.7
88.3
SCONE
51.6
69.4
70.0
55.4
55.0
52.1
33.1
66.3
66.0
62.6
55.7
53.0
45.8
49.9
51.4
57.1
52.5
88.2
DUL (Ours)
51,0
69.1
70.5
S5.L
54,5
51.5
32.4
662
65.9
62.6
55.7
53.0
45.4
49.4
50.9
56.9
49.1
89,3
23

---

C.6 Empirical evidence
Here we provide empirical supports our intuition about energy- based OOD detection regular- ization. We calculate the entropy of predicted distribution before and after finetuning with Energy regularization [8]. The results show can support our claim in Section 4.
Table 14:
Predictive entropy of predicted distribution on covariate-shifted OOD dataset before and after finetuning with energy-based OOD detection regularization [8].
PID
PSEM
Before
After
CIFARIO CIFARIO CTFAR 1O0 CIFARIOU
ImageNet-RC TIN-597
0.11
1.05
0.11
0.14
ImageNet-RC
0.92
3.39
TIN-597
0.92
1.15
D Discussions
D.1 Math derivation
The proposed DUL calculates differential entropy as OOD detection mea- surement. Here detail how to calculate the differential entropy of Dirichlet distribution. The following derivation of differential entropy taken from [9]. The differential entropy Dirichlet parameterized by is calculated by
Differential entropy.
hp(p4>=]] = / p(ulz) In(p(ulz))dd
K Σ II((aw)= ImI((a)) Σ -) (v(ax) - (aao)
(30)
where on the strength of Dirichlet, i.e., ag => agg. 0g denotes the k-th element a. the Gamma function and vi is the digamma function. Here we provide PyTorch implementation on how to calculate distribution uncertainty measured by differential entropy.
def diff_entropy (alphas):
alpha0 torch, sum (alphas, dim-1)
return torch. sum torch. 1gamma (alphas) (alphas- 1) (torch. digamma (alphas) torch digamma (alpha0). unsqueeze
dim=1) - torch. 1gamma (alpha0)
logits model
alpha torch. Relu (logits)+
diff entropy diff_entropy (alpha)
We refer interested readers to [9] and Gal's PhD Thesis [59] for more detailed math derivations.
D.2 Discussion about Disparity Discrepancy
In section 4, we claim that limited disparity discrepancy between test-time semantic OOD and covariate-shifted OOD is practical. Here provide some empirical evidence and discussion to support such a claim.
The key challenge of OOD detection lies in identifying ID-like semantic OOD. As mentioned in recent works [38], effectively distinguishing between the most challenging OOD samples that are much like in in-distribution (ID) data is the core challenge of OOD detection. Recent works regularize models on ID-like OOD to enhance the OOD detection performance. Since the ID-like semantic OOD samples more difficult to be detected and more informative. For example, NTOM [17] and

---

POEM [11] utilizes greedy and Thompson sampling strategies to find semantic OOD samples which are more closely ID. [38] proposes explicitly discover outliers near ID by prompt learning.
Semantic OOD and covariate OOD can be very similar in practice. As shown in Fig. 4. There exists many similar samples from semantic OOD and covariate OOD in large-scale commonly used benchmarks. We borrow some examples from recent works [47] to show case.
cal-faced
Galap. Fur seal
spagh. bolognese
chicken quesadila
L. Trea
glass milk
donut
marbled newt
spider
kangaroo
(a) Semantic OOD classes from NINCO
bar spider
lion
carbonara
eggnog
bagel
common
burrito
indri
(b) ID classes from ImageNet
Figure 4: Semantic OOD samples can be very similar to ID.
D.3
Social Impact
Al safety and trustworthiness are closely related our work. This paper presents work to harmonize the conflicts between out-of-distribution detection methods and model generalization. The proposed method puts effort to enhance machine learning models for their safely deployment on out-of- distribution data, avoiding both undesirable behavior and degraded performance in challenging high-stake tasks. However, due to the bias from data used by current OOD detection benchmark, e.g., large-scale ImageNet, the ones using the proposed method need to carefully consider the selection of auxiliary outliers for safecy-critical applications.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In conclusion and Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
The paper provide the full set of assumptions and a complete (and correct)
proof in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 6 and Appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
See Appendix and supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
onlliicyn)))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.cooc wwtppppplpggcpiieeceeeeeooooeppe gwte//gggee///eee CCodeSubmiieeeeece
/ttpa////ggiieeeeeeeuuuubiiiiiii iaatommsssioooooooi (eionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides above details in Section 6 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: http///eerrpp.eeeee httppub.ic/ttiiccsuude.......e
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The authors have carefully reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
30

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "QX1ZITIY5Y", "reviewer": "Reviewer_gJee", "summary": "This paper addresses out-of-distribution (OOD) detection and generalization problems. The authors show the sensitive-robust dilemma in learning objectives of OOD detection and generalization and propose a decoupled uncertainty learning (DUL) method to harmonize the above conflict. The proposed method only encourages high distribution uncertainty on OOD data and explicitly discourages high entropy in the final prediction. Experiments on some datasets show the DUL method achieves better performance compared to several baselines.", "questions": "1. According to Eq.9, if keep the overall uncertainty and enlarge the distributional uncertainty (for OOD detection), the data uncertainty must be reduced. Could the authors provide the visualization of data uncertainty?\n\n2. The reviewer suggests two more baselines. (1) In the original Entropy [7] or EBM (finetune) [8], reduce the weight of outlier regularization terms and show the performance on both OOD detection and generalization. (2) Add the unchanged overall uncertainty term in Eq.12 to the original Entropy [7] or EBM (finetune) [8], and report the performance. Those two baselines would help to demonstrate the effectiveness of the bayesian framework in decoupled uncertainty learning.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provide a detailed analysis of the sensitive-robust dilemma between OOD detection and generalization by leveraging transfer learning theory.\n\n2. Develop a decoupled uncertainty learning algorithm that explicitly discourages high entropy in the final prediction to keep the OOD generalization ability of the model. \n\n3. Experiments on benchmarks demonstrate the decoupled uncertainty learning objective achieves better performance compared to baselines.", "weaknesses": "1. While analyzing the sensitive-robust dilemma via transfer learning theory is novel, the phenomenon that existing SOTA OOD detection methods may suffer from catastrophic degradation in terms of OOD generalization has been reported in [4]. \n\n2. The decoupled uncertainty learning objective introduces an additional regularization term named unchanged overall uncertainty. It is difficult to understand why DUL even performs better than Entropy [7] and EBM (finetune) [8] on OOD detection tasks.\n\n3. Only simple Gaussian noise is used to reflect the OOD generalization ability, while there are many types of out-of-distributions like corruptions. Therefore, the results in Table 1 can not represent the performance of evaluated methods because different kinds of OOD data often behave differently under the same model [A]. \n\n[A] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and thorough comments on our paper and for recognizing the contribution of our theoretical analysis and the superiority of our DUL method over current SoTA methods.\n\n- **W1.** While analyzing the sensitive-robust dilemma via transfer learning theory is novel, the conflict between OOD detection and generalization has been reported in [4].\n\nThanks for mentioning [4]. We are pleased to acknowledge that the conflict between these two tasks is first reported in [4]. Our contribution lies in theoretically demystifying the underlying reasons and providing a theory-inspired solution. We will further clarify the difference and highlight this point in the revision.\n\n- **W2.** DUL introduces an additional regularization. It is difficult to understand why DUL even performs better than Entropy and EBM on OOD detection tasks.\n\nAlthough Entropy and EBM are specified for OOD detection. Their techniques may suffer from potential pitfalls. **Entropy uses MSP for OOD detection which has been proven to be overconfident and sub-optimal in [8].** In contrast, our DUL uses differential entropy which enjoys a good interpretability from a Bayesian perspective. Besides, **EBM enforces the energy score of all ID samples to be less than a fixed threshold**, which can be problematic when there are abnormal ID samples in the dataset (e.g., label noise). However, in Eq.12, DUL finetunes the differential entropy on ID to be relatively higher than that before finetuning. The differential entropy threshold in DUL is set in a **data-adaptive manner**. Thus DUL can perform better than these baselines. Thanks for your insightful comments. We will further elaborate on this in the revision.\n\n- **W3.** Only Gaussian noise can not represent the performance of evaluated methods, since the same model behaves differently on various OOD data [A].\n\nThanks for your suggestion. Please kindly remind that we have provided a comprehensive evaluation involving 15 different types of corruption (e.g., snow, rainy, defocus...) on CIFAR10/100-C and ImageNet-C [52] in Appendix C (Tab. 8, 9, and 10 in our manuscript). **We have reorganized them into Table 1 in the global response PDF**. The overall performance of our DUL consistently outperforms its counterparts.\n\n- **Q1.** According to Eq.9, if keep the overall uncertainty and enlarge the distributional uncertainty (for OOD detection), the data uncertainty must be reduced. Could the authors provide the visualization of data uncertainty?\n\nVery insightful question. According to your suggestions, **we visualize the data uncertainty on semantic OOD test data (Textures) when CIFAR10 is ID in Figure 1 in the global response PDF**. The investigated methods are 1) pretrained model training on ID dataset only, 2) finetuned model with OOD detection regularization (ablating the last term in Eq.12), and 3) finetuned model with the full DUL method described by Eq.12. **The results meet your expectation**. We will add these results to the revised paper.\n\n- **Q2.** The reviewer suggests two more baselines. (1) In the original Entropy [7] or EBM (finetune) [8], reduce the weight of outlier regularization terms and show the performance on both OOD detection and generalization. (2) Add the unchanged overall uncertainty term in Eq.12 to the original Entropy [7] or EBM (finetune) [8], and report the performance. Those two baselines would help to demonstrate the effectiveness of the Bayesian framework in decoupled uncertainty learning.\n\nAccording to your advice, we conduct additional experiments. See also **Table 2 and Table 3 in the global rebuttal PDF**.\n\n(1)  We tune the weight of outlier regularization term from [0,1e-4, 1e-3, 1e-2, 1e-1] for EBM [8], [0, 5e-4, 5e-3, 5e-2, 5e-1] for Entropy [7] and report the FPR (OOD detection metric) and error rate (Err, OOD generalization metric).\n\n| Entropy   |       |       |       |       |       |      | Energy    |       |       |       |       |       |\n| --------- | ----- | ----- | ----- | ----- | ----- | ---- | --------- | ----- | ----- | ----- | ----- | ----- |\n| $\\lambda$ | 0     | 5e-4  | 5e-3  | 5e-2  | 5e-1  |      | $\\lambda$ | 0     | 1e-4  | 1e-3  | 1e-2  | 1e-1  |\n| FPR.      | 35.15 | 8.36  | 6.37  | 5.71  | 5.60  |      | FPR.      | 20.57 | 14.69 | 13.54 | 8.15  | 6.11  |\n| Err.      | 9.55  | 13.58 | 15.48 | 17.97 | 18.53 |      | Err.      | 9.55  | 9.46  | 10.32 | 16.43 | 24.38 |\n\n\nWhen the regularization strength increases, OOD detection performance improves (lower FPR.), while the OOD generalization performance degrades (higher error rate).\n\n(2) We add the unchanged overall uncertainty term in Eq.12 to Entropy [7] and EBM (finetune) [8], and report the performance on CIFAR10/ImageNet-RC.\n\n| Method      | ID-Acc. | OOD-Acc. | FPR.  | AUROC |\n| ----------- | ------- | -------- | ----- | ----- |\n| EBM      | 95.93   | 81.47    | 5.58  | 97.75 |\n| EBM+reg  | 95.19   | 87.45    | 6.17  | 98.45 |\n| Entropy     | 96.04   | 72.57    | 6.63  | 98.72 |\n| Entropy+reg | 96.10   | 87.41    | 29.56 | 95.92 |\n| DUL         | 96.04   | 88.01    | 5.71  | 98.61 |\n\nThe results show that DUL regularization can also benefit EBM. However, combining Entropy with our regularization can not improve the accuracy substantially. This is not surprising, since the target of Entropy (high entropy prediction) and our DUL (non-increased entropy) directly conflict according to Theorem 1 in our manuscript. Besides, our DUL generally outperforms the suggested baselines.\n\n[4] Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection, ICML'23\n\n[8] Energy-based out-of-distribution detection. NIPS'20\n\n[52] Benchmarking neural network robustness to common corruptions and surface variations. ICLR'19", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJiNytLhQ8", "reviewer": "Reviewer_31hb", "summary": "This paper reveals the trade-off dilemma of OOD detection and OOD generalization for current SOTA OOD detectors from both theoretical and empirical perspectives. The authors employ a transfer learning framework to analyze the generalization error bound for MSP-based OOD detectors and identify the optimization conflicts between the training objectives of OOD detection and OOD generalization. They find similar patterns for Energy-based detectors. From the theoretical analysis, this paper derives a decoupled uncertainty learning technique for the dual-optimal performance for OOD detection and generalization. Experiments on CIFAR10/100 and ImageNet-200 validate its effectiveness.", "questions": "Although OOD detection and OOD generalization are both frequently discussed in the literature, it is still weird when they come up together. It is basically because the D(istribution) for OOD detection and OOD generalization has different meanings. For OOD detection, it means the collection of label y. For OOD generalization, it means the data distribution for sample x. A proper clarification or modification can be taken into consideration", "limitations": "Please see weakness.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research problem is valuable and well-presented.\n2. The theoretical analysis for the objective conflicts for MSP- and Energy-based OOD detectors is sound.\n3. The experiments show promising performance for the proposed DUL technique, which successfully maintains the OOD generalization capability while achieving competitive OOD detection results.", "weaknesses": "1. This paper only discusses the MSP- and Energy-based OOD detectors, which adopt exactly the same outputs (i.e., logits) from a neural network to perform ID classification (and OOD generalization) and OOD detection, where the optimization conflict is intuitive. However, other superior OOD detectors employ extra output branches aside from the classification logits (even if they share the same backbone for feature extraction) for OOD detection, such as [1][2][3], which shows promising robustness for covariate shifts[2]. Is the theoretical analysis and DUL technique still applicable to those types of OOD detectors?\n2. As the authors claim,  DUL can lead to a dual-optimal performance for both OOD detection and OOD generalization. However, according to Table 1, DUL does not always win first place across different experimental settings. For instance, DUL and $\\dagger$ DUL are surpassed by POEM on OOD detection performance by a non-negligible margin in the CIFAR10/100+ImageNet-RC setting. It seems DUL just achieves a better trade-off rather than the dual-optimal results. A proper explanation should be given.\n3. DUL seems to basically add a normalization/regularization on the model's output logits with knowledge (measured by Bayesian framework) distilled from a pre-trained ID classifier. It limits the application and novelty of this paper. What happens if the model is trained from scratch?\n4. Besides, adding Gaussian noise is a too trivial setting to evaluate the robustness of covariate shifts. How about evaluating OOD detectors in standard domain adaptation settings?\n\n[1] J Bitterwolf et al. “Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities”. ICML 2022.\n\n[2] L Kai et al. \"Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions\". NeurIPS 2023.\n\n[3] W Miao et al. \"Out-of-Distribution Detection in Long-Tailed Recognition with Calibrated Outlier Class Learning\". AAAI 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the clear presentation and effective method. We provide detailed responses to the constructive comments.\n\n- **W1.** The paper only discusses the MSP- and Energy-based OOD detectors, which are both adapted to logits from a neural network. There exist superior OOD detectors that use extra output branches aside from the classification logits for OOD detection (e.g. [1] [2] [3]) which shows promising robustness for covariate shifts[2]. Is the theoretical analysis and DUL technique still applicable to these OOD detectors?\n\nThanks for your insightful suggestions. According to your advice, we conduct additional experiments on OOD detectors with the K+1 class branch. First, the observed dilemma also exists in these types of OOD detectors. **We would like to acknowledge that, our theoretical analysis is not directly applicable for OOD detectors with K+1 class for now. However, the DUL technique is still applicable.** The training strategy we suggest for the K+1 classifier is: 1) First train a standard K-class classifier on the ID dataset. 2) Then replace the last fully connect layer to K+1 (random initialize its parameters). 3) During finetuning, one can obtain the predicted distribution $p^{k\\leq K}(\\hat{y}|\\tilde{x})$ of the first K class by normalizing their logits $[f_1,\\dots,f_k]$ with softmax. Then explicitly penalize high entropy on these ID classes as in Eq.12. The loss function is CE loss and $(f_{k+1}(x)-m_{in})^2+(f_{k+1}(\\tilde{x})-m_{out})^2+KL(p^{k\\leq K}(\\hat{y}|\\tilde{x})||p_0^{k\\leq K}(\\hat{y}|\\tilde{x}))$. The results show that the DUL technique is potentially useful for detectors with an additional branch. **We promise to make the limitation transparent in the revision.** Further analysis from a feature learning perspective may be needed in future work.\n\n|         |OOD-ACC|FPR|AUROC|\n| -------------------- | -------| -----| ----- |\n|MSP|88.11|37.04|90.91|\n|K+1 classifier|84.77|3.22|99.11|\n|K+1 classifer w. DUL|87.99|7.12|98.35|\n\nBesides, please kindly remind that DUL is devised in a finetune manner. Compared to K+1 classifier that requires re-training the classifier from scratch, DUL can be applied to any pre-trained model (e.g., torchvision, huggingface), with modest compute overhead ($\\leq$ 20 epochs).\n\n- **W2.** The authors claim that DUL can lead to dual-optimal performance for both OOD detection and generalization. However, DUL is surpassed by POEM on OOD detection in the CIFAR10/100+ImageNet-RC setting. A proper explanation should be given. \n\nThanks for the comments. Our DUL is initially devised in a finetune manner for computationally effectiveness. **DUL is finetuned on ImageNet-RC for only 20 epochs**. However, the official implementation of **POEM is trained from scratch for 200 epochs**. Thus it is reasonable that DUL does not outperform POEM in a few certain settings. According to your advice, we finetune DUL in CIFAR10/ImageNet-RC setting for more epochs (100). We observe that **DUL achieves even better OOD detection performance compared to POEM on CIFAR10/ImageNet-RC with longer training.**\n\n|             | OOD-ACC | FPR  | AUROC |      |\n| ----------------- | ------- | ---- | ----- | ---- |\n| POEM (200 epochs) | 78.89   | 3.32 | 98.99 |      |\n| DUL (100 epochs)  | 88.13   | 2.75 | 98.06 |      |\n\n- **W3.** DUL seems to basically add a normalization/regularization on the model's output logits with knowledge (measured by Bayesian framework) distilled from a pre-trained ID classifier. It limits the application and novelty of this paper. What happens if the model is trained from scratch?\n\nThanks for the interesting question and careful reading. Initially, we devise DUL in a finetune manner following EBM [4] for **effectiveness**. Thus it can be applied with any pre-trained model. **If the model is trained from scratch, one can simply introduce a two-stage training strategy.** First, train an ID classifier, and then finetune on OOD data with our DUL. It would take only 20 additional epochs for the second stage finetuning to get the superior performance we reported.\n\n- **W4.** Besides, adding Gaussian noise is a too trivial setting to evaluate the robustness of covariate shifts. How about evaluating OOD detectors in standard domain adaptation settings?\n\nThanks for your valuable suggestions. Please kindly remind that we have provided a comprehensive evaluation involving 15 different types of corruption (e.g., snow, rainy, defocus...) on CIFAR-C/ImageNet-C [52] in Appendix C (Tab. 8, 9, and 10). To our best knowledge, CIFAR10/100-C and ImageNet-C are widely used in domain adaption settings. **We have reorganized the results into Table 1, global response PDF. The overall performance of DUL consistently outperforms its counterparts.**\n\n- **Q1.** Although OOD detection and OOD generalization are both frequently discussed in the literature, it is still weird when they come up together. It is basically because the D(istribution) for OOD detection and OOD generalization has different meanings. For OOD detection, it means the collection of label y. For OOD generalization, it means the data distribution for sample x. A proper clarification or modification can be taken into consideration.\n\nThanks for your suggestion. In the classification context, distribution shift means the joint distribution $p(x,y)$ is different during training and testing. In this paper, OOD detection targets semantic OOD where the label $y$ of which do not belong to any known classes. OOD generalization aims to properly classify data where the $x$ undergoes changes in outlook or shift in style but still belongs to known classes. **These types of data can organically arise when deploying models in the open world [4]**. Thus it deserves to find out how to handle them in a unified framework. We will further clarify the definition of different types of OOD samples in an open-world setting according to your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kqTsmeOcgS", "reviewer": "Reviewer_Tcyq", "summary": "This paper theoretically characterizes the underlying dilemma in SOTA OOD detection method.  The authors find that OOD detection performance of state-of-the-art methods is achieved with tradeoff between OOD detection and classification. Accordingly, the authors provide an uncertainty-based strategy which decouples these two tasks and thus addresses the dilemma. Plenty of experiments are conducted which well support the theory and validate the effectiveness of the proposed model.", "questions": "1. Figure 1 is very helpful and intuitive, while it seems not clear - which dataset the results are reported in the right figure.\n2. The authors should explain more about the “tradeoff area”. Although it seems ok, but I think more detailed description is necessary. \n3. “Reducing the dependency on auxiliary OOD data can be an interesting research direction for the future exploration.” I think it is interesting to reduce the dependency on auxiliary OOD data. However, could the authors provide some potential directions?", "limitations": "See Weaknesses", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The findings of this paper are very interesting. This work studies the dilemma between OOD detection and classification generalization. For the first time, the authors provide a theoretical perspective which well reveals the underlying tradeoff. This theoretical result provides strict and intuitive explanations for this phenomenon, which is important and enlightening.\n2. The proposed uncertainty-decoupled strategy is reasonable for the dilemma, and the paper is technically solid. The proposed method is developed according to the induced theory which validates the potential of the theory in inspiring other models. \n3. The organization and writing of this paper are clear, making the contributions easy to catch. The experiments sufficiently validate both the theory and the proposed method.", "weaknesses": "I am not sure are there any existing works implicitly considering decoupling these two tasks, i.e., OOD detection and generalization. Please provide more clarifications. \n\nThe theory is very promising. So it will be better if the authors could provide more discussion on other potential strategies that could also addressing the dilemma. I think it will be very useful for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the interesting findings, important and enlightening theoretical results as well as sufficient experiments. We provide detailed responses to address your concerns.\n\n- **W1.** I am not sure are there any existing works implicitly considering decoupling these two tasks, i.e., OOD detection and generalization. Please provide more clarifications.\n\nThanks for your suggestions. Here we provide a brief survey of existing works related to both these two tasks. [A] considers both OOD detection and generalization in vision-language models (i.e., CLIP), they propose to hierarchically construct the text description for a certain category and enhance ID classification and OOD detection. [B] utilizes the diffusion model to generate virtual OOD samples which can be used for enhancing OOD detection and generalization simultaneously. [C] focus on improving OOD generalization performance in a realistic open-set setting, which is capable of simultaneously handling covariate-shifted OOD data and detecting semantic OOD data. Though there exist a few works that pursue these two targets altogether, the instinctive relationship between them is notably unexplored, not to mention an ideal solution. Thus we believe our findings is distinguished from these works. We will add more detailed clarifications in the revision according to your suggestion.\n\n[A] Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions. NIPS'23\n\n[B] Dream the Impossible: Outlier Imagination with Diffusion Models. NIPS'23\n\n[C] ATTA: anomaly-aware test-time adaptation for out-of-distribution detection in segmentation. NIPS'23\n\n- **W2.** The theory is very promising. So it would be better if the authors could provide more discussion on other potential strategies that could also address the dilemma. I think it will be very useful for readers.\n\nThanks for your suggestion. According to our analysis, previous OOD detection methods explicitly/implicitly encourage high-entropy prediction resulting in the dilemma. The most straightforward way to address this limitation is enforcing unchanged entropy as we devised in this paper. Besides, we notice that many recent OOD generalization works utilized unsupervised entropy minimization loss to further enhance the classification performance [28, 29, 30]. Their core idea is coherent with our DUL. It is worth trying new strategies such as conducting entropy minimization on unlabeled test data in the future.\n\n[28] Tent: Fully test-time adaptation by entropy minimization. ICLR'21\n\n[29] Memo: Test time robustness via adaptation and augmentation. NIPS'22\n\n[30] Towards stable test-time adaptation in dynamic wild world. ICLR'23\n\n- **Q1.** Figure 1 is very helpful and intuitive, while it seems not clear - which dataset the results are reported in the right figure?\n\nWe visualize the results on CIFAR-100 when TIN-597 serves as auxiliary OOD data in Figure 1 (b). Consist with the tabular results in Table 1. We will clearly explain this in revision.\n\n- **Q2.** The authors should explain more about the \"tradeoff area\". Although it seems ok, I think a more detailed description is necessary.\n\nThanks for your suggestion. The trade-off area in Fig. 1 (b) denotes the area where the investigated OOD detection methods perform better than the baseline MSP (the pretrained model without any OOD detection regularization) but yield a higher error rate compared to MSP. These methods exhibit better OOD detection performance but sacrifice OOD generalization ability. Thus we say that they lie in a \"trade-off area\". We will clearly explain this in the revised paper.\n\n- **Q3.** I think it is interesting to reduce the dependency on auxiliary OOD data. However, could the authors provide some potential directions?\n\nSure. As far as we can tell, there exist three directions in this literature. I. **Outlier sampling strategies** that choose the most informative outliers for model regularization. For example, greedy sampling[17] and Thompson sampling[11]. These methods are of better data efficiency compared to others. II. **Leveraging external knowledge from pretrain models**. Recent [34] utilizes the diffusion model to generate virtual outliers which can remove the dependency on real auxiliary OOD data. III. **Directly devising OOD detection methods without auxiliary OOD data.** Many works aim to enhance OOD detection performance given only a pretrain model. Though their practical performance is often sub-optimal compared to the counterparts explicitly regularized on OOD data, they are usually easier to deploy in applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzZU3bn7xU", "reviewer": "Reviewer_DUjY", "summary": "This paper proposes a novel method for finetuning a classifier for both OOD generalization and OOD detection. The paper presents a novel objective function for OOD generalization and detection based on the Bayesian framework and OOD generalization theory. Extensive experimental results suggest that the proposed method is effective.", "questions": "* Is there a reason why we should enforce the \"non-increased overall uncertainty\" constraint (the constraint in Eq. (11)) only on semantic OOD data? For example, what if we enforce this on the in-distribution dataset? Is there a theoretical justification for this choice, or is it more empirical?", "limitations": "* The paper addresses its limitation in the conclusion section. The limitation of the paper does not raise a concern.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The paper addresses an important problem of achieving OOD generalization and OOD detection simultaneously. This task has high significance regarding the safety of ML systems.\n* Overall, the exposition of the paper is clear and easy to follow.\n* The paper presents a sound theoretical analysis on why some existing OOD detection approaches have failed in OOD generalization.\n* The experiments are conducted in a reasonable setting, and the results are promising.", "weaknesses": "* It is possible that I missed it, but the paper does not provide a direct theoretical argument supporting the main objective function (Eq. (11)). The most important term in Eq. (11) is the constraint that the entropy of p(y|x) should not change before and after the fine-tuning. This term maintains OOD generalization, but the justification is weak. \n* Related to the above point, the statement in Lines 244-245 (\"While generalization is related to 245 the overall uncertainty of p(yˆ|x) as we mentioned in related works (both AU and DU).\") argues that this entropy (i.e., \"overall uncertainty\") is connected to the generalization performance. Still, I could not find the supporting information in the related work section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the importance of our research problem, clear exposition, sound theoretical analysis, and promising results. We appreciate your support and constructive suggestions and address your concerns as follows.\n\n- **W1.** It is possible that I missed it, but the paper does not provide a direct theoretical argument supporting the main objective function (Eq. (11)). The most important term in Eq. (11) is the constraint that the entropy of p(y|x) should not change before and after the fine-tuning. This term maintains OOD generalization, but the justification is weak.\n\nThanks for raising a concern about direct theoretical support to Eq.11. Theorem 1 can directly support the learning objective in Eq.11. In Theorem 1, the OOD detection loss is negatively correlated with OOD generation loss. Please note that the OOD detection loss we discuss here is the cross entropy between model prediction and uniform distribution proposed by [7]. This term explicitly encourages high entropy on OOD samples and is proven to be harmful to model generalization by Theorem 1. Thus in Eq.11, we disencourages high-entropy prediction, which is a natural and effective solution. More importantly, in section 5, line 230-248, we show that this constraint is not harmful to OOD detection theoretically by revisting the property of various uncertainty. We will add more discussion and smoothen the logic in the revised paper.\n\n- **W2.** Related to the above point, the statement in Lines 244-245 (\"While generalization is related to the overall uncertainty of p(y|x) as we mentioned in related works (both AU and DU).\") argues that this entropy (i.e., \"overall uncertainty\") is connected to the generalization performance. Still, I could not find the supporting information in the related work section.\n\nThanks for your comments. Many recent works in OOD generalization literature can support our statement. Here we provide a brief discussion. The recent success of OOD generalization [A] [C] [D] [E] has shown that overall uncertainty (entropy) is closely connected to the generalization performance. By iteratively minimizing the predictive entropy, **the model classification accuracy on test data can be significantly improved and vice versa**. The initial intuition from [C] is based on the observations that models tend to be more accurate on images for which they make predictions with higher confidence. Besides, earlier works [B] in semi-supervised literature also show that decreasing predictive entropy can improve model accuracy. The natural logical extension of this observation is to enforce models to not increase their entropy on test samples, which is the key motivation of our DUL. We will elaborate on this in revision.\n\n- **Q1.** Is there a reason why we should enforce the \"non-increased overall uncertainty\" constraint (the constraint in Eq. (11)) only on semantic OOD data? For example, what if we enforce this on the in-distribution dataset? Is there a theoretical justification for this choice, or is it more empirical?\n\nThanks for your interesting question. **Our choice is supported by both empirical and theoretical evidence.** The answer to your question is threefold. First, for covariant-shifted OOD. As we have shown before, many previous works in OOD generalization have demonstrated both theoretically [A, B] and empirically [C, D, E] that entropy on covariant-shifted OOD is negatively related to classification performance. Secondly, when it comes to semantic OOD, we have theoretically proved that high entropy will also result in degraded classification performance in Theorem 1. Thirdly, however, enforcing low overall uncertainty on the in-distribution dataset may be not necessary. Since during pretraining and finetuning, the standard cross-entropy loss has enforced the model to be highly confident on ID data [F]. **To summarize, we should discourage high overall uncertainty on all three types of data. However, only constraining on the semantic OOD is enough. Because in our settings, we cannot access covariant-shifted OOD and the entropy on ID has been constrained by standard CE loss.** We will add these explanations to the revised paper.\n\n[A] The Entropy Enigma: Success and Failure of Entropy Minimization. ICLR'24\n\n[B] Semi-supervised learning by entropy minimization. NIPS'04\n\n[C] Fully test-time adaptation by entropy minimization. ICLR'21\n\n[D] Memo: Test time robustness via adaptation and augmentation. NIPS'22\n\n[E] Towards stable test-time adaptation in dynamic wild world. ICLR'23\n\n[F] On calibration of modern neural networks. ICML'17", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear PCs, SACs, ACs, and Reviewers,

Thanks for your valuable feedback and insightful reviews, which greatly improved the paper. We are deeply encouraged that all the reviewers gave positive assessment on our work. This is a **clear** and **well-presented** (Reviewer 31hb, Tcyq) manuscript with a **valuable** and **important** research problem (Reviewer DUjY, 31hb) and **sound**, **strict**, **important, and enlightening** theoretical analysis (Reviewer DUjY, Tcyq, 31hb). The proposed method is **reasonable** and **novel** (Reviewer DUjY, Tcyq). The **promising** and **extensive** experimental results **can validate the effectiveness of the proposed method**. (Reviewer gJee, 31hb, DUjY, Tcyq).

In the rebuttal, we addressed the following raised concerns/misunderstandings. 

- We have provided a comprehensive evaluation involving 15 diverse corruptions from commonly-used domain adaption benchmarks i.e., CIFAR10/100-C and ImageNet-C (Table 1, global response PDF).
- We have provided a visualization of various uncertainties (Figure 1 in the PDF).
- We have shown how to extend DUL to the k+1 classifier and model trained from scratch and promise to make the potential limitations transparent in revision (Table 2 in the PDF).
- We have added experiments on two baselines suggested by Reviewer gJee (Table 2, Table 3 in the PDF).
- We have carefully revised all the presentation issues throughout the paper.

We believe these clarifications and additional results strengthen our paper and address the reviewers' concerns. We understand the workload that reviewers and AC face, and appreciate the effort already put into evaluating our work. If there are any additional insights, questions, or clarifications that you would like to discuss with us, we would be very grateful to hear them, your feedback is invaluable for the improvement of our research.

Best regards,

Authors

Meta-review: This paper investigated a very interesting problem: the relationship between OOD detection ability and OOD generalization ability, suggesting us be aware of purely pursuing the OOD detection ability in practice and theory. All reviewers agree that this contribution is valuable to the field.
Decision: Accept (poster)