Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution
Authors: Kai Liu
Zhihang Fu
Sheng Jin
Chao Chen
Ze Chen
Rongxin Jiang
Fan Zhou
Yaowu Chen
Jieping Ye
Venue: NeurIPS 2024 poster (2024)
Keywords: Out-of-Distribution Detection
Imbalanced Recognition
Primary area: machine_vision

Abstract: Detecting and rejecting unknown out-of-distribution (OOD) samples is critical for deployed neural networks to void unreliable predictions. In real-world scenarios, however, the efficacy of existing OOD detection methods is often impeded by the inherent imbalance of in-distribution (ID) data, which causes significant performance decline. Through statistical observations, we have identified two common challenges faced by different OOD detectors: misidentifying tail class ID samples as OOD, while erroneously predicting OOD samples as head class from ID. To explain this phenomenon, we introduce a generalized statistical framework, termed ImOOD, to formulate the OOD detection problem on imbalanced data distribution. Consequently, the theoretical analysis reveals that there exists a class-aware bias item between balanced and imbalanced OOD detection, which contributes to the performance gap. Building upon this finding, we present a unified training-time regularization technique to mitigate the bias and boost imbalanced OOD detectors across architecture designs. Our theoretically grounded method translates into consistent improvements on the representative CIFAR10-LT, CIFAR100-LT, and ImageNet-LT benchmarks against several state-of-the-art OOD detection ap- proaches. Code is available at https://github.com/alibaba/imood.

Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution
Kai Liul:2; Zhihang Fu Sheng Jin?, Chao Chen', Ze Chen',
Rongxin Jiang' 1 Fan Zhou', Yaowu Chen', Jieping Ye²
'Zhejiang University, 2 Alibbba Cloud
Abstract
Detecting and rejecting unknown aut-of-distributia (OOD) samples critical for deployed neural networks void unreliable predictions. In real-world scenarios, however, the efficacy of existing OOD detection methods is often impeded by the inherent imbalance of in-distribution (ID) data, which causes significant perfor- mance decline. Through statistical observations, we have identified two common challenges faced by different OOD detectors: misidentifying tail class ID samples OOD, while erroneously predicting OOD samples as head class from ID. To explain this phenomenon, we introduce a generalized statistical framework, termed ImOOD, to formulate the OOD detection problem on imbalanced data distribution. Consequently, the theoretical analysis reveals that there exists a class-aware bias item between balanced and imbalanced OOD detection, which contributes to the performance gap. Building upon this finding, we present a unified training-timg time regularization technique to mitigate the bias and boost imbalanced OOD detec- tors across architecture designs. Our theoretically grounded method translates into consistent improvements and ImageNet-LT benchmarks against several state-of of OOD detection ap- proaches. Code is available at phttps://gitee com htt///albbbbaimmooo
the representative CIFAR I0-LT, CIFAR 100-LT,
Introduction
Identifying and rejecting unknown samples during models' deployments, aka OOD detection, has garnered significant attention and witnessed promising advancements recent years [57 S 41
48.30 30). Nevertheless, most advanced OOD detection methods are designed and evaluated in ideal settings with category- di-distribution (ID) data. However, in practical scenarios, long-tain class distribution (a typical imbalance problem) not only limits classifiers' capability 77) but also causes a substantial performance decline for OOD detectors [51].
As Wang et al. 151 reveal, a naive combination of long-tailed image cognition 39 and general techniques cannot simply mitigate this issue, and several efforts have been
OOD detection applied to study the joint imbalanced OOD detection problem [51, 22 45) They mainly attribute the performance degradation to misidentifying samples from tail classes as OOD (due to the lack of training data), and concentrate improving the discriminability for tail classes and out-of-distributtin samples [51 [55]. Whereas, we argue that the confusion between tail class and OOD samples presents only one aspect of the imbalance problem arising from the long-tailed data distribution.
To comprehensively understand the imbalance issue, we investigate a wide range of representative OOD detection methods (i.e., OE (20). Energy B2). and PASCL (5T) on the CIFARIO-LT dataset [8]. For each model, we statistic the distribution wrongly detected samples and wrongly detected
WWrk done during Kai Liu's 's's research internship at Alibaba Cloud. Email: kail kail@zju.eda www..educnn.n... Corresponding authors. Email: rongxinj @ edu.ed, zhihang fzh alibaba-ine.com.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

OE
Energy
PASCL
Price Free
Price Freq
Prior Freq Lahck
dectrion
Lahile
Labale
boundarry
number
x
classes OE
classes Energy
classes PASCL
Poier Faeq
Prior Faoq not' Prods
Price Freq
not Prode
OOD Prode
number
Wrong ID OOD detections
(a) Statisties on wrongl-detecctdd and OOD samples.
(b) Dicision illustration.
Figure 1: Issues of OOD detection on imbalanced data. (a) Statistics of the class labels of ID samples that are wrongly detected as OOD, and the class predictions of OOD samples that are wrongly detected ID. (b) Illustration of the OOD detection process in feature space. Head classes' huge decision space and tail classes' small decision space jointly damage the OOD detection.
OOD samples, respectively. The results in Fig. reveal that different approaches encounter the same two challenges: (1) ID samples from tail classes are prone to detected as OOD, and (2) OOD samples are prone to be predicted ID from head classes. As illustrated in Fig. [b] argue that the disparate ID decision spaces on head and tail classes jointly result in the performance decline for OOD detection, which has also been confirmed by Miao et al. 40]
To mitigate this problem, Miao et [40] developed a heuristic outlier class learning approach (namely COCL) to respectively separate OOD samples from head and tail ID classes in the feature space. Different from COCL, this paper introduces generalized statistical framework, termed ImOOD, to formulate and explain the fuundamental issue of imbalanced OOD detection from probabilistic perspective. We start by extending closed-set ID classification to open-set scenarios and derive unified posterior probabilistic model for ID/OOD identification. Consequently, that between balanced and imbalanced ID data distributions exists a class-aware bias item, which concurrently explains the inferior OOD detection performance both head and tail classes.
find
Based on ImOOD, we derive unified loss function to regularize the posterior ID/OOD probability during training, which simultaneously encourages the separability between tail ID) classes and OOD samples, and prevents predicting OOD samples as head ID classes, Furthermore, ImOOD can readily generalize to various OOD detection methods, including OE (20). Energy (32), and BinDise (5), Mahalanobis-distanee (28), etc. Besides, our method can easily integrate with other feature-level optimization techniques, including PASCL [51] and COCL [40], to derive stronger OOD detectors. With the support of theoretical analysis, our statistical framework consistently translates into strong empirical performance on the CIFARIO-LT, CIFAR100-LT (8), and ImageNet-LT [51] benchmarks.
Our contribution can summarized as follows:
Through statistical observation and theoretical analysis, we reveal that OOD detection approaches collectively suffer from the disparate decision spaces between tail and head classes in the imbalanced data distribution.
We establish generalized statistical framework to formulate and explain the imbalanced OOD detection issue, and further provide a unified training regularization to alleviate the problem.
We achieve superior OOD detection performance three representative benchmarks, outper- forming state-of-the of-the-art methods by large margin.
2 Related Work
Out-of-distribution detection. To reduce the overconfidence on unseen OOD samples 4 surge of post-hoc scoring functions has been devised based on various information, including output

---

confidence [9
29 33 free energy 32 14 26 Bayesian inference [6 91, gradient information 21 model/data sparsity 46
60
13
and visual distance 47 49 etc. Vision-langual models like
CLIP 44 have been recently leveraged to explicitly collect potential OOD labels [17 [6] or
conduct shot OOD detections [41]. Other researchers add open-set regularization in the training
time 20 54
53
35 30), making models produce lower confidence or higher energy on OOD data.
Manually
20 52 or synthesized [14] 49 outliers are required for auxiliary constraints.
Works insofar have mostly focused on the ideal setting with balanced data distribution for optimization and evaluation. This paper aims OOD detection on practically imbalanced data distribution.
OOD detection on imbalanced data distribution. In real-world scenarios, the deployed data frequently exhibits long-tailed distribution, and Liu et al. [4]] start to study the open-set classification on class-imbalanced setup. Wang et al. [51] systematically investigate the performance degradation for OOD detection imbalanced data distribution, and develop partial and asymmetric contrastive learning (PASCL) technique to tackle this problem. Consequently, Wei et al. (55] and [40] extend the
feature-space optimization by introducing abstention classes outlier class learning and integrating with data augmentation or margin learning techniques, respectively. Sapkota and Yu [45] employ adaptive distributively robust optimization (DRO) quantify the sample uncertainty from imbalanced distributions. Choi et al. [TO] focus on the imbalance problem in OOD data, and develop an adaptive
regularization for each OOD sample during optimization. In particular, Jiang et [23] also utilize class prior to boost imbalanced OOD detector, which is however constrained as a heuristic post-hoc normalization for pre-trained models and unable to translate into training a better detector.
Different from previous efforts, this paper establishes generalized probabilistic framework to formulate and explain the imbalanced OOD detection issue, and provides unified training-time regularization technique to alleviate this problem across different OOD detectors.
3 Rethinking Imbalanced OOD Detection
In this section, we start from revisiting the closed-set imbalanced image recognition (in (in-distribution classification), and extend to open-set disttiibbiiiitoo detection problem. Finally, we will reveal the class-aware bias between balanced and imbalanced OOD detectors, and derive a unified training-time time regularization technique alleviate the bias for different detectors.
3.1 Preliminaries
Imbalanced Image Recognition. Let X and yan 21, -11,.... K} denote the ID feature space and label space with K categories in total. Let æ E X and y E join be the random variables with respect v and yin. The posterior probability for predicting sample into class y is given by:
P(ælly) P(y) P(x)
P(y]x)
x P(xlv) P(v)
Given learned classifier f: X' + K that estimates Ply ar), in the class-imbalance setting where the label prior P(y) is highly skewed, f evaluated with the balanced error (BER) 6 38
39
BER(F)
Σ Pa # f
(2)
This can be seen as implicitly estimating a class-balanced posterior probability [39
P(y|æ)
phal (y)())) K
P(ælyy
P(y)
The ideal Bayesian-optimal classification becomes y" = argmax,vexxj phal (y/zz)
Out-of-distribution Detection. In the open world, input sample x may also come from the OOD space you! Let be the random variable for unknown label of ymn and be the union variable

---

of all ID class labels (i.e., i U y). Given an input from the union space X U 2 - X, the posterior probability for identifying æ as in-distribution is formulated as:
P(i|z) Σ P(ylæ) = P(o]z) #1
(4)
Correspondingly, 2()) measures the probability that sample does not belong to any known ID class, aka OOD probability. Hence, the OOD detection task can be viewed as a binary classification problem [5]. Given learned OOD detector g: X U X + R' that estimate the P(ila),, samples with lower scores g(x) are detected as OOD and vice versa.
3.2 Analysis of OOD Detection on Imbalanced Data Distribution
When ID classification meets OOD detection, slightly different from Eq. I the classifier f is actually estimating the posterior class probability for sample x from ID space X merely, that is, P(y(=, i) [20 [1]] Considering sample æ from the open space E 2 U Xoout the classification posterior Eq. is re-formulated as P(y]æ) = P(ylar, i) (i)lær), the probability that sample comes from ID data multiply the probability that sample belongs to the specific y-th ID) class.
According to Eq. (3. we assume the proportion between balanced classification pbal (y/²) and vanilla P(y(z) for each class y still holds.
Lemma 3.1. For each ID class y in open-set, there exists non-negative variable "/y (a(x), so that phul (y) (z) (2))
P(ylm),
where (x)) K P(wly) P(y)
(all)) oo), P(ylæ), phol (ylæz), P(v) E [o, 1].
In fact, 7y (z)
(x)) captures the likelihood difference for the same sample æ between balanced and imbalanced distributions, and also plays the role in constraining the multiplication results to (0, 1)-
The proof can be found in Appendix A. Using Lemma 3.1 we reveal that there exists a class-aware bias term ((x) between the OOD detection on balanced and imbalanced data distribution.
Theorem 3.2. According to Lemma 3.1 there exists bias term B(x) Ey (2)
P(ylas) P(y)
between
(ilxz) and P(ilz), i.e., phol (ilæ) B(x) P()]z).
Proof. Since P(y[œ) = P(y [2. i) P(ilæ). from Lemma 3.1 pbal (yl (y/z)) can be further expressed as
phol (y x)) %o (z)) Pig (.))
P(ilz:) According Eq. (4 can be formulated as: ppall (ilæz)
P(a)
Σy phal Pyyl(y)))) = Eyn 7/(2) P())
P(yla.s) P(i]x) è ((x))P((il) x)). where ((x) = Ey (x(x)) flgla..))
P(u)
we conclude that the original OOD posterior P(i]z) estimated by the detector g(xr), with the bias term ((r)), causes the performance gap for OOI) detection balanced and imbalanced data distributions. To understand this further, we will first discuss the scenario under ideal conditions and then extend our analysis to real-world scenarios. attempting to analyze the intrinsic bias of the out-of fuittf-distribution problem in both cases.
Based on Theorem 3.2
On ideal class-balanced distribution, the data likelihood P(xly) phul pxll()))) (xly), so that (,(x)) k
From Theorem 3.2 ((x) E, K P(y) Mcanwhile, the class prior P(y) = - and the summary of in-distribution classification probabilities equals! 1 (i.e., Ey P(ylar, i) 1), making the bias item 3(x) = Σ, P(yla, i) = 1. Ultimately, Theorem 3.2 indicates phal (il=)) = P(i]z), where the detector g(x) exactly models the balanced OOD detection.
P(yla.i)
On more challenging class-imbalan for ID categories, and 3(x) Σ, (z) P(glæ.)) # In previous works 20 51], the class posterior P(a)
distribution, the class prior Pyy) is class- variable
P(y|æ, i) is usually estimated with a softmax function by classifier f, and the class prior P(y) adopts the sample frequency for each ID class. 7g (x) is under-explored and simply treated as a constant. Under this circumstance, since Σ, P(ylæ, i) = 1, the bias item can be viewed as weighted sum of 1 Theorem 3.2 explains how 3(x) causes the gap between balanced (ideal) the reciprocal prior P(y).
ID/OOD probability pbal and imbalanced (learned) P(i]æ);

---

Given sample from an ID tail-class with small prior P(u), when the classification probability P(y. x. i) gets higher, the term ((x) becomes larger. Compared to the original P())x:) (learned by g). the calibrated probability phll (ila:) i.e., P(ilar) ((x)) is more likely to identify the sample a as in-distribution, rather than OOD.
Given a sample x' from OOD data, as the classifier f tends to produce a higher head-class probability P(y a2', i) and lower tail-class P(ye (z', (23], the term (xx') becomes smaller. Compared to the original P(ilæ'), the calibrated probability phil i.e. P(ila=) ((x') is more likely to identify the sample x' as out-of-distribution, rather than ID.
The above analysis is consistent with the statistical behaviors (see Fig. 1 of vanilla OOD detector g. Compared to an ideal balanced detector ghal 9 is prone to wrongly detect ID samples from tail class as OOD, and simultaneously wrongly detect OOD samples head class from ID.
3.3 Towards Balanced OOD Detector Learning
To address the identified bias in OOD detection due to class imbalance, we present a unified approach from statistical perspective. Our goal is to push the learned detector y towards the balanced gaal. We outline the overall formula for estimating the OOD posterior below.
Specially, we use the common practices
39
23 to estimate the probability distribution phul (i|z))
Σ, (ba((( (y/a=) My (x))
P(ylz.a) P(y)
P(ila:) in Theorem 3.2
First, for the class prior P(y), we use the label frequency of the training dataset 88 39, expressed as P(y) Ey ny, where My refers to the number of instances in class y. For the class posterior Plyl,, i). given learned classifier f. the classification probability can be estimated using a softmax
function 39 23
P(ylar, 2) == E. (2)
cJy(m)
Pylæ,,, where fg represents the logit for class y.
For the OOD posterior P(i) (r), since OOD detection is a binary classification task (5), the posterior probability for arbitrary OOD detector g 20
be estimated using sigmoid function: P(ixz)) = 1+c-9(a)) where g(x) is the ID/OOD logit. Finally, for the class -pasifcpecific scaling factor
32 28
(z), estimating Ty (x) is a sophisticated problem in long-tailed image recognition [3 25]. To
focus on the OOD detection problem, we use a parametric mapping Ty:0 æ ++ (0, oo). where @ are learnable parameters that are optimized through gradient back cropapoppgttiio.
This unified approach allows us systematically estimate and correct for the bias introduced by class imbalance, thereby improving the performance of OOD detection in real-world scenarios. According to Theorem 3.2 the balanced OOD detectors المو modeled as:
(x)) ) = Σ (w))(w) Polm.i Ty
(g((z)) e B(x) a(g(x))
(5)
y
Substitute the sigmoid function o(2) Ite- into Eq. 5 we have:
g(x) = 9 hall (x) (x) log (B(x)) - 1) (m) + 8(x))
(6)
The derivation is displayed in Appendix A.3 In order to make the detector gax=) directly estimate the balanced OOD detection distribution, we can apply the binary cross-cntropy loss on calibrated logits:
Lood = LBCE (9(x)) - log [6(xz) - 1) ((() + {(x))) LECE (g(x) A(x).t)
(7)
where t = 1fx E Xin) indicates whether the sample x comes from ID or not. For a ID sample from tail classes with t discussed in Sec. 3.2 the bias 3 is larger than samples from head classes. In this situation, the punishment A correspondingly increases, which encourages the detector g to generate a higher score g(x)) to reduce the loss. On the other hand, for a OOD sample that predicted as a head ID class by classifier f. 3 and A become smaller than those predicted as tail ID classes, and g are further forced reduce the score g(xr) to decrease the loss, as the label for OOD samples is O. In practice, to alleviate the optimization difficulty, the A(x) for ID samples (with t 1) is cut
5

---

off to be non-negative value, and (xx) for OOD samples (with = 0) is cut off to be non-positive values, ensuring the optimization direct dose not conflict with the vanilla BCE loss function.
Meanwhile, according to Lemma 3.1 we add an extra constraint on Yy:0 to ensure the posterior estimate phal (3) (z) = (x)
Pulma (g(x))
and (il()) = Ey pbal (y ()) will not exceed I:
C max 0, Σ (w:)
Pyla. a(((2))) -1 max {0, o(g(x=)) 1} (8)
By
Note that for each class the term Palma a (g(xr) > 0 always holds, so that we only have to constrain the summary on all classes phal (i)zz) Σ, pbal (yl (z) within 1, as indicated in Eq. (8) and
phl (x) for each class y will be constrained well.
Combining with Land and C.y for optimization, the learned OOD detector g" has already estimated the balanced pbal (il We thus predict the ID/OOD probability as usual: p(ilaz) = T+e ""(w)- The other terms like 7, 3, and are no longer needed to compute, maintaining the inference efficiency and simplicity for OOD detection applications.
4 Experiments
In this section, we empirically validate the effectiveness of our ImOOD several representative imbalanced OOD detection benchmarks. The experimental setup is described in Sec. 4.1 based on which extensive experiments and discussions are displayed in Sec. 4.2 and Sec. 4.3
4.1 Setup
Datasets.
Following the literature (51] 23 10 40
we use the popular CIFARIO-LYY CIFAR100-
LT 88, and ImageNet-LT 34 as imbalanced in-distribution datasets.
For CIFARIO/100-LT benchmarks, the imbalance ratio (i.e., P = max,, (n.)) min, (ny,) is set as 100 8 (1). The original CIAFR test sets are kept for evaluating the ID classification capability. For OOD detection, the Tinylmages80M [50] is adopted as the auxiliary OOD training data, and the (SC-OOD) benchmark (56].
test set is semantically coherent
For the large-scale ImageNet-LT benchmark, training samples are sampled from the original ImageNet-1k [12] dataset, and the validation set is taken for evaluation. We follow the OOD detection setting as Wang et al. [51] to use ImageNet- Nxt-Extra as auxiliary OOD training and ImageNet- Ik-OOD for testing. Randomly sampled from ImageNet-22k [12). ImageNet-Extte contains 517.711 images belonging to 500 classes, and ImageNet- Ik- OOD consists of 50,000 images from 1,000 classes. All the classes in ImageNet-LT, ImageNet-Extee Extra, and ImageNet- Ik-OOD not overlapped.
Evaluation Metrics. For OOD detection, we report three metrics: (1) AUROC, the area under the receiver operating characteristic curve, (2) AUPR, the area under the precision-recal curve, and (3) FPR95, the false positive rate of OOD samples when the true positive rate of ID samples are 95%. For ID classification, we measure the macro accuracy of the classifier. We report the mean and standard deviation of performance (%) over six random runs for each method.
Implementation Details. For the ID classifier f. following the settings of Wang et al. 151, we train R [8] models on the CIFARIO/100L1 benchmarks, and leverage ResNet50 models for the ImageNet-LT benchmark. Logit adjustment loss (39] is adopted to alleviate the imbalanced ID classification. Detailed settings are displayed in Appendix B. For the OOD detector g. as Bitterwolf et al. S suggest, we implement g as a binary discriminator (abbreviated as BinDisc) to perform ID/OOD identification. Detector g shares the same backbone (feature extractor) as classifier f. and only attaches an additional output node to the classification layer of f. In addition, we also add linear layer on top the backbone to produce the y factors to perform the training regularization with Eq. ( and Eq. (8). To reduce the optimization difficulty, the gradient is stopped between Eq. 7 and Eq. 8) (but still shared the backbone), where Eq. (7) aims at training g while Eq. (8) only optimize
y. Furthermore, to verify the versatility of our method, we also implement several representative

---

Table OOD detection evaluation on CIFARI0/100-L1 benchmarks. The best results are marked in bold, and the secondary results are marked with underlines. The base model is ResNet18.
CIFARIO-LT
CIFARI00-LT
Method
AUROCT
AUPRT
FPR95
ACC1
AUROCT
AUPRT
FPR95
ACC1
MSP
72.28
70.27
66.07
72.34
61.00
57.54
82.01
40.97
OECC
87.28
86.29
45.24
60.16
70.38
66.87
73.15
32.93
EnergyOE
89.31
88.92
40.88
74.68
71.10
67.23
71.78
39.05
89.77
87.25
34.65
73.84
72.91
67.16
68.89
39.04
PASCL
90.99
89.24
33.36
77.08
73.32
67.18
67.44
43.10
OpenSampling
91.94
91.08
36.92
75.78
74.37
75.80
78.18
40.87
ClassPrior
92.08
91.17
34.42
74.33
76.03
77.31
76.43
40.77
BalEnergy
92.56
91.41
32.83
81.37
77.75
78.61
73.10
45.88
EAT
92.87
92.40
28.83
81.31
75.45
70.87
64.01
46.23
COCL
93.28
92.24
30.88
81.56
78.25
79.37
74.09
46.41
PASCL + Ours
92.93
92.51
28.73
78.96
74.23
68.63
65.65
44.60
COCL + Ours
93.55
92.83
28.52
81.83
78.50
79.96
71.65
46.80
OOD detection methods (e.g., OF (20), Energy etc.) into binary discriminators, and equip them with our ImOOD framework. For more details please refer to Appendix B.2
Methods for comparison. In the following sections, mainly compare method on three benchmarks with the typical OOD detectors including OE [20], Energy [2), and BinDise [3], as
well as some state-of-the-an detectors such PASCL [51], ClassPrior [3], EAT [55], COCL [40],
etc.. Specifically, as the results on the ImageNet-Li benchmark reported by COCL share a large
discrepancy against PASCL (especially the AUPR measure), re-implement COCL based on their released code and report the aligned results in Tab.
4.2 Main Results
ImOOD significantly outperforms previous SOTA methods on CIFAR10/100-LT benchmarks. As shown in Tab. [ our ImOOD achieves new SOTA performance on both of CIFARI0//00-LT
benchmarks. Built on top of the strong baseline PASCL [51. our method leads to 1.9% increase of AUROC, 3.2% increase of AUPR, and 4.6% decrease of FPR95 on CIFAR 10-LT, with 0.9% 1.8% enhancements of respective evaluation metrics on CIFAR 100-LT. By integrating with COCL 44 our ImOOD further pushes the imbalanced OOD detection on CIFARI0/100-LT benchmarks towards higher performance, e.g., achieving 93.55%/78. 55% of AUROC respectively. To further demonstrate the efficacy, we validate our method on the wel-world large-scale ImageNet-LT the results are displayed below.
34 benchmark, and
Table 2: OOD detection evaluation the ImageNet-LT benchmark. The base model is ResNet50.
Method
AUROCT
AUPR1
FPR95
ACC1
MSP
53.81
51.63
90.15
39.65
OBCC
63.07
63.05
86.90
38.25
EnergyOB
64.76
64.77
87.72
38.50
OE
66.33
68.29
88.22
37.60
PASCL
68.00
70.15
87.53
45,49
EAT
69,84
69.25
87.63
46.79
COCL
73.87
72.63
76.35
51.00
PASCL Ours
74.69
73.08
74.37
46.63
COCL Ours
75.84
73.19
74.96
52.43

---

ImOOD achieves superior performance on the ImageNet-LT benchmark. As Tab. 2 implies. our ImOOD brings significant improvements against PASCL, e.g., 6.7% increase on AUROC and 13.2% decrease FPR95, and further enhance the SOTA method COCL. for better OOD detection performance, e.g.: : 75.84 of AUROC. Since the performance enhancement is much greater than those on CIFAR 10/100-LT benchmarks, we further statistic the class-aware error distribution on wrongly detected ID/OOD sample in Fig. AT] The results indicate our method builds a relatively better-balanced OOD detector on Net-L, which leads to higher performance improvements. Besides, as we follow the literature (51 40) to employ ResNet18 on CIFAR 10/100-LT while adopt ResNet50 ImageNet-LT, the model capacity also seems play vital role in balancing the OOD detection on imbalanced data distribution, particularly in more challenging real-world scenarios.
Additional comparison following ClassPrior's 's setting. Since ClassPrior [23] uses totally different setting against the literature [5T 40 on the ImageNet datasets, including different imbalance ratio and OOD test sets, we additionally compare with ClassPrior in Tab. 3 For fair comparison, as ClassPrior does not leverage real OOD data for training. we eliminate the auxiliary ImageNet-Exiie dataset and utilize the recent VOS [14] technique to generate OOD syntheses for regularization. According to Tab. our method consistently outperforms ClassPrior by a large margin on all subsets.
Table 3: Comparison on ClassPrior's ImageNet-LT-ae benchmark. The base model is MobileNet.
iNaturalist
SUN
Places
Textures
Method
AUROCT
FPR95
AUROCT
FPRY5
AUROC1
FPR95
AUROC1
FPR95
ClassPrior
82.51 86.15
66.06
80.08
69.12
74.33
79.41
69.58
78.07
Ours
59.13
81.29
65.88
77.57
76.26
72.82
72.73
4.3
Ablation Studies
In this section, conduct in-depth ablation studies on the CIFARIO-LT benchmark to assess the validity and versatility of our proposed ImOOD framework, and the results are reported as follows.
Table 4: Ablation on the % estimates and technique integration on the CIFAR10-LT benchmark.
Ty Estimates
AUROCT
AUPR1
FPR95]
ID ACC1
none
90.06
88.72
33.39
78.22
== const
89.75
86,28
32.67
78.50
= Por0
92.04
91.32
31.24
79.16
Yv = Tv:0 (x:)
92.23
91.92
29.95
79.56
PASCL
92.93
92.51
28.73
78.96
COCL
93.55
92.83
28.52
81.83
Lemma 3.1 (phal (y|=)) (z))))
P(g is consistent with empirical results. To validate that P(a)
the coefficient ),(()) (x)) depends on input sample æ and differs for each class we perform a series of ablation studies in Tab. 4 We first build a baseline model with BinDisc [5] only, and no extra regularization is adopted to mitigate the imbalanced OOD detection. As shown in the first row from Tab. 4. the baseline (denoted as none of 7 estimates) presents a fair OOD detection performance (e.g., 90.06% of AUROC and 33.39% of FPR95). Then, we simply take Ty constant for all classes (denoted as % = const) by assuming phol p (xlv) = P(x(y) and %2 = 1 to apply the training regularization in Sec. 3.3) According to Tab, 4 the OOD detection performance receives a slight decline of AUROC (from 90.06% to 89.75%), despite the better FPR95 result. Consequently, after treating Ty as learnable variable for each class y (denoted as 7y == 79:0), the detector receives significant improvement all the three measures of AUROC, AUPR, and FPR95. Finally, setting Yy as an input-dependent and class-aware learnable variable (denoted as Ty == 7y.0 (x)) brings further OOD detection enhancement.
R (see Appendix A.1

---

ID Data
ID Data
ID Data
Prier
Pfor
Prior
Distribution
0.3
gatema
0.3
0.3
0.1
0.1
0.1
0.0
0.0
Class Label
Class Label
Class Label
OOD Data
0OD Data
ODD Data
Frior ganema
Frior
Distribution
0.3
0.3
0.2
0.0
-0.2
0.1
-0.4
0.0
0.0
4
Class Prediction
Class Prediction
Class Prediction
Figure 2: Statistics %, 8, and A from the CIFAR1O-LT benchmark. (1) Upper: distributions on ID samples from head tail (left to right) class indices; (2) Lower: distributions on OOD samples predicted head to tail (left to right) ID classes.
In addition, we further statistics the practical distribution of in Fig. 2 where Vy (a:) (x:) is relatively
1 phh (aly)
higher for the ID sample from head classes. It is consistent with (xx) = K P( (V) (see Ap-
pendix A. I, as the data likelihood P(xly) is close to the balanced situation (Pbal (a:: (zly)) for head samples while over-estimated for tail classes, leading higher fraction of (mll) for head classes P(wly)
and lower for tail classes. Detailed discussion is presented Appendix A.2
The designed training-tine time regularization in Sec. 3.3 is effective. As shown in Tab. compared with the BinDise baseline in the first row, adding Eq. 7 and Eq. (8) by setting 7y Yy:0 (x) leads to significant improvement on the OOD detection performance (i.e., 2.2% increase of AUROC and 3.4% decrease of FPR95). As Fig. 2 shows, the automatically learned adjustments (e.g. and А) are consistent with our motivation, where we aim to punish the ID data from tail classes with large positive A, well the OOD samples predicted as ID head classes with large negative A (Sec. 3.3 Moreover, as indicated by Tab. integrating with PASCL [5I) and COCL 40) techniques further boosts the imbalanced OOD detection, ultimately resulting in new SOTA performance.
ImOOD generalizes to various OOD detection methods. To verify the versatility of our statistical framework, implement the ImOOD framework with different OOD detectors beside BinDise, including the OE (20), Energy (32) and the Mahalanobis distance [24]. For those detectors, we add extra linear layer conduct logistic regression top of their vanilla OOD scores (see Appendix B.2)), and leverage our training-time regularization to optimize those detectors in a unified manner. According to the results presented in Tab. our ImOOD consistently boosts the original OOD detectors by a large margin on all the AUROC, AUPR, and FPR95 measures. In real-world applications, one may choose a proper formulation of our ImOOD to meet specialized needs.
Table 5: Generalization to OOD detectors.
Table 6: Robustness to OOD test sets.
OOD Detector
Method
AUROC
AUPR1
FPR95
OOD Datase
Method
AUROC1
AUPR1
FPR95L
Prob-Based
OF
89.91
87.32
34.06
Far-OOD
PASCI.
96.63
98.06
12.18
91.52
97.50
Energy-Based
Energy +Ours
90.27
88.73
34.42
Near-OOD
PASCL
84.43
82,99
$7.27
91.41
90.63
31.81
+Ours
86.61
85.71
55.51
Dist-Based
88.26
Spurious-OOD
79.00
+Ours
89.30
88.68
39.55
+Ours
83.27
84.19
$7.69
ImOOD is robust to different OOD test sets. In the preceding sections, we evaluated our method on the CIFAR10-LT benchmark, where the SCOOD test set (56) comprises 6 subsets covering different scenarios. As suggested by Fort et al. [17]. the SVHN subset can be viewed as far OOD, and the

---

CIFARI00 subset can be seen as near OOD (with CIFARIO-LT as ID), According to the detailed results in Tab. [ our ImOOD brings consistent enhancement against the strong baseline PASCL regardless of the near or far OOD test set. Furthermore, we also report the spurious OOD detection evaluation in Tab. 6 Specifically, we follow Ming al. 42 to take WaterBird as the imbalanced ID dataset, which also suffers from the imbalance problem (on water birds and land birds), and a subset of Places [59] as the spurious OOD test set (with spurious correlation to background). Results in Tab. [ also demonstrate our method's robustness in handling spurious OOD problems, with considerable improvement 4.3% increase on AUROC and 5.96% decrease on FPR95. The robustness of ImOOD to various OOD testing scenarios is verified.
4.4 ImOOD's Inference-time Application
Despite our main focus on training more balanced OOD) detectors, also make some attempts to apply our method during pre-trained models' inference. According to our Theorem [3.2] for an existing OOD detector P()]z) (e.g., trained with BinDisc), we can calculate the bias term ((x) to
regulate the vanilla scorer P(i|z) into balanced phul (i|z) ((x) P(i]z). However, as B(x)
Ey Tyy (a) P(y)
P(ylz..), the estimation of (x) presents considerable difficulty without training, but we
have also tried some trivial approaches in Tab. 7
Table 7: Attempts to apply our ImOOD into pre-trained models' inference stages.
Method
Detector
AUROCT
AUPRT
FPR95.
BinDisc
P(i]z)
90.06
88.72
33.39
+Ours (infer)
P(il()
90.34
88.45
32.10
+Ours (infer)
P(il))
90.86
88.95
30.80
+Ours (train)
((x))
92.23
91.92
29.95
First, we simply treat (x)) constant (e-g., ()(,))y()=)))))))))))))()))))( = 7 1) for arbitrary input x: and class y to calculate the bias term (denoted as 31 (x:)), and the results on CIFAR10-LE benchmark immediately witness a performance improvement (e.g., 0.28% increase on AUROC and 1.29% decrease on FPR95) compared to the baseline OOD detector. However, the improvement is relatively insignificant, and the phenomenon is consistent with our ablation studies in Tab. 4 which demonstrates the importance of learning a class deassdde and seputddependent 7y (z) during training.
Then, inspired by the statistical results in Fig. we take further step use polynomial (rank=2) to fit the curve between the predicted class y and 7 (x) learned by another model, and apply the coefficients to estimate a class-dependent 9y for the baseline model (denoted as (1x) P(ilx)). This operation receives further enhancement on OOD detection and gets close to our learned model (e.g.. 30.80% v.s. 29.95% of FPR95).
In conclusion, our attempts illustrate the potential of applying our method an existing model without post-training, and we will continue to extend fy to an input-dependent version (say (y)) in our future work.
5 Conclusion and Discussion
This paper establishes a statistical framework ImOOD to formulate OOD detection on imbalanced data distribution. Through theoretical analysis, we find there exists class-aware biased item between balanced and imbalanced OOD detection models. Based on it, our ImOOD provides a unified training- time regularization technique alleviate the imbalance problem. On three popular imbalanced OOD detection benchmarks, extensive experiments and ablation studies to demonstrate the validity and versatility of our method. We hope work can inspire new research in this community.
Limitations. Following the literature, ImOOD utilizes auxiliary OOD training samples to refine the the ID/OOD decision boundary. However, unforeseen OOD samples in real-world applications could potentially challenge this boundary. To mitigate this issue, integrating online-learning strategies for adaptive decision decision-making during testing is promising avenue. We view this our future work.
10

---

Acknowledgments and Disclosure of Funding
This work was supported part by the Fundamental Research Funds for the Central Universities, in part by Alibaba Cloud through the Research Intern Program, and in part by Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23F01013F01.
References
[1] Yong Hyun Ahn, Gyeong-Mowe Park, and Seong Tae Kim. Line: Out-of-diitribution detection by leveraging important neurons. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19852-19862, 2023.
[2] Mouin Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi. Neco: Neural collapse based nut-of-distributin detection. In International Conference Learning Representa- tions, 2024.
[3] Yichen Bai, Zongbo Han, Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang. Id-like prompt learning few-shot out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17480- 17480-17489, 2024.
[4] Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEFE conference on compuder vision and pattern recognition, pages 1893-1902, 2015.
[5] Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of- distribution detection: Many methods based on ood training data estimate a combination of the same core quantities. In International Conference Machine Learning, pages 2041-2074. PMLR, 2022.
[6] Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. The balanced accuracy and its posterior distribution. In 2010 20th international conference pattern recognition, pages 3121-3124. IEEE, 2010.
[7] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem convolutional neural networks. Neural networks, 106:249-258 2018.
[8] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-- aware margin loss. Advances in neural information procexsing systems, 32, 2019.
[9] Senqi and Zhongfei Zhang. Deep hybrid models for out-of-distribution detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4733-4743, 2022.
[10] Hyunjun Choi, Hawook Jeong, distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages -15900,,00, 2023.
Young Choi. Balanced energy regularization loss for out-of-
[11] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3606-3611 3613, 2014.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[13] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In International Conference on Learning Representations, 2023.
[14] Xuefeng Du, Zhaoning Wang, Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In International Conference on Learning Representations, 2022.
[15] Xuefeng Du Yiyou Sun, and Yixuan When and how does in-distribution label help out-of-distribution detection? In Forty-first International Conference on Machine Learning, 2024.
[16] Sepidch Esmacilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-traincd model clip. Proceedings of the AAAI conference on artificial intelligence, 36(6): 6568-6576, 2022.
[17] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of diit-ribititttionnn detection. Advances in Neural Information Processing Systems,
2021.

---

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 770-778, 2016.
[19] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distriuution examples in neural networks. In International Conference on Learning Representations, 2017.
[20] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[21] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-689, 2021.
[22] Zhuo Huang, Xiaobo Xia, Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In International Conference on Learning Representations, 2023.
[23] Xue Jiang, Feng Liu, Zhen Fang. Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting out-of- distribution data through in-distribution class prior. In International Conference on Machine Learning. 2023.
[24] Ren Jic, Fort Stanislav, Liu Jeremiah, Roy Abhijit Guha, Padhy Shreyas, and Lakshminarayanan Balaji. simple fix to mahalanobis distance for improving near-ood detection, 2021.
[25] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of- distribution data, Advances in neural information processing systems, 13:20578-20589, 2020.
[26] Mare Lafon, Flias Ramzi, Clément Rambour, and Nicolas Thome Hybrid energy based model in the feature space for out-of-distribution detection. International Conference on Machine Learning, 2023.
[27] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3, 2015.
[28] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neurul information processing systems, 31, 2018.
[29] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of -uttof-diistribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[30] Kai Liu, Zhihang Fu, Chao Chen, Sheng Jin, Ze Chen, Mingyuan Tao, Rongxin Jiang, and Jieping Ye. Category-extensible out-of distribbbt detection via hierarchical context descriptions. In seirth-seventh Conference on Neural Information Processing Systems. 2023.
[31] Litian Liu and Yao Qin. Fast decision boundary based sut-of-distributio detector. In Forty-first Interna- tional Conference on Machine Learning, 2024.
[32] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-basee out-of -istiiiiibttiinn detection. Advances in neural information processing systems, 0121144-214777 2020.
[33] Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax- based out-of- distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23946-23993 2023.
[34] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2537-2546, 2019.
[35] Fan Lu, Kai Zhu, Wei Zhai, Kecheng Zheng, and Yang Cao. Uncertainty-awer aware optimal transport for semantically coherent
detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282 3291, 2023.
[36] Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.
[37] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
[38] Aditya Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical con- sistency of algorithms for binary classification under class imbalance. In International Conference on Machine Learning, pages 603-6 6611. PMLR, 2013.
12

---

[39] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kurmar. Long-tail learning via logit adjustment. In International Conference Learning Representations, 2021.
[40] Wenjun Miao, Guansong Pang, Tianqi Li, Xiao Bai, and Jin Zheng. Out-of-distribution detection in long-tailed recognition with calibrated outlier class learning. Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024.
[41] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-t--distribution detection with vision-language representations. Advances in Neural Information Processing Syxtems, 35: 35087-35102, 2022.
[42] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10051-10059, 2022.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-87887 8763. PMLR, 2021.
[45] Hitesh Sapkota and Qi Yu. Adaptive robust evidential optimization for open set detection from imbalanced data. In International Conference on Learning Representations, 2023.
[46] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In Advances in Neural Information Processing Systems, 2021.
[47] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest. neighbors. In International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[48] Leitian Tao, Xuefeng Du Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In International Conference on Learning Representations, 2023.
[49] Leitian Tao, Xuefeng Du Jerry Zhu, and Yixuan Li. Non- Non-parametric outlier synthesis. In International Conference on Learning Representations, 2023.
[50] Antonio Torralba, Rob Fergus, and William Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 19:18 1970, 2008.
[51] Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for Conference on Machine Learning, pages 23446-2344 -34588. PMLR, 2022.
detection in long- tailed recognition. In International
[52] Qizhou Wang. Junjie Ye, Feng Liu, Quanyu Dai, Mareus Kalander, Tongliang Liu, HAO Jianye, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In International Conference on Learning Representations, 2023.
[53] Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, HAO Jianye, and Bo Han. Out-of-distribution detection with implicit outler transformation. In International Conference on Learning Reprexentations, 2023.
[54] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng. An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization, In Internutional Conference on Machine Learning, pages 23631- 23644. PMLR, 2022.
[55] Tong Wei, Bo-Lin in Wang, and Min-Ling Zhang. Fat: Towards long-tailed out-of-distribution detection. Proceedings of the 38th AAAI Conference on Artificial Intelligence, 2024.
[56] Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent citt-f--iittiibutionn detection. In Proceedings of the IÉEE/CVF International Conference on Computer Vision, pages 8301-8309, 2021.
[57] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized survey, arXiv preprint 3rXiv:2110.1133..00 2021.
detection: A
13

---

[58] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.0333 03365. 2015.
[59] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40 (6):1452-1464, 2017.
[60] Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Rong Zhang. Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting out-of of-distribution detection with typical features. Advances in Neural Information Processing Systems, 35:20758 20769, 2022.
14

---

A Theorem Proofs
A.1 Proof for Lemma 3.1
From Bayesian Theorem, we have:
P(y P(xly) P(v)
(AI)
P(æ)
In baass-balanced scenarios, the class prior phl equals R. where K is the class number. The balanced classification posterior is given by:
phal (x (y)
phll (y]zz)
K
(A2)
phal
where the marginal probability phal (ar) P(x)) is independent from balanced or imbalanced class distribution. Combining with Eq. (A1) and Eq. (A2) we have:
phol (<: v) P(z
P(ylez)
P(ylux)
pbal (b)aa)
& (,(x)
(A3)
P(y)
P(y)
where (x)) = (h P(mly) captures the likelihood difference for the same sample between balanced and imbalanced distributions.
A.2 Discussion for Lemma 3.1 and Statistic Results in Fig. 2
According to Fig. that depicts the statistical distribution the CIFARIO-LT benchmark, 7g (xx) is higher for the ID sample from head classes than tail classes. This phenomenon is consistent with
Ty (x) K P(o)y) As the model has seen sufficient training examples for head classes, the data
likelihood P(æ| (n is approaching the balanced situation of ph the fraction of (all)) is P(x
closed to 1. Meanwhile, since large number of tail samples are not presented during training, the
sample space is narrowed down and the data likelihood P(ælv) ((xly) for tail classes is over-estimated
(even higher than (xly)) leading to lower fraction of (*l()))))
(x)
P(mly) Therefore, Ty (2) K p(()))
presents higher values for head classes than tail classes. The statistical results are well-aligned with the theoretical analysis.
A.3 Derivation for Eq. (6)
Substituting the sigmoid function (z) I+e-=
I into Eq. 5
we have:
1 1 Te--guun g = B(x)
(A4)
1te-(() s(x)
Consequently, have:
6 (((x)) = B(x)) -850 1(x()
(A5)
And then:
g(x)
log B(x) (x) -(x()))
log e (x) 1+ (8(x) - 1) ca)
(A6)
= (xr) - log (3(x) - 1) (()) +1
15

---

B Experimental Settings and Implementations
B.1 Training Settings
For fair comparison, we mainly follow PASCL's 51 settings. On CIFARI0/100-LIT benchmarks, train ResNet18 [T8] for 200 epochs using Adam optimizer, with batch size of 256. The initial learning rate is 0,001, which is decayed to 0 using a cosine annealing scheduler. The weight decay is x 10-4 On the ImageNet-LT benchmark, we train ResNet50 [18] for 100 epochs with SGD optimizer with the momentum of 0.9. The batch size is 256. The initial learning rate is 0.1, which is decayed by factor of 10 at epochs 60 and 80. The weight decay is 5 10-5 During training, each batch contains an equal number of ID and OOD data samples (i.e., 256 ID samples and 256 OOD samples). We use NVIDIA V V100-32G GPUs in all our experiments.
For a better performance, one may carefully tune the hyper--arameters to train the models.
B.2 Implementation Details
In the manuscript, we proposed a generalized statistical framework to formularize and alleviate the imbalanced OOD detection problem. This section provides more details on implementing different OOD detectors into unified formulation, e.g., a binary ID/OOD classifier 5
For BinDisc [ we simply append an extra ID/OOD output node to the classifier layer of a standard ResNet model, where ID classifier and OOD detector g share the same feature extractor. Then we adopt the sigmoid function to convert the logit g(x) into the ID/OOD probability ((i(z))
For Energy 322 following Du et al. [4], we first compute the negative free- free-ynergy
E(x: f) = log En efulm), and then attach an extra linear layer to calculate the ID/OOD
logit gax) = E(::; f) + b, where w, b are learnable scalars. Hence, the sigmoid function is able to convert the logit g(x) into the probability p(i]æ:)
For OE (20). similarly, we compute the maximum softmax-probabiing MSP(æ; f) cu(()
maxy E. E co and use another linear layer to obtain the ID/OOD logit g(x)
MSP(x; f) fb +6.
For Mahalanobis distance 24 we maintain an online feature-pool for each ID class y, as to calculate the Mahalanobis distances for the test samples as Dm (x)... Then, an extra linear
layer is adopted transform the distances ID/OOD logits as g(x) w Dm (x) + b.
By doing so, one can exploit the unified training-time regularization in Sec. 3.3 to derive strong
OOD detector.
C Additional Experimental Results
C.1 Additional Error Statistics
In this section, we present the class-aware error statistics for OOD detection on different benchmarks (see (T] and different detectors (see Fig. A2) J. For each OOD detector each benchmark, first compute the OOD scores g(x) for all the ID and OOD test data. Then, a threshold À is determined ensure high fraction of OOD data (i.e., 95%) correctly detected out-of-distribution. Recall that g(x) indicates the in-distribution probability for a given sample (i.e., P(ilar)). Finally, we statistic the distributions of wrongly detected ID/OOD samples.
Specifically, in Fig. AI and Fig. A2 the first row displays class labels of ID samples that are wrongly detected OOD (L.e., g(x) A), and the second row exhibits class predictions of OOD samples that are wrongly detected as ID (i.e., g(x) > A). In each subplot, we statistic the distribution over head, middle, and tail classes (the division rule follows Wang et al. (51)) for simplicity. Note that the total count of wrong OOD samples (in the second row) is constant, and a better OOD detector will receive fewer wrong ID samples (in the first row).
Fig. AT compares our method with PASCL (51) on CIFAR10/100-11 and ImageNet-I.T benchmarks. The results indicate our method (dashed bar) performs relatively more balanced OOD detection
16

---

all benchmarks. We reduce the error number of ID samples from tail classes, and simultaneously decrease the enror number of OOD samples that are predicted as head classes. In particular, our method achieves considerable balanced OOD detection ImageNet-LIT (the right column), which brings a significant performance improvement, as discussed in Sec. 4.2
Fig. A2 compares our integrated version and vanilla OOD detectors (i.e., OE, Energy, and BinDisc) on the CIFAR IO-LT benchmark. Similarly, the results indicate our method performs relatively more balanced OOD detection against all original detectors. The versatility of our ImOOD framework is validated, as discussed Sec.
However, the statistics in Fig. Al and Fig. A2 indicate the imbalanced problem has not been fully solved, since the scarce training samples of tail classes still affect the data-driven learning process. More data-level re-balancing techniques may he leveraged to further address this issue.
CIFARIO-LT
CIFAR100-LT
ImageNet-LT
PASCL
PASCL
PASCL
ID Lahel
head
middle
tail
head
middle
tail
head
middle
Ixill
PASCL
PASCIL
PASCL
O0D Prede
ouns
ours
head
middle classes
tail
head
middle
tail
head
middle classes
taill
classcs
Figure A1: Class-aware error statistics for OOD detection on different benchmarks.
OE
Energy
BinDisc
vanilla
vanilla
vanilla
TD
middle
vanilla ours
vanilla ours
vanilla
OOD Preds
ours
head
middle classes
tail
head
middle
tail
head
middle
taii
classcs
classes
Figure Class-aware error statisties for different OOD detectors on CIFARIO-LT.
C.2 Detailed Results CIFAR10/100-L17 Benchmarks
For CIFAR 10/100-LT benchmark, Textures SVHN 43 CIFAR100/10 (respectively), TinyIma- geNet (27), LSUN (58), and Places365 [59] from SC-OOD dataset [56] adopted OOD test sets.
The mean results on the six OOD sets are reported Sec.
In this section, we reported the detailed measures on those OOD test sets in Tab. Al and Tab. A2 the supplementary to Tab. [ The results indicate our method consistently outperforms the state-of-hh-an PASCL [5]] most of the subsets.
17

---

Table A1: Detailed results CIFARIO-LT.
Table A2: Detailed results CIFARIOO-LT.
Di
Method
AUROC (1)
AUPR (1)
FPR95 (1)
Dan
Method
AUROC (1)
AUPR (T)
FPR95 (4)
PASCL
93.16 d: 0.37
84,80 1.50
23.26 1: 0.91
PASCL
76.01 4: 0.66
58.12 à: 1.06
67.43 + 193
Texture
96.58
94.09
Textre
Ours
0.55
62.76
PASCL
96.63 # 0.90
98.06
0.56
12.18 3.53
PASCI.
80.19 2.19
88.49 + 1.59
53.45 3.60
SVIIN
Ours
97.50 0.47
0.41 98.26
9.65
SVIN
Ours
84.20 :: 0.34
91.86 1:
47.50
PASCL
84.43 1: 0.23
82.99
:: 0.48
51.27
1: 0.88
PASCL
62.33 1: 0.38
57.14 1: 0.20
79.55 0.84
CIFAR100
Ours
86.61 0.19
85,71
0.12
55.51
0:41
CIFARIO
Ours
61.53 +: 0.43
56.56 4 0.42
79.19 0.65
PASCL
87.14:1...1
81.54
0.38
47.69
0.59
PASCL
68.20 0.37
51.53 1:
76.11 0.80
TIN
Ours
88.75 = 0.35
86.27 = 0.50
40.52 +: 0.32
TIN
Ours
68,42 :: 0.27
52.15 0.21
75.54 0.60
PASCL
93.17 0.15
91.76 + 0.53
26.40 + 1.00
PASCL
77.19 0.44
61.27 +
63.31
LSUN
Ours
94.55 =: 0.23
93.70 = 0.34
22,02 +: 0.37
LSUN
Ours
77.68 =: 0.20
61.66 + 0.38
60.32 0.32
PASCL
91.43 0,17
96.28 4: 0.14
33.40 +: 0.88
PASCL
76.02 :: 0.21
86.52 0.29
64.81
Places365
Ours
93.57 0.14
97.04
0.21
28.43 1: 0.61
Placos365
Ours
76.19 4: 0.13
86.79 1: 0.20
62.48 0.45
PASCL
90.99 + 0.19
89.24
+ 0.34
33.36 + 0.79
PASCL
73.32 + 0.32
67.18 + 0.10
67.44 0.58
Average
92.51
= 0.20
+ 0.48
Average
+ 0.35
68.60 +
65.65
C.3
Comparison on Different Imbalance Ratios
In our manuscript, we mainly take the default imbalance ratio (p- 100), which means the least frequent (tail) class only has 100 of training samples than the most frequent (head) class). Here, we follow PASCL (51) investigate another imbalance ratio of p = 50 (relatively more balanced than م 100) CIFARIO-LT According to Tab. A3 our method consistently surpasses PASCL on various imbalance levels, and gains a larger enhancement (e.g., near 2.0% of AUROC) on more imbalanced scenario with p 100, which further demonstrates our effectiveness in mitigating imbalanced OOD detection.
Table A3: Evaluation on different imbalance ratio p for the CIFARIO-LT benchmark.
100
50
Method
AUROCT
AUPRT
FPR95
AUROCT
AUPR
FPR95
OE
89.77
87.25
34.65
93.13
91.06
24.73
PASCL.
90.99
89.24
33.36
93.94
92.79
22.08
Ours
92.93
92.51
28.73
94.37
94.24
19.72
C.4 Investigation on SOTA General OOD Detection Methods
Besides the method specifically designed for imbalanced OOD detection, we also compare with several recently published detectors that aims at general (or, balanced) OOD detection:
NECO 2
studies the neural collapse phenomenon develops a novel post-hoc method to leverage geometric properties and principal component spaces identify OOD data.
investigates model's decision boundaries and proposes to detect OOD using the feature distance to decision boundaries.
iDBD 31
theoretically delineates the impact of ID labels on OOD) detection, and utilizes ID labels to enhance O0D detection via representation characterizations through spectral decomposition on the graph.
IDL.abll
15
ID-like
3 leverages the powerful vision-language model CLIP to identify challenging OOD samples/categories from the vicinity space of the ID classes to further facilitate OOD detection.
The experimental results are displayed in Tab. A4 On the CIFARIO-LT benchmark, our method surpasses the recent OOD detectors by large margin. For imbalanced data distribution, IDLabel [[5] struggles to capture the distributional difference across ID classes, while the pure post-hoc methods NECO 2 and fDBD [I even get worse results because they fail to generalize the scenario with highly skewed feature spaces and decision boundaries. On the ImageNet-LT benchmark, even the
18

---

incorporation of the powerful CLIP model cannot well address the imbalance problem for ID-like [3] while our method can specifically and effectively facilitate imbalanced OOD detection with the help of our theoretical groundness.
Table A4: Comparison with SOTA general OOD detectors.
Benchmark
Method
Pub& Year
AUROCT AUPR1
FPR95
NECO
ICLR'24
85.15
82.39
40.44
IDBD
ICML'24
87.90
83.07
41.98
CIFARIO-LT
IDLabel
ICML'24
90.06
88.29
34.66
Ours
93.55
92.83
28.52
ID-like
CVPR'24
72.05
71.37
78.36
ImageNet-LT
Ours
75.84
73.19
74.96
C.5
Additional Analysis on Correctly-detected ID and OOD Samples
In Fig. aa we reveal that the class labels for wrongly-identified II) samples and the class predictions for wronglyydetecter OOD samples are both sensitive the the ID class distribution prior. Here, indicates the correctly-detected ID and OOD samples are insensitive to the class distribution.
Fig. A3
OE
Energy
PASCL
Prior Freq
Prior Freq
Prior Freq
number
ID Labols
ID Labels
ID Labels
classes
classes
classes
OE
Energy
PASCL
Prioe Freq
Freq Preds
Freq
OOD Predk
Preds
number
classes
classes
classes
Figure A3: Statistics on correctly-detected ID and OOD samples.
In particular, the per-class prediction quantity of correctly-detected OOD samples in Fig. A3 may not precisely describe the statistical distribution, as the maximum ID-class prediction probability is relatively low (e.g., Payx,,P(yyl i) = 0.12 for 10-category classification) and introduce extra noise. Therefore, supplement the statistics of per-class prediction probability in Tab. A5 and all of the distributions for OE, Energy, and PASCL are nearly even.
Table A5: Average per-class prediction probability for correctly-detecte OOD samples.
Method
clsı
clsu
cls3
cls4
clsa
clsc
cls7
clss
clsg
cls10
OE
0.12
0.11
0.14
0.14
0.13
0.11
0.16
0.11
0.11
0.12
Energy
0.28
0.25
0.31
0.26
0.29
0.29
0.39
0.32
0.36
0.33
PASCL
0.15
0.12
0.12
0.14
0.11
0.12
0.13
0.12
0.11
0.11
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately made the claims to reflect the paper's contributions and scope
in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of our work in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Please refer to Sec. 3.2 and Appendix
A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided the experimental details in Sec. and Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
Please refer to http:p://g.ttuub.co//aaiiaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
aassooooiccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwiii/gguie///eeeeeeoecc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec. and Appendix B.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Appendix C.I
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported the compute resource requirements in Appendix B.I B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences that have been discussed in previous works, none of which feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
Please follow the LICENSE in mtttts:://gtthiii com www..//aaaaaammmm//oooaooooooaaa
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All our experimental data and models open source and obtainable.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer http:////ggttuub..ccaaaaaaaaaaa
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "tBUdUyXMxu", "reviewer": "Reviewer_5Lex", "summary": "The paper focuses on imbalanced data distribution, and finds that there is a bias term between balanced and imbalanced classification which can be used to explain the performance gap. To account for this bias, the authors introduce a regularization term, and show improved results on benchmarks.", "questions": "- Have you tested the benefits of your method with varying levels of imbalance? How does the performance change?", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is interesting to quantify the performance gap, and the proposed regularization technique is theoretically grounded and addresses the issue.\n- ImOOD can be generalized to existing OOD detection techniques and can also be combined with methods which use auxiliary data, making it easily adaptable. \n- The method achieves outperforms other OOD detection schemes across datasets and metrics.\n- The ablations presented are compelling and highlight the intended performance of ImOOD.", "weaknesses": "- Train-time regularization techniques cannot be applied to existing pretrained models, which may limit their adoption.\n- The method requires OOD data during training, and results may not generalize to other unseen OOD settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 5Lex:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: Train-time regularization techniques cannot be applied to existing pretrained models, which may limit their adoption.\n\n**A**: We have also tried to apply our method during pre-trained models' inference. According to our Theorem 3.2 in our manuscript, for an existing OOD detector $P(i|x)$ (e.g., trained with BinDisc), we can calculate the bias term $\\beta(x)$ to regulate the vanilla scorer into balanced $P^{bal}(i|x) = \\beta(x) \\cdot P(i|x)$. However, as $\\beta(x) = \\sum_y{\\gamma_y(x)\\frac{P(y|x,i)}{P(y)}}$, the estimation of $\\gamma_y(x)$ presents considerable difficulty without training, but we have also tried some trivial approaches to adapting our method to inference time.\n\n|Method|Detector|AUROC|AUPR|FPR95|\n|-|-|:--:|:--:|:--:|\n|BinDisc|$P(i\\|x)$|90.06|88.72|33.39|\n|+Ours (infer)|$\\beta_1(x)P(i\\|x)$|90.34|88.45|32.10|\n|+Ours (infer)|$\\hat{\\beta}(x)P(i\\|x)$|90.86|88.95|30.80|\n|+Ours (train)|$\\beta(x)P(i\\|x)$|**92.23**|**91.92**|**29.95**|\n\nFirst, we simply treat $\\gamma_y(x)$ as a constant (e.g., $1$) for arbitrary input $x$ and class $y$ to calculate the bias term (denoted as $\\beta_1(x)$), and the results on CIFAR10-LT benchmark immediately witness a performance improvement (e.g., 0.28% increase on AUROC and 1.29% decrease on FPR95) compared to the baseline OOD detector.\nHowever, the improvement is relatively insignificant, and the phenomenon is consistent with our ablation studies in Table 3, which demonstrates the importance of learning a _class-dependent_ and _input-dependent_ $\\gamma_y(x)$ during training.\n\nThen, inspired by the statistical results in Figure 2, we take a further step to use a polynomial (rank=2) to fit the curve between the predicted class $y$ and $\\gamma_y(x)$ learned by another model, and apply the coefficients to estimate a _class-dependent_ $\\hat{\\gamma}_y$ for the baseline model (denoted as $\\hat{\\beta}(x)P(i|x)$).\nThis operation receives further enhancement on OOD detection and gets close to our learned model (e.g., 30.80% v.s. 29.95% of FPR95).\n\nIn conclusion, our attempts illustrate the potential of applying our method to an existing model without post-training, and we will continue to extend $\\hat{\\gamma}_y$ to an _input-dependent_ version (say $\\hat{\\gamma}_y(x)$) in our future work.\n\n\n> **W2**: The method requires OOD data during training, and results may not generalize to other unseen OOD settings.\n\n**A**: We follow PASCL[1]'s setting to evaluate our method in imbalanced OOD detection with auxiliary OOD data during training, which is a commonly used setting in the literature[2][3][4] because auxiliary OOD data is relatively easy to obtain and usually generalizes well to unseen OOD samples[2][5]. \n\nOn the other hand, to further evaluate our method without real OOD training data, we also integrate with feature-level synthesis methods (e.g., VOS[6] and NPOS[7]) to generate pseudo-OOD data for training. \nIn particular, we follow ClassPrior[8]'s setting to train a MobileNet model on ImageNet-LT-a8 with VOS for OOD synthesis, and evaluate the performance on four OOD test sets. \nOur method outperforms the SOTA ClassPrior method by a large margin on all subsets.\n\n|Method|iNaturalist||SUN||Places||Textures||**Average**||\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n||AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|\n|ClassPrior|82.51|66.06|80.08|69.12|74.33|79.41|69.58|78.07|76.63|73.16|\n|**Ours**|**86.15**|**59.13**|**81.29**|**65.88**|**77.57**|**76.26**|**72.82**|**72.73**|**79.45**|**68.50**|\n\n\n> **Q1**: Have you tested the benefits of your method with varying levels of imbalance? How does the performance change?\n\n**A**: Yes. In our original manuscript, we mainly take the default imbalance ratio ($\\rho=100$), which means the least frequent (tail) class only has $\\frac{1}{100}$ of training samples than the most frequent (head) class). Here, we follow PASCL to investigate another imbalance ratio of $\\rho=50$ (relatively more balanced than $\\rho=100$) on CIFAR10-LT, and the results are presented as follows:\n\n|Method|$\\rho=100$|||$\\rho=50$|||\n|--|:----:|:---:|:----:|:---:|:----:|:-----:|\n||AUROC|AUPR|FPR95|AUROC|AUPR|FPR95|\n|OE|89.77|87.25|34.65|93.13|91.06|24.73|\n|PASCL|90.99|89.24|33.36|93.94|92.79|22.08|\n|**Ours**|**92.93**|**92.51**|**28.73**|**94.37**|**94.24**|**19.72**|\n\nAccordingly, our method consistently surpasses PASCL on various imbalance levels. Specifically, our method gains a larger enhancement (e.g., near 2.0% of AUROC) on the more imbalanced scenario with $\\rho=100$, which further demonstrates our effectiveness in mitigating imbalanced OOD detection.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.\n\n[2] Hendrycks et al. Deep Anomaly Detection with Outlier Exposure. ICLR'19.\n\n[3] Wei et al. EAT: Towards Long-Tailed Out-of-Distribution Detection. AAAI'24.\n\n[4] Miao et al. Out-of-distribution detection in long-tailed recognition with calibrated outlier class learning. AAAI'24.\n\n[5] Yang et al. Generalized out-of-distribution detection: A survey. IJCV'24.\n\n[6] Du et al. Vos: Learning what you don't know by virtual outlier synthesis. ICLR'22.\n\n[7] Tao et al. Non-parametric outlier synthesis. ICLR'23.\n\n[8] Jiang et al. Detecting out-of-distribution data through in-distribution class prior. ICML'23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n0q52VcRvp", "reviewer": "Reviewer_E388", "summary": "The paper addresses the challenge of detecting and rejecting out-of-distribution (OOD) samples by neural networks, particularly when the in-distribution (ID) data is inherently imbalanced. The authors observe that existing OOD detection methods struggle under these conditions primarily because they either misclassify ID samples from minority (tail) classes as OOD or mistakenly identify OOD samples as belonging to majority (head) classes.\n\nTo tackle this issue, the authors introduce a new statistical framework called ImOOD, which is designed to understand and improve OOD detection in the presence of imbalanced data. The framework considers the class-aware biases that arise due to data imbalance and incorporates a unified training-time regularization technique aimed at mitigating these biases.", "questions": "Please refer to the Weaknesses", "limitations": "Please refer to the Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction of the ImOOD framework provides a new way to conceptualize and address the problems arising from data imbalance in OOD detection.\n\nThe paper not only proposes a theoretical model but also validates it with empirical results, showing consistent performance improvements across multiple datasets.", "weaknesses": "A minor issue is that, when we know in advance that the data are long-tailed, it is a common practice that we will use the long-tailed learning methods to counteract its impacts. In this case, is the study of long tailed OOD detection an important topic? Or, even with long-tailed learning, the OOD detection therein still be a critical issue?\n\nI am not sure if the prior information can be properly estimated, considering the situations that the adopted models are not well calibrated (otherwise, OOD detection will not be a challenging task).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer E388:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: A minor issue is that, when we know in advance that the data are long-tailed, it is a common practice that we will use the long-tailed learning methods to counteract its impacts. In this case, is the study of long tailed OOD detection an important topic? Or, even with long-tailed learning, the OOD detection therein still be a critical issue?\n\n**A**: Yes, this issue has already been investigated and verified by PASCL[1]. According to the results on the CIFAR10-LT benchmark below, a simple combination of popular long-tailed learning methods (e.g., Reweighting[2], $\\tau$-norm[3], LA[4], etc.) does not address well the imbalanced OOD detection problem. Instead, PASCL treats imbalanced ID classification and OOD detection as a **joint** problem and achieves considerable improvement over the baselines, and our method further boosts imbalanced OOD detection significantly with the help of our theoretical groundedness.\n\n|OOD + LTR|AUROC|AUPR|FPR95|\n|:-------------|:---------:|:---------:|:---------:|\n|OE + None|89.92|87.71|34.80|\n|OE + Reweight|89.34|86.39|37.09|\n|OE + $\\tau$-norm|89.58|85.88|33.80|\n|OE + LA|89.46|86.39|34.94|\n|PASCL|90.99|89.24|33.36|\n|**Ours**|**92.93**|**92.51**|**28.73**|\n\n> **W2**: I am not sure if the prior information can be properly estimated, considering the situations that the adopted models are not well calibrated (otherwise, OOD detection will not be a challenging task).\n\n**A**: In previous studies, statistics on label frequency is a commonly-adopted estimate for class prior information (i.e., $P(y) := \\frac{n_y}{n}$)[4][5], and automatically learning the data density $P(x)$ (e.g., by normalizing flow[6]) is also proved applicable. Therefore, we follow the literature to estimate the class-aware and input-dependent scaling factor $\\gamma_y(x) = \\frac{1}{K}\\frac{P^{bal}(x|y)}{P(x|y)}$ by automatic optimization, and our empirical statistics on Figure 2 has validated the estimated results.\nOn the other hand, we also believe that a more precise estimation will further enhance our methods.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.\n\n[2] Cui et al. Class-balanced loss based on effective number of samples. CVPR'19.\n\n[3] Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR'20.\n\n[4] Menon et al. Long-tail learning via logit adjustment. ICLR'21.\n\n[5] Jiang et al. Detecting out-of-distribution data through in-distribution class prior. ICML'23.\n\n[6] Kumar et al. Normalizing flow based feature synthesis for outlier-aware object detection. CVPR'23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3Bb1MYix3", "reviewer": "Reviewer_QTK3", "summary": "This manuscript introduces ImOOD, a statistical framework addressing the OOD detection problem in imbalanced data distributions, identifying common issues such as misidentifying tail class ID samples and erroneously predicting OOD samples as head class ID. It reveals a class-aware bias between balanced and imbalanced OOD detection and proposes a unified training-time regularization technique to mitigate this bias. The method demonstrates consistent performance improvements on benchmarks such as CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, surpassing several state-of-the-art OOD detection methods.", "questions": "Why not obtain the class-specific scaling factor using methods from the long-tailed image recognition domain? Additionally, there is a lack of sufficient experimental evidence to show that using parametric mapping to obtain the class-specific scaling factor is more effective. The paper compares relatively few OOD detection methods; it is recommended to include more recent OOD detection approaches. The manuscript also lacks a summary of related works in class-imbalanced learning.", "limitations": "Please see Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Theoretical Analysis and Insight: The manuscript introduces a theoretical analysis that reveals a class-aware bias between balanced and imbalanced OOD detection, providing a deeper understanding of the underlying challenges in OOD detection tasks.\n\n2. Strong Experimental Results: The experimental results showcase the method's effectiveness and scalability, demonstrating consistent improvements across various benchmarks, which highlights the robustness and practical applicability of the proposed solution.", "weaknesses": "Although the manuscript provides insights through theoretical analysis and proposes a method to improve out-of-distribution (OOD) detection in class-imbalanced scenarios, it essentially estimates a class-specific scaling factor, which is a core aspect of long-tailed recognition and class-imbalanced learning. This is neither a new nor an interesting problem. Additionally, the proposed parametric mapping increases the difficulty of network optimization. It seems that OOD detection under class imbalance is merely an overlap of the two tasks: class-imbalanced learning and OOD detection. Using techniques from both fields can address this issue quite well. Therefore, the significance of the problem addressed in the manuscript is questionable and warrants further consideration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer QTK3:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: It seems that OOD detection under class imbalance is merely an overlap of the two tasks: class-imbalanced learning and OOD detection. Using techniques from both fields can address this issue quite well.\n\n**A**: OOD detection and class-imbalanced learning are _not_ seperate tasks. This issue has already been investigated and verified by PASCL[1]. According to the results on the CIFAR10-LT benchmark below, a simple combination of popular long-tailed learning methods (e.g., Reweighting[2], $\\tau$-norm[3], LA[4], etc.) does not address well the imbalanced OOD detection problem. Instead, PASCL treats imbalanced ID classification and OOD detection as a **joint** problem and achieves considerable improvement over the baselines, and our method further boosts imbalanced OOD detection significantly with the help of our theoretical groundedness. We hope the discussion and experiments can help establish the significance and contribution of our paper.\n\n|OOD + LTR|AUROC|AUPR|FPR95|\n|-|:--:|:--:|:--:|\n|OE + None|89.92|87.71|34.80|\n|OE + Reweight|89.34|86.39|37.09|\n|OE + $\\tau$-norm|89.58|85.88|33.80|\n|OE + LA|89.46|86.39|34.94|\n|PASCL|90.99|89.24|33.36|\n|**Ours**|**92.93**|**92.51**|**28.73**|\n\n> **Q1**: Why not obtain the class-specific scaling factor using methods from the long-tailed image recognition domain?  \n\n**A**: In our paper, the estimate of class prior $P(y)$ is actually borrowed from the long-tailed recognition literature[4] (using label frequency), while we have not currently[5] seen a reliable method to calculate the class-specific factor over balanced and imbalanced data likelihood $\\gamma_y(x) = \\frac{1}{K}\\frac{P^{bal}(x|y)}{P(x|y)}$. Therefore, we choose to learn it by automatic parametric mapping, which has been proven effective[6] when estimating the data density $P(x)$, and the empirical and statistical results in Table 3 and Figure 2 have further confirmed the effectiveness of our estimated $\\gamma_y(x)$. Despite that, we are also glad to integrate our method into reliable estimates of $\\gamma_y(x)$ for a further study if applicable.\n\n\n> **Q2**: The paper compares relatively few OOD detection methods; it is recommended to include more recent OOD detection approaches. \n\n**A**: Thanks for this suggestion. We have additionally compared with several SOTA OOD detection methods recently published in top conferences (ICLR'24, CVPR'24, and ICML'24): \n\n* NECO[7] studies the neural collapse phenomenon develops a novel post-hoc method to leverage geometric properties and principal component spaces to identify OOD data. \n* fDBD[8] investigates model's decision boundaries and proposes to detect OOD using the feature distance to decision boundaries.\n* IDLabel[9] theoretically delineates the impact of ID labels on OOD detection, and utilizes ID labels to enhance OOD detection via representation characterizations through spectral decomposition on the graph.\n* ID-like[10] leverages the powerful vision-language model CLIP to identify challenging OOD samples/categories from the vicinity space of the ID classes, so as to further facilitate OOD detection.\n\nThe experimental results are displayed as follows:\n\n|Benchmark|Method|Pub&Year|AUROC|AUPR|FPR95|\n|:--|:--:|:--:|:--:|:--:|:--:|\n|CIFAR10-LT|NECO|ICLR'24|85.15|82.39|40.44|\n||fDBD|ICML'24|87.90|83.07|41.98|\n||IDLabel|ICML'24|90.06|88.29|34.66|\n||**Ours**|**-**|**93.55**|**92.83**|**28.52**|\n|Imagenet-LT|ID-like|CVPR'24|72.05|71.37|78.36|\n||**Ours**|**-**|**75.84**|**73.19**|**74.96**|\n\nOn the CIFAR10-LT benchmark, our method surpasses the recent OOD detectors by a large margin. For imbalanced data distribution, IDLabel[9] struggles to capture the distributional difference across ID classes, while the pure post-hoc methods NECO[7] and fDBD[8] even get worse results because they fail to generalize to the scenario with highly skewed feature spaces and decision boundaries. On the ImageNet-LT benchmark, even the incorporation of the powerful CLIP model cannot well address the imbalance problem for ID-like[10], while our method can specifically and effectively facilitate imbalanced OOD detection with the help of our theoretical groundness.\n\nWe will add those experiments and discussions to further illustrate the novelty and contribution of our paper.\n\n> **Q3**: The manuscript also lacks a summary of related works in class-imbalanced learning.\n\n**A**: Thanks for this suggestion. We will supplement the discussion of current class-imbalanced learning methods[5] (via class re-balancing, information augmentation, module improvement, etc.), and specifically illustrate that the imbalanced ID recognition and OOD detection are a joint problem in the literature (the response to W1) to highlight our contribution.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.\n\n[2] Cui et al. Class-balanced loss based on effective number of samples. CVPR'19.\n\n[3] Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR'20.\n\n[4] Menon et al. Long-tail learning via logit adjustment. ICLR'21.\n\n[5] Zhang et al. Deep long-tailed learning: A survey. TPAMI'23.\n\n[6] Kumar et al. Normalizing flow based feature synthesis for outlier-aware object detection. CVPR'23.\n\n[7] Ammar et al. NECO: NEural Collapse Based Out-of-distribution Detection. ICLR'24.\n\n[8] Liu et al. Fast Decision Boundary based Out-of-Distribution Detector. ICML'24.\n\n[9] Du et al. When and how does in-distribution label help out-of-distribution detection? ICML'24.\n\n[10] Bai et al. ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection. CVPR'24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Pxs2hg6Bl", "reviewer": "Reviewer_XZSm", "summary": "The paper proposes the ImOOD framework to address class imbalance issues in OOD detection. Through Bayesian analysis, the authors identify critical biases and provide a unified regularization technique to improve detection performance. They conduct extensive experiments, demonstrating ImOOD's effectiveness on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets.", "questions": "- Why do the ID labels in Figure 1a show similar fluctuations across OE, Energy, and PASCL?\n- I believe the OOD prediction results in Figure 1a do not support the authors' claim. If I am mistaken, please correct me. The distribution of OOD predictions appears consistent with the prior distribution, with no significant deviation in the head region. For OOD samples, since the model hasn't been trained on them, it would predict according to the training sample distribution (prior distribution). I don't see this behavior as problematic.\n- Other questions: Please refer to the weaknesses section above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The paper offers a solid theoretical foundation, explaining the class-aware bias and providing a unified regularization approach.\n- Extensive experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "- The method relies on external OOD data, which is often difficult to obtain in OOD detection settings.\n- The comparisons in CIFAR and ImageNet experiments seem inconsistent, with fewer methods evaluated on ImageNet. Notably, ClassPrior seems to perform better on ImageNet.\n- Some non-long-tail-specific OOD detection methods in the ClassPrior paper also perform well. Given the emergence of many excellent OOD detection methods since ClassPrior's publication, comparing and discussing these methods with the proposed approach would provide a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XZSm:\n\nWe thank the reviewer for the valuable time and constructive suggestions, and our point-to-point responses are presented below:\n\n> **W1**: The method relies on external OOD data, which is often difficult to obtain in OOD detection settings.\n\n**A**: We follow PASCL[1]'s setting to evaluate our method for imbalanced OOD detection with auxiliary OOD data during training, which is a commonly-used setting in the literature[2][3][4] because auxiliary OOD data is relatively easy to obtain in practice[4][5]. \n\nOn the other hand, to further evaluate our method without real OOD training data, we also integrate with feature-level synthesis methods (e.g., VOS[6] and NPOS[7]) to generate pseudo-OOD data for training. \nIn particular, we follow ClassPrior's setting to train a MobileNet model on ImageNet-LT-a8 with VOS for OOD synthesis, and evaluate the performance on four OOD test sets. \nOur method outperforms the SOTA ClassPrior method by a large margin on all subsets.\n\n|Method|iNaturalist||SUN||Places||Textures||**Average**||\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n||AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|AUROC|FPR95|\n|ODIN|64.68|93.78|74.29|79.42|69.94|89.70|69.06|82.23|69.49|86.28|\n|GradNorm|70.87|78.12|69.70|67.59|66.00|85.75|63.09|74.89|67.41|76.59|\n|Dice|65.61|86.40|69.35|66.38|65.95|88.42|68.85|68.19|67.44|77.35|\n|ClassPrior|82.51|66.06|80.08|69.12|74.33|79.41|69.58|78.07|76.63|73.16|\n|**Ours**|**86.15**|**59.13**|**81.29**|**65.88**|**77.57**|**76.26**|**72.82**|**72.73**|**79.45**|**68.50**|\n\n> **W2**: The comparisons in CIFAR and ImageNet experiments seem inconsistent, with fewer methods evaluated on ImageNet. Notably, ClassPrior seems to perform better on ImageNet.\n\n**A**: Thanks for pointing out this. The reason is that ClassPrior uses a totally different setting against the literature[1-4] that we follow, and the differences are detailed as follows:\n\n|Settings|Model|ID Data|OOD Training Data|OOD Test Data|\n|:--|:--|:--|:--|:--|\n|ClassPrior|ResNet101/MobileNet|ImageNet-LT-a8|-|iNaturalist & SUN & Places & Textures|\n|Others|ResNet50|ImageNet-LT|ImageNet-1k-Extra|ImageNet-1k-OOD|\n\nIn Table 1 of our manuscript, we take ClassPrior's results on CIFAR10/100-LT from COCL's paper. Here, we provide a further comparison on ImageNet following ClassPrior's setting, and the results are reported in the above table (in the response to W1), which demonstrates our superiority against ClassPrior and other OOD detection methods.\n\n> **W3**: Some non-long-tail-specific OOD detection methods in the ClassPrior paper also perform well. Comparing and discussing these methods with the proposed approach would provide a more comprehensive evaluation.\n\n**A**: Thanks for this suggestion. On one hand, after aligning with ClassPrior's setting, we have further illustrated our method's efficacy over other OOD methods. \n\nOn the other hand, we have also compared it with several SOTA OOD detection methods published in the most recent top conferences (ICLR'24, CVPR'24, and ICML'24). \nDue to the space limit, please kindly refer to our response to Reviewer QTK3 (R3)'s Question 2 (Q2). \n\nWe will add those experiments and discussions to further illustrate the novelty and contribution of our paper.\n\n> **Q1**: Why do the ID labels in Figure 1a show similar fluctuations across OE, Energy, and PASCL?\n\n**A**: In Figure 1a, the `ID labels` actually means `label distribution for **wrongly-detected** ID samples`, and OE, Energy, and PASCL show similar fluctuations because they face the same problem: wrongly detecting samples from tail ID classes as OOD data, which inspires us to formulate this phenomenon and develop a unified solution.\n\n> **Q2**: I believe the OOD prediction results in Figure 1a do not support the authors' claim. If I am mistaken, please correct me. The distribution of OOD predictions appears consistent with the prior distribution, with no significant deviation in the head region. For OOD samples, since the model hasn't been trained on them, it would predict according to the training sample distribution (prior distribution). I don't see this behavior as problematic.\n\n**A**: The fact that models tend to predict according to training data distribution is exactly the core problem for imbalanced OOD detection. \n\nFirst, we may kindly clarify that Figure 1a displays the class predictions for OOD samples that are **wrongly-detected** as ID data, and all three OOD detectors tend to wrongly detect OOD data as head ID classes. On the other hand, we have shown in our _uploaded PDF file_ that both of the **correctly-detected** ID and OOD samples are **insensitive** to the class distribution. \n\nTherefore, we argue that this behavior (the model will predict according to training distribution) is exactly the problem to solve (e.g., _wrongly_ detecting OOD samples as ID data from head class), which is similar to the problem studied by the long-tailed recognition community (wrongly recognizing test samples as head classes).\nWe are thus motivated to formulate the distribution gap on imbalanced OOD detection, and try to regulate the imbalanced detectors towards balanced models using proposed techniques.\n\nWe hope our responses can address the reviewer's concerns, and we are more than happy to provide further explanations if there are additional questions.\n\nBest regards,\n\nAuthors\n\n---\n\n[1] Wang et al. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. ICML'22.\n\n[2] Cui et al. Class-balanced loss based on effective number of samples. CVPR'19.\n\n[3] Kang et al. Decoupling representation and classifier for long-tailed recognition. ICLR'20.\n\n[4] Menon et al. Long-tail learning via logit adjustment. ICLR'21.\n\n[5] Yang et al. Generalized out-of-distribution detection: A survey. IJCV'24.\n\n[6] Du et al. Vos: Learning what you don't know by virtual outlier synthesis. ICLR'22.\n\n[7] Tao et al. Non-parametric outlier synthesis. ICLR'23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable time and constructive suggestions when evaluating our manuscript. We are really encouraged to see **ALL** reviewers find our method **interesting and theoretically grounded** to formulate the gap for imbalanced OOD detection, and **effective and generalized** across datasets and metrics with consistently significant improvements.

We have provided point-to-point responses to reviewers' comments below, and here is a brief summary of the included experiments and explanations:

* **Preliminary investigation/verification on the joint imbalanced OOD detection problem**. We have supplemented the preliminary discussions (originated from PASCL) on the joint problem of imbalanced recognition and OOD detection, so as to enhance the significance and contribution of our research topic.

* **Additional comparison with more general OOD detection approaches**. We have newly provided more comparison and discussion with several SOTA approaches on general OOD detection from recent top conferences, and emphasize the necessity and efficacy of building our imbalanced OOD detection method with theoretical groundness.

* **Further exploration on ClassPrior's setting without auxiliary OOD data**. We have explored the scenario without auxiliary OOD training data after aligning with ClassPrior's experimental settings, so as to better illustrate the effectiveness and applicability of our proposed method.

* **Promising attempts to adapt our methods to models' inference stage**. We have attempted to apply our approach to pre-trained models during the inference stage, and shown promising results and potential applications under proper estimations.

We believe reviewers' comments have made our paper much stronger, and we hope our work can further inspire the community to a deeper study into the imbalanced OOD detection problem.

Meta-review: An imbalanced situation will indeed influence the OOD detection performance, which is an important research topic in this area. This paper thoroughly investigated both theoretical and empirical perspectives of this problem. All reviewers appreciate the main contribution of this paper and agree that this paper can have a valuable contribution to this field. The authors should carefully revise their paper to reduce the confusing portion of the paper, based on reviewers' comments. Another note from my side is that the first phenomenon "misidentifying tail class ID samples as OOD, while erroneously predicting OOD samples as a head class from ID" was empirically discovered by Jiang (2023) as well. The authors should carefully shape their contributions in the next version.
Decision: Accept (poster)