Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 05:22:03

Abstract: Detecting and rejecting unknown out-of-distribution (OOD) samples is critical for deployed neural networks to void unreliable predictions. In real-world scenarios, however, the efficacy of existing OOD detection methods is often impeded by the inherent imbalance of in-distribution (ID) data, which causes significant performance decline. Through statistical observations, we have identified two common challenges faced by different OOD detectors: misidentifying tail class ID samples as OOD, while erroneously predicting OOD samples as head class from ID. To explain this phenomenon, we introduce a generalized statistical framework, termed ImOOD, to formulate the OOD detection problem on imbalanced data distribution. Consequently, the theoretical analysis reveals that there exists a class-aware bias item between balanced and imbalanced OOD detection, which contributes to the performance gap. Building upon this finding, we present a unified training-time regularization technique to mitigate the bias and boost imbalanced OOD detectors across architecture designs. Our theoretically grounded method translates into consistent improvements on the representative CIFAR10-LT, CIFAR100-LT, and ImageNet-LT benchmarks against several state-of-the-art OOD detection ap- proaches. Code is available at https://github.com/alibaba/imood.

Main paper: 
 introduction: 
 Identifying and rejecting unknown samples during models' deployments, aka OOD detection, has garnered significant attention and witnessed promising advancements in recent years (Yang et al., 2021b; Bitterwolf et al., 2022; Ming et al., 2022; Tao et al., 2023a) . Nevertheless, most advanced OOD detection methods are designed and evaluated in ideal settings with category-balanced indistribution (ID) data. However, in practical scenarios, long-tailed class distribution (a typical imbalance problem) not only limits classifiers' capability (Buda et al., 2018) , but also causes a substantial performance decline for OOD detectors (Wang et al., 2022) .Several efforts have been applied to enhance OOD detection on imbalanced data distributions (Wang et al., 2022; Huang et al., 2023; Sapkota & Yu, 2023) . They mainly attribute the performance degradation to misidentifying samples from tail classes as OOD (due to the lack of training data), and concentrate on improving the discriminability for tail classes and out-of-distribution samples (Wang et al., 2022) . Whereas, we argue that the confusion between tail class and OOD samples presents only one aspect of the imbalance problem arising from the long-tailed data distribution.To comprehensively understand the imbalance issue, we employed several representative OOD detection methods (i.e., OE (Hendrycks et al., 2019) , Energy (Liu et al., 2020) , and PASCL (Wang et al., 2022) ) on the CIFAR10-LT dataset (Cao et al., 2019) . For each model, we statistic the distribution of wrongly detected ID samples and wrongly detected OOD samples, respectively. The results in Fig. 1a reveal that different approaches encounter the same two challenges: (1) ID samples from tail classes are prone to be detected as OOD, and (2) OOD samples are prone to be predicted as ID from head classes. As illustrated in Fig. 1b , we argue that the disparate ID decision spaces on head and tail classes jointly result in the performance decline for OOD detection. This paper introduces a generalized statistical framework, namely ImOOD, to formulate imbalanced OOD detection (see Section 3.1). Specifically, we extend closed-set ID classification to open-set scenarios and derive a unified posterior probabilistic model for ID/OOD identification. Consequently, we find that between balanced and imbalanced ID data distributions exists a class-aware bias item, which concurrently explains the inferior OOD detection performance on both head and tail classes. Based on the generalized ImOOD framework, given an imbalanced OOD detector g, we develop a naive post-hoc normalization approach to adaptively calibrate the OOD detection probability via ID classification information (see Section 3.2). To further enhance the OOD detection on imbalanced distribution, we design a unified loss function to regularize the posterior ID/OOD probability during training (see Section 3.3), which simultaneously encourages the separability between tail ID classes and OOD samples, and prevents predicting OOD samples as head ID classes. Furthermore, ImOOD can readily generalize to various OOD detection methods, including OE (Hendrycks et al., 2019) , Energy (Liu et al., 2020) , and BinDisc (Bitterwolf et al., 2022) , etc. With the support of theoretical analysis, our generalized statistical framework consistently translates into strong empirical performance on the CIFAR10-LT, CIFAR100-LT (Cao et al., 2019) , and ImageNet-LT (Wang et al., 2022) benchmarks (see Section 4).Our contribution can be summarized as follows:• Through statistical observation and theoretical analysis, we reveal that OOD detection approaches collectively suffer from the disparate decision spaces between tail and head classes in the imbalanced data distribution. • We establish a generalized statistical framework to formulate and explain the imbalanced OOD detection issue, and further provide a unified view of post-hoc normalization and loss modification techniques to alleviate the problem. • We achieve superior OOD detection performance on three representative benchmarks, e.g., outperforming the state-of-the-art method by 1.7% and 5.2% of AUROC on CIFAR10-LT and ImageNet-LT, respectively. 
preliminaries: 
 Imbalanced Image Recognition. Let X in and Y in = {1, 2, • • • , K} denote the ID feature space and label space with K categories in total. Let x ∈ X in and y ∈ Y in be the random variables with respect to X in and Y in . The posterior probability for predicting sample x into class y is given by:EQUATIONGiven a learned classifier f : X in → R K that estimates P (y|x), in the class-imbalance setting where the label prior P (y) is highly skewed, f is evaluated with the balanced error (BER) (Brodersen et al., 2010; Menon et al., 2013; 2021) :EQUATIONThis can be seen as implicitly estimating a class-balanced posterior probability (Menon et al., 2021) :EQUATIONThe ideal Bayesian-optimal classification becomes y * = argmax y∈[K] P bal (y|x).Out-of-distribution Detection. In the open world, input sample x may also come from the OOD feature space X out . Let o be the random variable for an unknown label o / ∈ Y in , and i be the union variable of all ID class labels (i.e., i = ∪ y). To ensure OOD detection is learnable (Fang et al., 2022) , we make a mild assumption that the ID and OOD data space is separate (i.e., X in ∩X out = ∅) and finite (i.e., |X in |, |X out | < ∞ and P (i), P (o) < 1). Given an input x from the union space X in ∪ X out ≜ X , the posterior probability for identifying x as in-distribution is formulated as:EQUATIONCorrespondingly, P (o|x) measures the probability that sample x does not belong to any known ID class, aka OOD probability. Hence, the OOD detection task can be viewed as a binary classification problem (Bitterwolf et al., 2022) . Given a learned OOD detector g : X in ∪ X out → R 1 that estimate the P (i|x), samples with lower scores g(x) are detected as OOD and vice versa. 
balanced ood detection on imbalanced data distribution: 
 When ID classification meets OOD detection, slightly different from Eq. (1), the classifier f is actually estimating the posterior class probability for sample x from ID space X in merely, that is, P (y|x, i) (Hendrycks et al., 2019; Wang et al., 2022) . Considering a sample x from the open space x ∈ X in ∪ X out , since P (i, y|x) = P (∪y, y|x) = P (y|x), Eq. ( 1) can be re-formulated as:EQUATIONRecall that P (y|x, i) and P (i|x) are estimated by f and g, respectively. Combining Eq. ( 5) with Eq. ( 3), the balanced posterior class probability in the open-set is formulated as:EQUATIONCombining Eq. ( 6) with Eq. ( 4), the ideal balanced ID/OOD detection probability is given by:EQUATIONFor simplicity, Eq. ( 7) is summarized as:EQUATIONCompared to the original OOD probability P (i|x), the bias item y P (y|x,i) P (y)≜ β(x) causes the performance gap for OOD detection on balanced and imbalanced data distribution.On class-balanced distribution, the class-priority P (y) is a constant that equals to each other of the ID categories, i.e., P (y) = const. Since the summary of in-distribution classification probabilities equals to 1 (i.e., y P (y|x, i) = 1), the bias item β(x) = y P (y|x,i)const = 1 const y P (y|x, i) = 1const is also a constant. Thus, according to Eq. ( 8), P bal (i|x) ∝ P (i|x), as g(x) exactly models the balanced OOD detection.On class-imbalanced distribution, the class-priority P (y) is a class-specific value for ID categories. Since y P (y|x, i) = 1, the bias item β(x) = y P (y|x,i) P (y)can be viewed as a weighted sum of the reciprocal prior 1 P (y) . According to Eq. ( 8), β(x) causes the gap between balanced (ideal) ID/OOD probability P bal (i|x) and imbalanced (learned) P (i|x), as P bal (i|x) ∝ P (i|x) • β(x):• Given a sample x from an ID tail-class y t with a small prior P (y t ), when the classification probability P (y t |x, i) gets higher, the term β(x) becomes larger. Compared to the original P (i|x) (learned by g), the calibrated P (i|x) • β(x) is more likely to identify the sample x as in-distribution, rather than OOD. • Given a sample x ′ from OOD data, as the classifier f tends to produce a higher head-class probability P (y h |x ′ , i) and a lower tail-class P (y t |x ′ , i) (Jiang et al., 2023) , the term β(x ′ ) becomes smaller. Compared to the original P (i|x ′ ), the calibrated P (i|x ′ ) • β(x ′ ) is more likely to identify the sample x ′ as out-of-distribution, rather than ID.The above analysis is consistent with the statistical behaviors (see Fig. 1 ) of a vanilla OOD detector g. Compared to an ideal balanced detector g bal , g is prone to wrongly detect ID samples from tail class as OOD, and wrongly detect OOD samples as head class from ID.In order to mitigate the imbalanced OOD detection, based on Eq. ( 8), this paper provides a unified view of post-hot normalization and training-time regularization techniques in the following sections.Given a vanilla OOD detector g learned on imbalanced data distribution, Eq. ( 8) suggests one may use the posterior classification probability P (y|x, i) and prior P (y) to adaptively calibrate P (i|x) to get close to the balanced P bal (i|x). We now show how to perform the naive post-hoc normalization from a statistical perspective.(1) P (i|x). Since OOD detection is a binary classification task (Bitterwolf et al., 2022) , for an arbitrary OOD detector g (Hendrycks et al., 2019; Liu et al., 2020) , the posterior probability can be ultimately estimated by a sigmoid function P (i|x) := e g(x) 1+e g(x) , where g(x) is the ID/OOD logit.(2) P (y|x, i). Given a learned classifier f , the classification probability can be estimated by a softmax function P (y|x, i) := e fy (x)y ′ e f y ′ (x)≜ p y|x,i , where f y (x) presents the logit for class y.(3) P (y). An empirical estimation of the prior P (y) is using the label frequency of the training dataset (Cao et al., 2019; Menon et al., 2021) . In the open-set, it becomes P (y) := ny no+ y ′ n y ′ ≜ πy , where n o is the number of auxiliary OOD training samples, and n y indicates the number of class y.For numerical stability (see Appendix A.3), we take the logarithm of both sides of Eq. ( 8), and the balanced OOD detector g bal is derived as:EQUATIONBy adding the distribution-dependent offset log y p y|x,i πy , g bal is able to bridge the gap between balanced and imbalanced data distributions. As analyzed in Section 3.1, g bal produces a higher ID probability for the sample x from ID tail classes, and generates a lower ID probability (i.e., a higher OOD probability) for an OOD sample x ′ predicted as ID head classes.To directly strengthen the OOD detection capability during training, we introduce the way to regularize the loss function in the following Section 3.3.According to Eq. ( 8), a more effective approach to balanced OOD detection is directly modeling the balanced posterior probability P bal (i|x), rather than the vanilla P (i|x). To do so, consider the following regularized ID/OOD binary cross-entropy loss function:EQUATIONwhere t = 1{x ∈ X in } indicates whether the sample x comes from ID or not. Given a detector g * that minimizes the above Eq. ( 10), it has already estimated the balanced P bal (i|x). We thus predict the ID/OOD probability as usual: p(i|x) = 1 1+e -g * (x) . Note that in Eq. ( 10), we did not use the empirical statistics πy = ny no+ y ′ n y ′ to regularize detector g. The reasons are two-fold: (1) πy is sensitive to n o , which means the choice of auxiliary OOD training set will change the prior estimates, while collecting an optimal OOD training set is generally prohibitive; (2) the empirical frequencies of πy are not precise estimations for prior probabilities of P (y) (Cao et al., 2019; Menon et al., 2021) . Therefore, we estimate P (y) by a set of learnable and input-agnostic parameters π y (θ) ≜ π y in Eq. ( 10) via automatic gradient back-propagation during training. This idea is motivated by the recent prompt tuning frameworks (Lester et al., 2021; Jia et al., 2022; Zhou et al., 2022) , which also aim to learn some prior properties from the joint distribution of X × Y. The learnable π y is initialized with the statistical πy to ensure stable convergence.The ultimate framework, termed ImOOD, is verified through extensive experiments in Section 4. 
experiments: 
 In this section, we empirically validate the effectiveness of our ImOOD on several representative imbalanced OOD detection benchmarks. The experimental setup is described in Section 4.1, based on which extensive experiments and discussions are displayed in Section 4.2 and Section 4.3.Datasets. Following the literature (Wang et al., 2022; Jiang et al., 2023; Choi et al., 2023) , we use the popular CIFAR10-LT, CIFAR100-LT (Cao et al., 2019) , and ImageNet-LT (Liu et al., 2019) as imbalanced in-distribution datasets.For CIFAR10/100-LT benchmarks, training data's imbalance ratio (i.e., ρ = max y (n y )/ min y (n y )) is set as 100, following Cao et al. (2019) ; Wang et al. (2022) . The original CIAFR10/100 test sets are kept for evaluating the ID classification capability. For OOD detection, the TinyImages80M (Torralba et al., 2008 ) is adopted as the auxiliary OOD training data, and the test set is semantically coherent out-of-distribution (SC-OOD) benchmark (Yang et al., 2021a) .For the large-scale ImageNet-LT benchmark, we follow Liu et al. (2019) ; Wang et al. (2022) to select training samples from the whole ImageNet-1k (Deng et al., 2009) dataset, and take the original validation set for evaluation. We take the same OOD detection setting as Wang et al. (2022) , where the ImageNet-Extra is used as auxiliary OOD training data and ImageNet-1k-OOD for testing. Randomly sampled from ImageNet-22k (Deng et al., 2009) , ImageNet-Extra contains 517,711 images belonging to 500 classes, and ImageNet-1k-OOD consists of 50,000 images from 1,000 classes. All the classes in ImageNet-LT, ImageNet-Extra, and ImageNet-1k-OOD are not overlapped.Evaluation Metrics. For OOD detection, we follow Hendrycks et al. (2019) ; Yang et al. (2021a) ; Wang et al. (2022) to report three metrics: (1) AUROC, the area under the receiver operating characteristic curve, (2) AUPR, the area under precision-recall curve., and (3) FPR95, the false positive rate of OOD samples when the true positive rate of ID samples are 95%. For ID classification, we measure the macro accuracy of the classifier, as in Eq. ( 2). We report the mean and standard deviation of performance (%) over six random runs for each method.Implementation Details. For the ID classifier f , following the settings of Wang et al. (2022) , we train ResNet18 (He et al., 2016) models on the CIFAR10/100LT benchmarks, and leverage ResNet50 models for the ImageNet-LT benchmark. Logit adjustment loss (Menon et al., 2021 ) is adopted to alleviate the imbalanced ID classification. Detailed settings are displayed in Appendix A.1. For the OOD detector g, as Bitterwolf et al. (2022) suggest, we implement g as a binary discriminator (abbreviated as BinDisc) to perform ID/OOD identification with Eq. ( 9) and Eq. ( 10). Detector g shares the same backbone (feature extractor) as classifier f , and g only attaches an additional output node to the classification layer of f . To verify the versatility of our method, we also implement several representative OOD detection methods (e.g., OE (Hendrycks et al., 2019) , Energy (Liu et al., 2020) , etc.) into binary discriminators, and equip them with our ImOOD framework. For more details please refer to Appendix A.2.Methods for comparison. In the following sections, we mainly compare our method with the recent OOD detectors including OE (Hendrycks et al., 2019) , Energy (Liu et al., 2020) , BinDisc (Bitterwolf et al., 2022) , etc. Specifically, we conduct comprehensive comparisons with the advanced baseline PASCL (Wang et al., 2022) , and also report the results of standard MSP (Hendrycks & Gimpel, 2017) and the modern variant PR+MSP (Jiang et al., 2023) scores on pre-trained models. In addition, Choi et al. (2023) have recently achieved superior performance on CIFAR10/100-LT benchmarks, whereas both of the training and testing procedures are different from PASCL's and our settings. We thus separately take their settings to assess our method in Appendix B.3 for a fair comparison. ImOOD consistently boosts existing OOD detectors via post-hoc normalization. To validate the efficacy of our generalized statistical framework, we first train several standard OOD detectors on the CIFAR10-LT (Cao et al., 2019) benchmark using popular methods like OE (Hendrycks et al., 2019) , Energy (Liu et al., 2020) , and BinDisc (Bitterwolf et al., 2022) . Then we implement our ImOOD as a post-hoc normalization via Eq. ( 9) to adaptively adjust those OOD detectors' predictions with imbalanced classification information. The results are displayed in Table 1 , which suggests our ImOOD brings consistent performance improvements for existing OOD detectors (e.g., 0.3% increase on AUROC). Specifically, our ImOOD decreases 1.3% of FPR95 for Energy and 2.3% for BinDisc, and the post-hoc normalization does not affect ID classifications.Though Table 1 indicates our ImOOD is effective in naively• normalizing the imbalanced OOD detectors, the final performance (e.g., 90% of AUROC and 31% of FPR95) is still far from satisfaction for safe deployments. Hence, we employ our ImOOD as the training-time regularization via Eq. ( 10) to directly learn a balanced BinDisc detector on CIFAR10/100-LT and ImageNet-LT benchmarks (Wang et al., 2022) , and the results are illustrated as follows.ImOOD significantly outperforms previous SOTA methods on CIFAR10/100-LT benchmarks. As shown in Table 2 , our ImOOD achieves new SOTA performance on both of CIFAR10/100-LT benchmarks, and surpasses the advanced PASCL (Wang et al., 2022 ) by a large margin. Specifically, ImOOD leads to 1.7% increase of AUROC, 2.1% increase of AUPR, and 5.1% decrease of FPR95 on CIFAR10-LT, with 0.8% -1.7% enhancements of respective evaluation metrics on CIFAR100-LT. The detailed comparison with PASCL on six OOD test sets is reported in Appendix B.2. The results suggest that training-time regularization is more effective in alleviating the imbalanced OOD detection problem. To further demonstrate the efficacy, we validate our method on the realworld large-scale ImageNet-LT (Liu et al., 2019) benchmark, and the results are displayed below.ImOOD achieves superior performance on the ImageNet-LT benchmark. As Table 3 implies, our ImOOD brings significant improvements against PASCL, e.g., 5.2% increase on AUROC and 12.7% decrease on FPR95. Since the performance enhancement is much greater than those on CIFAR10/100-LT benchmarks, we further statistic the class-aware error distribution on wrongly detected ID/OOD sample in Fig. A1 . The results indicate our method builds a relatively better-balanced OOD detector on ImageNet-LT, which leads to higher performance improvements. Besides, as we employ ResNet18 on CIFAR10/100-LT while adopt ResNet50 on ImageNet-LT, the model capacity also seems to play a vital role in balancing the OOD detection on imbalanced data distribution, particularly in more challenging real-world scenarios. In addition, our ImOOD brings better ID classification accuracy on all three imbalanced benchmarks. We owe it to the relatively disentangled optimization of ID classifier f and OOD detector g, which is incidentally discussed in the last part of Section 4.3.In this section, we conduct in-depth ablation studies on the CIFAR10-LT benchmark to assess the validity and versatility of our proposed ImOOD framework, and the results are reported as follows.Automatically learning the class-prior P (y) ≜ π y is effective. To validate our training-time regularization in Eq. ( 10), we first build a baseline model with BinDisc (Bitterwolf et al., 2022) only, and no extra regularization is adopted to mitigate the imbalanced OOD detection. As shown in the first row from Table 4 , the baseline (denoted as none of P (y) estimates) presents a fair OOD detection performance (e.g., 90.06% of AUROC and 33.39% of FPR95). Then, we leverage the statistical estimates of label frequency (denoted as π y = πy ) to perform the loss regularization, and the performance receives an immediate 1.7%-2.3% improvement of all the AUROC, AUPR, FPR95 measures. Furthermore, automatically learning the class-prior (denoted as π y = π y (θ)) leads to a consecutively better OOD performance (e.g., 92.19% of AUROC and 29.29% of FPR95). The learned class-prior is visualized in Fig. 2 , which indicates automatic prior estimation from existing data distribution is practical and effective. ImOOD generalizes to various OOD detection methods. To verify the versatility of our statistical framework, we implement the ImOOD framework with different OOD detectors, including OE (Hendrycks et al., 2019) , Energy (Liu et al., 2020) , and BinDisc (Bitterwolf et al., 2022) . For OE and Energy detectors, we append an extra linear layer to conduct logistic regression on their vanilla OOD scores (e.g., the maximum softmax probability for OE) (see Appendix A.2), which are thus universally formulated as the detector g(x) in Eq. ( 10). Then, our training-time regularization is leveraged to optimize those detectors (denoted as +ImOOD). According to the results presented in Table 5 , our ImOOD consistently boosts the original OOD detectors with stronger performance on all the AUROC, AUPR, and FPR95 measures. In particular, after utilizing the feature-level contrastive learning technique from PASCL (Wang et al., 2022) (denoted as +ImOOD+PASCL), our method achieves a superior OOD detection performance (e.g., 92.73% of AUROC and 28.27% of FPR95). It implies our ImOOD framework does not conflict with current feature-level optimization approaches, and the combination of them may lead to better results. In Fig. A2 , we also statistic the class distribution on wrongly-detected ID/OOD samples, which suggests our method continually mitigates the imbalanced problem for those original detectors. In real-world applications, one may choose a proper formulation of our ImOOD to meet the specialized needs. In addition, as OE adopts the softmax output from ID classifier f to perform OOD scorer g, the deeply coupled optimization on f and g may sacrifice the ID classification accuracy (Wang et al., 2022; Choi et al., 2023) . Meanwhile, Energy utilizes the logits (before softmax function) to perform OOD detection, whose ID accuracy obtains an increase (from 75.75% to 77.92%). As for BinDisc that only shares the features for f and g, the relatively disentangled optimization leads to a higher classification accuracy (78.22%). 
related works: 
 Out-of-distribution detection. To reduce the overconfidence on unseen OOD samples (Bendale & Boult, 2015) , a surge of post-hoc scoring functions has been devised based on various information, including output confidence (Hendrycks & Gimpel, 2017; Liang et al., 2018; Liu et al., 2023) , free energy (Liu et al., 2020; Du et al., 2022; Lafon et al., 2023) , Bayesian inference (Maddox et al., 2019; Cao & Zhang, 2022) , gradient information (Huang et al., 2021) , model/data sparsity (Sun et al., 2021; Zhu et al., 2022; Djurisic et al., 2023; Ahn et al., 2023) , and visual distance (Sun et al., 2022; Tao et al., 2023b) , etc. And the vision-language models like CLIP (Radford et al., 2021) have been recently leveraged to explicitly collect potential OOD labels (Fort et al., 2021; Esmaeilpour et al., 2022) or conduct zero-shot OOD detections (Ming et al., 2022) . Another promising approach is adding open-set regularization in the training time (Malinin & Gales, 2018; Hendrycks et al., 2019; Wei et al., 2022; Wang et al., 2023b; Lu et al., 2023) , making models produce lower confidence or higher energy on OOD data. Manually-collected (Hendrycks et al., 2019; Wang et al., 2023a) or synthesized outliers (Du et al., 2022; Tao et al., 2023b) are required for auxiliary constraints.Works insofar have mostly focused on the ideal setting with balanced data distribution for optimization and evaluation. This paper aims at OOD detection on practically imbalanced data distribution.OOD detection on imbalanced data distribution. In real-world scenarios, the deployed data frequently exhibits long-tailed distribution, and Liu et al. (2019) start to study the open-set classification on class-imbalanced setup. Wang et al. (2022) systematically investigate the performance degradation for OOD detection on imbalanced data distribution, and develop a partial and asymmetric contrastive learning (PASCL) technique to tackle this problem. Consequently, Jiang et al. (2023) propose to adopt class prior as a post-hoc normalization for pre-trained models, and Sapkota & Yu (2023) employ adaptive distributively robust optimization (DRO) to quantify the sample uncertainty from imbalanced distributions. Choi et al. (2023) focus on the imbalance problem in OOD data, and develop an adaptive regularization item for each OOD sample during networks' optimization.Different from previous efforts, this paper establishes a generalized probabilistic framework to formularize the imbalanced OOD detection issue, and provides a unified view of post-hoc normalization and training-time regularization techniques to alleviate this problem. 
conclusion: 
 This paper establishes a statistical framework, termed ImOOD, to formularize OOD detection on imbalanced data distribution. Through theoretical analysis, we find there exists a class-aware biased item between balanced and imbalanced OOD detection models. Based on it, our ImOOD provides a unified view of post-hoc normalization and training-time regularization techniques to alleviate the imbalance problem. On three popular imbalanced OOD detection benchmarks, extensive experiments and ablation studies are conducted to demonstrate the validity and versatility of our method. We hope our work can inspire new research in this community.Limitations and future work. To ensure satisfactory OOD detection performance, ImOOD requires full-data learning, where auxiliary OOD training samples are needed. It has two major limitations:(1) the training cost is inevitable when the deployed environment changes, and fast adaptation across scenarios should be explored. (2) the manually collected auxiliary OOD data is relatively expensive, and recent outlier synthesis techniques may reduce the burden. We view those as our future work.Reproducibility statement. This work follows the settings and code as PASCL (Wang et al., 2022) to implement our method. For reproducibility, necessary information has been provided in Section 4.1 and Appendix A in detail, involving adopted datasets, training and evaluation settings, implementation details, etc. 
 
 Appendix: A.1 TRAINING SETTINGS For a fair comparison, we mainly follow PASCL's (Wang et al., 2022) settings. On CIFAR10/100-LT benchmarks, we train ResNet18 (He et al., 2016) for 200 epochs using Adam optimizer, with a batch size of 256. The initial learning rate is 0.001, which is decayed to 0 using a consine annealing scheduler. The weight decay is 5 × 10 -4 . On ImageNet-LT benchmark, we train ResNet50 (He et al., 2016) for 100 epochs with SGD optimizer with the momentum of 0.9. The batch size is 256. The initial learning rate is 0.1, which are decayed by a factor of 10 at epoch 60 and 80. The weight decay is 5 × 10 -5 . During training, each batch contains an equal number of ID and OOD data samples (i.e., 256 ID samples and 256 OOD samples).For a better performance, one may carefully tune the hyper-parameters to train the models.In the manuscript, we proposed a generalized statistical framework to formularize and alleviate the imbalanced OOD detection problem. This section provides more details on implementing different OOD detectors into a unified formulation, e.g., a binary ID/OOD classifier Bitterwolf et al. (2022) :• For BinDisc (Bitterwolf et al., 2022) , we simply append an extra ID/OOD output node to the classifier layer of a standard ResNet model, where ID classifier f and OOD detector g share the same feature extractor. Then we adopt the sigmoid function to convert the logit g(x) into the ID/OOD probability p(i|x) = 1 1+e -g(x) . • For Energy (Liu et al., 2020) , following Du et al. (2022) , we first compute the negative free-energy E(x; f ) = log y e fy (x) , and then attach an extra linear layer to calculate the ID/OOD logit g(x) = w •E(x; f )+b, where w, b are learnable scalars. Hence, the sigmoid function is able to convert the logit g(x) into the probability p(i|x). • For OE (Hendrycks et al., 2019) , similarly, we compute the maximum softmax-probability MSP(x; f ) = max y e fy (x)y ′ e f y ′ (x) , and use another linear layer to obtain the ID/OOD logit g(x) = w • MSP(x; f ) + b.By doing so, one may leverage Eq. ( 9) as the post-hoc normalization for a given model, or exploit Eq. (10) as the training-time regularization to derive a strong OOD detector.In Section 3.1, we model the balanced ID/OOD probability as P bal (i|x) ∝ P (i|x)• y P (y|x,i) P (y) , and respectively use the softmax-output p y|x,i and label-frequency πy to estimate P (y|x, i) and P (y). An intuitive quantification of P bal (i|x) is P bal (i|x) ∝ P (i|x) • y p y|x,i πy ≜ P (i|x) • β(x), which however may cause numerical instability. Taking CIFAR10-LT (Cao et al., 2019) with imbalance ratio ρ = 100 as an example, the reciprocal priority for tail-class y t becomes 1 πy t ≈ 248. Even if the classifier f produces a relatively lower posteriority (e.g., p yt|x,i = 0.1), the bias item becomes β(x) ≜ y p y|x,i πy > 0.1 × 248 = 24.8, which may overwhelm P (i|x) := e g(x) 1+e g(x) ∈ (0, 1). As Table A1 shows, compared to P (i|x), the regularized P (i|x)• β(x) even leads to worse OOD detection performance (e.g., 33.6% v.s. 35.7% of FPR95) . Hence, we propose to turn the probability-level regularization into logit-level, which is derived as g bal (x) := g(x) + log β(x). The term log β(x) plays the same role in calibrating the imbalanced g(x) to the balanced g bal (x), as discussed in Section 3.1. One may also take the sigmoid function to get the probability P bal (i|x), whose numerical stability is validated in Table A1 as well. We thus develop the logit-level post-hoc normalization and training-time regularization techniques in Section 3.2 and Section 3.3, respectively. In this section, we present the class-aware error statistics for OOD detection on different benchmarks (see Fig. A1 ) and different detectors (see Fig. A2 ). For each OOD detector on each benchmark, we first compute the OOD scores g(x) for all the ID and OOD test data. Then, a threshold λ is determined to ensure a high fraction of OOD data (i.e., 95%) is correctly detected as out-of-distribution. Recall that g(x) indicates the in-distribution probability for a given sample (i.e., P (i|x)). Finally, we statistic the distributions of wrongly detected ID/OOD samples. Specifically, in Fig. A1 and Fig. A2 , the first row displays class labels of ID samples that are wrongly detected as OOD (i.e., g(x) < λ), and the second row exhibits class predictions of OOD samples that are wrongly detected as ID (i.e., g(x) > λ). In each subplot, we statistic the distribution over head, middle, and tail classes (the division rule follows Wang et al. ( 2022)) for simplicity. Note that the total count of wrong OOD samples (in the second row) is constant, and a better OOD detector g will receive fewer wrong ID samples (in the first row). Fig. A1 compares our method with PASCL (Wang et al., 2022) on CIFAR10/100-LT and ImageNet-LT benchmarks. The results indicate our method (dashed bar) performs relatively more balanced OOD detection on all benchmarks. We reduce the error number of ID samples from tail classes, and simultaneously decrease the error number of OOD samples that are predicted as head classes. In particular, our method achieves considerable balanced OOD detection on ImageNet-LT (the right column), which brings a significant performance improvement, as discussed in Section 4.2. Fig. A2 compares our integrated version and vanilla OOD detectors (i.e., OE, Energy, and BinDisc) on the CIFAR10-LT benchmark. Similarly, the results indicate our method performs relatively more balanced OOD detection against all original detectors. The versatility of our ImOOD framework is validated, as discussed in Section 4.3. However, the statistics in Fig. A1 and Fig. A2 indicate the imbalanced problem has not been fully solved, since the scarce training samples of tail classes still affect the data-driven learning process. More data-level re-balancing techniques may be leveraged to further address this issue. For CIFAR10/100-LT benchmark, Textures (Cimpoi et al., 2014) , SVHN (Netzer et al., 2011) , CIFAR100/10 (respectively), TinyImageNet (Le & Yang, 2015) , LSUN (Yu et al., 2015) , and Places365 (Zhou et al., 2017) from SC-OOD dataset Yang et al. (2021a) are adopted as OOD test sets. The mean results on the six OOD sets are reported in Section 4.In this section, we reported the detailed measures on those OOD test sets in Table A2 and Table A3 , as the supplementary to Table 2 . The results indicate our method consistently outperforms the stateof-the-art PASCL (Wang et al., 2022) on most of the subsets. BalEnergy (Choi et al., 2023) has recently achieved advance on CIFAR10/100LT benchmarks by adding an instance-specific loss regularization for each OOD training sample. However, its training and testing settings both differ from the PASCL (Wang et al., 2022) baseline. For example, PASCL trains all the models from scratch, while BalEnergy fine-tunes the OOD detector from a pre-trained classifier. During ID/OOD testing, PASCL treats ID samples as positive while BalEnergy treats OOD samples as positive, which causes a slight difference on the FPR95 measure (Tao et al., 2023b) .Thus, we separately evaluate our method on BalEnergy's settings for a fair comparison. The results in Table A4 demonstrate that our ImOOD can still effectively boost the OOD detection performance.As we discussed in Section 4.3, the combination with ImOOD and advanced techniques may bring better OOD detectors.  


Decision: An imbalanced situation will indeed influence the OOD detection performance, which is an important research topic in this area. This paper thoroughly investigated both theoretical and empirical perspectives of this problem. All reviewers appreciate the main contribution of this paper and agree that this paper can have a valuable contribution to this field. The authors should carefully revise their paper to reduce the confusing portion of the paper, based on reviewers' comments. Another note from my side is that the first phenomenon "misidentifying tail class ID samples as OOD, while erroneously predicting OOD samples as a head class from ID" was empirically discovered by Jiang (2023) as well. The authors should carefully shape their contributions in the next version.

Reviews: review: paper_summary: paper_summary: The paper focuses on imbalanced data distribution, and finds that there is a bias term between balanced and imbalanced classification which can be used to explain the performance gap. To account for this bias, the authors introduce a regularization term, and show improved results on benchmarks.
strength_weakness: strength_weakness: - Train-time regularization techniques cannot be applied to existing pretrained models, which may limit their adoption.
- The method requires OOD data during training, and results may not generalize to other unseen OOD settings.
questions: questions: - Have you tested the benefits of your method with varying levels of imbalance? How does the performance change?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper addresses the challenge of detecting and rejecting out-of-distribution (OOD) samples by neural networks, particularly when the in-distribution (ID) data is inherently imbalanced. The authors observe that existing OOD detection methods struggle under these conditions primarily because they either misclassify ID samples from minority (tail) classes as OOD or mistakenly identify OOD samples as belonging to majority (head) classes.

To tackle this issue, the authors introduce a new statistical framework called ImOOD, which is designed to understand and improve OOD detection in the presence of imbalanced data. The framework considers the class-aware biases that arise due to data imbalance and incorporates a unified training-time regularization technique aimed at mitigating these biases.
strength_weakness: strength_weakness: A minor issue is that, when we know in advance that the data are long-tailed, it is a common practice that we will use the long-tailed learning methods to counteract its impacts. In this case, is the study of long tailed OOD detection an important topic? Or, even with long-tailed learning, the OOD detection therein still be a critical issue?

I am not sure if the prior information can be properly estimated, considering the situations that the adopted models are not well calibrated (otherwise, OOD detection will not be a challenging task).
questions: questions: Please refer to the Weaknesses
limitations: limitations: Please refer to the Weaknesses
paper_summary: paper_summary: This manuscript introduces ImOOD, a statistical framework addressing the OOD detection problem in imbalanced data distributions, identifying common issues such as misidentifying tail class ID samples and erroneously predicting OOD samples as head class ID. It reveals a class-aware bias between balanced and imbalanced OOD detection and proposes a unified training-time regularization technique to mitigate this bias. The method demonstrates consistent performance improvements on benchmarks such as CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, surpassing several state-of-the-art OOD detection methods.
strength_weakness: strength_weakness: Although the manuscript provides insights through theoretical analysis and proposes a method to improve out-of-distribution (OOD) detection in class-imbalanced scenarios, it essentially estimates a class-specific scaling factor, which is a core aspect of long-tailed recognition and class-imbalanced learning. This is neither a new nor an interesting problem. Additionally, the proposed parametric mapping increases the difficulty of network optimization. It seems that OOD detection under class imbalance is merely an overlap of the two tasks: class-imbalanced learning and OOD detection. Using techniques from both fields can address this issue quite well. Therefore, the significance of the problem addressed in the manuscript is questionable and warrants further consideration.
questions: questions: Why not obtain the class-specific scaling factor using methods from the long-tailed image recognition domain? Additionally, there is a lack of sufficient experimental evidence to show that using parametric mapping to obtain the class-specific scaling factor is more effective. The paper compares relatively few OOD detection methods; it is recommended to include more recent OOD detection approaches. The manuscript also lacks a summary of related works in class-imbalanced learning.
limitations: limitations: Please see Weaknesses and Questions.
paper_summary: paper_summary: The paper proposes the ImOOD framework to address class imbalance issues in OOD detection. Through Bayesian analysis, the authors identify critical biases and provide a unified regularization technique to improve detection performance. They conduct extensive experiments, demonstrating ImOOD's effectiveness on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets.
strength_weakness: strength_weakness: - The method relies on external OOD data, which is often difficult to obtain in OOD detection settings.
- The comparisons in CIFAR and ImageNet experiments seem inconsistent, with fewer methods evaluated on ImageNet. Notably, ClassPrior seems to perform better on ImageNet.
- Some non-long-tail-specific OOD detection methods in the ClassPrior paper also perform well. Given the emergence of many excellent OOD detection methods since ClassPrior's publication, comparing and discussing these methods with the proposed approach would provide a more comprehensive evaluation.
questions: questions: - Why do the ID labels in Figure 1a show similar fluctuations across OE, Energy, and PASCL?
- I believe the OOD prediction results in Figure 1a do not support the authors' claim. If I am mistaken, please correct me. The distribution of OOD predictions appears consistent with the prior distribution, with no significant deviation in the head region. For OOD samples, since the model hasn't been trained on them, it would predict according to the training sample distribution (prior distribution). I don't see this behavior as problematic.
- Other questions: Please refer to the weaknesses section above.
limitations: limitations: Yes.

Comments: title: Thanks for Your Timely Feedback
Thanks for Reviewer's Positive Feedback!
Kind Remind of the Ending of Discussion Period
Kind Remind of the Ending of Discussion Period
Kind Remind of the Ending of Discussion Period
Thanks for your timely feedback
The discussion has begun
comment: Dear Reviewer XZSm,

We sincerely thank you for your timely feedback and for adjusting your rating.   Here are our further explanations for your new questions.

Firstly, we are grateful to clarify that the statistics in our original paper are consistent with our motivation.   As for the OOD prediction distribution in Fig 1(b) in our rebuttal PDF file, it describes the _per-class prediction quantity_ of correctly identified OOD samples. In fact, when an OOD sample is identified as OOD data, its maximum ID _prediction probability_ is relatively low (i.e., $max_y{P(y|x,i)} = 0.12$ for 10-category classification), and the quantity statistics on class prediction may thus be biased and unable to capture the true picture.  When we statistic the _per-class prediction probability_, all of the distributions for OE, Energy, and PASCL are nearly even:

|        | class1 | class2 | class3 | class4 | class5 | class6 | class7 | class8 | class9 | class10 |
|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:-------:|
|   OE   |  0.12  |  0.11  |  0.14  |  0.14  |  0.13  |  0.11  |  0.16  |  0.11  |  0.11  |   0.12  |
| Energy |  0.28  |  0.25  |  0.31  |  0.26  |  0.29  |  0.29  |  0.39  |  0.32  |  0.36  |   0.33  |
|  PASCL |  0.15  |  0.12  |  0.12  |  0.14  |  0.11  |  0.12  |  0.13  |  0.12  |  0.11  |   0.11  |

Therefore, we claim _both the correctly-detected ID and OOD samples are insensitive to the class distribution_.   We will clarify this in our revised manuscript.

Secondly, regarding the contribution of our ImOOD paper, we believe our theoretically grounded analysis framework and consistent improvements across datasets and metrics can support our novelty and bring insights to the community.

Finally, for the curves in the figures, we also statistic the distribution on the ImageNet benchmark with 1,000 categories, which presents similar patterns in the previous analysis.   As the rebuttal PDF cannot be updated during this period, we will include it in our final version.

We thank you once again for evaluating our paper, and we hope our responses can help address your new concerns.   If there are any questions left, we are glad to provide more explanations.

---

Best regards,

Authors
Thank you for your response. (W1) While there are many cases in the literature that rely on OOD data, OOD detection, by definition, assumes the OOD distribution is unknown, so it's important not to make assumptions about it. However, this is not a critical weakness. The responses to w2 and w3 were helpful, thank you. 

In the rebuttal PDF, the OOD prediction distribution in Figure 1(b) also appears odd. In both OE and PASCAL, there is a large distribution centered on a single class, whereas in Energy, the distribution seems more aligned with the class priors. The authors mentioned in their comments that "both the correctly-detected ID and OOD samples are insensitive to the class distribution," but this doesn't align with my observations. I see two potential issues: First, the OOD prediction distribution in OE and PASCAL might suggest that the OOD sample distribution itself is uneven. Second, the results in Energy do not seem consistent with the authors' conclusions.

I have also read all the reviews and your rebuttal. My main concern is about the contribution that IMOOD brings to the community, as the improvements in the experimental results are not particularly impressive. Additionally, I still have some questions about the visualizations. Specifically, the curves in the figures seem unusually erratic and not very smooth, which I find puzzling. I raised this concern in my initial review, but the authors did not address it directly. I suspect this might be related to the limited number of classes.

Overall, thank you for your responses. I will adjust my rating to 5, as the concerns outlined above prevent me from giving a higher rating at this time.
Dear Reviewer QTK3,

Thank you for your positive feedback and for raising your score! We are glad that our responses addressed your concerns and will ensure to include them in the final version.

---

Best regards,

Authors
Thank you authors' response. I will increase the score.
Dear Reviewer XZSm,

We thank you again for your time and efforts in evaluating our paper. As the discussion period will **end in 24 hours**, we are eager to receive your feedback on our responses, which is very important to us. If there are any further questions, we are more than happy to provide more clarifications as needed.

---

Best regards,

Authors
Dear Reviewer QTK3,

We thank you again for your time and efforts in evaluating our paper. As the discussion period will **end in 24 hours**, we are eager to receive your feedback on our responses, which is very important to us. If there are any further questions, we are more than happy to provide more clarifications as needed.

---

Best regards,

Authors
Dear Reviewer E388,

We thank you again for your time and efforts in evaluating our paper. As the discussion period will **end in 24 hours**, we are eager to receive your feedback on our responses, which is very important to us. If there are any further questions, we are more than happy to provide more clarifications as needed.

---

Best regards,

Authors
Dear Reviewer 5Lex,

We are sincerely glad that our responses have successfully addressed your concerns. Thanks again for your valuable time in evaluating our work.

Best regards,

Authors
Dear Reviewers,

Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended.

Thanks for your continued efforts and contributions to NeurIPS 2024.

Best regards,

Your Area Chair
Dear Reviewers,

Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended.

Thanks for your continued efforts and contributions to NeurIPS 2024.

Best regards,

Your Area Chair
Thank you for the clarifications and additional experimental results! My questions have been addressed.
Dear Reviewers,

Thank you for taking the time to review the paper. The discussion has begun, and active participation is highly appreciated and recommended.

Thanks for your continued efforts and contributions to NeurIPS 2024.

Best regards,

Your Area Chair