On the Importance of In-distribution Class Prior for Out-of-distribution Detection
Authors: Xue Jiang
Feng Liu
Zhen Fang
Hong Chen
Tongliang Liu
Feng Zheng
Bo Han
Venue: Submitted to ICLR 2023 (2023)

Abstract: Given a pre-trained in-distribution (ID) model, the task of inference-time out-of-distribution (OOD) detection methods aims to recognize upcoming OOD data in inference time. However, some representative methods share an unproven assumption that the probability that OOD data belong to every ID class should be the same, i.e., probabilities that OOD data belong to ID classes form a uniform distribution. In this paper, we theoretically and empirically show that this assumption makes these methods incapable of recognizing OOD data when the ID model is trained with class-imbalanced data. Fortunately, by analyzing the causal relations between ID/OOD classes and features, we identify several common scenarios where probabilities that OOD data belong to ID classes should be the ID-class-prior distribution. Based on the above finding, we propose two effective strategies to modify previous inference-time OOD detection methods:  1) if they explicitly use the uniform distribution, we can replace the uniform distribution with the ID-class-prior distribution; 2) otherwise, we can reweight their scores according to the similarity between the ID-class-prior distribution and the softmax outputs of the pre-trained model. Extensive experiments show that both strategies significantly improve the accuracy of recognizing OOD data when the ID model is pre-trained with imbalanced data. As a highlight, when evaluating on the iNaturalist dataset, our method can achieve ~36% increase on AUROC and ~61% decrease on FPR95, compared with the original Energy method, reflecting the importance of ID-class prior in the OOD detection, which lights up a new road to study this problem.

Under review as a conference paper at ICLR 2023
ON THE IMPORTANCE OF IN-DISTRIBUTION CLASS PRIOR FOR OF -FT-SF-DIISTIIUIION DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Given pre-trained in-distribution (ID) model, the task of inference-time out-of- distribution (OOD) detection methods aims to recognize upcoming OOD data in inference time. However, some representative methods share an unproven assump- tion that the probability that OOD data belong to every ID class should be the same, i.e.. probabilities that OOD data belong to ID classes form a uniform distribution. In this paper, we theoretically and empirically show that this assumption makes these methods incapable of recognizing OOD data when the ID model is trained with class-imbalanced data. Fortunately, by analyzing the causal relations between ID/OOD classes and features, we identify several common scenarios where proba- bilities that OOD data belong to ID classes should be the ID-class prioor distribution. Based on the above finding, we propose two effective strategies to modify previous inference-time OOD detection methods: 1) if they explicitly use the uniform distri- bution, we can replace the uniform distribution with the ID-class-prior distribution; 2) otherwise, we can reweight their scores according to the similarity between the ID- ID-class-price distribution and the softmax outputs of the pre-trained model. Extensive experiments show that both strategies significantly improve the accuracy of recognizing OOD data when the ID model is pre- pre-trained with imbalanced data. As a highlight, when evaluating on the iNaturalist dataset, our method can achieve ~36% increase on AUROC and ~-1 % decrease on FPR95, compared with the original Energy method, reflecting the importance of ID-class prior in the OOD detection, which lights up new road to study this problem.
INTRODUCTION
How to reliably deploy machine learning models into real-world scenarios has been attracting more and more attention (Huang et al., 2021 Liang et al. 2018 Liu et al.) 2020p. In real-world scenarios, test data usually contain known and unknown classes (Hendrycks Cimpel| 2017). We expect the deployed model to eliminate the interference of unknown classes while classifying known classes well. Nevertheless, current models tend to be overconfident in the unknown classes (Nguyen et al.) 2015), and thus confusing known and unknown classes, which increases the risk of deploying these models in the real world. Especially if the scenarios are life-critical (e.g., car-driving scenarios), we cannot take the risks of deploying unreliable models in them. This motivates researchers to study (OOD) detection, where we need to identify unknown classes (i.e., OOD classes) and classify known classes (i.e., in-distribution (ID) classes) well the same time (Hendrycks & Gimpel 2017: Hendrycks et al. 2019).
In the OOD detection, a well-known branch is to develop the inference-time/por hoc OOD detection Liang et al.) 2018 Liu et al., 2020) Hendrycks & Gimpel 2017 Lec et al., 2b1 Sun et al. 2021). where we are given pre-trained ID model and then aim to recognize upcoming OOD data well. The key advantage of inference-time OOD detection methods is that the classification performance on ID data will be unaffected since we only use the ID model instead of changing it. A general way to design a large-salle-frinnde to propose a score function by using the ID model's s information. For example, maximum softmax probability (MSP) uses the ID model's outputs (Hendrycks & Gimpel| (017). and GradNorm uses the ID model's 's gradients (Huang et al. 202T). If the score of data point is smaller, then this data point is OOD data point with higher probability.
methods (Huang et al.) 2021
inference-time OOD detection method is

---

Under review as a conference paper at ICLR 2023
x
Features
X
OOD Features
Y
Classes
Yeu
OOD Classes
8 DDD Style
:00D Slyle
:Shared Slyle
yin
Yout
Yin
You
yiu
Youl
X
Xout
X
X
X
Xout
S
S
S
(a) No Styles
(b) Different Styles
(c) Shared Style
Figure 1: Three common causal graphs OOD detection. Under these graphs, we prove that probabilities
that an OOD data point x x belongs to ID classes should be the ID-class-pries distribution Py (Theorem I However, some representative OOD detection methods (Huang ct al. [202]] Hendrycks & Gimpel] (017)) assume such probabilities be uniform distribution (c.g.. GradNorm in Eq
Bq. 2). In this figure, each node represents random variable, and gray ones indicate observable variables. stands for features, stands for classes, and stands for styles. In the three graphs, features are generated by classes (i.e.,
+ X) (Gong et al.] 2016 (Stojanov et al. 2021 or generated by classes and styles (i.c., Y + X <- 5) Yao et al. 2021). The three causal graphs broadly exist in common datasets. For example, (a) corresponds to datasets consisting of sketch images. like ImageNet-Sketch (Wang al. 2019) where ID classes could be cars and OOD classes could be animals; (b) and (c) correspond datasets consisting of common images, like ImageNet (Deng et al. (009)) and MNIST (LeCun et al.) 1998). In (b), the ID classes could be cars in ImageNet, and the OOD classes could be numbers in MNIST (different styles). In (c), the ID classes could be numbers in MNIST, and OOD classes could be classes in Pashion-MNIS, (Xiao et al. (017) (the same style). Through these graphs, it is clear that yrin X i.e., yii and X are independent.
However, some representative methods share an unproven assumption: the probability that an OOD data point out belongs each IDD class is always the same. Namely, for any xout they assume
[Pr(x out belongs to class 1),...,
......,P((e5 Pr(x out belongs to class K)] [1/K, 1,k.,,1/KK] 1/K]IxX == (1)
where is uniform distribution and K is the number of ID classes. Taking the GradNorm (Huang
et al. 2021) ). a state-of-the of-the-art art OOD detection method, as an example" let (x be ID model's
output of a data pointx, x, and the score function of GradNorm
a @D.L (fftmax(ff
ScradNorm (fo x)
(2)
00
where ② represents the ID model's parameters, softmax(fo) (x)) is vector consisting of predicted probabilities that x belongs to ID classes, and DKL |] is the Kullback-Leibler divergence. It is clear that GradNorm considers data. If the divergence between softmax higher probability. Nonetheless, since we do not have this assumption proven, we do not know whether it is correct. If not, the u-based score functions (e.g., Eq. 2 are ill-defined because they cannot guarantee that the lowest score corresponds to the most OOD-ness data.
reference distribution to distinguish between ID and OOD and u is smaller, then x is an OOD data point with
In this paper, we theoretically analyze the above assumption (i.e.. [ under three common causal graphs (Figure D] and find that the above assumption holds only when the ID-class prior is u, i.e., the ID model is trained with class-balanced data. In other cases, the reference distribution of OOD data should be the ID-class-prics distribution Pyn (Theorem [ i.e.,
Pr belongs to class 1)...., Pr(x belongs to class K)] IPy
Specifically, assume that we have K classes in training data (i.c., ID data). Let ny be the number
of samples in class j, then the total number of samples is N = Σx nz. Thus, we have Py = [nI/N, N2/N N.
Note that, MSP also has this assumption, will discuss it in Section 3.2
2

---

Under review as a conference paper at ICLR 2023
Sample Number
1200
1000
FPR95 (%)
80
(96
90
80
70-
600
70
400
60
200
50-
50
40
600
(Naturalist
Textures
Sample Class
0OD datasets
OOD datasets
(a) Data Distribution
(b) GradNorm
(c) Energy
Figure 2: (a) Plot showing the data distribution of balanced and imbalanced datasets. OOD detection performances of (b) GradNorm and (c) Energy. Smaller FPR95 values are better. Cyan (left) bar: the original method
balanced dataset. Yellow (middle) bar: the original method on an imbalanced dataset. Red (right) bar: the original method with our method on an imbalanced dataset. For a fair comparison, the sample numbers of balanced and imbalanced datasets are the same. More detailed results are shown in Appendix A.
Empirically, we test the performance of OOD detection methods when the data are not class-balanced (Figure 20. i.e., Pymm jf u. We find that the GradNorm, state-of-the-arr OOD detection method, will suffer from the imbalanced situation (see cyan and yellow bars in Figure 2). Besides, is interesting to find that Energy (Liu al.) (220), the other one of representative OOD detection methods that do not explicitly use u. also suffers from this situation (see cyan and yellow bars in Figure Based on Theorem and Eq. 3 propose effective strategies to modify previous score-based OOD detection methods using the ID-class-prior distribution: replacing (RP) strategy and reweighting (RW) strategy. In RP strategy, previous methods explicitly use the uniform distribution (like GradNorm), we can modify them by replacing u with the ID-class-prior distribution Pyi.. For example, we can modify score functions of GradNorm by replacing u in Eq. with Pyin:
@D y softmax
SpP-GradNorm (fee
For the methods that do not explicitly use the uniform distribution to compute scores (like Energy (Liu ctt al. 2.20)), we can use the RW strategy to reweight their scores according to the similarity between the -laas--prior distribution Py and the softmax outputs of the pre-trained model softmax(fe) (x)).
Namely,
Mew (fe, x) = SMethoa (fes, x) cos(sofmaax((fo( (x) Pyn.,,
where SMethod (fe, x) a score function proposed in previous studies (like Energy (Liu et al. 2020
We conduct extensive experiments to verify the effectiveness of RP and RW strategies. After our modification, the results (red bars Figure 2 show significant improvement, which illustrates the effectiveness of our theory. Meanwhile, our method achieves state-of-the-an performance four evaluation tasks. As a highlight, when evaluate the OOD detection performance on iNaturalist dataset, method can achieve ~36% increase on AUROC and ~61% 1% decrease on FPR95, compared with the original Energy (Liu et al. 2020) (see Table TT. It further validates that we cannot default the reference distribution of OOD data the uniform distribution. To improve the generalizability of OOD detection methods, the class class-prior distribution of the training data should taken into account, which might benefit future researches in the community.
PRELIMINARIES
Let X' C RRd and yin (1, ..., K} be the feature space and the ID label space. Denote X X, Xout E X' and yiu E yin by the random variables with respect to L' and you, P(Xin yin) the ID joint distribution, IP xin is the ID marginal distribution, and Px the OOD marginal distribution.
OOD Detection. Given the training data Dirain {(x , ..., (x",y") ))") independent and identi- cally distributed (i.idd)) drawn from P(Xin yia), the aim of OOD detection is to learn model
3

---

Under review as a conference paper at ICLR 2023
using Dimin such that for any test data drawn from Pxii or Pxou:: 1) if x is drawn from Pxt., then can classify x into correct ID classes; and 2) if x is drawn from Xxout, then h can detect x as OOD data.
Inference-time OOD Detection. A well-known branch of OOD detection methods is to develop the timme OOD detection (or post hoc OOD detection) methods (Huang et al. [202]] Liang
let al. 2018 Liu et al. 2020; Hendrycks & Gimpel 2017 Lee et al. 2018b Sun et al. [021] where
we are given pre-trained ID model and then aim to recognize upcoming OOD data well. The key
advantage of inference-time OOD detection methods is that the classification performance on ID data will be unaffected since we only use the ID model instead of changing it.
Score Functions. Many representative OOD detection methods use score-based strategy: given threshold 7, an ID model fe and scoring function S, then x detected as ID data if ((fe x) 2 y
ID, OOD,
(fe, x) 2 if S(fo, x) <>
(x)
(6)
The performance of OOD detection depends on how to design a scoring function S to make OOD data obtain lower scores while ID data obtain higher scores-thus thus, we can recognize ID/OOD data.
3 METHODOLOGY
Clearly, without any assumptions or conditions, OOD detection cannot be addressed well due to the unavailability of OOD data (Zhang et al. 2021). Therefore, investigate the feasibility of OOD detection, in this section, we consider a natural case that ID classes and OOD features do not interfere with each other.
3.1 ASSUMPTION AND THEOREM
Assumption 1. Random variables Xout and Y are independent, i.e., P(Xout, yri)) = Px Pym.
Justification of Assumption 1 To justify that Assumption iss realistic, we conclude three common causal graphs in Figure []] These graphs illustrate how the data is generated through the lens of causality. Notably, in Figure D we can observe that X and Xoutt are actually dependent, which very common our daily life. It seems that the dependence of X and X could result in the failure of Assumption 1 However, since X and Xout are dependent only because of the same style (S in Figure | instead of classes (Y in Figure 1 Yao et al. 2021), the condition that X and out are dependent does not conflict with Assumption I In fact, there exist many practical scenarios which meet the causal structure in Figure D e.g., MNIST and Fashion-MNIST (Xiao et al., 2017). According to this assumption, can prove main theorem, which provides the theoretical foundation for our paper.
Theorem 1. If Assumption
1 holds, then Py (5/x) Py ) for any y E yin
The proof of Theorem is in Appendix B.I In the inference-time OOD detection, researchers assume that the pre-trained model fe (x) can simulate the ID conditional distribution, i.e.. the probabilities that x belongs to each ID class: softmax(fe, (x)) == yP y
(1] (1xx), Py .y.... i (K|x), when the data point x is from OOD distribution. Therefore, Theorem I implies that if Assumption Ircnaptiiooooolss... hholds, then we hope that (ffttax(fffxx))) discuss how to utilize this novel observation improve existing boseed OOD methods. When the labels of the training dataset are not available, we can use the predictions made by the model as alternative to simulate empirical ID-class-prics distribution Pyin. The specific analysis and experiments can be found in the Appendix B.2
(1), yn (K)] Pyin, for any OOD data x. Next, we
3.2 RETHINKING MSP AND GRADNORM BY THEBREM II
According to Eq. 6 we discover that the score-based strategy has an implied assumption that if data point x has a lower score, then the data x has a higher probability detected as an OOD data point. Based on this assumption, we consider the ideal case that if data point has the smallest score, then what will happen? We answer this issue, when the score function is MSP.
Rethinking MSP by Theorem 1. We consider the MSP score and answer above issue by Theorem
4

---

replace the Under review as a conference paper at ICLR 2023
Theorem 2. Given data point X, (x) arg ( SMsp(fe, then
softmax (x)) = where u l= 1/K/ ER
The proof of Theorem is in Appendix B.3
According to the implied assumption, know that when the data point has the smallest score, then x has the largest probability detected as an OOD data point. Then, Theorem (ammmm2sswwwww distribution u, which is conflict with our observation (i.e., softmax (x)) R Pyi., if x is an OOD data point) in the ID class-imbalance case. Therefore, avoid the contradiction, uniform distribution in MSP as follows:
shows that in the ideal case, the output of this data point is uniform
SRP-MSP(fo: x) x)= max (ffx - Py (1))- iE()...,K)
Rethinking GradNorm by Theorem 1 Here, we discuss how to adjust the GradNorm score. By Eq. 2 is clear that in the ideal casc, we can conclude that softmax(fe(x)) se u, i.c.
lim sfftmax(fo(x))) = u, where (x() satisfies SGradNorm (fe, x) <9 7-00
(8)
Therefore, Eq. is inconsistent with our observation (i.c., softmax(fe(x)) se Pyin, if is an OOD data point) in the ID class-imbas duss-imbalance case. Similar to the MSP scenario, the basic idea is to the ID-class-pree celass-prroor distribution Py - (1), (1),...,Pyel Py (K)] to replace the uniform distribution u, i.e.,
@DRL (P i softmax(fe(x))
SRp-GraNNom xf
00
L7
3.3 OUR PROPOSAL: REPLACING AND REWEIGHTING STRATEGIES
Replacing (RP) Strategy. For those methods (e.g., MSP and GradNorm) whose score functions are deeply related to the uniform distribution the simple and straight way of modifying them is to replace the uniform distribution u with the ID-class-pricc distribution Py As mentioned in Section 3.2 we modify the score functions of MSP and GradNorm as
SRPAMSP(fer
max aE KE.....,K
(x)) (1)),
(10)
@Dri. (Pri (f
SRP-GradNorm (fo, x)=
(11)
00
Reweighting (RW) Strategy. For the methods that have no obvious correlations with the uniform distribution (e.g., ODIN (Liang et al. 2018) and Energy (Liu al., 2020)), design the RW strategy as a complementary to the RP strategy. RW strategy reweights their scores according to similarity between the ID-class-prior distribution Pyi. and sof (x)). Here, expect that the weights do not impact on the OOD scores seriously. In this paper, we use the cosine function as the weight function, which is one the most popular distances and similarity functions contrastive learning (Chen et al.) 2020 Grill et al., 2020 He et al. 2020). The main reason we choose cosine function that cosine a bounded function and suitable as a weighting parameter after normalization. Specifically.
MgW+Melhou (fo, x) SMettwa fee x) cos(softmax((e((x)) Pym)
softmax(fe(xx) P
(12)
SMeeehou x)
softmax(fe
|
where SMethoa (fon (fo, x) is a score function proposed in previous studies, e.g., ODIN and Energy. Next, introduce the details about the reweighted ODIN and reweighted Energy in the following.
5

---

Under review as a conference paper at ICLR 2023
Compared to MSP, the main improvement of ODIN is the use of temperature scaling strategy. We can modify ODIN as follows for a temperature T > O,
exp (f())/T)
SRW+ODIN (fe, x) = max
(os(softmax(fe(x)),
(13)
exp (x)
Energy (Liu et al. 2020) proposes replace the softmax function with the energy function (LeCun et al. 2006) for OOD detection. The energy function has a property that is highly correlated with the distribution: the system with a more concentrated probability distribution has lower energy. while the system with more divergent probability distribution (more similar to the uniform distribution) has higher energy (LeCun al. 2006). Thus, the energy of ID data is smaller than OOD data. Based we modily Energy as follows:
Eq. 12
Sgw+Energy (fe. x) = log
L(x)/T
cos(softmaxx(fe (
(14)
In this paper, we mainly realize our strategies using Eqs.
10
and
EXPERIMENTS
In this section, we construct series of imbalanced ID datasets whose data are sampled from the ImageNet-14 1K (Deng et al. 2009). Then, we train the ID classifiers on them as pre-trained ID models, and use large-scale ImageNet OOD detection benchmark (Huang & Li [021] evaluate our methods, i.e. RP+MSP (Eq. [O),, RP+GradNorm (Eq. [T], [T) RW+ODIN (Eq. [3). and RW+Energy (4). In addition, we also evaluate our methods on a real-world imbalanced dataset iNaturalist (Horn et al.. 2018), see Appendix A.2
4.1 EXPERIMENT SETUP
Dataset. Following Liu et al. (019),, we construct a series of imbalanced datasets that are sampled by the Pareto distribution in ImageNet-1K KK dataset. The definition of Pareto distribution
Imagehes
in Eq. (15).
Sample Number
Imapeben-
am
p(x)
(15)
za+1
600
In Appendix B.5 have shown that the parameter does not affect the level of imbalance. Thus, we set m 1. Additionally, we note that the level of imbalance depends the tail index a (see FigureB). thus, to evaluate the performance of our methods different imbalanced cases, take different tail index a.
400
200
200
400
600
000
1000
mpmlle Class
The frequency distributions of classes of the sampled datasets are shown in Figure 3 As the increase of the tail index a, the sampled datasets become more imbalanced, thus, the ImageNet- LT-a8 dataset the most imbalanced.
Figure 3: Data distribution with dif- ferent tail index a.
In the inference time, we use the large-scale benchmark proposed by Huang & Li (2021). In this benchmark, the OOD datasets include the subsets of iNaturalist (Horn al.) 2018), SUN (Xiao) et al. 2010), Places (Zhou et al. 2018), and Textures (Cimpoi et |l.] 2014). Note that, there are no overlapping classes between datasets and OOD datasets (Huang & Li
Evaluation Metrics. We two common metrics to evaluate OOD detection methods (Huang ct al. 202T): the false positive rate that OOD data are classified as ID data when 95% of ID data correctly classified (FPR95) (Provost et al. 1998) and the area under the receiver operating characteristic curve (AUROC) (Huang et al. 2021).
In fact, ODIN uses the modified softmax function with temperature T', which is also related to the uniform
distribution, so we can also modify ODIN with RP strategy. We can map the class- prior distribution to the same feature space with ODIN's OOD scores by temperature T. However, if following the default setting (T 1000)
ODIN, ||P yiu / ss 0. Thus, RP+ODIN may not work. We will discuss this issue in Appendix B.4

---

Under review as a conference paper at ICLR 2023
Table OOD detection performance comparison with other competitive score-based OOD detection methods. All methods are based on ResNet101 trained on ImageNet-LT MeteeNet-LT-a.... indicates larger values are better and indicates smaller values are better. All values are percetages. The bold indicates the best performance while the underline indicates the second.
Method
INaturalist
SUN
Places
Testures
Average
AUROCT
FPR95L
AUROC
FPR95L
AUROC*
FPR95
AUROCT
FPRO5]
AUROC1
FPR95]
MSP Hendracks & Gimgel
2017
63.95
97.72
93.13
92.11
42.74
18.79
60.03
CDDN Taleng Ri 2013
60.14
9%.70
70.63
93.13
70.14
91.96
41.83
18:
64.66
95.52
Mahalanoois Lex
20T8b
60.72
95.87
56.79
94.50
55.27
93.78
49.43
86.99
55.55
92.78
Energy dl
98.
GradNorm [Tunang 0
82.51
72.19
74.57
78.10
70.67
86.58
57.31
84.95
71.26
80.45
Maxlogit
(Hemiyy
60.14
98.70
70.64
93.13
70.15
91.96
41.83
98.30
60.69
96. 52
Dice
85.80
58.96
73.17
76.90
67.83
87.89
58.43
80.59
71.31
76.
KP+MSPROur))
64.95
91.79
67.46
43.05
98.51
60.71
RW (WDDD)N(Ours)
86.66
93.85
71.59
97.67
67.55
91.24
68.04
95.
73.46
96.01
RW+EaxgyyOuus)
91.92
37.89
80.81
76.22
77.15
81.18
64.48
86.19
78.59
70.37
RP-GradNom/Oenn
RW MWW-Maalogit Ours)
78.16
77.68
77.26
81.91
25.15
83.28
48.34
91.06
69.73
84.23
INsturalist
SUN
90
Places
88
Textures
70
80-
(%6
2.64
60
75-
EPR95
50
12
78
70
Data Type
Data Type
Type
Figure 4: Performance comparison of different data type. The figures shows the OOD detection performance of GradNorm and RP+GradNorm in four OOD datasets.
Baselines. In order to verify the effectiveness of our strategies, we select MSP, ODIN, Energy, GradNorm and Dice as the baselines, where Dice is the state-of state-of-the-ar (SOTA) method. Following Liang al. 2018), the temperature parameter T in ODIN is set be 1000 and
(Huang et al. 2021
in GradNorm
Models and Hyperparameters. We use mmclassification (Contributors 2020) with Apache-2.0 license to train ID models. The training details of ResNet (He et al. (016)) and MobileNet (Howard al. 2019) follow the default setting in mmclassification. Note that, all methods are realized by Pytorch 1.60 with CUDA 10.2, where we use several NVIDIA Tesla V100 GPUs.
4.2 EXPERIMENTAL RESULTS AND ANALYSIS
Verification of Two Strategies. Our strategies are applicable various score functions used by OOD detection methods. The performance of our methods and basclines are shown in Table [ Overall, after modifying previous methods using our strategies, their performance are significantly improved, indicating the effectiveness of our strategies. Specifically, RW+Energy achieves the highest AUROC (78.5%%) compared to all methods. As a highlight, + shows the most significant performance improvement all four datasets: ~61 % FPR95 decrease in iNaturalist, 177 FPR95 decrease SUN, ~ 10% FPR95 decrease in Places and 12% FPR95 decrease Textures.
Besides, strategies outperform the existing baseline methods in all evaluation tasks. Compared with the best baseline, RW RW+Energy increases AUROC from 71.31% to 78.59%, while RP+GradNorm reduces FPR95 from 76.1 17.11% % to 70.12% Experimental results have shown that strategies can significantly outperform the baselines in the ID-class-imbalanced scenarios.
Analysis of Detection Results Different ID Classes. Since the training dataset is imbalanced, follow Liu et al. (2019) to divide all ID classes into three categories (ID Head, ID Mid, ID Tail) for further analysis. In detail, ID Head category includes the classes containing more than 100 samples in the training dataset:ID Mid category includes the classes whose number of samples is between 20 and 100 in the training dataset; and Tail category includes the classes containing less than 20
httpp::/ www.thhubb... http//ppnneeeee 1 ass fitiiion

---

Under review as a conference paper at ICLR 2023
MSP ODIN
GradNorm
100
MSP
Gra dNomm
84
Dice O
ODIN
Dice
Mahalanobis
Average FPR95 (%)
96
Mahalanobis
RP GraaNoooomtuu
Average AUROC
76
Energy
92-
Energy
68
60
76
72-
52
2
3
Tail Index
Tail Index
(a) AUROCT
(b) FPR95,
Figure 5: OOD detection performance with ResNet101 trained on different imbalanced ID datasets. 1 indicates larger values are better and indicates smaller values are better.
samples in the training dataset. Then, we evaluate the OOD detection performance on three datasets: ID Head+OOD, ID Mid+OOD and ID Tail+OOD (details be found in Appendix (oppendi(AA..3)) If the performance of one method on ID Tail+OOD is better than that on ID Head+OOD, then this method performs better when facing tailed classes and OOD data.
In the case of GradNorm, experiment results in Figure 4 show that our method RP+GradNorm improves the performance the above three datasets (II) Head+OOD, ID Mid+OOD, and ID Tail+OOD). When we take closer look at the performance improvement, we notice that the overall performance improvement RP+GradNorm mainly due the significant improvement on the Tail+OOD dataset. This result might indicate that the previous method, like GradNorm, confuses OOD data and ID tailed classes, which hinders their OOD detection performance. And our strategies can overcome this issue. More detailed results are shown in Appendix A.1.3
4.3 ABLATION STUDY
Analysis regarding Tail Index a. Here, we report the performance of our method and baselines when changing the tail index E 3........ .). We conduct repeated experiments on these seven datasets (ImageNet-LT-a2, ImageNet-LTTa3...... ImageNet-LT-a8)) and the results are shown in Figure Overall, our method RP+GradNorm always outperforms other baselines with different imbalance degrees. More importantly, the performance improvement between RP+GradNorm and each bascline gradually increases, as the increase of the imbalance degree. This indicates that RP+GradNorm can handle different imbalanced scenarios better. More detailed results are in Appendix A.1.2
Analysis regarding Network Architecture, We evaluate methods on a different network archi- tecture, MobileNet- V3 (Howard et al.) 2019). Experiment results in Table 2 show that methods (RP+GradNorm and RW+Energy) still outperform baselines on four evaluation tasks even when we change the network architecture. In addition, RP+GradNorm has better performance FPR95 while RW+Energy has higher AUROC, corresponding to the performances of GradNorm and Energy.
Analysis regarding Model Size. We provide experiment about the model size of RP+GradNorm. We compare ResNet50, ResNet101 and ResNet152 52 trained on ImageNet-LT-a8 datasets. The results shown in Table 3 The optimal model is the smallest one (ResNet50), and we observe that the increase of the model size, the performance decreases. One of the reasons may be that small models are more difficult to overfit and thus more suitable for OOD detection in imbalanced scenarios.
More Experiments and Exploration. First, we can also regard the cosine similarity weights in the RW strategy as a score function, and conduct several experiments in Appendix 1.1.4 Second, to evaluate the stability of our strategies, we conduct 10 independent replicate experiments in Appendix A.1.5 Third, to further explore the collocation of existing methods and our design strategy, we conduct experiments regarding RW+MSP RW+GradNorm, RW+RP+MSP/Gran Appendix 1.6 Then, we further consider what will happen if the ID-class prior is not accurate in the practical applications and conduct relevant experiments in Appendix A.1.7 Finally, show in Appendix A.1.8 that our methods still work well when models are trained with long-tailed learning Park al. 2022] during the training phase.
shown in
strategies (Cao al. 2019
8

---

Under review as a conference paper at ICLR 2023
Table 2: OOD detection performance with MobileNet trained on ImageNet-LT-a8, t indicates larger values are better and + indicates smaller values are better. All values are percentages. The bold indicates the best performance while the underline indicates the second.
Method
iNaturalist
Textures
Aserage
AUROCT
FPR95]
AUROC
FPR95L
AUROC*
FPR951
AUROCI
FPR95]
AUROC1
FPR95]
MSP Hendsycks & Gimpel 2017
63.47
92.38
67.27
85.62
64.16
89.62
59.88
89.06
63.69
89.17
ODDN
2015
64.6%
66.06
82.23
69.44
86.28
Eneryy
et 2012
63.12
96.37
74.95
77.86
70.30
90.50
70.69
80.83
69.81
36.39
GradNorm PHiners sell 2021
70.87
78.12
69.70
67.59
66.00
85.75
63.09
74,89
67.41
76.59
Dice
65.61
86.40
69.35
66.38
65.95
88.42
68.85
68.19
67.44
77.35
RP+MSP(Ours)
$8.73
RW (DDIN(Ours)
80.20
73.32
72.46
77.48
68:25
89.08
67.62
73.24
72.13
78.28
RW+ Energy(Our))
81.64
69.94
76,10
80.36
70.41
86.71
65.58
85.82
73.43
$0.71
#P-(GradNoomm(((amn))
77.25
70.75
Table 3: OOD detection performance with model size increases. The RP+GradNorm method is
trained on ImageNet-LT-a8. All values are percentages.
Model
INaturalist
SUN
Places
Textures
Average
AUROCT
FPR95
AUROCT
FPR95
AUROCT
FPR95
AUROC1
FPR95
AUROC1
FPR95
ResNe150
89.85
50.03
80.73
64.52
74.69
78.18
63.31
77.73
77.14
67.62
ResNet101
91.23
43.87
77.36
73.53
72.67
83.29
62.94
79.80
76.05
70.12
ResNet152
88.24
53.14
73.45
78.96
68.61
87.14
59.41
83.71
72.43
75.74
5 RELATED WORKS
Inference- weapsnnceeiim//psst-ttee OOD Detection: Some methods (Huang et al. 2021 Liang et al. 2018
Liu et al. 2020, Hendrycks & Gimpel 2017: Lee et al. 201 8b) [un et al. 2021) focus designing OOD score functions for OOD detection in the inference time and are easy to use without changing the model's 's parameters. This property is important for deploying OOD detection methods in real- world scenarios where the cost of re-training is prohibitively expensive and time-consuming. MSP (Hendrycks & Gimpell 2017) directly takes the maximum value of the model's prediction the OOD) score function. Based on MSP. ODIN (Liang et al. 2018)) uses a temperature scaling strategy and input perturbation to improve OOD detection performance. Moreover, [Liu et al. (2020) and Wang et al. 222 propose to replace the softmax function with the energy functions for OOD detection. Recently, GradNorm (Huang et al., 202T) uses the similarity of the model preeictpe probability distribution and the uniform distribution to improve OOD detection and achieve state-of titte-of-theaate -he-hrt- art performance. In this paper, we mainly work on the inference-time OOD detection methods and aim at improving the generalizability of OOD detection in real-world scenarios.
Other methods (Hsu et al., 2020 Hein et al. 2019 Bitterwolf et al., 2020; Wang et al., 2021bb will complete ID tasks and OOD detection simultancously in the training time. Bitterwolf et al. (2020) uses adversarial learning to process OOD data in training time and make the model predict lower confidence scores for them. Wang al. (2021b) generates pseudo OOD data by adversarial learning to re-training a K+1 model for OOD detection. These methods usually require auxiliary OOD data available in the training process. Thus, the model will be affected by both ID data and OOD data. It is important for these method to explore an inherent trade-off (Liu
Training-time OOD Detection:
al. 2019 Vaze al. 2022
Yang et al. 202T) between ID tasks and OOD detection.
CONCLUSION
This paper theoretically and empirically shows that the unproven assumption of uniform distribution previous methods is not valid when the training dataset imbalanced. Moreover, by analyzing the causal relations between ID/OOD classes and features, we point out that the best reference distribution for OOD data is the D--clas--prior distribution. Based on this, we propose two simple and effective strategies to modify the uniform distribution assumption in previous inference-time OOD detection methods. RP strategy is suitable for the methods that directly use the uniform distribution to design the OOD score function, while RW strategy is designed for methods that potentially use the assumption. Extensive experiments show that both strategies can significantly improve the performance of OOD detection large-scale image classification benchmarks.

---

Under review as a conference paper at ICLR 2023
7 ETHIC STATEMENT
This paper does not raise any ethics concerns. This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, inseriminationbiissfaaeees concerns, privacy and security issues, legal compliance, and research integrity issues.
8 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of experimental results, we provide main codes the Appendix D The experimental setups for training and evaluation as well the hyperparamters are detailedly described in Section 4.1
REFERENCES
Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of-distribution data. In NeurIPS, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distributionnaawar margin loss. In NeurIPS, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton, A simple framework for contrastive learning of visual representations. In ICML, 2020.
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
MMClassification Contributors. Openmmlab's image classification toolbox and benchmark. https
2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of diittriiditii detection. In NeurlPS, 2021.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In ICML, 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In NewrIPS, 2020.
Matteo Guarrera, Baihong Jin, Tung-Wei Lin, Maria A Zuluaga, Yuxin Chen, and Alberto Langiovanni-Vincentelle Class-wise thresholding for robust out-of-distribution detection. In CVPR, 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2023
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. ICML, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Andrew Howard, Mark Sandler. Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3. In 2019,
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: detecting out-of- distribution image without learning from out-of-distribution data. In CVPR, 2020.
Rui Huang and Yixuan Li. MOS: towards scaling aut-of-distribution detection for large semantic space. In CVPR, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts the wild. NeurIPS, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Kibok Lee, Kimin Lee, Kyle Min, Yuting Zhang, Jinwoo Shin, and Honglak Lee. Hicrarchical novelty detection for visual object recognition. In CVPR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018b.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. In ICLR, 2018
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurlPS, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boging Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-taile tailed classification. In CVPR. 2022.
Pramuditha Perera and Vishal M. Patel. Deep transfer learning for multiple class novelty detection. In CVPR, 2019.
Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. Generative-disriminatie feature representations for open-set recognition. In CVPR, 2020.
Foster J. Provost, Tom Fawcett, and Ron Kohavi. The case against accuracy estimation for comparing induction algorithms. In ICML, 1998.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI, 2021.
11

---

Under review as a conference paper at ICLR 2023
Gabi Shalev, Yossi Adi, and Joseph Keshet. Out-of-distribution detection using multiple semantic label representations. NeurlPS, 2018.
Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime G. Carbonell, and Kun Zhang. Domain adaptation with invariant representation learning: What transformations to learn? In NeurIPS, 2021.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribttion detection. In ECCV, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurlPS, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICML, 2022.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier all you need. ICLR, 2022.
Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurlPS, 2019.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't 't know? In NeurIPS, 2021a.
Haotao Wang. Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for outtof-distribuutio detection in long-tailed recognition. In ICML,, 2022.
Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based upen-world uncertainty modeling for confidence calibration. In ICCV, 2021b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms, 2017.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Jingkang Yang. Haoqi Wang, Litong Feng. Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In ICCV, 2021.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. In NeurlPS, 2021.
Lily H. Zhang. Mark Goldstein, and Rajesh Ranganath. Understanding failures in -ii-tiiiititttionn detection with deep generative models. In ICML, 2021.
Bolei Zhou, Ágata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4026))1452-1144 2018
12

---

Under review as a conference paper at ICLR 2023
Table of Contents of Appendix
Further Experiments
14
A.1
Evaluation on ImageNet Benchmark
14
A.I.
Evaluation on Imbalanced Data and Balanced Data
14
A.1.2
Analysis regarding Tail Index
14
A.1.3
Analysis of Detection Results on Difterent ID Classes
15
A.1.4
Cosine Similarity A Score Function
17
A. 1.5
Performance Evaluation under Random Sampling
17
A.1.6
Ablation Study between Proposed Strategies
19
A.1.7
Robustness to Inaccurate Class Prior Distribution
20
A.1.8
OOD Detection with Long-taile Learning
20
A.1.9
Evaluation on the Balanced Dataset
A.2
Evaluation on iNaturalist Benchmark
21
A.2.1
Experiment Setup
................................
A.2.2
Experimental Results
22
Further Analysis
22
Proof of Theorem
22
B.2
Alternative Choice for ID-class-price prior Distribution
................................
22
Proof of Theorem 2
23
B4 4
Discussion about RP+ODIN
................................
23
B.5
Discussion about m in Pareto Distribution
24
B.6
Discussion about Feature-based Methods
24
B.7 7
Discussion about Class Dependen Thresholding
24
B.8
Discussion about possibility for RP+MSP
24
Detailed Related Works
25
D
Codes
25
13

---

Under review as a conference paper at ICLR 2023
A FURTHER EXPERIMENTS
A.1 EVALUATION IMAGENET BENCHMARK
A.1.1 EVALUATION IMBALANCED DATA AND BALANCED DATA
We randomly sample balanced dataset from ImageNet- IK dataset, which has the same sample numbers with the imbalanced datasets. conduct experiments on the balanced data as shown in Figure @ggure@aand and Figure 7 All methods shows similar trend, i.c., the performance drop lot when the training dataset becomes imbalanced (from cyan bars to ycllow bars). Moreover, our method shows a significant improvement with previous methods on all evaluation tasks (from yellow bars to red bars).
-/18
Ours/i
-/1B
Ours/88
75
70
80
AUROC (%)
70
AUROC
60
50
50
45
40
iNaturalist
SUN
Places
Textures
iNaturalst
SUN
Places
Textures
OOD datasets
OOD datasets
(a) MSP
(b) ODIN
-/B
-11
-+OursiB
-/8
-18
-+-OrrnnB
AUROC (96)
70
75
60
50-
INaturalist
SUN
Places
Textures
INaturalist
SUN
Places
Textures
datasets
(c) Energy
(d) GradNorm
Figure 6: OOD detection performance (AUROC) of (a) MSP (b) ODIN (c) Energy and (d) GradNorm. Larger AUROC values are better. Cyan (left) bar: the original method balanced dataset. Yellow (middle) bar: the original method on imbalanced dataset. Red (right) bar: the original method with our method on imbalanced dataset.
1.1.2 ANALYSIS REGARDING TAIL INDEX
We can obtain sampled datasets with different levels of imbalance based on the Pareto distribution. ImageNet-LT-at dataset is the most imbalanced, while ImageNet-LL-a2 the most balanced. We conduct repeated experiments on these seven datasets and results are shown in Table 4 Obviously, the more imbalanced the training dataset becomes, the more our methods (RP+GraNNrmm and Cosine Similarity) demonstrates their superior performance of OOD detection, compared other methods on all evaluation tasks.
It is noticeable that the detection performance of GradNorm is relatively stable matter how imbalanced the ratio changes, compared with other existing methods (such as MSP, ODIN, Energy). These methods explicily/implicityy uniform distribution. Thus, they will be affected lot if the prior distribution changes from the uniform distribution imbalanced/tailed one.
the discrepancy between the classifier's output and the
14

---

Under review as a conference paper at ICLR 2023
-/8
-11
~+OursiBB
-/B
100.0
97.5
95
FPR35 (%)
95.0
FPR95 (%)
52.5
90
90.0
87.5
85.0
80
62.5
INaturalist
SUN
Places
Textures
INaturalist
SUN
Places
Textures
OOD datasets
OOD datasets
(a) MSP
(b) ODIN
100
80
FPR9S (%)
70-
60
50
OOD datasets
0OD datasets
(c) Energy
(d) GradNorm
Figure 7: OOD detection performance (FPR95) of (a) MSP (b) ODIN (c) Energy and (d) GradNorm. Smaller FPR95 values are better. Cyan (left) bar: the original method on balanced dataset. Yellow (middle) bar: the original method on imbalanced dataset. Red (right) bar: the original method with our method on imbalanced dataset.
As for GradNorm, we conjecture that considering the gradient space might be robust to the changes of priors (such as from a uniform prior to an imbalanced prior). To verify this conjecture, we conduct the experiment that KL divergence is directly used to measure the discrepancy between the output of the classifier and the uniform distribution (i.c.. GradNorm without gradient-norm process). The results are shown in Table 5 Obviously, this KL-based method is also significantly affected by the imbalanced situation, then we can verify this conjecture. Thus, we confirm that GradNorm's robustness of the imbalanced ratio depends the gradient space.
A.1.3 ANALYSIS DETECTION RESULTS DIFFERENT ID CLASSES
We calculate the evaluation metrics for the three categories (ID-Head. ID-Mid, ID-Tail) by randomly sampling OOD data in equal proportions corresponding to the number of samples in each category. For example, we have 50000 ID samples and 10000 OOD samples in total. If the number of samples in category Head is 10000 and accordingly we will sample 2000 OOD samples, then we use the 12000 samples to calculate AUROC and FPR95. The results reflects the confusion degree between ID Head data and OOD data in the view of OOD detection methods. We conduct experiments to analyze the performance of different data types, shown in Table
We try to analyze tailed categories using confusion matrix, and there are some cases in Table [A @ is the result, where A represents the number of ID samples in the current class that correctly classified as ID while C represents the number of ID samples in the current class that are misclassified ID. D represents the number of OOD samples close ID samples in the current class that correctly classified as OOD while B represents the number OOD samples close to ID samples in the current class that misclassified OOD.
In Class #88 and #94, more OOD samples are correctly classified after applying our strategies, while the performance of Class #671 remains stable. In Class #671, more ID samples correctly classified,
15

---

Under review as a conference paper at ICLR 2023
Table 4: OOD detection performance with ResNet101 trained on different imbalanced ID datasets. indicates larger values are better and indicates smaller values are better. All values are percentages.
Dasssets
Method
AUROCT
INttrrlist
FPRUS
AUBOCY
FFROOS
AUTRCC
FPROST
AUROC
Testures
FPRUS.
AUROCY
Averago
FFRR9S]
Hisllnckkk Gourel
71.57
80.73
DDDN Laggr al. 201
01.17
71.77
54.20
95.76
Malaland lracetiie Fn purdb
92.51
InageNass1Ta2
Euergy Lhrseei Piee
72.00
95.18
GradNoent contane cell
7582
72.86
75.33
70.13
86386
Dice 11p2004
72.17
77.31
70.28
63.85
50.12
86.38
GrinNiii Donst
70.15
70.33
56.88
Cosine Similartty/Oun))
Hendpacks Gimpel
90.91
Loopz AM133
Mahalanessz (zzett Kupph
ImageNes-.T-a
Energy
70.94
82.25
71.42
82.45
Stnilarity/Ous)
90.66
87.73
2017
72.41
89.92
20.36
70.35
92.76
82.05
71.35
95.50
Mahalan raleseeeee
49.37
96.48
93.36
54.95
85.45
ImageNst-LT.4
Energy
66.27
85.85
69.63
95.14
91.44
GradNocn
80.87
71.75
71.00
69.86
57.87
81.72
77.35
75.60
72.35
73.36
68.94
82.18
78.58
86.20
60.90
75.29
21.95
70.40
$4.43
59.85
86.51
72.93
74.32
Coiine
89.21
48.11
70.06
$7.25
86.60
73.31
MSM llendrucks Gimpel 2017
72.18
91.15
85.42
20.38
48.59
97.13
65.89
90.36
69.55
94. 68
75.92
$4.36
73.07
87.04
46.49
96.47
66.22
90.63
Mahaland (1 kirrh
48.35
95.03
58.07
91.11
$7.20
90.50
50.00
83.95
$3.41
50.58
ImagoNet-.ITos
Energy hrrrat pra3
66.85
95. 98
38
85.79
73.25
K7.89
46.25
95.89
64.68
91
GradNern Pinsn 2011
84.01
67.28
30
75.25
69.82
65.35
58.53
82:52
71.91
77.11
Dis: (Sur
82.88
65.51
73.14
76,80
66.94
80.04
$7.04
82.25
70.00
78.40
39.59
51.01
72.95
20.91
81.05
74.53
72.42
Cosine Similanty(Ouns
91.33
41.44
78.57
71.83
74.41
79.89
60.13
86.74
76,16
69.97
MS3 Henlnykk Gingel 2017
70.99
91.08
71.63
86.67
70.50
96.62
45.49
97.66
64.65
90.50
7082
92.37
74.34
87.81
72.18
19.25
43.49
96.33
65.13
91.36
Mahalandii Mi 1o:
kuz4h
$7.62
89.82
64..18
89.3
59.07
$9.86
49.49
84.92
56.59
$8.39
ImageNot LT
Energy Liberri ui
67.93
93.85
72.62
89.16
70.64
$9.81
43.55
95.94
63.63
92.20
GradNornt Connll 2021
$3.49
07.76
74,00
76.58
69.12
$5.87
55.75
85.46
70.60
79.92
Dice SUr 2i2llie
82.31
68.27
71.70
79.35
66.20
$9.82
56.04
85.60
68.56
30.76
RP.Crin Luis
89.67
$1.55
75.09)
77.47
69.89
$6.28
5%.90
84.35
73.21
74.92
Cosine Similartty(OOs)
91.22
43,01
7M.35
78.35
74.64
$2.94
61.08
88.63
76.07
73.23
MSH llendrycke Gimrel
2017
65.60
96.81
68. 50
90.68
67.22
91.26
46.65
98.16
61.09
94.25
broN (Zagrr
201k
63.13
72
71.29
91.17
69 16
91.20
43.35
97.64
61.23
94.43
Mahalan Trera mi
50.52
97.88 85
$4.1 18
92. 02.96
54.10 1.19
90.31
51.56
88.78
$2.71
92. 70
ImageNat-l.eo1
Energy Irrraa
59446
98.20
70.56
91 89
68.07
91.90
43.77
97.41
60.47
94.85
GraiNeen Peotan Lalla
20
80.38
77.04
73.00
77.45
66.86
$8.60
55.65
83.21
60.74
$1.57
Dice (0l
2012
81.61
70.42
71.9
78.95
63.79
91.40
57.72
81.95
58.58
$0.68
88.59
55. 40
75.11 17
75.35
68.75
66-41
63,98
80383
23.01
74,50
Conine (imilrrity(mm))))
90.12
49.49
78.62
77.87
73.4
$2.00
62:45
87.54
76.25
74.48
MSP Henlpska Gingel 2017
63.95
97.72
66.60
93.13
66.84
92.11
42.74
98.79
60.11
95.44
bD Sanilemma miningg or 2013
61.14
95.70
70.63
93.13
70.14
91.96
41.83
98.30
60.69
95.52
Mahalanduu 2i dur
60.72
95.87
36.79
94.50
53.27
93.78
49.43
88.99
53.55
92.78
ImageNet LT
Energy Literii king
35.99
98.74
71.12
93.11
70.24
91.30
42.38
98.07
59.93
95,30
GradNoen Mintk c 11 2021
82.51
72.19
74.57
78.10
70.67
$6.58
$7.31
81.95
71.26
30.45
Dise Sue & 201
85300
$3.96
73.17
76.00
67.83
$7.89
54.43
80169
71.31
76.11
RP GraGrene CuisT
91.23
43.07
77.36
73.55
72.67
$3.29
62.94
79.80
76.05
70.32
Cosing Similattty/OOss)
39.51
$1.42
$1.55
73.25
77.47
75.38
62.41
87.59
77.65
72.66
Table 5: OOD detection performance with ResNet101 trained on different imbalanced ID datasets. KI. stands for using only KI. divergence as the OOD) detection function. 1 indicates larger values are better and 4 indicates smaller values are better. All values are percentages.
Datasets
Method
AUROCY
INaturalist
FPROS]
AUROC
SUN
FPR95]
AUROCT
FPR95L
AUROCT
Textures
FPROS]
AUROCT
Avenge
FPROS]
ImageNes-LT-a2
60.66
95.51
75.21
83.06
86.15
40.36
95.25
62.04
ImageNasLLTa3
KL
68.14
94.26
74.91
83.22
70.32
88.10
44.55
94.10
64.48
39.92
ImageNet-LT-a
KL.
66.58
94.78
72.97
86.35
69.66
89.62
44.97
95.25
63.54
91.50
65.14
75.02
86.18
88.03
96.01
64.56
91.53
ImngeNat-LT-a6
KL
68.14
93.77
72.30
89.82
70.44
89.86
43.10
96.15
63.50
92.40
ImageNet-LT-a7
KL.
59.71
98.10
70.32
92.18
68.12
91.81
43.49
97.41
60.41
94,88
ImagoNet-LT-o8
98.72
71.09
91.13
42.27
98.12
while the performance of Class #94 of ID samples is sacrificed for more performance improvement of OOD samples.
Moreover, we visualize the OOD score distributions in Figure Obviously, the results and figures show that the previous methods tend to confuse OOD data and the minority classes, which hinders their performance of OOD detection. And our strategies can reduce the confusion to improve OOD detection performance.
16

---

Under review as a conference paper at ICLR 2023
Table 6: Performance comparison of different data type, All methods are based on ResNet101 trained on ImageNet-LT-a8. All values are percentages.
Method
Dau Type
AUTBCCT
FFROS
FFR001
AUBOCT
EPROSS
AUTROCT
FFROSS
Hiann
Hoad
73.15
75.63
$2.66
55.90
80.75
71.64
38.85
58.62
96.62
Overall
91.22)88722)
43.831 28 26:
77.05(2.48)
74.20 3.90)
72.306+1.09)
53,331
65,05+5.74)
(P4GraiNoomm(Cmm)
Head
87.50x+4.85)
50,94 14.54
68.13 5.00)
78.770+5.59 5.599
61.7113.155
52.56++0996 (9.021-5.36)
83.99
Mid
91 11.42x7.50)
42. 11.20-27.224
76.87(-0.59)
23.00--2.99
72. 22.1220.22)
K3-30(-1.11)
63.324+7.38 333
5.50
Tail
13.38(+10.67)
37.581-38.19)
(1.83(46.69)
10.33(-1045)
78,51(46.57)
80.9817.857)
69.40[+10.78)
77.016-9.61)
Table 7: Confusion matrix some tailed categories.
Class
GradNorm
RP+GradNorm
88
47 14
47 4
94
106
29
18
227
671
35
11
15
9 1
Table 8: Performance comparison under random sampling. All methods are based ResNet101 trained on different imbalanced ID dataset with tail index 8. The results are means + standard errors among ten randomly sampled datasets. t indicates larger values are better and & indicates smaller values are better. All values are percentages.
Method
INatunlist
SLN
Places
Textures
AUROCT
FPRM5
AUROCT
HPRRS]
AUROCT
FPR95]
AUROCT
FPRY5]
Hendrscks Gimpel 2017
65.19100995
93.77.00.88
59.83.11.19
93.4420.47
97.16.00.26
ODIN Liage & au 2016
59.3010..8 86
96.99100.31
66.82.13.6
94,210.48
59.84.1..35
91.6510.10
39.7910.62
32.10.1000
Mahulanobin Lav di 20r
$2,70:1.055
94. 04.56.10.72
50. 12:15.09
95.79-1.33
49.72:13.50
95 95.66:2:1000 1.04
54.79113.94
91. 36:12.222
Energy La ptau
53.96-20.77
9% 0077
65.44-+3.2
96. 96.26-0.49
5867+1-46 16
95. 95.54+0.51
57.09+0.76
09-07--1111
GradNorm
2121
81.03=0.556
68.73+114)
76.03-1.77
70.39-1.3 1.44
70.87+0.77
83.39-1.18
61.78+0.83
79.00+0.80
Nome RP4GrndNonn Ours)
83.48:1.03
61.93::2.26
78.74:2-43
62.66-1.81
74.20:10.78
77.36
64.11:1..33
76.80:10.69
Cosine Similariy(Quus)
71.99-
84.29+13.22
71.72+3877
82.04 3.06
66.63+44.70
87.99- 3.45
51.46-1-.44
02.95+1199
Table 9: OOD detection performance with model size increasing. The RP+GradNorm method is trained on ImageNet-LTT-8. All values are percentages.
Model
INaturalist
SUN
Places
Textures
Average
AUROCT
FPR95]
AUROC1
FPR95
AUROCI
FPR95
AUROCT
FPR95
AUROCT
FPR95..
ResNe150
89.18
51.42
81.55
73.25
77.47
78.38
62.41
87.59
77.65
72.66
ResNet101
90.34
47.35
81.30
75.55
77.84
80.01
62.04
88.05
77.88
72.74
ResNet152
89.19
50.08
79.53
79.20
76.25
82.63
61.11
89.17
76.59
75.27
MobileNet
82.55
65.76
79.73
69.31
73.84
79.47
69.09
78.21
76.30
73.19
A.1.4 COSINE SIMILARITY AS A SCORE FUNCTION
We also regard the cosine similarity weights in the RW strategy several experiments in Table Table g and Table We notice that the cosine similarity also achieves significant improvement compared with baselines in main evaluation tasks. Yet notice that the cosine similarity is sensitive to the ID data distribution, since the performance in random sampling experiments (see Table is not good enough compared with RP+GradNorm.
function, and conduct
A.1.5 PERFORMANCE EVALUATION UNDER RANDOM SAMPLING
To further evaluate the performance of our method RP+GradNorm, we conduct experiments in 10 different ID datasets, which are generated randomly by the Pareto distribution with a = 2 and = 8 from ImageNet- The results on ImageNet-a8 dataset are reported in Table From this table, can be observed that, our method RP+GradNorm can outperform baseline methods on all evaluation tasks. As a highlight, RP+GradNorm reduces FPR95 from 70.39% to 62.66% All these results show the method RP+GradNorm still outperforms all baselines under random sampling. ImageNet-a2 dataset is more similar to the balanced dataset. The results are reported in Table TO, It can be observed that, our method RP+GradNorm can still outperform baseline methods all evaluation tasks.
17

---

Under review as a conference paper at ICLR 2023
Jmapethont)
D-dd
Density
0.00 Poourallet
Density
000 (Pacees)
003 [Tooture)
OOD Scores
DOD Scores
DOD Scores
OOD Scores
(a) MSP
D drraggllet D-Fiead
13 ciraghheti
11m Im-spelert
ID Imageevee) ID-leed
ID-Heed
ID-Htedd
D-Hid
1D-MBc
ID-Ni
ID-Ta
ID-Tu Tal
Tal
Densiyy
OCD (Paalrra GU)
CCD (SUNI
Dunsity
OOD (Pacos)
Dunsiit
000 [Tuxturss)
GOD Scores
0.00 Scores
000 Scores
00D Scores
(b) RW+MSP
Figure 8: OOD score distribution of (a) MSP and (b) RP+MSP.
ID (maggeNet) ID Heod ID-Wi
D Orragelee D-HosE E-Hid
(maggeMet)
Emagetiet Head
D D-Head
D-Hid
ID-Mic
Denstty
OOD (Nat.rraist) (s))
ID-aal
Demsity
p-Tai OCD (5UN)
Demalty
OCD (Paccal
p-tail
OOD (Thoda us)
ID-Tal
DOD Scores
DDD Scores
0D0 Scores
00D Scores
(a) ODIN
Imagehez
I0 (imagehee))
UetgeNet)
lirageNett
DDiiedd
ID- Head
E-lies:
ID-M
ID-Mid
B-dd
D-
Donsty
COO (Naturatett)
1D-Tx|
000 (SUN)
ID-THi
OCB (Placos)
Dunaly
0CD (Tletuunn)
G0D Scores
o0D Scores
OOD Scores
OOD Scores
(b) RW+ODIN
Figure 9: OOD score distribution of (a) ODIN and (b) RW+ODIN.
Table 10: Performance comparison under random sampling. All methods are based on ResNet101 trained on different imbalanced ID dataset with tail index 2. The results are means + standard errors among ten randomly sampled datasets. 1 indicates larger values are better and | indicates smaller values are better. All values are percentages.
Method
iNaturalist
SUN
Textures
AUROC:
FPR95
AUROC
FPR951
AUROC*
FPR951
AUROC!
FPR95]
MSP Hendrycks & Gimpel 2017
72.55=00.99
67.64-0.55
69.23-40.3
87.33=90.2
64.82-0.25
90.21-00.77
50.58-00.48
96.02-0.26
ODLS
70.70=0.63
89.4120.75
72.03-00.43
00.5
67.26:0.38
89.28-10.46
4488::0.99
95.57=0.28
Energy
91.14.0.83
72.52.10.63
85.27
67.33-1.554
43.1910.74
95.42-0.27
GradNorm Hanng ah 2021
83.66=00.93
61. 21.90=2..25
78. 78.40=00.76
66.78=1.31
72.09=00.67
79.47=1.00
61.80=0.72
80.21
RP-Gramnarmttmmm
83.77=0..5
61.83:2.300
78.84-0.77
65.79_1.40
72.50-:0.69
78.65=1.117
61.91:0.74
8..1
18

---

Under review as a conference paper at ICLR 2023
(inageNet) Driisd
D-Hoad
Hi-d ID-TN1 uoo))
Deiod
Deld
D- OCD (Paztura
D-5M 2C0 (SUN)
Density
Demaity
Density
Denstry
OCD footurce
DOD Scores
OOD Scores
DOD Scores
O0D Scores
(a) Energy
Huod
D- DCD [Piatura
ID CCD (SUIN)
Density
Dendity
CCD cP1
000 (Trotures)
GOD Scores
OOD Scores
oOD Scores
oOD Scores
(b) RW+enyrgy
Figure 10: OOD score distribution of (a) Energy and (b) RW+Energy.
ID (ImageNet) ID Haad ID Kil ID-sil OOD Netura 8x0
ID (ImageNet) IG- Heee IG Hid I0-Tai OOD SUN)
ID Smagshee) ID-ftedd
D (imagelett E-Piaad
ID Do-92
D-Hid
ID
D-ggi
Density
Density
0OO (P aces)
Dennity
0CD /Rotured
OOD Scores
oOD Scores
000 Scores
00D Scores
(a) GradNorm
ID (raggNett
ID Imapebers ID-lledd
(mtagsaet)
D [imggelet D-lesed
ID- (D-iledd
ID-less
Mic
98
ID-98
D-iid
Donsity
CCD (Nuturatati
TD-Tal
Density
OOD (SUNI
1
Density
DCD (fteco))
ID
Density
DCD (Totureal
D-
0OD Scores
0OD Scores
0OD Scores
000 Scores
(b) -rrrGradoorm
Figure 11: OOD score distribution of (a) GradNorm and (b) RWrGrdNNorm.
A. 1.6 ABLATION STUDY BETWEEN PROPOSED STRATEGIES
To further explore the collocation of existing methods and our design strategy, we conduct experiments regarding RW+MSP, RW+GradNorm, RW-RP-MSP/GradNN
as shown in the table [T] Since the cosine distance is superior to the performance of MSP, RW strategy can modify the output of MSP to improve it better than RP strategy. However, for GradNorm, RP strategy performs better than RW strategy because the cosine distance does not perform much better than GradNorm. In contrast, RP strategy better matches the idea of the original method. Therefore, we suggest that the choice of the two strategies follows the idea of preferentially matching the original method. As for RP-RW+MSP/GradNomm they do not perform well. Taking GradNorm as an example, we think the main reason for this phenomenon is that RP+GradNorm has outstanding performance, but RW strategy performance is worse than RP+GradNorm. Thus, adding RW strategy to RP+GradNorm has negative effect, the performance is significantly reduced. On the other hand, for MSP,
19

---

Under review as a conference paper at ICLR 2023
Table 11: Ablation study between proposed strategies. All methods trained ImageNet-LT-a8 dataset with ResNet101.
Method
Strategy
iNaturalist
SUN
Places
Textures
Average
AUROCT
FPR95
AUROC1
FPR95
AUROC1
FPR95]
AUROC1
FPR95]
AUROCT
FPR95]
w/o
w/o
63.95
97.72
66.60
93.13
66.84
92.11
42.74
98.79
60.03
95.44
MSP
64.95
96.44
67.39
91.79
67.46
91.16
43.05
98.51
60.71
94.48
W/o
81.77
78.77
77.83
78,10
75.67
81.98
51.24
90.44
71.63
82.32
61.91
w/o
w/o
82.51
72.19
74.57
78.10
70.67
86.58
57.31
84.95
71.26
80.45
GradNorm
w/o
W/
91.23
88.96
43.87
$6.32
80.05
73.53
82.45
77.47
83.93
62.08
62.94
92.39
77.14
76.05
78.77
70.12
82.56
85.92
77.89
89.06
88.23
58.73
73.87
89.89
Table 12: OOD detection performances with different level of noises.
Method
Noise Intensity
NNaturalist
SUN
Places
Textures
Average
AUROCT
FPR95
AUROCT
FPR95
AUROCT
FPR95)
AUROC1
FPR95]
AUROC1
FPR95
61.95
96.44
67.39
91.79
67.16
91.16
43.05
98.51
60.21
94.15
0.1
65. 12
96-40
67.24
91.92
67.55
90.97
42.88
98.49
60.70
94.45
RP-MSP
0.2
65.06
96.41
67.19
92.007
67.51
91:08
42.86
98.51
60.65
94.52
64.80
96.75
67.04
67.38
91.32
42.79
98.
60.28
11
64.48
97.15
66.75
92.82
67.18
91.73
42.72
98.76
60.28
95.11
86,66
85
67
97.34
95.37
73.46
96.05
RW-ODIN
0.5
0.2
73 16.16
65.59
81 944
83.52
$7.79
62.75
96. 25
96.08
56,86
61.46
96.46
966 68
51.25
$2.55
944
944.36
62. 62.48
$7.87
92. 29
92.61
65.75
86.32
87
93.10
0
91.92
37.89
80.81
76.22
77.15
$1.18
64.48
$6.19
78.59
70.37
91.79
37.93
80.65
76.82
77.12
$1.24
61.33
86.49
78.47
70.62
Energy
70.66
91.41
39.09
80.31
77.42
76.91
81.51
62.79
87.38
77.86
71.35
89.53
45.44
79.06
78.57
76.44
81.35
60.48
89.
76.38
73.63
87.24
51.32
76.98
74.70
82.04
$8.05
88. 88.92
74.24
75.47
91.23
43.87
77.36
73.53
72.67
62.94
79 79.80
76.05
70.12
0.1
91.08
44.37
77.002
74.18
72.52
83.30
62.46
80.11
75.27
70.49
RPaGrreNNemm
90.89
44.70
76.69
74.65
72.22
83.67
62.24
80.09
75.51
70.78
90.27
89.89
48.70
75.66
75.94
71.24
84.70
61.37
80.59
74.54
72.48
88.05
55.07
71.82
76.33
70.51
85,30
60.73
80.60
73.52
71.33
the RP strategy has a negative impact on RW+MSP, resulting in the degradation of OOD detection performance.
A.1.7 ROBUSTNESS TO INACCURATE CLASS PRIOR DISTRIBUTION
We further consider what will happen the ID-class prior is not accurate the practical applications. In this regard, we conduct relevant experiments. Specifically, we simulate this kind of error regarding prior by adding noises with different intensities to the ID-class-prior distribution. Assume that the standard deviation of the ID-class-rrio distribution is std. The noises add will follow (0, std), where controls noise intensity. The results are shown Table [2] After adding noise, the performances of OOD detection do decrease, but not much, which shows that method robust to inaccurate class prior distributions.
A.1.8 OOD DETECTION WITH LONG-TAILED LEARNING
We train ResNe150 with some long-tailed methods, like LDAM (Cao et al. 2019) and CMO Park al. 2022). Then we evaluate with different OOD detection methods and our strategies, as shown
Table 13 The models with LDAM loss do perform better than those with CrossEntropy loss, but after applying our strategies, there are also significant improvements all methods. However, CMO does not bring the performance improvement of OOD detection LDAM does, and even performs worse than CrossEntropy. We think that this phenomenon indicates that not all long-tailed training methods are helpful to improve the OOD detector. But the results show that strategies still works well while the models try to overcome the class imbalance in training time.
At last, We would like reiterate view class-imbalanced OOD detection:
20

---

Under review as a conference paper at ICLR 2023
Table 13: OOD detection performance with long-tailed learning methods
Method
Long-tailee Method
INuturalist
SUN
Places
Textures
AURCCC]
FPR95]
AUROCT
FPROS
AUROCI
FPR95]
AUROCT
FPRMSL
AUROCT
PPR95]
MSP
CrossFarryy
62.50
97.30
67.46
92.14
66.89
91.49
42. 64
98.44
59.87
94.82
RPIMSP
CrossEntrpy
63.45
95. 95.88
67.83
90.85
67.41
90.47
42.74
98.26
60.36
93.87
MSP
LDAM
68.12
95.22
70.04
89.39
69.72
87.71
43.04
98.07
62.73
92.60
MSP
CMO
71.13
83.77
59.35
93.81
60.35
92.77
49.13
93.58
59,99
90.98
RP+MSP
CMO
70.82
$4,59
58.92
94.62
59.96
93.49
43.99
93.85
59.67
91.64
ODIN
CrossEntrogy
59.34
98.59
72.57
91.68
70.85
90.83
42.00
98.16
94.81
KW+ODIN
CrossEntropy
ODIN
LDAM
64.08
98.28
74,10
90.79
72.93
$9.01
39.74
98.37
62.79
91.11
RW+ODIN
LDAM
85.94
71.28
68.59
96. 92
66.77
966 63
50. 30
93.53
67.99
89. 59
ODIN
CMO
73.67
81.36
58 944
31
59.57
92.54
49 34
92.41
60.33
89.93
OWIN
CMO
77.19
$1.55
53.21
26
83.67
$8.60
Energy
CrossEttroyy
56.25
98.95
73,60
91.56
71.32
90.37
12.65
98.10
60.96
91.75
RW-Enery
CrossEnrrpy
90.57
42.68
80.09
76.41
76.12
80.78
64.68
86.17
77.87
71.51
Fnergy
IDAM
59.93
98 M0
74.74
91.99
72.74
900.99
39:51
98.37
61.73
94.89
RW- WW-eneegy
LDAM
92.32
$2.38
74.25
79.08
32
79.16
68.86
Energy
KW-Energy
CMO
79.75
71.54
54.32
97.12
53.63
96.51
52.31
90.32
60.00
$8.87
GradNomm
CrossEntrop
$0.61
74.33
78.73
68.77
72.78
$2.15
58.12
81.97
72:56
76.80
XPaGmdNamm
CrocsEntroyy
$9.85
50.03
80.73
64.52
63.31
77.73
77.14
67.62
RP+GradNorm
LDAM
92.87
36.75
82.70
63.74
77.41
77.21
60.50
79.01
78.37
64.18
GradNomm
CMO
78.56
77.25
$7.91
95.74
56.94
96.06
$2.88
89.79
61.57
89.71
RP+GmdNorm
CMO
84.90
63.05
63.65
92.25
61.24
94.42
39.17
86.06
67.24
83.95
Data imbalance is a common phenomenon, and even a slight imbalance (like the ImageNet- LT-a2 dataset) can still lead decrease in the performance of the OOD detector. After applying our strategies, this phenomenon can be improved.
Developers do not necessarily use strategies to overcome data imbalance during the training phase of the model, depending whether developers need to pay more attention to the minority in specific applications.
Even developers use strategies to overcome data imbalance during the training time, is very hard to obtain class balanced classifier. Experiment results show that method can achieve performance improvements with or without a strategy to overcome data imbalance.
1.1 EVALUATION THE BALANCED DATASET
As for RP strategy, when the training dataset is balanced (the class-prior distribution is uniform distribution in Eqs. 10 and RP+Method will be same as the original method.
For RW strategy, we conduct experiments on full ImageNet dataset (note that it is balanced) and the results are shown Table [4] The performances of Energy and ODIN are quite close to the performnce of Energy and ODIN using RW strategy on the balanced dataset.
Table 14: OOD detection performances full ImageNet dataset.
INaturalist
SUN
Places
Textures
Average
Method
RW Strategy
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC:
FPR95
ODIN
w/o
89.81
45.74
83.01
63.84
82.50
66.48
81.31
65.51
84.16
60.39
W/
89.87
45.37
83.05
63,59
82.54
66.32
81.36
65.27
84,20
60.14
Energy
w/o 93.28
37.62
49.48
87.36
54.61
86.80
53.40
89.01
48.78
W/ 93.48
35.86
88.48
50.13
87.29
54.80
86.72
53.24
88.99
48.51
A.2 EVALUATION ON INATURALIST BENCHMARK
A.2.1
EXPERIMENT SETUP
Datasets. We use iNaturalist as ID dataset, which is a real long-tailed dataset, and use Texture as OOD dataset, which has overlapping classes with the former.
21

---

Under review as a conference paper at ICLR 2023
Table 15: OOD detection performance on the iNaturalist benchmark.
Textures
Method
AUROCT
AUPR-INT
AUPR-OUTY
FPR95..
MSP
51.41
91.10
9.66
98. 16
RP+MSP(Ours)
51.54
91.12
9.70
97.96
ODIN
37.65
86.53
7.44
99.56
RW+ODIN(Ours)
70.80
94.96
19.73
85.00
Energy
30.54
84.61
6.68
99.93
RW+Energy(Ours)
66.99
94.38
17.32
88.03
GradNorm
31.29
83.84
6.75
99.61
RP+GradNorm(Ours)
36.26
85.10
7.34
98.71
Cosine Similarity(Ours)
63.64
93.66
14.58
91.26
Model and Hyperparameters. We use mmclassífication Contributors 2020) with Apache-2.0 2.0 license to train ID models. The training details ResNet (He et al. (016) follow the default setting in mmclassification. We the model trained in ImageNet as the pre-trained model and finetune in iNaturalist. Note that, all methods are realized by Pytorch 1.60 with CUDA 10.2, where we use several NVIDIA Tesla V100 GPUs.
A.2.2 EXPERIMENTAL RESULTS
We evaluate our methods and previous methods on the proposed benchmark. The results in Table 15 illustrate that our methods show significant improvement previous methods. In addition, the RW strategy appears to be more insensitive to the performance of the original method than the RP strategy. Specifically, the RW strategy is able to bring substantial growth when the original method performs particularly poorly, but the RP strategy is unable to do so.
OOD detection, compared with
B FURTHER ANALYSIS
B. PROOF OF THEBREM 1I
Theorem 1. If Assumption holds, then Pympxoo (2/xx Pym (v), for any Vin
Proof. Using Assumption the second equation, we have
P Py (y/xx) P(yin y Xout xx) P(yin PPyin= y)P =x) Pyym P(Xout= = xx)
P(Xout = =x)
B.2 ALTERNATIVE CHOICE FOR ID-CLASS-PRIOR DISTRIBUTION
When the labels of the training datasct not available, model as an alternative to simulate empirical ID-class-price prior distribution. Specifically, for each sample x; in training dataset, the prediction made by the model is softmax(fa (x:)). Thus, have Fy 1/N V (ofttax((f((x))) Ei=o
the predictions made by the
We also conduct experiments to confirm the assumption, and the results are shown in Table I6 Noticeably, OOD detection performances with kinds of ID-class-prior distribution are similar.
https://gi
fiation
22

---

Under review as a conference paper at ICLR 2023
Table 16: Performance comparison of different ID-class-prios distribution acquisition methods. All methods are trained on ImageNet-LT-a8 dataset with ResNer50.
Method
ID-class-grior Distribution
INotumlist
SUN
Places
Texture
Average
AUROC1
FPR95)
AUROCT
FPR95]
AUTROCT
FFR055
AUROCT
FPR95,
AUROCCT
FPRSS]
RP+MSP
Model Prediction
Label
63.36
63.34
96.09
96.05
68.19
68.19
90.44
90.40
67.43
67.44
90.61
90..8
42.90
42.91
96.17
98.17
60.18
00.45
93.84
92.80
RW+ODIN
Medel Predicrion
53.21
Fieegy
78.05
Data Label
90.56
42.09
80.45
73.34
76.27
S0.48
64.79
85.90
79.02
71.10
RP+GrddNorm
Model Prediction
Data Labe!
89.35
89.89
50.03
49.89
80.75
80.73
64.52
64.33
74.69
74.70
78.18
78.11
63.36
63.31
77,33
77.70
77.17
77.14
67.51
67.60
B.3 PROOF THEOREM 2
Proof of Theorem
According to the definition of softmax function, is clear that
(fx and (ff) 2 for V 1,... K.
Existence. we assume that for all i 1,... K, softmax, (fo (f (x)) < K then
Σ (fflm) 1, which is conftict with (fo
Therefore, there is at least one such that (fftmax(ffexx))) 2 K which implies that
min SMsP(fe, fe(x)
K
Note that when softmax(fe(x))) (x)) = SMSP SMSP(fe, (fe, = t. which implies that there exists fe arg (x) SMSP(fe) x) such that
" (ff
Uniqueness. If there (x) arg min SMSP x) such that softmax(f5 (x)) & u, is clear that
Σ softmax, (fa (x)) <
which is conflict with Σ -i (fo = Therefore, softmax(f5(x))
Combining the results in existence and uniqueness, we have completed this proof.
B.4 DISCUSSION ABOUT RP+ODIN
ODIN (Liang et al. 2018) an enhanced version MSP, whose main improvement is the introduction of temperature scaling strategy. The temperature parameter T smoothes the prediction distribution of the softmax function and thus making the prediction sparser and more similar the uniform distribution.
exp (fx(x)/T)
SODIN fe. x) = max プ5=1 exp (x)(x)
(16)
Since ODIN maps the prediction distribution of the softmax layer another distribution space while we need to measure the similarity between the class-prior distribution and the model-predicted distribution, need to the same mapping method to deal with the class-prior distribution Pyim = [Pi P2,
.2., ..., pc]. as follows:
(PI/T) ES exp (p; E exp (p:
exp (p2/T)
exp (pc/PTT) E exp (D:/T)
Pyn =
(17)
23

---

Under review as a conference paper at ICLR 2023
Then, use this new class class-prior distribution Pyin in to modify ODIN with RP strategy Eq. 19
exp(fi(x)//)) exp (f(()/T)
exp (x)/T)
hx = 1 exp (x/()) ΣS exp (f, (x)
(18)
Σjn exp (x)(x)
SRP+ODIN (fe, x) = max [ yi.)
(19)
When we follow the default setting 1000 ODIN, we notice that P will be quite close to the uniform distribution, where each element close to 1/K. Thus, Eq. (19) can be regarded he (x) minus a constant.
B.5
DISCUSSION ABOUT M IN PARETO DISTRIBUTION
For each class Ts, the sample number is
am a+I
Y/ N x p(z:) N
(20)
2
where tail index, is constant and N is the sample number of the ImageNet-IK dataset.
After sampling, the new data distribution for each class is
am N
Vi
T 1
p(y))
(21)
Σ 1/i
ΣK
EE
am 2.41
2
2at1
Li=1
Obviously, the value of m do not affect the imbalance degree of sampled datasets. Thus, keep m unchanged.
B.6 DISCUSSION ABOUT FEATURE-BASED METHOSS
Feature- based methods, like KNN (Sun et al. 2022 need a training set to generate class prototypes, ie., an average feature vector for each category. Under class- maas-immalanceed situations, prototypes of tailed classes would be more unreliable than the majority due to the limitation training samples. We think using ID-class-prior distribution to reweight features may be an effective way to solve the imbalanced problem in feature space.
B.7 DISCUSSION ABOUT CLASS-DEPENDER THRESHOLDING
The paper (Guarrera et al. 2022 designs an optimization for threshold selection rather than designing an OOD score. Class-dependent thresholding is designed for Ptrain * Ptest and not for data imbalance issues. This approach only affects the precision and recall metrics the deployment stage of the OOD detector, not the AUROC and FPR95 metrics that our paper focuses on. This can be seen in Table in Guarrera et al. 2022
B.8 DISCUSSION ABOUT POSSIBILITY FOR RP+MSP.
In order discuss about the possibility for aligning the minimizer of the score function with the class priors, we conduct experiments for results in the helow table. Our experiments show that (ffftmmxx(((x))))) (i)) also performs
(f (i)) and show the corresponding
very well.
Table 17: OOD detection performances on ImageNet-LT-a8 dataset.
Method
iNaturalist
SUN
Places
Textures
Average
AUROC
FPR95
AUROC
AUROC
AUROC
AUROC
81.76
69.75
57.80
94.80
$4.90
94.68
$2.01
88.97
61.62
87.05
24

---

Under review as a conference paper at ICLR 2023
C DETAILED RELATED WORKS
OOD Detection. OOD detection is crucial problem for reliably deploying machine learning models into real-world scenarios. OOD detection can be divided into two categories according to whether the classifier will be re-trained for OOD detection or not.
1) Inference-time/pone hoc OOD Detection: Some methods (Huang et al. 2021 Liang al. 2018
Liu et al. 2020, Hendrycks & Gimpel 2017: Lee et al. S018b]|um et al. 2021) focus on designing OOD score functions for OOD detection in the inference time and are easy to use without changing the model's parameters. This property is important for deploying OOD detection methods in real- world scenarios where the cost of re-training is prohibitively expensive and time-consuming. MSP (Hendrycks & Gimpel (017) directly takes the maximum value of the model's prediction as the OOD) score function. Based on MSP, ODIN (Liang et al. 2018) uses temperature scaling strategy and input perturbation to improve OOD detection performance. Moreover, Liu et al. (2020) and Wang al. (2021a) propose to replace the softmax function with the energy functions for OOD detection. Recently, GradNorm (Huang et al., (021) uses the similarity of the model- predicted probability distribution and the uniform distribution to improve OOD detection and achieve state-of-the-an performance. In this paper, we mainly work on the inference-time OOD detection methods and aim at improving the generalizability of OOD detection in real-world scenarios.
2) Training-timm OOD Detection: Other methods (Hsu et al. 2020 Hein et al. 2019 Bitterwolf (ct al., 22200) Wang et al.) 202 lb) will complete ID tasks and OOD detection simultaneously in the training time. [itterwoll et al. (020) uses adversarial learning process OOD data in training time and make the model predict lower confidence scores for them. Wang et al. 2021b) generates pseudo OOD data by adversarial learning to re-training K+1 model for OOD detection. These methods usually require auxiliary OOD data available in the training process. Thus, the model will be affected by both ID data and OÔD) data. It is important for these method to explore inherent trade-off (Liu
iet al. 2019 Vaze et al.
2022
Yang et al. 2021) between ID tasks and OOD detection.
The paper (Wang al. 2022 training-time OOD detection and uses OOD data to train the model. After being finetuned, the model can deal with the imbalanced issue and OOD problem. The problem similar our paper, but the setting is completely different with our paper (inference-tim) OOD detection). In our paper, we do not change any parameters of the model and design methods to deal with the imbalanced issue on OOD detection. Note that our work and this work are not comparable due to the different problem settings.
Open Set Recognition. open set recognition, machine learning models (Huang & 2021 Lee et al. 2018al Sal |eerera & Patel, 2019) Perera et al.) 2020 Shalev et al. 2018 Radford et al. 2021
2021) are required to both correctly classify the known data (ID) from the closed set and detect unknown data (OOD) from the open set. Some works (Lee et al., 201 Sa Huang & Li 2021) use the information in the label space for OOD detection, and they divide the large semantic space into multiple levels for models to easily understand. Perera & Patel 2019) designs two parallel networks training on different dataset and use the membership loss to encourage high activations for ID data while reducing activations for OOD data. Perera et al. (2020) uses self-supervision and data augmentation to improve the network's ' ability to detect OOD data. Input images are augmented with the representation obtained from a generative model. In this paper, we consider a more complex open set, large scale and imbalanced, to achieve OOD detetcion.
Fort et al.
D CODES
import torch
import numpy as np
from autograu import Variable
def MSP (data_loader, m ood_scores []
m = torch. nn. Softmax (dim=-1), cuda ()
for b, (x, y) in enumerate (data_ loader)
25

---

Under review as a conference paper at ICLR 2023
with torch. no ()
x - x.cuda ()
logits, model_fozwarr (model, x)
softmax softmax_output m (Logits)
ood_score, torch (softmax_output,, dim--1)
ood ood_scores extend (ood_score.data)
return sod_es
def MS?_RP
model, ID_Prior):
ood_acores []
m torch. nn. Softmax (dim=-1) cuda ()
for b, (x, y) enumerate (dada_loader)
with torch. norcad))_grad()=
x cudaa
logits, - model_forward (model, x)
softmax cettmax_output m (logits)
RP strategy
sim softmax_output
Priir
ood score, torch. (sim, dim=-1)
ood scores extend (ood score.
return ood_
def ODIN (data_ _oader, model, epsilon, temper)
criterion corch....croeEnntoo.... Crosssnttrryyoosssssssss....ssse () cuaa
ood acores []
for b, (x, y) enumerate (data_loader):
Variable (x. c)da requires sraaurree_srdd-rreeee
outputs,
foaeellfowwrdddooe (model, x)
maxIndexTemp = argmax (
data (pata.cp()
numpy (), axis=1)
outputs = octputs temper
labels Variable (torch. LongTensor (maxIndexTemp) cuda ())
loss criterion (outputs, labels)
loss backward
Normalizing the gradient binary in (0,
gradient torch. ge (x. grad. data, 0)
gradient = (gradient #loat 0.5) + Adding small perturbations to images
tempIoputs torch add (x. data, -opsilon, gradient)
outputs, model_forwart (model, Variable (tempinputs))
outputs outputs temper
Calculating the confidence after adding perturbations
anoutputs outputs data cpu
anOutputs nnOutputs mnoutputs.numpy/ numpy() ()
anOutputs nnOutputs np max (nnOutputs, axis=1, cropdims-Truq
nnOutputs rp.exxp (nnOutputs) / np. sum (np.ex))))) exp (nnOutputs),
4 axis-1, keepdi ma ins-True)
ood vod_scores. www.sooo.........eeoooooo (np.max max (nnOutputs, axis=1))
return cod_scores
def ODIN_RW (data_loada) model, epsilon, temper, ID_Prior): criterion torch. nn. ros.ssooooooooyossssss () cuda
ood scores []
26

---

Under review as a conference paper at ICLR 2023
= torch. nn. wn..nn...oooo.... Softmax (dim=-1) cuda ()
for b, (x, y) in enumerate (data_loader).
x Variable (x, c)da requires__rac-Trus outputs, model @odee__oowarrdmoooee x)
aoftmax_output m (outputs)
softmax_output softmax_ourput..dar
softmax_output softmax_outpptt numpy
maxIndexTemp = ao.aargaa (outppuss data. numpy (), axis=1)
outputs - octputs temper
labels Variable (torch. LongTensor (maxIndcxTemp) .cdaa ())
loss criterion (outputs, labels)
Loss backward())
Normalizing the gradient binary in (0,
gradient torch. ge (x. grad. data, 0)
gradient = (gradient #loat (1 0.5) + Adding small perturbations to images
tempInputs torch add (x. (x. data, -epsilon, gradient)
outputs, model_forward (model, Variable (tempinput:s)
outputs outputs temper
Calculating the confidence after adding perturbations
anOutputs outputs data.apuu
nnOutputs nnŌutputs (ompup ()
anoutputs nnOutputs np. max (nnOutputs, axis-1, keepdims-Tee -True)
nnOutputs ex.exp (nnOutputs) / np. sum (np. exp (nnOutputs),
axis=1, keepdims-Trus RW strategy
sim -oftmax_output ID_Prior
sim sim. (axis=1) (np linalg (nnOutputs, axis=-1)
mp.linal norm [D_Prior,, axis=-1))
sim expard dims (sim, axis=1)
noutputs sim nnOutputs
ood_ (np. max (nnOutpute, axis=1))
return ood scores
def Energy (data_loader, model, temper):
ood_scores []
for b, (x, y) in enumerate (data loader)
with torch. mo_grad
x c)
logits, model mndel_forward (model, x)
sco_e temper torch. Logsumexp (logits temper, cim=l) ood_scores extend food__scoee data)
return cod_eco
def (dataa__dddaaaaadddddd model, temper, ID_Frior):
ood scores = []
m torch. nn. Softmax (dim=-1) 1).dma cuda ()
for b, (x, y) enumerate (dade_)
with torch (rrch.oo_grad():
euda
logits, - model_forward (model, x)
conf = temper torch. logsumexp (logits / temper, dim=1)
RW strategty
27

---

Under review as a conference paper at ICLR 2023
softmax output (logits)
sim -owfssaaoouteepppaaaaa
ID_Prior
sim sim. sum (1) (torch.norm wtt.:ooootoottaaoorreeeee dim=1)
- torch. Noreh..orm (ID Prior, (ID_Prrr,, dim=l))
ood sod_score conf sim
ood scorec extend (ood_ scod_scoreddda return scores
def GradNorm adata_loader) model, temperature, num_classes)
ood_acores []
logsoftmax torch nn. LogSoftmax (dim=-1) cuda
for b, (x, y) enumerate (data_
inputs Variable cx.c raquires_grad-Tnuu
model. perel.zero_gral
outputs, model forward (model, inputs)
targets = torch. oncs (Laputs.shape(00) num_classes)) ) cuda ()
outputs = outputs temperature
loss torch sum (torch, mean (-targets logso (outputss)
dim=-1))
loss backward ()
layer_ model. head. weight grad. data
ood_acore torch. sum (torch. abs (layer_grad)
vod_scores.appon (ood_score)
return ood scores
def GradNorm_ _ladde, model, temperature, nun_classes, ID_Prior)
ood_scores
Logsoftmax torch. Corch..n.Loggottee..... LogSoottaax cuda
for b, (x, y) in enumerate (dada_loader):
inputs Variable cuda (), saquires_grad-Truee model zeeo_gradd outputs,
inputs)
RP strategy
target torch. tensor (10_Prior)
targets target. unsqueeze (dim=0) (udda
outputs outputs temperature
loss = torch.sum sum(-targete Logsoftmax (outputs), dim=-1)
loss. backward)
layor_grad model. fc. weight. Beligh.ggrr...mm data
cod_acore corch.sun (torch. abs (layer_grad))
vod_scores.appett append (cod__core))
return ood_scores
def Cosine Similarite Loader, model, (D_Prior): ood_scores []
m = torch. nn. Softmax cuda
for b, (x, y) enumerate (data_loader)-
with torch roreh.ro__rra())
x x. cuda
logits, model_ [orwed x)
softmax_outpur m (logits)
sim = -softmax_outpur
ID_Frior
sim = sim.sum / (torch. worrch.normmeetee www.mmoottta..outtoooooooooooooe output, dim=l)
+ norm Priio,, dim=1))
28

---

Under review as a conference paper at ICLR 2023
ood score wim.unsqueeze (1)
ood_scores.ext extend (ood_score. return ood_scores
29


Reviews:
[{"review_id": "y0Dt8ZJPHe", "reviewer": "Reviewer_G6VC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper modifies OOD detection methods which assume a balanced class distribution such that they are applicable to distributions with unbalanced classes.", "summary_of_the_review": "The paper contains interesting approaches to OOD detection with a class-imbalanced in-distribution. However, as detailed above, particularly its methodology currently contains several problematic issues.", "strength_and_weaknesses": "Strengths:\n\n- The paper tackles the important problem of transferring OOD detection methods that are known to work for class-balanced distributions to unbalanced ones.\n\n- The experimental results show good improvements compared to the presented implementations of the baseline methods.\n\nWeaknesses and open questions:\n\n- Assumption 1: What does $P_{X^{out} Y^{in}}$ mean exactly? This is central for the paper and as I read it it is not explained.\n\n- It is not clear why causal graphs are necessary to understand the independence of certain variables.\n\n- The claim that \"researchers assume that the pre-trained model f can simulate the ID conditional distribution\" should be substantiated. The methods proposed by this paper are motivated by achieving this simulation, so it is important to exhibit why it is necessary or desired.\n\n- The presented theorems are trivial.\n\n- Eq. (7) should be justified, as this is not the only possibility for aligning the minimizer of the score function with the class priors. For example, $max_i softmax_i f(x) / P_{Y^{in}}(i)$ would have the same minimizer but different behaviour away from the minimum.\n\n- For the reweighting strategy of Eq. (12), having a minus sign in front of the base score seems extremely unintuitive. That the cosine should get a minus sign is clear, but why does it also affect $S_{Method}$?\n\n- \"the system with a more concentrated probability distribution has lower energy, while the system with a more divergent probability distribution (more similar to the uniform distribution) has higher energy (LeCun et al., 2006). Thus, the energy of ID data is smaller than OOD data.\" -- This is not clear for the logits of a classifier trained with cross-entropy loss. The cited slides (LeCun et al., 2006) mainly describe models where the energy is regularized. How can those be connected with inference-time OOD detection?\n\n- The sign of Eq. (14) is wrong, since the *negative* energy is the base OOD detection score $S_{Method}$. This means that $-S_{Method}(f,x) = -T·\\log \\sum_k e^{f_i(x)/T}$. What does this mean that the AUROCs in Table 1 for RW+Energy?\n\n- In Figure 4, I would suggest showing only one of the similar metrics AUROC and FPR, and using the space to show all datasets.\n\n- Results for the balanced standard datasets should definitely be shown. Both for assessing the baselines compared to other publications and for checking how the proposed method (should only affect RW) impacts performance.\n\n- I think this paper could benefit from an analysis how the base methods which do not account for class imbalance fail. E.g. is it because ID samples from rare classes are being rejected too often, or because OOD samples receive imbalanced predictions that are close to the class priors rather than to the uniform distribution? The \"Analysis of Detection Results on Different ID Classes\" answers this partially, but it focuses on the improvements rather than on nature of the original problems.\n\n- The following previous works investigate class-dependent thresholding and should be discussed and potentially compared to.\n  - [Guarrera et al. 2022: Class-wise Thresholding for Robust Out-of-Distribution Detection]\n  - [Wang et al. 2022: Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition]\n\n- It is not clear to me whether the compared and improved methods are state of the art in the regarded setting as the paper claims. Many publications have worked with Vision Transformers pretrained on larger datasets -- which can be considered a different setting for OOD detection in particular as pointed out by [Hendrycks et al. 2022: Scaling Out-of-Distribution Detection for Real-World Settings] -- and achieve much better performance. This as well as other related works that work only with ID data should be discussed and compared to in order to provide context. It could also help emphasize that all methods that ignore potentially imbalanced classes need to be reconsidered.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S2pbtqy9DQ", "reviewer": "Reviewer_uGHp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a very simple solution to an important problem of OOD detection. The authors study the effect of class imbalance in the training set. The idea is that if there is class imbalance, the current techniques that perform inference time OOD detection will fail as they assume uniform class probably distribution. To remedy this, the authors propose a simple fix in which: (1) If the methods explicitly use uniform prior, they replace it with training class probability distribution, or (2) For the methods that do not use class probabilities, they have a simple reweighting mechanism. Strong experimental results are obtained.", "summary_of_the_review": "I think the paper proposes a simple solution, which they have demonstrated to be quite effective. Overall, I like the paper.", "strength_and_weaknesses": "The method is really simple, yet elegant. The authors clearly describe the motivation and present the fix they propose. It is well explained.\n\nOne of the most appealing features about the method was its ability to work with many other techniques. The authors show experiments of four such methods - MSP, ODIN, Energy and GradNorm. The method improves performance across board on all experiments.\n\nI also liked how extensive ablations were performed.\n\nOne question I had was what happens when you use your method on balanced dataset? I think it should go back to the original formulation and there should be no loss in performance compared to baseline. Does that happen?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MI4R-cHC6P", "reviewer": "Reviewer_vuMZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of OoD detection in the case where the distribution of training classes is not-uniform. In other words, the authors tackle OoD detection in the case of long-tail image recognition. The authors first make the observation that a number of existing OoD methods (either implicitly or explicitly) make the assumption that the training set is balanced, and then propose a simple re-weighting strategies to address this. \n\nOn an ImageNet evaluation, with varying degrees of long-tailness, the authors show their re-weighting strategies boost the performance of a number of baselines (including maximum softmax probability and the more recent GradNorm). Specifically, the re-weighting takes into account the empirical distribution of training classes.\n\nThe authors theoretically analyze the problem through the lens of causality, by presenting different candidate causal graphs which model the data generation process. ", "summary_of_the_review": "Overall, I think this is an interesting paper which proposes a simple solution to an important problem, which nonetheless demonstrates substantial boosts on a large-scale benchmark. Significant extra exploration is undertaken to provide additional details to explain the method. My main issue is that prior work in this space is not discussed or compared to, and hence I am hesitant to recommend a strong acceptance.\n\n**UPDATE AFTER AUTHOR RESPONSE:**\n\nAfter reading the other reviews and the (especially) the authors' responses, I have decided to upgrade my rating. Overall, I like this paper because it provides strong empirical results with a very simple method, to tackle an important and realistic problem. \n\nHowever, I would strongly suggest the following improvements to the paper:\n1) The authors should discuss [1]. Though it is a valid distinction that [1] trains with OoD data, it is not a completely different task. For instance, Outlier Exposure is still often referred to inference-only OoD detection papers, with reasonable discussion given on why inference-only methods operate in a different setting.\n2) I concur with the other reviewers that the theoretical causal framework does not add much to the paper's narrative, but rather distracts from the main findings.\n\n[1] Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition, Wang et al., ICML 22", "strength_and_weaknesses": "Strengths:\n* The authors tackle an interesting problem, OoD detection for long-tailed training data. It is interesting because real-world data is rarely balanced and, as the authors observe, the performance of existing methods drops substantially in this regime. \n* The re-weighting strategies which the authors propose are intuitive and easy to implement. It makes sense to incorporate the empirical class prior of the in distribution data to the OoD scoring rule, and the authors show an easy way in which it can be done. Despite this, they markedly improve the performance of a number of baselines. \n* The authors provide a detailed ablation, controlling for model architecture, different amounts of data long-tallness, as well as a useful breakdown of the OoD performance when only the 'Head','Middle' or 'Tail' classes are taken from the closed-set categories. Intuitively, they find that the biggest problem is distinguishing 'Tail' closed-set categories from open-set classes.\n\nWeaknesses:\n* The main weakness for me is that the authors don't seem to mention or compare to [1], which is published work on the problem of OoD detection for long-tail recognition. This method also reports substantial boosts over baselines (though on small-scale OoD benchmarks). The authors should discuss and compare to this method if possible, or else explain why it cannot be done.\n* While I appreciate the efforts to impose a theoretical framework on the problem, I believe it makes the paper more confusing than clarified. I do not think that the proofs and arguments derived from the causal framework add anything to the argument which could not be more clearly expressed through a few lines describing the intuitions.\n* (Minor): There are a number of typos throughout the paper. For example, P_xout * P_yin = P_xout * P_yin in Assumption 1. \n\n[1] Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition, Wang et al., ICML 22", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "APmAaGapqF", "reviewer": "Reviewer_dyEV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the importance of ID class prior for OOD detection; two methods are proposed, replace and reweight, which can enhance the current logit-space OOD detection scores and improve the performance a lot. Both medical and theatrical analyses are conducted and provided. The experiments on benchmark datasets support their motivation.", "summary_of_the_review": "The idea of this paper is new and looks novel; the analysis of this paper supports their motivation. They also provide a good theatrical analysis of their discovery and idea. In general, I think this is a good finding for OOD detection.", "strength_and_weaknesses": "Another branch of OOD detection is feature-space detection scores, such as KNN. I think a discussion of the potential extension of their method on feature space may be good since logit-space scores may not well fit some real-world applications, even since not all applications have a good assumption.\n\nFig 2 shows the performance of the balanced and imbalanced dataset, which, if an OOD set is close to the tailed categories of ID data, such as the OOD set highly close to the minor categories, and what's the performance of their method? I think reporting some per-category or hard case OOD is also interesting instead of reporting the full dataset score.\n\nAssumption 1 seems good for natural images, while for those images with a huge domain gap between natural images, not the MINIST vs Fasion-MINIST one, how does assumption 1 fit for this case?\n\nAs for model selection, they only selected the ResNet for verification, it is reasonable for the standard setting, but considering that there are so many pretrained models existing, VIT, CLIP which was pretrained on large-scale data, reporting the scores for pretrained model with supervised or zero-shot setting is also necessary for the extension of this method since most of the recent OOD tasks are suing pretrained models.\n\nWhat's the performance on other scores, such as Maxlogit ? \n\nThe performance of AUROC and FPR95 are pretty low on Textures, which is interesting; even though Detection methods with the help of their method on other more natural-image-like datasets look good, textures still fail to perform OOD detection.\n\nThose selected datasets are pretty standard, and I wonder what the performance of zero-shot performance is with pretrained models (i.e, CLIP) when we use this method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject