A Closer Look at Generalized BH Algorithm for Out-of-Distribution Detection
Authors: Xinsong Ma
Jie Wu
Weiwei Liu
Venue: ICML 2025 poster (2025)
Keywords: Generalized BH algorithm
Out-of-distribution detection
Primary area: general_machine_learning

Abstract: Out-of-distribution (OOD) detection is a crucial task in reliable and safety-critical applications. Previous studies primarily focus on developing score functions while neglecting the design of decision rules based on these scores. A recent work (Ma et al., 2024)  is the first to highlight this issue and proposes the generalized BH (g-BH) algorithm to address it. The g-BH algorithm relies on empirical p-values, with the calibrated set playing a central role in their computation. However, the impact of calibrated set on the performance of  g-BH algorithm has not been thoroughly investigated. This paper aims to uncover the underlying mechanisms between them. Theoretically, we demonstrate that conditional expectation of true positive rate (TPR) on calibrated set for the g-BH algorithm follows a beta distribution, which depends on the prescribed level and  size of calibrated set. This indicates that a small calibrated set tends to degrade the performance of g-BH algorithm. To address the limitation of g-BH algorithm on small calibrated set, we propose a novel ensemble g-BH (eg-BH) algorithm which integrates various empirical p-values for making decisions. Finally, extensive experimental results validate the effectiveness of our theoretical findings and demonstrate the superiority of our method over g-BH algorithm on small calibrated set.

A Closer Look at Generalized BH Algorithm for Out-of-Distribution Detection
Xinsong Ma
Jie Wu
Weiwei Liu'
Abstract
distribution the training data (Hendrycks et al., 2022; Djurisic et al., 2023). In real-world applications, models trained on in-distribution (ID) data from specific domain often encounter OOD data from unseen distributions dur- ing deployment. This discrepancy between the training and deployment leads to poor model performance (Liang et al., 2018; Sastry & Oore, 2020; Kaur al., 2022). The im- portance of OOD detection has grown with the increasing reliance on deep learning models in safety-critical appli- cations, such as autonomous driving (Li et al., 2022) and medical diagnosis (Frolova et al., 2022).
Out-of-distribution (OOD) detection a crucial task in reliable and safety-critical applications. Previous studies primarily focus on developing score functions while neglecting the design of de- cision rules based on these scores. A recent work (Ma et al., 2024) is the first to highlight this issue and proposes the generalized BH (g-BH) algo- rithm to address it. The g-BH algorithm relies on empirical p-values, with the calibrated set playing a central role in their computation. However, the impact of calibrated set on the performance of g-BH algorithm has not been thoroughly investi- gated. This paper aims to uncover the underlying mechanisms between them. Theoretically, we demonstrate that conditional expectation of true positive rate (TPR) on calibrated set for the g- BH algorithm follows a beta distribution, which depends on the prescribed level and size of cali- brated set. This indicates that small calibrated set tends to degrade the performance of g-BH algorithm. To address the limitation of g-BH algo- rithm on small calibrated set, we propose a novel ensemble g-BH (eg-BH) algorithm which inte- grates various empirical p-values for making deci- sions. Finally, extensive experimental results vali- date the effectiveness of theoretical findings and demonstrate the superiority of method over g- g-BH algorithm on small calibrated set.
Numerous studies have proposed various methods to address the OOD detection problem (Liu et al., 2020; 2023; Regmi et al., 2024: Lu et al., 2024). These methods mainly focus on designing the score functions which enables learn critical discriminative information in training data. A recent work (Ma et al., 2024) is the first to point out that existing OOD detection methods neglect the systematic study of the decision rule based on the score functions, and propose a novel generalized BH (g-BH) algorithm to tackle this problem. The g-BH algorithm establishes a connection between score functions and multiple hypothesis testing framework through the empirical p-values. The calibrated set is crucial for computing empirical p-values and thus has profound impact on the detection performance of the g- BH algorithm. However, to the best of our knowledge, the impact of the calibrated set on the performance of the g-BH algorithm remains unexplored. This paper aims address the above issue.
Intuitively, larger calibration set enhances the performance of the g-BH algorithm. Our experimental results in Figure confirm this conjecture: as the size of calibrated set in- creases, both the TPR and F1-score monotonically increase. Theoretically, we demonstrate that the TPR expectation con- ditional on calibrated set for the g-BH algorithm follows a beta distribution, with its shape parameters determined by the prescribed significance level and the size of calibrated set. This shows that a smaller calibrated set tends to de- grade the detection performance of the g-BH algorithm. To address the limitation of the g-BH algorithm on small cal- ibrated set, we propose a novel ensemble g-BH (eg-BH) algorithm which integrates multiple empirical p-values for decision-main making. Moreover, we extend the theoretical re- sults the g-BH algorithm from (Ma et al., 2024) and
1. Introduction
nut-of-Distributii (OOD) detection is critical task in ma- chine learning and computer vision (Hendrycks & Gimpel, 2017; Liu et al., 2020). addresses the challenge of deter- mining whether a given input sample belongs to the same
'Shooo of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence and Hubei Key Laboratory of Multimedia and Network Communi- cation Engineering, Wuhan University, Wuhan, China. Correspon- dence to: Weiwei Liu <liuweiwei863 gmail.com>.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
TPR.
55
53
40
5:0 of ca Brotnd IN:
Sias eteaeffllle bruled sut
(a) SVHN
(b) Place365
(c) TinyImageNet
Figure OOD detection performance of g-BH algorithm with varying size calibrated set. The score function the Energy (Liu al., 2020). The x-axis corresponds to the size of the calibrated set, and the y-axis represents the values of metrics.
demonstrate that the eg-BH algorithm controls the false discovery rate (FDR) for p-values without clear structural dependence.
may originate from unseen distributions, where the corre- sponding label space may be disjoint from V. These OOD samples should be identified and excluded from prediction.
Finally, we conduct extensive experiments to verify the theoretical results on the conditional expectation of TPR and the effectiveness of the eg-BH algorithm. Experimental results demonstrate the superiority of our method over the g-BH algorithm small calibrated set.
The objective of OOD detection is to identify OOD exam- ples in the testing set. In prior work, the OOD detection task is formulated as a binary decision problem:
sf) { ID, LOOD, if if <
<(x)
(1)
We summarize our core contributions as follows:
where s(.) is the score function and the threshold empirically selected so that the ture positive rate (TPR) on ID validation set is 95% before testing (Sun et al., 2022; Wei et al., 2022).
We empirically find that a larger calibrated set improves the performance of the g-BH algorithm, whereas smaller calibrated set adversely affects its performance.
We theoretically demonstrate that the TPR expecta- tion conditional on the calibrated set follows a beta distribution, with its shape parameters determined by the prescribed significance level and the size of the calibrated set, which supports our findings.
Prior work on OOD detection has primarily focused on de- signing powerful score functions to capture discriminative information in ID data (Hendrycks & Gimpel, 2017; Liu et al., 2020; Djurisic et al., 2023; Liu et al., 2023). However, Ma al. (2024) highlight that these studies lacks system- atic research on the decision rule on the score functions. Morcover, the decision rule in Eq (1) is empirical and lacks theoretical guarantee for its outputs. Different from the pre- vious studies, Ma et al. (2024) studies the OOD detection problem from the perspective of multiple hypothesis testing, and propose the g-BH algorithm to tackle it.
To address the limitation of the g-BH algorithm on the small calibrated set, we propose a novel eg-BH algo- rithm that integrates multiple empirical p-values for decision bocciion-mmaiin... Besides, our theoretical results pro- vide a statistical guarantee for the integrated p-values in our eg-BH algorithm.
3. Multiple Hypothesis Testing Framework for OOD Detection
Extensive experimental results demonstrate the superi- ority of our method over the g- g-BH algorithm on small calibrated set.
We first introduce the hypothesis testing framework for OOD detection in Ma al. (2024). For mathematical convenience, we follow the notations of Ma et al. (2024). Given a testing set Trast {Xfest Xttest ...., X..,, X }. For
2. Background
We donote by X' C Rdd the feature space and y the label space with unknown joint dis- tribution P. and X' has marginal distribution Dz--
1=1, = n. the OOD detection task is formulated as the following multiple hypothesis testing problem:
21,2,3,.... .., K}
H1 :lest Dz. Hia Xtest ~ Dz
During the prediction phase, it is typically assumed that the testing data are drawn from the same distribution Dz as the training data. However, in practical applications, test inputs
(2)
H : Xlest Dz. Hm xlest Dz
2

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
where H4D0 and Haj are called null hypothesis and alterna- tive hypothesis, respectively. Then, if Hgo is rejected, we declare that X is OOD.
Ma et al. (2024) demonstrate that if p-values independent or positive regression dependence subset (PRDS), the g- BH algorithm controls the FDR at prescribed level a. FDR is defined as
In statistics, the decision to accept or reject the null hypoth- esis is made based on the concept of the p-value, which is generally defined as follows:
| n Hol
FDR E
max 1, [R]]]
Definition 3.1. (p-value (Casella & Berger, 2002)] Given sample X 4, A statistic p(X) is called p-value correspond- ing to the null hypothesis Ho. if p())
where R. is the set of indices of the rejected null hypotheses and Hg is the set of indices for the true null hypotheses. (Ma et al., 2024) demonstrates that the g-BH algorithm can control the FDR at prescribed level if p-values are mutually independent or satisfy the positive regression dependence on subset (PRDS) condition (Benjamini & Yekutieli, 2001).
satisfies
Plp(X)= S Ho) S
(3)
for every 0 1.
If the statistic p(X) follows the uniform distribution on (0, 1) under the null hypothesis, it is a valid p-value. A small p-value typically provides strong evidence against the null hypothesis. It is noteworthy that the p-value has clear statistical interpretation. For example, if the p-value of a OOD lesting example Xtest is 0.01, this implies that, for any subsequent testing example Xj the probability that Xjett is more similar to OOD data than X/eet 0.01, In other words, it is highly unlikely find an example less similar to the OOD data than Xteat, Hence, it provides strong evidence that X/est is OOD.
4. Impact of Calibrated Set on Generalized BH Algorithm
In most multiple hypothesis testing literature (Benjamini & Hochberg. 1995; Benjamini & Yekutieli, 2001; Blanchard &
Roquain, 2008; Delattre & Roquain, 2015; Cao et al., 2022), the p-values or the distribution of the testing statistic are assumed be known. Denote by F()) the cumulative dis- tribution function of s(X) where s(-) is the score function and X ~ Dz. Then, for a given example Xtest, its p-value can be expressed as
Remark 3.2. In statistics, the following terminology char- acterizes the distribution of null p-values: if P PP)(()= s E H|Ho] = f, the p-value p(X) is called exact or uniform; if P(p(X)= < Ho) < 1, p(X) is called conservative. Com- pared to an exact p-value, a conservative one tends to under- state the evidence against the null Hypothesis.
p(X ((tttt) = PXxDD (s(X) S s(Xteat))
= =((((X(
(5)
According to the Definition 2, under the Ho (Xtest is the ID data), we have
P (F (asss)) 5x) =P) = S F (x))
Based on the work (Benjamini & Hochberg, 1995), Ma et al. (2024) propose the B- algorithm to tackle the OOD detection problem. Define two function classes:
=F (z)) = I,
where F-1(.) is the inverse function of F(.). Therefore, the random variable ((x(X(ea()) follows the uniform distribu- tion on (0, 1). namely, p(Xtestt) is a valid p-value and is exact. Obviously, small score s(xtest) results in small p-value, which aligns with the classical setting of OOD detection in (1) and the interpretation of the p-value.
0, S f(x) dr 1
F = (f(x) : =,(0 ~0.
F2 = (f(z) f+ - f(x) 1}.
where (0) = lim f(x) for x E (0, 1). Based on Fi and F2, the g-BH algorithm is defined as follows:
However, in the context of the OOD detection, we often have little prior information about underlying distribution F(.). Hence, Ma et al. (2024) propose using the empirical p-values in the g-BH algorithm, which is a nonparametric estimation method for the p-value p(X p(Xtes)). Given cal- ibrated set Teal {X ..sal ..... Xcal) consisting of the ID examples, for testing example Xlest, the empirical v-vulue p; corresponding to null hypothesis Hap is defined as
Definition 3.3 (g-BH algorithm (Ma al., 2024)). Given the null hy- potheses Hi,0 H2 .... Hn. let Pıt) be the i-th order statistics from the smallest to the largest. For pre-specified level a E (0, 1), define
the p-values P1. . Pn corresponding
G-BBH = -maxfic E [n]: f(p() s i _a].
(4)
where ʃ()) E FFF. Then, the null hypothesis H Ho is rejected if i 1 B-B-
E ((s(X < s s(X(est)) +1
Pt = p ((xteat)
m 1
*A sample means a sequence of examples.
(6)
3

---

A Closer Look at Generalized BH algorithm for Out-of-Distribation Detection
StoFf or
200
20
500
68
65
18
100
68
030
130
08
081
190
385
18
(a) 0 0.05
(b) a 0.08
(c) a 0.12
Figure 2. Density distribution functions of the TPR conditional on calibrated set for the g-BH algorithm with varying level and size of calibrated set.
where s(-) is certain score function. According to Arlot et al. (2010), Eq. (6) satisfies the Definition 3.1.
Theorem 4.1. Given the calibrated set Teal (X X2 ...., Xeal) and the score function s(.) that is continuous. For prescribed level a, the density function of B(TPR|Tool) for the g-BH algorithm can be expressed as
easily verify that empirical p-value in
Note that the g-BH algorithm directly makes the decisions based on the empirical p-values. Therefore, the calibrated set plays critical role in the OOD detection performance of g-BH algorithm. However, to the best of our knowl- edge, there literature that thoroughly investigates the influence of calibrated set the performance of the g-BH algorithm. This paper aims systematically study this important problem.
m 9- x 10 8 11) if <
(x)
otherwise,
where
8 1= +1]
and E distribution Be(m - 1)
E(TPR|TOu) follows the beta
In this paper, we focus on the situation where only ID data available before testing. Thus, we first investigate how
the size m of calibrated set influences the conditional ex- pectation of TPR the calibrated set Teal for the g-BH algorithm: E ((TP)(To())
The proof of the Theorem 4.1 is presented in Appendix A.I. Theorem 4.1 indicates that the pre-specified level a and the size of the calibrated set m play pivotal roles in determin- ing the distributional characteristics of the g-BH algorithm. We visualize the probability distribution of [E(TPR|Toa) with different and the size of calibrated set in Figure 2. From Figure 2, we find that if we choose a larger the g-BH algorithm tends to achieve smaller TPR with appre- ciable probability. For example, with the calibration exam- ples m 200, we have F(E(TPR|TTC) for a 0.08 and P(E(TPR(TOg) S 0.85) 0.81 for a 0.12. The reason behind this phenomenon is that larger induces the g-BH algorithm to adopt more aggres- sive decision decision-making strategies. In other words, the g-BH algorithm tends to classify more testing examples as OOD, More importantly, a small calibrated set causes the g-BH al algorithm to achieve poor detection performance in terms of TPR. contrast, a large calibrated set easily ensures high TPR with a high probability. For example, with a 0.05. obtain (TE( Teat) < 0.9) 0.32 for m 200. and (P fore, a large calibrated set significantly improves the perfor- mance of the g-BH algorithm. In Section 6, we also conduct extensive experiments on the real datasets to validate this conclusion.
To derive the distribution characteristics of conditional expectation of TPR, we need the concept of empirical distribution function. Given the calibrated set Teat = (Xtat, X.gal Xeal), ) for any input v, the empirical dis- tribution F(-) of score function s--) on Teal can be ex- pressed
< 0.9) 0.14
1 F(z) = (s((xo))) S s(x))
Σ
1=1
0, s(x) < ((
s(X S szx) s (x(+t))
m
1, if s(x) (X))))).
where & = 2, ... and X(k) is the k-th order statistic
of , X ..... cal from the smallest to the largest. Obviously, we have
0.9) 0.01 for 500. There-
k
F(X
In addition, we denote |-] the floor function. Then, we have
the following theoretical result.

---

A Closer Look at Generalized BH algorithm for Out-of-Distribation Detection
5. Ensemble Generalized BH Algorithm for Small Calibrated Set
Algorithm eg-BH algorithm
1: Input: Training data T. testing set Thest
[Xteat : X2 ...., ...,,,aal prescribed level a E (0, 1).
In Section 4, we demonstrate that large calibrated set tends to improve the detection performance of the g BH algorithm, but a small calibrated set easily leads to a poor results. In practice, we usually separate a portion of data from the training set to serve as the calibrated set. If the training set small, large calibrated set takes up more of the training data and further leads to the underfitting of neural networks. To address this problem, we propose the ensemble g- g-BH (H) algorithm to address this challenges faced by the g-BH algorithm.
2: partition 7 into T1, T2, ,,, TT, and let T:eal = T, and T T,,
for 1 do
Train the score function s(x) Ttrain, denote by s,(-) the score trained on T
end for
6: for i to do
7: for j = 1 to L do
8:
Compute the empirical p-values for testing exam- ple Xteat based on 85(+) and T
Our motivation arises from the practical significance of p- values. small calibrated set leads to reneetrepresentative empirical p-values, which fail capture the distributional characteristics of the ID data. To address this issue, natural approach is to generate multiple empirical p-values using the entire training set, thereby fully utilizing the available information. We then integrate these empirical p-values to make decisions.
P (X EXET,aL 15(()) S 85 (Xx +1
(T
end for Integrate the empirical p-values Pr, L., l for x Xtest:
10:
For the given training data T. we first partition into Ti, ......, Tt. Denote T.cal = T; and Tirain T\T where "I is the difference operation. Then, train the score function s(.) based Ttrain For simplicity, we denote s.).) the score trained on Tirain Besides, de- note by |T,cul| the size of T.cal. Hence, for a testing ex- ample tesst, we enable to compute various empirical p- values using trained score function and calibrated set pairs (s)()), (e..),T;aa T. R=1
(n +
wyPiis
11: end for
12: Compute i* = max(i E [m]
f(P() I a. where Pte) is the i-th order statistic from the smallest to the largest for P1.
13: Output: Declare that X() is OOD if S i", and the rests are ID.
p: (x(ttss)) 1(Xeal E T.cal (xeea)) S 8/ (Xtest) |+1
[Toall +
arithmetic mean when g(x) = x; 22(-) is the geometric mean when g = log x; 2(-)) is the harmonic mean when g(x) = 1 defined as
f = 1,2, L. After computing the empirical p-values P1 (X (2 P7 (Xtest) the next problem is how to integrate these empirical p-values. direct approach average them:
x For a random variable X, its a-quantile is
a(X,a)= < x) < a). zER
((Xtest)= Pi (Xtest) + p2 + pr..,a
Clearly, Q(X, 1) is the essential supremum of X. In ad- dition, denote by P the set of all p-values. Suppose that the function i(-)) : [0, 1]_ + [0, oo) is continuous and increasing, we define
Unfortunately, p( Xleet) does not necessarily satisfy the definition of the p-value (Rüschendorff, 1982; Meng, 1994).
To obtain more general method of integrating the p- values, we first introduce universal notion of average (Kol- mogorov & Castelnuovo, 1930): given the p-values p {wi,
P.EP Q"(h, inf (Q(h(p).... ())
where p (pI.... ..L....P. .PL). The following theoretical results provide concise method that integrates the multiple p- values.
(pI, P2, }Pr} and weights
W2, {wi,w2,"
.r.)
where w; and El= wi 1, define
(pp, w) = -1 (wig(()) w2g(p2) +... wLL(PL))
Theorem 5.1. Given the empirical p-values pt, P1, PL. and the function g(x) = x" where K. >> 0, then
where g(-) is a continuous and strictly monotonic function, and g (1) is its inverse function. If w; = +. 2(-) is the
((K 1)(wipf + w2p + wrPpi))
5

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
is a valid p-value. Specifically,
follow the experimental implementation in (Yang al., 2022; Zhang al., 2023a), and our codes based on (Zhang et al., 2023a). More details are found in (Zhang et al., 2023a).
2(1 + p2 + 4...4 Pt.)
L
and
max (pi, m
P2: P2. .PL)
6.2. Impact of Calibrated Set on Generalized BH Algorithm
are the valid p-values.
The proof of Theorem 5.1 is presented in Appendix A.3. According to Theorem 5.1. we can choose appropriate function 9(x) = xd to integrate various empirical p-values for Decision-making The following theorem provide an consideration for the choice x in g(x).
In this experiment, we aims to reveal how the calibrated set influences the detection performance of the g-BH algorithm. We first split the training data equally into two parts. One part is employed to train the neural networks for construct- ing the score function, and the other serves as the largest calibrated set 'M Then, from Tool we extract samples various proportions to construct several relatively smaller calibrated sets, where = {0.2, .... ,1.0.. The experi- mental results of practical metrics based on the Energy (Liu et al., 2020) are presented in Tables and 3. The results based on the MSP (Hendrycks & Gimpel, 2017) are pre- sented in Tables 2 and 4. Because of the space limitation, all experimental results of using MNIST as OOD data presented in Appendix B.
Theorem 5.2. Given the empirical p-values P1. P2. PL x², denote w²
and the function g(ar) max w1, W2,' .WL If wn"= min( 14x 14 Jethe we have
PP (T(()) sa) }=a.
sup P:EP
where h(p) = ((1s + ()(ppp + w2pt + ....... wLPL))
The proof of Theorem 5.2 is presented in Appendix A.4.
Theorem 5.2 indicates that if we choose w* such that 1 a h. the integrated p-value h(p) can w" S. min 1+x 14x be exact, which benefits the improvement of power for the hypothesis testing algorithm. Based on the analysis above, we summarize our proposed method in Algorithm called ensemble g-BH ( algorithm.
From Table 1, calibrated set, the evaluation metrics TPR and Fl-score con- siderably increases, accompanied by a marginal rise in FPR. For example, we use the SVHN as the OOD data, and use the Energy as our score function based on ResNet18. When sampling ratio
we find that with the increase of size for the
0.2, the F1-score, TPR and FPR of g- BH algorithm are 25% 35.41% % and respeeespyy...lll. When r = 0.8, the corresponding F1-score, TPR and FPR are 75. 78%, 61.51 11.51% and 0.31%, 1%, respectively, which leads to the direct improvements of 23.53% and 26. 26.10% for F1- score and TPR, at a negligible cost of 0.26% increase in FPR. Notably, as shown in Tables 2, and 4, this trend is consistent for other socre function MSP, network architec- ture WideResNet and OOD data. Therefore, large calibrated set improves the performance of the g-BH algorithm with- out the dependence on the distribution assumptions of OOD data. The above analysis demonstrates the effectiveness of Theorem 4.1.
6. Experiments
In this section, we aims to verify the effectiveness of Theo- rem 4.1 and the superiority of our proposed eg-BH algorithm over the g-BH algorithm. Our experimental framework is based on Ma et al. (2024) and use the same evaluation met- rics. The experimental results show the superiority of the eg- -g-BH algorithm over the g- B-HH algorithm on small cali- brated set.
6.1. Experimental Settings
Scores. We choose two famous methods MSP(Hendrycks & Gimpel, 2017) and Energy(Liu et al., 2020) the score functions in our method.
6.3. Comparison between g-BH and eg-BH on Small Calibrated Set
In this experiment, we aim to compare the detection perfor- mance between vanilla g- g-BH algorithm and our proposed eg-BH algorithm on the small calibrated set. We first ran- domly divide the training data into L equal parts. For the g- g-BH algorithm, one of these parts is used as the calibrated set. Note that a larger L implies smaller calibrated set. For our proposed method, we directly apply the strategies in the algorithm 1. When L 5, the corresponding experimental results of practical metrics are presented in Tables 5 and 6
Benchmarks. We use CIFAR-10 (Krizhevsky et al., 2009) as ID data, and use C00, ImageNet (Krizhevsky et al., 2017). SVHN (Netzer et al.. 2011), 1), Fashion-MNISS (F-MNIST) (Xiao et al., 2017), Places365 (Zhou al., 2018) and MNIST (Deng, 2012), as OOD data.
Metrics. We use the same practical evaluation metrics as Ma et al. (2024), including TPR, FPR and Fl-score.
Model. The score functions in this paper are based on the R8 and Nete respectively. We mainly
As tables 5 and 6 shown, we observe the FPR of our pro-
6

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
Table Experimental results (%) of practical metrics on CIFAR-10 as data. Energy (Liuu et al., 2020) used as the score function
based the ResNet18. We compare the detection performance of BH algorithm with different sizes of calibrated set.
CIFAR- 100
TinylmageNet TPR
SVHN TPR
Place365
F-MNIST
Ratio
F1
TPR
FPR
F1
FPR
FPR
F1
TPR
FPR
F1
TPR
FPR
0.2
63.63
47.12
0.98
33.06
32.47
0.29
52.25
35.41
0.05
51.84
35.35
0.19
62.94
45.99
0.68
0.3
66.06
49.91
1.20
39.76
34.58
0.35
54.81
37.80
0.05
54.01
37.53
0.24
65.74
47.83
0.97
0.4
67.88
52.07
1.35
43.34
36.19
0.40
58.03
40.95
0.07
56.00
39.31
0.30
68.16
52.79
1.42
0.5
70.19
54.92
1.56
45.27
40.63
0.55
62.27
45.31
0.08
58.11
41.48
0.35
70.95
54.98
1.46
0.6
71.90
57.99
1.61
53.81
43.04
0.67
65.66
49.00
0.10
59.67
42.37
0.44
72.78
57.51
1.69
0.7
74.35
60.54
2.31
57.75
46.94
0.87
73.18
58.05
0.23
61.27
44.92
0.47
75.58
59.25
2.15
0.8
76.16
63.53
3.30
62.05
52.61
1.41
75.78
61.51
0.31
66.15
48.26
0.87
76.52
62.98
2.47
0.9
77.90
64,94
3,95
65.91
53.63
1.49
79.71
82.34
5.42
68.92
56.14
1.11
78.43
67.61
4.79
79.26
69.50
5.87
70.52
64.03
3.51
84.03
86.68
11.64
73.84
71.76
5.36
80.74
70.68
6.51
Table 2. Experimental results (%) of practical metrics on CIFAR 10 as ID data. MSP (Hendrycks & Gimpel, 2017) is used as the score function based the ResNet18. We compare the detection performance of g-BH algorithm with different sizes of calibrated set.
CIFAR- 100
TinyImageNet TPR
SVHN TPR
Place365
F-MNIST
Ratio
F1
TPR
FPR
F1
FPR
F1
FPR
F1
TPR
FPR
F1
TPR
FPR
0.2
62.27
45.62
0.91
48.49
32.47
0.29
51.55
34.76
0.04
50.31
33.81
0.16
62.12
45.36
0.68
0.3
64.89
48.56
1.11
50.08
33.95
0.33
55.67
38.63
0.06
53.42
36.75
0.23
64.89
48.45
0.89
0.4
67.88
52.07
1.35
52.38
36.19
0.40
58.03
40.95
0.07
54.19
37.51
0.25
67.84
51.93
1.17
0.5
70.19
54.92
1.56
53.81
37.64
0.45
61.22
44.21
0.08
57.02
40.37
0.33
70.20
54.83
1.39
0.6
71.81
57.00
1.76
55.64
39.53
0.51
65.66
49.00
0.10
60.16
43.68
0.42
71.81
56.89
1.56
0.7
74.35
60.54
2.31
58.74
42.97
0.67
67.76
51.39
0.12
62.77
46.63
0.53
74.37
60.50
2.20
0.8
75.26
62.03
2.82
62.05
46.94
0.87
71.67
56.10
0..77
65.41
49.92
0.74
75.19
61.76
2.51
0.9
78.49
67.87
5.06
65.42
51.84
1.33
75.78
61.51
0.31
68.82
54.55
1.09
76.98
64.80
3.56
81.28
77.63
13.40
69.63
70.05
6.23
79.71
86.34
11.64
74.26
70.23
5.18
79.58
70.25
6.30
Table 3. Experimental results (%) of practical metries CIFAR-10 10 as ID data. Energy (Liu et al., 2020) is used the score function
based on the WideResNet. We compare the detection performance of g-BH algorithm with different sizes of calibrated set.
CIFAR- TPR
Tiny ImageNet
SVHN TPR
Place365
F-MNIST TPR
Ratio
Fl
FPR
Fl
TPR
FPR
F1
FPR
F1
TPR
FPR
F1
FPR
0.2
63.52
50.19
4.53
46.42
35.22
2.49
38.08
36,97
1.97
48.78
36.41
1.32
64.30
47.85
0.3
65.84
50.88
5.99
48.92
36.49
3.24
52.95
39.86
2.49
52.64
37.58
1.79
65.89
51.21
2.25
0.4
67.22
52.79
7.24
50.15
38.84
4.08
56.71
41.48
3.05
54.88
39.73
2.04
67.19
53.79
2,86
0.5
68.85
56.17
8.06
50.79
40.16
4.99
57.46
43.15
3.41
56.49
40.89
2.65
70.53
55.84
3.59
0.6
70.28
59.06
8.55
53.84
42.55
5.71
59.25
45.89
3.97
58.34
44.25
3.02
73.48
59.39
4.01
0.7
73.43
61.72
9.09
54.75
44.63
6,38
62.94
47.74
4.33
61.75
48.84
3.48
75.59
62.41
4.64
0.8
74.58
64.66
13.27
56.88
48.18
7.45
64.77
50.91
4.82
64.48
53.79
3.99
77.11
64.29
5.29
0.9
77.53
67.18
14.62
60.13
66.09
10.04
65.96
52.76
5.12
66.52
56.68
4.51
79.49
69.49
6.89
78.97
78.73
22.79
61.52
67.75
12.84
67.18
56.57
5.79
70.16
71.41
8.79
83.09
82.23
16.52
Table 4. Experimental results (%) of practical metrics on CIFAR- -00AR-10-A as ID data. MSP (Hendrycks & Gimpel, 2017) is used as the score function based WideResNet. We compare the detection performance g BH algorithm with different sizes calibrated set.
CIFAR-
TinylmageNet TPR
SVHN TPR
Place365
F-MNIST TPR
Ratio
Fl
TPR
FPR
FPR
F1
FPR
F1
TPR
FPR
F1
FPR
0.2
62.65
48.10
5.45
45.95
34.11
2.87
43.45
36.49
2.13
49.95
35.13
1.52
64.30
48.30
1.93
0.3
63.95
49.74
5.82
47.27
35.68
3,06
53.15
38.56
2.51
52.33
37.66
1.72
65.80
50.09
2.15
66.19
52.69
6.52
48.99
37.84
3.33
55.24
40.85
2.71
53.86
39.34
68.04
52.88
67.60
54.74
7.21
49,96
39.12
3.50
56.90
42.82
2.96
55.39
41.05
69.76
55.18
69.91
58.43
8.72
52.21
42.06
3.81
58.45
44.69
3.16
58.34
44.62
72.01
58.31
0.7
70.69
59.79
9.37
53.01
43.26
3.99
60.24
46.97
3.44
60.37
47.35
2.61
74.07
61.43
0.8
73.31
64.87
12.11
55.25
47.07
4.66
61.43
48.49
3.60
64.49
53.90
3.63
75.52
63.91
5.34
0.9
74.66
67.93
14.03
59.25
66.70
11.69
63.33
51.20
4.03
65.23
55.37
3.94
78.27
68.94
7.21
76.76
78.66
26.30
59.56
63.51
14.95
65.50
55.82
5.61
68.14
69.35
9.37
81.22
81.04
18.51

---

A Closer Look at Generalized BH algorithm for Out-of-Distribation Detection
74
20
99
50
90
44
54
(a) CIFAR-100
(b) TinyImageNel
12
(c) Place365
(d) SVHN
Figure 3. Comparison between g-BH algorithm our eg-BH algorithm terms of Fi-score and TPR. The x-axis corresponds to the number L in Algorithm 1, and the y-axis represents the value of metrics.
Table 5, Experimental results (%) of practical metrics on CIFAR- 10 as data. The score function Energy based on ResNet18. We compare the performance between g-BIl and eg-BH based on the same score function.
OOD data, compared to g- g-BH algorithm, our method re- duce reduce the FPR from 34.45% 30.02%, improve the TPR by 16.92% and the Fi-score by 20.91%. 9.. Obviously, this improvement still exists for Different OOD data and score function MSP. The above analysis demonstrates the superiority of our method over the g-BH algorithm on small calibrated set.
g-BH TPR
Bg-BH TPR
Data
F1
FPR
F1
FPR
CIFAR-100
82.16
88.59
27.05
91.17
92.91
25.80
TinyImageNet
59.34
53.51
34.45
73.25
70.43
30.02
SVHN
75.16
59.13
29.06
89.00
79.43
31.30
Place365
56.07
64.16
33.09
78.55
84.71
31.50
To assess the impact of L on both the g- BH algorithm and eg- BH algorithm. we set L {6, 7, 8, 10} and conduct the corresponding experiments using Energy as score func- tion based on the ResNet18. The experimental results are presented in Figure 3. From Figure 3, we find that our proposed method outperform the g-BH algorithm in terms of Fi-score and TPR across different values of L. More- over, for larger value T. (i.e. smaller calibrated set), the performance gap between our method and g-BH algorithm becomes more pronounced, especially when L = 9 and L = 10. This demonstrates the superiority of our proposed eg-BH algorithm over the g-BH algorithm on the smaller calibrated set.
F-MNIST
84.89
87.91
23.82
88. 10
90.26
24.05
MNIST
79.09
88.23
14.87
82.77
89.91
12.60
Average
69.45
73.59
27.06
83.81
84.61
25.88
Table 6. Experimental results (%) of practical metrics on CIFAR- 10 as ID data. The score function is MSP based on ResNet18. We compare the performance between g- --HH and eg-BHl based the same score function.
g-BH TPR
eg-BH TPR
Data
Fl
FPR
F1
FPR
CIFAR-100
80.70
82,53
25.60
89.09
87.28
25.48
TinyImageNet
73.25
78.43
33.02
82.78
88.05
31.55
SVHN
86.00
87.43
4.30
91.17
92.15
4.06
Place365
78.55
83,99
14.50
86.08
90.17
13.09
F-MNIST
85.26
82.63
21.79
90.11
89.27
19.05
7. conclusion
MNIST
81.26
85.30
19.87
86.78
90.92
16.70
In this paper, we thoroughly analyze the g-BH algorithm and demonstrate that a large calibrated set improves the perfor- mance of the g-BH algorithm in terms of TPR, while a small calibrated set weakens its performance. To address this is- sue, we propose a novel eg-BH algorithm that integrates multiple p-values for decision-making. Extensive experi- ments demonstrate the validity of our theoretical results and
average
80.84
83.39
19.85
87.67
89.64
18.32
posed BH algorithm achieves certain degree of improve- ment compared with g-BH algorithm. More significantly, the TPR and Fl-score are considerably improved. For exam- ple, when using Energy as score function and I'nyemtgeNNet
8

---

A Closer Look at Generalized BH algorithm for Out-of-Distribation Detection
the superiority of method over the g-BH algorithm.
Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely
simple activation shaping for out-of-distribution detection. In ICLR, 2023.
Acknowledgment
This work is supported by the Key R&DD Program of Hubei Province under Grant 2024BAB038, the National Key R&D Program of China under Grant 2023 (D2YFF3304402,.t Fun- damental Research Funds for the Central Universities under Grant 2042025kf0045.
Frolova, D., Vasiliuk, A., Belyaev, M., and Shirokikh, B. Solving sample-level out- distrrbtittiit detection on 3d medical images. arXiv preprint arXiv:22/2.06500 2022.
Gong, X., Yuan, D., and Bao, W. Understanding partial multi-label learning via mutual information. In Ran- zato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P. and Vaughan, J. W. (eds.), NeurlPS, pp. 4147-4156, 2021.
Impact Statement
To our best knowledge, this work has no negative social im- pact. This work mainly provides a solid theoretical support for the field of the OOD detection and improves the perfor- mance of existing methods obviously. Hence, our work may promote the development of the related applications.
Gong, X., Yuan, D., and Bao, W. Partial label learning via label influence function. In ICML, volume 162, pp. 7665-7678, 2022.
Gong, X., Yuan, D., and Bao, W. Discriminative metric learning for partial label learning. IEEE Transactions on Neural Networks and Learning Systems, 34(8):4428- 4439, 2023a.
References
Arlot, S., Blanchard, G., and Roquain, E. Some nonasymp- totic results resampling in high dimension, i: Confi- dence regions. Annals of Statistics, 38(1):51-88)) 2010.
Gong, X., Yuan, D., Bao, W., and Luo, F. A unifying probabilistic framework for partially labeled data learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8036-8048, 2023b.
Benjamini, Y. and Hochberg, Y. Controlling the false dis- covery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1 0:289-300, 1995.
Hendrycks, and Gimpel, baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Benjamini, Y. and Yekutieli, D. The control of the false dis- covery rate in multiple testing under dependency. Annals of statistics, pp. 1165-115 1188, 2001.
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-disttrbutton detection for real- world settings. In ICML, volume 162, pp. 8759-8773, 2022.
Bernard, C., Jiang, X., and Wang. R. Risk aggregation with dependence uncertainty. Insurance: Mathematics and Economics, 54:93-108, 2014.
Kaur, R., Jha, S., Roy, A., Park, S., Dobriban, E., Sokolsky, O., and Lee, I. idecode: In-distribution equivariance for conformal out- -id-tri-iitiiittionn detection. In AAAI, pp. 7104 2022.
Blanchard, G. and Roquain, E. Two simple sufficient condi- tions for fdr control. Electronic Journal of Statistics, 2: 963-992, 2008
Cao, H., Chen, J., and Zhang, X. Optimal false discovery rate control for large scale multiple testing with auxiliary information. Annals of statistics, 50(2):807, 2022.
Kolmogorov, A. N. and Castelnuovo, G. Sur la notion de la moyenne. G. Bardi, tip. della R. Accad. dei Lincei, 1930.
Casella, G. and Berger, R. L. Statistical inference. Cengage Learning, 2002.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Chen, Y. and Liu, W. A theory of transfer-based black-box attacks: Explanation and implications. In NeurIPS, 2023.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84990, 2017.
Delattre, S. and Roquain, E. New procedures controlling the false discovery proportion via romano-wolf" wolf's 's heuristic. Annals of Statistics, 43(3): (113):1141-111 41-1177, 2015.
Li, K., Chen, K., Wang. H., Hong, I.., Ye, C., Han, J.. Chen, Y., Zhang, W., Xu, C., Yeung, D., Liang, X., Li, Z., and Xu, H. CODA: real-world road corner case dataset for object detection in autonomous driving. In ECCV, volume 13698, pp. 406 406-423, 2022.
Deng, L. The MNIST database of handwritten digit images for machine learning research [best of the web). IEEE Signal Process. Mag., 29(6):141 141-142, 2012.

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-oo-distribuutin image detection in neural networks. In ICLR, 2018.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 7708.07747, 2017.
Liu, W., Shen, X., Du, B., Tsang, I. W. Zhang, W., and Lin, X. Hyperspectral imagery classification via stochastic hhsvms. IEEE Transactions on Image Processing, 28(2): 577-588, 2019.
Xu, and Liu, W. On robust multiclass learnability. In NeurIPS, 2022.
Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., Du, X., Zhou, K., Zhang, W., Hendrycks, D., Li, Y., and Liu, Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS, 2022.
Liu, W., Wang, X., Owens, J. D., and Li, Y. Energy-basee out-of-distribution detection. In NeurIPS, 2020.
Liu, X., Lochman, Y., and Zach, C. GEN: pushing the limits of softmax- out-o--distribution detection. In CVPR, pp. 23946-23955, 2023.
Yu, C., Ma, X., and Liu, W. Delving into noisy label detec- tion with clean data. In ICML, volume 202, pp. 40290- 40305, 2023.
Lu, H., Gong, D., Wang, S., Xue, J., Yao, L., and Moore, K. Learning with mixture of prototypes for out-of- distribution detection. In ICLR, 2024.
Zhang. J., Yang, J., Wang, P., Wang, H.. Lin, Y., Zhang, H., Sun, Y., Du, X., Zhou, K., Zhang, W., Li, Y., Liu, Z., Chen, Y., and Li, Openood v1.5: Enhanced benchmark for -nttt---iittiuttionn detection. CoRR, abs/2306.09301, 2023a.
Ma, X., Wang, Z., and Liu, W. On the tradeoff between robustness and fairness. In NeurlPS, 2022.
Ma, X., Zou, X., and Liu, W. provable decision rule for ouitt--diitr detection. In ICML, 2024.
Zhang, S., Zhou, C.,, Zhang, P., Liu, Y., Li, 7.,, and Chen, H. Multiple hypothesis testing for anomaly detection in multi-type event sequences. In ICDM, pp. 808-817, 2023b.
Ma, X., Wu, J., and Liu, W. SAC-BL: A hypothesis testing framework for unsupervised visual anomaly detection and location. Neural Networks, 185:107147, 2025.
Zhang, S., Zhou, C.,, Liu, Y., Zhang, P., Lin, X., and Pan, S. Conformal anomaly detection in event sequences. In ICML, 2025.
Meng, X.-L. Posterior predictive p-values. The annals of statistics, 22(3):1142-116) 1994.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Reading digits in natural images with unsupervised feature learning. 2011.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recog- nition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6): 1464, 2018.
Regmi, S., Panthi, B., Dotel, S., Gyawali, P. K., Stoyanov,
D., and Bhattarai, B. T2fnorm: Train-time feature nor- malization for OOD detection in image classification. In CVPR, pp. 153-162, 2024.
Zou, X. and Liu, W. On the adversarial robustness of out-of- distribution generalization models. In NeurlPS, 2023a.
Zou, X. and Liu, W. Generalization bounds for adversar- ial contrastive learning. Journal of Machine Learning Research, 24:1 14:1-114:54 2023b.
Rüschendorf, L. Random variables with maximum sums. Advances in Applied Probability, 14(3):623 632, 1982.
Sastry, C. S. and Oore, S. Detecting out-of-distribution examples with gram matrices. ICML, volume 119, pp. 8491-8501, 2020.
Sun, Y., Ming, Y., Zhu, X., and Li, Y. Out-of-distribution detection with deep nearest neighbors. In ICML, volume 162, pp. 20827-20222 20840, 2022.
Wang, B. and Wang, R. Joint mixability. Mathematics of Operations Research, (1(3):8088824 2016.
Wei, H., Xie, R., Cheng. H., Feng, L., An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. In ICML, volume 162, pp. 23631-23644, 2022.
10

---

A Closer Look at Generalized BH algorithm for Out-of-Distribation Detection
A. Proofs
A.1. Proof of Theorem 4.1
Proof. To obtain an explicit analytical solution, we assume that the testing data comes sequentially in a stream. Then, given a testing set T {X1est X2 ...... ....,.,....... X consisting of ID data (TPR only focus on the detection performance of ID data), the expectation of TPR conditional calibrated set Teal for the g-BH algorithm can be expressed
((TPR = ((f(((( a)
Note that the data Xiest Xgest Xtest are independent and identically distributed, we have
E E(TPR/T = =E (1((6((X7))))) > a)(Tea))
=P([e((X1 > alTool).
Note that the empirical p-value
p(x 2 1(0(X;a)) S s(X(ea))) +
m +
Since E F2, (() > 0 and thus f(-) is increasing. Denote by f-( (.) the inverse function of f(). Then, we obtain
T(T P(S(p(X((3t) a|Toll)
EE (xx(x)a)) S s(X(es))) +1 > f (a)]
m+1
m
{ + 1) -1/ (s (((xaa)) S s( ((X(est)) f (a)(m
/=1
E ((s(X)at) S s(x(cot))
(a) + 1)
Treal
m
m
((s((test))) (a) +1 Teal
m
Without loss of generality, we suppose that the random variable s(X s(x(ett) follows continuous distribution. Denote by F(.)
the real cumulative distribution function of s (((eat) and by f-) the inverse function of F(-.) addition, we denote [-] the floor function. Then, we get
f (a)(m 1) Traa E(TPR/70a) = ((((Xtea))) 5
m
((s((((est) [ (a=)(m + 1)] Teeal
[f (a)(m) + +1) 1 |Tool s(Xtest) < F
- P
p-1 [1- 1)]
F
For simplicity, we denote 3 = (a) + 1)] - Note that F(Xcal) (1)
Therefore, we have
E( = - F(X
(g))
According to Eq. (5), (T -1- (8)
To complete our proof, we need the following technical lemma.

---

A Closer Look at Generalized BH algorithm for Out-of-Distribatoon Detection
Lemma A.1. Suppose the continuous random variable X have the cumulative distribution function F(.). Then, the random variable (XX) follows the uniform distribution on (0, 1).
m and (x)) is the 3-th order statistic from the smallest to the largest. By Lemma A. p(X;a) follows the uniform distribution on (0, 1). Next, we aims to
For the examples X( X2 ..... X we denote p(X;al)= F(X;")), 1,2. ,0...,,,,111
derive the probability density function of (() According to the Definition of p(X;""),, p(X(""), p( p(xg"))) pp( (X are independent and identically distributed. Denote by (-(-) the cumulative distribution function of p(X;ul). For any æ in the support set Ep(-) and sufficiently small ö, we have
IP (s S p( (()) <x 6) = PP( one of the pl o( E [1, 6) and B - of the others < x:)
(p(X;eu) E [x, + 6) and exactly 3 - of the others < xx)
nP (p(Xpa)) E [x, 6) and / of the others < <x)
(8)
nP (p(XFo)) - 6)) P(B of the others < <x)
nP (p(X)al) E |2, 8)) ( P(p(X < ==)p(1((((((a) x)m-9
- 1
Then, the probability density function of p(X (() is
x S (()) < <x+d)
fa(x)
lim 5-0
m m 1
1 (z) ((
3 1
m (m-
9-1
(1 m) if0 <
otherwise.
Therefore, the probability density function of E(TPR] = F( can be expressed as
(x)))))Tea))x)) f,(1 - x)
m m- 1 8-
m- (1 0-1 f00x<< <1
otherwise.
The above result indicates that B(TPR]Tecl) follows beta distribution with shape parameters m 8 and 8, which completes the proof.
A.2. Technical Lemmas and Their Proofs
Lemma A.2. Suppose that the function g(-) is continuous and monotonically increasing [0, 1]. Then, for any (0, 1). we have
I." g(z) glz)do= do sa L. g(x) dz.
Proof.
Since function 9(-) increasing, g(z) is integrable, namely. g(x) dr oo. Note that
a Sf g(z) dx = a 1 g(x) do + a g(x) dx
12

---

A Closer Look at Generalized BH algorithm for Out-of-Distribation Detection
then, have
J," g(x) dr a Lam-aaa o. ((o)dd dx
/ g(o) de a1 /'' g(x) dx a S: g(z) dr.
Laddr-aao a a g(o) de a S: g(x) de
By the first mean value theorem for integration, there exist Eι E (0,a) and &a E (a, such that
L g(x) de ag((t), [( g(())de = (1 a))((())
Obviously, E S 22. Since g(x) is increasing, then g(Ei) - ((2)). Therefore, we have
[ g(z) dr a L g(z) dr a(1 - a) -0(2)) = 0,
namely, for any (0, 1),
/ g(x) dx Sa aaf g(z)dz.
Based the lemma A.2, have following lemma.
Lemma A.3. Suppose that the function g(-) is continuous and monotonically increasing on [0, 1]. Then, for any a (0, 1), have
(( (
g(x) de)
a.
Proof. Denote Y; = g(p;) where pæ E P. Without loss of generality, we assume that the p-values PI,". ,r, are exact. Note that for any & E (0, 1). we have
PVY s = PP(()) ( (p: I ( 9 (t)-
Therefore, (-) is the cumulative distribution function of Y,. Based on the theoretical results in Bernard et al. (2014), we obtain
('h = inf (Q(R(p. w).
MEP
where h(pr.... ... ".P.....PPP..... .PL) = =Ω(p, w) and
A(p, w) (wngg) + w2g(p2) +... wr.g(pr))
Note that (((w)g(p) + (2g(22) +.. (..,(wr))))) 1) is the essential supremum of wig(p)) + wag(pz) 4.. "...,,
thus the following relations holds:
@((w1g(P)) + w2g(p2) +... (.t....ww((p))) 1) 2 E (wig(p1) + w2g(ps) +... ,.r.,,
Since g(pr), s(22), .. s(LL) are identically distributed sharing the cumulative distribution function y --(), we have
E (wig(p1) + w2g(p2) +..., w..((p)))) wi R(g(p)) w2 W2 +... "...,,,,E((((E)))) (wi + ... wt. (()(i((())))
((g(p))) = S g(x) de
13

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
By Lemma A.2, we get Q((rrg(p)) + w2g(p2) + + (wo(pL)) - / g(z) dæ
Because g(-) is continuous and increasing, 9 (-) is also continuous and increasing. Hence, for any ps P have Q(g ((w1g(p1) + w2g(p2) + +.... 1)) (2 g o(z)dx)
namely, 9 g- (2 Jo g(x) dx) is the lower bound of Q(g ((wig(p)) w2g(p2) t.. (Lg((L))) 1). Then, we get inf
(h)
P.EP {Q(g-1 ((w1g(p)) w2g(p2) t.. wro(p)))) 1)}
J / g(x) dx
2 ~g
According to the definition of a-quantile, obtain
P 2(p. w) وو65 (2 (z))d)) )) (P(pp w) S SQQO(, a)))) Sa.
Lemma A.3 provide significant region for the level By Lemma A.3, we can choose appropriate function g(-) integrate various p-values.
A.3. Proof of Theorem 5.1
Proof. When g(x) x², gg'g((x)) x². According to the Theorem A.3, we have
g (1 (n g(z) (x+ = (K + =
a²
l
and
P 0(p, w) و و6 ( (" ((x) dx)
=P -P(wwipt + u2pt + ....4 wIPL)# - (N + 1) 1)=+aa
=P ( ( + w + ... WLP sa) s a
Therefore,
(ks + (wirf + wopp + ... wLPi)
is a valid p-value. Specifically, when and w; 1,
(ne + (wipf + wops + +.... + = 2(p1 + + + ... .....7 PLL)
L
Denote wk- max(p1, P2, pr.. Note that
(N +++1)# (w p ) < (xc 1) (wIp) + war + ... wLpt) < (N ()+
further,
lim (x + =) (wi (w K=000
(wx-pp-) ) lim (n + 1) ( Pk-
k-000
Hence, when & + 00, we have
lim (K 1) (Wipf + wyps + WLFf)+ max P2.. ... .PL). 8-000
14

---

A Closer Look at Generalized BHf algorithm for Out-of-Distribution Detection
A.4. Proof of Theorem 5.2
Proof. According the proof of Theorem Af3, g(x) xx, we have
()) inf Q (h, inf p.PP
{Q((wiP + wops + #
P.EP
Then,
a
P.EP (Q'( inf (Q((wpP waps +....... wLPt), K
Note that A > 0, the probably density function of g(p:) is monotone its support set. By Wang & Wang (2016),
(Q" aa))* inf (Q((wip + wapp + ... + wrrf), 1))
mep
x+1
if and only if the "mean condition"is satisfied:
w'a" a
(wi + u2 + + FL - w²aa = (1 w w')aa.
K
Equivalently, w S min 1
J. Further, we have
Tix x+1
@'h,p,,a)= inf D.EP
{o((x= + 1) (wipf w2p5 + #LPF) 1)} a,
where h(p) = (N + (wipf + wppt + ... + WLPF) A. Next, based on the condition in Eq. (9), we aim to demonstrate
Sa) ((il
P
sup
P.PP
If Q" p, a, for any a E (0, 1) and arbitrary p-values Pl, P1,P2.- Pr where E P, we have Q(h(),,) a) 2 according to the definition Q*"(hhhh p. a). By the definition of a-quantile, P(h(p) < a) S a. It follows that
(h(p) sa) < (h((p) <a sa+5,
Since is arbitrary, we have
P (h(p) Sa) aa
On the other hand, according to the definition of infimum, for any 8 E (0, 1), there exist the p-values Pi," "M .PL E P such that Q(h(p"(h()) a) + 5, and thus P(h(pp) + 5) where p* {pi. ,PL) Since 8 is arbitrary, then have
P (h(p) sa)saa }=a.
sup PEP
B. Additional Experimental Results
In this section, we present additional experimental results. The results on MNIST as OOD data are presented in Tables 7. Table 7 shows the same conclusions as those of tables in main text.
15

---

A Closer Look at Generalized BH algorithm for Out-of-Distribution Detection
Table Experimental results (%) practical metries CIFAR-10 ID data. The MNIST OOD data. Energy and MSP used
the score functions. We compare the detection performance of g- BH algorithm with different sizes of calibrated set.
Energy
MSP
Score
ResNet18
WideResNet
ResNet18
WideResNet
Ratio
Fl
TPR
FPR
F1
TPR
FPR
F1
TPR
FPR
F1
TPR
FPR
0.2
63.48
47.94
4.94
63.16
45.87
1.98
62.87
48.25
5.25
62.46
46.45
2.29
0.3
65.15
50.15
5.78
65.09
49.12
2.35
64.23
50.10
5.90
64.95
49.40
2.72
67.57
52.95
6.95
66.97
53.99
3.04
66.67
53.69
6.37
67.95
53.13
0.5
69.53
57.39
7.27
70.15
56.75
3.59
66.67
54.68
7.59
69.76
55.61
3.82
0.6
72.94
59.82
8.54
71.59
59.44
4.25
69.91
58.59
9.02
70.88
57.20
4.19
0.7
73.82
62.29
10.11
75.53
63.17
6.11
71.19
60.79
10.00
72.91
60.20
4.94
74.75
65.87
13.68
77.46
66.57
6.79
71.82
62.19
74.83
63.37
0.9
75.59
68.57
15.35
79.85
69.44
8.26
73.52
66.55
14.49
78.13
69.69
8.70
76.81
75.49
19.54
81.74
73.39
10.79
75.59
81.51
34.14
79.15
72.47
10.65
16


Reviews:
[{"review_id": "eQuAEbfiNt", "reviewer": "Reviewer_ZPgt", "summary": "This paper investigates the impact of the calibrated set on the generalized BH (g-BH) algorithm[1] for Out-of-Distribution (OOD) detection. The authors provide a theoretical analysis showing that the conditional expectation of the true positive rate (TPR) follows a beta distribution, demonstrating that a small calibrated set negatively affects performance of g-BH algorithm. To address this problem, they introduce the ensemble g-BH (eg-BH) algorithm, which integrates multiple empirical p-values for decision-making. Extensive experiments validate the theoretical findings, and show that eg-BH algorithm outperforms g-BH algorithm, particularly on small calibrated set.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__W1__: The proposed eg-BH algorithm depends on multiple p-values. the advantages of integrating multiple p-values should be discussed\n\n__Ans1__.   A small calibrated set leads to under-representative empirical p-values, which fail to capture the distributional\ncharacteristics of the ID data. To address this issue, we utilize the available\ninformation in training set to generate multiple empirical p-values. We then integrate these empirical p-values to a single p-value with more  information of ID data, making it more discriminative.\n\n__W2__. The proofs in line 572 - line 583 in appendix should be more detailed.\n\n__Ans2__. Following your suggestions, the detailed steps are as follows:\n\n$$ \n \\begin{aligned}\n    \\mathbb{E}(\\mathrm{TPR}| \\mathcal{T}^{cal}) & =  \\mathbb{P}(f(p(X_{1}^{test})) > \\alpha  | \\mathcal{T}^{cal}) \\\\\\\\\n    & = \\mathbb{P}\\left( \\frac{ \\sum_{j=1}^m \\mathbf{1}(s(X_{j }^{cal})\\leq  s(X_{i}^{test}))+1}{m+1} > f^{-1} (\\alpha)  | \\mathcal{T}^{cal} \\right) \\\\\\\\\n    & = \\mathbb{P}\\left(  \\sum_{j=1}^m \\mathbf{1}(s(X_{j }^{cal})\\leq  s(X_{i}^{test})) > f^{-1} (\\alpha)(m+1 ) - 1  | \\mathcal{T}^{cal} \\right) \\\\\\\\\n    & = \\mathbb{P}\\left(  \\frac{ \\sum_{j=1}^m \\mathbf{1}(s(X_{j }^{cal})\\leq  s(X_{i}^{test}))}{m} > \\frac{f^{-1} (\\alpha)(m+1 ) - 1 }{m} | \\mathcal{T}^{cal} \\right) \\\\\\\\\n    & =  \\mathbb{P}\\left( \\hat{F} ( s(X_{i}^{test}))  > \\frac{f^{-1} (\\alpha)(m+1 ) - 1 }{m}   |\\mathcal{T}^{cal} \\right)\n  \\end{aligned}\n  $$\n\n__Q1__. What are the advantages of controlling FDR?\n\n  __Ans3__. It is well-known that there is a tradeoff between the detection performance of ID and OOD examples for a trained score functions. Therefore, we cannot only consider the true positive rate (TPR) or false positive\nrate (TPR) when designing the OOD detection algorithm. Factually, an ideal OOD detection algorithm should achieve low FPR while maintaining a high TPR, which leads to a small FDP. Thus, controlling FDR can achieve a well tradeoff between the detection performance of both ID and OOD examples.", "claims_and_evidence": "Yes.the claims made in the submission appear to be supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes. the proposed method makes sense for the OOD detection.", "theoretical_claims": "Yes. I check some proofs, including Theorem 4.1, Lemma 5.1, Theorem 5.2 and Theorem 5.3.", "experimental_designs_or_analyses": "Yes. The experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "This paper focus on studying the influence of calibrated set on the performance of the g-BH algorithm. Besides, to address the problem caused by small calibrated set, this paper proposes the eg-BH algorithm to integrate the multiple empirical p-values for making decision.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The analytical framework of this paper is well-grounded in the statistical hypothesis testing. Hence, the conclusions have strong theoretical guarantee. The paper presents a rigorous theoretical analysis for the g-BH algorithm, demonstrating that small calibrated set tends to weaken the performance of it. The proposed eg-BH algorithm effectively enhances OOD detection performance on small calibrated set by integrating multiple empirical p-values, compared with g-BH algorithm. Extensive experimental results demonstrate the effectiveness of the proposed method.\n\nSome weaknesses are listed below:\n\nW1: The proposed eg-BH algorithm depends on multiple p-values. the advantages of integrating multiple p-values should be discussed.\n\nW2 The proofs in line 572 - line 583 in appendix should be more detailed.", "other_comments_or_suggestions": "No.", "questions_for_authors": "What are the advantages of controlling FDR?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hk4EF4KsmQ", "reviewer": "Reviewer_dWQD", "summary": "Based on the recent work [1], this paper aims to study the influence of the calibrated set on the generalized BH (g-BH) algorithm for out-of-distribution (OOD) detection task. By theoretical analysis and experimental results on the real data, the authors show that the small calibrated set tends to degrade the performance of the g-BH algorithm. Then, the authors propose an enhanced approach, the ensemble g-BH (eg-BH) algorithm, which integrates multiple empirical p-values to solve this issue. The claims in this paper are built on strong theoretical foundations and supported by extensive empirical validation", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Weakness 1__: the proposed method requires a hold-out set. However, since the hold-out set consists of ID examples, this is not a significant restriction.\n\n__Ans1__: We emphasize that the calibrated set consists of ID data, without the need of OOD data.  We directly extract some examples from training data to construct calibrate set. Hence, it is easy to obtain a calibrated set for our proposed method.\n\n__Weakness 2__: why does the authors only use the empirical p-values in Algorithm 1?\n\n__Ans2__: We must clarify that in practice, any valid computation methods that satisfy the definition of p-values can be used. Since the distribution of ID data is unknown, we choose a non-parametric method: empirical p-values to estimate real p-values. Essentially, it is equivalent to use empirical distribution to estimate real distribution. \n\n\n__Weakness 3__: Suggest checking and standardizing the format in Section 4.\n\n__Ans3__: Thanks for your careful review. We have checked the typos in Section 4. We will fix these typos in new version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. I check some proofs in Appendix, but do not carefully read them step by step.", "experimental_designs_or_analyses": "Yes. they seems quite sound.", "relation_to_broader_scientific_literature": "This paper provides a new decision framework based on the existing score functions, which enables to be adapted to small calibrate set.", "essential_references_not_discussed": "No, the paper includes all essential and relevant references.", "other_strengths_and_weaknesses": "Strengths \n\n(1) Theoretical analysis. The authors develop a novel theoretical understanding of the role of calibrated set in the g-BH algorithm.\n\n(2) Novel method. The proposed method extends g-BH algorithm by integrating multiple empirical p-values, mitigating the problem due to small calibrated set.\n\n(3) Extensive performance. The experimental setup, including the variation of calibrated set sizes, provides strong empirical support for the theoretical results. Additionally, the the authors use different score functions, such as Energy-based and Maximum Softmax Probability, to assess the robustness of the proposed approach on various benchmarks.. The evaluation is comprehensive, ensuring the generalizability of the results.\n\n\nWeaknesses\n\n(1) The proposed method requires a hold-out set. However, since the hold-out set consists of ID examples, this is not a significant restriction.\n\n(2) Why does the authors only use the empirical p-values in Algorithm 1?\n\n(3) Suggest checking and standardizing the format in Section 4.", "other_comments_or_suggestions": "No", "questions_for_authors": "See Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O3gp5eK3vT", "reviewer": "Reviewer_5FFL", "summary": "The paper explores the role of the calibrated set in the performance of the g-BH algorithm for OOD detection. Theoretical results indicates the large calibrated set will improve the performance of the g-BH algorithm but small calibrated set tends to degrade the performance of\nIt. Then, the authors propose a novel eg-BH algorithm to tackle the limitations of g-BH algorithm on the small calibrated set. Finally, the authors conduct extensive experiments to demonstrate the correctness of the theoretical results and the validity of proposed method. In summary, this paper represents a meaningful step forward in OOD detection study and provides valuable insights into multiple hypothesis testing applications.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Weakness 1__: about the interpretations of some concepts, including type-1 error, significant level and FDR.\n\n__Ans1__: we interpret these concepts as follows:\n\nSignificance level: If the probability of obtaining a result as extreme as the one obtained, supposing that the null hypothesis were true, is lower than a pre-specified cut-off probability (for example, 5%), then the result is said to be statistically significant (the null hypothesis is rejected), and the cut-off probability is called significant level.\n\nType-1 error and FDR: In statistical hypothesis testing, a type-1 error is the rejection of the null hypothesis when it is actually true.\nFDR can be considered as the generalization of the probability of type-1 error in single hypothesis testing. In multiple testing, the null hypotheses rejected by the detection algorithms are called discoveries. FDR is used to describe the expected proportion of erroneous discoveries among all discoveries. \n\n__Weakness 2__: in Theorem 4.1, what are the assumptions about the score functions?\n\n__Ans2__. We just require that the score function is continuous, without more strong assumptions. For example, two famous score functions MSP and Energy satisfy this demand.\n\n__Weakness 3__: in Algorithm 1, how do you choose the number L?\n\n__Ans3__: In practice, we can obtain a well performance of OOD detection when L is set to 3 or 4.", "claims_and_evidence": "Yes. This paper first verify the new finding about the g-BH algorithm on the small calibrated set by rigorous theoretical analysis and numerous experiments on real-world datasets. Then, this paper conducts extensive experiments to demonstrate the superiority of the proposed method over the g-BH algorithm on the small calibrated set.", "methods_and_evaluation_criteria": "Yes. This paper uses many evaluation criteria, including TPR, FPR, F1-score, AUROC, AUPR and FPR95. These criteria are suitable for the OOD detection problem.", "theoretical_claims": "Yes. I check the proofs in the Appendix. especially for Theorem 4.1 and Theorem 5.3, since these theorems are the key contributions of this paper. \n\n- Theorem 4.1 derives the distribution of conditional TPR on calibrated set, the distribution parameters are determined by the significance level and size of the calibrated set;\n\n- Theorem 5.3 provides a concrete method for integrating multiple p-values, which is the basis of the proposed eg-BH algorithm.", "experimental_designs_or_analyses": "Yes. In experiments, this paper first verifies the correctness of theoretical results. The experimental results show the same conclusions as Theorem 4.1. Then, this paper conducts extensive experiments to demonstrate the superiority of the proposed method over the g-BH algorithm on the small calibrated set.", "relation_to_broader_scientific_literature": "This paper finds the limitations of the g-BH algorithm on the small calibrated set and proposes a novel method to address this problem.", "essential_references_not_discussed": "No, the paper includes all essential and relevant references.", "other_strengths_and_weaknesses": "Strengths \n\n- This paper reveals the influence of calibrated set on the g-BH algorithm.\n\n- This paper establishes a mathematical relationship between the size of the calibrated set and the conditional TPR expectation of the g-BH algorithm.\n\n- This paper proposes a novel method to improve OOD detection performance on the small calibrated set by aggregating multiple empirical p-values.\n\n- This paper show the impact of different calibrated set sizes using various OOD datasets (CIFAR-10, SVHN, TinyImageNet, etc.). Besides, this paper shows that the proposed method consistently outperforms g-BH algorithm on the small calibrated set.\n\n\nWeaknesses\n\nI have not found any obvious weaknesses. It would be more comprehensive if the following questions are addressed:\n\n- The paper uses numerous terms from hypothesis testing, such as type-1 error, significant level and FDR, which may lead to difficulties for the readers unfamiliar with hypothesis testing. So the author should provide more explanations for these concepts.\n\n- In Theorem 4.1, what are the assumptions about the score functions?\n\n- In Algorithm 1, how do you choose the number L?", "other_comments_or_suggestions": "No", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQZhXWECHi", "reviewer": "Reviewer_rS6P", "summary": "The paper investigates methods for setting a decision threshold for out-of-distribution (OOD) detection using a calibration set, ensuring that the resulting detector achieves a desired performance on unseen data. The authors examine the g-BH algorithm, a recently proposed method for threshold selection [Ma 2024], and extend it in two ways. First, they characterize the distribution of the expected True Positive Rate for decision rules using thresholds set by the g-BH algorithm, demonstrating that the TPR follows a Beta distribution with parameters dependent on the calibration set size and the target $\\alpha$-level. Second, they propose an ensemble-based algorithm to address the inefficiency of the g-BH algorithm when using small calibration sets. Both contributions are empirically validated on standard OOD benchmarks.\n\n## update after rebuttal\nI acknowledge that I failed to fully grasp the setup and purpose of the proposed algorithm based on the current presentation. While I did not consult [Ma et al., 2024], which may provide additional context, I believe the paper should be self-contained and provide a clear and complete definition of the problem it addresses.\n\nThe authors' rebuttal offered some clarification, but key aspects remain unclear to me. For instance, the claim that \"a larger calibration set improves the performance of the g-BH algorithm in terms of TPR\" is difficult to interpret meaningfully without addressing other performance metrics such as FPR. Since the method revolves around setting a decision threshold on a small calibration set, reporting improvements in TPR alone seems insufficient for a full evaluation.\n\nThe paper may contain valuable ideas, but the current presentation does not clearly convey them to me. This limits my ability to fully understand the core contributions and assess their significance. While I recognize the possibility that the misunderstanding lies on my side, I made an effort to engage with the paper.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: about the claim and goal of our paper.\n\n**Ans1**: We first emphasize that FDR is closely related to TPR and FPR. Based on [1], we have\n$$ FDR = E(\\frac{1}{1 + \\frac{P}{N}\\cdot \\frac{1-FPR}{1-TPR} }) $$\nwhere P is the number of ID data in test set and N is the number of OOD data in test set. Obviously, larger TPR and smaller FPR leads to smaller FDR. [1]  points out that traditional decision rule just considers TPR when choosing decision threshold. By contrast, FDR can balance the TPR and FPR. __By controlling FDR, g-BH can better trade off the performance between ID and OOD data and further improve total decision performance (F1)__ (The details can be seen in third paragraph of Section 5.2 of [1]). The experiments in [1] also show that they care more about TPR, FPR and F1, instead of only controlling FDR.  The meaning of our claim is that larger calibrated set can improve the TPR of g-BH and decrease its FPR, leading to smaller FDR and larger F1. Our Theorem 4.1 and experimental results of Figure 1 and Tables 1-4 support this claim. Hence, our claim is align with the purpose of g-BH, and is correct without misleading.  \n\nThe goal of our paper is to __deeply study the influence of calibrated set on g-BH__, which has been repeatedly pointed out in Abstract (line 20-24) and Introduction (line 32-38, right). Overall, our paper consists two parts: 1, we find that small calibrated set degrades the performance of g-BH, and establish Theorem 4.1 and conduct extensive experiments (Tables 1-4) to verify this phenomenon. 2, we propose the eg-BH algorithm  to solve the issue caused by small calibrated set. The performance of hypothesis testing algorithm  depends on precise p-value. Since the distribution of ID data is unknown, [1] directly uses empirical p-value to estimate real p-value.  Glivenko-Cantelli theorem indicates that this estimation method performs well for large calibrated set. When calibrated set is small, the estimation error of p-value increases, leading to a poor performance of g-BH. Our core goal is to mitigate estimation error problem of p-value caused by small calibrated set. The entire Section 5 focuses on discussing how to obtain a good computation method of p-value based on small calibrated set. Combining our estimation method of p-value and g-BH, this algorithm is called eg-BH. Hence, our core goal is not simply to address the randomness caused by calibrated set, nor to set a threshold such FDR remains below a specified threshold.\n\n**Q2**: about the understanding of some concepts.\n\n__Ans2__: we first give a correct interpretation of FDR. FDR is the expectation of false discovery proportion (FDP). FDP can be expressed as A/B where A is the number of ID examples in test set falsely classified as OOD and B is the number of examples classified as OOD in test set. Hence, \n\"False Discovery Rate (the proportion of true OOD samples incorrectly identified as OOD by the detector) on unseen data\" is incorrect.  Besides, FDR is a real number without randomness. \"at a given confidence level\" is often used to describe random event. So, \"at a given confidence level\" is not suitable to describe FDR control. Different from conformal prediction, in hypothesis testing, p-value can not be called \"confidence level\". $\\alpha$ is only upper bound of FDR  and not used to describe confidence level. Moreover, p-value is the function of socre of testing example, instead of hyper-parameter, and thus can not be prespecified. \n\n**Q3**: about experimental settings and some notations.\n\n**Ans3** Our code is based on [2] and we use same experimental setup as [2],  which has been emphasized in Section 6.1 (line 362-363, left). The details, such as optimizer, learning rate, epoch, can be found in \nSection 4.1 of [2]. Besides,  the code of [2] run 5 times for every model, and all experimental results are the mean of 5 trials. You can find these information in Section 4.2 of [2]. We will provide these details in Appendix.  \nOur theoretical framework is based on [1] and we use the same notations and definitions as [1], which have been emphasized in Section 3 (line 102-104, right). The computation method of empirical p-value has been clearly presented in Section 5 (line 196-198, right) and Algorithm 1 (step 8) for two times. Following [1], $\\alpha$ is set as 0.05 (see Section 2.3 of [1]). The weights in eg-BH is 1/L, which has been clearly pointed out in Theorem 5.3. We will describe the weights in Experiments again. $n _0$ is the number of ID data in test set, which has clear description in the proof of Theorem 5.5. You can also find its detailed description in Sections 2.2 and 3 of [1]. \n\n\n[1] A Provable Decision Rule for Out-of-Distribution Detection\n\n[2] OpenOOD: Benchmarking Generalized Out-of-Distribution Detection", "claims_and_evidence": "A central claim, reiterated multiple times in the paper, is that larger calibration sets enhance the performance of the g-BH algorithm in terms of the true positive rate (TPR). However, this claim is somewhat misleading and, in my opinion, fundamentally incorrect, as it does not align with the intended purpose of the g-BH algorithm or its guarantees. The algorithm is designed solely to set the decision threshold such that the actual false discovery rate (FDR) or TPR does not exceed a specified threshold at a given confidence level.", "methods_and_evaluation_criteria": "The authors devote a significant portion of the paper to empirically analyzing how the calibration set size affects the performance of the OOD detector configured by the g-BH algorithm. However, the experimental setup is not clearly described, and its purpose remains somewhat unclear. In particular, the confidence level (p-value), and the desired FDR threshold are not consistently defined across all experiments.\n\nFurthermore, since the core issue addressed in the paper is the randomness introduced by small calibration sets, the experiments should account for this variability. This could be done by running multiple trials and reporting summary statistics (such as the mean and standard deviation) rather than presenting single-value metrics (e.g., FPT, F1), which are merely realizations of random variables with high variance.", "theoretical_claims": "I did not verify the proofs in detail, but I believe Theorem 4.1 is correct. The setup and results closely resemble those found in problems related to characterizing the coverage of a conformal predictor. Specifically, the claim that coverage—computed in the same manner as in the BH algorithm—follows a Beta distribution has been previously established in:\nVovk, \"Conditional validity of inductive conformal predictor,\" ACML, 2012.", "experimental_designs_or_analyses": "As previously noted, the design and analysis of the experiments have the flaws mentioned earlier.", "relation_to_broader_scientific_literature": "The authors cite relevant literature. I see a connection between Theorem 4.1 and a similar problem in conformal prediction, as discussed earlier. However, I do not view the omission of this reference as a significant issue.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "As mentioned earlier, the problem to be addressed is not clearly defined, the claims lack clarity and precision, and the descriptions of the algorithms and experiments are insufficiently detailed. Examples of issues related to problem formulation and vague claims have already been provided. Additionally, several key details are missing regarding the algorithms and experimental setup. For instance, it is unclear how the weights of the ensemble algorithm are determined, what $n_0$ represents in Theorem 5.5 and Corollary 5.6, and how the p-values and FDR thresholds were set across all experiments.\n\nCharacterizing the distribution of the true positive rate (TPR) for the g-BH algorithm, along with the analysis of the proposed ensemble algorithm, could provide solid contributions. However, the paper's presentation is far from ideal, making it difficult to thoroughly assess its validity and clearly grasp its main message.", "other_comments_or_suggestions": "No.", "questions_for_authors": "One of the shortcomings of the paper is the lack of a clear problem definition, which should be stated explicitly at the beginning. Based on my understanding, the goal is to set a threshold on the calibration set such that the False Discovery Rate (the proportion of true OOD samples incorrectly identified as OOD by the detector) on unseen data remains below a specified threshold with at least $1-\\alpha$ confidence. Can you confirm if this is the intended problem being addressed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper analyzes the relationship between the size of the calibration set and the expectation of true positive rate (TPR) when performing out-of-distribution detection based on empirical p-value computed over the calibration set. To address the limitation caused by small calibration sets, the authors propose a novel detection algorithm that utilizes multiple empirical p-values computed over multiple calibration sets, which are obtained by partitioning the data. The proposed method is guaranteed to control the false discovery rate (FDR) at a prescribed level. Theoretical claims are supported by extensive experiments.

Overall, reviewers agree that the paper makes a meaningful contribution to the development of OOD detection theory, and that the proposed method is supported both theoretically and empirically. Meanwhile, authors are encouraged to enhance clarity in the presentation, especially provide more explanation on the statistical concepts from hypothesis testing.
Decision: Accept (poster)