Boosting Out-of-Distribution Detection with Multiple Pre-trained Models 
Authors: Feng Xue
Zi He
Chuanlong Xie
Falong Tan
Zhenguo Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Out-of-Distribution Detection
Model Zoo
Ensemble

Abstract: Out-of-Distribution (OOD) detection, i.e., identifying whether an input is sampled from a novel distribution other than the training distribution, is a critical task for safely deploying machine learning systems in the open world. Recently, post hoc detection utilizing pre-trained models has shown promising performance and can be scaled to large-scale problems. This advance raises a natural question: Can we leverage the diversity of multiple pre-trained models to improve the performance of post hoc detection methods? In this work, we propose a detection enhancement method by ensembling multiple detection decisions derived from a zoo of pre-trained models. Our approach uses the p-value instead of the commonly used hard threshold and leverages a fundamental framework of multiple hypothesis testing to control the true positive rate for In-Distribution (ID) data. We focus on the usage of model zoos and provide systematic empirical comparisons with current state-of-the-art methods on various OOD detection benchmarks. The proposed ensemble scheme shows consistent improvement compared to single-model detectors and significantly outperforms the current competitive methods. Our method substantially improves the relative performance by $65.40\%$ and $26.96\%$ on the CIFAR10 and ImageNet benchmarks.

Under review as a conference paper at ICLR 2023
BOOSTING OUT-OF-DISTRIBUTION DETECTION WITH MULTIPLE PRE-TRAINED MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Nit-of-DDittibution (OOD) detection, i.e., identifying whether an input is sampled from a novel distribution other than the training distribution, is critical task for safely deploying machine learning systems in the open world. Recently, post hoc detection utilizing pre-trained models has shown promising performance and can scaled to large-scale problems. This advance raises a natural question: Can we leverage the diversity of multiple pre-trained models to improve the performance of post hoc detection methods? In this work, we propose a detection enhancement method by ensembling multiple detection decisions derived from zoo of pre- trained models. Our approach uses the p-value instead of the commonly used hard threshold and leverages fundamental framework of multiple hypothesis testing control the true positive rate of In-Distribution (ID) data. We focus on the usage of model 200s and provide systematic empirical comparisons with current state-of- the-art methods on various OOD detection benchmarks. The proposed ensemble scheme shows consistent improvement compared to single-model detectors and significantly outperforms the current competitive methods. Our method substan- tially improves the relative performance by 65.40% and 26.96% on the CIFAR10 and ImageNet benchmarks.
INTRODUCTION
Deep neural networks have achieved empirical success in many applications, but generalization robustness has always been thorny problem deep learning. sophisticated and well-trained deep neural network can provide excellent test performance on identically distributed (ID) test data but may fail to make accurate predictions on inputs from outside the training distribution Nguyen et al. (015). This poses a big obstacle to the generalization of deep neural network models. Especially in safety-critical applications, is better to identify mut-of-distribution (OOD) inputs ahead of time rather than letting the model make predictions that may be unreliable.
On the basis of pre-trained deep neural networks, many recent works on post hoc OOD detection have proposed diverse score functions to distinguish OOD samples utilizing the output probabil- ity, logits, gradients, and features of the pre-trained classifier. At the same time, some works also propose new training strategies to encourage the network to learn more features that may not be relevant to the OOD classification task. For example, MSP (Hendrycks & Gimpel 2017) uses the maximum softmax probability, Energy score (Liu et al. (0200) considers the logits, and GradNorm (Huang et al. 2021) employs the vector norm of gradients. Based on these frameworks, several improved methods such as ODIN (Liang al.. 2018), Adjusted Energy Score (Lin al. 1021), ReAct (Sun et al. 202T) are proposed to enhance the performance of OOD detection. These score functions above measure the similarity between a test input and the training (ID) data through pre- trained feature extractors or classifiers. There are also many distance- algorithms that directly quantify the distance of samples the embedding space extracted from pre-trained model and regard test input the conditional distribution of features given the class label is a Gaussian distribution and derives confidence score based on the Mahalanobis distance. SSD (Sehwag et al. 2020 considers self- supervised pre- pre-training and a Mahalanobis distance. Taak et al. (2020) uses contrastive learning with distributionally shifted augmentations for pre-trane tre-trainig and proposes detection score specific to their training scheme. [Snn et al.) (2022) studies the nearest-neighbor distance and demonstrates the efficacy of norpperamtrric modeling of the feature distribution for OOD detection tasks.
OOD sample when it is far from the ID data. [Lee et al. (2018) assumes

---

Under review as a conference paper at ICLR 2023
The performance of post hoc detection highly depends on the quality of pre-training. The most com- monly used model architectures in OOD detection include convolutional networks such as ResNet (He et al. 2016), DenseNet (Huang et al. (017) and Wide-ResNet Zagoruyko & Komodakis and of course Transformer models such as Swin (Liu et al. 2022) or ViT (Dosovitskiy et al. 2021 In general, the pre-trained models focus the features related to classification tasks and the learnt representation may be insufficiently rich for OOD detection. Therefore, researchers have proposed ideas such as contrastive learning (Winkens al. 2020 Tack et al. 22020), adversarial training Big- gio & Roli 2018);; Miller et al. 2020); Chalapathy & Chawla (2019), outlier exposure (Hendrycks et al. 2018; Papadopoulos et al. 2021) or other auxiliary artificially synthesized data (Lee et al. 2017 and auxiliary loss function (Vyas et al., 2018) to encourage models to learn high-level, task- agnostic and comprehensive features, which makes the model more robust and efficient in the down- stream detection task. These models trained with different architectures and training strategies can extract diverse features that may complement each other well. So, a natural question is raised:
2016
Can we leverage the diversity of multiple pre-trained models to improve the performance of post hoc OOD detectors?
To answer this question, we first build model zoo that captures as many properties of the input as possible and remains sensitive to distributional changes. Then we reformulate the OOD detection task to check whether there exists model in the model zoo that can identify the test input as an OOD sample. Section [.I shows that the naive ensemble of multiple OOD detection decisions cannot maintain the true positive rate of the ID data (TPR). Therefore, we propose ensemble scheme to integrate the results of multiple OOD detectors and provide theoretical guarantees that our method can keep TPR at the target level. In Section 4 method, which is close to the target TPR level.
we also report the empirical TPR of our
Ensembling not new to OOD detection. Morningstar al. (2021) combines multiple test statistics from generative models to differentiate ID and OOD data. Haroush et al.] (2022) uses both the Simes' method and Fisher's method to summarize p-values computed for each channel and layer of a deep neural network. [Bergamin et al. (2022) shows that combining different types of test statistics using Fisher's 's method overall leads to a more accurate out-of-distriburt
test. Recently,
Magesh et al. (002)) proposes ensemble framework that combines any number of different test statistics using the
procedure (Benjamini & Yekutieli, 200T) and conformal p-value estimator (Vovk et al.) (199). In this work, we develop a simple and fundamental ensemble scheme for using model zoos in OOD detection and name our method Enhancement (ZODE). Our method directly estimates the p-values according to its definition and employs the Benjamini-Hoccherg procedure (Benjamini & Hochberg 1995) to control TPR. Then, we provide theoretical guarantees and empirical validation to show that ZODE can maintain the TPR close to its target level. On the other hand, we focus on the settings of the model zoo and conduct systematic experiments to demonstrate the superiority of our approach. First, we show that ZODE can consistently improve current OOD detectors. Second, by comparing modll detectors with the ZODE-ensembled detector, we find that ZODE can exploit the diversity of multiple pre- trained models and leverage complementarity among single-model detectors. Finally, our approach significantly improves current SOTA performance.
OOD Detection
We summarize our contributions as follows:
We provide novel insights into OOD detection from the perspective of the model zoo. We propose an enhancement scheme, ZODE, for OOD detection by exploiting the diversity of pre-trained models. The proposed method is inspired by a simple and fundamental frame- work of multiple hypothesis testing. Our theoretical results and experiments clearly show that ZODE can leverage the complementarity among single-model detectors to improve performance.
We point out that the naive ensemble of multiple OOD detectors leads to lower TPR. Then we provide theoretical analysis and empirical validation to demonstrate that our proposed method can maintain TPR well under the settings of the model zoo.
Extensive experiments show that our method can effectively and consistently improve the power of identifying OOD samples. On commonly used CIFAR10 benchmark, our method significantly improves the SOTA result of the average false positive rate from 11.07% 3.83%. For challenging OOD detection task based ImageNet, we show
2

---

Under review as a conference paper at ICLR 2023
that our method is scalable to large-scale problems and significantly improves the SOTA result of the average false positive rate from 38.47% to 28.10%.
2 PRELIMINARIES
cut-of-Distribution Detection aims to check whether a test input generated from the training distribution not. It is one-sample hypothesis testing problem if we can only access the training data. We denote and y as the input and label space respectively and let Pa be the training distribution over X' X V. Suppose that o(x) is neural network trained data drawn from Pia to predict the label of input x E X. Let Did denote the marginal distribution on X. Then we call ~ Did in-distribution (ID) sample, otherwise, identify it "unknown" input, called out-of-distributio (OOD) data. At test time, OOD detection distinguishes OOD samples and ID samples by using a decision function:
{ {
ID S(x*) 2 X; OOD S(x*) A;
G(x')=
where x" is test input, S(.) is score function that gives higher scores for IID data and lower for OOD data, and À is the threshold. In this work, we consider post hoc OOD detection in which the score function S is derived from pre-trained classifier i.e. S(x*) ;()
We denote (s; Φ) as the distribution of 5(x; Φ) with x ~ Dia and any pre-trained model éb Then, if x" is an ID sample, the score (xx"; () is an ID value following the distribution F(s; Φ). Therefore, given a pre-trained model zoo M = {Øi...., Om J. we strengthen the OOD detection problem to:
Is there @ E M that would allow us to identify x* as an OOD sample?
In this work, proposed approach to achieve the goal of this OOD detection problem.
3 METHODOLOGY
3.1 NAIVE ENSEMBLE CANNOT MAINTAIN TPR
To leverage the model zoo M for OOD detection, straightforward way is to execute the detection procedure in Eq.(T based on each pre-trained model:
{
ID
;(x"; 2 Asi
G(x"; x)
{OoD
OoD ;(x);0) < Aoi
and identify x* as an OOD sample if there exists & E M such that G(x"; ()) = 00D, i.e.
ID G(x"; OOD if ( Ao. 36 M;
if S(x": 2 Ao. Vo E M:
(2)
In other words, x* is classified x" is an ID sample. However, this simple approach is not easy to control the true positive rate of the ID data (TPR). In practice, the threshold X is chosen that high fraction (e.g. 95%) of ID data is correctly identified. We denote the target level of the true positive rate of the ID data as TPR, and write a
ID sample only if all detectors G(x"; oi), 6, E .M agree that
TPR.- Therefore, each detector G(x"; (b)) has a probability of misidentifying an ID sample as an OOD sample. When ensembling multiple single-model detectors, the probability of making mistakes also accumulates. It is easy to see that the detector G(x";A) can misidentify an ID sample as an OOD sample with probability more than a, specifically 1 - (1- a)m when more and more pre-trained models become available, this error probability of G(x";M increases until it becomes 100%. This implies that the naive ensembled detector cannot maintain the target TPR level. On the other hand, by fixing m, we can assign a low probability to make sure 1 - (1 at) ) = 5%. In this case, TPR, should be very large and even close to 1. This greatly reduces the probability of successfully identifying OOD data, as each single-mon model detector becomes very conservative and can only identify extreme OOD data. In this work, we develop an ensemble scheme that can maintain the target TPR level while keeping a high probability of successfully identifying OOD data.
detectors are independent.
3

---

Under review as a conference paper at ICLR 2023
3.2 USING P-VALUE FOR OOD DETECTION
However, directly integrating score functions is uninterpretable and lacks theoretical guarantees. Therefore, we use the p-value for OOD detection. P-value (Abramovich Ritov 2013) is defined in the framework of statistical hypothesis testing. In OOD detection, the p-value is probability measure that quantifies how extreme the observed score is when the input is an ID sample (Cai & Koutsoukos 2020 Morningstar et al.) 202T Haroush et al. 2022, Bergamin et al. 2022 Magesh et | Kaur et 22022 For example, we identify an input x as an OOD sample (reject the null hypothesis) when the observed detection score S(x) is smaller than a critical value y. Given test sample x", , the lower value of S(x*), the more likely x* is not drawn from the training distribution. Hence, the p-value of x" is the probability that ((x) is less than S(x*) under the ID distribution, that is,
(3) In general, if the p-value of x* is less than 0.05, we can determine that x" is an OOD sample the significance level 0.05. In Appendix C
P-value of x" (x)) I S(x') Dia).
we show that using the p-value is equivalent to using the
hard threshold in Eq. (T]
Suppose the test input x" is an ID sample that x* ~ Dia and the detection score S(x") is contin- uous random variable. We write po as the p-value of x* and let F(s) the cumulative distribution
function of S(x) with x ~ Dig. Then we have Po P(S(xx) S S(x*) x Pia) F(S(x)).
follows from the continuity of S(x")) and Lemma 21.1 of Van der Vaart 2000 that
P(Po- < <a) = 1 ---((((((x))) 2 a)
= 1 11 P(((x)) 2 (o) = FF( = =a.
This implies that the p-value of x" follows a uniform distribution U00, 1]. In the following, we will use this property to develop an ensemble scheme (Theorem and Lemma
3.3 TPR CONTROLLING FOR ENSEMBLE
According to Eq. 3 the v-value relies on the score function S(x), which is derived from pre- trained model Φ, i.e. S(x) S(x; Φ). Given one pre-trained model, we can construct score function and compute the p-value of test input. But when multiple pre-trained models are acces- sible, how to fuse the single-model results to leverage the diversity of multiple pre-trained models while strictly maintaining TPR on ID data?
We borrow the idea of the Benjamini- Benjaanni-Hoccherrg procedure (Benjamini & Hochberg, 1995) and pro- pose an ensemble scheme for OOD detection via p-value correction. Consider a model zoo with m and score function S(x; o). Given test input x* and a pre- pre-trained model Φ.. we compute the score value S(x" Φ1) and obtain the corresponding p-value p:- Going through all pre-trained models,
pre- pre-trained models: M {$1. 02, . ....... 2m
obtained m p-values: {PI. P2,- P ). and sort them in ascending order: P(1) S P(2) S ... S P(m)- Then, we identify the test input x" as an OOD sample if there exists an integer 1 < k S yn such that P())
& (1 - TPRo). Here "TPR,' is predetermined TPR level of the ID data. In general, it is taken to be 95%. We call the proposed method Zoo-based OOD Detection Enhancement (ZODE) and present the details of ZODE in Algorithm I Next, we
provide theoretical guarantees that Algorithm [aan maintain the target TPR level on ID data.
Theorem Suppose a pre-trained model zoo {01, .z..... dm is accessible and the score function is S(x; Φ). Let TPRo > 0.5 be a predetermined TPR level for the ID Data. If the test impul x" is an ID sample that x" Dia and ;(x";(x) ) is independent of S(x": :bg) (b))for o Vi # j. then Algorithm can identify x" as an ID data with probability larger than T PRR-
Remark.
Here we assume that S(x*: (t)) is independent of S(x"; (;)), which leads to the indepen- dence between pt and pj for Vi a j. This assumption can hold if different pre-trained models learn completely different features. In this case, the model zoo haves the desired diversity. In practice, the pre-trained models can still be very diverse but different models may extract related features. There- fore, report the empirical TPR of our method in Section 4 One can find that ZODE can still maintain the empirical TPR not less than the target level though the p-values may be related. The proof is postponed to Appendix A In Appendix we analyze the detection power of Algorithm
4

---

Under review as a conference paper at ICLR 2023
Algorithm ZODE: Zoo-based OOD Detection Enhancement
Require: Training data {x, )< pre-trained model zoo føi...., pm J. test sample x", detection score S(x; 6), TPR level for ID data "TPRo";
1: Stage Inference
2: Compute the score value of ((x,, $5), LA sis n and LA < S m;
Stage 2. Testing
4: for 1 S j S. m do
5: Estimate the p-value of given Øj:
# # S(x,, Ø3) s S(x",, ffi)
Pj
6: end for
7: Stage Ensemble
8: Sort {Pi...., Pm in ascending order: {P)),.... ................................ P(m)) }:
9: if 31 I k S m such that P(a) S m (1 (TR))) then
10: return x" is an OOD sample;
else
12: return x" is an ID sample.
end if
as m tends to infinity, which implies that FPR is guaranteed as the size the model zoo increases. Appendix [D more discussions about the ensemble scheme and compares the BH procedure with three baseline ensemble schemes.
Computational complexity. In Algorithm 1, decompose ZODE into three stages: inference, testing, and ensemble. The inference stage requires computing the score of all validation samples for all pre-trained models, and its computational complexity is times that of post hoc OOD detection using a single pre-trained model since ZODE uses m pre-trained models. However, the inference stage only needs the ID data and can be done before deploying the OOD detector. Therefore, its computational complexity does not increase the detection time when testing new inputs. In addition, the inference stage is only feed-forward and is easily parallelizable. Therefore, the computational burden is not heavy. In this work, all experiments can be done using one NVIDIA V100 GPU.
Interpretability. One of the benefits of ZODE is interpretability. If a test input is classified as an OOD sample, we can track which pre-models lead to this detection decision. At Step 9 of Algo- rithm if P(k) < (1 - TPRo) and P(y) > 1(1 TPRo), Vi > k. then there are k pre-trained models, corresponding to P(i)...... ................................ P(k.) respectively, that identify the test input an OOD sample. In our experiments, we exploit this interpretability to find that there are OOD images that only one pre-trained model can detect. This implies that ZODE leverages the complementarity between all single-model detectors.
Limitation. The limitation of ZODE is that the testing stage takes up lot of storage space. Post hoc OOD detection computes the score of all validation samples and selects hard threshold by the quantile of the empirical distribution of the detection score. Therefore, post hoc OOD detection only passes the threshold from the inference stage to the testing stage. In Algorithm | the testing stage requires the score validation samples to compute p-values. Therefore, the testing stage of ZODE takes up more storage space than post hoc OOD detection methods.
4 EXPERIMENTS
In this section, demonstrate the effectiveness of our proposed method. First, we evaluate whether our model zoo and ensemble scheme can enhance OOD detectors. Second, we demonstrate that ZODE exploits the diversity of pre-trained models and leverages the complementarity between the single-model detectors to achieve superior performance. Finally, we show that our method can significantly improve the current SOTA results.
Dataset: We evaluate our proposed method on the CIFAR benchmarks. We use CIFARIO Krizhevsky al., 2009) as the IDD data and evaluate OOD detectors on six OOD datasets: SVHN
5

---

Under review as a conference paper at ICLR 2023
Table 1: Results on CIFARIO. Comparison with competitive OOD detection methods. The results
of all competitors are from Sun et al. 2022). are better and vice versa.
All values are percentages. & indicates smaller values
OOD Dataset
Mothod
SVHN
LSUN
ISUN
Texture
Places365
Average
TPR
FPR|
AUCI
FPR.
AUC
FPR
AUC
FPR
AUC
FPR.
AUC1
FPR|
AUC
MSP
95.00
59.66
91.25
45.21
93.80
54.57
92.12
66.45
88.50
62.46
88.64
57.67
90.86
ODIN
95.00
20.93
95.55
7.26
98.53
33.17
94.65
56.40
86.21
63.04
86.57
36.16
92.30
Energy
95.00
54.41
91.22
10.19
98.05
27.52
95.59
55.23
89.37
42.77
91,02
38.02
93.05
GODIN
95.00
15.51
96.60
4.90
99.07
34,03
94,94
46.91
89.69
62.63
87.31
32.80
93.52
Mahalanobis
95.00
9.24
97.80
67.73
73.61
6.02
98.63
23.21
92.91
83.50
69.56
37.94
86,50
KNNN
95.00
24.53
95.69
25.29
95.96
25.55
95.26
27.57
94.71
50.90
89.14
30.77
94.15
CSI
95.00
37.38
94.69
5.88
98.86
10.36
98.01
28.85
94,87
38.31
93.04
24.16
95.89
SSD+
95.00
1.51
99.68
6.09
98.48
33.60
95.16
12.98
97.70
28.41
94.72
16.52
97.15
KNN+
95.00
2.42
99.52
1.78
99:48
20.06
96.74
8.09
98.56
23,02
95.36
11,07
97.93
ZODE MSP
95.04
52.44
92.86
15.11
97.62
30.98
95.63
43,16
94.68
43.58
94,35
37.05
95.07
ZODE Energy
95.07
50.05
92.26
3,12
99.29
16.03
97.09
37.34
95.
19.52
96.95
25.21
96.15
ZODE Mahulanobis
94.99
18.24
96.30
6.28
98.48
7.17
98.55
3.88
99.12
72.25
85,93
21.56
95.68
ZODE KNN
94.96
2.12
99.43
1.50
99.61
5.48
98.70
0.16
99.88
9.91
97.99
3.83
99.12
Netzer et al. 2011). LSUN Yu et al. 2013), iSUN (Xu et al. 2015), Texture (Cimpoi al. 2014),
Places365 (Zhou et al., 2017). ). and CIFAR 100 (Krizheskky et al., 2009). We then consider more challenging benchmarks based on ImageNet, i.e., large-scale OOD detection tasks. The ID data is ImageNet- 1K (Deng et al. 2009). We evaluate OOD detectors on four test datasets that are subset of: Places365 (Zhou al. 2017). iNaturalist (Van Horn al. (2018). SUN (Xiao et al. 2010), and
Texture (Cimpoi et al. 2014) with different categories of each other.
We evaluate OOD detection methods by the following three metries: (1) the true positive rate of the ID samples (TPR); (2) the false positive rate of OOD samples when the true positive rate of the ID samples is about 95% (FPR); (3) the area under the receiver operating characteristic curve (AUC). For single-model detectors, the hard threshold is determined by TPR 95%. Therefore, the first metric aims to check whether our ensemble scheme can maintain the TPR level close to 95%. FPR and AUC are often used in the literature to reflect the capabilities of OOD detectors. For the AUC metric, we use grid values of TPR ranging from 0 to with a gap of 0.0005 and obtain the corresponding FPR to compute the area under the receiver operating characteristic curve.
Metrics:
Enhanced OOD detection: We consider three OOD detection methods: MSP (Hendrycks & Gim- 2017), Energy (Liu et al., (2200) and KNN (Sun et al. 2022). MSP is simple baseline method that uses maximum softmax probabilities as the detection score. In some experiments, MSP can yield surprisingly good results when used on top of large pre-trained model that has been fine- tuned on the ID data (Fort et al. (02T). The energy-based model (LeCun et al., (006) maps a test input scalar that higher for OOD samples and lower for the training data. Liu al. (2020) proposes an energy score that uses the logits output by pre-trained classifier. [Sun et al. (0022) uses the feature distance between the test input and the k-th nearest ID sample and proposes KNN- based detector. These three OOD detection methods represent three kinds of detectors based on probability, logit, and distance, respectively. We take them as the baseline methods and denote our enhanced methods by 'ZODE-MSP", 'ZODE-Energy', and 'ZODE-KNN' respectively.
4.1 EVALUATION ON CIFARIO BENCHMARKS
Model Zoo. We build a model zoo with seven pre- tre-triind models: ResNet18, ResNet34, ResNet50, ResNet101, ResNet152 (He et al. 2016), DenseNet (Huang et al.) (017) and ResNet18* (Sun al.) 2022). Here ResNet and Net are two backbones routinely used in the literature on OOD detection. Therefore, we consider different architectures and use six models trained by cross-entropy loss. In addition, also notice the effect of the loss function and introduce the model ResNet18 which is trained with contrastive loss. In summary, our model zoo contains diversity derived from different architectures and different training strategies.
ZODE maintains TPR. According to Section 3.1 one of the challenges of ensembling OOD detec- tors is control the true positive rate of the ID data. Theorem I states that if different pre-trained models learn completely different features, ZODE can keep TPR close to the target level. In Table report the empirical TPR of ZODE, which is close to the target level 95%.

---

Under review as a conference paper at ICLR 2023
Table Results CIFARIO. We compare the ZODE-KNN detector with the single-model KNN
detector. All values are percentages. & indicates smaller values are better and vice versa.
OOD Dataset
Method
SVHN
LSUN
iSUN
Texture
Places365
Average
TPR
FPRL
AUC
FPR
AUC
FPR
AUC
FPR.
AUC
FPR.
AUC1
FPR
AUC1
ResNet18
95.00
27.97
95.49
18.50
96.84
24.68
95.52
26.74
94.97
47.95
90.02
29.17
94.57
ResNet18*
95.00
2.42
99.52
1.78
99.48
20.06
96.74
8.09
98.57
22.82
95.32
11.03
97.93
ResNet34
95.00
26.53
95.85
10.22
98.39
29.45
95.15
31.65
94.53
36,59
92.75
26.89
95.33
ResNet50
95.00
17.31
97.40
7.10
98.83
17.32
97.26
20.85
96.59
41.35
91.61
20.79
96.34
ResNet101
95.00
25.73
96.12
6.65
98.90
96.80
18.42
40.57
92.15
22.24
96.17
ResNet1 52
95.00
34,96
94,98
7.22
98.88
22.30
96.66
20.76
96.60
38.57
92.36
24.76
95,90
DenseNet
95.00
10.22
98.18
7,90
98.60
10.87
97.94
20.78
96.25
50.14
88.92
19,98
95.98
ZODE-KNN
94.96
2,12
1,50
99.61
5.48
98.70
0.16
99.88
9.91
97.99
3,83
99.12
(a) ResNet18
(b) ResNet34
(c) ResNet50
(d) ResNet101
(e) ResNet152
(l) DenseNet
(g) ResNet18
Figure 1: Places365. Example OOD images that only one single-model detector can identify.
ZODE-KNN achieves superior performance. We compare our method with competitive OOD detection methods, including MSP (Hendrycks & Gimpel| 2017). ODIN (Liang al., (2088), Energy (Liu et al. 2020). GODIN (Hsu et al. 2020). Mahalanobis (Lee et al. 2018), KNN (Sun et al. 2022), CSI (Tack al. 2020), SSD+ (Sehwag et al.) 2020). as well as KNN+ (Sun et al. 2022). We cite the results of the competitors reported in [Sun et al. (2022). For a fair comparison, we set ZODE-KNN which is the same as Sun et al. (2022). We can find that compared to the best baseline KNN+, ZODE-KNN reduces the FPR from 11.07% to 3.83%, which significantly improves the relative detection accuracy by 65.40%. Note that ZODE-KNN significantly reduces FPR when OOD samples drawn from iSUN, Texture, and Places365. For LSUN, ZODE-KNN slightly improves the performance of KNN+. In addition, SSD+ outperforms ZODE-KNN SVHN. Overall, ZODE-KNN significantly improves the performance of existing methods on these five OOD datasets.
= 50 in the experiments
ZODE achieves consistent improvements. We consider three different kinds of OOD detection scores. MSP (Hendrycks & Gimpel, (017) is based on the probabilities, Energy (Liu et al.) 2020)
uses the logits, and Mahalanobis (Lee al. 2018) and KNN (Sun et al. 2022) quantity the distance in the embedding space. Then we compare them with the corresponding enhanced detectors: ZODE-
and PODE-KNN, For ZODE MSP, ZODE-Energe we use the same settings as Hendrycks & Gimpel (2017) andTiu et al. (2020). We find that ZoDF-enhanced detectors consistently improve the performance of the corresponding baselines (Table D..
MSP, ZODE-Energy
and
ZODE leverages the complementarity between the single-model detectors. Table 2 reports the results of all single-model detectors derived from our model zoo and KNN score. It is easy to that the ZODE-ensemblee KNN detector significantly outperforms all single-model KNN detectors on LSUN, iSUN, Texture, and Places365. Compared with the best single-model baseline. ZODE reduces the FPR from 11.03% to 3.83%, which significantly improves the relative detection accuracy by 68.28%. Moreover, ZODE improves the performance sharply Texture and Places365. This

---

Under review as a conference paper at ICLR 2023
Table 3: Results on CIFARIO for CIFARI00 as OOD, The results of GRAM and MaSF are from Haroush et al. (022). We cite the results of SSD and SSD+ reported in (Sehwag et al. (020). All values are percentages. & indicates smaller values are better and vice versa.
(a) Comparison with baseline methods
(b) Ensembled vs Single-model
Method
TPR
FPR.
AUCT
Method
TPR
FPR.
AUC1
GRAM
95.00
51.00
83.30
ResNet18
95.00)
52.24
89.69
MaSF
95.00
58.20
86.10
ResNet18*
95.00
38.83
92.75
SSD
95.00
50.78
90.63
ResNet34
95.00
46.74
91.04
SSD+
95.00
38,50
93.40
ResNet50
95.00
47.14
90.64
KNN
95.00
52.54
89.69
ResNet101
95.00
47.07
90.87
KNN+
95.00
38.83
92.75
ResNer152
95.00
47.72
90.84
ZODE-KNN
94.96
18.29
97.12
DenseNet
95.00
49.43
89.80
ZODE-KNN
94.96
18.29
97.12
implies that the superior performance ZODE does not fully come from any single-model detector. Therefore, our ensemble procedure works and is necessary for the improvements.
We further take Place365 as an example to illustrate that ZODE exploits the diversity of multiple
pre-trained models. At step 9 of Algorithm if P(1) I (1 TPRg) and Pt)) 21 TPRg), Vj 2 2, then there is only one pre-trained model that can help to identify the test input as an OOD sample. Figure [ presents seven such images and each image corresponds to one pre-trained model in our model Zoo.
Evaluations on CIFARIO vs CIFAR100. We consider a challenging OOD detection task that iden- tifies OOD samples drawn from CIFARI00 when the ID data is CIFARIO. Table 3a summarizes
detailed comparison with GRAM (Sastry Oore 2019), MaSF (Haroush et al.) 2022). SSD (Se-l hwag et al., 2020), and KNN (Sun 2022). Compared with the best baseline SSD+, ZODE reduces the FPR by 20.21%. which is relative 52.49% improvement in detection power. The re-
sults in Table 3h clearly show that ZODE significantly outperforms the ingle-model-barte KNN detectors and our ensemble scheme fully leverages the complementarity between the single-model detectors.
4.2 EVALUATION ON IMAGENET BENCHMARKS
Model zoo and implementation details. We use five pre-trained models to build model zoo, con- sisting of models with different architectures and different pre- pre-training strategies. The models are as follows: ResNet50 (Sun et al. 2022), semi-weekly supervised ResNeXt101 32x16d (Yalniz al.) 2019). Swiny2-B256, Swinv2-B338 and Swinv2-L226 (Liu et al. 2022). Significantly, resolutions of Swiny2-B256 -w84 and Swinv2-L255 are 256x256. 256x256, and 384x384 respec- tively. ResNet50* is trained with SupCon loss (Khosla et al. 2020). which pulls points belonging to the same class together in the embedding space and separates samples from different classes. ResNeXt101 is pre-trained on Billion-scale images associated with meta information semantically relevant ImageNet, which achieves 84.8% top-1 accuracy on ImageNet. The three Swinv2 models are pre-trained at higher resolution, and their top-1 accuracy on Imagenet all exceed 84%. In the following, we only report the results of ZODE-KNN based on the model zoo. The hyperparameter TPR, is taken to be 93.50%, which makes the empirical TPR of ZODE -NN close to 95%. We use k 1000 for ResNet50*. which same Sun et 2022).. For the rest models, we selected from {100, 200, 500, 700, 800, 900, 1000, 3000, 5000} that minimize the FPR.
KNN achieves superior performance. In Table 4 we compare ZODE-KNN with com- petitive OOD detection methods, including MSP (Hendrycks. & Gimpel (017)) ODIN (Liang al.) 2018 Energy (Liu et al. 2020), GODIN (Hsu et al.) (020)). Mahalanobis (Lee et al. 2018), KNN SSun et al. 2022), SSD+ (Sehwag et al. 2020), well as KNN+ (Sun et al. 2022). ZODE-KNN outperforms the best baseline KNN+ uniformly on all four OOD datasets, substantially reducing the
average FPR from 38.47% to 28. 10%, which achieves relative 26.96% improvement in detection
power. Especially when test datasets are iNaturalist and Textures, ZODE-KN KNN reduces the relative
FPR by 83.40% and 70.61% 61% respectively, which highlights the effectiveness of ZODE.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Results on ImageNet. All results of the competitors are cited from Sun al. (0022)
Methods reported are all based on ID data only (ImageNet-1k, All values are percentages. indicates smaller values are better and vice versa.
OOD Dataset
Method
iNaturalist
SUN
Places
Textures
Average
TPR
FPR
AUCT
FPR
AUCt
FPR.
AUCT
FPR.
AUC1
FPR,
AUC1
MSP
95.00
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
ODIN
95.00
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85.62
56.48
85.41
Energy
95.00
55.72
89.95
59.26
85.89
64.92
82.86
53.72
85.99
58.41
86.17
GODIN
95.00
61.91
85.40
60.83
85.60
63.70
83.81
77.85
73.27
66.07
82.02
Mahalanobis
95.00
97.00
52.65
98.50
42.41
98.40
41.79
55.80
85.01
87.43
55.47
KNN
95.00
59.00
86.47
68.82
80.72
76.28
75.76
11.77
97.07
53.97
85.01
SSD+
95.00
57.16
87.77
78.23
73.10
81.19
70.97
36.37
88.52
63.24
80.09
KNN+
95.00
30.18
94.89
48.99
88.63
59.15
84.71
15.55
95.40
38.47
90.91
ZODE KNN
94.89
5.01
98.60
48.87
90.37
53.96
88.07
4.57
98.93
28.10
93.99
Table 5: Results ImageNet. Comparison with single-model detectors and ZODE, All values percentages. | indicates smaller values are better and vice versa.
OOD Dataset
Method
iNaturalist
SUN
Places
Textures
Average
TPR
FPR
AUCT
FPR.
AUC1
FPR
AUC
FPR
AUC1
FPR.
AUC1
ResNet50
95.00
30.18
94.89
48.99
88.63
59.15
84.71
15.55
95.40
38.47
90.91
ResNext101 32x16
95.00
15.24
96.78
56.06
88.60
61.74
86.29
26.06
93.53
39.78
91.30
Swiny2-B256
95.00
9.11
97.93
58.16
88.78
58.66
87.13
41.24
89.67
41.79
90.88
Swinv2-B B384
95.00
5.65
98.50
49.59
90.28
52.27
88.44
38.37
89.99
36.47
91.80
Swinv2-1.256
95.00
6.98
98.44
52.43
89.49
53.81
88.07
39.26
89.92
38. 12
91.48
ZODE-KNN
94.89
5.01
98.60
48.87
90.37
53.96
88.07
4.57
98.93
28.10
93.99
(a) R
(b) ResNeXtt111
(c) Swinv2-B256
(d) Swiny2-B384 (c) Swiny2-1.256
Figure 2: Textures. Example OOD images that only one modlll detector identify.
ZODE combines the advantages of the single-model detectors. In Table 5 we report the per- formance of every single-model detector derived from our model zoo. We highlight three trends: (1) ZODE-KOA KNN outperforms the best single-model KNN detector with a relative 22.95% improve- ment in FPR. This implies that ZODE works the ImageNet benchmarks and the ensemble scheme of ZODE-KNN is necessary for the improvements. (2) ZODE combines the advantages of single- model detectors. Table 55] we can observe that ResNet50* and ResNeXt101 32x16 perform well on Textures, but underperform mance. However, the ZODE-ensembled detector achieves strong and stable performance all test datasets. (3) ZODE leverages the complementarity between the single- model detectors. Similar to the discussions Figure [] we find some images in Textures that can be successfully identified OOD samples and the detection decision depends only presents five such images and each image corresponds to one pre-trained model in our model zoo.
iNaturalist, while the Swin models show the opposite perfor-
single-model detector. Figure
5 CONCLUSION
In this paper, we exploit the diversity of multiple pre-trained models in model zoo to improve the performance of post hoc OOD detection. We propose, ZODE, an efficient and fundamental ensemble scheme for combining multiple detection decisions. Extensive experiments show that ZODE can effectively solve the missed detection problem the complementarity of multiple detectors. We find that ZODE combined with the KNN detector (Sun et al.] 2022) works very well. On a wide range OOD detection benchmarks, ZODE-KNN significantly improves the current SOTA results.
mogle-model detectors by exploiting

---

Under review as a conference paper at ICLR 2023
REFERENCES
Felix Abramovich and Ya' acov Ritov. Statistical theory: a concise introduction. CRC Press, 2013.
Jerone Andrews, Thomas Tanay, Edward Morton, and Lewis D Griffin. Transfer representation- learning for anomaly detection. JMLR, 2016.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1893-1902, 2015.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
Yoav Benjamini and Daniel Yekuticli. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pp. 1165-1188, 2001.
Federico Bergamin, Pierre-Alexandre Mattei, Jakob Drachmann Havtor,, Hugo Senetaire, Hugo Schmutz, Lars Maalóe, Soren Hauberg, and Jes Frellsen. Model-agnostic out-of -ift----iittibttion de- tection using combined statistical tests. In International Conference on Artificial Intelligence and Statistics, pp. 10753-10776. PMLR, 2022.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pullern Recognition, 94:317-331, 2018.
Feiyang Cai and Xenofon Koutsoukos. Real-time -utto--distribution detection dearning-enablle cyber-physical systems. In 2020 ACM/IEEE 1 Ithh International Conference Systems (ICCPS), pp- 174-183. IEEE, 2020.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey, arXiv preprint 1rXiv: 003407,,0 2019.
Zhiyi Chi. On the performance of fdr control: constraints and partial solution. The Annals of Statistics, 35(4):1409-1431, 2007.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248.255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openzeview. net http:n/ee//oommmd/iiiiiiioee. CNNTY
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5414 (443,,22,,4 2021.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-o--disttibution detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Matan Haroush. Tzviel Frostig, Ruth Heller, and Daniel Soudry. A statistical framework for efficient out of distribution detection in deep neural networks. In International Conference on Learning Representations, 2022.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 41- 50, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributiin examples in neural networks. Proceedings of International Conference on Learning Representa- tions, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2018
Yen- Ven-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from oft-o--distribution data. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
Ramneet Kaur, Susmit Jha, Anirban Roy, Sangdon Park, Edgar Dobriban, Oleg Sokolsky, and Insup Lee. idecode: In-distribution equivariance for conformal out-of out-of-distribution detection. Proceed- ings of the AAAI Conference on Artificial Intelligence, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 136611661,,77, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji sakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc' Aurelio Ranzato, and Fu Huang. A tutorial on energy-based learning, 2006.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence- classifiers for detecting out-of-distribution samples, arXiv preprint arXiv:1711.09322 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting nut-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability duitof-disttibutionn image detec- tion in neural networks. In International Conference on Learning Representations, 2018.
Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribution detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15313- 15323, 2021.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. Advances in Neural Information Processing Systems, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 10012-10022, 2021.

---

Under review as a conference paper at ICLR 2023
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In International Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Akshayaa Magesh, Venugopal V Veeravalli, Anirban Roy, and Susmit Jha. Multiple testing frame- work for out-of-distribution detection. arXiv preprini arXiv:2206.09522, 2022.
David Miller, Zhen Xiang, and George Kesidis. Adversarial learning targeting deep neural network classification: A comprehensive review of defenses against attacks, Proceedings of the IEEE, 108 (3):402 433, 2020.
Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. In International Conference on Artificial Intelligence and Statisties, pp. -3240200. PMLR, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
AristotelisAAngelos Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang- Outlier exposure with confidence control for out-of-distribution detection. Neurocomputing, 441: 138-150, 2021.
Mohammad Sabokrou, Mohsen Fayyaz, Mahmood Fathy, Zahra Moayed, and Reinhard Klette. Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Computer Vision and Image Understanding, 172:88- 97, 2018.
Chandramouli S Sastry and Sageev Oore. Zero-shot distii-iittiitttion detection with feature cor- relations. 2019.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Frfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference information processing in medical imaging, pp. 146- 157. Springer, 2017.
Vikash Sehwag, Mung Chiang, and Prateck Mittal. Ssd: A unified framework for self supervieed outlier detection. In International Conference on Learning Representations, 2020.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activa- tions. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-disribbution detection with deep nearest neighbors. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 00827-20840. PMLR, 17-23 Jul 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11839 1152. 2020.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Grant Van Hom, Oisin Mae Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7769-8778, 2018.
Volodya Vovk. Alexander Gammerman, and Craig Saunders. Machine- applications of al- gorithmic randomness. In Proceedings of the Sixteenth International Conference on Machine Learning, pp. 444 453, 1999.
12

---

Under review as a conference paper at ICLR 2023
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L Willke. dustt---distibuttinn detection using an ensemble self supervised leave-out classifiers. In Proceedings of the European Conference Computer Vision (ECCV), pp. 550-564, 2018.
Haoqi Wang, Zhizhong Li, Litong Feng. and Wayne Zhang. Vim: logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4921-492299 2022.
with virtual-
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWillams,, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, al. Contrastive training for improved out-of-distribution detection. arXiv preprini arXiv:2007.05566 2020.
Jianxiong Xiao, J Hays, KA Ehinger, A Oliva, and A Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint N5l::1504.06755. 2015.
Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv:/905.0000 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: 506. 506.:36500.03365, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6)) 2017.
A PROOF OF THEOREM
Theorem. Suppose a pre-trained model zoo {61, 02,..., dm 1 is accessible and the score function is S(x; Φ). Let TPR, > 0.5 be the target TPR level for the ID data. If the test input x" is ID sample that x* ~ Dia and Sxx"; 6s) is independent of S(x*, ; 05) for Vi # j. then Algorithm can identify x"
ID data with probability larger than T TPRo
Proof. Before proving the theorem, we state a useful lemma that provides the distribution of p-values under the ID distribution.
Lemma Suppose the test input x" an ID sample that x" ~ Dia and the detection score S(x"; is continuous random variable. We write the p-value of x" as
Po P(S(x; o) & S(x"; $) ~ Dia).
Then Po follows the uniform distribution U(0. 1].
Proof. Let Fs(s) be the cumulative distribution function of S(x; σ) with x ~ Dia. Then,
Po P (S(x Φ) S(x"; p) Dia) = Fx(S(x*; +)).
By the continuity of ;(x*: and Lemma 21.1 of Van der Vaart 2000), we have
P(po < <a) = 1 - (FF 2 a))
= 1 - P(S(x Φ)) 2 Fr (a)) FF = a.
Hence Po follows the uniform distribution 0 1).
13

---

Under review as a conference paper at ICLR 2023
According to Lemma 2 for any o, E M,
P - P(S((; - ; x ~ Dsa) ~ U10, 1|,
and the density function of pæ is
{u 1 [0,
fp. (x)
otherwise.
Then, the joint probability density the ordered values P(1),P(2): -.. P(m)) is
$
Pes.. (I1, T2, ..., Tm.))..)) mm!
Sp. (2:) = m!
i=1
We denote a 1 TPRo and define event &: V1 S m, PU) 2 La Then have
m
P(E|x Dza)= f
ppnn. P(m) (2,, Im Emdddydy . drm
P(m)
2a
(4)
2
ml(11
a1
a) a).
m
m
m
Next, we prove that for any m 2 and < 0.5,
1
m
ml
a1 a) a)...(1
a 1-a
(5)
m
m
m
It is easy to see that Eq. m ma + 1, we have
5 holds when m 1. Suppose Eq.
holds for m = mm. Then for
1
m0 + 1
(mo + 1)!(1
a1
mo + a)...(1
a) mo +
mg +
2(mo + 1)!(1 a)((
1
2
a.... m0
mu
a)
mo 1 a))
+ 1 my
(6)
mg
mo
2(mo + 1)(1 2 2 1 - a.
which implies that 55) also holds for m = mo + 1. Hence, the proof is finished.
B THE POWER OF ALGORITHM
In this section, we study the FPR of Algorithm from the asymptotic perspective, i.e., m tends to infinity. According to Section 3.2 and Appendix A the p-value of ID sample follows the uniform distribution [0, 1]. Also, if a pre-trained model fails to identify OOD samples, the p-value derived from the model follows the uniform distribution. For an OOD dataset, we assume that there is fixed proportion of pre-trained models that can recognize the OOD data points. We call these models active models and denote the set active models as A. Then pre-trained models that fail to identify OOD samples belong to the set A M - A. We have, for any 0 S u < I.
P(p; S ulo E A
and P(P, I u/o E A) G(u),
where G(u) is cumulative distribution function different from that of the uniform distribution U(0, 1]. Therefore, the p-values of the OOD dataset are sampled from a mixture model with a cumulative distribution function:
F(u) (1 TT + #G(u).
Let be the number of pre-trained models that classify the OOD input an OOD sample. That is L La, Vj > It with 1 -TPR, Table summarizes the OOD detection
P(a S m
a and P() >
14

---

Under review as a conference paper at ICLR 2023
Table 6: OOD detection.
Truth
ID
OOD
ID
U
Detect
0OD
V
result with Algorithm 1
S 2 1, Algorithm
successfully detects an OOD sample. Next, we
study the average power:
S m1
k
g I
(1
E
E
m
m2
m
converges to a positive value (a, F) as m - oo, which serves as the limit of the proportion of rejected p-values. By Theorem and its lemma in Benjamini & Hochberg 1995).
According to Chi 2007
E
7700 = - r).
k
Therefore, if m is sufficiently large, then we have
S 7781
pa(a, (1
E
7711
This implies that if m is sufficiently large, S is greater equal to with high probability. In other words, the number of pre-trained models sufficiently large, Algorithm can identify OOD samples with high probability.
C USING P-VALUE FOR OOD DETECTION
In OOD detection, the p-value is probability measure that quantifies how extreme the observed
score is when the input is an ID sample (Cai & Koutsoukos 2020 Morningstar et al. 2021 Haroush
al. 2022 Bergamin al. 2022 Magesh al. [022]] Kaur al. 2022). Given test sample x",
the lower value of S(x"; d), the more likely x" is not drawn from the training distribution. Hence, the p-value of x" is the probability that S(x: o) is less than S(x";
o) under the ID distribution:
P-value xf (S Φ) S S(x"; o) Día).
In practice, using the p-value is equivalent to using the hard threshold S(x*) < X. We denote {(x, y, }= as validation data sampled from the ID distribution Pis and sort their detection score in ascending order:
S(xn)) x) S S(x(2)) x) S S {((((n))))) Ø).
In post hoc OOD detection, the threshold A is determined by correctly classifying 95% of validation data as ID samples, i.c., TPR at least 95%. Therefore,
((x00)) Φ) s As S (110:/.00m=+1100000 Ø).
where [+] is the floor function. On the other hand, the p-value of x" less than 0.05 implies that
p(S(x;0) <
D.a) s 0.05 + S(x"; ()) $ S(x(
where Dia is the empirical distribution of (x, 1 Therefore, when the sample size is sufficiently large, the OOD region derived from the critical value {x S(x; ø) < Aa is the same as the OOD region determined by the p-value (x P-value of 0.05).
15

---

Under review as a conference paper at ICLR 2023
Table 7: Results on CIEARIO. Comparison with three baseline ensemble schemes. All values are percentages. & indicates smaller values are better and vice versa.
OOD Dataset
Method
SVHN
LSUN
ISUN
Texture
Places365
Average
TPR
FPR.
AUC1
FPR.
AUC
FPR.
AUC
FPR.
AUC1
FPR
AL'1
FPR.
AUC1
Naive
69.89
0.16
99.18
0.12
99,39
0.53
98.21
0.00
99.74
0.14
97.36
0.95
98.78
Averuge
100.00
19.21
99.98
6.92
100.00
20.04
99.98
23.69
100.00
69.76
99.97
27.92
99.98
Voting
99.64
5.43
99.82
2.46
99.95
6.68
99.82
0.39
99.99
8.99
99.86
4.79
99.89
BH
94.96
2.12
99.43
1,50
99.61
5.48
98.70
0.16
99.88
9.91
97.99
3.83
99.12
Table 8: Results on ImageNet. Comparison with three baseline ensemble schemes. All values are percentages. 4 indicates smaller values are better and vice versa.
OOD Dataset
Method
iNaturalist
SUN
Places
Textures
Average
TPR
FPR4
AUC1
FPR.
AUC
FPR.
AUC1
FPR.
AUC1
FPR.
AUCT
Naive
88.53
2.00
98.54
29.53
89.60
36.72
86.93
1.45
99.00
17.43
93.52
Average
97.41
11.64
98.57
58.21
91.43
61.40
89.13
44.77
93.55
44.05
93.15
Voting
93.59
3,99
98.66
43.16
90.26
48.18
87.98
11.74
97.14
26.76
93.51
BH
94.89
5.01
98.60
48.87
90.37
53.96
88.07
4.57
98.93
28.10
93.99
D USING BH PROCEDURE FOR ENSEMBLE
In this work, we use the Henjaminii-Hocheee procedure (Benjamini & Hochberg 1995 to ensemble multiple detection decisions (p-values) to exploit the diversity of pre-trained models. Please see Steps 7 13 Algorithm D Recall that TPR, is the target TPR level and = - TPRg- According to Theorem L this procedure controls the true positive rate of ID data at a level greater than TPR, by assuming independent p-values. [ennjamini & Yikutieli (201T) points out that the procedure is also available for cases with certain types of positively related p-values.
In this section, we consider three ensemble schemes, Naive, Average and Voting, as competitors to illustrate the superior of the BH procedure. Here 'Naive' represents the naive ensemble scheme in Eq (2) 'Average' refers to the ensemble scheme that identifies a test input as an OOD sample if the average p-value is smaller than a, and 'Voting' is the majority voting that identifies a test input as an OOD sample if more than half of the p-values are less than a. The score function is the KNN score (Sun al. 2022) with & 50 and TPR, 95%. The results are presented in Table and 8 We can find that the empirical TPR of our method is well-controlled and is close to the target TPR level. The naive ensemble scheme cannot maintain the target TPR level. Therefore its low FPR is unreliable. This observation is consistent with our theoretical results in Section 3.1 The average ensemble scheme can maintain its empirical TPR larger than 95% but its FPR very large. Overall, the voting ensemble is comparable to our scheme. Its TPR on ImageNet is bit out of control.
E INFLUENCE OF HYPERPARAMETER
we find that the ZODE-KNN detector works very well and significantly improves the best baseline method on a wide range of OOD detection benchmarks. The KNN detector (Sun) al. (002) uses the distance between the test input and the k-th nearest ID sample as the detection score. Here k is hyperparameter that needs to be predetermined. In this section, we use CIFARIO to investigate the effect of the choice of & on ZODE. We consider results in Table One can find that the choice of k affects the performance of ZODE-KNN But the influence is not significant compared to the improvements of ZODE-KNN Both (ODE-KNN(K=I) we report
In Secction 4
and 50, and report the
and ZODE-KNN(K=50) significantly outperform the best baseline methods. In Table TO the detailed comparison between the ZODE-ensembled KNN detector and the single-model KNN detectors derived from our model zoo. We observe similar phenomenon in that the choice of k affects the performance of both ensembled and single-mode detectors, while the influence is not significant compared to the improvement caused by the ensemble scheme.
16

---

Under review as a conference paper at ICLR 2023
Table Results on CIFARIO. Comparison with competitive OOD detection methods. The results
of all competitors are from Sun et al. 2022). All values are percentages. y indicates smaller values are better and vice versa.
OOD Dataset
Mothod
SVHN
LSUN
ISUN
Texture
Places365
Average
TPR
FPR
AUCT
FPR
AUC1
FPR
AUCT
FPR
AUCT
FPR
AUCt
FPR
AUC1
MSP
95.00
59.66
91.25
45.21
93.80
54.57
92.12
66.45
50
62.46
88.64
57.67
90.86
ODIN
95.00
20.93
95.55
7.26
98.53
33.17
94.65
56.40
86.21
63.04
86.57
36.16
92.30
Energy
95.00
54.41
91.22
10.19
98.05
27.52
95.59
55.23
89.37
42.77
91.02
38.02
93.05
GODIN
95.00
15.51
96.60
4.90
99.07
34,03
94.94
46.91
89.69
62.63
87.31
32,80
93.52
Mahalanobis
95.00
9.24
97.80
67.73
73.61
6.02
98.63
23.21
92.91
83.50
69.56
37.94
86.50
KNN
95.00
24.53
95.69
25.29
95.96
25.55
95.26
27.57
94.71
50.90
89.14
30.77
94.15
CSI
95.00
37.38
94.69
5.88
98.86
10.36
98.01
28.85
94.87
38.31
93.04
24.16
95.89
SSD+
95.00
1.51
99.68
6.09
98.48
33,60
96.16
12.98
97.70
28.41
94.72
16.52
97.15
KNN+
95.00
2.42
99.52
1.78
99.48
20.06
96.74
8.09
98.56
23.02
95.36
11.07
97.93
ZODE-KNNI-=]]
2.60
2.34
7.29
98
0.44
97.77
4.89
ZODE-KNN(K=50
94.96
2.12
99.43
1.50
99.61
5.48
98.70
0.16
99.88
9.91
97.99
3.83
99.12
Table 10: Results on CIFARIO. Comparison with single-model detectors at different k levels. Setting and k 50 respectively, we compare the performance with single-model detectors and ZODE.
OOD Dataset
K
Method
SVHIN
LSUN
ISUN
Texture
Places665
Averuge
EPR
AUC1
FPR
AUC
EPR
AUC1
EPR
AUC
HPR
AUC1
FPR
AUC
ResNec18
95.00
50.33
91.45
25.44
95.67
30.53
54.71
33.44
93.69
48,40
50.05
37.63
93.11
ResNec18*
2.16
99.56
4.45
21.99
96.56
12.47
ResNet34
95.00
28.09
95.55
11.80
98.13
30.90
94.97
32.19
94,24
38.67
92.41
28. 19
95.06
A
ResNe150
ReaNer101
95.00
25.45
95.96
96.25
7.15
8.46
98.62
20.92
96.54
21.03
23.63
95.72
96.31
41.07
92.18
22.51
23.39
95.92
R5sNet152
95.00
33,79
94,16
8.48
98.63
19.06
96.93
20.62
96.50
37.97
92.52
23.90
95.75 75.75
DensNet
95.00
13.46
97.76
9.41
98.31
13.70
97.58
24.29
95.46
51.49
89.07
22.
95.64
ZODE KNN
95.00
2.60
99.42
2.34
99.46
7.29
98.44
0.44
99.73
11.79
97.77
4.89
98.97
ResNer18
95.00
27.97
95.49
18.50
96.84
24.66
95..2
26.74
94.97
47.95
50.02
29.17
94.37
ResNet18*
95.00
2.42
99.52
1.78
99,
20.06
96.74
8.09
98.57
22.82
95.32
11.03
97.93
ResNet34
95.00
26.53
95.85
10.22
98.39
29.45
95.
31.65
94.53
36.59
92.75
26.89
95.33
k ReaNer101
ResNet50
95.00
17.31
97.40
7.10
98.83
17.32
97.26
20.85
96.59
41.35
91.61
20.79
96.34
95.00
25.73
96.12
6.65
98.90
19.84
96.80
18.42
96.89
40.57
92,15
22.24
96.17
ResNet152
95.00
34.96
94,98
7.22
98.88
22.30
96.66
20.76
96.60
38.57
92.36
24.76
95.90
DenaNet
95.00
10.22
98.18
7.90
98.66
10.87
97.94
20.78
96.25
50.14
88.92
19.98
95.98
ZODE-KNN
94.96
2.12
99.43
1,50
99.61
5.48
98.70
0.16
99.88
9.91
97.99
3.83
99.12
F MORE RELATED WORK
Score function for OOD detection. Many recent works have proposed use models pre-trained with multi-class classification tasks to do OOD detection. A natural idea to distinguish OOD samples to use the softmax confidence score based on the classifier output and a normalized probability vector. However, neural networks might give OOD sample high confidence prediction (Guo) ct al. 2017| Hein et al., 2019). To deal with this issue, several studies have improved scoring methods and also proposed new scoring measures, such as OpenMax score (Bendale & Boult 2015), maximum softmax probability (Hendrycks & Gimpel 2017). ODIN score (Liang et al.]|2018 deep ensembles (Lakshminarayanae
al. (017). Mahalanobis distance-based score (Lee et al. 2018).
Gram matrix (Sastry & Oore, 2019). energy score (Liu et al.) 2020), (Lin et al. 2021 1). activation
rectification (ReAct) (Sun et al. 2021], gradient-based score (GradNorm) (Huang et al. 202T), and
ViM score (Wang et al. (022). In this work, we consider KNN score Sun et al. (2022) that employs the non- non-nonantee parameemeteicc nearest neighbor approach to quantify the distance between the test input and the training data.
Model-training strategies for OOD detection. Besides proposing efficient score functions, an- other area of research focuses on choosing strategies for training deep models, including generative adversarial networks (Schlegl et al. 2017). convolutional network (Sabokrou et al. 2018) and trans- fer representation--eerning (Andrews et al. 2016). These classical neural network architectures, such as ResNet (He et al. 2016). DenseNet (Huang et al., 2017) and Swin (Liu et al.] 2021) etc, show different effects in downstream tasks when trained with different training strategies regu- larization methods (Ericsson et al.) 2021). For example, utilizing auxiliary OOD training data like artificially synthesized data from GANs [Lee al. (017) and unlabeled data for outlier exposure
17

---

Under review as a conference paper at ICLR 2023
(Hendrycks al. 2018 explicitly regularize the model. Another method to fine-tune the model is to modify the loss function or use auxiliary objectives. Some loss functions encourage the pre- dictive distribution of OOD sample toward uniform distribution (Lee et al. 2017 Hendrycks al. 2018), and also some by adding contrastive loss (Winkens et al. 2020). margin loss (Vyas et al, 2018) objective function of adversarial learning (Biggio & Roli 2018 Miller et al., 2020 Cha- lapathy & Chawla, 2019) can force models to learn more high-level, task agnossic,, comprehensive features from the training dataset, to enable it robust enough for downstream tasks with various distribution shifts. These works either modify the neural network and objective function or require additional data, at the of computational cost.
18


Reviews:
[{"review_id": "X51wesfj2S", "reviewer": "Reviewer_Wjqr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the method ZODE, an ensemble scheme using pretrained models for OOD detection. The proposed scheme employs the Benjamini–Hochberg procedure to decrease the false discovery rate of OOD samples by combining the base model outputs. The authors conduct experiments on CIFAR10 and Imagenet datasets to evaluate the ZODE performance compared with SOTA ODD methods, and they show that the method improves the ODD detection performance.", "summary_of_the_review": "The trustworthiness of ML approaches is significantly related to their ability to recognize out-of-distribution samples. This is an important topic with a lot of applications. The proposed paper presents one interesting approach but fails to analyze different perspectives of the problem. The authors focus on error and do not consider the computational cost of the proposed method or the comparison with other ensemble methods. The author should discuss the proposed approach's limitations and explore the qualitative aspects of the results.", "strength_and_weaknesses": "- The proposed approach is interesting and presents promising results. \n- The idea of using an ensemble of classifiers for OOD is not new, but using the BH procedure to combine the base classifiers is clever. BH generally produces fewer Type I errors and performs best in sparse cases. What about the non-sparse cases?\n\n\n- I miss some qualitative discussions. What is the impact of the pretrained models on the results? Is there a minimum quantity of base classifiers? Do they need to be trained in some specific dataset? Is it necessary that the training dataset of the base classifiers be related to the test samples? Does the increase in the number of base classifiers improve the results? Is there a limit to that? What is the impact of adding diversity in the base classifiers in this context?\n- The authors do not consider the computational cost of using all base classifiers in the inference process when comparing the performance with the standalone SOTA methods. Could distilling the ensemble's knowledge using a distillation approach improve the process for computational cost purposes? \n- Is the approach generic to another type of data or just work with images?\n- How the SOTA methods were chosen? Are those the SOTA?\n- The authors did not discuss the limitation of the proposed method. It will be meaningful to discuss the gap between the experiments in the current version of the paper and the real-world applications.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Z2isNvqJ1", "reviewer": "Reviewer_Ty7t", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to detect out-of-distribution (ODD) samples by an ensemble of pre-trained models. For a given test sample, OOD scores are computed based on a set of pre-trained models and a suitable p-value is determined whether the sample is OOD while maintaining a given true positive rate. experiments are conducted on common OOD benchmarks such as CIFAR-10, ImageNet, and Places365.", "summary_of_the_review": "The idea of considering an ensemble of models for detecting OOD is interesting. However, the overall novelty is lacking. Please address the concerns in the weaknesses section.", "strength_and_weaknesses": "Strengths:\n\n1. Considering an ensemble of models to detect OODs is a simple yet effective idea.\n\n2. The draft is clearly written and easy to follow.\n\n3. Authors conducted extensive experiments of the standard benchmarks and the results are state of the art.\n\nWeaknesses:\n\n1. Overall novelty is limited as p-value based OOD detection is explored in conformal prediction literature.\n\n2. What is the score function used in this work? This is important and needs to be discussed.\n\n3. In OOD detection, FPR is more important than TPR. Does theorem 1 provide any guarantee on FPR?\n\n4. Authors are encouraged to discuss the computational complexity of Algorithm 1. For each test sample, it requires computing the score \nof all the training samples for all the models. This can be expensive.\n\n5. Does the OOD detection performance depend on the quality of pre-trained models?\n\n6. Some related references are missing \n\na. Cai, F.; and Koutsoukos, X. Real-time out-of-distribution detection in learning-enabled cyber-physical systems. ICCPS, 2020: \nb. Kaur, R. et al., iDECODe: In-distribution equivariance for conformal out-of-distribution detection. AAAI, 2022\n\n(a) considers p-value based conformal scores for OOD detection.\n(b) proposed an approach to combine multiple p-values with a bound on TPR. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "44H0qjVm9qR", "reviewer": "Reviewer_FQTf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to ensemble multiple pre-trained models with different architecture by estimating and thresholding p-value for out-of-distribution detection. Experimental results show the effectiveness of the proposed method.\n", "summary_of_the_review": "I am mostly happy with this paper, except for missing ablation study on the choice of ensembling strategy. Again, I believe ensemble method almost always improves the performance of machine learning models, so comparison with other ensemble methods should be more reasonable than standalone models. Please answer my concerns above.\n\n\n**post-rebuttal**\n\nThe additional experiments for comparison with simple ensemble methods addressed my concern, but it turned out that the performance of the proposed method is not so better than baseline ensemble methods, in terms of AUC. Ensembling based on the p-value and corresponding analysis are interesting, but by looking at the performance, I can't see the reason why we need such a sophisticated ensemble strategy.\nAt this point, I decided not to change my initial rating.\n\nBy the way, authors claim that TPR is \"out of control\" and show FPR under somewhat random TPR for other ensemble methods, which I do not fully understand what they actually mean by. Maybe authors observed discrete changes on TPR?\n\nIt seems authors also count the fact that their work is first to take advantage of \"(publicly available) model zoo\" for OOD detection as the main contribution, but by looking at other reviews, it seems other reviewers do not care about it much?", "strength_and_weaknesses": "Strengths\n\n+ Theoretical analysis on the p-value looks interesting.\n\n+ The proposed method clearly outperforms other methods in most cases.\n\nWeaknesses\n\n- No ablation study on the ensembling strategy. For example, authors could simply average the detection score or do majority voting and take them as baselines. \n\n- The comparison is not fair. Model ensembling is a kind of guaranteed strategy to improve the performance in machine learning. Even authors showed consistently good performance, empirical contribution is somewhat limited in that sense. Rather, comparison with other (baseline) ensemble strategies should be fair, but missed.\n\n- Is there a reason why the proposed method is combined with MSP, Energy, and KNN only? I wonder if the rank of methods holds, e.g., if ZODE-Mahalanobis is on average worse than ZODE-KNN. Authors could add more results in the appendix.\n\n- Typo: DensNet @ Table 2", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject