ExCeL: Combined Extreme and Collective Logit Information for Out-of-Distribution Detection
Authors: Naveen Karunanayake
Suranga Seneviratne
Sanjay Chawla
Venue: Accepted by TMLR ()

Abstract: Deep learning models often exhibit overconfidence in predicting out-of-distribution (OOD) data, underscoring the crucial role of OOD detection in ensuring reliability in predictions. Among various OOD detection approaches, post-hoc detectors have gained significant popularity, primarily due to their ease of implementation and competitive performance. However, recent benchmarks for OOD detection have revealed a lack of consistency in existing post-hoc methods. This inconsistency in post-hoc detectors can be attributed to their sole reliance either on extreme information, such as the maximum logit, or on collective information (i.e., information spanned across classes or training samples) embedded within the output layer. In this paper, we propose ExCeL, which combines both extreme and collective information within the output layer for enhanced and consistent performance in OOD detection. We leverage the logit of the top predicted class as the extreme information (i.e., the maximum logit), while the collective information is derived in a novel approach that involves assessing the probability of other classes appearing in subsequent ranks across various training samples. Our idea is motivated by the observation that, for in-distribution (ID) data, the ranking of classes beyond the predicted class is more deterministic compared to that in OOD data. Experiments conducted on CIFAR100, ImageNet-200, and ImageNet-1K datasets demonstrate that ExCeL consistently is among the five top-performing methods out of twenty-one existing post-hoc baselines when the joint performance on near-OOD and far-OOD is considered (i.e., in terms of AUROC and FPR95). Furthermore, ExCeL shows the best overall performance across all datasets, unlike other baselines that work best on one dataset but have a performance drop in others.

Published in Transactions on Machine Learning Research (01/2025)
ExCeL: Combined Extreme and Collective Logit Information for Out-of-Distribution Detection
Naveen Karunanayake School of Computer Science The University of Sydney
Suranga Seneviratne
wwwangg..emeeoinnneeeayyyyy Ssydee edu.
School of Computer Science The University of Sydney
Sanjay Chawla
schawla@hbbkus wew.:bbbuuuuuu...... qu
Qutar Computing Research Institute, HBKU
Reviewed on OpenReview: https: // openreview. net/ forum? id= 4X2OWBA
Abstract
Deep learning models often exhibit overconfidence in predicting nut-of-distribution (OOD) data, underscoring the crucial role of OOD detection in ensuring reliability in predictions. Among various OOD detection approaches, post-hoc detectors have gained significant pop- ularity, primarily due to their case of implementation and competitive performance. How- ever, recent benchmarks for OOD delection have revealed a lack of consistency in existing post-hoc methods. This inconsistency in post-hoc detectors can be attributed to their sole reliance either on entreme information, such as the maximun logit, or on collective infor- mation (i.c., information spanned across classes or training samples) embedded within the output layer. Tn this paper, we propose FxCeT., which combines both extreme and collec- tive information within the output layer for enhanced and consistent performance in OOD detection. We leverage the logit of the top predicted class the extreme information (i.e., the maximum logit), while the collective information is derived in novel approach that involves assessing the probability of other classes appearing in subsequent ranks across vari- ous training samples. Our idea is motivated by the observation that, for in-distribution (ID) data, the ranking of classes beyond the predicted class is more deterministic compared to that in OOD data. Experiments conducted on CIFAR100, ImageNet-200, and ImageNet-IK IK datascts demonstrate that ExCeL consistently is among the five top-performing methods out of twenly-one existing post-hoc baselines when the joint performance on near-OOD and for is considered (i.e., in terms of AUROC and FPR95). Furthermore, ExCeL shows the best overall performance across all datasets, unlike other baselines that work best on one dataset but have a performance drop in others.
1 Introduction
Deep neural networks (DNNs) deployed in the open world often encounter diverse range of inputs from unknown classes, commonly referred to as out-of-distribation (OOD) data. However, DNNs' tendency to be overly confident, yet. inaccurate about such inputs makes them less reliable, particularly in safety-crilical applications such as autonomous driving (Filos et al. 2020) and healthcare (Roy et al. (022). Therefore, DNN should be able to identify and avoid making predictions on these OOD inputs that differ from its training data. For instance, in an autonomous vehicle, the driving system must promptly alert the driver and transfer control when it detects unfamiliar scenes objects that were not encountered during its

---

Published in Transactions on Machine Learning Research (01/2025)
Arggenggenn Wean Over O kenk
15
Hostin
Ravirfort
Opentold
Gredioor
KLN
SHF
OpenMox
E8Q
KBW
tempsSaale
Mad.oge
RMOS
ASH4
(Duns)
Figure 1: The performance comparison of 22 post-hoc OOD detection algorithms based on aggregate mean overall rank (i.c., mcan overall rank averaged across all datascts). ExCeL exhibits the highest consistency across all datasets, indicated by its lowest aggregate mean overall rank (cf. Section 6.1
training NNitsch et al.
2021
Accordingly, addressing the challenge of OOD detection has gained significant
attention in recent studies
Yang al. 2021b).
Among various OOD detection methods, post-hoc inference techniques (Hendrycks & Gimpel 2016 Liang
ct al.
2017
Loe et al.,
2018
Liu ct al., 2020 Sun et al. 2021 Hendrycks et al., 2219aa) Wang et al. 2022 Sun
lel al.
2022 Sun & Li 2022) stand out due to their ease of implementation and compelitive performance Yang et al.
perpeiiiiaeeeoommanneeoom 2021b Zhang et al. 2023b). They can be applied to any pre-trained model, making them versatile and
applicable without the need for modifications during the training phase. These tochniques extract crucial
information from intermediate (Sastry & Oore 2020 Huang et al., 2021) or output (Hendrycks & Gimpel
2016 Hendrycks et al. 2019a] Liu et al.; 2020 Lee et al.) 2018) layers of DNNs, establishing OOD score to distinguish between in-distribution (ID) and OOD samples. Since the output layer (i.e., normalised or unnormalised probabilitics) in a DNN adeptly captures high-level semantics (i.c., objects, seenes ctc.). researchers have increasingly focused on employing its features for OOD detection. For example, Hendrycks) & Gimpel (2016) proposed the marimum softmax probability (MSP) as an initial baseline for OOD detection. Subsequently, Hendryycks et al. (2019a) and [iu et al. (2020) further harnessed more exbreme information from the output layer for OOD detection, utilising the marimum logit (commonly referred to as Maxlogit) and energy (i.e., smooth approximation for the maximum logit), respectively. Moreover, recent studies have incorporated advanced post-procossing techniques (Liang al. 2017 Sun & Li 2022 Sun et al. 2021 to enhance the performance exhibited by MSP and energy scores. While all these methods focus on extreme information, some studies have adopted broader perspective, considering information spanned across ID classes or training samples (Sun et al. 2022 Lee ct al.) 2018 Ren et al., 2021). which we refor to here as collective information. For example, Lee et al. (2018) proposed a method to fit a class-conditional Gaussian distribution on the penultimate layer features based on training samples and derived an OOD score using Mahalanobis distance.
Furthermore, most of these methods, in their respective experimental setups, outperform reported baselines such as MSP and MaxTogit. However, as reported by /Zhang et al. (2023b), these methods perform poorly when evaluated within a common benchmarking setup, with some even performing worse than MSP (e.g.. ODIN [iang et al. 2017), GradNorm (Huang et al. 2021), KLM (Hendrycks et al. 2019a)). )
In this paper, we introduce novel approach ExCeL, designed to enhance the consistency in OOD detection by incorporating both extreme and collective information at the output layer. While the logit of the top predicted class (i.e., MaxLogit) (Hendrycks et al. 2019a) captures the extreme information, we show that the probability of other classes in subsequent ranks yields the collective information required to improve the distinguishability of OOD samples. Our approach is motivated by the observation that, during inference, when an input is predicted as a specific ID class, rankings of the remaining classes are more consistently predictable for ID data compared to OOD data. Therefore, each ID class can be characterised by a unique class rank signature, that is represented as a class probability matric (CPM) with rows corresponding predicted ID classes and columns to their ranks. Each element signifies the probability of particular ID class occurring at specific rank, computed by analysing predicted class rankings across training samples.
2

---

Published in Transactions on Machine Learning Research (01/2025)
In Figure [ show the consistent performance exhibited by ExCeL, compared to existing post-hoc baselines in terms of the aggregate mean overall rank. To summarise, we make the following contributions.
We show that collective information spanning all classes and training samples, embedded in the output layer of trained DNNs through predicted class ranks, can be used to improve OOD detection. Consequently, emphasise the existence of a class rank signature for each ID class, frequently evident in ID data but not in OOD data.
We represent the class rank signabure as a two-dimensional class probubility matric and propose a novel post-hoc OOD detection score named ExCeL, that combines the extreme information provided by the max logit and the collective information provided by the class probability matrix.
We validate ExCeL through extensive experiments conducted in OpenOOD benchmarking library. Compared to twenty-one existing post-hoc baselines, our method consistently ranks among the top five methods overall (near and far) OOD detection on CIFAR100, ImageNet-2000, and ImageNel- 1K datasets. Furthermore, ExCeL exhibits the best performance in terms of the mean overall rank across all datasets outperforming all other basclines that work best performance drop in others.
datasct but has
The rest of the paper organised as follows. In Section - we present the related work, while Scction 3
provides the background related to OOD detection. We provide an overview of our methodology in Section followed by detailed explanation of our experiment setups in Section 5 Next, present the results of our discuss
experiments in Section 6 together with analysis of the findings and outcomes. In Section the implications of our findings and the limitations. Finally, Section concludes the paper.
2 Related Work
A plethora of recent work attempled to address the challenge of OOD detection (Yang et al. 2021b). These techniques can be broadly categorised into three main groups: post-hoc inference methods, training methods without outlier data, and training methods with outlier data (Zhang et al. 2023b
2.1
Post-hoc inference methods
Post-hoc inference methods (Hendrycks & Gimpel 2016 Liang et al. 2017 Lee et al. 2018 Liu et al. 2020 Sun et al. 2021 Hendrycks et al.) 2019aa Wang et al. 2022 Sun et al. 2022 Sun & Li 2022 Djurisic ct al. 2022 Zhang ct al. 2022) utilise post-processon applicd to the base classifier. These works formulate an 0OD score, which, in lurn, is employed lo produce binary TD/OOD prediction through thresholding. These methods are active during the inference phase and generally assume that the classifier has been trained using the standard Pross-entropy loss. They extract information from either intermediate layers (Sastry & Oore,
2020 Dong al. 2022 Huang al.)
2021 the output layer (Hendrycks & Gimpel 2016 Hendrycks ct a.. 2019a Liu et al. 2020 Lee et al. 2018 Ren et al.] 2021) of DNN to establish the OOD score.
Given that high-level semantics are more effectively captured in the output layer, much attention has been directed towards exploiting output layer features for OOD detection. Early work by Hendrycks & Gimpel (2016) proposed the maximum softmax probability (MSP) as a reliable baseline for detecting OOD inputs. Building upon this, later studics adopted a similar approach, leveraging more extreme information from the output layer. For instance, subsequent work by [Hendrycks et. al. (2019a) used the maximum logit directly (i.e., MaxLogit), whereas Liu et al. (2020) employed the energy score, which is smooth approximation for the maximum logit.
Expanding on these strategies, some studies incorporated advanced post-processing techniques (Liang et al. 2017 Sun & Li 2022 Sun et al.) 2021] to elevate the performance exhibited by MSP and energy scores. For example, Sun al. (2021) introduced ReAct by rectifying activations at an upper limit, obtaining a modified logit vector with improved OOD separability. While these methods focus on extreme information, others explored a more comprehensive view of the information provided by the output layer (Hendrycks)
3

---

Published in Transactions on Machine Learning Research (01/2025)
et al. 2019a Sun et al. 2022 Lee et al. 2018 Ren al. 2021). For instance, Hendryeks et al. (2019a] employed KT. divergence between the softmax prediction vector and reference vector to define an OOD score, considering predictions for all ID classes. Furthermore, Sun et al. (2022) leveraged information across training samples, defining an OOD score based on the distance to the 1thh nearest neighbour. However, none of the prior works utilised collective information embedded within the output layer across both ID classes and training samples.
2.2 Training methods without outlier data
These methods incorporate regularisation techniques during training without relying on auxiliary OOD data,
often referred as outliers (DeVries & Taylor 2018; Hsu et al., 2020 Tack et al., 2020 Wei al., 2022;
Ming et al. 2022b). They include a diverse range of approaches, such constraining vector norms (eei
et al. 2022). modifying the decision boundary (Huang & Li 2021 and applying sophisticated learning
methods (Hendrycks et al. 2019b). For instance, Wei et al. (2022) enforced a constant vector norm the logits to prevent their continuous increase throughout the model training. Furthermore, Huang & Li (2021) proposed to simplify the decision boundary between ID and OOD by dccomposing the large semantic space into smaller groups with similar concepts. While the majority of these techniques followed a supervised learning approach, other work [Hendrycks et al., 2019b: Khalid et al. 2022) adopted self-supervised learning for OOD detection.
2.3 Training methods with outlier data
In contrast to the methods discussed in Section 2.2 these techniques harness the knowledge derived from auxiliary OOD data during model training (Hendrycks al. 2018 Yu & Aizawa 2019 Yang et al. 202 Zhang et al. 2023a). This allows OOD detectors to detect OOD inputs more effectively at test time. Within these approaches, some merely incorporate a set of outliers, while others attempl to mine the most informative outliers, a process known as outlier mining (Chen et al. 2021 Ming et al. 2022a). Generally, these methods have the capability to outperform post-hoe and training-basee based approaches without outlier data, as they expose the model to OOD characteristics some extent during the training phase. Nonetheless, they have limitations in generalisation since the model gets exposed to only certain types of OODs (ef. Section A.3 of the Appendix).
Overall, post-hoc inference methods emerge as the standout choice for OOD detection, owing to their ease of implementation and competitive performance (Yang et al., 2021b). While existing post-hoc detectors predominantly concentrate on either extreme or collective information, we propose ExCeL that combines both aspects avoilable within the output layer.
3 Background
In classification tasks, the problem of out-of-distribution detection can be defined using the following setup, Let X' Rd be the input space and y 2,., C} be the oulput space. Assume deep neural network f x + y is trained on set of data D = [(T, 1))) drawn from distribution P defined on X' x V. The network outputs a logit vector which is used to predict the label of an input sample. Furthermore, let. Din denote the marginal distribution of P for X', which represents the distribution of ID data. At test time, the model may encounter inputs from other distributions, denoted as Douts that differ from Dins and are recognised as out-of-iistribution. Thus, the goal of OOD detection is to define decision function g to differentiate between ID and OOD inputs. Accordingly, post-hoc detectors leverage a scoring function S(x) and make this decision via threshold (A) comparison as follows.
inn if S(;; 1) > { ffi
(x(x) =
(1)
out if if <>
This approach different from the MaxLogit approach in Hendrycks ct al. two separate approaches, one relying solely on extreme information (l.e., Loztog while the other on collective information (i.e., KT. matching).
2019a i.c. Hendrycks et al. 2019a proposed
4

---

Published in Transactions on Machine Learning Research (01/202))
Class Careel
Class Elephant
Class Table
Class Cheir
Predicted class
accember Ropie ballry bouds touvee yort 'seedd
uatartum fict
musehen Reppla
607
ballr beed
bally baune Beid
bally las
604
toee
tore
beetil
Nocycle
boitly
601
Rank
Rank
Rank
Ramc
Figure 2: Class rank signatures for top ten ranks for four base ID classes in CIFAR100. Notably, Bear often ranks in the top five for Camel and Elephant classes, while Bed frequently appears in the top five for Chair and Table classes. This is the central concept employed in ExCel. for OOD detection.
4 Methodology
In this section, we first explain the intuition behind our proposed method. Following that, we outline the ExCeL score computation algorithm. Lastly, we analytically justify our motivation.
4.1 Intuition
idea is motivated by the observation that, during inference, when input is predicted specific ID class, the rankings of the subsequent classes more deterministic for ID data compared to OOD data. To illustrate this more clearly, we depict the probability of subset of ID classes ranking among the top ten for four base classes of the CIFAR100 dataset in Figure 2 Tt is important to note that the first rank is always assigned to the respective base class. We notice that when a test input is predicted as either an elephant or a camel, there is a strong probability of the class bear appearing within the top five predictions, given its semantic proximity to these classes. Similarly, in the case of an input predicted chair or table, the class bed is highly likely to be among the top five predictions. We refer to these distinct patterns associated with each class as class rank signatures. Notably, such trends are absent in OOD data, enabling us to leverage this information to design ExCeL based on the predicted class ranking for efficient OOD detection.
As discussed in Section
4.2 ExCeL score computation
The ExCeL score computation comprises four steps; two steps involving pre-computation using training samples and two steps exccuted at test time. Firstly, we calculate the class probability matrir (CPM) for each ID class by leveraging correctly classified training samples specific to that class. Following this, the CPM undergoes a smoothing process to amplify the influence frequently occurring classes while penalising less prevalent ones. During test time, based on the top predicted class of an input, a rank score is computed using the relevant CPM. This score captures collective information from the ranking of predicted classes. Finally, the rank score is linearly combined with the max logit to compute the final ExCeL score for OOD detection. These steps are further discussed in the following sections.
4.2.1 Generating the Class Probability Matrix
The objective of the CPM is to model the probability mass function (PMF) across all ID classes within each rank. To achieve this, we start by filtering correctly classified training samples for each specific class, and rank the remaining C - classes based on their corresponding logit values. Within each rank, we then Figure 2 when an input is
calculate the probability a particular class across the training samples. predicted as a chair, the probability of class bed appearing one of the next four ranks (i.c., ranks 2-5) is approximately 0.08. Similarly, when an input is predicted elephant, class bear appears in ranks 2-4, with probability of approximately 0.08, in each position.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Likewise, for each ID class c, an element Pi in the CPM (Pe RCxC) indicates the probability of class occurring the yth rank when an input predicted as class e as shown in Equation 2
Pi
Pia
Pic
P21
P2
D2C
n
Pe
Pig
(2)
No
VC1
PC2
PCC/
Here, nij represents the number of occurrences where class appears rank among correctly classified samples in class Furthermore, N₂ denotes the total number of correctly classified samples in class c, and C represents the total number of ID classes. It is important to highlight that when calculating the probability matrix for any class exclusively take into account the correctly classified training samples belonging to that class. Therefore, the top rank in the probability matrix is invariably occupied by the class c itself.
{
if otherwise
(3)
4.2.2 Smoothing the CPM
DNNs tend to exhibit some degree of overfitting to the training set. Therefore, some samples may lose the correlation between similar classes during training. This will induce noise in the CPM computed in Section 4.2.1 Hence, to extract high-level information from the CPM, we incorporate smoothing step, employing a piecewise function based on the following criteria:
For classes frequently occurring in a specific rank, a fixed high reward is assigned.
If the probability of a class, though not highly significant, surpasses that random prodiction, a small reward given.
If the probability is worse than a random prediction but not zero, a small penalty is imposed.
Classes that do not appear in specific rank receive fixed high penalty.
Specifically, let P. be the smoothed probability matrix of Pa. Each element Pi in P. is determined based on the corresponding value pf in P. based on Equation
if ύ-ج if c- S p4, < C-1
Pl =
(4)
if pf ن
pf =0
Here, corresponds to the reward, while denotes the high probability threshold. We use the validation set to determine these hyperparameters. The smoothed CPM is then employed to calculate the rank score that provides measure of how closely a prediction aligns with the distinct class rank signature.
4.2.3 Computing the rank score
For a test image x, let the predicted class ranking be [ci, C2..., ec], where e₁ and ec are the classes with the highest and the lowest logit values. Since the top predicted class is c1, we use the smoothed CPM of class C1 denoted by P., for the rank score calculation. Thus, compute the rank score (RS) of as,
RS(x) = Σ po
(5)
i=1

---

Published in Transactions on Machine Learning Research (01/2025)
Since we consider the CPM associated with the top-ranked class when computing the rank score, it is worth noting that the first term (i.e., Pej ) consistently yields 1 (i.e., analogous to с-ニ in Pc) for all inputs, in accordance with Equation 3 Consequently, the presence of the first term in Equation | merely introduces a constant shift to the score, without actively contributing to the discrimination between ID and OOD. However, for completeness, we retain the first term in rank score computation.
Moreover, the rank score can also be computed efficiently via matrix operations. In order to achieve this, the predicted ranking is represented as one-hot encoded matrix (p) with the predicted classes the rows and ranks columns. For example, in four-class classification problem, if the predicted class ranking for an input & is 21,4,2,3) 2,3], Pa would be as follows.
0
0
Pa
0
(6)
0
0
Accordingly, for input x, if the one-hot encoded predicted class ranking matrix is Pas we can compute the rank score as,
RS(x) = tr((PP)"Ppa pxl
(7)
A higher rank score is indicative of accumulating more rewards as per Equation 4 This suggests a strong alignment between the predicted class ranking and the rank patterns observed in training samples, indicating that the input is highly likely to be ID. this way, the rank score encompasses the collective information spanning all ID classes and training samples, which then utilised improve OOD detection.
4.2.4 Combining with the maximum logit
The rank score draws on collective information from subsequent ranks, excluding the top rank, which itself holds valuable information inherent to ID data. Therefore, as the final step, we combine the rank score with the logit value of the top predicted class, referred to as MazLogit by Hendrycks et al. (2019a). to compute the ExCeL score for OOD detection. Since MaxLogit contains extreme information within the output layer, the final ExCeL score incorporates both extreme and collective information embedded in the output layer. We define the ExCeL score as a linear combination of the rank score and the MaxLogit as per Equation
ExCeL(x) a RS(x) + (1 a)) MaxLogit
Here, a balances the use of collective and extreme information for OOD detection. We fine tune a using a validation set following the same approach used Section 4.2.2 to fine-tune a and b.
4.3 Analytical justification
We next analytically justify the existence of distinct patterns in the class ranking that can be exploited to improve OOD detoction. Suppose for any predicted ID class, the remaining C-1 classes occur uniformly distributed across the subsequent ranks. Then, for any class c, the probability matrix, denoted as Equation takes the following form:
if (i = c and j= if (i + and j= or (i and j 1). otherwise
0
(9)
C-T
Thus, for any predicted class ranking of an input x, the rank score would be,
RS(x)
1 C-1
(10)
C 1
C-1
1
1 k (constant)
C
7

---

Published in Transactions on Machine Learning Research (01/2025)
Subsequently, we can compute the ExCeL score as,
ExCeL(x) = a. a RS(x) + (1 - -a) MaxLogit
(11)
= a + (1 - a) MaxLogit
As per Equation [1 when classes appear uniformly al random, the rank score remains coustant, leading the ExCeL score correspond linearly transformed MaxLogit. In this case, the OOD detection performance of ExCeL would be identical to that of the MaxLogit, since a linear transformation would not impact the separability between ID and OOD. Hence, if ExCeL demonstrates enhanced OOD detection performance compared to MaxLogit, it would affirm the presence of unique class rank signatures within ID classes.
5
Experiments
We evaluate ExCeL over common OOD detection benchmarks. To ensure fair comparison with various basclines, we use the OpenOOr library /Zhang et al., 2023b).
5.1 Datasets
ImageNet-200 (a.k.a., TinylmageNet) (Le & Yang 2015 and ImageNet-1K (Deng et al. 2009) ID data in our experiments. Each ID dataset is evaluated against near-OOD and far-OOD datasets. As near- OOD data follow a closer distribution to ID compared to far- OOD, RoD detection is more challenging than far-OOD detection. For CIFAR100, CIFAR10 and TeeelmageNet datasets serve as near-OOD, while MNIST (Deng) 2012), SVHN (Netzer et al. 2011). 1 Tex- tures (Cimpoi al. 2014), and Places365 [Zhou et al. (017)) are considered as far-OOD, Similarly, for both TinylmageNNer and ImageNet-1K,, SSB-hard (Vaze al., 2021 and NINCO (Bitterwolf et al. 2023 datasets used as near-OOD, while iNaturalist (Van Horn et al. 2018 Textures (Cimpoi et al. 2014 and OpenTmage-O (Wang et al.; 2022) datasets are used as far-OOD. For consistency, we adopt the same train, validation, and test splits used by the OpenOOD benchmark in implementing our method.
We use CIFAR100 (Krizhevsky et al. 2009
5.2 Models
For both CIFAR-100 and Tiny Imageete datasets, use ResNet-18 (He et al. 2016 Each model is trained for 100 epochs using the standard cross-entropy loss. We use the SGD optimiser with a momentum of 0.9, a learning rate of 0.1, and cosine annealing decay schedule [Loshchilov & Hutter 2016). Furthermore, we incorporate a weight decay 0.0005, and employ batch sizes of 128 and 256 for CIFAR100 and ImageNet-200, respectively. For ImageNet-11K, we use the pre-trained ResNet-50 (He et al. model from TorchVision (maintainers contributors, 2016).
the base model.
2016
5.3
Comparison with baselines
We compare ExCeT. with twenty-one existing post-hoc methods provided by the OpenOOD library. These basclines include carly OOD detoction methods such as maximum softmax probability (MSP) (Hendrycks &
Gimpel| 2016), Mahalanobis distance (MDS) (Lee al.) 2018). and OpenMax (Bendale & Boult 2016),
well state-of-the-an approaches like SHE (Zhang et al. 2022). ASH (Djurisic et al.) 2022), and DICE (Sun
& Li 2022 Furthermore, it is reported that OpenGAN [Kong & RRaaaaan 2021) has not shown success
on Net-eK and requires substantial changes to make it work with Image.Net Net-Neet- IK models (Zhang et al.
2023b). Therefore, we have not included the results for the OpenGAN baseline for ImageNet-
5.4
Evaluation metrics
We employ two metrics to evaluate the OOD detection performance: i) FPR95, which measures the false positive rate of OOD samples when the true positive rate of ID samples is at 95%; ii) AUROC, representing the area under the receiver operating curve. An effective OOD detector will exhibit low FPR95 alongside
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Comparison of post-hoc OOD detectors for CIFARIO0 (ID). Top five values are highlighted, with
ExCeL and RMDS showing the best performance (M.O.R: 1.5), alongside ReAct, GEN, and MaxLogit.
Post-hoc
AUROC (%)
FPR95 (%)
M.O.R.
method
Near-OOD
Far-OOD
Overall
Near-OOD
Far-OOD
Overall
OpenMax
76.41 0.25 (15)
79.48
0.41 (11)
17.95 (14)
56.58 = 0.73 (9)
64.30 = 0.68 (6)
55.54 (4)
9.0
MSP
80.27 + 0.11 (7)
77.76 + 0.44 (14)
79.02 (12)
54.80 + 0.33 (3)
58.70 = 1.06 (12)
56.75 (10)
11.0
TempScule
80.90 0.07 (4)
78.74
(13)
79.82 (8)
54.49
(2)
57.94
(11)
56.22 (8)
8.0
ODIN
79.90 1: 0.11 (10)
79.28 4: 0.21 (12)
79.59 (10)
57.91 +: 0.51 (10)
58.86 = 0.79 (13)
58.39 (13)
11.5
MDS
58.69 0.09 (20)
69.39 +: 1.39 (18)
64.04 (20)
83,33 + 0.60 (19)
72.26 1.56
(21)
77.90 (19)
19.5
MDSEES
46.31 0.24 (22)
66.00 0.69 (22)
56.16 (22)
95.88 0.04 (22)
66.74
1.04 (17)
81.31 (21)
21.5
RMDS
80.15 + 0.11 (9)
82.92 + 0.42 (1)
81.54 (1)
55.46 + 0.41 (5)
52.81 + 0.63 (3)
54.14 (2)
1.5
Gram
51.06 0.77 (21)
73.36 1.08 (17)
62.51 (21)
92.28 0.29 (21)
61.1 2.37 (16)
78.36 (20)
20.5
EBO
80.91 - 0,08 (3)
79.77 = 0.61 (8)
80.34 (7)
55.62 = 0.61 (7)
56.39 : 1.38 (8)
56.11 (7)
7,0
OpenGAN
65.98 1.26 (18)
67.88 1: 7.16 (20)
66.93 (18)
76.52
(16)
70.49
(19)
73.51 (16)
17.0
GradNorm
70.13 + 0.47 (17)
69.14 - 1.05 (19)
60.64 (17)
85.58 + 0.46 (20)
83.68 1.92 (22)
84.63 (22)
19.5
ReAct
80.77 + 0.05 (5)
80.39 = 0.49 (6)
80.58 (4)
56.39 1 0.34 (8)
54.20 1 1.56 (5)
55.30 (3)
3.5
KLM
76.56 0.25 (14)
76.24 1 0.52 (16)
76.40 (16)
77.92 1.31 (17)
71.65 2.01 (20)
74.79 (17)
16.5
VIM
74.98 + 0.13 (16)
81.70 + 0.62 (4)
78:34 (13)
62.63 + 0.27 (14)
50.74 + 1.00 (1)
56.09 (9)
11.0
KNN
(8)
82.40
(2)
81.29 (3)
(13)
53.65 +
(4)
57.41 (12)
7.5
DICE
79.38 1 0.23 (11)
80.01 = 0.18 (7)
79.70 (9)
57.95 + 0.53 (11)
56.25 + 0.60 (7)
57.10 (11)
10.0
RankFeat
61.88 1.28 (19)
67.10 1.42 (21)
64.49 (19)
80.59
(18)
69.45
(18)
75.02 (18)
18.5
ASH
78.20 + 0.15 (13)
80.58 + 0.66 (6)
79.39 (11)
65.71 + 0.24 (15)
59.20 2.46 (14)
62.46 (15)
13.0
SHE
78.95
0.18 (12)
76.92 + 1.16 (15)
77.94 (15)
59.07 + 0.25 (12)
61.12 = 2.70 (15)
61.60 (14)
11.5
GEN
81.31 0.08 (1)
79.68 = 0.75 (9)
80.50
54.42 + 0.33 (1)
56.71 - 1.59 (9)
55.57 (5)
5.0
MaxLogit
81.05 = 0.07
(2)
79.67 + 0.57 (10)
80.36 (6)
55.47 = 0.66 (6)
56.73 1.33 (10)
56. 10 (6)
6.0
ExCol.
80.70 + 0.06 (6)
82.04 1 0.90 (3)
81.37 (2)
55.21 + 0.56 (4)
22.24 1 1.90 (2)
53.73 (1)
Table 2: Comparison of post-hoc OOD detectors for ImageNet-2000 (ID). Top five values are highlighted,
with ExCeL and GEN showing the best performance (M.O.R: 3.0), alongside KNN, ASH, and TempScale.
Post-hoc
AUROC (%)
FPR95 (%) 1
M. M.O.R.
method
Near-OOD
Par-OOD
Overall
Near-OOD
Far-OOD
Overall
OpenMax
80.27
0.10 (13)
90.20 1 0.17 (12)
85.24 (13)
63.48 1 0.25 (12)
33.12 1 0.66 (8)
48,30 (12)
12.5
MSP
83.34 + 0.06 (3)
90.13 4: 0.09 (13)
86.74 (8)
54.82 1 0.35 (2)
35.43 - 0.38 (13)
45.13 (7)
7.5
TempScale
83.69 0.04 (1)
90.82 0.09 (10)
87.26
54.82 + 0.28 (2)
34.00 0.37 (9)
44.41 (6)
5.0
ODIN
80.27 + 0.08 (13)
91.71 0.19 (5)
85.99 (11)
66.76 + 0.26
(14)
31.23 = 1.05 (11)
50.50 (14)
12.5
MDS
61.93
(19)
74.72
(18)
(19)
79.11
(17)
(17)
70.39 (17)
18.0
MDSERS
54.32 1: 0.24 (22)
69.27 1: (21)
61.80 (21)
91.75 1: 0.10 (21)
80.96 - 0.38 (20)
86.36 (21)
21.0
RMDS
82.57 = 0.25 (6)
88.06 + (16)
85.32 (12)
54.02 - 0.58
(1)
32.45 + 0.79 (7)
43.24 (3)
7.5
Gram
67.67 1.07 (18)
71.19 0.24 (20)
69.43 (18)
86.40 1.21 (20)
84.36 0.78 (21)
85.38 (20)
19.0
EBO
82.50 +: 0.05 (6)
90.86 = 0.21 (9)
86.68 (9)
60.24 - 0.57 (9)
34.86 = 1.30 (12)
47.55 (11)
10.0
OpenGAN
59.79 3.39 (20)
73.13 + (19)
66.47 (20)
84.15 3.85 (19)
64.16
(18)
74.16 (18)
19.0
GradNorm
72.75 + 0.48 (17)
84.26 - 0.87 (17)
78.51 (17)
82.67 + 0.30 (18)
65.45 = 0.22 (19)
74.56 (19)
18.0
ReAct
81.8 0.98 (9)
92.31 0.56 (3)
87.09 (6)
62.49
(11)
28.50 1
(5)
(8)
7.0
KLM
80.76 4 0.08 (12)
88.53 1 0.11 (15)
84.65 (16)
70.26 + 0.64 (16)
40.90 - 1.08 (15)
55.58 (16)
16.0
VIM
78.08 0.24 (16)
91.26 0.19 (7)
84.97 (1G)
59.19 = 0.71
(G)
27.20 + 0.30 (1)
43.20 (2)
8,5
KNN
81.57 0.17 (11)
93.16 1 0.22
87.37 (3)
60.18 1 0.52
(8)
27.27 1 0.75 (2)
43.73 (5)
4.0
DICE
81.78 + 0.14 (10)
90.80 + 0.31 (11)
86.29 (10)
61.88 + 0.67
(10)
36.51 = 1.18 (14)
49.20 (13)
11.5
RankFeat
56.92 1.59 (21)
38.22 3.85 (22)
17.57 (22)
92.06
(22)
97.72
(22)
91.89 (22)
22.0
ASH
82.38 + 0.19 (8)
93.90 1 0.27 (1)
88.14 (1)
64.89 1 0.90 (13)
27.29 1 1.12 (3)
46,09 (9)
5.0
SHE
80.18 0.25 (15)
89.81 0.61 (14)
85.00 (14)
66.80
(15)
42.17 1.24 (16)
54.49 (15)
14.5
GEN
83.68 - 0.06 (2)
91.36 = 0.10 (6)
87.52 (2)
55.20 - 0.20
(4)
32.10 - 0.59 (6)
43.65 (4)
3.0
MaxLogit
82.90 - 0.04 (4)
91 = 0.19 (8)
87.01 (7)
59.76 - 0.59 (7)
31.03 1.21 (10)
16.90 (10)
8.5
ExCeL
82.40 0.01 (7)
91.97 1 0.27 (4)
87.19
57.90 1 0.40 (3)
Σ (4)
43.18
3.0
high AUROC. We also measure the overall performance of each method by computing the mean of near and far-OOD performances. Finally, to compare the performance of a method with other basclines across ID datasels, we define the Mean Overall Rank (M.O.R) as the average of overall ranks for AUROC and FPR95.
Mean Overall Rank RAROO + Roveral
(12)
Moreover, we report the mean and the standard deviation of the above metrics computed over three indepen- dent runs for CIFAR100 and ImageNet-200, while, for ImageNet- the result for a single run is reported to ensure consistency with the OpenOOD benchmark.
5.5 Hyperparameter tuning
Fine-tuning hyperparameters on a validation set widely adopted in prior OOD detection work (Hendrycks)
et al. 2018 Sun et al. 2022 Liu et. al. 2020 Following similar approach, we determine the three
9

---

Published in Transactions on Machine Learning Research (01/2025)
parameters associated with ExCeL using the validation set. By performing grid search a, b, and a, we discovered the best hyperparameter combination for both CIFAR100 and ImageNet-200 datasets, is = 10, b = 5, and a = 0.8. For ImageNet-IK, they were a = 10, b = 5, and a = 0.6. We visualise the variation of AUROC for CIFAR100 (ID) in Figure
6 Results and Analysis
6.1 Comparison with baselines
We present our results in Tables 2 and 3 for CLFAR-100, ImageNet-200, and ImageNet-IK, respectively, Note that due to space limitations, provide the results on ImageNet-1K (i.e., Table 3 in Section A. of the Appendix. Furthermore, we report the average performance for both near-OOD and far-OOD across all OOD benchmarking datasets in each group (cf. Section 5.1). along with the overall OOD performance. Per-dataset results can be found in Section A.2 Additionally, we compare ExCeL with outlier-based training methods in Section A.3 of the Appendix.
According to the results, ExCeL consistently stands out among the five top-performing methods in most instances. For instance, in CIFAR100, ExCeL is ranked second in overall AUROC, first in overall FPR95, and has the equal best mean overall rank with RMDS. Similarly, in ImageNet-200, ExCel. is ranked fifth in overall AUROC and first in overall FPR95. Again, ExCeL is ranked equal first in mean overall rank, but this time, sharing it with a different method, GEN. Furthermore, in ImageNet- IK, ExCeL is ranked third in overall AUROC and fifth overall FPR95, and ranked third in terms of the mean overall rank.
We can also observe from the results that most of the other baselines exhibit strong performance in spe- cific cases but demonstrate only moderate or poor performance in others. For example, RMDS excels in CIFAR100, sharing equal best mean overall rank with ExCeL. However, its performance lags behind in ImageNet-200 and ImageNet-1K. Similarly, GEN performs well in TmageNet-200, but falls behind in CT- FAR100. In contrast, ExCeL delivers consistent results in all datasets, achieving a mean overall rank of 1.5 in CIFAR100, 3.0 in ImageNet-2000 200, and 4.0 in ImageNet-1K:
Furthermore, the results also show that ExCeL performs slightly better in far-OOD detection than near-OOD detection. For example, in CIFAR100, ExCeL is ranked second and fourth in terms of FPR95 for far and near OOD, respectively. Similarly, in ImageNet-200 ExCeL is ranked fourth and fifth in terms of FPR95 for far and near OOD, This can be explained using the characteristics of probability matrices in different datasets (cf. Section 6.3
From the three Tables, we can see the variations in the performance of different methods across different evaluation criteria. For example, while ASH performs significantly well on ImageNet-IK (cf. Table 3 it falls short on CIFAR-100 (cf. Table D. Similarly, in CIFAR-100 ID scenario (ef. Table D. KNN performs well in terms of AUROG yet falls short with respect to FPR95. Furthermore, on (cf. Table 2), RMDS performs well on near OOD detection but lags behind on far OOD detection in terms of AUROC. Therefore, to combine the performance across ID datasets and measure the consistency across various evaluation criteria, we define aggregate mean overall rank, computed as the average of mean overall ranks for different datasets. This metric enables capturing performance insights across all evaluation aspects (i.e., near and far-OOD detection, AUROC and FPR, across three ID datasets). We showed the aggregate mean overall rank of all the methods in Figure D and ExCeL clearly stands out for its higher consistency.
6.2 The effect of hyperparameters
perform the grid search to find (a.b) pair that yields the best performance for the rank score. Next, we show an analysis of the effect of these hyperparaneters on the performance using the CIFARI00 dataset as an example. Figure 3a shows the variation of the AUROC on the validation set with a and when CIFAR100 serves as ID. Notably, the AUROC tends to be particularly low for higher values of both a and b. The optimal region for both hyperparameters is found to be a.b E (5,10).
As mentioned in Section 5.5
After determining the optimal values for and b, we proceed to fine-tune a balance between extreme (i.e., MaxLogit) and collective (i.e., rank score) information. Figure 3b illustrates the validation AUROC's
10

---

Published in Transactions on Machine Learning Research (01/2025)
100
0853
1850
20
0.8486
0.8628
0.8491
10
0.840
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
-1885
Rank score
a
MaxLogit
2-
0.800
1
1.825
0.850
0.852
0.854
0.856 AUROC
0.858
0.860
0.862
10
20
100
(a) Tuning and b
(b) Tuning
Figure 3: Hyperparameter tuning for CIFAR100 (ID).
Wedeot
Merlest
FPRO5 (%)
F9R95
FPR95
20
10
Textures
Places365
Textress Openimage-O
Tettuess Openimage-0 NINCO (c) ID Net-
(a) ID CIFAR100
(b) ID ImageNet-200
Figure 4: FPR95 comparison between MaxLogit and ExCeL.
variation with respect to a. We observe from the extremes in the figure that both the rank score (i.e., a = 1) and MaxLogit (i.e., = 0) alone achieve relatively low performance compared to their combinations. For CIFAR100, a = 0.8 yields the bost performance, as indicated by Figure 3b We obsorved similar trends for both ImageNet-200 and ImageNet-11K datasets.
We also note that the performance of the rank score slightly lower than that of MaxLogit. This is because these classification models trained onc-hot encoded vectors where the only target is the ground truth class. Therefore, the model is only trained to predict the ground truth class. What propose from the rank score is discovering some hidden knowledge (i.e., high-level semantic relationships between ID classes) that the model is not specifically trained for and exploiting this knowledge for OOD detection. Therefore, unlike MaxLogit, the rank score itself does not use any information from where the model excels (i.c., leveraging the information from the highest prediction). Therefore, is possible that the rank score alone would not be able outperform methods such as MSP, Energy, and MaxLogit that utilise information from the maximum prediction. While it is expected for the rank score to perform slightly lower than the MaxLogit, the performance is not considerably lower than the MaxLogit. In Figure 3b difference in terms of AUROC between MaxLogit and the rank score is 0.0005. More importantly, the main advantage of the rank score is that it can enhance the performance of MaxLogit without diminishing its original effectiveness, as demonstrated both analytically in Section 4.3 and empirically in Section 6.1
that the performance
6.3 ExCeL and Maxlogit
As analysed in Section 4.3 improvement in the OOD detection in ExCeL compared to MaxLogit confirms the existence of unique class rank patterns in ID classes. First, to validate this, we show the FPR95 comparison of ExCeL and MaxLogit in Figure on all datasets. There, we observe two key behaviours. For some OOD datasets, ExCeL achieves a significant improvement in FPR95 compared to MaxLogit, while in others, the performance of ExCeL is on par with MaxLogit. For example, with CIFAR100 as ID, ExCeL significantly improves the OOD detection in SVHN and Textures, while the performance on Places365 is similar to MaxLogit.
11

---

Published in Transactions on Machine Learning Research (01/2023)
CIFAR100 (ID)
Textures (OOD)
Places365 (OOD)
000
Predicted Class
0.00
0000
-1.23
-0.75
Rank
Figure 5: Class probability matrices for ID and OOD samples predicted as a selected ID class in CIFAR100. We can see clear clusters of classes occurring mainly within the top ranks for ID data. In Textures, the predicted class rankings show random and sparse behaviour, while in Places365, the occurrence of classes is close to uniformly random distribution.
The reason for this can be explained using the CPMs of ID and OOD data shown in Figure 5 Here, we show how the subsequent classos are ranked in ID) and OOD samples for a selected class in CIFAR100. Specifically, for CIFAR10O, we see clear clusters of classes occurring mainly within the top ranks for ID data, indicating a unique class rank signature. In contrast, the class occurrence in Textures data looks random and sparse which allows the FxCel. score to separate the two datasets effectively. On the other hand, the occurrence of classes Places365 is close to uniformly random distribution, making the rank score less informative. Consequently, ExCeL performs better against. OOD data whose predicted class rankings are sparse and random. In general, this happens more often in far-OOD than near-OOD.
7 Discussion
We next discuss the implications of our results and limitations.
Overall Consistency. We measure the overall consistency of an OOD detection method using the aggregate mean overall rank (ef. Section 6.1 This metric embeds the performance information with respect to the two OOD tasks (i.c., near and far- OOD detection), two evaluation metries (i.c., AUROC and FPR95), and across three ID datasets. A low aggregate mean overall rank with a low standard deviation would indicate consistent OOD detection performance.
Among the compared post-hoc OOD detection methods, ExCeL achieves the lowest aggregate mean overall rank with low standard deviation, showcasing its consistency across different ID datasets. GEN (Liu et al. 2023) and ReAct (Sun et al. 2021 | secure second and third positions in terms of the overall consistency cxhibiting competitive performance with ExCeL. Even though ASH (Djurisic et al. 2022 performs excep- tionally well in ImageNet-1K its performance falls short, particularly on the CIFAR100 dataset. Similarly, RMDS (Ren et al., 2021) performs well on 11FAR100, but falls short on ImageNet-IK. Due to their excep- tional performance on one particular dataset, both ASH and RMDS achieve low aggregate mean overall rank. However, the high standard deviation of both ASH and RMDS, as shown in Figure I indicates their lack of consistency. This consistency issue was recently highlighted by [Zhang et al. (2023b). In this regard, ExCeL's stable results in both near and far OOD detection across multiple datascts highly promising.
Moreover, we evaluate ExCeL in common framework compared with many existing baselines since one of the main drawbacks of existing OOD detection methods is that they outperform reported baselines in their own experimental sct-ups yet are shown to fail in the common framework (Zhang et al.] 2023b). We intend to include ExCeL in the OpenOOD benchmark so that our results can be reproduced and compared with future work.
Limitations. ExCeL performs better in scenarios where ID classes consist of a diverse range. For instance, CIFAR100 encompasses a wide range of object calegories, including animals (e.g,, dogs, cats), vehicles (e.g.. cars, aeroplanes), and household items (e.g., chairs, tables), each exhibiting distinct visual features. Similarly, both ImageNet-200 and ImageNet-1K present broad spectrum of objects, including animals, plants, vehiclos, and various other everyday objects. In such situations, we can clearly observe classes in semantic proximity appear in neighbouring ranks, which is exploited by ExCeL. However, when faced with lack of diversity
12

---

Published in Transactions on Machine Learning Research (01/2025)
among classes, ExCeL's performance may be constrained. Consider CIFAR10, which comprises only ten classes, including six animal classes and four vehicle classes. In this scenario, the distribution of classes within the same category may exhibit high randomness across subsequent ranks. Consequently, the rank signatures are not as prominent as datasets such as CIFAR100 and ImageNet. Thus, ExCeL's effectiveness is limited under such circumstances.
8 Conclusion
In this paper, we proposed a novel OOD score, ExCel,, which combines extreme and collective information within the output layer for consistent OOD detection. We utilised the MaxLogit as extreme information and proposed novel class rank score that captures information embedded across all ID classes and training samples. We demonstrated that each ID class has a unique signalure, determined by the predicted classes in the subsequent ranks, which becomes less pronounced in OOD data. Experiments on CIFARI00, ImageNet- 200, and ImageNet-1K showed that ExCeL consistently ranks among the top five methods out of twenty-one baselines. Furthermore, ExCeL showed the equal best performance with RMDS and GEN methods on CIF and ImageNet-2200 datasets, respectively, in terms of the overall mean rank. With regard to the overall consistency across datasets, ExCeL surpasses the other post-hoc baselines.
Acknowledgements
We sincerely thank the Action Editor, Yanwei Fu, and the three anonymous reviewers for their constructive comments. This research was supported by the Australian Government through the Australian Research Council's Discovery Projects funding scheme (Project ID DP220102520).
References
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1563 1572, 2016.
Julian Bitterwolf, Maximilian Müller, and Matthias Hein. In or out? fixing imagenet out-of-distribution detection evaluation. arXie preprinl arXiv:2306. 00826, 2023.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pp. 430-445. Springer, 2021.
Mircea. Cimpoi, Subhransu Maji, Tasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606 3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 258-255, 2009. doi:
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2916):141-142, 2012.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802. 04865, 2018.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distributtion detection. arXiv preprint arXiv:2299.08 099888 2022.
Xin Dong, Junfeng Guo, Ang Li, Wei-Te Ting, Cong Liu, and HT Kung. Neural mean discrepancy for efficient distr--ditt detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 19217- 2022.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinchart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In International Conference on Machine Learning, pp. 3145 3153. PMLR, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A bascline for detecting misclassified and out-of-distributioo examples in neural networks. arXiv preprint arXiv:6610.02136, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1rx:::8822.00 04606, 2018.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling oustof-distributio detection for real-world settings. arXiv preprint 1rXiv:1911.11199 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in newral information processing systems, 32, 2019b.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt. Kira. Generalized odin: Detecting out-of-distribution image without learning from dustri-iiitiitution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8710-8719, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-689, 2021.
Umar Khalid, Ashkan Esmacili, Nazmul Karim, and Nazanin Rahnavard. Rodd: A self-supervised approach for robust out-of-distribution detection. In 2022 IEEE/CVE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 163-170. IEEE, 2022.
Shu Kong and Deva. Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 813 822, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1766 706.02690, 2017.
Weitang Lin, Xiaoyun Wang, John Owens, and Yixuan Li. Eneegy-based out-of-distribution detection. Advances in neural information processing systems, 33:21464 21475, 2020.
Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gem Pushing the limits of softmax-based out-of- distribution detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern 2023.
Recognition, pp.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 170.0:1608.09983, 2016.
14

---

Published in Transactions on Machine Learning Research (01/2025)
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://github.
hwm/pyytorch/iision 2016.
Yifei Ming, Ying Fan, and Yixuan Li. POEM: Out-of-distribution detection with posterior sampling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.); Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15650-15665. PMLR, 17-23 Jul 2022a. URL https://proceedding mlr. wwtppsssee//ming2222/eee html
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out- of-distribution detection? arXiv preprint or arXiv:2203044400 2022b.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Julia Nitsch, Masha Itkina, Ransalu Senanayake, Juan Nieto, Max Schmidt, Roland Siegwart, Mykel Kochenderfer, and Cesar Cadena. Out-of-distribution detection for automotive perception. In 2021 JEEE International Intelligent Transportation Systems Conference (ITSC), pp. 2938 2943. IEEE, 2021.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood delection. arXiv preprint arXiv:2106. 090022 2021.
Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, et al. Does your dermatology classifier know what it doesn't know? detecting the long-tail of unseen conditions. Medical Image Analysis, 75:102274, 2022.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pp. 8491 8501. PMLR, 2020.
Yue Song, Nicu Sebe, and Wei Wang, Rankfeat: Rank-1 feature removal for out-of-distributioo detection. Advances in Neural Information Processing Systems,
2022.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In European Conference Computer Vision, pp. 691-708. Springer, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144 157, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributioo detection with deep nearest neigh- bors. In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11830 11852, 2020.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set
classifier is all you need? arXiv preprint arXi:2210006200 2021.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit
matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4930, 2022.
Hongxin Wei, Renchunzi Xic, Hao Cheng, Lei Feng Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631- 23644. PMIRR 2022.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In Proceedings of the IEEE/CVP International Con- ference on Computer Vision, pp. 8301 8309, 2021a.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distributiio detection: A survey. arXic preprint orXiv:2110. 11334, 2021b.
Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrep- ancy. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9518-9526, 2019.
Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution delection in fine-grained environments. In Proceediiggs of the IEEE/CVP Wünter Conference on Applications of Computer Vision, pp. 5531-5544 -5440, 2023a.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood vl. 5: Enhanced benchmark for out-of- distribution detection. arXiv preprint ar Xrv:22306. 09301, 2023b.
Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, Shi Han, Dongmei Zhang, et al. Out-of- distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In The Eleventh International Conference on Learning Representations, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEF transactions pattern analysis and machine intelligence, 40(6): 1452 1464, 2017.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A Appendix
A.1 Performance on ImageNet-1K
Table 3: Comparison of post-hoc OOD detectors for ImageNet-IK (ID). Top five values are highlighted,
with ASH showing the best performance (M.O.R: 1.0), alongside ReAct, ExCeL, GEN, and VIM.
Post-hoc
AUROC (%)
FPR85 (%) +
M.O.R.
method
Near-OOD
Par-OO1)
Overall
Near-OOD
Far-OOO))
Overall
OpenMax [Bendale & Boult
2016
74.77 (11)
89.26 (13)
82.02 (12)
69.07 (10)
34.31 (5)
51.69 (6)
9.0
MSP (Hendrycks & Grupel
2016
76.02 (9)
85.23 [17)
80.63 (17)
65.68 (5)
51.45 (17)
58.57 (15)
16.0
TempScale (Gto
77.14
8T.56 (15)
82.35 (8)
64.50 (2)
46.64 [15)
55.57 (11)
9.5
ODIN [Liang of al:
74.75 (12)
89.47 (11)
82.11 (11)
72.50 (14)
43.96 (13)
58.23 (14)
12.5
MDS Leo
2
65.44 (19)
74.25
64.85 (19)
85.45 (18)
62.92 (18)
74.19 (18)
18.5
MDSEns
Lee el sL.
2018
19.67 (21)
67.52 (20)
58 60 (20)
93.52 (21)
82.77 (20)
88,15 (20)
20.0
RMDS
Bt. 1212T
(16)
(14)
(10)
(8)
Gram
SustEy Xr Core
20200
61.70 (18)
79.71 (18)
70.71 (18)
86.63 (19)
67.36 (19)
77.00 (19)
18.5
EBO Liu el 9J: (051
75.89 (10)
89.47 (11)
82.68 (6)
68.56 (9)
38,39 (9)
53.48 (10)
8.0
OpenGAN [Kong Tamanan
2021
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Noam
Huung et al
2021
72.96 (15)
90.25 (6)
81.61 (15)
78.89 (17)
47.92 (16)
63.41 (17)
16.0
ReAct
(Sun
77.38
93.67 (2)
85.53
06.69
46.50
KLM
Hendrycke cl al
2019a
76.61 (7)
87.60 (14)
82.12 (10)
72.54 (15)
16.60 (14)
59.57 (16)
13.0
Wang
72.08 (16)
(3)
(7)
(12)
(2)
KNN
5100 Di mL
2022
71.10 (17)
90.18 (7)
80.64 (16)
70.87 (11)
34.13 (4)
52.50 (7)
11.5
DICE Sun XC
2022
73.07 (14)
90.95 (4)
82.01 (13)
72.43 (13)
41.83 (12)
57.13 (12)
12.5
Rank [Song
2022
50.99 (20)
53.93
52.46 (21)
91.83 (20)
87.17 (21)
89.50 (21)
21.0
ASH Djurisic of al. 2022
78.17 (1)
95.74 (1)
86.96 (1)
63.32 (1)
19.49 (1)
41.41 (1)
1.0
SHE
Znaux
73.78 (13)
(5)
(8)
(11)
57.23 (13)
10.5
GEN
Liu et aL. (026)
76.85 (6)
89.76 (8)
83.31 (4)
65.32 (4)
35.61 (6)
50.47 (4)
4.0
Maxl.ogit (Hendrycks
2019a)
76.46 (8)
89.57 (10)
67.82
38.22 (8)
(9)
ExCeL (Ours)
77.03
89.70 (9)
83.37 (3)
67.00 (7)
36.14 (7)
51.37 (5)
4.0
reported that OpenGAN has not shown success on ImageNet-IK, and require substantial changes to make
it work with ImageNet-1K models (Zhang et al. 2023b
A.2 Per-dataset results
This section presents the per-dataset performance for near and far-OOD detection. Table and Table demonstrate the FPR95 and AUROC results, respectively, when CIFAR100 serves as ID. Similarly, Tables and 7 report FPR95 and AUROC results, respectively, with ImageNet-2000 as the ID dataset, while Tables and present the same metrics with ImageNet- as the ID dataset.
As can be seen from Table 4 in terms of FPR95, ExCeL exhibits the most consistency, ranking among the top five performing methods in five cases (i.e., against ImageNet-200, near-OOD, SVHN, Textures, and far-OOD), which is the highest for any method when CIFAR100 is considered
ID. Similarly, when ImageNet-2200 serves ID, ExCeL ranks among the top five methods in all seven scenarios, Table 0 Compared to CIFAR100 and ImageNet-200 datasets, ExCeL drops slightly short on the ImageNet- 1K dataset, reaching among the top five methods only in two instances, as shown in Table g However, ExCeL ranks within the top seven methods all instances for ImageNet-1K in terms of FPR95.
shown in
With regard to AUROC, when CIFAR100 is considered as ID, ExCeL ranks among the top five methods five cases (i.e., against ImageNet-200, SVHN, Textures, Places365, and Far-OOD), being the most consistent method out of twenty-one baselines, shown in Table 5 When ImageNet-200 and ImageNet-1K3 serves as ID, ExCeL ranks among the lop five methods in three and two instances, respectively, out of the seven scenarios. Consequently, when the overall consistency is considered, FxCel outperforms all the other baselines as shown in Figure in terms of the aggregate mean overall rank (cf. Section 7 in the main text).
17

---

Published in Transactions on Machine Learning Research (01/2025)
+ 1.06 (2)
58 86 + 0.79 (13)
72 - 1.56 (21)
66.74 1 1.04 (17)
52.81 = 0.63 (3)
61.41 - 2.37 (10)
56 56.59 + 1.38 (8)
70.49 + 7.38 (19)
83 08.08 + 1.92 (22)
54.20 1.5.1.666 (3)
71.65 4 2.01 (20)
50.74 = 1.00 (1)
53.65 0.28 (1)
56 + 0.60 (7)
60.45 + 1.01 (18)
50 50.20 - 2.45 (34)
64.12 = 2.70 (15)
50.71 1.50 (9)
56.73 # 1.33 (10)
52.24 = 1.90 (2)
79.48 + 0.41 (11)
1 0.44 (14)
79.22 4 0.21 (12)
00-30 - 1.39 (18)
66. 00.00 - 0.69 (22)
82 0.42 (1)
75.06 4. 1.08 (17)
79. 1 0.61 (8)
67. 88 7.16 (20)
69.14 4 1.05 (19)
80.30 0.49 (6)
76.21 = 0.32 (36)
81.70 0.62 (4)
82.40 0.17 (2)
80.01 1 0.18 (7)
07-10 = 1.42 (21)
60.58 0.66 (5)
76.02 1.16 (15)
79.68 + 0.75 (9)
79.67 1 0.37 (10)
0,90 (3)
Table 4: FPR95 comparison of post-hoc OOD detectors for CIFAR100 (ID). The performance rank of each method is indicated within brackets. Top live values are marked in bold.
Far-OOD
04 14.50 0.68
57.94 1.14
Table 5: AUROC comparison of post-hoc OOD detectors for CIFARIOO (ID). The performance rank of each method indicated within brackets.
Par-OOD
78 0.51
38.70
82.04
77.
54, .6.8 1..2 (2)
56.62 0.87 (6)
56.46 0.94 (5)
59.71 - 0.92 (11)
79.61 - U.34 (18)
96.61 1 0.17 (22)
53.57 1 0.43 (1)
57.75 # 0.86 (8)
73.75 4 8.32 (16)
85.32 - 0.14 (20)
55.30 0.42 13)
81 81.40 4. 1.58 (19)
61-57 4 0.77 (13)
59.39 + 1.25 (10)
77.42 4 1.96 (17)
62.93 - 0.00 114)
(15)
(4)
(7)
58.02 1.28 (9)
79.29 +: 0.40 (10)
79.22 4 0.20 (11)
79.45 0.26 (8)
+ 0.40 (20)
42.27 - 0.73 (22)
53.40 0.46 (1)
46.38 1. 1.21 (21)
79-32 + 0.23 (7)
69.69 4 0.17 (17) (17)
80.03 0.11 (3)
75.70 + 0,21
73.85 # 0.37 (15) (15)
79.43 0.47 (9)
T8.33 4 0.00 (13)
78.76 + 0. (12)
76.30 + 0.51 (14)
80.28 + 0.27 (2)
79.75 0.24 (6)
0,57 (4)
Places303
91.67 - 0.00
60.70 1.03
65.24 + 0.98
1.01
57.03 - 0.91
Places365
79. 0.25
69.13 4 7.08
63.82 188
79.98
56.
(0)
61.88 + 1.28 (10)
(9)
(2))
7U.19 + 2. (16)
84.94 1. 0.83 (0))
53 53.09 + 1.06 (4)
02 # 2.06 (11)
74.77 1.78 (8)
02.51 - 0.61 (22)
55.04 + 0.82 (3)
81.80 1. 5.80 (1))
46. + 2.29 (1)
53.56 2.32 (3)
04.27 + 1.65 (14)
66 86.87 4 3.80 (3)
07.27 1 2.74 (8)
73 73:29 + 3.22 (17)
1.40 (7)
(13)
51.14 1 0.14 (2)
(6)
(14)
(8)
(15)
(19)
83.65 + 0.51 (3)
70.79 1. 1.32 (18)
78.30 4 0.83 (11)
3.43 (1)
64.58 4 0.13 (2))
80.15 0.40 (7)
75.77 4 0. (16)
85.91 $ 0.78 (1)
83.08 0.83 (2)
77.63 4 0.34 (13)
1 3.08 (20)
80.72 4 0.70 (5)
73.04 + 1.28 (17)
78.74 + 0.81 (9)
78 78.30 1 0.84 (10)
(4)
Textures
60. 12 4 1.01
61.56 1.43
02.37 - 2.96
89 99.51 233
02 30 4 2.13
Textures
80.56 0.09
77.32 4: 0.71
78.11 + 0.72
79.33 4 1.08
70.20 4 0.69
09.75 75 + 1.14
+ 0.58
61 11.23
63.28
65
53. 1.78 (10)
59.07 = 2,53 (16)
(14)
67.41 = 3.88 (10)
67 = 6.09 (28)
82.57 - 2.58 22)
31.65 + 3.68 (8)
53.62 = 3.14 3.14(11) (11)
70.35 2.08 (21)
60.00 = 7.91 (20)
50.41 4 2.02 (T)
30.30 1 7.04 (6)
46.22 $ 5.46 (4)
31.75 3.12 19)
49.58 + 3.32 (5)
58.40 2.30 (16)
46.0 - 2.67 (3)
50.15 = 7.61 (17)
55.45 - 2.76 (13)
53.90 = 3.04 (12)
45.13 1 0.33 (2)
76.01 + 1.59 (17) 82.07 + 1.53 (9)
78-42 - 0.89 (16)
79.79 1.05 (14)
74.54 = 0.76 (18)
70.08 6.40 (20)
53.76 = 1.63 (22)
84.89 + 1.10 (4)
95.55 0.60 (1)
82.03 = 1.74 (10)
08.40 2.15 (22)
76.96 = 4.73 (17)
83.01 - 0.97 (8)
79.31 = 0.44 (15)
53.14 + 3.71 3,71 (7)
84.15 1 1.00 (6)
84.22 - 2.00 (6)
85.00 .0 + 1.40 (3)
80,97 = 3.8 (13)
81.41 = 1.50 (12)
81.65 1 1.49 (11)
85.91 = 0.73 (2)
20 00.00 1.96
SVHN
SVHN
Top five values are marked in bold.
57.71
74.14
(10)
(15)
(2)
TL. - 2.0 (19)
2.83 1 0.86 (1)
02.03 1 0.28 (6)
52.62 # 3.83 (7)
63.09 = 21.25 (17)
86.07 - 1.11 (22)
36.04 - 5.06 (13)
73.09 1. 6.67 (2))
38.32 4 1.07 (3)
48.58 4.67 (1)
51.79 + 3.67 (5)
75.01 4 5.83 (21)
66. - 388 (18)
58.78 4 2.70 (16)
53.92 4 5.71 (1)
03.82(2 (8)
34.67 4 5.86 (12)
76.08 - 1.86 (16)
83.79 4 1.31 (2)
67.47 4 0.81 (20) (28)
98.21 + 0.78 (1)
79.74 2.19 (7)
80.71 4.15 (5)
79.18 4 1.37 (8)
08.14 18.78 (19)
65.35 : 1.12 (21)
78.37 + 1.50 (1))
71.15 - 2.59 (1))
81.89 + 1.02 (4)
82.36 1.52 (3)
79.86 4 1.80 (6)
63.03 3,80 (22)
77.23 4 0.46 (14)
76.76 - 1.07 (15)
(12)
78.91 1 1.47 (10)
1.73 (9)
MNIST
53.82 - 4.74
+ 4.08
56.05 4.61
45.04 + 3.29
53.53 7.15
52.93 - 3.82
MNIST
T7:27 - 1.85
78.29 - 2.05
57.23
78.99
00 08 = 0.73 (9)
54.80 + 0.33 (3)
(2)
57.91 + 0.51 (10)
83.53 0.60 (19)
95.88 0.04 (22)
55. 1 0.41 (3)
55 + 0.61 (7)
76.52 2.50 (16)
85.58 4 0.46 (20)
56.30 0.34 (8)
77.92 1: 1.31 (17)
02.63 1 0.27 (14)
(1))
57.95 + D.53 (11)
80.59 1.10 (18)
65.71 1 0.24 (13)
59.07 0.25 (12)
54.42 1. 0.33 (1)
(0)
55.21 1 0.56 (4)
76.41 + 0.25 (15)
80 80:27 = 0.11 (7)
80. +. 0.07 (d)
79.90 4 0.11 (10)
58.09 0.00 (20)
46.31 0.24 (22)
80.15 = 0.11 (9)
51.66 0.77 (21)
80.91 1 0.08 (3)
70.13 4 0.47 (17)
80.77 0.05 (5)
76.36 4 0.25 (14)
74.98 0.13 (16)
80 0.15 (8)
79.38 1 0.23 (11)
61.88 1.28 (19)
78.20 4 0.15 (13)
78.95 0.18 (12)
81.31 + 0.08 (1)
81.05 1. 0.07 (2)
= (0)
Near-OOD
54.49 0.48
92.28 + 0.29
05.47 = 0.66
Near-OOD
65.98 1.26
61.22
52. 9. + 0.51 (10)
50.70 0.34 (6)
50.28 0.16 (5)
59.19 19 - 0.37 (33)
7U - 1.22 (20)
05. 95.82 1. 0.12 (22)
40.56 = 0.00 (2)
91.85 - 0.86 (2)
52.03 = 0.50 (9)
74.21 1.25 (17)
80.83 + 0.62 (20)
51.47 = 0.47 (7)
71.07 4. 0.39 (6)
64.06 - 0.42 (1))
49.65 0.37 (3)
54.93 + 0.33 (1)
78.40 0.93 (18)
63 03.33 - 0.90 15)
57 # 0.73 (14)
49.98 0.05 (4)
51.53 = 0.70 ( (8)
49.35 = 0.31 (1)
78.44 + 0.14 (15)
82.07 = 0.17 (9)
81.63 4 0.08 (10)
61.50 4 0.28 (20)
48.78 - 0.19 (22)
82.55 0.02 (8)
53.91 4. 1.38 (11)
82 82.76 = 0.08 (7)
68.74 2.29 (18)
95.95 4 0.79 (17)
62.88 0.08 (6)
79.22 - 0.28 (14)
77.76 4 0.16 (36)
83.34 0.16(1) (1)
80.7 72 1 0.30 (11)
92.9244 4 0.20 (12)
(13)
0.13 (3)
(4)
(2)
ImaggeNet-200
ImageNet-2200
82. 90 0.05
0.03
70. - 0.36
82. 52.79
05. 72.72-
83 83.25 25
53.28
60.17 + 0.07 (6)
58 91 0.93 (3)
58.72 0.81 (1)
60.61 + 0.66 (8)
0U = 0.49 (20)
95.04 0.16 (22)
61.37 4 0.4 (12)
92.71 - 0.04 (21)
59.21 0.75 (5)
78.83 4 3.94 (16)
81.30 - 0.36 (18)
61. = 0.33 (11)
84.77 4. 2.0 19)
70.00 : 0.4 (14)
(15)
00.98 +1 1.10 (9)
82.78 4: 1.56 (17)
68. 06 - 0.44 (33)
60.41 4 0.51 (7)
59.11 4 0.64 (4)
61.07 1 0.81 (10)
74.38 + 0.37 (14)
78-47 1 0.07 (6)
78.18 4 0.14 (7)
55.87 - 0.22 (20)
43. = 0.31 (22)
TT.75 + 0.19 (11)
49.41 1 0.58 (21)
79.05 1 0.100.111 (3)
63.23 2.44 (18)
70.32 # 0.20 (17)
78.65 0.05 (6)
73.91 - 0.25 (15)
72.21 : 0.41 (16)
77.02 1 0.25 (2)
78.04 - 0.32 (10)
76.48 + 0.50 (3)
78. 15.15-2 0.03 (8)
70. 38 0.04 (1)
79.21 0.10 (2)
78.14 0.09 (9)
CIFAR-10
58 87 0.09
CIFAR-10
79.02 0.06
58.04 233
72. =
Post- processor
ExCeL (Ours)
Post- processor
ExCeL (urrs
Opx
TempSrale
MDSEES Ens
OpenGNN
GradNorm
Rank
MaxLogit
OpenAlax Mlax
TempSeale
MDSEns
OpenGAN
GradNorm
Rank Real
MaxLogit
MSP
ODIN
MDS
RMDS
Gram
EBO
ReAct
KLM
VIM
KNN
DICE
ASH
SHE
GEN
MSP
ODIN
MDS
RMDS
Gram
EBO
Re Aet
KLM
VIAI
KNN
DICE
ASH
SHE
GEN
18

---

Published in Transactions on Machine Learning Research (01/2025)
Table FPR95 comparison of post-hoc OOD detectors for ImageNet-228 MaggeNet-200 200 (ID). The performance rank of each method is indicated within brackets. Top live values are marked in bold.
Table 7: AUROC comparison of post-hoc OOD delectors for ImageNet-200 200 200 (ID). The performance rank of each method indicated within brackets. ts.
33.1 13.12 0.80 (8)
35. = 0.38 (15)
0.37 (9)
31.23 = 1.05 (11)
61. = 0.27 (17)
80. - 0.38 (20)
32.45 1 0.79 (?)
81.30 0.78 (21)
34.86 + 1,30 (12)
64.16 4 9.78 (18)
66. 65.45 = 0,22 (30)
28.50 0.95 (5)
40.90 1 1.08 (15)
27 .0 + 0.30 (1)
36.51 + 1.18 (14)
97-72 = 0.75 (22)
27 27-29 = 1.12 (3)
42.17 = 1:24 (26)
31.03 = 1.21 (10)
28.45 1 0.80 (4)
90. 20 + 0.17 (32)
90. 90.13 - 0.09 (13)
91.71 1 0.19 (5)
74.72 = 0.20 (18)
00.27 = 0.57 (21)
88.00 = 0.34 (16)
71.19 1. 0,24 (20)
90.86 0.21 (9)
64.26 # 0.87 (17)
92.31 0.56 (3)
88.53 = 0.11 (15)
91.26 0.19 (7)
93.16 0.22 (2)
80.80 = 0.31 (11)
38.2882222 - 3.85 (22)
93.90 + 0.27 (1)
80.81 = 0.01 (14)
91 91.36 0.10 (6)
91.11 1 0.19 (8)
Far-OOD
27.27 0.75
32.10 0.50
Far-OOD
82.82 - 0.09
73.15 4.07
91.97 0,27
OpenImage- O
38.03 4 0.49 (13)
35.23 = 0.18 (9)
0.31 (7)
37.30 + 0.00 (12)
68 + 28 (18)
87.26 1 0.10 (21)
34.85 = 0.31 (8)
36.74 + 1.14 (11)
06.34 8.44 (17
71.16 4 0:23 (19)
82 4 0.74 (2)
45.81 1. 0.50 (15)
33.80 = 0.63 (6)
18.91 + 1.18 (14)
95.77 + 0.85 (22)
33.72 72.72 1 0.07 (3)
46.34 + 1:34 (16)
35.76 - 0.74 (10)
32.91 1 0.78 (4)
OpenImage-O O
88.07 + 0.14 (13)
89.24 = 0.02 (9)
90. 4 0.15 (5)
09.87 + 0.14 (19)
61.96 - 0.51 (21)
86.77 + 0.22 (16)
67.72 1. 0.58 (20)
89.23 4 0.2 (10)
73.54 1 4.48 (18)
80.66 4 1.0 (17)
00.40 0.85 (2)
8T.66 + 0.17 0.17 (14)
88.20 + 0.18 (12)
90.19 T 0.32 (3)
89.06 4 0.44 (11)
(20)
91.82 4 0.25 (1)
87.19 + 7.70 (16)
90. 90.13 1 0.08 (1)
89 0.21 (8)
89.75 0.32 (7)
82.00 22.00 + 1.12
42. 25 1 0.31
86.00 - 1.27
52.48 4
89
(6)
(14)
0.77 (15)
(12)
08 + 0.84 (17)
72 1 0.48 (20)
37 - 1.32 (7)
41.43 + 1.85 (10)
66.00 9.97 (18)
06 88 + 3.39 119)
29. 29.67 - 1.35 (6)
50.24 1.36 1:36 (16)
20.39 + 0.17 (1)
40.96 + 1.87 (8)
00.40 +. 0.68 (22)
25. .5.65 = 0.80 (3)
15.58 4 2.42 (15)
42.01 0.92 (1)
41. + 1.86 (0)
30.14 4 0.64 (5)
(12)
(14)
90. - 0.20 (8)
0.33 (20)
80 70 - 0.48 (18)
86. + 0.38 (15)
80.53 0.37 (19)
00 70 4 06 (7)
86.07 4. 0.36 (17)
92.80 22 - 0.47 (4)
86 13 4 0.12 0.12 (16)
94.61 + 0.12 (3)
95. 29.29 1 0.02 (1)
01.03 - 0.21 (6)
29 12002200 2.57 122)
94.77 + 0.19 (2)
00.51 + 0.19 (10)
90 25 + 0.10 111)
(0)
92.40 = (5)
Textures
30. 80 - 0.55
44.58 + 0.68
99 + 1.56
80 87 1.20
24.45 0.29
Textures
90 21 + 0.07
88 1 0.13
70.58 4.66
90 1 0.16
89 +
79
Top five values are marked in bold.
24.53 - 0.00 (8)
26. 26.48 = 0.73 (12)
24.39 0.79 (6)
22.30 - 1.87 (3)
58. 0.75 (17)
63.37 0.70 20)
24.70 + 0.00 (9)
85.04 0.40 (21)
26.41 + 2.29 (11)
60.13 9.79 (18)
61 61.31 2.86 (19)
22.07 4 2.25 (5)
26.06 1.61 (13)
27.34 = 0.38 (14)
21.16 - 1.00 (7)
29 29.06 2.02 (15)
0.80 (2))
22 40 + 2.24 (4)
34.38 = 3.48 (16)
22 .2. + 0.98 (1)
25.00 = 2.04 (10)
22 + 1.00 (2)
92. 02 0.32 (11)
92.8 -240000 0. 0.25 (9)
93.39 - 0.25 (7)
94. 94.37 4 0.41 (2)
75.03 0.70 (10)
62. 16 0.73 (21)
90.64 0.46 (16)
65. 0.20 (20)
92. 03 = 0.50 (10)
75.32 3.32 (18)
56.06 1.90 (17)
93.65 105440550 0,88 (6)
91 80 0.21 (13)
900 06 0.36 (15)
93.99 1. 0.36 (3)
91.81 = 0.79 (12)
33.08 4.08 (22)
95.10 4 0.47 (1)
91.43 1.28 (14)
93. 70 + 0.18 (5)
93.12 1 0.45 (8)
93.76 + 0.43 (4)
iNaturalist
iNatoralist list
98.
0.26 (12)
54.82 + 0.35 (2)
(2)
06. 76 + 0.26 (14)
79.11 0.31 (1T)
91.75 0.10 (2)
54.02 1 0.58 (1)
86.40 46-40 1.21 (20)
60. + 0.57 (9)
84.15 3.85 (19)
82.65 67.67 1 0.30 (18)
62 49 2.19 (11)
70.26 0.64 (16)
00 # 0.71 (0)
00 10 0.32 (8)
61.88 + 1.67 (10)
92.06 0.23 (22)
1 0.90 [13)
(15)
1 0.20 (4)
00 + 0.59 (7)
57 90 1 0.40 (5)
80.27 27 0.10 (13)
83.34 + 0.06 (3)
(1)
20.27 D.08 (13)
01.03 0.51 (19)
54.32 34 + 0.24 (22)
82.57 + 0.25 (5)
67.67 1 1.07 (18)
82.50 1 0.05 (0)
(20)
72.75 72.75 4 0.48 (17)
81.87 0.98 (9)
80.76 76.76- - 0.08 (12)
0.24 (16)
11.57 0.17 (11
81.78 1 014 (10)
36.92 1.59 (1)
82.08 4 0.19 (8)
80.18 0.25 (15)
83. 83.68 + 0.08 (2)
0.04 (4)
0.04 (7)
Ne Near-OOD
Near-OOD
0.74
+
63.
64 80
66 06.80
55. 20
59.79
78.65
82.
82.40
43.65 0.75 (4)
0.70 (2)
(14)
74.37 4 0.15 (17)
91.36 1 0.16 21)
42.13 = 1.04 (1)
87.13 1 1.89 (20)
50.70 = 0.89 (9)
80.23 4: 5.71 (18)
83.17 17 1 0.21 (10)
53. - 2.46 (21)
62.33 1 2,66 (6)
47.10 = 1.10 (?)
52.91 + 1.20 (10)
03-32 4 0.11 (22
57 63 1 0.98 13)
60. 96 4 1.33 (15)
(3)
(8)
46.51 = 0.20 (5)
53.01 03.011 + 0.17 (15)
86. 29 = 0.11 (3)
88.67 0.08 (1)
83.34 4 0.12 (13)
05. 48 +: 0.46 (19)
58 18 1 0.12 (21)
84.91 0.28 (9)
60.40 1 1.07 (18)
85.17 = 0.11 (8)
(20)
73.39 4 0.63 (17)
84.76 - 0.64 (10)
96 - 0.12 12)
83. # 0.19 (14) (21
88.10 0.12
84.49 4 0.2 (1)
55.10 2.5 (2)
65.24 = 0.08 (7)
07 + 0.33 (10)
86.60 0.08 (2)
85.65 0.09 (6)
(6)
NINGO
64. 09.994 + 0.34
60. 00 - 0.80
16.61 0.73
43.61 0.61
19.57 = 0.94
NINCO
61.49
83 83.96
55,40
72. 37 = 0.11 (12)
10.00000 = 0.10 (2)
06. 06.45 0.26 (3)
73.61 - 0.38 (14)
83. 63.05 - 0:47 (18)
92.13 0,05 (2))
65.91 1 0.27 (1)
69.77 0.32 (7)
88.07 4 2.23 (20)
82.17 - 0.02 (17)
T1.51 = 192 (10)
78.19 1. 2:30 16)
71.28 4 0.49 (9)
70.84 + 1.50 (8)
90.79 + 0.37 (21)
12.14 = 0.07 (11)
T2 64 # 0.30 (13)
0.26/6 (4)
69.64 04.64110 4 0.37 (6)
09. 09.28-1 1 0.60 (5)
77.53 + 0,08 (13)
80 38 + 0.03 (3)
(2)
80. 19 : 0.06 (14)
58. - 0.58 (20)
50. 40 - 0.36 (22)
80.20 0.23 (4)
65.05 1. 1.08 (18)
79.83 1 0.02 (6)
72.12 : 0.43 (17)
78.07 - 1.33 (10)
77.56 - 0.18 (2)
T4.04 # 0.31 (16)
77.03 1 0.23 (5)
79.06 1 0.06 (9)
79.52 4 n.37 (7)
75. 30 = 0.20 (11)
80.75 75 0.03 (1)
80.15 0.01 (5)
(8)
SSB- hard
SSB-hard hard
85.08 - 0.80
73.71 = 0.31
58.74 = 0,94
66. 79
55.08
79.39
Post- processor
ExCoL (Ours)
Post- processor
OpenMax
MSP
TempSrale
ODIN
MDS
MDSEES Ens
RMDS
Gram
EBO
OpenGAN
GradNorm
ReAct
KLM
VIM
KNN
DICE
Rank
ASH
SHE
GEN
MaxLogit
Openhlax
MSP
TempSeale
ODIN
MDS
MDSEEs
RMDS
Gram
EBO
OpenGAN
GradNorm
Aet
KLM
VIAI
KNN
DICE
Rank Real
ASH
SHE
GEN
MaxLogit
ExCeL
19

---

Published in Transactions on Machine Learning Research (01/2025)
Table 8: FPR95 comparison of post-hoc OOD detectors for ImageNet-IK (ID). The performance rank of each method is indicated within brackets. Top five values are marked in bold.
Post-proeessor
SSB-haad
NINCO
Near-OOD
iNaturalist
Textures
OpenImage-O
Far-OOD
OpenMax
77.33 (10)
60.81 (12)
69.07 (10)
25.29 (3)
40.26 (6)
37.39 (6)
34.31 (5)
MSP
74.49 (3)
56.88 (6)
65.68 (5)
13.31 (17)
60.87 (19)
50.13 (15)
51.45 (17)
TempScule
73.90 (2)
64.50 (2)
37.63 (14)
(17)
45.40 (11)
(15)
ODIN
76.83 (9)
68.16 (15)
72.50 (14)
35.98 (13)
49.24 (15)
46.67 (12)
43.96 (13)
92.10 (20)
78.80 (17)
86.45 (18)
73.81 (19)
42.79 (8)
72.15 (18)
62.92 (18)
MDSEas
95.19 (21)
91.86 (20)
93.52 (21)
81.23 (20)
73.31 (20)
90.77 (21)
82.77 (20)
RMDS
77.88 (12)
$2.20 (1)
65.04 (3)
33.67 (11)
48,80 (14)
40.27 (9)
40.91 (10)
Gram
89.39 (18)
83.87 (19)
86.63 (19)
67.89 (18)
58.80 (18)
75.39 (19)
67.36
EBO
76.54 (8)
60.58 (11)
68.56 (9)
31.30 (8)
45.77 (11)
38.09 (8)
38.39 (9)
OpenGAN"
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GradNorm
78.24 (14)
79.54 (18)
78.80 (17)
32.03 (9)
43.27 (9)
68.46 (17)
47.92 (16)
ReAct
77.55 (11)
55.82 (5)
66.69 (6)
16.72 (2)
29.64
32.58 (2)
26.31 (3)
KLM
84.71 (17)
60.36 (10)
72.51 (15)
38,52 (15)
52.40 (16)
48.89 (14)
16.60 (14)
VIM
80.41 (15)
62.29 (13)
71.35 (12)
30.68 (6)
10.51 (1)
32.82 (3)
24.67 (2)
KNN
83.36 (16)
58.39 (8)
70.87 (11)
40.80 (16)
17.31 (3)
44.27 (10)
34.13 (4)
DICE
77.96 (13)
66.90 (14)
72.43 (13)
33.37 (10)
44.28 (10)
47.83 (13)
41.83 (12)
RankFeat
89.63 (19)
94.03 (21)
91.83 (20)
94.40 (21)
76.84 (21)
90.26 (20)
(21)
ASH
73.66 (1)
52.97 (2)
63.32 (1)
14.04 (1)
15.26
29.15 (1)
19.49 (1)
SHE
76.30 (7)
69.72 (16)
73.01 (16)
31.06 (12)
35.27 (5)
55.02 (16)
11.45 (11)
GEN
75.73 (4)
54.90 (3)
65.32 (4)
26.10
46.22 (13)
34.50 (4)
35,61 (6)
MaxLogit
76,20 (6)
59.44 (9)
67.82 (8)
30.61 (5)
46.17 (12)
37.88 (7)
38.22 (8)
ExCeL (Ours)
75.82 (5)
58.18 (7)
67.00 (7)
30.85 (7)
41.07 (7)
36.51 (5)
36.14
reported that OpenGAN has not shown success on ImageNet- and require substantial changes to make it work with ImageNet-1K models (Zhang et al. 2023b).
Table 9: AUROC comparison of post-hoc OOD detectors for ImageNet- (ID). The performance rank of cach method is indicated within brackets. Top five values are marked in bold.
Post-processor
SaB-hard
NINCO
Near-OOD
iNaturalist
Textures
OpenImage-O
Far-OOD
OpenMax
71.37 (13)
78.17 (13)
74.77 (11)
92.05 (7)
88.10 (13)
87.62 (10)
89.26 (13)
MSP
72.09
(9)
76.02
(15)
(19)
84.86 (16)
TempScale
72.87 (4)
81.41 (6)
77.14 (3)
90.50 (13)
84.95 (17)
87.22 (12)
87.56 (15)
ODIN
71.74 (11)
77.77 (14)
(12)
91.17 (8)
89.00 (10)
88.23 (9)
80.47 (1))
MDS
48.50 (20)
62.38 (19)
55.44 (19)
63.67 (19)
89.80 (8)
69.27 (19)
74.25 (19)
MDSEas
43.92 (21)
35.41 (20)
49.67 (21)
61.82 (20)
79.94 (20)
60.80 (20)
67.52 (20)
RMDS
71.77 (10)
82.22
76.99 (5)
87.24 (16)
86.08 (16)
85.84 (15)
86.38 (16)
Gram
57.39 (18)
66.01 (18)
61.70 (18)
76.67 (18)
88.02 (14)
74.43 (18)
79.71 (18)
EBO
72.08 (7)
(10)
(10)
90.63 (12)
(11)
(7)
(11)
OpenGAN"
N/A
N/A
N/A
N/A
N/A
N/A
N/A
GradNorm
71.90 (9)
74.02 (17)
72.96 (15)
93.89 (3)
92.06 (6)
84.82 (17)
90.25 (6)
ReAct
73.03 (2)
81.73 (4)
77.38 (2)
96.34 (2)
92.79 (5)
91.87 (2)
93.67 (2)
KLM
71.38 (12)
81.90 (3)
76.64 (7)
90.78 (11)
84.72 (18)
87.30 (11)
87.60 (14)
VIM
65.54 (16)
78.63 (12)
72.08 (16)
89.56 (14)
97.97 (1)
90.50 (3)
92.68 (3)
KNN
62.57 (17)
79.64 (11)
71.10 (17)
86.41 (17)
97.09
87.04 (13)
90.18 (7)
DICE
(15)
(16)
(14)
92.54
88.26 (8)
90.95 (4)
RankFeat
55.89 (19)
46.08 (21)
50.99 (20)
40.06 (21)
70.90 (21)
50.83 (21)
53.93 (21)
ASH
72.89 (3)
83.45 (1)
78.17 (1)
97.07 (1)
96.90 (3)
93.26
95.74 (1)
SHE
71.08 (14)
76.49 (15)
73.78 (13)
92.65 (4)
93.60 (4)
86.52 (14)
90.92 (5)
GEN
72.01 (8)
81.70 (5)
76.85 (6)
92.44 (6)
87.59 (15)
89.26 (4)
89.76 (8)
MaxLogit
72.51 (5)
80.41 (8)
76.46
91.17 (8)
88.39
89.17
89.57
ExCeL (Ours)
73.28 (1)
80.80 (7)
77.03 (4)
90.79 (10)
89.24 (9)
89.07 (6)
89.70 (9)
It is reported that OpenGAN has not shown success ImagoNet-IK, and require substantial changes to make it work with ImageNet-1K models (Zhang et 2023b).
A.3 Comparison with outlier-based training methods
In Section 2.3 discussed OOD detection methods that utilise auxiliary outliers during training, On the positive side, they have the potential to outperform other approaches, such as post-hoc methods, due to the additional knowledge provided by outliers. However, they are likely to fall short in standard ID classification accuracy as focusing on outliers during training may lead to a lack of generalisation to the ID data distribution (Zhang et al., 2023b). The model may bocome specialised detecting outliers, sacrificing performance on the ID data. Furthermore, they have the tendency to overfit to the seen outliers, which may degrade their performance in OOD detection (Yang et al. 2021b) Accordingly, post-hoc methods are
20

---

Published in Transactions on Machine Learning Research (01/2025)
PPR95 (76
ID Tost
Method
Near-OOD
Far-OOD
Overall
Accuracy (% 1)
CIFAR-100
OF
Hendryrkss et al.
2018
30.73 (1)
54.82 (3)
42.78 (1)
76.84
MCD
Yu
55.88 (4)
54.39 (2)
55:14 (3)
75.83
UDG
Yang el al 2021a
61.42 (5)
59.00 (4)
60.21 (5)
71.51
MixOE
Zhang
(3)
(5)
(4)
ExCol.
(Post-hor)
55.21 (2)
52.24 (1)
53.73 (2)
77.25
ItuageNet-200
200
OE (Hendryoks et al
2018
52.30 (1)
34.17 (3)
43.24 (3)
85.82
MCD
Yu AlAay
2019
54.71 (2)
29.93 (2)
42.32 (1)
86.12
UDG
Yang al 2021a
68.89 (5)
62.04 (5)
65.47 (5)
68.
MixOE
Zhaan et AL 2023a
57.97 (4)
40.93 (4)
19.45 (4)
85.71
ExCeL (Post-ltoc)
(3)
(1)
(2)
Table 10: FPR95 comparison of ExCeL with outlier-based training methods. leads in far-OOD detection and ranks second overall on both ID datasets.
ExCeL, a post-hoc method,
generally popular due to their competitive performance and case of implementation. For completeness, in
Table 10 we compare FxCel with four outlier-based training methods: Outlier Exposure (OE) (Hendrycks
et al. 2018), Maximum Classifier Discrepancy (MCD) (Yu & Aizawa 2019 Unsupervised Dual Grouping
(UDG) Yang al. 2021ah, and Mixture Outlier Exposure (MixOE) Zhang et al. 2023a), evaluate the
aforementioned pros and cons,
From Table we observe that the ID classification accuracy of outlier-based training methods has decreased compared to standard training methods. For example, the Lest accuracy of UDG has dropped by approx- imately 6% and 18% on the CLEAR-100 and ImageNet-200 datasets, respectively, compared to standard training, which is used post-hoe methods. In OOD dctection, ExCeL has shown competitive performance with outlier-based methods, achieving the best performance in far-OOD detection and the second-best overall performance on both ID datasets in FPR95.
We observe significantly higher performance for the OE method in near-OOD detection compared to other methods for the CIFAR-100 dataset. This improvement is attributed to the bias in seen outliers. In the OpenOOD setup, data from non-overlapping classes in ImageNet-1K is used as auxiliary outlier data for training OE models on CIFAR-100 and ImageNet-2200. Consequently, when Tiny Imagene (i.e., the first 200 classes of ImageNet-1K)) is evaluated as a near-OOD dataset for the CIFAR-100 ID model, OE performs well because it has already encountered relatively similar OOD inputs during training. This bias of auxiliary outliers during training allows OE to perform significantly better in near-OOD detection for the CIFAR-100 ID scenario. This is further evidenced by OE's performance not being significantly high in other scenarios, even underperforming compared to ExCeL in far-OOD detection.
21


Reviews:
[{"review_id": "1vHHRj8It0", "reviewer": "Reviewer_pa32", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Figure 3 clearly shows the considerations for the three most important hyperparameters.\n\n* It is impressive that Table 1 can compare this method to 21 other methods. Also, Table 4, 5, 6 and 7 display a wide range of results, which are useful for future works that want to study this problem.\n\n*  The proposed method can be applied to any pre-trained model, making it versatile and applicable without the need for modifications during the training phase.\n\n\nWeaknesses:\n\n* The out-of-distribution detection problem is not well defined. Section 3 speaks of ‘marginal probability distributions,’ but any distribution on the image space [0, 255]^{H x W x C} will assign density to any image. The threshold is also not well defined, e.g. ImageNet contains many dogs and no cherries, so an image of a cherry might be OOD, but since computer screens and paintings are in ImageNet, that could be a painting of a cherry.\n\n* The method introduces Class Probability Matrices to compare predictive rank between samples. However, it is not clear where the near-class ranks are learned, is that the test set, train set, or a separate validation set? If it's a separate validation set, how would this method compare to just using that validation data for training? From Figure 3 it seems that the best setting of this method scores only 1 point higher than existing works (at alpha=0), and this gap might well be closed by using the validation set for training.\n\n* Section 5.5 describes how the hyperparameters are different between CIFAR100 or ImageNet. However, there misses an explanation of why these hyperparameters are different. For example, is it coincidental that the a,b hyperparameters are the same, and what can be learnt from the optimal alpha-value being higher for one than the other?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of out-of-distribution detection. The approach aims to determine if a certain sample originated from the training distribution or not and have that decision benefit further steps in the automated decision pipeline. Experiments on CIFAR100, ImageNet-200, and ImageNet-1K datasets show that ExCeL is among the top-performing methods in terms of AUROC and FPR95. However, I find that the paper misses a crucial step: the opening sentence of the abstract states 'Deep learning models often exhibit overconfidence in predicting out-of-distribution (OOD) data,' but this assertion was never established. In the context of the datasets used in the experimental section, could this overconfidence actually be shown?", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* From Equation (4), it seems that \\hat{p}_{ij}^c can only assume one of four values, was this meant as a scaling factor instead? Or how would one calculate the correlation of a categorically-valued vector?\n\n* The writing frequently uses the phrase 'to be more deterministic' or 'to be less deterministis.' For example, in 'the rankings of classes are more deterministic for ID data compared to OOD data' (section 4.1). What does that mean, 'to be more deterministic'?\n\n* Could Section 3 be expanded and give a few examples of ID and OOD images? E.g. for ImageNet, would one consider an image of a cherry OOD. Or are we talking about spectral properties, e.g. all ImageNet images are encoded with JPEG, so would a PNG image be consider OOD?\nSimilarly, in Section 5.1, CIFAR10 is considered and OOD class for CIFAR100. How does that work? They were made by the same three authors, who state on their website ‘This dataset [CIFAR100] is just like the CIFAR-10, except it has 100 classes containing 600 images each.’ [https://www.cs.toronto.edu/~kriz/cifar.html].\n\nAbout the degree of OOD-ness, the paper states that 'the degree of OOD-ness is not a binary concept.' However, the paper does not define what it is. Could the authors provide a definition of OOD-ness, or a reference to a paper that does?\n  * A foundational work from 2011 seems relevant to this approach: Torralba, Antonio, and Alexei A. Efros. \"Unbiased look at dataset bias.\" CVPR 2011. IEEE, 2011.\n  * Similar to section 5.1, earlier work has already tried to define near-OOD and far-OOD: Djolonga, Josip, et al. \"On robustness and transferability of convolutional neural networks.\" CVPR 2021, IEEE, 2021, c.f. the OOD-ness scale in the ImageNet-C and ImageNet-P datasets.", "broader_impact_concerns": "* The paper addresses a problem of distribution shift that is relevant to many applications. I do not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RNZ0aOJYSu", "reviewer": "Reviewer_ojac", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The idea is simple, well presented, and potentially generalizable (though see below) beyond immediate applications. Therefore, this work satisfies the criteria that TMLR's audience would be interested in knowing the findings of this paper.\n\n- Experiments on the benchmark are rigorous, details meticulously provided, and thereby reproducible. If the codebase is also made public, depending on the Action Editor's discretion, this work can receive a reproducibility badge. \n\n- I truly appreciated the discussion of limitations, but I think some of it needs to be extended (see below).\n\n\n## Weaknesses\n\n- \"Our idea is motivated by the observation that, for in-distribution (ID) data, the ranking of classes beyond the predicted class is more deterministic compared to that in OOD data.\" There are supporting empirical observations and a theoretical motivation in Section 4.3, however, I am not sure how generalizable this observation is. Please correct me if I am wrong, but Section 4.3 simply suggests that if ExCel works, then this motivation is justified. However, there is no apriori prediction as to when ExCel will be successful or not; apart from just testing it. But, in real datasets, we do not know the ground truth, so without a clear theoretical *apriori* motivation, how can we trust in ExCel's generalization capabilities? I think this is one major weakness since generalization is the main claim of this paper (i.e., ExCel works on all benchmarks as opposed to other methods). I will recommend two paths for the authors in the requested changes below to address this.\n\n- Figure 1 claims \"ExCeL exhibits the highest consistency across all datasets, indicated by its lowest aggregate mean overall rank.\" Such claims require statistical tests. Moreover, this figure focuses only on ranks, which is not necessarily interpretable. This needs to be replaced with average accuracies, or this needs to be added as a second figure if the authors like to keep the rank figure. Currently, it makes me think that maybe ExCel was not the one with highest average accuracies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes ExCeL, a method for out-of-distribution (OOD) detection that combines MaxLogit and a novel class rank-based score from a deep neural network's output layer. The method improves OOD detection based on an empirical observation that class rank signatures are more predictable for in-distribution data compared to OOD data. This approach is particularly good because for a uniform distribution of class ranks, it reverts back to an interpretable approach, MaxLogit. Finally, extensive experiments across CIFAR-100, ImageNet-200, and ImageNet-1K demonstrate that ExCeL consistently ranks among the top-performing methods across various benchmarks, outperforming 21 baselines in terms of AUROC and FPR95 metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please address the weaknesses above.\n\n- Can you please update Figure 1 to include the data points in a swarm plot + box plots? I do not see any reason why this should be a bar plot. \n\n- Please perform significance tests on Figure 1 (with multiple testing corrections, ranksum or wilcoxon depending on whether the data is paired) and let us know which algorithms are significantly worse than ExCel. I doubt ExCel will be significantly better than all algorithms as claimed.\n\n- Can you please quantify the class rank signatures discussed in Fig. 2? How do we know that this observation is statistically significant compared to chance level? This example is an helpful illustration, but not a clear evidence.\n\n- To remedy the first weakness, I believe it is important to show one real life application or a realistic simulation (of authors' choosing) beyond just testing on benchmarks. For example, a public brain computer interface dataset or a set of synthetic datasets with controlled OOD data generation processes etc. If the authors simply want to write in the limitation that they have not tested on real life applications, that is also fine, though I strongly encourage them to take the former path.\n\n- Figure 5 suggests that ExCel will simply revert to MaxLogit when the main assumption does not hold, but are there any cases where ExCel would be worse?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y5row1sCZr", "reviewer": "Reviewer_fTa6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The authors propose an interesting perspective about the rankings of the remaining ID classes.\n- Based on OpenOOD library, they have comprehensive experiments on different datasets.\n- The proposed method, ExCel, exhibits the best overall performance.\n\n## Weaknesses or Questions\n### Major\n- There are a few hyper-parameters to tune in ExCel. I’m afraid that the method relies on heavy hyper-parameter tunning in reality. In Section 5.5, the authors mention that they use validation set to tune the hyper-parameters. What validation set do they use? What are the OOD images in the validation set? \n\n- I am not sure whether linear combination of the rank score and the MaxLogit is a contribution of the paper. In principle, we can combine any OOD methods to get possibly better performance. I appreciate the explanation in Section 4.3, but I’m still concerned with it. \n\n(1) What is the performance of the rank score itself for OOD detection? \n\n(2) Is it also possible to combine the rank score with other method that use extreme information, such as MSP or energy?\n\n(3) Previous works, such as ReAct and ASH, also report that their methods can be combined with others to get better performance. If they authors combine the rank score with MaxLogit, I’m not sure whether it is a fair comparison.\n\n- The OpenOOD library has supported newer models like ViT and reports that some methods’ performance drops a lot on different models (for example, ASH on ViT). Could the authors also report the performance of ExCel on the new models, now that they build their study upon OpenOOD as well?\n\n\n### Minor\n- (Page 2) “… with some even performing worse than MSP (e.g., ODIN (Liang et al., 2017), GradNorm (Huang et al., 2021), KLM (Hendrycks et al., 2019a)).” \n\nThe citations here can be improved. I feel it’s better to write like “with some even performing worse than MSP, such as ODIN (Liang et al., 2017), GradNorm (Huang et al., 2021), KLM (Hendrycks et al., 2019a)” than to use double parentheses.\n- Performance of methods on ImageNet-1k is one of main results for the paper, and is very important because ImageNet-1k is a large-scale dataset for OOD detection evaluation. I think it’s better to consider putting Table 3 in the main manuscript instead of in Appendix. \n- I’m curious about the design choice that the authors filter correctly classified training samples for each specific class for building CPM. I feel there are multiple options to explore. \n\n(1) Have the authors tried to use all training samples assigned by the model to the specific class no matter whether their classification is correct? \n\n(2) Or have the authors tried to use training samples with ground truth being that class?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors identify the existence of a class rank signature for each ID class, frequently evident in ID data but not in OOD data. That is, when a test-time input is predicted as a specific ID class, rankings of the remaining classes are more consistently predictable for ID data compared to OOD data.\n- They construct a class probability matrix (CPM) with rows corresponding to predicted ID classes and columns to their ranks, capturing the class rank signature for each ID class. They calculate the matrix based on the training set and smooth the matrix with a proposed transformation.\n- The authors propose ExCeL, an OOD detection method by linearly combing the CPM and the MaxLogit, utilizing both collective and extreme information.\n- In experiments, ExCeL consistently ranks among the top 5 in overall (near and far) OOD detection on CIFAR100, ImageNet-200, and ImageNet-1K datasets. ExCeL exhibits the best performance in terms of the mean overall rank across all datasets outperforming all other baselines, showing better generalization ability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the Strengths And Weaknesses section. If the authors can address my concerns on the Weaknesses of the paper, I would like to recommend acceptance.", "broader_impact_concerns": "No concersn on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision to recommend acceptance for the paper ExCeL: Combined Extreme and Collective Logit Information for Out-of-Distribution Detection is based on a combination of its contributions, empirical rigor, theoretical grounding, and responses to reviewer feedback. Below are the key points supporting this decision.


The paper introduces ExCeL, a method that integrates extreme (maximum logit) and collective (rank-based class information) logit features to improve out-of-distribution (OOD) detection. Unlike prior approaches that focus solely on one type of information, this combination provides a more consistent and robust framework. This novel dual-feature perspective is well-motivated and represents a significant contribution to the OOD detection domain.

The theoretical reasoning behind using rank-based class signatures is well-justified. The authors show that deterministic ranking patterns emerge in ID data, making this feature useful for distinguishing OOD samples. 

The authors have thoroughly responded to reviewer comments, improving clarity and addressing specific weaknesses, and all reviewers accept this paper: 
They provide expanded explanations of OOD detection challenges, include examples of ID vs. OOD cases, and discuss the limitations of ExCeL in small datasets like CIFAR-10.
Statistical significance tests and improved visualization (swarm plots and box plots) enhance the robustness of their conclusions.
The combination with MaxLogit was justified with empirical results showing that ExCeL outperforms similar combinations involving collective methods (e.g., KLM and KNN).
Decision: Accept (poster)