Igeood: An Information Geometry Approach to Out-of-Distribution Detection
Authors: Eduardo Dadalto Camara Gomes
Florence Alberge
Pierre Duhamel
Pablo Piantanida
Venue: ICLR 2022 Poster (2022)
Keywords: out-of-distribution detection
anomaly detection
deep learning

Abstract: Reliable out-of-distribution (OOD) detection is fundamental to implementing safer modern machine learning (ML)  systems. In this paper, we introduce Igeood, an effective method for detecting OOD samples. Igeood applies to any pre-trained neural network, works under various degrees of access to the ML model, does not require OOD samples or assumptions on the OOD data but can also benefit (if available) from OOD samples. By building on the geodesic (Fisher-Rao) distance between the underlying data distributions, our discriminator can combine confidence scores from the logits outputs and the learned features of a deep neural network. Empirically, we show that Igeood outperforms competing state-of-the-art methods on a variety of network architectures and datasets.

Published as a conference paper at ICLR 2022
IGEOOD: AN INFORMATION GEOMETRY APPROACH TO OUT-OF-DISTRRBUTION DETECTION
Eduardo D. C. Gomes, Florence Alberge & Pierre Duhamel
Laboratoire des signaux et systèmes (I.2S) Université Paris-Saclay CNRS CentraleSupélec 91190, Gif-sur-Yvetee France. feduarde. dadalto, florence alberge, plt..r..cciaaclccccrrreeel......
Pablo Piantanida
International Laboratory on Learning Systems (ILLS) McGill ETS MILA CNRS Université Saclay CentraleSupélec
H3C IK3 Quebec, Canada
piant @mila. quebee
ABSTRACT
Reliable out-of-distribution (OOD) detection is fundamental to implementing safer modern machine learning (ML) systems. In this paper, we introduce IGEOOD, effective method for detecting OOD samples. IGEOOD applies any pre-trained neural network, works under various degrees of access to the MI. model, does not require OOD samples assumptions the OOD data but also benefit (if available) from OOD samples. By building on the geodesic (Fisher- Rao) distance between the underlying data distributions, our discriminator can combine confidence scores from the logits outputs and the learned features of a deep neural network. Empirically, we show that IGEOOD outperforms competing state-of- saate-of-the-ad art methods on variety of network architectures and datasets.
INTRODUCTION
Deep neural networks (DNNs) reach the state-of-the-art in several classification tasks as they are known to generalize well on data with a distribution close to the training set. Whereas, in many prac- tical applications, the training set does not reflect well enough the real-life environment (Quionero- Candela et al. 2009) which is often non-stationary and sometimes with unpredictable events. There- fore, matching the training scenario to reality can be impossible or too complex. The inability of machine learning (ML) models to adapt to non-stationary distributions could limit their adoption in mission-critical systems (e.g., autonomous devices, healthcare applications).
Out-of-Distributioon (OOD) novelty detection is of the main objectives in conceiving reliable MI systems (Amodei et al. 2016). A typical application monitoring MI--based online services for periodically shifting distributions. However, tracking changes in the underlying data distribution is challenging as they contain unusual (irregular or unexpected) events and have large dimensions. For instance, relying on the intrinsic properties of ML models and their statistical behavior in the presence of in-distribution data is essential to identify OOD samples. Classic approaches to OOD detection consist of deriving metries for detecting those abnormalities from the lens of ML models (e.g., softmax output, latent representations across layers), provided that often only a single test example available. Furthermore, these metrics are subject to potential limitations inherent in practical scenarios depending the level of access to information in the ML model, c.g., having access only to the last layer or to all intermediate layers.
The baseline approach for OOD detection relies the predictive uncertainty of DNNs. Hendrycks & Gimpel (2017) demonstrated that OOD samples, in general, induce DNN classifiers to output less confident softmax scores, while existing state-of-the-ar methods classification problems still output high accuracy even under dataset shift. For instance, Ovadia et al. (2019) show that the accuracy of the underlying DNN increases, the supervisors outlier detection accuracy also

---

Published as a conference paper at ICLR 2022
improves. Unfortumately, also the variance increases. Henriksson al. (2021) observed that small changes in model parameters that marginally impact the accuracy could have a degrading impact on the performance of the OOD discriminator. This challenge is not exclusive to discriminative models. Deep generative models also fall short in discerning OOD from in-distribution samples. [Nalinnic]] et al. (2019) raise awareness of the fact that deep generative models also may output higher likeli- hood to OOD samples. They show that, even though the samples from the in-distribution CIFAR-I0 (Krizhevsky al. 2009) dataset (e.g., cats, dogs, airplanes, ships) are conceptually and visually different from house numbers from SVHN (Netzer et al. 2011) dataset, DNN-based classifiers may still assign a high likelihood to SVHN samples.
In this paper, we propose IGEOOD, new unified and effective method to perform OOD detection by rigorously exploring the eeoommtiin--eommerie properties of the feature space on various depths of a DNN. IGROOD provides a flexible framework that applies to any pre-trained softmax neural clas- sifier. A key ingredient of IGEOOD is the Fisher-Rao distance. This distance is used as an effective differential geometry tool for clustering and as a distance in the context of multivariate Gaussian pdf's (Pinele et al., 2020, Strapasson et al. 2016). In our context, we measure the dissimilarity be- tween probability distributions (in and out), as the length of the shortest path within the manifold induced by the underlying class of distributions (i.e., the softmax probabilities of the neural classi- fier or the densities modeling the learned representations across the layers). By doing so, we can explore statistical invariances of the geometric properties of the learned features (Bronstein et al., 2021). Our method adapts to the various scenarios depending on the level of information access of the DNN and uses only in-distribution samples but can also benefit (if available) from OOD samples.
1.1 CONTRIBUTIONS
Our work investigates the problem of OOD detection and advances tfae-oo-the--a in different ways.
i To the best of our knowledge, this the first work studying information geometry tools to devise a unified metric for OOD detection. We derive an explicit characterization of the Fisher-Rao distance based on the information- -eeommrri properties of the softmax probabilities of the neural classifier and the class of multivariate Gaussian pdfs. In general terms, our the mismatch-in in the geometry space- space-betweet between the probability density functions of the pre-trained DNN classifier conditioned on test and in-distribution samples. Section Daction3detee details IGBOOD.
metric measures
Experiments on BLACK- -Bo and GREY-BOR setups using various datasets, architectures, and classification tasks show that IGFOOD is competitive with state-of-the-art methods. In the BLAACK- Box setup, we assume that only the outputs, i.c., the logits of the DNN, are available. In the GREY-BOX setup, we allow access to all parameters of the network; however, the detection must be performed using only the output softmax probabilities. The latter permits input pre- rrepprccssiigg which introduces small (additive) noise in the direction of the gradients w.r.t the test sample. This pre-processing allows for further discrimination between in- and out-of-distribution samples. Our benchmark contains two DNN architectures, three in-distribution datasets, and nine OOD datasets.
ii In a WHITE-BOX setting, we combine the logits with the low-level features of the DNN to leverage further useful statistical information of the encoded in-distribution data. We model the pre-trained latent representations as a mixture of Gaussian pdfs with a diagonal covariance matrix. Under this assumption, we derive a confidence score based on the Fisher-Rat Rao distance between con- ditional pdfs corresponding to the test and the closest in-distribution samples. Experiments based on various datasets, architectures, and classification tasks clearly show consistent improvement of IGBOOD, achieving new state-of-the-an -hettrrrtt performance increased the average TNR at 95% TPR by 11.2% with tuning on OOD data and by 2.5% with tuning on adversarial data compared to Lee et al. (0018).
a couple of benchmarks. In particular, we
1.2 RELATED WORKS
OOD discriminators consist of binary classifier to distinguish between in- and
samples. A few works (Shalev et al. 2018 Hendrycks et al.] 2019 Bitterwolf et al. 2020 Mohseni et al. 2020; Winkens al. 2020 V Vyas et al. 2018 Hein et al. 2019) propose retraining the base (or an auxilarry model with synthetic or ground truth OOD samples to serve as a classifier and OOD discriminator. Disposing of both OOD and in-distribution samples during training en-
2

---

Published as a conference paper at ICLR 2022
ables the latent representations to learn the decision boundaries to facilitate OOD detection. These methods will not be compared to ours in this work, as they entail retraining or modifying the base neural network by using OOD data to further train parameters. Nagarajan et al. (2021) studies fail- ure modes of OOD detection methods to better understand how to improve them, especially how spurious features like the background can vastly degrade detection performance. [Lee et al. (202T) leverage OOD data as a regularization technique to improve the generalization and robustness of current neural networks. References (Schlegl al. 2017: Kirichenko et al.) 20200, Choi & Jang 2018 Vernekar et al. 2019; Xiao et al. 2020 Ren et al. 2019) et [ Mahmoood et al.,
2021 Zhang et al. [020] mas & Tamar 2020) study OOD detection in the context of gen-
erative models for density estimation. Open set recognition (Bendale & Boult 2016), outlier or anomaly detection (Pimentel et al. 2014), concept drift detection (Quionero-Candela et al. 2009). and adversarial attacks detection (Goodfellow et al. 2015
Madry et al. 2018 are related topics.
BLACK-BOS and GREY-BOX scenarios. It is often the case on MI. as service (Ribeiro et al.) 2015) that the model's parameters knowledge and access are not allowed to the end-user, granting access only to the computation of the forward and the logits or softmax outputs. The baseline work (Hendrycks & Gimpel| 2017) for BLACK-BOR techniques simply consider the unscaled maximum value of the softmax (MSP) as OOD score. In some cases, this confidence score is enough to distin- guish between in-distribution and values to OOD examples (Hein et al. 2019)] ODIN's (Liang et al. (018)) method has two variations. The BLACK-BOX variation consists of temperature scaling the softmax outputs. While the GREY- Box variation also uses an input pre-processing technique that calculates the gradient of the model parameters and adds to the input in an adversarial manner for a more effective OOD detection. Hsu et al. (2020) proposes variation of ODIN that does not need access to OOD data for validation. Liu et al. (020) proposes an energy- based OOD score. They substitute the softmax confidence score with the free energy function with a temperature parameter without retraining. They also propose GREY-BOX variation with posterior processing for improved results. Fine-tuning is done differently across the literature and should be considered when comparing methodologies.
examples, but it also may assign overconfident
THX scenario. This class of OOD detectors has access to all intermediate layer outputs. Naturally, discriminators have access to more information than the BLACK-BOX GREY-BOx statistics between layers are used (Quintanilha et al. 2019) to fit a logistic regression that serves as an OOD detection score. [aas try & Oore (2020) proposes high order Gram matrices to perform OOD detection by computing class-conditional pairwise feature correlations between the test sample and the training set across the hidden layers of the network. Lee et al. (2018) assume that latent features of DNN models trained under the softmax score follow a conditionl Gaussian mixture distribution with tied covariance matrix and different class-conditional mean vectors. They calculate the Mahalanobis distance between a test sample as a single estimator of the mean of a caas-coondiiinal Gaussian distribution with a tied covariance matrix estimated on the training set. The importance of each low- level component and hyperparameters are tuned using validation data, Ren et al. (202T) modifies this method to improve detection of near-OOD data. They fit the layer-wise background distribution with a Gaussian distribution fit from the training set. They subtract the Mahalanobis distance be- tween the test example and this distribution from the score proposed in Lee et al. (2018), reducing the importance of features shared by in- and out-of- oi--iiitiittiion data.
tups, warranting greater detection capacity.
BACKGROUND
Let X' C Rdd be the feature space (continuous) and y label space. Moreover, let pxy be the un- derlying unknown probability density function (pdf) over X' x Y. We define the in-distribution training dataset as DN A {( (1),,))) ~ Pxy, where x; E X is the input feature data, Vi E y A (1,...,C) is the output class among C possible classes and N denotes the number of training samples. The training dataset is characterized by the joint pdf pxy with in-distribution marginals X px and Y ~ Py. The predictor denoted by fDN : X + y is based on the inferred model Ppix- i.., fDr (x) = fz; DN) & arg maxzey Peix (1/ xx; Dw). In order to model the underlying problem, we introduce an artificial binary random variable Z E {0. 1) indi- cating with z 1 that the test sample x is OOD and otherwise, it is in-distribution. The open- world data can then be modeled a mixture distribution px|z defined by (x)2 0) a
3

---

Published as a conference paper at ICLR 2022
Gaussian parametric space
Data pdf
Mahalanobis
Fisher-Re Rao
In-dist.
Frequeney
In-dist.
Standard deviation
OOD
OOD1 I
OOD
0'2
Mahalanchis
Frequency
OOD II
OOD
Fisher-Ro Rao
Frequency
In-dist.
ln-dist.
OOD I
O
OOD
Mi
Hz
OOD score distribution
Mean
(a) Contour lines.
Synthetic data distribution.
(c) OOD detection histogram.
Figure I: Example comparing Fisher-Rai Rao with Mahalanobis distances to distinguish between 1D Gaussian distributions, showcasing the motivation to use of Fisher-Rao metric for OOD detection.
(x), and (x|z = 1) (x). The intrinsic difficulty arises from the fact that very lit- lle can be assumed about the unknown distributions px and qx in particular for out-of-distribution.
IGEOOD: OOD DETECTION USING THE FISHER-RAO DISTANCE
This section introduces IGEOOD, a flexible framework for OOD detection. IGEOOD implemented in two ways: at the level of the logits using temperature scaling (Section 3.2), which mitigates the high-confidence scores assigned to OOD examples, and layer-wise level (Section 3.3 The key ingredient of IGEOOD is the Fisher-Rao distance that allows for effective differentiation between in-distribution and out-of ditoof-distribution samples. This distance measures the dissimilarity between two probability models within class of probability distributions by caleulating the geodesic dis- tance between two points on the learned manifold. This measure connects information geometry and differential geometry through the R. Fisher information matrix (Fisher 1922). Closed-form pressions of this distance are known to multivariate normal distributions under certain assumptions, among others distributions (Pinele al. 2020).
3.1 MOTIVATION FOR THE USE THE FISHER- RAO DISTANCE FOR OOD DETECTION
We introduce a simple example to demonstrate conceptually how Fisher-Rao distance instrumental OOD detection. It should be noted that this example is limited to dimension. However, expect similar behavior with more complex data under the Gaussianity assumptions.
Consider the case where we try distinguish between samples from distinct Gaussian distributions on ID. Assume that the in-distribution data follows Gaussian M(u, σ1) while OOD data is drawn according to either ((12 01) or ( (2). These distributions are illustrated in Figures la and 1b In this setup, iittance-based approaches which are invariant to the variance of the distributions would have the performance limited to the information given by the difference between the means of the underlying distributions. For instance, in the case of the Mahalanobis distance, we would rely our discrimination on the difference between the sample and the in-distribution mean, rescaled by the in-distribution standard deviation only, but nothing further could be obtained. However, if can estimate OOD standard deviations from actual or pscudo OOD data, we expect the Fisher-Ran distance between Gaussian distributions be more effective in distinguishing between distributions. Figure shows that the Fisher-Ran Rao distance distinguishes better between "In-dist." and "OOD II" samples, while the other distances fail.
3.2 IGEOOD SCORE USING THE SOFTMAX PROBABILITY
The Fisher-Rai distance AAkinson & Mitchell 1981 takes as input two probability distributions.
For the classification problem, take the temperature T scaled softmax function (Eq. (T) as
4

---

Published as a conference paper at ICLR 2022
an approximation of a class- probability distribution:
go (ylf()):T) & exp (xp(fx(x)/T)
(I)
Evey exp (fv (x)/T)
where / X - RC vectorial function with / - ( f2...., Sc) and [2(+) denotes the y-th logits output value of the DNN classifier. The Fisher-Rao distance dER Logits between two distributions resulting from the softmax probability evaluated at two data points is (see Appendix[A]:
dFR dFR-Logits (1) - arccos
ao (o WEY
Σ
(2)
Class conditional centroid estimation. We model the training dataset class-conditional posterior distribution by calculating the centroid of the logits representations of this set. Precisely, we compute the empirical centroid for the logits of each class E E = {1, C} of the in-distribution training dataset Dy corresponding the Fisher-Ras Rash distance, i.e.,
Σ Vi:ym
My
min MERC
N
where N, is the amount of training examples with label y. We optimize this expression offline using SGD algorithm, where the parameter be tuned is la in the logits space. This equivalent finding the centroid of cluster using the Fisher-Rao distance, after each example has been assigned to a cluster. Please refer to the appendix (see Section B) for further details on this optimization.
OOD and confidence score. Using the softmax probability, we can define a confidence score to be the minimum of the Fisher-Rao distance between [(x) and the class-conditional centroids. As sanity check, we show empirically in the appendix (see Section C) that this confidence score does not degrade the in-distribution test classification accuracy. Thus, the estimated class OFR follows as:
OFR (z)
arg min dFR- (go( (=0(-((()))) ge(-)w,)) yEy
However, we obtained slightly better OOD detection performance by using Eq. 3 instead of the minimal value. A likely explanation would be that this metric uses extra information from the other logits dimensions. We provide an empirical study comparing both methods in the appendix (see Section E. Thus, we propose the Fisher-Rao distance-based OOD detection score FRo(#) for the logits to be the sum of the distances between f(x) and each individual class conditional centroid p,y given by Eq. (3). By taking the sum instead of the minimal distance, we leverage useful information related to the example's 's confidence score for each class y. We denote it by
(Ro
A Σ dFR-Logita (go (-1[(z)) go (111))
(5)
yEy
Input pre-processing. In consonance with the literature (Liang et al. 2018 Liu et al. 2020: Lee et al. 2018), we also perform input pre-processing to enhance the detection between in distsrribution and OOD samples and potentially improve OOD detection performance for the GREY-Bop dis- criminator. We add small magnitude perturbations in Fast Gradient-Sign Method-style (FGSM) (Goodfellow et al. 2015) to each test sample æ to increase the proposed metric, that is:
.े + sign V (R)
The OOD detector. The detector consists of a threshold-based function for discriminating between in-distribution and OOD data. This threshold and parameters set so that the true positive rate, ie., the in-distribution samples correctly classified as in-distribution, becomes 95%. Mathemati- cally, the BLACK- Box OOD detector JBB and the GREY-BOx OOD detector JGB writes:
9BB (a:; (B 5, = {
1
FR, (x) < and JGB (a: T,T,5) 1 if FR, (x)
if FR,
a T
<
(7)
0
if FR
5

---

Published as a conference paper at ICLR 2022
3.3 IGEOOD SCORE LEVERAGING LATENT FEATURES
For each layer, we define a set of class-conditional Gaussian distributions with diagonal standard deviation matrix ((1) and class-conditional mean M4 the latent feature. We compute the empirical estimates of these parameters according to
(8) where y (I....,c) and is the index of
W (z.)
Σ Nu Vi:WE=
(2)), and o (E) diag N DEDVE:BEy
(2) pb.
(8)
where j E (l,...,k), k is the size of feature E, and (-) is the output of the network for feature The Fisher-Rao distance prn between two arbitrary univariate Gaussian pdfs N (11, of) and N(22, oB) is given by (See Section
prn (1) (12, V2log
11 1 -
01)
v2
12 -02
+
V2:01
V2"
01
122 V2
02)
Mi 1/2
Me
Mi /2
-
Similarly, the Fisher-Rao distance dFR- Gause between two multivariate Gaussian pdfs with diagonal standard deviation matrix is derived from the univariate case and is given by
dFR-Gause (), (w', o'))
Σ PEr ((H), os,)) (44, o
(10)
where & is the cardinality of the distributions [(,,, σ)) and NNu', o'), br is the i-th component of the vector Ja, and og, is the entry with index (i, of the standard deviation matrix
Experimental support for a diagonal Gaussian mixture model. It is known that intermediate fea- tures of DNN can be valuable for detecting abnormal samples demonstrated Lee al. (2018). Nonetheless, we observed that the latent features covariance matrices are often ill-conditioned and are diagonal dominant. In other words, the condition number of the covariance matrix often di- verges, and the magnitude of the diagonal entry in a row is greater than or equal to the sum of all the other entries that row for most rows. Thus, diagonal covariance matrix will be a favorable compromise for OOD detection. See Appendix, Section B.3 for further details.
Fisher-Rao Rao distance-based feature-wise confidence score. We derive confidence score by ap- plying the Fisher-Rao distance between the test sample æ and the closest class-conditions diagonal Gaussian distribution. Contrarily to the logits, taking the sum did not improve results, so we kept the minimal distance. We can consider two scenarios: (i) We do not have access to any validation OOD data whatsoever. In this case, the natural choice is to model the test samples as Gaussian distribution with the same diagonal standard deviation as the learned representation, i.e..
FR, min dpR-Gauss ((xr, a ())), (p (()));
(11)
yey
and (ii) we dispose of validation OOD dataset which the features' diagonal standard deviation matrices of and the means (f) can be estimated, as well as the quantity:
(x)
min drR- drR-Gause ((2, o (9), (()(() (()).
(12)
yey
This validation dataset could be obtained from synthetic dataset, dataset different from the testing one, or even by adversarially creating OOD data by attacking the classifier model on the training dataset. In the appendix (Section B include pseudo-codes for calculating the IGEOOD score for the BLACK-BOX -OX, GREY-BOX, and WHITE-BOX settings.
Feature ensemble. To further improve performance, combine the confidence scores of the logits and the ones from the low-level features through linear combination. Similarly to the strategy in Lee et al. (2018), we choose the weights ag, Og and af E R by training logistic regression detector using validation samples. Thus, we ensure that the metric emphasizes features that demonstrate a greater capacity for detecting abnormal samples. IGEOOD score for the WHITE- Box setting
FR(x) (x) (R)()) + Σ as FRe(2) + oa FR)()),
(13)

---

Published as a conference paper at ICLR 2022
where FR, is given by equation 5 FR, is given by equation 11 and FR' considers different validation diagonal covariance matrix for the test samples (equation 122 We also apply imput pre-processing similarly to the GREY-BOX setting (equation ( obtaining FR( as final score.
Unified metric. For the three settings, the metric is the same but has different formulations given the family of the distributions. For the DNN outputs, we the softmax posterior probability distribution formulation. For the intermediate layers, it under the model of diagonal Gaussian pdfs. Therefore, we have derived unified OOD detection framework that combines a single distance for both the seftmax oulputs and the latent features of a neural network. Figure 2 illustrates how each of the presented techniques contributes towards separating m-diisrribttion and OOD samples. Additional histograms of the detection scores are relegated to the appendix (see Section())
0.75
In-distribution
Indisttributin
In-tistribution
Fequeney
Out-of-distribution
Outtof-ditrribttion
Out-of-distrbution
0.50
0.25
0.00
15
20
25
15
20 (b) Crey box
25
80
90
100
110
(u) Black bux
(c) White box
Figure 2: Probability distributions of the IGEOOD score under three different settings for pre-
trained DenseNet on CIFAR-10 for in-distribution and OOD data (TinylmageNet downsampled).
4 EXPERIMENTAL RESULTS
We show the effectiveness of IGFOOD comparing to state-of-the-art methods. Details about the experimental setup] I and additional results are given in appendices (see Sections D and F
4.1
SETUP
The experimental setup follows the setting established by Hendrycks & Gimpel
2017 Liang et al.
2018) and [ee et al. (018). We use two pre-trained deep neural networks architectures for image classification tasks: Dense Convolutional Network
BenneNettBBC-10) (Huang et al. 2017) and
a Residual Neural Network (ResNet-34) (He et al. 2016). We take as in-distribution data images
from CIFAR-10 (Krizhevsky al. 2009). CIFAR- 100 and SVHN (Netzer et al. 2011 datasets.
For out-of--distribution data, we use natural image examples from the datasets: Tiny-ImageNer (Le &
Yang, 2015), LSUN (Yu et al., 2015). Describable Textures Dataset (Cimpoi et al. 2014), Chars74K
(de Campos et al. 2009), Places365 (Zhou et al. 2017). iSUN (Xu et al. 2015) and a synthetic dataset generated from Gaussian noise. For models pre-trained on CIFAR-10. 10, data from CIFAR-100
and SVHN are also considered OOD; for models pre-trained on CIFAR- 100, data from CIFAR-10
and SVHN considered OOD, and for models pre-trained
SVHN CIFAR-10 and CIFAR- 100 datasets are considered OOD. We resize the images to dimension 32 x 32 by downsampling and applying center crop when needed. We only use test data for evaluation. Even though ran experiments with image data, IGEOOD could be applied to any neural-based classification task.
We measure the effectiveness of the OOD detectors with three standard evaluation metrics: (i) The true negative rate at 95% true positive rate (TNR at TPR-95%); (ii) the area under the receiving operating curve (AUROC); and (iii) the area under the precision-recat curve (AUPR). We use the scores over the test set of in-distribution and OOD datasets to calculate them. For the BLACK-Bop and GREY-B Box experimental settings, we tune hyperparameters for all of the OOD detectors only based on the DNN classifier architecture, the in-distribution dataset, and validation dataset. The iSUN (Xu et al. 2015) dataset chosen as a source of OOD validation data, independently from OOD test data. We choose the parameters that maximize the TNR at TPR-95% on the validation OOD dataset. For the WHITE-Boo -Bx framework, we allow both the benchmark and our method to
Our code is publicly available at httppps//tttttb...///eaaaaaaaaaa

---

Published as a conference paper at ICLR 2022
tune either adversarially generated data from in-distribution training samples separate vali- dation dataset containing 1, 000 images from the OOD test dataset with feature ensemble described in Section 3.3 In this case, we evaluate performance on the remaining test samples.
4.2 RESULTS FOR THE BLACK-BOO BLAC--BOX AND THE GREY-BOX SETUPS
For comparing IGEOOD under the hypothesis of a BLACK Box scenario, we consider the Baseline (Hendrycks & Gimpel 2017) method, ODIN (Liang et al. 2018) with temperature scaling only, and the free-energy-basee based metric (Liu et al. 2020) with temperature scaling only. The results for the BLACK-BOX setting are available in Table where
show the average and one standard deviation OOD detection performance for each of the eight OOD detection method in six different image classification contexts (couple DNN model and in-distribution dataset). The extended results for each OOD dataset can be found in Table 13 For comparison under the GREY-Box assumption, methods, both with input pre-processing. The results we consider ODIN and the free-energy-basee
for the GREY-Bok Box setup are provided in the appendix (see Section Eld Table For the BLACK- Box setting, IGEOOD slight improves the benchmark by less than in TNR TPR-95%. While for the BRREY-BX setting, results show IGEOOD is outperformed by <1% in few benchmarks by ODIN, which is greatly improved by input pre-procee processing techniques.
%
Table 1: Average and standard deviation OOD detection performance across eight OOD datasets for each model and in-distribution dataset in a BI.ACK- Box setting. IGROOD compared to Baseline (Hendrycks & Gimpel 2017), ODIN (Liang et al. 2018). and Energy (Liu et al. 2020) methods.
The extended results can be found Table 13 in the appendix.
TNR at TPR-95%
AUROC
Model
In-dist.
Baseline / ODIN/ / Energy / IGROOD (ours)
C-10
91.8:an:/9..11444221111.
DenseNet
C-100
69.1111/71.....2771......
SVHN
6844::w66..4-:77702217777...82
C-10
44.2215519H1//553:1555.111
ResNet
C-100
15.02ss//16.02333/1133..////
SVHN
762=:a//7..2210784555//8...11 84.8..o888.88s::55522.....
Average and Std.
449.:0950.312255511225566..
Temperature scaling and input pre-processing. We observed that low values of temperature and moderate noise magnitude yield better detection performance for IGEOOD on the logits. For most models and datasets, we obtained better results for temperatures between and 6 and noise magni- tudes below 0.002. Detailed results and the best hyperparameters found for each configuration, as well as figures of their impact on performance, are delegated to the appendix (see Section E).
How the choice of validation dataset impacts performance. We include in the appendix (see Section[]] the average OOD detection performance for each method when we change the validation set among the nine available ones. We show that the average TNR at TPR-95% for IGBOOD ranges between 63% and 72% on a BLACK-BOX scenario and between 65% and 74% on GREY Box scenario. The performances among the compared methods are consistent across validation datasets.
4.3 RESULTS FOR THE WHITE-BoL SETTING
For benchmarking IGFOOD on the WHITE-BOX setting, we compare results to the Mahalanobis (Lee al. 2018) method with input pre-processing and feature ensemble. For both of them, we extract features from every output of the dense (or residual) block of the DenseNet (or ResNet) model and the first convolutional layer. The size each feature is reduced by average pooling in the spatial dimensions. Thus, the initial dimension Fe We He is reduced to Fe. where Fe is the number of channels in block For DenseNet, this reduction translates to features of sizes F, (24, 108, 150, 342); and for ResNet, to features of sizes F, (64, 64, 128, 256, 512).
We consider two scenarios for tuming hyperparameters for both Mahalanobis and IGEOOD: one with adversarially generated (FGSM) and in-distribution data and another with 1, 1,000 OOD samples
8

---

Published as a conference paper at ICLR 2022
and in-distribution data. We derive two methods: IGEOOD+, which is given by equation (3) and considers that we can calculate the statistics from OOD data as additional information; and IGEOOD, which doesn't 't consider any prior on OOD data, i... set o/ 0 on equation (13)
Comparison with current literature. For each DNN model and in-distribution dataset pair, we report the average and one standard deviation OOD detection performance for Mahalanobis (Lee ett al. 2018), IGEOOD and IGEOOD+. Table 2
validates the contributions of our techniques. We observe substantial performance improvement all experiments for the left-hand side of the table, where we outperform Mahalanobis on average for all test cases. IGEOOD+ show improvements of least 2.1% up to 23% on TNR at TPR-95%. Since the results are usually above 90%, these improvements are significant. To assess the consistency of IGEOOD to the choice of validation data, we measured the detection performance when all hyperparameters are tuned only using in- distribution and generated adversarial data, observed in the right-hand side of Table 2 IGROOD record improvements up to 10.5%, and improves by 2.5% the average TNR TPR-95% across all datasets and models. We provide an extra benchmark against other WHITE- Box methods (Sastry & Oore 2020
Hsu et al. 2020 Zisselman & Tamar 2020
(see Table in the appendix).
Table Average and standard deviation OOD detection performance for the WIITE-BOXX settings. The abbreviation TNR-95% C-10 and C-100 stands for TNR at TPR-95% CIFAR-I1 10 and CIFAR-
100, respectively. The extended results can be found in Tables I5 and 16 in the appendix.
Validation on OOD data
Validation on adversarial data
TNR-95%
AUROC
TNR-95%
AUROC
Model
In-dist.
Mahalanobis/ IGEOOD+ (ours)
Mahalanobis IGEDOD (ours)
C-10
76.6:31/92.6:14
92.1=12/98.4:33
75.9#m/77.9942
91.7:12/94.0=99.0
DenseNet
C-100
67.2+2w990.2 90.2=13/97.7:550
60.4134/70.9433
85.3:19/90.8:11
SVHN
93.3=x.0/8.. 98/92..1
93.7211/92.2:490
98.6:2.n/98.42111
C-10
82.512:/91.6116 96.514.n/88.4..00
78.6124/77.3.222
15.31..090000115
ResNet
C-100
70.41x/86..411
10.9.1097..11.50
57.41.6665.115
86.9113/88.66111
SVHN
96.8=6.//98.9422
99.2=1.099.7
96.3+a../3.6:+14
99.1+1.0988.4330
Average and Std.
81 11/.:1992..
94.854.
77.0=15/79.5+11
92.8t5../93.4455
Ablation study. IGEOOD has three components, FRg, FR,, and FR', that together compose the final metric of equation (13). The outputs of the network provide limited OOD detection capacity as observed in T When available, the intermediate features, i.e., FR,, are valuable resource for OOD detection. Moreover, when few reliable OOD data are available, calculating FR' can further improve the detection performance (left-hamd side column of Table 2). Also, data from source other than in-distribution, c.g., adversarial samples, is enough for tuning hyperparameters and combining features (right-hand side column of Table 2). The detection capacity of each hidden layer before any tuning is studied in Appendix [B.4] Experiments show that the Fisher-Rao metric effectively separates in- and out-of-distribttion data for each of the features individually as well.
SUMMARY AND CONCLUDING REMARKS
This paper introduces IGEOOD, effective and flexible method for OOD detection that applies to any pre-trained neural network. The main feature of IGEOOD relies on the geodesic distance of the probabilistic manifold of the learned latent representations that induces an effective measure for OOD detection. First, in a (GREY-) BLACK-BOX setup, we calculate the sum of the Fisher- Rao distance between the softmax output, corresponding to the test (pre-processed) sample, and reference probability, corresponding to the conditional-class. of softmax probabilities. Similarly, in WHITE-BOX setup, we model the low-level features of DNN as a diagonal Gaussian mixture. The Fisher-Rao distance between the pdf of the latent feature, corresponding to the test sample, and reference pdf, corresponding to the conditional-class of pdfs, provides an effective confidence score. We considered diverse testing environments where prior knowledge of OOD data may may not be available, reflecting diverse application scenarios. observed that IGEOOD significantly and consistently improves the accuracy of OOD detection several DNN architectures across various datasets for a WHITE- Box setting. Some perspectives for future work include studying causal factors, explainable components for OOD detection, and extensions textual data.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work has been supported by the project PSPC AIDA: 2019-PSPC-09 funded by BPI-France.
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. CoRR, Abss/606.00565,, 2016. URL (attp:///aaxxi
abs/1 606..6666... 06565
Colin Atkinson and Ann S. Mitchell. Rao's distance measure. Sankhyā: The Indian Journal of Statistics, Series A (1961-2002), 43(3):345-365, 1981. ISSN 0581572X. URL http: /wwww.
jstor.
Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1563-1572, 2016. doi: 10. 109/CVPR. 2016.173.
Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of duitt--dist data. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16085- 16095. Cur-
ran Associates, Inc., 2020, / www.eedddinnnnnnriiipp.eemm wwtpsnewww...ppeeeeeeeeeee wttppppppppp.c///////p// 2adeee22eeeeeaaaeeeeeeaa022
fi
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges, 2021.
Hyun-Jae Choi and Eric Jang. Generative ensembles for robust anomaly detection. ArXiv, abs/1810.01392, 2018
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Patlern Recognition (CVPR), 2014.
T. E. de Campos, B. R. Babu, and M. Varma. Character recognition in natural images. In Pro- ceedings of the International Conference on Computer Vision Theory and Applications, Lisbon, Portugal, February 2009.
J. Deng, W. Dong, R. Socher, L.-J Li, K. Li, and I.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
R. A. Fisher. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, A, 222:309-368. 1922.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015. URL http://
arxiv. org. /a0os.141..555 6572
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10.1109/CVP..2016.990
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 41-50, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019. URL https:
openreview net (etes/ooorrmmmdlxyxxxxxceer
10

---

Published as a conference paper at ICLR 2022
Jens Henriksson, Christian Berger, Markus Borg, Lars Tomneer, Sankar Raman Sathyamoorthy, and Cristofer Englund. Performance analysis of out -uttt---iittiuution detection on trained neural networks. Information and Software Technology, 130:106409, 2021. ISSN 0950-5849. doi:
106499. URL httpt::/t///... /wwww sci encedi rect. com/
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10948-109577 10957, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2261-222222222222 2269, 2017. doi: 10. 109/CVPR.2017.222
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail detect out-of-distribution data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20578-20589. Cur- ran Associates, Inc., 2020. URL
newrins wttp:/ippp.ccppppe////222/2 paf
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensembles. In L. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL
Vishwanathan, and R. Garnett
Paper. pdf
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified frame- work for detecting out-of-distribttio samples and adversarial attacks. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7167-7177. Curran Associates, Inc., 2018. URL http::/papers..ips...... www.aaprrr.iiss..omm wwt.:::s..inss.cc.pparaaa 7947- a- simple-unified framework- for hta.aa/eeeaaaaaaaeoe.eeooooeoorr
pdf
Saehyung Lee, Changhwa Park, Hyungyu Lee, Jihun Yi, Jonghyun Lee, and Sungroh Yoon. Re-
moving undesirable feature contributions using out-of-distribution data. In International Confer- ence on Learning Representations, 2021. URL https::/ /ttps:://pperevvee...ett www.eeeee/ooiiiiiee
@IHYL6fpbkA,
Shiyu Liang, Yixuan Li, and R. Srikant. Enhaneing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018. URL htt.a//aeaaepeeaeeeeeaeeeeeeeaaa
net/ http:/oormmmddikkkiI
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out-of-distribution detec- tion. Advances in Neural Information Processing Systems, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Leam- ing Representations, 2018. URL htt |Rttttts:: /tttps:://penreviieee net wtte://oormmmiddIi11e.
Prasanta Chandra Mahalanobis. On the generalized distance in statisties. Proceedingy of the Na- tional Institute of Sciences (Calculta), 2:49-55, 1936.
Ahsan Mahmood, Junier Oliva, and Martin Andreas Styner. Multiscale score matching for out-of- distribution detection. In International Conference on Learning Representations, 2021. URL https:://ppenewiier .etteeewweeeeeeeeeeeeeeeeeeeeeee htw..etee/orrummllooooooooogogg

---

Published as a conference paper at ICLR 2022
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-super learning for generalizable out-of-distribution detection. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04))5216-522) Apr. 2020. doi: 1..1600/aai.vv34000 v34104.5966. URL https://
/ ojs. wtt.://ro//ii..iooooo http://AAA/atttt.cl/////e//////s
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. In International Conference on Learning Repre- sentations, 2021. URL
net/ ittp://rwewieettttttooorooeeaaae
Ib
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do
deep generative models know what they don't know? In International Conference on Learning Representations, 2019. htt.sttts: http//pprrrvvveeee net/ forum? c= xwNhCcYm
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading pdf
digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http:://fld.l.. www.ssrr.aa....o.ooo /du/
housenumbers///ipp.... wwwp:si
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snoek. Can you trust your model's uncertainty? evalu- ating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and Garnett (eds.), Advances Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL bttpst http://pppppppppppeeeeeeeeeee//e www.:iirinnnnnn.nsss...o.o....se Par.
Marine Picot, Francisco Messina, Malik Boudiaf, Fabrice Labeau, Ismail Ben Ayed, and Pablo Piantanida. Adversarial robustness via fisher-rao regularization. ArXiv, abs/2106.06685, 2021.
Marco Pimentel, David Clifton, Lei Clifton, and L. Tarassenko. A review of novelty detection. Signal Processing, 99:215-249 06 2014. doi: 10. 2..0066jsigppo22011....22.
Julianna Pinele, João E. Strapasson, and Sueli I. R. Costa. The fisher-rao distance between multi-
variate normal distributions: Special cases, bounds and applications. Entropy, 22(4), 2020. ISSN 1099-4300. doi: 10.3390/e22040404. URL https: www. http:p....mmmm0 1tt.:o/mm/19994300/002 4/404
Igor M. Quintanilha, Roberto de E. Filho, José Lezama, Mauricio Delbracio, and Leonardo O. Nunes. Detecting out-of-distribution samples using low-order deep features statistics, 2019, URL
https:://pperrey: net forum? fo
Joaquin Quionero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence.
Dataset Shift Machine Learning. The MIT Press, 2009. ISBN 0262170051.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL http::/ htwp://cweedddng...eie.iee...eee cc httpp/pppp/////11/eee.
pdf
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. A simple fix to mahalanobis distance for improving near-ood detection, 2021.
Mauro Ribeiro, Katarina Grolinger, and Miriam A.M. Capretz. Mlaas: Machine learning as a service. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 896-902, 2015. doi: 201100/2ML.22201..522222
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribuue examples with Gram matrices. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. PMLR, 13-18 Jul 2020. URL https: httpss//proceednggg http:///poceeeeengeeeeee mlr wtwss/v1s9//
Bastrry20 html
12

---

Published as a conference paper at ICLR 2022
Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth. and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, 2017.
Gabi Shalev, Yossi Adi, and Joseph Keshet. Out-of-distribttion detection using multiple semantic label representations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL https: /ttp:t/tss/pppooedasss...eeereee
http://g.....aaca..ooo//ooo////a 2151b4c7 6b dcb 6d. 24222292226662 Papeeerrbbrpppp..p.p..... pdff
S. S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591-611 1965. ISSN 00063444. URL http::///e http:://wwwwsstr.... jstor. org/
Stable/233377
João E. Strapasson, Julianna Pincle, and Sueli I. R. Costa. Clustering using the fisher-rao distance. In 2016 IEEE Sensor Array and Multichannel Signal Processing Workshop (SAM), pp. 1-5, 2016.
doi: 2.1.......117777777777
Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, and Krzysztof Czarnecki. Out-of-distribution detection in classifiers via genera- tion. In Neural Information Processing Systems (NeurIPS 2019). Safety and Ro- bustness in Decision Making Workshop. https://sites. http://tteee.oooo...ooo//we/eeee nafe-robust-workkhhr 12/2019 2019.
URL http://ww...ooooo.c.c/m/////eeed
Apoory Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. Out-of-distribution detection using an ensemble of self supervised leave-out clas- sifiers. In ECCV (8), pp. 560-574, 2018. URL /tttps://doi.org/10...
978 978-3-030-01237-33-3
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Led- sam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon A. A. Kohl, tay- lan. cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved out-of- distribution detection. ArXiv, abs/2007. 0oos/200.00566, 2020.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485-3492 2010. doi: 10. 0..1099CCVRR220005539999
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-en In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20685-20696. 20696. Cur- ran Associates, Inc., 2020, URL UULLa nww.upss..........ooo..m........ cc/ wttppppp..ccpppp///2////// fi httee/dddaa222225522222222 htt.55222244411ee6eeceeeeeeeeea (cc5Fc/eeaa00--Paeeer....
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking, 2015.
Yu, Y. Zhang, Shuran Song, Ari Seff, and Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. ArXiv, abs/1500.03366. 2015.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In ECCV, 2020.
Yufeng Zhang, Wanwei Liu, Zhenbang Chen, Ji Wang, Zhiming Liu, Kenli Li, and Hongmei Wei. detection with distance guarantee in deep generative models, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Ev Zisselman and Aviv Tamar. Deep residual flow for out distribution detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

Published as a conference paper at ICLR 2022
A REVIEW OF FISHER-RAL RAO DISTANCE (FRD)
In this section, we review some results from references Atkinson & Mitchell (1981); Pinele et al. (2200) We intend clarify some basic concepts surrounding the Fisher-Rao distance while moti- vating this measure in the context of OOD detection.
In few words, the Fisher-Rao's distance is given by the geodesic distance, i.e., the shortest path between points in Riemannian space induced by parametric family. Consider the family of probability distributions over the class of discrete concepts or labels: y (l.....,Cr C}, denoted by é {-1() x E X S }
We interested in measuring the distance between probability distributions go(- (ala:) with respect to the testing input æ and a population of inputs drawn accordingly to the in-distribution data set. To this end, we first need to characterize the Fisher-Rao distance for two inputs or for two probability distributions Qe % E C.
Assume that the following regularity conditions hold (Atkinson & Mitchell 1981
(i V (v((z) exists for all æ, y and 0 E O;
(ii)
Σ Ve (o(())) = 0 for all æ and 0 E O;
(iii) G (x)) = (y) V log to V log (Y(=)) positive definite for any a and
0 E θ.
Notice that if (i) holds, (ii) also holds immediately for discrete distributions over finite spaces (as- suming that Eney and aæ are interchangeable operations) as in our case. When (i)-(iii) are met, the variance of the differential form V log (Yl=)) (n)da:: can be interpreted as the square of a differential are length ds? in the space C, which yields
ds = (dx, G(x)) = da T G(()dd
(14)
Thus, G. which is the Fisher Information Matrix (FIM), can be adopted as metric tensor. We now consider a curve 7 [0, 1] + X' connecting pair of arbitrary points x; x' in the input space X, i.e., 7(0) = and (1) x'. Notice that any curve y induces curve go (t)) for t [0 in the space C. The Fisher-Rao Rao distance between the distributions Go = 9-(-1a) and de = ?o (-(2x)) will be denoted dR,c(40, %6) and is formally defined by the expression:
(t) dgo %))= $ inf V.VV dr d
G(r(t)) dr(t)) dT()) dt
Sinf
(15)
dt
where the infimum is taken over all piecewise smooth curves. This means that the FRD is the length of the geodesic between points and x' using the FIM the metric tensor. In general, the minimization of the functional in equation 15) is problem that can be solved using the well-known Euler-Lagrange differential equation.
A.1 DERIVATION OF FISHER-RAO DISTANCE FOR CLASS SOFTMAX PROBABILITY DISTRIBUTIONS
The direct computation the FIM of the family with go (y(())) in the form of the softmax probabil- ity distribution function given by equation (T) can be shown to be singular, i.e., rGnx))( S C-1, where 1 is the number of degrees of freedom of the manifold C. To overcome this issue, we introduce the probability simplex P defined by
P q::0 - 10 Σdの)=
(16)
VEY
Next, we consider the following parametrization for any distribution g P:
<ull)) = 2/
4 yE (I.
(17)
14

---

Published as a conference paper at ICLR 2022
From this expression, {q(+1z) ||= 4, 2y 2 0, Vy E V}. Note that the parameter vector x belongs to the posi- tive portion of sphere of radius 2 and centered at the origin in R². The computation of the FIM for = on D yields:
consider the statistical manifold D
Gizz)
gllll
[ log e(yl=)
log [()l=)]
2
2
Σ
Ey
_
e
4
Fyy
Zzy
VEy
(18)
Σ
eyey
I.
where {ey} are the canonical basis vectors in RC and is the identity matrix. From equation 18 can conclude that the Fisher-Rao metric in this parametric space is equal to the Euclidean metric. Also, since the parameter vector lies on a sphere, the FRD between the distributions g = q(1 and = q (-I=²) can be written the radius of the sphere times the angle between the vectors and z'.
Which leads to expression:
z T=
dRD (q,9') 2 arrcos
2 arccos
Σ
(alviz)g(ulz))
(19)
WEY
Finally, we can compute the FRD for softmax distributions in as
Logits (4o, %o) 2 arccos
Σ DEy
(o((yk))g (2/x²)
(20)
obtaining the same form of equation 2 Notice that O < drR-Logie (40, 96) S π for all
x, x E X' s RC, being zero when (o(- = qo (-12') and maximum when the vectors (0(())))......(( and (yo (1|z"),..... , Qa (Clx') are orthogonal.
A.2 DERIVATION OF FISHER-RAA DISTANCE FOR MULTIVARIATE GAUSSIAN DISTRIBUTIONS
Consider a broader statistical manifold S A {Po p(æ; 0) 0 (0.,0.....,, Om) E θ of multi- variate differential probability density functions. The Fisher information matrix G(0) [g:;(0) in this parametric space provided by:
2 log pl:;
a log p(>;; 0))
9sj (0) IEg
a0,
00,
(21)
/
i ae,
log logp(x: 9
log p x: 0)p(2: 0 0)dr.
00,
Next, consider a multivariate Gaussian distribution:
(2))-
(x
T (x 11)
p(æ; u4
exp
(22)
/Del(E)
2
where x E R the variable vector, μ E R is the mean vector, E (R)R) is the covariance matrix, and Ps (R) is the space of & positive definite symmetric matrices. We can define the statistical manifold composed by these distributions as M {po: 0 = (u, Σ) E R$ x Ps (IR) By substituting equation (22) in equation 21). can derive the Fisher information matrix for this parametrization. obtaining:
0p
0u
1 tr Σ
0 ツー 2
gh (0)
(23)
30,
202
2
30,
00,
which induces the following square differential are length in M:
ds² du
T du
2 tr ( dz)
(24)
15

---

Published as a conference paper at ICLR 2022
Here, du (dui..... djv.) E R and dΣ invariant to affine transformations (Pinele et al. 2020), i.c., for any (c. Q) R (LR, with (LR(RR) the space of non- non-singue ain-sngular order l matrices, the map (n. + (Qu+ QIQT) is an isometry in M. Thus, the Fisher-Rao distance between two multivariate normal distributions with parameters 0, - (J1,, E1) and O2 (122, Σ2) in M satisfies:
[doyp]
(RR... We observe that this metric is
(25) Unfortunately, a closed-form solution for the Fisher-Rao distance remains unknown. This is still an open problem for an arbitrary covariance matrix Σ and mean vector μ. Fortunately, the FRD is known for the univariate case and hence, for the submanifold where E is diagonal. Notice that in this case equation (24) admits an additive form.
dR.M (01, dR.M ((Qu) + +c, QE,"E)") +0
From Pinele et al. 2020), we obtain the analytical expression of the Fisher- Rao in the 2-dimensional submanifold of univariate Gaussian probability distributions M2 = (po = (1,, (2, (0,+)
E R x
PFR (uu), oî) (2)) = V2log
11 /2
(2:01 01
/2
H2
02
+
H1 /2
01
02 V2"
H22
(26)
(/2 0
v2
-
11 oj
V2"
02
where i the Euclidian norm in R2 and a denotes the standard deviation. Consequently, the FRD for Gaussian distributions with diagonal covariance matrix Σ diag (oî oz...., in the 2k- dimensional statistical submanifold MD {pa 0 (H, E) = diag (oî 02.... ... of) , >
0, i 1....,,k k) is
drn- Gauss (0,, 02)
Σ dR (111
(12)) (2))
(27)
A.3 FISHER-RAL RAO vs. MAHALANOBIS DISTANCE
There is intricate relationship between the FRD for multivariate Gaussian distributions and the Mahalanobis distance. We borrow the result from Pinele et al. (2020), which states that in the k- dimensional submanifold My of M where Σ is constant, i.c.. My (pe 0 = (μ. Σ), E Eo e (R)), the Fisher-Rac Rao distance dR.Mg between two distributions is given by the Mahalanobis distance (Mahalanobis 1936): d (W(m,, (,,,, E)) = ((m - 122 T 122).
(28)
The Mahalanobis distance is also used for OOD detection (Lee al. 2018) and its performance is compared to the FRD through several experiments Section Since the covariance matrix for the hidden layers' outputs is often not full rank, the pseudo-inverse calculated instead of the inverse.
B IGEOOD ALGORITHMS AND COMPUTATION DETAILS
In this section, we provide pseudo-code for calculating the IGEOOD score from the logits (Algo- rithm [T) and from the latent features (Algorithm 2 The BLACK-BoO IGEOOD score is obtained with Algorithm[]] by setting = O, while the GREY-BOu IGEOOD score is obtained with E > 0. We calculated the centroid of the logits for the in-distribution training set by optimizing the objective function given by equation (3) through a gradient descent algorithm for each DNN. We used a con- stant learning rate 0.01 and a batch size of 128 for 100 epochs. Finally, the WHITE-BoR IGEOOD score is obtained by combining the outputs of Algorithms I [andd] through fitting the multiplicative weights a through a logistic function classifier on a labeled mixture dataset composed from in- and out-of-distribution data according to validation dataset, which leads to expression equation (33)
Note that the calculation of the training logits centroids My, as well as the latent representations' mean vectors By (t) and standard covariance matrices (()) performed beforehand, prior to infer- ence. In this way, we retrieve the objects from memory at inference time. Also, we define k as the cardinality of feature f, or [f(()] and pFr as the Fisher-Rao distance between univariate Gaussian distribution given by expression equation
16

---

Published as a conference paper at ICLR 2022
Algorithm 1: Evaluating IGEOOD score based on the logits.
Input : Test sample æ. temperature T and noise magnitude parameters, and training set
Dx { )=))
Output: FRg: IGEOOD score in the logits level.
// Offline computation
Calculate the logits centroids from the training data:
My A miR Ny Evi arccos V (oo (v'lu)
// Online computation
Add small perturbation to 2:
δ + x. + E @ sign
V E, arccos (Evey v 10 ( )ool)))oooll (ky)
return FRo ( + Ey 2 arccos (Eyey V (1 (z) )oo (v' (1y)
Algorithm 2: Evaluating feature-wise IGEOOD score.
Input : Test sample x and training set DN = {(xi, v:l Output: FRy: feature-wise IGEOOD scores.
for each feature E (1....,L) L} do
// Offline computation
Calculate the means: Byy (F)
N, Ei 1 (m:)
Calculate the diagonal standard deviation matrix:
N
Eyey Evi:gmy
(5)
(xx) (xx) y4y3)
2
jj
// Online computation
Compute the OOD score for f:
FRe(#) miny V Σ}
1 PFR (11)) (1))))))))))))))))
end
return (xr)... (R)
B.1 LOGITS CENTROIDS ESTIMATION DETAILS
In order to obtain the logits centroids given the Fisher-Rao distance in the space of softmax prob- ability distributions, we designed simple optimization problem. This problem aims to minimize the average distance between the class conditional training samples and the centroids as given by equation (3. We initialized the C' centroids, where C' is the number of classes of a given model, with the identity matrix of size
C. Note that the initial centroid for class is given by the matrix's line number i. We minimized the expression in equation (3) with gradient descent optimizer for 100 epochs with a fixed learning rate equal to 0.1 for every DNN model and in-distribution dataset.
The computation of the logits centroid is done offline, and the loss of the centroid estimation con- verges fast. We show in Table 3 the execution time for some operations in the OOD detection pipeline accelerated by one GPU. The left-hand column shows the offline computations needed to run our setup. They are as follow:
Save train set logits: We first do forward pass through all the training sets and save in memory the resulting logits for given network, which takes on average 83s for CIFAR-10 and CIFAR-100;
Centroid estimation: We load the training logits from memory and run the Gradient Descent algorithm, which takes average 1.2s for CIFAR-10 and 1ss for CIFAR-100:
17

---

Published as a conference paper at ICLR 2022
Algorithm 3: Evaluating feature-wise IGEOOD+ score.
Input Test sample æ, training N {.,,,,))) y:)) and M OOD samples
OM = ={
Output: FRc and FR;: weature-wise IGEOOD+ scores.
for each feature (1...., I.) do
// Offline computation
Calculate class conditional means: Py N2
(E) 1 Eineyy 110 (a..)
Calculate OOD samples mean: ( 1 ΣE p( (x.)
M
Calculate the diagonal standard deviation matrix from training data:
1 EvED Em... (1, (2x)
(0)
N
Pb.
Calculate the diagonal standard deviation matrix from OOD data:
(EY "j
1 M E
(xt)) mg m
M
// Online computation
Compute the OOD scores for f:
(R) < miny V Σラ= PFR
(142. (1)
FR/ f miny Σ PFR (1)" ;033 (6))
(1) (z)) (8) 2
end
return (FR, (2), (x) (R),
The right-hand side of Table 3 shows the average online computation time for one test sample in BLACK- Box setting.
Model inference: The average time needed to complete one forward pass for a DenseNet- BC-100 model is 28 ms and ms for CIFAR-10 and CIFAR-100, respectively.
MSP and BLACK- BLACK-Box IGEOOD computations. Computing the OOD detection scores from the calculated softmax output roughly 100 to 1000 times faster than the inference time taken by the model.
Hence, computing the Fisher-Rao distance between test sample and the class-conditional centroids does not account for considerable overhead in execution time.
Table 3:
Execution time analysis for experimental set accelerated by single GPU for Den-eNet-BC-100 architecture pre-trained on CIFAR-I0 and CIFAR-100. We show the average value for runs.
Offline computation
Online computation
In-dist. Dataset
Save train set logits
Centroid estimation
Model inference
MSP
BLACK-BOX IGHOOD computation
computation
CIFAR-10
83
1.2
ms
63 ps
66 js
CIFAR-100
83
11 s
19 ms
34 ps
171 μs
B.2 COVARIANCE MATRIX ESTIMATION DETAILS
We model the latent output probability distributions as Gaussian distributions with diagonal covari- matrix calculated with equation g We chose this model motivated by closed form for
18

---

Published as a conference paper at ICLR 2022
the FRD and by observing that the standard covariance matrix for the latent features is often ill- conditioned and diagonal dominant. The condition number of a matrix correlates to its numerical stabiliity, ie., small rounding error in its estimation may cause large difference in its values. So, a matrix with a low condition number is said to be well-conditioned, while a matrix with a high condition number is said to be ill-conditioned. We calculate the condition number of the covariance matrices with the formula x(2)) = | || ||> where || lloo is the infinity norm. For each of the four dense blocks outputs of DenseNet trained on CIFAR-IO, we obtained the condition num- bers kg = f2.8c10,3 3.5c6, 3.1c5, 3. 3.5c21}. While for the diagonal covariance matrix, we obtained smaller values of condition numbers: KEn = {1.0c3, 3.0c1, 1.4c1, 7.6c20}. We associate the high value for the last feature mainly because the last feature is high dimensional and coarse, i.e., most of the values in the diagonal are close to zero.
B.3 GAUSSIANITY TEST OF THE HIDDEN LAYERS' OUTPUTS
In order to test if the Gaussian assumption is valid for the outputs of the hidden feature, we conduct [Shapiro & Wilk (1965) normality test for each coordinate of the features for the training data of a DenseNet model. We calculated the test's W statistic for each coordinate and class and averaged them. We chose univariate normality test because they often powerful and the problem is high dimensional, which would be unfavorable for a multivariate statistic test. Thus, this study should be considered with caution, given the considered hypothesis. In Figure we also show the standardized histograms for the first coordinate of each layer. Note that, apart from the penultimate layer, if we consider the coordinates of the hidden features independently, the Gaussianity assumption holds, as we obtain W statistics close to 1. However, for the last block, this assumption sometimes does not hold. Hence, modeling the penultimate layer with more powerful density estimator, and using a metric that considers this more complex distribution, may be favorable for OOD detection.
B.4 FEATURE IMPORTANCE REGRESSION DETAILS
For both Mahalanobis and IGEOOD methods, we fitted a logistic regression model with cross- validation using 1,000 OOD and 1,000 in-distribution data samples. Each regression parameter multiplies the layer scores outputs with the objective function of maximizing the TNR at TPR-95%. We set the maximum number of iterations to 100.
In order to investigate which hidden feature assists the most in OOD detection, we calculate the TNR at TPR-95% for the scores in the outputs of Blocks 1, 2 and of a DenseNet pre-trained on CIFAR-10. We took OOD data the SVHN dataset. Figure 4 shows the histogram and detection performance for each layer as well as the results from the logistic regression. Note that for the İGROOD score in this study, we did not consider the logits.
C DETAILED EXPERIMENTAL SETUP
C.I DNN MODELS AND TRAINING DETAILS
We describe the DNN models used in the experiments:
DenseNet. Densely Connected Convolutional Networks (Huang al. 2017), DenseNet for short, are compositions of dense blocks, which are composed of multiple layers di- rectly connected to every other layer in feed-forward fashion. In this work, we use the DenseNet-BC-IUe 100 architecture. The BC stands for a model with 1xl convolutional bot- tleneck (B) layers and channel number compression (C) of 0.5. The models have depth I. = 100 and growth rate & 12. We consider the outputs of each dense block after the transition layer (3 in total) and the first convolutional layer output as the latent features. After an averaging pooling, the latent features have dimensions F [24, 108, 150,342)
ResNet. Residual Networks (He et al., 2016), ResNet, are deep neural networks com- posed of residual blocks. Each residual block is composed of layers connected in a feed-forware manner plus skip connection. We use the ResNet with 34 layers pre- trained on CIFAR-10, CIFAR-100, and SVHN datasets. We take the output of every residual block (4 in total) and the first convolutional layer for calculating the score
19

---

Published as a conference paper at ICLR 2022
W
stat.: 0.99 (In); 0.97 (Out)
W stat.: 0.99 (In): 0.97 (Out)
W stat.: 1.00 (In); 0.99 (Out)
W stat.: 0.88 (In); 0.87 (Out)
0.5
0.-
0.4
Std. Training dist.
0.4
0.6
Std. Out dist.
0.3
Normal dist.
0.3-
0.2
0.2
0.4
0.2
0.1
0.1
0.1
0.2
0.0
0.0
0.0
0.0
-5
5
-2.5
0.0
2.5
5
5
Conv O, Coordinate
Block 1. Coordinate
Block 2. Coordinate
Block Coordinate
(a) DenseNet pre-trained on CIFAR-10.
stat.: 0.98 (In): (Out)
W stat.: 0.99 (In); 0.98 (Out)
stat.: 0.99 (In): 0.99 (Out)
stat.: 0.86 (In): 0.86 (Out)
0.6
0.4
0.6
Std. Training dist.
Out dist.
Normal dist.
0.3
0.4
0.4
0.2
0.2
0.2
0.2
0.1
0.1
0.0
0.0
-5
0
-2.5
0.0
2.5
5
Conv 0, Coordinnte
Block 1, Coorddinate
Blook 2, Coordinate 1
Block 3, Coordinate
(b) DenseNet pre-trained on CIFAR-100.
W stat.: 0.99 (In); 0.98 (Out)
W star.: 0.98 (In): 0.95 (Out) W stat.: 0.99 (In); 0.93 (Out)
W stat.: 0.93 (In); 0.91 (Out)
Training dist.
0.4
04-
0.4
Std. Out dist.
Normal dist.
0.3
0.3
0.4
0.3
02-
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
-2.5
0.0
2.5
-5
0
-10
-5
0
-5.0
-2.5
0.0
2.5
Conv 0, Coordinate
Block 1, C.Cordinstee
Block 2, Coordinate
Block Coordinate
(c) DenseNet pre-trined on SVHN.
Histograms of the standardized first coordinate output of each hidden feature of DenseNet model for in-distribution and out-of-distribution (TinyImageNet) compared to a I-D Nor- mal distribution. The Average Shapiro- Will test's W statisties is close to one for Conv 0 Block 1 and Block 2, which indicates that the coordinates, and potentially the feature vector, are prov- ably Gaussian. The penultimate layer (outputs of Block 3) has a lower test statistic for the given experiments.
Figure 3:
the WHITE-BOX setting. After an averaging pooling, the latent features have dimensions F2 (64, 64, 128, 256, 512).
We train each model by minimizing the cross-entropy loss using SGD with Nesterov momentum equal to 0.9, weight decay equal to 0.0001, and a multi-step learning rate schedule starting at 0. for 300 epochs. The pre-trained models is available at We report their test set accuracy in Table with the softmax function and by replacing it with the Fisher-Rao distance between the training cass-conditional centroids and the test sample outputs. Also, it is worth noting that one high-end GPU is sufficient for running every experiment presented in this work.
htt. woc.lltoooooooooooooo geood
20

---

Published as a conference paper at ICLR 2022
Mahalanobis: 63%
Mahalamobis: 87%
Mahalanobis: 48%
0.3
In
Proquency
0.2
Froquency
0.2
Out
Frequenccy
0,04
Out
0.1
0.1
0.02
20
20
50
IGEOOD: 88%
IGEOOD: 94%
IGEOOD: %
In
0.2
0.2
Frequency
Out
Frequency
Out
Frequency
Out
0.1
0.0
0.0
0.0
20
10
10
15
20
25
Score
Score
Score
(a) Block 1.
(b) Block 2.
(c) Block 3.
Mahalanobis: 89%: IGEOOD: 94%
Frequency
20
Maha. In
Maha. Out
10
Igeood In
Igeood Out
0.0
0.5 Combined confidence
1.0
(d) Logistic regression result.
Figure 4: Histograms of the Mahalanobis and IGEOOD scores for the output of each hidden block of a DenseNet model for CIFAR-IO (in-dstribution) and SVHN (out-of-distribution)) The title shows the TNR at TPR-95% considering only the scores of the outputs of the given layer. The logistic regression found as coefficients: (1.0, 336, 00.13) for Mahalanobis and a (1.0, 1.3, 1.2) for IGEOOD.
Table 4: Test set accuracy in percentage for ResNet and DenseNet architectures pre-trained CIFAR-10, 10, CIFAR-100 and SVHN.
ResNer-34
DenseNet-BC-1000
In-Dataset
Softmax
Fisher-Rao
Softmax
Fisher-Rao
CIFAR-10 10
93.52
93.53
95.20
95.20
CIFAR-100
77.11
77.09
77.62
77.63
SVHN
96.61
96.61
95.16
95.16
C.2 EVALUATION METRICS
We introduce below standard binary classification performance metrics used to evaluate the OOD discriminators.
True Negative Rate at 95% True Positive Rate (TNR at TPR-95% (%)). This metric measures the true negative rate (TNR) at a specific true positive rate (TPR). The operating point chosen such that the TPR of the in-distribution test set is fixed to some value, 95% in this case. Mathematically, let TP, TN, FP, and FN denote true positive, true negative, false positive and false negative, respectively. We measure TNR = TN/(FP + TN), when TPR = TPP(TP- FN) is 95%.
21

---

Published as a conference paper at ICLR 2022
Area Under the Receiver Operating Characteristic curve (AUROC (%) ). The ROC curve is constructed by plotting the true positive rate (TPR) against the false positive rate (= (PP + TN)) at various threshold values. The area under this curve tells how much the OOD discriminator can distinguish in-distribution and OOD data in a threshold- independent manner.
Area Under the Precision-Rectt curve (AUPR (%)). The PR curve plots the precision (= TP/(TP + FP)) against the recall (= TP/(TP FN)) by varying threshold. For the experiments, in-distribution data are specified as positives while OOD data as negative.
Note that the TNR TPR-95% is significant because want to identify OOD data and preserve sufficiently good performance on identifying in-distribution data, which is not the case for the other metrics.
C.3 DATASETS
We use natural image examples from the following image classification and synthetic datasets in our experiments. We normalize the test samples with the in-distribution dataset statistics.
CIFAR-10. The CIFAR-I0 (Krizhessky et al. 2009) dataset is composed of 32 x 32 natu- ral images of 10 different classes, e.g., airplane, ship. bird, etc. The training set comprises 50,000 images, and the test set is composed of 10,000 images. The classes are approxi- mately equally distributed (5,000 examples each label). The CIFAR-10 dataset under the MIT license.
CIFAR-100. The CIFAR-100 (Krizhevsky et al. 2009) dataset contains similar natural images to the CIFAR-10 dataset, but with 90 additional categories. Its set repartition is 50,000 for training and 10,000 for the test set. We expect around 500 samples for each class of the training set. It is also under the MIT license.
SVHN. The SVHN Netzer et al. 2011) dataset collects street house numbers for digit classification. It contains 73,257 training and 26,032 test RGB images of size 32 x 32 of printed digits (from 0 to 9). We take only the first 10,000 examples of the test set for evaluating the methods to have a balanced dataset of in-distribution and out-of off-iffiiibuttinn data. This dataset subject to non-commercial license.
Tiny-ImageNee
The Ema-eemeeNett (Le & Yang 2015) dataset is subset of the large-
scale natural image dataset ImageNet (Deng et al. 2009). contains 200 different classes and 10,000 test examples. We downsize the images from their original resolution to images of dimension 32 x x 3.
LSUN. The LSUN (Yu et al., 2015) dataset, which has equally 10,000 test examples, is used for the large-scale scene classification of different scene categories (e.g.. bedroom, bridge, kitchen, etc.). Similarly, we resize the images following the same procedure for the Tiny-ImageNet dataset. LSUN under the Apache 2.0 license.
iSUN. The iSUN (Xu et al. 2015) dataset consists selected natural scene images from the SUN (iao al. 2010) dataset. The test set has 8925 images, which downsample to 32 x 32 x 3. We use this dataset as a source of OOD for validation purposes as an independent dataset from the test OOD data.
Textures. The Describable Textures Dataset (DTD) (Cimpoi et al. 2014) is a collection of textural pattern images observed in nature. It contains 47 categories totaling 5640 images of various sizes, which are resized and center cropped to fit into the input size of 32 x 32.
Chars74K. The Chars74K dataset (de Campos al. 2009) contains 74,000 samples of 62 classes of characters found in natural images, handwritten text, and synthesized from computer fonts. We used OOD data only the Englishimg dataset split, which contains 7705 characters from natural scenes. We resized and center-cropped the images.
Places365. The Places365 dataset (Zhou et al. 2017) contains images of 365 natural scenes categories. We used the small images validation split as OOD data in our experiments. contains 36,500 RGB images which were downsampled from 256 x 256 32 x 32.
22

---

Published as a conference paper at ICLR 2022
Gaussian. For the Gaussian dataset, generated 10,000 synthetic RGB images from 2D Gaussian noise, where each RGB pixel is sampled from an i.dd Gaussian distribution with mean 0.5 and variance 1.0. The pixel values clipped to [0. 1] interval. This synthetic data was introduced in previous work as an easy benchmark (Hendrycks & Gimpel 2017).
C.4 ADVERSARIAL DATA GENERATION
We generate adversarial samples from the in-distribution dataset using the fast gradient sign method (FGSM). This method works by exploiting the gradients of the neural network to create non- targeted adversarial attack. For an imput image a, the method computes the sign of the gradients of the loss function J with respect to the input image create new image a;d' that maximizes the loss as given by equation (29). This fabricated image is called an adversarial image, which for tuning the hyperparameters of the OOD detection methods in the WHITE-BOX case. Mathematically,
2 adv x4 + adv @ sign( Vz, ((0,
(29)
where g gadv > 0 the additive noise magnitude parameter. Table 5 shows the resulting Loo mean perturbation and classification accuracy adversarial samples.
Table 5: The Loo mean perturbation used generate adversarial data with FGSM algorithm and classification accuracy on adversarial samples for the DNN models and in-distribution datasets.
CIFAR-10
CIFAR-100
SVHN
L
Acc.
Loc
Acc.
I
Acc.
DenseNet-BC--00
0.21
19.5%
0.20
4.45%
0.32
54.7%
ResNet-34
0.21
23.7%
0.20
12.49%
0.25
50.0%
D BENCHMARK METHODS
This section briefly introduces the benchmark OOD detection methods with standardized notation.
D.1 BASELINE
DNNs tend to assign lower confidence for OOD samples. So, calculating the Maximum Softmax Probability (MSP) (Hendrycks Gimpel, 2017) natural baseline for OOD detection. In other words, provided an input data æ, pre-trained neural network f()), and confidence threshold 6, the OOD score, and the discriminator are given by
(fv(
{
1
s() < if sf >
(xx = max yEY Eveyefyter (w) and S(x; 8)
(30)
respectively. Here, (x) indicates the y-th logits output. A limitation of this method is that un- scaled softmax posterior distributions are usually spiky, i.e., softmax trained deep neural models are incorrectly calibrated, which does not favor OOD detection (Lakshminarayanan et al. 2017).
D.2 ODIN: OOD DETECTOR FOR NEURAL NETWORKS
In summary, ODIN Liang et al. 2018 explores the weaknesses of the MSP criterion by recalibrat- ing the output's confidence the task of OOD detection. They improve the MSP baseline by using the temperature scaled softmax function (equation (1) instead. Also, ODIN adds small adversarial noise perturbation to the inputs, i.e.,
O sign (-V log (x)
(31)
where E is the perturbation magnitude. Hyperparameters T and tuned validation dataset without requiring prior knowledge of test OOD data. They calculate the confidence score by taking the maximum of the perturbed input temperature scaled softmax outputs.
23

---

Published as a conference paper at ICLR 2022
D.3 EASRGY-BASED OOD DETECTOR
An energy-based based OOD discriminator is proposed by Liu et al. (2020), where the differences of ener- gies between in-distribution and OOD samples allow for distribution distinction. The energy-based model substitutes the softmax function with the Helmholtz free energy equation to extract a confi- dence score. They observed that examples with higher energy have a low likelihood of occurrence, concluding that they are likely OOD. The free energy expression is:
E(x: f) log E clulx)/T
(32)
Note that, differently from ODIN and MSP, they use the information of all of the logits output values through the sum operation. Besides, they apply input pre-processing for further separating OOD data from in-distribution.
D.4 MAHAALANOBIS DISTANCE-BASER CONFIDENCE SCORE
The Mahalanobis-base method in Lee et al. (2018) fits the DNN training data features as class- conditional Gaussian distributions. These the outputs of every DNN latent block to leverage useful information for discrimination. For a test sample æ, the confidence score from the f-th feature calculated based on the Mahalanobis distance between (x)(x) and the closest class-conditional distribution:
Mc(x) = max - (f(0(()) p!)) ES (F(0(z)
-
(33)
(e) where ((0 is the f-th latent feature output, and Pig and Ex are, respectively, the empirical class mean and covariance matrix estimates. The covariance matrix often not full rank, so the pseudo- inverse is calculated instead of the inverse. In addition, input pre-processing and feature ensemble are also used to boost performance. A logistic regression model learns the multiplicative weights ag for each layer score, which predicts 1 for in-distribution and 0 for OOD examples from mixture validation dataset. Finally, the Mahalanobis- based discriminator given by thresholding expression E, (x)).
E ADDITIONAL OUT-OF-DISTRIBUTTI DETECTION RESULTS
E. FISHER-RAv DISTANCE VERSUS KULLBACK- DIVERGENCE
From Picot et al. 2021), the Dallback-Leibler divergence (KL) is connected to the Fisher-Rao dis- tance between softmax probability distributions (dR,D) by the inequality:
ddDD (g0. %o)
1 KL (40,6))
1 1 cos
(34)
2
2
To verify how the KL divergence would behave for OOD detection, we ran experiments with our BLACK-BOL Box setting, where we calculated the class conditional centroids with the KL divergence. We calculated the divergence of the test sample w.r.t each of these centroids during test time, then aggregated the results with sum by taking the minimal value. The results are displayed in Table
6
We can conclude from these experiments that taking the sum of the outputs instead of the minimal value is overall advantageous for Fisher-Rao distance and KL divergence.
E.2 HYPERPARAMETERS TUNING
For temperature T, we ran a Bayesian optimization for 500 epochs in the interval of temperature values between I and 1000, where the objective function was to maximize the TNR at TPR-95% metrie for the validation set. We took the best temperature among five runs with different random seeds. For the input prepprcesssing noise magnitude tuning, ran grid search optimization with 21 equally spaced values in the interval [0, 0.002]. Table showw the best hyperparameters we found for the methods in the BLACK-BOOL Box, GREY-BOX. and WHITE-BOS settings.
24

---

Published as a conference paper at ICLR 2022
Table Performance comparison between the Fisher-Rao distance and the KL Divergence for OOD detection in BLACK- Box setting. The numerical values in the Table are TNR at TPR-95% in percentage for a DenseNet and ResNet models pre-trained on CIFAR-I0, CIFAR-100 and SVHN datasets. FISHER-RAY (sum) corresponds to the IGEOOD score.
OOD
CIFAR-10
CIFAR-100
SVHN
dataset
FiSHER-RAC (sum)
FISHER -AR (min) KL (min) KL (sum)
Chars
55.1/45.0/56.9%/
17.2/14.6/20. 1/1.77///
47.9/46.6/ 1/1.666/50.1/4....
Gaussian
99.9/97.9/97.9/99
0.0/0.0/0.0/0.0
98.0/97.2/73 2/1.1/////..//11111
TinyImgNet
87.8/73.6/72.3/88
1/tp//85.//1
85,1/84. 1tt.1/119999944 4/85.0
DenseNet
LSUN
93.3/81 4//86.4/93.4
85.0/83. 2.65..
Places365
20.7/20.9/18.2/20.8
71.9/71. 1/7113
Textures
35.8/46.//51.0344
22.8/19 1/1...5/17.6//3.11
56.5/57. 115.74465..3/ 33/55.4
CIFAR-10
17.0020.0/18.5/11..77
67.0/66 $7.0/66.0/56..211 0/56.2/65.6..8
CIFAR-100
50.8/48.7/49 7/5..6
65.4/65.0/59..1664
SVHN
00.1/47.//50.7449
36.7/29.
average
65.6/61.4/653//65.1
72.1/71 4/6.3/1/6
Chars
$1.1/45.0/4/ 0//.1///199...//
15.2/14.6/14.55155
58.5/57 4/46.2/58.4
Gaussian
99.0/86.4/62.3/86.8
0.6/1.7/3.9/0.9
$7.3/87.5/76.7/88
TinyImageNet
58.2/5 (552/51.4/51.444 1/55
82.2/8 118.1//66888111118
ResNet
LSUN
62.0/53.
20.6/15.4/10.2/19.5
77.4/77.5/64.677
Places365
48.2/40 0/39 0/399//488..8.1
16.9/17.3316.7/11..88
79.0/79.1/67.2777 8
Textures
50.3/44.0/45 6/49.8
23.4/20.9/14.1/22.2
80.9/88..9/7..888000
CIFAR-10
18.0/18..116.8/1888
81.2/81 0/6..////1......
CIFAR-100
45.9/38 9/36.8/43.....
8/6.799//66.///999
SVHN
48.8/31. 16/31.5/47.... 55/47.0
13.3/14.3/15.7/14.3
average
06.7/48.9/455660
16.4/15.0/1..811.44
7N.3/78.1/66.377
Table 7: Best temperatures T for the BLACK-BI -BO setup, best temperature and noise magnitude (T', =) for the GREY-BOX setup, and best for the Mahalanobis score and (T, E) for IGEOOD and IGBOOD+ in the WHITE-Bol Box setup with adversarial tuning.
In-dist.
BLACK-Bus Energy
GREY-BOX Energy
WHITE-Bor
Model
dataset
ODIN
IGEOOD
ODIN
IGBOOD
Maha.
IGEOOD,+
C-10
1000
4.6
5.3
(1000. 0.0014)
(4.6, 0.0012)
(5.3. 0.0012
0
(5. 0.0013
DenseNet
C-100
1000
2.1
(1000, 0.0020)
(1.1. 0.0020
(2.1.0.0020)
0
(5.
SVHN
1.1
1.1
(1. 0.0010)
(1.1. 0.0006)
0.0006
0.001
(5. 0.0015)
C-10
1000
5.4
5.3
(1000, 0.0014)
(5.4, 0.0012)
(3.3. 0.0012)
0.0005
(2.0)
ResNet
C-100
1000
1
(1000. 0.0020)
(9.1, 0.0024)
(12.7, 0.0024)
0.0005
(1. 0)
SVHN
1000
1.7
1
(1000, 0.0004)
(1.7,000002
(1.0, 0.0004
0
(5.
E.3 TEMPERATURE SCALING AND NOISE MAGNITUDE PLOOTS
In Figure aaddd we plot on the left hand side column the effect of the temperature parameter in the setup. We set the noise magnitude zero and measured the TNR
performance for the BL.ACK- at TPR-95% for 500 different temperatures values found by Bayesian optimization for variety of DNN models. The performance is evaluated on the iSUN dataset. The right hand side column of Figure 5 and show the effect of the noise magnitude parameter in the performance of IGEOOD score in the GREY-Box Box setup. We set the temperature to the best found in the BLACK-Bol case. Then, measured the OOD performance for 21 values of noise magnitude equally spaced in the interval [0, 0.004.. The best couple ET) for each method and model is used to evaluate the GREY-Box performances. The best hyperparameters found are detailed in Table 7
25

---

Published as a conference paper at ICLR 2022
TNR at TPR-955 (%
IGEOOD logits
Odin
Energy
10
10
10
0.000
0.001
0.002
0.003
0.004
Temperature:
Noise magnitude,
(a) DenseNet on CIFAR-10.
TNR at TPR-95% (%6
40
IGEOOD logits
Odin
Energy
20
10'
10'
10a
10'
0.000
0.001
0.002
0.003
0.004
Temperature,
Noise magnitdde,
(b) DenseNet on CIFAR-100.
TNR at TPR-95% (56)
IGEOOD logits
Odin
Energy
10
0.000
0.001
0.002
0.003
0.004
Temperature, T
Noise magnitude, e
(c) DenseNet SVHN.
Figure 5: OOD detection performance against temperature and noise magnitude parameters for ODIN Liang al. 2018). Energy (Liu et al. 2020) and IGEOOD (ours) the iSUN (Xu et al., 2015
OOD dataset for DenseNet- 100 architecture.
E.4 CONSISTENCY OF IGEOOD SCORE CONCERNING THE CHOICE OF THE VALIDATION DATA
To verify the consistency of IGEOOD and other methods to the choice of validation data, we mea- sured the TNR TPR-95% after tuning our method BLACK-BOx and GREY-BOX scenario on nine validation datasets. In Table 8, the first column shows the validation dataset, while we used the remaining OOD datasets to evaluate performance. We obtained consistent results, ranging from 63.4% to 72.0% the average TNR at TPR-95% in the BLACC- Box case and from 65.0% to 73.4% in the GREY-Ro Box setting. We show that input pre-processing provides mild amelioration for our method and can be considered a fine-tuning step.
E.5 ERROR BARS AND STANDARD DEVIATION
We conduct all of our experiments during inference time. Provided that we fix the DNN, the in- distribution, and the nut-o--distribution datasets, there is not a source of randomness to our algorithm because the weights of the feature ensemble method and centroids are initialized deterministically Thus, the OOD scores for the same experimental setting do not change. To confirm this, we ran the same experiment five times and obtained the same results in all of them. However, if we allow for retraining the DNN from scratch, we might obtain different parameters, leading to slightly different model accuracy and potentially OOD detection performance. With this in mind, retrained DenseNet-B BCC10000 model on CIFAR-10 five times with five different random seeds. The results for OOD detection in MACK-BBS setting for the 5 models can be found in Table g
26

---

Published as a conference paper at ICLR 2022
TNR at TPR-95% ($e
70
IGEOOD logits
Odin
Energy
50
100
10
10
10
0.000
0.001
0.002
0.003
0.004
Temperature:
Noise magnitude,
(a) ResNet on CIFAR-10.
TNR at TPR-05% (%)
IGEOOD logits
Odin
30
Energy
10
0.000
0.001
0.002
0.003
0.004
Temperature, T
Noise magnitude, E
(b) ResNet CIFAR-
TNR at TPR-95% (56)
80
IGEOOD logits
Energy
10"
10
10a
10
0.000
0.001
0.002
0.003
0.004
Temperature, T
Noise magnitude, e
(c) ResNet SVHN.
Figure 6: Temperature and noise magnitude tuning for OOD detection performance for ODIN (Liang al. 2018). Energy (Liu al. 2020) and IGROOD (ours) on iSUN (Xu et al. 2015) OOD dataset for Net-34 architecture.
Table 8: BLACK-Box and GREY-Bot Box settings average performance across different OOD datasets for validation. The hyperparameters are tuned using one validation dataset (column 1), and evalua- tion is done on the remaining eight OOD test datasets. The DNN is DenseNet-BC-100 pre-trained on CIFAR-10, 10, and the values are TNR at TPR-95% in percentage.
BLACK-BoL
GREY-BOX
Validation set
Baseline
ODIN
Energy
IGEOOD
ODIN
Energy
IGEOOD
iSUN Chars
52,5
64.3
64.9
65.6
66.8
64.8
65.3
55.0
70.8
71.1
71.4
72.5
72.0
73.4
CIFAR-100
55.4
68.6
69.1
72.0
68.6
71.7
71.3
Gaussian
49.4
62.8
65.6
63.4
70.4
64.0
68.0
TinyImgNer
53.0
64.7
65.2
63.5
67.0
65.0
65.5
LSUN
52.1
63.9
63.7
63.6
66.6
65.3
65.0
P1aces365
55.3
68.5
69.0
71.8
70.0
71.5
70.9
SVHN
55.4
68.7
69.3
69.5
70.0
69.4
70.1
Textures
55.4
71.2
73.1
71.4
71.5
72.4
71.6
average and std.
53.7+22
67.143.0
67.943.
68.0
69.31220
68.4114
69.043.0
27

---

Published as a conference paper at ICLR 2022
Table 9: Experiment using five different training seeds for DenseNet-100 on CIFAR-10 for the
BLACK-BOX scenario. The average test accuracy of the 5 models is 94.58% 0.13%. All values are percentages.
TNR at TPR-95%
AUROC
Dataset
Baseline / ODIN Energy IGEOOD
Chars
CIFAR-100
3711:ss44..8...44...22555..
8822:..8888:118888011888888.
Gaussian
42.2::07744.222296::2:282222
95.22..96..222246661:::::
TinyImgNet
92.5::.9957:11190:0::::/...
LSUN
Places365
9,502211552222::5222225555331113
38.420.499.9:1122944111
SVHN
Textures
average and std.
Table 10: Average and standard deviation OOD detection performance across eight OOD datasets for each model and in n--iitribution dataset in a GREY-BOX setting.
TNR TPR-95%
AUROC
Model
In-dist.
ODIN Energy / IGEOOD
C-10
66.8:223/4484222/55.... 24
91.9=6./91 15..:///99.556//11111
DenseNet
C-100
76.6:12776.4117788...A. 28822
SVHN
75.4::1777.6411772..44
#L.6::../89.226669999...
C-10
57.34mm/57.7119557..888
$9.2255./88.7455:89..06
ResNet
C-100
31.1=://30.22:22332222 76.9=117744441174.3311
SVHN
78.55:78/78.5:::/888..111 86.1+6.7855.277705577777
90.44..499099:::440777.33
Average and Std.
55.8#22/54.4222544.927
E.6 IGEOOD COMPARED OTHER WHITE-BOR Box METHODS.
Even though Lee et al. (0118 shares the closest setup to ours, recent literature also shows promising results for OOD detection WHITE-BI Box setting, achieving state-of-the-ee in a few benchmarks. Notably, the works from Sastry & Oore 2020 Hsu et al. (2020); Zisselman & Tamar (2020) achieve remarkable performance in a range of benchmarks. Thus, we gathered the reported results from the original works and displayed them in Table II and [2] which considers that few OOD samples and only adversarial samples are available for tuning, respectively. We highlight that Sastry & Oore (2020) extracts, in addition to the outputs of the blocks, intra-block features for the ResNet and DenseNet models.
E.7 EXTENDED OOD DETECTION RESULTS
We show in Table 13 extended OOD detection results of Table 1 It contains the OOD detection performance for each model, in-distribution dataset and OOD dataset in a BLACK-Bos -Box setting. In Table 14 we show the performance of ODIN, energy-based, and IGEOOD scores in the task of OOD detection in GREY-BOX setup for each OOD dataset. In Table 15 and [6] we show additional results referring to the right-hand column and left-hand column of Table respectively.
HISTOGRAMS
Figures g and 9 display histograms for the OOD detection score for IGEOOD in the BLOCK- Box, GREY- BOX and WHITE-Bor Box settings, respectively.
28

---

Published as a conference paper at ICLR 2022
Table 11: TNR at TPR-95% (%) performance in a WIITEE-BOX setting considering the original
results from Lee et al. 2018 and Zisselman & Tamar (220)) with access to OOD samples. The
models are DenseNet- BC- 100 and ResNet-34 pre-trained on CIFAR-I0, CIFAR- 100 and SVHN.
OOD
CIFAR-10
CIFAR-I00
SVHN
dataset
Mahalanobis Res-Flow IGEOOD
IGEOOD+
iSUN
95.3/
197.7/99.8
87.0/ /93.8/99.7
99.9/ /98.3/99.9
DenseNer
LSUN
91.4//6.3/95.2/999
99.9/100/97.1/99
TinyImgNet
95.0/96.4/95.70
86.6/93.0/94.5/99
99.9/100/98. 2/99.
SVHN/C-10
90.8/94. 9/98, 9/99.9
22.5/84.9/93.33
96.8/99 .099
11199/9/91.6//8.. 16/98.3
average
94.6/96.5/97.7788
86.9/91 4/9999.4/942/999
99.1799.6796.335
iSUN
97.8/ /97.2/99.9
89.9/
/93.4/99.8
99.7/
/99.8/100
ResNer
LSUN
08.8/99.0/9 4/100
90.9/96.2/94.3/100
99.9/100/99 7/999.
TinyImgNet
97.1/97.8/96 8/96.39999999
90.9/94.6/90 1/99.6
99.9/100/99.7/999
SVHN/C-10
87.8/96.5/98.88999
1L.9/93.0/91.6/99 66/9..7
98.4/99 4/97.7/99.7
average
05.4/97./97.7/99..
90.9/94.6/92.35/99..
Table 12: TNR at TPR-959
performance in a WHITE-BOX setting considering the original Sastry & Oore (2020); Hsu et al. (020); Zisselman & Tamar (2020)
%
results from Lee al. 2013 8
without access to OOD samples for hyperparameter tuning.
OOD
CIFAR-10 10
CIFAR-100
SVHN
dataset
Mahalanobis / Gram Matrix / DeConf-C Res-Flow IGEOOD IGBOOD+
iSUN
94.3/99.0/99
/94.5/95.8
84.8/95.9/98..
/93.8/92.2
99.9/99.4/
198.2/98.6
DenseNet
LSUN
97.2/99.5/99 .499/.4/9811.. 1/96.4/97.2
91.4/97 2/999/ 8/95.1/94
100/99.5 S/
/100/97.3/97.0
TinyImgNet
94.9/98.8/99 1996.1199.11... 1/93.4/94.5
77.2//5.7/98.64
5/94.33/440
99,9/99. 1/
/99.9/98. 1/96.8
SVHN/C-10
89.9/96.
1t.99/6.1198.88/8 1/98.8/86...... 1/94.3/95.7
62.2/89.3/95.0
9/90. 1/90.6
90.0/30.4/
/90,0/89.5MM 5/86.6
average
11/99..//9.2/9934494
91.4999..99...97789999.9999
97.4/94.6/
0/6.6/95.8/44.3
iSUN
96.8/99.3/88.88 &/ /95.3/95.0
87. 97.99/9..775.3/
/89,4/91.0
100/99 4/
199.8/99.9
ResNet
LSUN
98.1/99.6/90
9999.6990.9//9 9/99999999..... 1/97.7/97.7
56. 8/7.76/677../88.66 6/93.9
99.9/99.66 6/
/100/99.8/
TinylmgNet
95.5/98.7/8/ 4/99 0/94.3/94.2
70.3/9448776.5777... 2/90.
99.2/99.3/
199.9/99.6/99,
SVHN/C-10
75.8/97. 6/89.59999 1 01/9.2/99.7
/1.9/80.8/55.1 1/4. 2/78.
94. 9/85/8
196.6/96. 7/97.3
average
TL.5/98.8/87./946 6/9.//9999999966....
98.3/96.07
198.8/99.0/99.2
29

---

Published as a conference paper at ICLR 2022
Table 13: Extended BLACK-BoL results for Table
Parameter tuning iSUN dataset.
In-dist,
OOD dataset
TNR at TPR-95%
AUROC
AUPR
(model)
Baseline ODIN Energy IGEOOD
Chars
43.5/57.2/54.6/55. 6/55.0
90.2/91.2/90.. 5
93.0/93.1/22.5/92.
CIFAR-100
40.6/53.1/50.555 5/50.7
19.4//0.4/89.7//
90.5/90.7/90. 1/9
Gaussian
88.1/99.88 11199/8/99.9/999 9/999.9
97 97.6/98.9/98.559
98.3/99.3/99 1/99.1
TinyImgNet
59.4/85.0/88 008.55008880/77..7
94. Nt../9..3/97.6/9/9
95.4/97.7/97.9/99 9/999
CIFAR-10
LSUN
/6/33/9.4/9933//33..
95.5/98.3/98.5999
96.5/98. 0t.5//8.5/98.77 7/9.8
(DenseNet)
Places365
40.8/54.2/51 5/522.0
88 2/99. .589977
74.4/74.8/73 9/74.3
SVHN
40,4/52.0/49.6/55
9t.99999990./9900
84.6/84.6/833 7/333/
Textures
40.5/42.1134.9115
0..5/85.1/82.411
93.1/88.3/86 3/86.//88888
average
52.3766.8765.3/66 3/63.6
91.7792.//22119991
90.7/90.9/90/3/990
Chars
15.1/17.//7.0017.
72.8/78.0/77.00 9/7778
79.6/83.8/83 8//3.9/83.8
CIFAR-10
17.7/18.1/17.
21118.1/17.1/17...
75.6774.877444755.4
78.3/74.3/74.0/77
Gaussian
0.0/0.000.0/0.0
30.2/19.4/19.555 4/19//30//
53. 53.2/44. 23.2/4.00/44 1/53.5
TinyImgNet
16.7/24.7/25.0//5..
72.0/79.477.66779 4/79.6/79.5
74.8/80.7/88 7/80.
CIFAR-100
LSUN
70.9/80 6/80.8/80.6
74.4/82. 6/82.9/82.5
(DenseNet)
Places365
2..8/21./220./222
75.9/78.0/77
3/77. 7/78.0
54.2/54.55 24.22/4../544 5/55.
SVHN
25.7/36.4/36.5/33
F...//8.4/88.4/88
2.../82.//8.4//222
Textures
18.0/22.4/22 2/222..//2../22..8
72.7774.4/44.3/755
80.8/79 1/79 0/8115
average
15.9/20.5/20.3/22 3/20.7
69.1/71.6 6//7/
71.3/12.7772.7774...
Chars
46,4/27.0/45.000
83,9/52 2/7.6/6677//...
918770.2788 9789.6
CIFAR-10
61,8/66. 0/666.0664,77. 0/6.,006677777777 7/670
92.3/90.9/90..00 9/90.3/90.9
96.2/95. 1/94.6/95.0
CIFAR-100
61.3/64.466..06654
91.9/90.3/88 3/999
0/1/9443/93.//94
Gaussian
93.6/97 8/97.9/98.8
97 47.9/ 009.98.09..00 0/9800
99 4/999//99.....
SVHN
TinylmgNet
80.4/84.4/84.1 1/85.1
35.5/95.3//44999
97.9/97 A/97 2/97/5
(DenseNet)
LSUN
10.1/84.4/84.33
95.5/95.3/95 19..1... 1/95.3
98.0/97 6/97 999757
Places365
66,8/71 11111.0/699990 9/71.99
93.0/91 1991.99921 1.999.9
89.1/85
8/84.6/8 6/4.6688888
Textures
36.4/55.2/52.445
2./84.6/82.558
95.5/93.3/99
average
95.3/68.8770,211
3/900/2999
95.4/91
5/941
Chars
36,8/45 8/545,855.,1110 7/51.1
89 4/.4/90.1/990 1/90.3/90.4
92.7/19.71/22.5579 1/9../92..
CITAR-100
33.6/41. 11111.8/56666/ 6/45,9
86.4/87.088 00877/077..0.0 1
87.0/86 6/86. 2/86.4
Gaussian
81.5/89.9/88.07 2/89.0
TinylmgNet
42.1//3.4/57.7/58.2
00.3/91.5/91 16/91.7
91.8/92./992. 1/92.2
CIFAR-10
LSUN
41.2/55. 1/61.7/62.0
90.1/91.5/9P 5/922.0/92.
91.5/92. 1/92.3/92.3
(ResNet)
Places365
2.9/42.444882//4
85 6/8.666686./86699
67.1/66. 666.6/65.7
SVHN
27.7/39. 9/48. 6/48.8
89 2/900.2990.//90.6
85.8/85. 35.8/85.1/88. 1/84.5
Textures
37.9/46. (1446.6/4966//11 6/500.
89 11.00/9..//88 4/88.6
33.9/99.3/992.49
average
41.7/51.9756 3/56.
89 47900
88.5788 3/7788.3/87.0 3/87.9/88.
Chars
14.3/15.3/15.1155.2
72.7773.0773.1//73
77.8/77 2/77./27773/77..
CIFAR-10 10
18.1/18.4/17.4111
(6.5/76.6/76.5778
78.3/77 8/77.8/78.
Gaussian
16600.8/0./0066
12.8776.0776.776..7
81.6/83.
81.6/83.8884.4/84 8/84.4/84.3
TinyImgNet
7.8/20.7/23.8/23
73.3776.6777447666
76. 7/7.877/9.2/78.8
CIFAR-100
LSUN
15.5/18.8/21.577
72.9975..2/5..775.2 70.4974.1/449944444
(ResNet)
Places365
7.3/17.2/16.2/165
74.1/73.2/72.9977..
44.6/42. 1/41,9//2/ .9/42.7
SVHN
14.2/13.8/12.5/113
74.8774.1/73.//744
59.4/56. 99.4/56.9/56.8557 9/56.8/57.7
Textures
20.9/22.8/23.3/22..
77.0/78.0/T
2/78.3
85. ./.9/86.. 2/86.3/86.4
average
15.0/16.0/16.3/66.
74.0775.2775 2/15 4/75.5
72. 12.1277..//2.4/72
Chars
56.9/58.2755 4/58.5
17.188.7783..1 7/84.0
92.3/91 1O/1..99/0000
17991.999913 10/1.33
CIFAR-10
79.0/80.6/81 1/8.0//81../811.3
93.0/92. 1/92.2/92.5
94.7/93 4/93.4/93.
CIFAR-100
78.0/79, 6/79 8/80.2
92.7/91.998 1999222
94.7/93 4/93.4/93.7
Gaussian
85,3/86. 7/86. 9/87.3
2..9/95.7/95.7999
97.7197. 1/97. 1/97.3
SVHN
TinyImgNet
79.8/81 4/81. 8/82.2
93.5/92. 9/93.2
95.3/94.3/94 3/94,
(ResNet)
LSUN
74.9/76.8/77 8/77.68/7777777...
9L.5/90.5990.6990
93.5/92.
1/92.4
Places365
77,0/78 4/78. 8/79.0
92.0/91 11991.999..1111 1/991.4
81.4/77 (LA/77..9/7 9/77 8/78.6
Textures
78.5/80.0/80.41 4/80.9
23.7/93.0/93.0/99
97.6/97 .0/9 7//972
average
76.2/71.3778.07788.
92.291 3/91 4791.7
93,4/92 0792.
Average of the average values
15.0051../52.1//52.
85.3/85.4885.1/85.
85.2/84.7/84.447
30

---

Published as a conference paper at ICLR 2022
Table 14: Extended GREY-BOX results. Parameter tuning on iSUN dataset.
In-dist.
OOD
TNR at TPR-955
AUROC
AUPR
(model)
dataset
ODIN
Energy IGEOOD
Chars
52.5/49 2/50.0
89.1/88.24 2/8887
90. 90.7/90.2/90.8
CIFAR-100
48.9/49. 7/50.0
88.3/88 5/88.7
88.4/88 6/88,9
Gaussian
100/100/100
100/99 999999 9/99.9
100/99.9/99.9 9/99,9
TinyImgNet
29./992.4/92.4
98.5/98 5/98.5
98.6/98 5
CIFAR-10
LSUN
06.2/96.2/96.2
99.2/99. 1/99.2
99.3/99 22999 2
(DenseNet)
Places365
52.4/52.0/52.4
89.0/88 9/89.2
SVHN
00.//118/42.
89.7/88 2/88.6
82.1/80.6/8
Textures
02.9/37.0/38.
$1.4/80.8/824.
average
66.8/64 3/66.5
91.9/91 .5991.9
91.6/91 01.6/11.3/9.. .3/91.8
Chars
19.8/19.2/19.5
76.6/76.3/75.8
80.8/80.6/79.7
CIFAR-10
15.3/16.4/16.9
72.4/72.8/74.9
72.6/72.8/76.
Gaussian
0,0/0.0/0.0
47.1/46.9/59.9
65.5/65 4/74.9
TinyImgNet
23.8/42.2/40.1
86.5/86.2/84.8
87.2/87 0//5.33
CIFAR-100
LSUN
42.2/40.6/38.99
86.8/86.4/84.7
8/88/777788.
(DenseNet)
Places365
25.4/23.8/23..
79.1/79.1/79.4
79.4/79
19,4//9.4/80. 4/80.0
SVHN
34.7/31.4/35.7
87.7/87.2/88.1
Textures
24.9/24.7/24.9
76.7/76.6/77.8
81.7/81 66/8.88
average
25.5/24.8/25.0
76.6/76.4/78.2
79.5/79.
Chars
53.3/45.8/48
8188/780079.3
90.4/88. 1/88.8
CIFAR-10
0088/5.1/67.11
91.3/89. 1L.9999.189..9
05.2/93.7/94.2
CIFAR-100
69.8/64.2/66.33
91.1/88.5/89.4
94.8/93.0/99 0/933.6
Gaussian
99.4/99.0/99 1/99.
98.9/98.5/98..
99.7/99.5/99
SVHN
TinyImgNet
88.4/85. 5/86.4
06.1/94.9/95.2
97 9/.9/9..0 3/97.3
(DenseNet)
LSUN
88.2/85. 8/86.5
96.2/95.1/95.4
98.0//7.3/97..
Places365
74.8/69.9/71.8
92.4/90.3/91.0
95.8/94.2/94.7
Textures
59.4/49.6/53.7
84.9/79 .0/11.4
93.4/90.4/91
average
75.4/70.6/72.4
91.6/89 2/90.0
95.7/94.2/94.7
Chars
54.7/54.2/54.8
88.3/87 8788,2
90.7/90.4790.6
CIFAR-100
38.9/41.7/41.5
83.8/83.8/88 8/84.1
83.7/83.5/1 8/8838
Gaussian
100/100/100
100/99.5/99.5
100/99.708 7/99.7
TinyImgNet
68.7/66.6/6 6/67.0
93.1/92.4/92.6
93.3/92.7/92.9
CIFAR-10
LSUN
70.4/68. 8/68.9
93.2/92.8/92.19
93.3/92.9/93.00
(ResNet)
Places365
40.4/43.2/43.0
84.1/84. 1/84.2
84.0/83 9/84.0
SVHN
38.6/39. 9/40.0
84.8/84.0/844
76.4/74.6/75.7
Textures
46.8/47.1/47.5
86.1/85.2/85.1
91.4/90.7/91.0
average
57.3/57. 1/57.8
89.2788, 7/89.0
89.1/88.
Chars
14.8/13.8/13.
68.0/65 1/65.0
71,0/88
5/68.4
CIFAR-10
70.8/65. 1/64.9
70.1/64.7/6455
Gaussian
78.7/73.8/74.3
96.8/95.9/96.0
97.7/97.4/9 4/97.4
TinyImgNet
44.9/49.2/49.3
86.4/86.5/86.5
86.1/85 6/85.6
CIFAR-100
LSUN
40.2/44.3/44.4
83.5/83.8/83.8
82.5/82.5/82.5
(ResNet)
Places365
15.9/10.2/10.1
67.7/59.7/59.5
64.5/58. 2/58.0
SVHN
10.2/10.6/10.6
64.3/65.4/65.3
42.2/43.6/43.
Textures
19.5/27.6/273
77.9/73.7/736
84.4/81. 1/811.
average
31.1/30.2/30.2
76.9774.4/74.3
74.8772.1/7266
Chars
99.9/59.4/60.J
82.8/83.2/83.2
90.4//0.7/90.6
CIFAR-10
80.9/81.0/81.1
90.9/91.5/91.2
92.2/92.8/92.5
CIFAR-100
80.2/80. 2/80.5
90.8/91.4/91.1
92.3/92.9/92.6
Gaussian
89,4/88. 7/89.4
95.9/95.9/9599
97.3/97.2/97.2
SVHN (ResNet)
TinyImgNet
82.6/82.3/82.9
92.2/92.6/92.5
93.5/93.9/93.8
LSUN
77.8/77.6777 6/78.3
89.6/90.1/89.9
91.3/91.7/9116
Places365
78.7/78.8/79
89.8/90.6/90.
91.2/919//1..5
Textures
78.6/79.8/79.
91.3/9222/91.7
96. 1/.1//666/96.3
average
78.5/78.5778.8
90.4/90.9/90.7
93.0/93.5/93.3
Average of average values
55.8/54.4/54.9
86.1/85.2/85.7
87.3/86.6787.0
31

---

Published as a conference paper at ICLR 2022
Table 15: WHITE-BOX extended results. Validation OOD data.
In-dist.
OOD dataset
TNR at TPR-95%
AUROC
AUPR
(model)
Mahalanobis
Ice 2018
IGFOOD+
Chars
91.3/99.4
97.5/99.9
97.7/99.9
CIFAR-100
21.4/56.6
67.3/90.7
64.4/90.8
TinylmgNet
96.9/99.8
99.3/99.9
99.3/99.9
LSUN
98.2/99.9
99.5/100
99.5/100
Places365
18.1/80.2
72.7/95.7
72.8/95.4
CIFAR- 10 (DenseNet)
SVHN
90.1/99.9
97.3/100
97.3/100
Textures
84.1/97.4
95.6/99.3
94.7/99.5
Gaussian
100/100
100/100
100/100
97.3/99.8
99.4/100
99.4/100
average
77.5=s192..=114
92.1 41/984440.
91.7+11988.4430
Chars
62.9/97.5
94.0799.4
95.8799.4
CIFAR-10
9.1/22.7
60.8/80.7
60.1/83.0
TinyImgNet
87.1/99.5
97.4/99.9
97.4/99.9
LSUN
91.1/99.9
97.8/100
98,1/100
Places365
5.9/58.2
54.8/90.0
54.7/89.2
CIFAR-100
SVHN
79.0/99.6
96.8/99.9
94.1/99.9
(DenseNet)
Textures
70.3/90.2
91.4/98.1
94.3/98.2
Gaussian
100/100
100/100
100/100
iSUN
86.4/99.7
96.8/99.9
97.7/99.9
average
67.7-2x/90.2.. 2.2
17.81 /1/17///7...00
88 18.0012//97.8 91.2/7..8.500
Chars
78.7/92.2
96.1/99.44
98.9/98.5
CIFAR-10
91.6/98.3
98.0//9.6
99.4/99.6
CIFAR-100
92.9/95.3
98.2/99.1
99.4/99.2 .2
TinyImgNet
99.9/99.9
99.8/99.9
99.9/99.9
LSUN
99.9/99.9
99.8/100
99.7/100
SVHN
Places365
94.7/98.3
98.3/99.6
98.4/99.7
(DenseNet)
Textures
98.2/98.5
99.4/99.6
99.9/99.6 6
Gaussian
100/100
100/100
100/100
iSUN
99.9/99.9
99.8/99.9
99.9/99.9
average
1=8.099.002..
98.8110/99.6100
99.3E11/99.6200
Chars
93.6/99.3
98.6/99.8
99.1799.8
CIFAR- 100
44.9/51.3
87.4/90.9
87.8/11.7
TinyImgNet
96.8/99.6
99.4/99.9
99.4/99.9
LSUN
98,3/99,9
99.6/100
99.6/100
Places365
45.8/77.6
88.1/95.6
88.1/95.5
CIFAR-10
SVHN
1699/9
99,0/99.9
98.1/99.9
(ResNet)
Textures
84.3/97.0
97.3/99.4
98.6/99.4
Gaussian
100/100
100/100
100/100
iSUN
97.2/99.9
99.4/100
99.5/100
average
94.1=2591.6611
96,
96.5
96.754u/98.5
Chars
63.8/97.8
94.0/99.5
96.0/99.5
CIFAR-10
18.0/30.8
76.6/85.3
76.4/87.8
TinyImgNet
90.1/99.6
97.9/99.9
98.0/99.9
I.SUN
92,4/100
98.3/100
98.5/100
Places365
23.5/59.1
76,8/91.2
76.0/91.4
CIFAR-100
SVHN
88,4/99.7
97.7/99.9
95.2/99.9 9
(ResNet)
Textures
71.6/90.7
93.9/98.2
96.6/98.1
Gaussian
100/100
100/100
100/100
iSUN
89.4/99.8
97.7/99.9
98.0/99.9
average
70.8-w/86.4220
90.5
92.72m
Chars
84.9792.4
97.0798.4
99,0798.5
CIFAR-10
98.0/99.7
99.2/99.9
99.7/99.9
CIFAR-100
98.3/99.1
99.3/99.7
99,8/99.8
TinylmgNet
99.9/99.9
99,9/100
100/100
I.SUN
99.9/99.9
99.9/100
100/100
SVHN
Places365
98.4/99.6
99.3/99.9
99.8/99.9
(ResNet)
Textures
99.0/99.9
99.7/99.9
99.9/99.9
Gaussian
100/100
100/100
100/100
iSUN
100/100
99,9/100
100/100
average
97.6.s6..98.9.224
99 99.44.10/9.77100
99.8.1.//998...
Avg. and std. avg. values
82.1=1/92.9440
94.554098.54110
94.754m98.6=110
32

---

Published as a conference paper at ICLR 2022
Table 16: WHITE-BOX extended results, Validation on adversarial (FGSM) data.
In-dist.
OOD dataset
TNR TPR-95%
AUROC
AUPR
(model)
Mahalanobis
Lee et 2018
IGEOOD
Chars
88,5/87.3
91.7/97.7
98.3/98.3
CIFAR-100 100
21.5/26.4
68.0/77.7
66.3/75.5
TinyImageNet
93.9/93.4
98.6/98.7
98.6/98.7
LSUN
96.3/96.4
99. 99.1/99.2
19.1/99.2
CIFAR-I0
Places365
17.8/23.2
70.0/77.9
1/7.3
(DenseNet)
SVHN
87.0/94.3
97.2/98.7
93.7/97.3
Textures
83.6/86.0
95.8/97.2
97.3/98.2
Gaussian
100/100
100/100
100/100
iSUN
94.3/94.5
98.8/98.9
98.9/99.0
average
75.92w/77.119
91.7212944.0=99
88
08.02///93662110
CIFAR-10
1.1/5.7
43.5/62.6
46.7/62.6
Chars
53.9/59.6
92.2/92.0
94.5/93.9
TinyImageNet
86,4/94.3
97.4/98.8
97.5/98.9
LSUN
88.6/95.1
97.6/98.9
97.9/98.9
CIFAR-100
Places365
5,5/13.0
56,6/71.0
57.5/71.0
(DenseNet)
SVHN
56.1/90.1
91.8/98.0
85,4/96.2
Textures
67.5/86.7
91.2/97.4
94.4/98.4
Gaussian
100/100
100/100
100/100
iSUN
84.8/93.8
97.2/98.7
97.6/98.8
average
60.4::4/70.9423
85.3411/90,88
85.74:w/91.00s1
CIFAR-
90.6/89.5
97.7/97.8
99. 1799.2
CIFAR-I00 100
91.8/88.4 4
98.0/97.7
99.2/99.1
Chars
72.3/70.5
95.2/94.5
98.5/98.3
TinylmageNet
99,5/98.1
99.6/99.3
99.5/99.8
SVHN
LSUN
99.9/97.3
99.8/99.1
99.9/99.7
(DenseNet)
Places365
94.3/91
98.3/98.2
98, 1/99.3
Textures
95.3/97.1
98.8/99.3
99.6/99.8
Gaussian
100/100
100/99.9
100/100
iSUN
99.9/98.2
99.8/99.3
99.9/99.8
average
93.7=xo/92.34990
19.3:1./99
CIFAR- 100
36.5/21.3
84.5/63.3
84,3/58.1
Chars
82.0/90.9
96.9/98.3
97.7/98.7
TinylmageNet
96,2/94.3
99.2/98.0
99.2/96.7
LSUN
98,2/97.7
99,5/99.2
99.5/98.9
CIFAR-10
Places365
34.8/15.9
85.0/60.1
84.2/24.4
(ResNet)
SVHN
81.0/98.2
96.6/99.3
93.7/97.5
Textures
81.7/81.6
96.7/93.4
98.2/94.3
Gaussian
100/100
100/100
100/100
iSUN
96.8/95.3
99.3/98.6
99.3/98.1
average
78.6=24/77.34.2
95.3+...990.E11
95.1scw/85.222
CIFAR- 10
3.0/5.0
61.0/59.6
63.7/60.6
Chars
39 19.9/55.1
85 35.6/90.4
88.1/92. 1/92.
TinyImageNet
88.7/86.2
97.6/97.3
97.6/97.3
LSUN
91.3/88.6
98.0/97.8
98.3/98.0
CIFAR-100
Places365
8.0/8.6
67.9/63.0
66,8/61.7
(ResNet)
SVHN
31.6/75.2
82.9/95.8
8/92.7
Textures
65.9/78.1
91.9/95.6
95.2/97.6
Gaussian
100/100
100/100
100/100
iSUN
87.9/89.4
97.4/97.8
97.6/97.7
average
57.44w66.114m
86,9 65
86.2+1u/88.7 75
CIFAR-
97.1/96.7
99.1/99.2
99.7/99.7
CIFAR- 100
97.5/96.2
99.1/99.1
99.7/99.6 6
Chars
75.4/55.1
95.3/89.1
98.5/96.0
TinyImageNet
99.9/99.6
99.9/99.9
99,9/99.9
SVHN (ResNet)
LSUN
100/99.5
99.9/99.9
100/100
Places365
98.1/97.0 O
99.2/99.2
99.2/99.0
Textures
98.9/98.4 4
99.6/99.6
99.9/99.9
Gaussian
100/100
99.9/100
100/100
iSUN
100/99.8
99.8/99.9
99.9/100
average
96.3=xw/93.60
99.141n/98
99.6+os/99.3.0m 3+10
Avg. and std. of avg. values
77.0=15/79544m
92.8+../93.4539
9/.3+s
33

---

Published as a conference paper at ICLR 2022
Baseline:
ODIN: 85%
Energy: 88%
IGEODD: 88%
2000
40
0.61
0.4
Out
0.4
20
1000
0.2
0.0
0.0
0.75
0.100
0.101
0.102
0.103
30
10
20
(a) DenseNet CIFAR-I0. 10.
Buseline: 17%
ODIN: 25%
Energy: 25%
IGBOOD: 26%
10000
0.02
In
0.10-
5000
0.85
0.25
0.50
0.75
1.00
0,0002
0,0104
250
500
(b) NetseNet on CIFAR- 100.
Baseline: 80S
ODIN: 84%
Energy: 84%
IGEOOD: 85%
3000
0.64
Out
2000
0.4
1000
0.2
0.25
0.50
0.75
1.00
0
0,1000
0. 0,1005
0,1n10 0,1015
0.0
10
1.5
24
26
25
(c) DenseNet SVHN.
Bascline: 42%
ODIN: 53%
Energy: 57%
IGEOOD: 58%
In
4000
1.5-
1,0-
0,5
0.5
0.0
0.0
(d) ResNet on CIFAR-10.
Baselinc:
ODIN: 21%
Energy: 24%
IGEOOD: 23%
20
15000
Oul
10000
S000
0.0
0.0
0.5
1.0
nunTORS NU10IC AUMA15
10
15
200
300
310
(e) ResNet CIFAR-100.
Baseline: 80%
ODIN: 81%
Fnergy: 82%
IGBOOD: 82%
10000
1.0-
7.5
In
5000
0,5-
5.0
2.5
0,0
0.11
(1) ResNet on SVHN.
Figure 7: BLACK-BOX setup. TinyImageNet as OOD dataset.
34

---

Published as a conference paper at ICLR 2022
ODIN:
Energy: 92%
IGEOOD: 92%
1500
0.4
In
1000
0.4
0.2
0
0.0
0111
0.102
(a) DenseNet on CIFAR-10.
ODIN: 44%
Energy: 42%
IGEOOD: 40%
10000
0.10
0.075
0.050
0.55
0.025
0.01025
0,00000
70
401
730
3001
(b) DenseNet on CIFAR-100.
ODIN: 88%
Energy: 85%
IGEOOD: 86%
In
40-
0.4
Oui
30
0100
0.25
0.50
0.75
1.00
10
15
25
26
27
(c) DenseNet on SVHN.
Energy: 67%
IGEOOD:
1.0
1.01
2000
0.5
0.5
0.0
0.0
0.1005
17
(d) ResNet on CIFAR-10.
ODIN: 45%
Energy: 49%
IGEOOD: 49%
20000
10
0.3
0.2
10000
0.1
0/1111
DA/12
47.00
42,25
42.50
20
30
(e) ResNet on CIFAR-
ODIN: 83%
Energy: 82%
IGBOOD: 83%
4000
100
Dut
2000
0.5
2.5
0-
0.0
0.101
0.102
10
26
(f) ResNet SVHN.
Figure 8: GREY-BOX setup. TinylmageNet as OOD dataset.
35

---

Published as a conference paper at ICLR 2022
Mahalanobis: 94%
IGEOOD:
IGEOOD+: 92%
0.06
0.075
0.1
0.050
0.025
0.02
0.1
0
0.00
(a) DenseNet on CIFAR-10.
Mahulanobis: 86%
IGEOOD: 94%
IGEOOD+: 94%
0.2
0.02
0.05
0.11
775
50
255
-11
701
0
1001
(b) DenseNet on CIFAR-IO0 -00.
Mahalanobis: 100%
IGEOOD: 98%
IGEOOD 99%
0.2
In
0.15
Out
0,150
0.1
0.10
0.05
01000
0.0
0100
-400
-200
-10
(c) DenseNet SVHN.
Maha
IGEOOD:
0.06
0.04
0.050
0.02
0.000
0.00
0.00
-100
(d) ResNet on CIFAR-10.
Mahalanobis: 85%
IGEOOD: 86%
IGEOOD+: 95%
0.075
0.06
0.04
0.050
002
0.255
0.02
2001
40
-10
-25
25
(e) ResNet on CIFAR-
Mahalanobis: 100%
IGEOOD: 100%
IGEOOD+: 100%
0.0015
0.10
Out
010010
0.05
01005
01000
0.00
0.00
20000
10000
0
60
(f) ResNet SVHN.
Figure 9: WHX setup with adversarial data validation. TinylmageNet as OOD dataset.
36

---

Published as a conference paper at ICLR 2022
Mahalanobis: 96.1%
IGEOOD: 97.5%
IGBOOD+: 99.9%
0.15
0.10
0.1
0.05
0.11
0.00
0.00
-100
-50
10
10
(a) DenseNet on CIFAR-10.
Mahalanobis: 86.8%
IGEOOD: 94.7%
IGEOOD+: 99.9%
0.10
0.10
0.2
0.1
0.05
100
-0
70
20
401
(b) DenseNet on CIFAR- 100.
Mahalanobis: 99.9%
IGEOOD: 98.0%
IGEOOD+: 99.8%
0.10
0.15
In
Out
0.10
0.05
015
0.0
0/0
0100
-200
-10
10
20
-10
(c) DenseNet SVHN.
Mahalanobis:
IGEOOD: 96.1%
99.8%
0.3
0.4
0.2
0.2
0.1
0.1
0.0
0.0
(d) ResNet on CIFAR-10.
Mahalanobis: 89.5%
IGEOOD: 87.1%
IGEOOD+: 99.8%
0.2
0.2
0.1
20
20
20
(e) ResNet on CIFAR-
Mahalanobis: 99 9%
IGEOOD: 99.6%
IGEOOD+: 100.0%
0.004
0.15
0.10
Out
0.10
0.002
0100
0.00
0.00
5000
20
(f) ResNet SVHN.
Figure 10: WHITE-BOX setup with validation on OOD data. TinyImageNet as OOD dataset.
37


Reviews:
[{"review_id": "pucHqFHRFcW", "reviewer": "Reviewer_R2Ss", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The theoretical basis of the proposed methodology is well described and the experiments appear to be well-designed overall. Empirically the proposed method outperforms other methods, especially in the white-box setting where OOD validation samples are available. \n\nAccording to Table S8 (Table 8 in the Appendix), the performance difference of the different methods appears to be marginal. Can you provide statistical significance for the performance differences? In addition, according to Table S9 (grey-box), ODIN performs better than the proposed method in most cases. Can you add some discussion on why? Moreover, the current description of the results given in the main text is a bit misleading because these observations are not properly explained.  \n\nIn the white-box setups, IGEOOD seems to perform best when OOD samples are available for validation (Table S10), but it does not when only the adversarially generated samples are used for validation(Table 2, Table S11). This result does not match the purpose of this study presented in Abstract and in Introduction (e.g., \"IGEOOD applies to any pre-trained neural network, does not require OOD samples or assumptions on the OOD data\").\n\nRecently, many SSL (self-supervised learning)-based OOD detection methods have been developed. Examples are SSD[1] and CSI[2] that use the same confidence score as in Lee et al.[3] while utilizing self-supervision (simclr). I wonder 1) if the proposed method can be compared to these self-supervision-based models, and 2) whether the proposed geodesic distance can be applied to the SSL-based approaches.\n\n[1] Sehwag, Vikash, Mung Chiang, and Prateek Mittal. \"SSD: A Unified Framework for Self-Supervised Outlier Detection.\" International Conference on Learning Representations. 2020.\n[2] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in Neural Information Processing Systems, 33, 2020.\n[3] Lee, Kimin, et al. \"A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\" Advances in neural information processing systems 31 (2018).\n\nThe proposed model is based on the assumption that the layer output follows a Gaussian distribution. It is possible to provide validation or discussion of this assumption? \n\nRegarding Eqs. (5) and (6), the authors note that taking the sum (5) instead of the minimum distance to the class conditional centroid produces better results. This looks interesting. Can you provide empirical results or more analysis on this?\n\nIt would be interesting to see the score distributions as shown in Figure 1(c) for the real datasets used in this study. ", "summary_of_the_paper": "This paper presents IGEOOD, a new method for detecting OOD samples by using geodesic (Fisher-Rao) distance in confidence scoring. It further combines confidence scores from the logit outputs and the layer-wise features of a deep neural network. The method is validated under various testing environments such as the availability of OOD data or the accessibility of latent features of a deep network. The idea of using Fisher-Rao distance for OOD detection seems novel and interesting. ", "summary_of_the_review": "This paper presents a novel idea of using Fisher-Rao distance for confidence scoring in OOD detection. The main idea is interesting, the paper is well structured in terms of methodological description and the problem/experimental setups, but the empirical results do not appear to be sufficient to validate the intended purpose of this study. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cJ6tf1YeLL2", "reviewer": "Reviewer_KjND", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Methodology\n\nThe authors propose two related methods: (1) measuring distance in the output space, and (2) measuring distance in the space of the hidden layer activations. (1) is quite intuitive, and as far as I know novel: to the best of my knowledge prior work typically considers just the confidence of the classifier, and not the full predictive distribution. (2) is quite similar to Mahalanobis [1], with the main difference being the distance metric used. However, the authors show significant improvements in performance compared to [1] in Section 4.3, justifying the proposed method.\n\n## Results\n\nThe empirical results constitute the main strength of this paper: the method outperforms the considered baselines across the board. However, I have two concerns:\n\n- The improvements in the Black-box setting appear very minor. In many cases, the method does not outperform the baselines, our improves the results by $O$(0.1%) (see e.g. the AUROC results for ResNet, in Table 1). It would potentially be helpful to add error bars to Table 1.\n- The paper claims to set the new *state-of-the-art* on visual out-of-distribution detection, but the considered baselines are somewhat limited. OOD detection is a very active field, with many papers claiming to improve on the considered baselines [see e.g. 2-6]. From a quick comparison it seems like the results reported by the authors are competitive with the best methods I could find, but I think the authors need to do a better job comparing to prior work, in order to claim SOTA.\n\nOn the other hand, I want to highlight that the authors perform a fairly exhaustive experimental evaluation in terms of the out-of-distribution datasets considered for each in-distribution dataset, including both near- and far-OOD.\n\n## Writing\n\nThe writing is generally clear, but there are several typos and minor inaccuracies.\n\n## Comparison to other distance metrics?\n\nAs far as I understand, it is possible to replace the Fisher-Rao metric with any other distance metric in the method proposed by the authors? I think it would be interesting to see a comparison of the results with a few other standard metrics in the Black-Box set-up, e.g. KL-divergence, total-variation distance, etc. If the Fisher-Rao metric provides significantly better results, this experiment would strengthen the paper.\n\n\n## References\n\n[1] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks\nKimin Lee, Kibok Lee, Honglak Lee, Jinwoo Shin\n\n[2] Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data\nYen-Chang Hsu, Yilin Shen, Hongxia Jin, Zsolt Kira\n\n[3] Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices\nChandramouli Shama Sastry, Sageev Oore\n\n[4] Hybrid Models for Open Set Recognition\nHongjie Zhang, Ang Li, Jie Guo, Yanwen Guo\n\n[5] Deep Residual Flow for Out of Distribution Detection\nEv Zisselman, Aviv Tamar\n\n[6] A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection\nJie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, Balaji Lakshminarayanan", "summary_of_the_paper": "The paper proposes a new group of methods for supervised OOD detection. In particular, the authors propose to use the Fisher-Rao distance between output distributions on the in-distribution data and test samples to detect OOD. The authors additionally propose to use Fisher-Rao distance in the hidden layer feature space, when possible (white-box setting). The method achieves strong empirical performance, improving upon standard baselines (such as Odin, Mahalanobis), especially in the white-box setting.", "summary_of_the_review": "In summary, I recommend a weak accept for this paper. The empirical results are good, and the method is generally novel. I am open to increasing my score, if the authors address the concerns I raised in my review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NRK9zby3971", "reviewer": "Reviewer_eVhF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths & Weaknesses \n\n**Clarity**: The paper is mostly clear and well written. The writing can be improved in some places (e.g. paragraph 2 in the Introduction)   \n\n**Novelty**: While the Fisher-Rao metric has been applied in the context of deep learning (natural gradient, regularization of training, etc. -- need references here), I've unaware of its use for anomaly/ out-of-distribution (OOD) detection. It's a reasonable extension to evaluate the utilization of Mahalanobis distance in the prior art (Hendycks and Gimpel, ICLR'17). The novelty meets the bar for a publication without rising to the level of being significant.  \n\n**Technically correctness**: The material is technical correct in the large. I went through the approach including the math in the paper and skimmed through the Appendix. I'm mostly familiar with the topic and the material seems correct though I didn’t check the appendix comprehensively. \n\n*(TC 1) Gaussianity*: There are two main parts for the proposed score – one based on the SoftMax output – this seems ok. My main concern is with the other part -- using the multivariate Gaussian model with a diagonal covariance for the feature spaces without first validating the premise with a Gaussianity test. While testing for high-dimensional multivariate Gaussians may pose difficulties, the diagonality assumption should make this feasible. I recommend that evaluation and analysis of this assumption is added to the paper. \n\n*(TC 2) PMFs and PDFs*: while the SoftMax formulation provides a posterior distribution in the label space, having this interpretation for the feature layers for a single input sample without further grounding in theory/ past literature makes the approach ad hoc. I'd like to see authors clarify this. \n\n*(TC3) Max and Average*: The authors comment that (for the SoftMax output) they obtain slightly better results using (6) -- average() rather than (5) -- min() over the classes.  \n\n(a) Would using the average be tantamount to computing the probability with respect to a mixture distribution and admit a model more applicable for the scenario (single sample case) than a PMF/ PDF estimate interpretation?  \n\n(b) Was this also tried with the features corresponding to equations (12) and (13)?  \n\n**Experimental Evaluation**: A good number of experiments have been conducted and presented in the main paper for the black-box (Table 1) and the white-box (Table 2) scenarios. These are further supported by additional experiments in the Appendix E (Tables 5-11) as well as ablation studies for various (hyper)parameters important to the proposed approach. This is very good.  \n\nOn the flip side, some parts of the experimental validation can be improved to support better the central claims – that OOD scores based on the Fisher-Rao information metrics outperform the previous state of the art. \n\n*(EE1) Difficulty of directly comparing with published results*: Since the experimental settings seem different from other papers (DNN models used, finetuning dataset, etc.), it is really hard to directly compare tables in the paper to those published in the compared SOTA. I tried to do this both for the Tables in the main paper as well as in the Appendix. Since this beats the due diligence review process, the authors should explain why this is ok. \n\n(a) Black-Box settings: Entries in Table 1 (and Table 8) can't be compared with Table 2 in Hendryks and Gimpel (ICLR 2017) referred to as Baseline, Table 1 or Table 2 in Liu et al (NeurIPS 2020) or with ODIN - Liang et al (ICLR 2018). \n\n(b) White-Box settings:  I compared some numbers across Tables 10 and 11 with Table 2 in Lee et al (NIPS 2018) -- discrepancies in results exist, sometimes very significant,  but mostly (not all) the comparative accuracy improvements hold. \n\nI gave up on a more comprehensive cross-checking of the results but since there are differences in the setups.  \n\n*(EE2) SOTA used for comparison*: Apart from Liu et al (NeurIPS 2020), all the compared approaches seem 3-4 years old. The authors should respond on whether those results are SOTA. \n\n*(EE3) Inconclusive evidence for improved performance*: Performance improvements vis-a-vis the reported SOTA is mixed – ref. Tables 8 – 11 though I consider the results to be promising. The authors should discuss this in their response. \n\n**Reproducibility**: I expect the results to be reproducible since enough details are shared in the paper and the code has been made publicly available at https://github.com/igeood/Igeood.   \n\n ", "summary_of_the_paper": "The paper proposes to use a score based on the Fisher-Rao information metric (the Riemannian metric in the space of probability distributions) for the detection of out-of-distribution samples input to a trained DNN. While the output SoftMax probabilities are used in the black-box and grey-box scenarios, the learnt features in the intervening DNN layers are additionally used in the white-box scenario. The approach models each sample as providing posterior probabilities – (a) the SoftMax probability in the label space, and (b) class-conditional PDFs over the corresponding feature spaces for each DNN layer. These latter are modeled as multivariate Gaussian distributions with diagonal covariance matrices. Extensive experiments on existing benchmarks are conducted for comparative results against the state of the art demonstrating promising results. ", "summary_of_the_review": "I recommend to accept the paper.  \n\nIt is a good addition to the set of methods on an important topic - OOD. The approach is novel, reasonably principled and the code has been made publicly available at [https://github.com/igeood/Igeood](https://github.com/igeood/Igeood) which is good for the community to able to put the above methods to test.  The validation is comprehensive though the results are somewhat inconclusive though promising. There are some concerns regarding the validation of assumptions and the experimental evaluation. My preliminary assessment is that the submission passes the criteria for acceptance to this venue. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L0ZUgFXMtmZ", "reviewer": "Reviewer_dbMS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper\n1) The concept of applying the Fisher-Rao distance to out-of-distribution seems novel and important to me. Furthermore, examples on the Gaussian strengthen the motivation of the FIsher-Rao distance's theoretical benefit.\n2) Proposed Fisher-Rao distance can be combined with pre-existing techniques, input-preprocessing, temperature scaling, and feature ensembling, to boost the OOD performance.\n3) The proposed method shows competitive results on the black-box OOD detection setting.\n\nWeakness of the paper\n1) One major issue of the proposed algorithm is that we have to tune the parameter of centroid parameter of each class. While the authors have noted that they optimized for 100 epochs, I wonder how the extra computation time scales compared to the baselines. (e.g., Mahalanobis distance, energy-based distance), especially in the CIFAR-100 dataset, where we have to evaluate the mean of the centroid 100 times.\n2) Furthermore, given the computation time of 1), experiment results are weak to champion IGEOOD as the OOD method suitable for pre-trained classifiers. In the grey-box setting, ODIN outperforms IGEOOD. Furthermore, in the white-box setting, the method is only compared against the Mahalanobis distance. \n\nComments\n\n\n1. As mentioned in the weakness section, the wall clock time of the IGEOOD on the various datasets can help to solve the time efficiency issue of the method.\n\n\n2. Furthermore, in the white-box setup, I suggest comparing the method not only against the Mahalanobis distance but also on the recently proposed methods (e.g., [1])\n\n\n3. In figure 1(c), while it's convincing that the Fisher-Rao distance improves detection against type-2 OOD data, the two histograms are not normalized enough to make comparisons. I suggest matching the area of in-distribution data frequencies between two histograms. \n\n\n4. .\" Empirically, we show in the appendix (see Section C) that this confidence score does not degrade and\nsometimes improves the in-distribution test classification accuracy\".<<<< I found the results and the claim are rather redundant. If there is no stark improvement, why should we take a look into it?\n\n\n5. For Figure 2, I suggest adding the distributions of the baselines for direct comparisons.\n\n\n6. How the choice of validation dataset impacts performance <<< I am skeptical about the absolute, or relative robustness of IGEGOOD against the other algorithms. First, 8% variation in TNR does not seem so robust since the paper's gain against baselines is not bigger than 8%. Furthermore, I cannot find a major difference against baselines given the results in E.3.\n\n==================================================================================================\nReferences\n[1] Detecting Out-of-Distribution Examples with Gram Matrices. C. S. Satry and S. OOre, \n", "summary_of_the_paper": "UPDATE:\n\nI acknowledge that I've read the author responses as well as the other reviews. \n\nWhile the authors added further analysis on the proposed method, I am skeptical of the performance of the method since the proposed method requires validation OOD data to achieve SOTA performance and the runtime is much larger than MSP and energy baseline. However, I think the rebuttal clarified much of my concerns and therefore raise the score to 5 weak reject.\n \n=================================================================\n\nThe paper proposes an out-of-distribution detection (OOD) metric based on Fisher-Rao distance which can be applied for pre-trained classifiers. First, the authors derive a Fisher-Rao distance applied to the distribution of softmax. Also, they motivate a toy example where the FIsher-Rao distance outperforms the conventional OOD metric, Mahalanobis distance. Furthermore, they formulate the Fisher-Rao distance-based framework, IGEOOD on Black(Grey)-Box, where we can only get access on the logit of the network output, and White-Box, where we can get access to intermediate feature layers. Finally, the authors compare IGEOOD against conventional OOD metrics on various out-of-distribution data and in-distribution data.", "summary_of_the_review": "The paper proposed a new OOD detection framework, IGEGOOD. My biggest concern is that the experiment results are fairly weak given the complex procedure of obtaining the detection scores. Therefore, I am leaning towards the rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel approach for out of distribution detection that generates scores from a trained DNN model by using the Fisher-Rao distance between the feature distributions of a given input sample at the logit layer and the lower layers of the model and the  corresponding mean feature distributions over the training data.

The use of Fisher-Rao distance is novel in the context of OOD, and the empirical evaluations are extensive.  The main concerns of the reviewers were the limitations of the Gaussianity assumption used in computing the Fisher-Rao distance and the use of the sum of the Fisher-Rao distances to the class-conditional distributions of the target classes rather than the minimum distance. These concerns were addressed satisfactorily in a revision. In terms of technical novelty, experimental evaluation and novelty, the paper is above the bar of acceptance.
Decision: Accept (Poster)