Zero-Shot Out-of-Distribution Detection with Feature Correlations
Authors: Chandramouli S Sastry
Sageev Oore
Venue:  (2020)
Keywords: out-of-distribution
gram matrices
classification
out-of-distribution detection

Abstract: When presented with Out-of-Distribution (OOD) examples, deep neural networks yield confident, incorrect predictions. Detecting OOD examples is challenging, and the potential risks are high. In this paper, we propose to detect OOD examples by identifying inconsistencies between activity patterns and class predicted. We find that characterizing activity patterns by feature correlations and identifying anomalies in pairwise feature correlation values can yield high OOD detection rates. We identify anomalies in the pairwise feature correlations by simply comparing each pairwise correlation value with its respective range observed over the training data. Unlike many approaches, this can be used with any pre-trained softmax classifier and does not require access to OOD data for fine-tuning hyperparameters, nor does it require OOD access for inferring parameters. The method is applicable across a variety of architectures and vision datasets and generally performs better than or equal to state-of-the-art OOD detection methods, including those that do assume access to OOD examples.

Under review as a conference paper at ICLR 2020
ZERO-SHOT OUT-OF-DISTRIBUTION DETECTION WITH FEATURE CORRELATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
When presented with yield confident, incorrect predictions. Detecting OOD examples is challenging, and the potential risks are high. In this paper, we propose to detect OOD examples by identifying inconsistencies between activity patterns and class predicted. We find that characterizing activity patterns by feature correlations and identifying anomalies in pairwise feature correlation values can yield high OOD detection rates for drrfoomdisttibttion examples. We identify anomalies the pairwise feature correlations by simply comparing each pairwise correlation value with its respective range observed over the training data. Unlike many approaches, this can be used with any pre-trained softmax classifier and does not require access to OOD data for fine-tuning hyperparameters. nor does it require OOD access for inferring parameters. The method is applicable across a variety of architectures and vision datasets and, for the important and surprisingly hard task of detecting far-from-distributio outliers, it generally performs better than or equal to state-of- the-art OOD detection methods (including those that do assume access to OOD examples)
(OOD) examples, deep neural networks
INTRODUCTION
Even when deep neural networks (DNNs) achieve impressive accuracy on challenging tasks, they do not always visibly falter on misclassified examples: in those cases they often make predictions that are both very confident and completely incorrect. Yet, predictive uncertainty is essential in real-world contexts tolerating minimal error margins such as autonomous vehicle control and medical, financial and legal fields.
In this work, we focus flagging test examples that do not contain any of the classes modeled in the train distribution. Such examples are often referred to as being out-of-distrrbuttin (OOD), and while their existence has been well-known for some time, the challenges of identifying them and baseline method to do so in a variety of tasks such as image classification, text classification, and speech recognition were presented by Hendryyks and Gimpel (2017). Recently, Nalisnick et al. (2019a) identified a similar problem with generative models: they demonstrate that flow-based models, VAES, and PixeICNNs cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR- 10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former. They report similar findings across several other pairs of popular image datasets.
While we might expect neural networks to respond differently to OOD examples than in-distribution (ID) examples, exactly where and how to find these differences in activity patterns is not at all clear. Hendrycks and Gimpel (2017) and others (Nguyen et al. 2015 Yu et al. 201I) showed that looking at the maximal softmax value is insufficient. In Section 2 we describe some other recent approaches to this problem. In this work, we find that characterizing activity patterns by feature correlations computed with extension of Gram matrices that we introduce- lets quantify anomalies allow state-of-the-art (SOTA) detection rates on OOD examples.
Intuition.
We identify out-of-distribution examples by jointly considering the class assigned at the output layer and the activity patterns in the intermediate layers. For example, if image is
The code for this work is available http::/tttthpp.//aa..oo.eocootea

---

Under review as a conference paper at ICLR 2020
predicted to be dog, yet the intermediate activity patterns are somehow atypical of those by the network for other dog images during training, then that is a strong indicator of an OOD example. This effectively allows us detect incongruence between the prediction made by the network and the path by which it arrived at that prediction.
Strengths. Unlike those previous works that assume access to OOD examples and train an auxiliary classifier for identifying anomalous activity patterns, our method finds differences in activity patterns without requiring access to any OOD examples, and it works across architectures. We hope this will also help further our understanding of how neural networks respond differently to OOD examples in general, not just how particular network responds to examples coming from particular distribution.
Contributions.
This work includes the following contributions:
1. We extend Gram matrices compute effective feature correlations.
2. Using the order Gram matrices, we present new technique for computing class- conditional anomalies in activity patterns.
3. We evaluate this technique on OOD detection, testing on
competitive architectures: DenseNet, ResNet;
benchmark OOD datasets including: CIFAR-10, CIFAR-100, SVHN, TinyImageNet, LSUN and iSUN.
4. Zero-shot:
crucially, our method does not require access to OOD samples for tuning hyperparameters or for training auxiliary models.
5. Nevertheless, we report results which, for the challenging and important cases of far-from- distribution examples, are generally better than or equal to the tate-of-the-art method for OOD detection that does require access to OOD examples.
2 RELATED WORK
Previous work which aims to improve OOD detection can be roughly grouped by several themes:
Bayesian Neural Networks. A nice early Bayesian approach (Gal and Ghahramani 2016) esti- mates predictive uncertainty by using an ensemble of sub-networks instantiated by applying dropout at test time. As opposed to implicitly learning distribution over the predictions by learning distribution over the weights, Chen et al. 2019) and Malinin and Gales (2018) explicitly parameterize Dirichlet distribution over the output class distributions using DNNs in order to obtain better estimate of predictive uncertainty; the main differences between these methods is that Chen et al. (2019) use ELBO, which only requires the in-distribution dataset for training whereas Malinin and Gales (2018 use a contrastive loss which requires access to (optionally synthetic) OOD examples.
Using any pre-trained softmax deep neural network with OOD examples. Lee et al. (0018) to the best of our knowledge, the current SOTA technique by a significant margin compute the Mahalanobis distance between the test sample's feature representations and the class-conditional gaussian distribution at each layer; they then represent each sample as a vector of the Mahalanobis distances, and finally train logistic regression detector on these representations to identify OOD examples. Another technique in this category is ODIN (Liang et al. 2018): they use mix of temperature scaling at the softmax layer and input perturbations to achieve better results. In fact, both Lee et al. (2018b) and Liang et al. (2018) add small input perturbations to achieve better results; the former do so increase the confidence score, while the latter do so increase the softmax score. Recently,Quintanilha
Recently, Quintanilha et al. (2019) achieve results comparable to that of et al. (2018)) 8b) by training a logistic regression detector that looks at the means and standard deviations of various channels activations. Unlike the previous two techniques, Quintanilha et al. (2019) achieves comparable results even without the use of input perturbations, which allows it to be applicable to non-continuous domains. Our work, too, does not involve input perturbations.
All of these techniques depend on OOD examples for fine- tutiitnnng hyperparameters (Liang et al. 2018) or for training auxiliary OOD classifiers (Lee et al. (2018b); Quintanilha et al. (019)). Furthermore, these classifiers neither transfer between one non non-training distribution and another. nor do they transfer between networks, separate classifiers must be trained for each (In-Distribution, OOD,
2

---

Under review as a conference paper at ICLR 2020
Architecture) triplet. In many real-world applications, this is infeasible: cannot assumee advance access to all possible OOD distributions. Our work does not require access to OOD samples.
Alternative Training Strategies. Lee et al. (2018a) jointly train a classifier, a generator and an adversarial discriminator such that the classifier produces more uniform distribution on the boundary examples generated by the generator; they use OOD examples to fine-tune hyperparameters. Veiisss and Taylor (2018) train neural networks with multi-task loss for jointly learning classify and estimate confidence. (hhaeey et al. (2018) use multiple semantic dense representations as the target instead of sparse one-hot vectors and use cosine-similarity based measure for detecting OODs. Building on the idea proposed by Lee et al. (2018a), Hendrrykk et al. (2019a) propose an Outlier Exposure (OE) technique. They regularize softmax classifier to predict uniform distribution on (any) OOD distribution and show the resulting model can identify examples from unseen OOD distributions; this differs significantly from previous works which used the same OOD distributions for both training and testing. Unlike other methods, they retain the architecture of the classifier and introduce just one additional hyperparameter- their model is quite robust to the choice of OOD examples chosen for the regularization. However, while the OE method is able to generalize across different non-training distributions, it understandably does not achieve the rates of Lee et al. (2018)) on most cases. Recently, [Hendrycks et al. (2019b) make sigmificant advances in detecting hear-distribution outliers without having any know weoge of the exact of-distribution examples by using in-distribution examples in a self-supervised training setting.
regularization rafe-and also demonstrate that
Generative Models. Ren et al. (2019) hypothesize that stylistic factors might impact the likelihood assignment and propose detect OOD examples by computing a likelihood ratio which depends on the semantic factors that remain after the dominant stylistic factors are cancelled out. On the other hand, Nalisnick et al. (2019b) argue that samples generated by a generative model reside in the typical set, which might not necessarily coincide with areas of high density. They demonstrate empirically that OOD examples can be identified by checking if an input resides in the typical set of the generative model. Unlike the standard experimental setting, they aim to identify distributional shift, which predicts a batch of examples are OOD.
EXTENDING GRAM MATRICES FOR OUT-OF-DISTRIBUTTI DETECTION
Overview In light of the above considerations, we are interested in proposing a method that does not require access to any OOD examples, that does not introduce hyperparameters that need tuning, and that works across architectures. Gram matrices can be used to compute pairwise feature correlations, and are often used in DNNs to encode stylistic attributes like textures and patterns (Gatys al. 2016). We extend these matrices as will be described below, and then use them to compute class-conditional bounds of pairwise feature correlations at multiple layers of the network. Starting with a tre-trained network, we compute these bounds over only the training set, and can then use them to effectively discriminate between in-distribution samples and out-of-distribttion samples at test time. Unlike other SOTA algorithms, we do not need to "look" at any fit-of-dditribution samples to tune any parameters; the only tuning required that of a normalizing factor, which we compute using a randomly-selected validation partition of the (in-distribution) test set.
Notation If the considered deep convolutional network has L layers and the hh layer has ny channels. we consider feature co-occurrences between the Σi<sie= ma pairs of feature- maps. (Note that by "layer" we refer to any set of values obtained i limediittlly after applying convolution or activation functions.) We use the following notation:
FI(D)
The feature map at the 1-th layer for input image D; when referring to an arbitrary image D, we just write Fj. can be stored in a matrix of dimensions ni Ps, where nı is the number of channels at the 1-th layer and pt, the number of pixels per channel, is the height times the width of the feature map. The predicted class for input image D The set of all train examples
DJ((D)
Train
Va
The set of all validation examples. 10% of the examples not used in training are randomly chosen as validation examples.
Te
The set of all test examples, disjoint usual from the training and validation sets. We assume that only the test set may contain -iitr---ditiiittioon examples.
3

---

Under review as a conference paper at ICLR 2020
Gram Matrices and Higher order Gram Matrices We compute pairwise feature correlations
between channels of the (-th layer using the Gram matrix:
G: = FiFR
(1)
where Fj is an ni x Pt matrix as defined above.
In order to compute feature correlations with more prominent activations of the feature maps, define a higher-order gram matrix, which we write GÝ, to be a matrix computed identically to the regular Gram matrix, but where, instead of using a raw channel activation a, we use a", the pd power of each activation. Gj is therefore computed using F`", where the power of Ft is computed element- wise; in an effort to retain uniform scale across all orders of Gram matrices for given layer, we compute the relement-wise p-th root. The p-th order gram matrix is thus computed as:
G = -FP 3
(2)
We show in Section [5] that higher p values help significantly in improving the OOD detectability. In our experiments, we limit the value of p to 10, as exponents beyond 10 are not worth the extra computation that is needed to avoid overflow error
The flattened upper (or lower) triangular matrix along with the diagonal entries is denoted as G.. The set of all orders of gram matrices (in case {1.... 10}) to be considered is denoted by P. The schematic diagram of the proposed algorithm is shown in Fig. [5| (in Appendix
Preprocessing If we compute G. for every layer / and every order p E P, we obtain a total of Ns Epep Σt (n + correlations for any image D. The preprocessing involves computing the class specific minimum and maximum values for the correlations: for every class the minimum and maximum values for each of the Ns correlations are computed over all training examples D classified as We keep track of the minimum and maximum values of the Ns correlations for
all the classes in 4-D arrays Mins and Maxs, each of the order C L P x
max (x(m++)) ISLE
Since each layer has different number channels, the 4-th dimension has been large enough to accommodate the layer with the highest number of channels.
Algorithm 1 Compute the minimum and maximum values of feature co-occurrences for each class, layer and order
Input:
C: Number of output classes
L: Number of Layers in entire network
P: Set of all orders of Gram Matrix consider
Train: The train data
Output:
Mins, Maxs
1: Mins[C][L][[]]
max +1()++) 2
f- Do
Stores Mins each class, layer order
2: Maxs[C][L][[]
max 1S1S
me(m++1)
f- -
D Stores the Maxs for each class, layer and order
3: for i [1,C)d do
4: -Train, = (DID D E Train s.l. ((D)
[D| for D E Train. do
All the training examples predicted as
6: for in [1, L| do
7:
for in P do
stat (D)
> The flattened upper triangular matrix
9:
for in m do
10:
min(Mins [p] [sta(()))))) Maxs [ max(Maxs [c] [ [t)sta(())
Mins [
11:
12:
return Mins, Maxs
The maximum activation values observed in the convolution layers of ResNet trained on Cifar- 10 (open- sourced Lee et al. (20T8h) are 6.5 and 6.3 train and partitions.
4

---

Under review as a conference paper at ICLR 2020
Computing Layerwise Deviations Given the class-specific minimum and maximum values of the Ns feature correlations, we can compute the deviation of the test sample from the images seen at train time with respect each of the layers. In order to account for the scale of values, compute the deviation as the percentage change with respect to the maximum or minimum values of feature co-oceurrences; the deviation of observed correlation value from the minimum and maximum correlation values observed during train time can be computed as:
if min S value S max
mmmi, =
value
if value < min
Jnnin
value max
if value > max
max
The deviation of test image with respect to a given layer is the sum total of the deviations with respect to each of the Epep 2 mni(m + correlation values:
P [nf(++11
5((D) = Σ
Σ
8 (Mins(D.[Albll, Maxs(DD.ll12 (f (Diid)
(4)
Total Deviation of a test image D (A(D)), is computed by taking the sum total of the layerwise deviations (D). However, the scale of layerwise deviations (ōz) varies with each layer depending on the number of channels in the layer, number of pixels per channel and semantic information contained in the layer. Therefore, we normalize the deviations by dividing it by Eva [ót], the expected deviation layer dj. computed using the validation data, Note that we use the same normalizing factor irrespective of the class assigned.
(i( Eva [6:]
A(D)
Threshold As is standard (Lee et al. 201 8b), threshold, r. for discriminating between out-of- distribution data and in-distribution data is computed as the 95th percentile of the total deviations of test data (A(D)) In other words, the threshold is computed so that 95% of test examples have deviations lesser than the threshold т; the wreshold-basee discriminator can be formally written as:
True if A(D) > T, if A(D) ST
isOOD(D)
False
Computational Complexity. In order reduce computational time, we in fact compute deviations based on row-wise sums rather than individual elements. This would mean that the variable stat, defined in line 8 of Algorithm [ would now contain row-wise sums of G? instead of the flattened upper triangular matrix; the inner loop of Eq. 4 would loop over ny elements instead of (n + 1) elements while also reducing the storage required for Mins and Maxs. practise, we found that computing the anomalies this way yields differences of less than 0.5%, and usually imperceptible, so the results described in the next section were computed this way.
EXPERIMENTS - DETECTING OOD
In this section, we demonstrate the effectiveness of the proposed metric using competitive deep convolutional neural network architectures such as DenseNet and ResNet on various computer vision benchmark datasets such as: CIFAR-10, CIFAR-100, SVHN, TinylmageNer, LSUN and iSUN.
For fair comparison and to aid reproducibility, we use the pretrained ResNet (He et al. 2016) and DenseNet (Huang et al. 2017) models open-sourced by Lee et al. (2018b). i.e. Res and DenseNet3 models trained on CIFAR-I0, CIFAR-100 and SVHN datasets. For each these models, we considered the corresponding test partitions the in-distribution (positive) examples. For CIFAR-10 and 100, we considered the oft-of- distrriibbttiin datasets used by Lee et al. (2018b): Tinylmagenet, LSUN and SVHN. Additionally, and DenseNet models trained on SVHN, we used considered CIFAR-10 10 dataset as the third OOD dataset. Details these datasets available in Appendix
also considered the iSUN dataset. For ResNet
5

---

Under review as a conference paper at ICLR 2020
Can work with pre-trained Net?
Can work without knowledge of OOD test examples?
DPN (Malinin and Gales 2018)
X
Semantic (Shalev al. (018))
X
7
Variational Dirichlet (Chen al. 2019)
X
7
Mahalanobis (Lec al. 2018b)
7
X
ODIN (Liang et al. 20188
X
OE (Hendrycks (2019a)
X
V
Baseline (Hendrycks and Gimpel 2017)
V
Ours
Table I: List closely related methods. Note: uses OOD examples during training, but unrelated to test
We benchmark our algorithm with the works listed in Table using the following metrics:
1. TNR INR@95TPR the probability that an OOD (negative) example is correctly identified when the positive rate (TPR) as high 95%. TPR be computed as TPR TP/(TP + FN), where TP and FN denote True Positive and False Negative respectively.
2. Detection Accuracy measures the maximum possible classification accuracy over all possible thresholds discriminating between in-distribution and out-of-distribution ex- amples. For those methods which assign higher-score to the in-distribution exam- ples, it can calculated as max, P0.5Pm m (f(x) 2 7) + 0.5P (f(x) < <));; for those methods which assign a lower score to in-distribution examples, it can be calculated as max, (0.5P... (f(x)= S 7) + 0.5P (f(z) > T)}.
3. AUROC is the measure of the area under the plot of TPR vs FPR. For example, for those methods which assign higher score to the in-distribution examples, this measures the probability that an OOD example is assigned a lower score than an in-distribution example.
Experimental setup:
We use a pre-trained network to extract class-specific minimum and maxi- mum correlation values for all pairs of features across all orders of gram matrices. Subsequently, the total deviation is computed for each example following Eq. Since the total deviation values depend on the randomly selected validation examples, we repeat the experiment 10 times get a reliable estimate of the performance. The OOD detection performance for several combinations of model architecture, in-distribution dataset and -ft-t-ddisttitttionn dataset are shown in Table The results for Outlier Exposure (OE) are available in Table some more results for OE and the results for DPN, respectively.
Variational Dirichlet and Semantic are available in Appendix E. and Appendix
The results of Table 2 show that at glance, over a total of 24 combinations of model architecture/in- www.aaaseeetootto--fsssriiiieeee -dttassss,, the proposed method outperforms the previous com- peting methods in 15 of them, is par in 6 of them, and gives second highest results on of th Furthermore, does so without requiring access to samples from the OOD dataset. the hyperparameters and/or parameters of Mahalanobis and ODIN algorithms fine-tuned using FGSM adversarial examples instead of the real OOD examples, their performance decreases. We also observe that our performance is similar for both architectures.
We also performed experiments with fully-connectee networks by using three different MLP architec- tures trained on MNIST; PaSHiOn-MNIST (Xiao et al. (017) and KMNIST (Clanuwat al. 2018 were considered as the out-of--istiibution datasets (Results are provided Appendix E.3
"This based on the TNR at TPR 95% value; AUROC and Detection Accuracy results are comparable.

---

Under review as a conference paper at ICLR 2020
In-dist
TNR at TPR 95%
AUROC
Detection Acc.
OOD
(model)
Baseline / /OIIN/ Mahalamohis Ours
ISUN
44.6/ 1666/39./97.8 99.88 99.3
91.0/94.0/ 00..0//99...0 /9.5 99.7
85.07 050786.579/11
98,1
CIFAR-10
LSUN
4 98.8 99.6
91.0/ 94,1 19.77 99.88
85.3/ /../886.7/9.. 98.6
(ResNet)
TinyImgNet
41.0/
97.1 98.7
91.0 94.0 99.5 99.6
96.3 97.8
SVHN
70.3/ 87.8 19.66
89.9 96.7 199. 99.4
85.1/91.1/ /9.8 96.7
iSUN
16.9/ 89.9 95,1
75.8 / 85,5 97.9/ 98.9
00.1/78.5799 95.2
CIFAR-100
LSUN
18.8 90.9 97.0
75.8 / 85.6 98.2 99.3
69.9 09.9/78.3/ 93.5 96.2
(ResNet)
TinyImgNet
20.4/33.1 90.9 95.1
77.2 87.6 98.2
7 80.1 93.3 95.1
SVHN
6 /814
79.5 /9.5/93.9 98.4 96.2
73.2 88.0 93.7 89.8
ISUN
62.5/93.2 / 95.37 99.1
9777998.7999777 99.8
89.2/94.37 95. 98.0
CIFAR-10
LSUN
66.6//6.2 /97.2 99.5
9 99.3 99.9
90.3/95.7 96.3 97.9
(DenseNet)
TinyImgNet
98.9/92.4 95.00 98.8
94.1 98.5 98.8 99.77
/ 95.0 / 97.9
SVHN
80.2/ 90.8 96.0
89.9 99.5 98.1 19.1
83.2/914/ 93.9 95.8
ISUN
14.9/ 3
69.57 97.4 99.1
73,477,.4999 92.4 95.7
JIFAR-100
LSUN
41.6 /91.4
70.8 / 85.5 98.0 99.4
64.9 993.9/999 96.4
(DenseNet)
TinyImgNet
17.6 42.6 86.6 95.8
71.7 / 85.2 97.4 99.0
65.7
92.2 95.6
SVHN
70.6/ 82.5 89.4
82.7 93.8 197.2 99744
7 86.6 91.5 92.4
ISUN
78.31 82.2 99.7 99.3
94,4 99.77 99.9 99.8
89.6 89.7 99.2 /8.33
SVHN
LSUN
77.1 99.9 99.5
94.1 / 94.5 99.9 19.88
89.1 89.2 99.3 98.5
(DenseNet)
TinyImgNet
19.1 99.9 99.1
94.8 99.1 99.9 99.7
90.2 90.4 98.9 97.9
CIFAR-10
69.3/71.7 96.8 80.2
/1.9/ / 98.9 95.5
86.6 85.8 95.9 89.0
iSUN
7 99.7 99.1
/ 99.8/ 99.8
89.7 / / 98.3 98.1
SVHN LSUN
/ 99.9 99.4
91.6 89.4 99.9 99.88
89.0 87 99.5 98.5
(ResNet)
TinyImgNet
79.0 82.0/ 99.9 99.3
93.5 92.0 99.9 99.7
90.4 89.4 99.1 97.9
CIFAR-10
7 98.4 85.7
92.9 / 92.1 99.3 97.3
90.0 89.4 / 96.9 / 91.9
Table 2: Comparison of OOD Detection Performance for all combinations of model architecture and training datasct are shown. The hyperparameters of ODIN and the hyperparameters and parameters of Mahalonobis are tuned using a random sample of the OOD dataset.
Mean TNR @ TPR95
In-dist
OE
Ours
OE
Ours
(Base)
(Base)
CIFAR- 10
65.1
90.5
51.7625
98.7
CIFAR-100
37.3
61.5
19.15
93.4
SVHN
93.7
99.9
76.65
95.2
Table Comparison of results with OE (Hendrycks et al. 2019a). Since OE uses a different model from ours, we also report the corresponding baseline accuracy. We extract the mean TNR @ TPR95 for our technique by considering both ResNet and DenseNet models. Some more results are available in Appendix E.T
The metric does not perform well for near-distribution outliers (for example, CIFARIO vs CIFAR100) the detailed results can be seen in Table 4 recent work which performs very well for near-distribution outliers and yet does not require knowledge of OOD distribution is Hendryeks et al. (2019b).
Near-distribution Outliers
The above results are obtained when we consider all elements of gram matrix (Algorithm 1. compute deviations from extrema (Eq and finally, compute the total deviation with normalized layerwise deviations (.. In order to better understand the source of performance gain, we conduct a detailed ablation study that considers alternative choices for the three steps outlined before; the alternative choices are chosen in order to answer the following questions: Q1) What if strictly diagonal elements or strictly off-diagonal elements are considered instead of complete Gram Matrix?; Q2) What happens if the deviation is computed from the mean instead of the extrema?: Q3) What happens we do not normalize the layerwise deviations? In all, we conduct 12 experiments: choices for QI x 2 choices for Q2 x 2 choices for Q3. As broad summary, we find that, while there is no single rule that unbroken by an exception, our proposed combination -.e using the complete Gram matrix, using the min/max metric, and using normalization in Eq generally more robust than any of the other combinations that we tried. More details on the ablation tests and discussions are available in Appendix C
Source of Performance Gain
7

---

we find Under review as a conference paper at ICLR 2020
5 DISCUSSION AND CONCLUSION
Beyond explicit OOD detection, this line of work may ultimately help better interpret neural networks' responses OOD examples. With this goal in mind, and at the same time clarify the internal mechanism of our method, we perform tests to address the following two questions:
1.
Which representations are most useful? In order to examine the role of the depth at which we compute G' in detecting OODs, we construct detectors which make use of correlations derived from just one residual or dense block at time; however, all orders of gram matrices are considered. Representative results are shown Figure T For all combinations of model/in-distribtionn// hatpii///ouuoo--iiissrrutttonddd are much more informative discriminating between in-distribution and cuttof-distribution datasets. However, the difference in detective power depends on the in-distribution dataset considered: for example, the difference in detective power between higher-level representa- tions and lower-level representations bigger for Cifar-100 than for Cifar- 10. More graphs are available in Appendix C.I
we find that the lower level representations
2. Which orders of gram matrices are most useful? In order understand which orders of gram matrices are most helpful in detecting OODs, we construct detectors which make use of only one order of gram matrix at a time; however, correlations are derived from the representations of all residual and dense blocks. Representative results shown in Figure 2 For all combinations of that the higher order gram matrices much more informative in discriminating between in- distriibbution and out-of-distribution datasets. Ignoring the variations at orders greater than 4, we find that the TNR @ 95TPR increases with higher orders and finally saturates. More graphs are available in Appendix C in Figure
Conclusion. Out-of-dee off.ofiiributtionn detection is challenging and important problem. We have pro- posed and reported on relatively simple OOD detection method based on pairwise feature correla- tions that gives new state of the art detection results without requiring access to anything other than the training data itself.
100
Outs.
100
Ours
Mahulanobis
Mahulanobis
TNR TPR95
TNR TPROS
60
ODIN
ODIN
40
Baseline
20
Bascline
40
2
3
2
Block
Blook
(a) ResNNe/CIFAR-10 vs Tiny Imagenet
(b) DenseNee/CIFAR-III -10 vs Tiny ImageNet
Figure 1: Significance of depth: The TNR @ TPR95 is computed by constructing detectors which make use of all the gram matrices but consider only one residual or dense block time. ResNet32 has residual blocks and DenseNet3 has dense blocks.
8

---

Under review as a conference paper at ICLR 2020
100
Our-
100
Oars
Malatnbbis
Mahalanobis
80
TNR @ TPR95
TNR @ TPR95
60
ODIN
60
ODIN
Baseline
Raseline
40
2
10
2
4
10
Order of Gram Matrix
Order of Gram Matrix
(a) ResNet/CIFAR- 10 vs Tiny Imagenet
(b) DenseNes/CIFAR-100 vs Tiny ImageNet
Figure 2: The importance of higher order gram matrices: The TNR@TPR9) is computed by constructing detectors which make use of only one of the gram matrices hut consider all layers.
REFERENCES
W. Chen, Y. Shen, W. Wang, and H. Jin. A variational dirichlet framework for detection, 2019, Ut//ntts:://ppenrriies wtwpeeewwwii.eeefffreeeeee fet:/oormmddddyyniiieee
9FQ
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 3606-3613, 2014. doi: 10.1109/CVPR.20144.22
URL
fitpss:/dooi ww.ddoi.org.. org/ www.1119//VPR.2222 2014.
Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and Ha. Deep learning for classical japanese literature, 2018.
DeVries and G. W. Taylor. Learning confidence for out-of-distribution detection in neural networks.
arXiv preprint arXiv: 08880.00085, 2018
Y. Gal and Z. Ghahramani. Dropout As a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33rd International Conference on Machine Learning Volume 48, ICML 16, pages 1050-1059, New York, NY, USA, June 19-24, 2016. JMLR.org. URL http://al. amm. org/ ei witaatttiiiiccc www.cccwwccc...m
3045502]
L. A. Gatys, A. S. Beker, and M. Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2414-2423, 2016. doi: 10..1099/VVP..211..2266 URL https: doi. org. wtt.g//...../.../......11 1109/CVPR. 6016.265)
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 770-778, 2016. doi: 10.1109/CVPR.2016.990 URL https:
7/doi 10. 1109/CVPR. 201 9.
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out- out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017. URL https:// openreview.nor http:/eee//oooooeeee htaes//forummmddHgggiio -id=Hkg41I9x1
D. Hendrycks. M. Mazeika, and G. Dietterich. Deep anomaly detection with outlier exposure. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019a. URL
1=HyxCxhReY7
D. Hendryeks, M. Mazeika, S. Kadavath, and D. Song. Using self-supervised learning improve model robustness and uncertainty. Advances in Neural Information Processing Systems (NeurIPS), 2019b.

---

Under review as a conference paper at ICLR 2020
G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, Honolulu, HI. USA, July 21-26 2017, pages 2261-2269, 2017. doi: 10.1109/CVPP.2011.2255
URI.tttps /ttp:://doi.oo org 10. O. 110 /9/CVPR.C///// 2017. .243.
K. Lee, Lee, K. Lee, and Shin. Training confidence-calibratee classifiers for detecting out-of- distribution samples. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. 2018a. URL
net for id=
K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems 31; Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pages 7167-7177, 2018b. URL
S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In 6th International Conference Learning Representatiinns ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL htttss://ppenre///
net Centt/forriie forum wwt.m/oodoummmooooooe dd1GHIIIIIIII
A. Malinin and M. J. F. Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neu- ral Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Mon-
tréal, Canada., pages 7047-7058, 2018. URL http: http:/ppprrrssspcccee.ceectc
E. T. Nalisnick, A. Matsukawa, Y. W. Teh, D. Görür, and B. Lakshminarayanan. Do deep generative models know what they don't 't know? In 7th International Conference Learning Representattons, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019a. URL xwNhCcYm
net/forum?inn
E. T. Nalisnick, A. Matsukawa, Y. W. Teh, and B. Lakshminarayanan. Detecting out-o--dist inputs deep generative models using test for typicality. CoRR, 0bs/1996.02994, 2019b. URL
http:/axvvvv.o org. abs/ 02994
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 20/ I, 2011. URL http::/ufldl....... st an wew.dedduuuuuuuue http://aaa.......oooooooeooe//ae
(r120111houuseumbeer pdf
A. M. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 427 2015. doi: 10. 109/ CVPR.20015.2988400 URL https::////....... http:/ii.gggii//////ee / 10. 1119/1110//// 2015. 7298640
I. M. Quintanilha, R. de M. E. Filho, J. Lezama, M. Delbracio, and L. O. Nunes. Detecting out-of-distribution samples using low-order deep features statistics, 2019. URL https://
openreview.nett forum? Ld=zkggCoRctm,
J. Ren, P.J. Liu, F. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dillon, and B. Lakshminarayanan. Likelihood ratios for -ut-of-distiibtiion detection. CoRR, (od//990002845, 2019. URL http:
//rxxiv.rrg/aa / 1906 02845
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision. 115(3):211 Dec 2015. ISSN 1573-1405. doi: 10. /tL org. 100.//00...00//.1
0816-y
G. Shalev, Y. Adi, and J. Keshet. Out-of-distribution detection using multiple semantic label representations. In Advances Neural Information Processing Systems 31: Annual Confer- ence Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018 Montréal, Canada., pages 7386-7396, 2018 URL ptattp:/pappes.... nips. httppps.ccpppppp...p
1967-out-of-distee
detect inneeettitouuuue on
tipl e- semant labe! reppesentat. ons
10

---

Under review as a conference paper at ICLR 2020
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnin novel image dataset for benchmarking machine learning algorithms, 2017.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene recognition from abbey Zoo. In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pages 3485- 3492. 2010. doi: 10. 10.1100/CVP2.2000.555335 URL https: org 1109//VVR 2010.553997
D. Yu, J. Li, and L. Deng. Calibration of confidence measures in speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 19()):441-22733, Nov 2011. doi: 10.1 109/ 1...14998..
F. Yu, Y. Zhang, S. Song, A. Seff, and J. Xiao. LSUN: construction of large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365,, 2015. URL http:
7/arxiv. wwwpi//.rrr/bb//15..oooo 03365

---

Under review as a conference paper at ICLR 2020
A
SCHEMATIC DIAGRAM
Activation Fune
Activation Fune
Fully- Comnected
Predicted class
Teat image (D)
Sofimax
Conv
Conv
(D.
Pairwise Correlations between the feuture-maps of
every layer computed using Gram matrices of various orders. In the preprocesingg stage, the class-specific element-wise minimum and maximum values noted for each of the gram-matrices.
{G Iper
[G] her
ber
G2 lper
Layerwise Deviation (ái(D) sum deviation of the entries in all gram matrices {G]) per from their corresponding minimum and maximum values extricted from training data points classified us D.. In other words, for all channel-puirs, if uny of the computed correlation values are greater (or lesser) than corresponding the maximum minimum) value extracted for training points classified the extent deviation noted.
61 (D)
6 ()
(2)
6z (D)
Total deviation (4) computed by summing across deviations of the layers. However, since the seale deviations of each layer different, normalize by dividing it with Fy, (&.] the expected deviation layer computed using the Validation Data.
A(D) Σ
6;(
Ev. [6;]
1m1
Figure 3: The Schematic Diagram demonstrating the proposed algorithm
B DESCRIPTION OF OOD DATASETS
The following includes the description of the out-of-distribution datasets:
1. TinyImagenet, subset of ImageNet (Russakovsky et al. 2015) images, contains 10,000 test images from 200 different classes. Each image downsampled size 32 x 32 and all 10,000 images are used, as given in the opensourced version by Liang et al. (2018).
2. LSUN, the Large-scale Scene UNderstanding dataset Yu et al.
2015 has 10,000 test images from 10 different scenes. Each image is downsampled to size 32 x 32 and 10,000 images are used, given in the opensourced version Liang al. (2018).
3. ISUN, subset of SUN images (Xiao et 2010), consists of 8925 images. Each image is downsampled to size 32 32 and is used; the downsampled version of the dataset has been opensourced by [Liang al. (2018).
4. SVHN, the Street View House Numbers dataset (Netzer et al., 201T), involves recognizing digits 0-9 in natural scene images. The test partition consisting of 26,032 images is used.
C DETAILED ABLATION RESULTS
The results in the main paper correspond to the performance obtained when considering:
1. Feature Set: all gram matrix entries
2. Metric: layerwise deviations computed with respect the mins and maxs.
3. Aggregation Scheme: the total deviation is then computed using 5q
In this section, detailed ablation results are reported by considering other options. Specifically:
Alternate Feature Set: In addition to considering all gram matrix entries, we consider a proper partition of the gram matrix: strictly diagonal elements, and strictly off-diagonal
12

---

Under review as a conference paper at ICLR 2020
elements. The diagonal elements correspond to the unary features, while the off- -if-onago elements correspond to pairwise features. This can be done by appropriately changing the definition of variable stat in Line 7 of Algorithm []] In these experiments, row-wise sums wherever the size of stat is (n²)); ); in other words, we consider row-wise sums when considering off-diagonal elements and all gram matrix entries.
consider
2. Alternate Metric:
An alternative formulation for computing feature-wise deviations can be to compute the deviation from the means using the one-dimensional Mahalanobis distance. In the preprocessing stage, this would be done by storing the Means and Variances of stat (feature-wisd instead of their Mins and Maxs. Under this new alternative, the function 8 defined in Eg would be redefined as:
(value mean)
8( (mmann, variance,
variance
Accordingly, the layerwise deviation 6 can be defined as:
6(( = Σ (D
[Means[D.][blld) Variaanc/c/c/////llllldddd. Cf (D)(i)
(8)
where GP(D) would correspond to the statistic chosen in the previous step: diagonal entries only, row-wise sums of off-diagonal entries only or row-wise sums of complete gram matrix. We thus consider 2 options for computing the deviations: the Min/Max method presented in the main paper and the Mean/Variance method (Gaussian) described above.
3. Alternate Aggregation Scheme: In order to compute the total deviation A from the lay- erwise deviations 5i we can compute it by following or taking simple sum as shown:
A
8
We refer to 5 as the normalized estimate and Eq 9 as the unnormalized estimate.
In all, the Table 4 reports detection rates in 12 settings: 3 choices for stat (only the diagonal entries of gram matrix G. only the off off-diagonal entries of G, or all of G) x metrics for computing deviation (Min/Max
x 2 choices for computing total deviation (Normalized sum or Unnormalized sum). All layers and all orders of gram matrix are considered in Table 4.
DOD
Drapeal Beentt
(Bon ail Stand
(Rize usu Savro
Depen Bleents
(Sew aole Sanei
(Riz sdia
Dasgeaal treeens
(Rew ui Sanii
(Row SSN BAA
(ewale)
Mas!
Min
Mosa
Ma/
May
Yeu
Mtal
Mas
Meaa
Var
Mac
Vie
Min
Mors
102
Man
Mer
Max
Meul Vie
Maz
1ac
Mas!
Min
Moss
Mas
Vie
New'
Moar
Meer
Mas.
Mea!
Nax
Vic
Naal
Miu
Moril
Ma.
Mar
Naxt
9011
9633
18.7
133
05:1
27.5
24.2
08.3
09.1
V1.1
10.3
16.1
16.2
162
93.
16.1
66.7
95.4
16.0
313
05.5
821
10.1
10.1
36.0
00.5
15.3
00.00
15.4
CITAR
TackegNa
00.
60.8
06.3
18:3
01.0
07:2
35.1
66.5
06.7
073
66.7
674
59.6
99
18.3
65.5
996
06-
11.5
07.8
08.6
99.4
08.3
61:
81.5
656
64.1
01.3
4711
66.3
05.3
00.1
07.8
55.4
02.4
43.6
20.6
12.5
424
54.
85.1
13.35
$11
15.0
08.3
11.8
11.0
18.5
'1.3
11.4
12.5
102
34.5
36.
572
13.5
CHRMR-100
LSUN
TrateaNa
700
12.
97.2
920
525
15.1
23.5
28.3
06.5
46/1
15.0
99.1
10.
10.55
V1-2
54.
15.1
15.0
51.3
01.5
06.5
12.4
55.5
1835
10.1
15.1
15.1
$1.5
10.2
53.
2015
781
36.6
25-2
523
91.8
10.5
19.2
932
86.3
345
129
16.1
19.
11.
179
12.1
16.1
18.0
71.2
76.6
75.3
719
67.6
75.3
75.5
54.
360
4000
64.
09.5
65.3
TE.1
12.5
045
18.6
566
M92
56.
14:0
447
18.6
661
141
(Denchaa
SVHX
301
193
03.9
567
969
16:1
54.
16.5
97.6
60
59.1
15.3
10.2
911
0031
553
51.0
52.0
IKN
94-5
12.4
63.5
861
88.
251
357
05.
55.4
06.
07.3
15.6
682
190
079
919
960
66.1
616
08.9
69.0
657
654
11.
851
00.6
05.3
08.4
653
12.4
05.5
15.5
CIRAR (Denseott
TackegNor
45.
51/
02.4
03.3
053
360
353
655
56.4
05.
661
540
06.6
67
990
07.
66.7
66.0
98.9
667
455
464
663
05.5
015
655
51.
05.2
647
CIRAR-11 10
50.
16-
11.1
13.
186
156
303
139
10-5
15.6
163
15.
61.1
701
66.0
667
61.
66.5
617
66.
613
647
66
66.3
416
611
633
567
63.
016
631
411
010
6-5
735
773
911
163
353
SYIEN BBubbro
LSUN
916
1903
200
19.
15.0
15.4
911
1911
15.3
555
19.1
15.1
909
18.5
18.5
110
18.
51.5
$2.6
05.7
16.1
12.
95.
06.9
27.8
26.1
85.5
645
00.4
29.5
193
15.3
1936
930
19-1
95.5
953
97.8
AELL
625
96.2
11.9
963
55.3
02.6
642
SVIN (Dencheet
3011
1976
211
271
191
111
155
111
111
151
111
187
1511
10:
191
191
111
311
TealegNa
1912
10.4
692
191
1612
1001
1911
192
1922
16.1
991
1936
1916
15.7
595
19:0
919
16.5
96.2
18.11
10.1
61.
15.1
16.1
51.3
TEPAN
113
101
151
153
100
011
15
10.11
111
100
111
16.1
411
11.
161
055
153
914
413
155
Table 4: Detailed Ablation Results demonstrating the detection rates under 12 different settings. The MEAN and STD-DEV are computed by using all elements in the table excepting the CIFAR-I0 CIFAR- 100 and CIFAR-100 vs CIFAR-10 10 entries. Note: We will of course fix the formatring of this for future versions; this just an initial draft during discussion period.
By analysing the ablation results, we attempt answer the following questions:
13

---

Under review as a conference paper at ICLR 2020
1. Are pairwise features more useful than unary features? We observe that the Min/Max metric can work equally well with both unary and pairwise features; in some cases, the unary features are marginally better (Ex: ResNet/CIFAR-IO 10 vs SVHN) and in some cases, the pair- wise features are marginally better (Ex: ResNe/CCIFRR- 100 vs Interestingly, the behavior of the Mean/Var metrie is different: the performance with pair- wise features are significantly higher than with unary features in 19 out of 28 tested cases. For example, the TNR at TPR95 for ResNeu/CIFAR-000 vs TinyImgNet 68.0 with unary features and 84.2 with pairwise features.
We notice that using the unary features (diagonal entries) sometimes did well when pairwise features (off-diagonal entries) did not do well, and vice versa, so using both gives the kind of effect that we want in an ensemble: models that cover and work well over different parts of the space. Therefore, an overall message of our experiments is that is worthwhile to consider all elements of the gram matrix.
2. Is it neccessary to use Min/Max metric? Except in 6 cases (ResNet: CIFAR-10 vs CIFAR-100, ResNet: CIFAR-100 vs CIFAR-1O, DenseNet: CIFAR-I0 vs CIFAR-100, DenseNet: CIFAR-100 vs CIFAR-10, ResNet: SVHN CIFAR-10 and DenseNet: SVHN vs CIFAR-10), the min/max metric consistently performs better than the mean/var metric. Additionally, is not clear if the Mcan/Var estimate performs better with normalized sums or unnormalized sums: for example, observe that Mean/ Var estimate performs very poorly with the unnormalized estimate for ResNet/CIFAR-II while the performance of Mean/Var for DenseNet/CIFAR 100 is competitive with the performance of Min/Max only when an unnormalized estimate is computed.
One can observe that computing the miedimensional Mahalanobis distance for each com- ponent of the statistic derived from the Gram Matrix and later computing the total sum is equivalent to representing each input image by a big vector (say, Z) derived from the Gram Matrices computed across various layers, constructing class-conditional distributions of Z (assuming that each component of Z is normally distributed and independent of the other components) and subsequently computing the probability of an unseen Z. In early rescarch, we noticed the following problems with the Mean/Va estimate:
(a) The individual components of gram matrices do not follow normal distribution strictly and Mean/Var assigns lower probabilities to the in-distribution images as well.
(b) The total deviation A computed by simply summing across the layerwise deviations, 6 was not able to accurately summarize the information contained in the different ō1s. Specifically, information about the layer where the input example had higher deviation was lost when a simple sum was taken.
The proposed Min/Max idea solves problem (a) by employing weaker metrie: from extrema instead of the mean. It can also be said that the Min/Max metric considers a uniform probability density between the extrema. Problem (b), which exists even for this newer metric, is solved by the normalization scheme described in the main paper for computing the sum total deviation.
deviation
Higher Order Gram Matrices The Min/Max metric is a weak approximation to the true prob- ability density. On conducting a thorough analysis of how the OOD examples were able to fool the metric, it appeared that the intermediate features had several tiny activations that could yield innocuous correlation values. Higher-order gram matrices as described in the main paper provide a natural way to mitigate these effects.
Notable observations from Figures through (all layers are considered but only one order of gram matrix is considered at a time):
Ensemble effect: In 24/28 cases, higher order gram matrices improve detection rates. Higher order gram matrices help both the Min/Max and the Mean/Var metrics. In most cases, the even powers are more helpful than the odd powers; in some cases, the odd powers are more helpful (Ex: DenseNet/CIFAR- 100 vs CIFAR-10), Despite these variations, it is possible to get an ensemble effect by considering all possible powers as demonstrated in the main paper.
14

---

Under review as a conference paper at ICLR 2020
In ResNet: CIFNR- 10 CIFAR- 100 and DenseNet:CIIAR-III vs CIFAR-100 100, the higher order gram matrices yield lower detection rates. We find these exceptions interesting, and would like understand them better in future.
Summary The unambiguous message from this ablation study is that the Gram matrix contains useful information which can be used for detecting OOD examples. While the standard Mean/Variane Variance metric does not always work well, the proposed Min/Max metric yields consistent performance competitive with state-of-the-an methods. The use of higher-order Gram matrices further boosts the overall performance. Although the Min/Max method can work very well for fan-from-distribution examples, it does not work well when fine grained estimate is needed (for example, CIFAR- 1- vs CIFAR- 100). We hope the strong empirical proof that Gram matrices contain useful information motivate the development of OOD detectors with powerful density estimators; eventually, such estimators might not need higher-order Gram matrices to boost their performance. Here they allow us propose a metric based on simple thresholding that, generally, works independently of the particular OOD examples and independently of the architecture, and we believe that the fact that this the case the fact that this approach scores relatively rell-itself raises several interesting new questions. We hope that answering these questions will ultimately strengthen our understanding of deep neural networks and give rise to more robust neural network models in the future.
M
Figure 4: ResNeu/CIFAR-10: The TNR at TPR95 trends for Min/Max and Mean/Var as the order of Gram Matrix is varied.
Figure 5: ResNet/CIFAR-100 The TNR at TPR95 trends for Min/Max and Mean/ Var as the order Gram Matrix is varied.
Figure 6: ResNet/SVHN: The TNR at TPR95 trends for Min/Max and Mcan/Var the order of Gram Matrix varied.
15

---

Under review as a conference paper at ICLR 2020
M
Figure 7: DenseNet/CIFAR-100 10: The TNR at TPR95 trends for Min/Max and Mean/Var as the order of Gram Matrix is varied.
MW
Figure 8: DenseNeu/CIFAR-100 The TNR at TPR9S trends for Min/Max and Mean/Va as the order of Gram
Matrix is varied.
Figure 9: DenseNee/SVHN: The TNR at TPR95 trends for Min/Max and Mean/ Var as the order of Gram Matrix is varied.
C.1
SIGNIFICANCE OF DEPTH
Figure 10: DenseNet/CIFAR-10: 10: The TNR TPR95 trends for Min/Max and Mean/Var go deeper in the network.
16

---

Under review as a conference paper at ICLR 2020
Figure 11: DenseNeuCIFAR- 100: The TNR at TPR95 trends for Min/Max and Mean/ Var as we go deeper in the network.
Figure 12: ResNet/CIFAR-10 10: The TNR at TPR95 trends for Min/Max and Mean/ Deean/Vara we go deeper in the network.
Figure 13: ResNe:/C/FAR-100 The TNR at TPR95 trends for Min/Max and Mcan/Vat Var go deoper in the network.
Figure 14: DenseNet/SVHN: The TNR at TPR95 trends for Min/Max and Mean/Var as we go deeper in the network.
Figure 15: DenseNet/SVHN: The TNR TPR95 trends for Min/Max and Mcan/Var go deeper in the network.
17

---

Under review as a conference paper at ICLR 2020
D COMBINING OE + OURS
SANTEE
In-Konntatie
DX10
AUBIC:
DUACC:
113
AUROI
DYTAEE
AUROC:
ISTAXX
ALBOC
DIACE
E1ST
AUROC
DESAEE
AIIRICC
ITTACC
LSUN
41.5
01.3
13.0
05.3
19.5
156.
99.3
19.4
93.3
93.0
55.4
19.5
103.0
1913
CIRAR
39.1
68.3
81.6
02.1
06.3
CIPARIOO
3330
$5.8
11.5
25.7
16.3
66.5
26.1
73.6
09.1
23.5
64.5
17.8
26.5
16.
1126
7622
151
1533
100
2233
10.3
3313
173
11.3
16.3
133
150
773
10.1
CHAAR-100
256
15.3
71.5
09.0
05.3
96.1
08.1
35.1
85.1
11.5
16.3
08.5
08.3
14.8
6188
354
96.8
91.3
88.9
91.3
96.3
88.5
875
Table 5: Table shows results from preliminary experiments combining OE with method. The experiment conducted with pretrained WideResNet open-sourced by Hyeedryyysss all (2019a). MSP uses Maximum Softmax Probability; "Ours" refers the metric A (Eq. 5 "Ours+MSP" obtained by dividing with MSP.
E
FEW MORE OOD RESULTS
E.
COMPARING WITH OE
OE
Ours (Base)
In-distribution
OOD
OE
Ours
(Base)
CIFAR-10
Gaussian
85.6
99.3
43.5
100.
Rademacher
52.4
99.5
48.3
100.
Blob
83.8
99.4
52.9
99.8
Texture
57.2
87.8
37.0
85.3
SVHN
71.2
95.2
45.4
96.1
LSUN
61.3
87.9
58.2
99.5
CIFAR-100
Gaussian
45.7
87.9
18.2
100.
Rademacher
61.0
82.9
15.6
100.
Blob
62.0
87.9
38.4
98.6
Texture
28.5
45.6
19.9
68.5
SVHN
30.7
57.1
23.5
85.4
LSUN
26.0
42.5
18.2
97.2
SVHN
Gaussian
94.6
100.
87.65
100.
Bernoulli
95.6
100.
92.25
100.
Blob
96.3
100.
93.35
100.
Texture
92.8
99.8
72.6
94.9
Cifar-10
94.0
99.9
73.8
83.0
LSUN
93.6
99.9
75.7
99.5
Table 6: Comparison of Mean TNR @ TPR95 values.
Following Hendryeks et al. (2019a), we created the gaussian, rademacher, blob and bernoulli synthetic datasets. Their descriptions are as follows: Gaussian anomalies have each dimension i.i.d. sampled from an isotropic Gaussian distribution. Rademacher anomalies are images where each dimension is or with equal probability, so each dimension is sampled from symmetric Rademacher distribution. Bernoulli images have each pixel sampled from Bernoulli distribution if the input range is [0, 1]. Blobs data consist of algorithmically generated amorphous shapes with definite edges. Textures is dataset of describable textural images (Cimpoi al. 2014)
18

---

Under review as a conference paper at ICLR 2020
E.2
COMPARING WITH DPN, VD AND SEMANTIC.
TNR
TNR
Detection Accuracy
Detection
Method
@
AUROC
Method
AUROC
TPR95
TPR95
Accuracy
DPN
42.60
90.20
79,50
Semuntic
41.50
85.20
88.40
VD
92.30
98,
10
VD
80.20
94.20
87.80
Baseline
49,80
91.00
85,
Baseline
16.89
75.80
70.11
LSUN
ISUN
ODIN
82.10
10
86.70
ODIN
45.21
85.48
78.47
Mahalanobis
98.80
99.70
97. 70
Mahalanobis
89.91
97.91
93.05
Ours
99.85
99.89
98.66
Ours
95.12
98.9
95.18
DPN
71.60
93.00
86,40
Semantic
20.50
79.00
57.80
VD
82.90
96.80
91.30
VD
85,50
95.
90.40
Tiny
Baseline
Baseline
75.80
69.90
IngNell
ODIN
67.90
94.00
86.50
LSUN
ODIN
23.20
85.60
78.30
Mahalanohis
Mahalanabis
98.2
Ours
99.48
99.72
97.82
Ours
97.14
99.28
96.19
DPN
79.90
95.90
87,30
Semantic
37.60
83.10
75. 13.60
SVHN
Baseline
50.50
89.90
85.10
Tiny ImgNet
Baseline
20.40
77.20
70.80
Mahalanobis
87,80
99.10
95.80
Mahalanobis
90.92
98.20
93.30
Ours
98.14
99.50
96.71
Ours
95.12
98.97
95.13
(a) ResNet/CCFAR-10 10
(h) ResNet/CIFFRR1100
Table 7: We compare method with DPN, VD and Semantic by reporting results where available.
E.3 RESULTS FOR FULLY-CONNECTED NETWORKS
Architecture
OOD
Method
TNR @ TPR95
AUROC
Detection Accuracy
Baseline
47.66
73.96
73.91
KMNIST
Ours
98.57
99.66
97.37
300
Baseline
44.93
66.93
71.07
Pashion-MNIST
Ours
93.51
98.64
94.36
Baseline
59.79
75.17
79.49
KMNIST
Ours
97.8
99.4
96.55
300-150
Baseline
70.73
77.10
83.00
Ours
95.2
99.00
95.17
Baseline
70.4
79.75
83.38
KMNIST
Ours
97.5
99.11
96.4
300-150-50
Baseline
73.92
76.54
84.67
Fashion-MNIST
Ours
95.7
98.94
95.48
Table The method even works quite well with fully-connected neural network trained MNIST, The results are shown for 300-unit single layer MLP, 300-150 two-layer MLP and 300-150 150-50 MLP.
19


Reviews:
[{"review_id": "rJe62ec0tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new scoring function for OOD detection based on calculating the total deviation of the pairwise feature correlations. The method only requires in-distribution data for tuning its hyper-parameters and can use a pre-trained classifier directly. Its performance is evaluated with small image datasets, which are commonly used in this line of work. Two additional experiments are performed to analyze the effect of two factors (the layer of the feature and the order of Gram matrix) in the method.\n\nThe overall vote for this paper is a weak reject. The clarity of the paper is good. The primary strength of the work is the very strong performance given the setting, which does not require OOD data for tuning. However, we give a rating slightly below the borderline, because of lack of in-depth explanation of why pairwise feature correlation is helpful for the problem. The explanation could be either theoretical or empirical, while the latter can be a set of carefully designed ablation studies. We do not see sufficient arguments or experiments to ensure the performance gain comes from the pairwise feature and is not from other parts of the design.\n\nThe above weakness could be addressed by answering the following questions:\n1. If the pairwise feature (G) is replaced by the unary feature (F) while keeping the other parts of the pipeline the same (including the use of p-th power), would it reach a similar performance or much worse?\n2. Why does the implementation use the statistics of min/max values instead of mean and variance? The latter can also calculate the deviation for the method using a Gaussian model, which is usually a more natural choice.\n3. What is the motivation for using p-th power, and why does it help?\n\nWe present the additional feedbacks in the form of questions which could further strengthen the work if answered:\n4. Based on Figure 2, having p=10 is sufficient to give a good result. Why does the method still use all p-th power (p=1..10)?\n5. The study in Figure 1 only presents specific OOD data (Tiny Imagenet). Does the trend still hold with other OOD data, such as SVHN or LSUN?\n6. Why are the networks used in figure 1(a) and (b) different? Does the trend still hold when the networks in both cases are changed?\n7. The statistics (Mins/Maxs) to be saved can be very large. The paper provides a strategy to reduce that by using the row-wise sums of G. However, the summation has an effect of mixing the features, causing a weaker signal from the deviation. Why does the method still works with the row-wise sums?\n8. Would the method work if the networks have no batch normalization layer?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xozffaFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses Gram matrices for OOD detection. This enables the reliable detection of far-from-distribution examples, which is a long-standing and surprisingly difficult problem in OOD detection.\nThis is the best paper in my batch due to the strength of the results. However, this paper should more accurately reflect the contribution: this helps with far-from-distribution examples, not near-distribution yet OOD examples.\nFor instance, I used their code and found that their technique leads to an AUROC of 79.01% when using CIFAR-10 as the in-distribution and CIFAR-100 as the OOD set. Likewise, with their code I found having CIFAR-100 as in and CIFAR-10 as out gives an AUROC of 67.95%. This is much worse than currently existing techniques. Hence the paper should re-frame or qualify their results as helping with far-from-distribution detection or the detection of obvious anomalies. This paper makes a solid stride in improving the detection of garbage inputs, but the paper should modify its message so as not to suggest this helps with all currently considered OOD detection tasks.\n\nSmall comments:\n\n> Lee et al. (2018b)—to the best of our knowledge, the current SOTA technique by a significant margin\nThis should be again qualified and expanded. If you assume access to knowledge of the test distribution, then Mahalanobis is easily the best. If not, the Outlier Exposure is best. If you assume access to no extra data during training, then the maximum softmax probability + rotation prediction is best [1].\n\n> However, while the OE method is able to generalize across different non-training distributions, it does not achieve the SOTA rates of Lee et al. (2018b) on most cases.\nThere are different senses of state-of-the-art and these should be qualified.\n\nDoes this technique do better if you do 5th and 9th percentile instead of min and max? Is it important to do the min and max with training examples instead of validation examples? (Not a pressing question.)\n\n> Can work without access to OOD validation examples?\nTable 1 is deceptive. OE does not need the \"validation\" examples.\nI suggest two columns instead of one. \"Can this work without knowledge of OOD test examples? Does this use OOD training examples?\"\n\nShow AUROC and AUPR. Detection accuracy is an unusual metric relative to AUPR (OOD as positive).\n\nShow CIFAR-10 vs CIFAR-100 in the tables or I'll downgrade my rating, since otherwise the paper is not leaving an accurate impression.\n\nSince OE is complementary, perhaps this technique can be combined to tackle these near-distribution cases?\n\nThe title is confusing. \"Zero-Shot\" could be applied to various techniques in this space. Perhaps emphasize Gram matrices?\n\nIn their code:\n        validation_indices = random.sample(range(len(all_test_deviations)),int(0.1*len(all_test_deviations)))\n        test_indices = sorted(list(set(range(len(all_test_deviations)))-set(validation_indices)))\nThese indices change with every power, which is unrealistic. Please fix the sets beforehand.\n\nSince neural style transfer, which uses Gram matrices, works much better with VGG architectures than ResNets, does this technique work better with VGG architectures?\n\n[1] Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. Hendrycks et al. NeurIPS 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bkxza1qaKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a strategy for detecting out-of-distribution samples based on feature representations obtained via neural networks. Given a test sample, the proposed strategy checks whether the correlation values between the features of the test sample obtained at different channels of the same layer are coherent with those of the training samples known to belong to the estimated class of the test sample. \n\nThe proposed strategy can be applied to pretrained networks, as it only requires the channel activation values to determine whether a sample is out of distribution or not. The studied problem is an important problem and the experimental results show that the proposed strategy leads to some performance gains in comparison to reference methods. However, in my view the main drawback of the study is that it is based on an ad-hoc methodology whose theoretical foundation is not quite clear. In particular, it would be good to provide some more explanations on the following issues:\n\n- What exactly motivates the assumption that the correlation values between different channels of the same layer provides a discriminative characteristics of the classes from each other? It would be natural to assume that different classes will have different activation levels at a certain channel of a certain layer. However, the idea of the paper is to look at how different channels correlate with each other. I cannot entirely grasp the motivation for this, as looking at the feature correlations is a bit more indirect than looking at the features themselves. It would be good to provide the justification of this choice.\n\n- What is the theoretical motivation behind using the p-th order Gram matrix, instead of using the original Gram matrix (e.g. p=1)? Some experimental justification is given, but it is also important to provide some theoretical insight if possible.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new scoring function for OOD detection based on calculating the total deviation of the pairwise feature correlations. This is an important problem that is of general interest in our community.

Reviewer 2 found the paper to be clear, provided a set of weaknesses relating to lack of explanations of performance and more careful ablations, along with a set of strategies to address them. Reviewer 1 recognized the importance of being useful for pretrained networks but also raised questions of explanation and theoretical motivations. Reviewer 3 was extremely supportive, used the authors' code to highlight the difference between far-from-distribution behaviour versus near-distribution OOD examples. The authors provided detailed responses to all points raised and provided additional eidence. There was  no convergence of the review recommendations.

The review added much more clarity to the paper and it is no a better paper. The paper demonstrates all the features of a good paper, but unfortunately didn't yet reach the level for acceptance for the next conference. 
Decision: Reject