Improving Entropic Out-of-Distribution Detection using Isometric Distances and the Minimum Distance Score
Authors: David Macêdo
Teresa Ludermir
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Out-of-distribution detection

Abstract: Current out-of-distribution detection approaches usually present special requirements (e.g., collecting outlier data and hyperparameter validation) and produce side effects (classification accuracy drop and slow/inefficient inferences). Recently, entropic out-of-distribution detection has been proposed as a seamless approach (i.e., a solution that avoids all the previously mentioned drawbacks). The entropic out-of-distribution detection solution comprises the IsoMax loss for training and the entropic score for out-of-distribution detection. The IsoMax loss works as a SoftMax loss drop-in replacement because swapping the SoftMax loss with the IsoMax loss requires no changes in the model's architecture or training procedures/hyperparameters. In this paper, we propose to perform what we call an isometrization of the distances used in the IsoMax loss. Additionally, we propose to replace the entropic score with the minimum distance score. Our experiments showed that these simple modifications increase out-of-distribution detection performance while keeping the solution seamless.

Improving Entropic Out-of-Distribution Detection using Isometric Distances and the Minimum Distance Score
Anonymous Author(s)
Affiliation
Address
email
Abstract
Current out-of-distribution detection approaches usually present special require- ments (e.g., collecting outlier data and hyperparameter validation) and produce side effects (classification accuracy drop and slow/ineffcciet inferences). Recently, entropic out-of -itt----iitribttion detection has been proposed as seamless approach (i.e., a solution that avoids all the previously mentioned drawbacks). The entropic out-of-distribution detection solution comprises the IsoMax loss for training and the entropic score for out-tt-distributiin detection. The IsoMax loss works as a SoftMax loss drop-in replacement because swapping the SoftMax loss with the IsoMax loss requires no changes in the model's architecture or training proce- prrcsstyperparametters.. In this paper, isometrization of the distances used in the IsoMax loss. Additionally, we propose replace the entropic score with the minimum distance score. Our experiments showed that these simple modifications increase out-of-distribution detection per- formance while keeping the solution seamless.
5
propose to perform what call an
11
13
Introduction
15
Neural networks have been used in classification tasks in many real-world applications [4. In such cases, the system usually needs to be able to identify whether a given input belongs to any of the classes which it trained. Hendrycks & Gimpel [9] called this capability out-of-distribution (OOD) detection and proposed datasets and metrics to allow standardized performance evaluation 20 and comparison. However, current OOD detection solutions still present limitations (e.g., special requirements and side effects) that prevent a more general use of OOD detection capabilities in
17
19
21
22
practical real-world applications 27 (Table
23 First, OOD detection solutions commonly present hyperparameters that usually presume access to 3 A consequence of presuming access to OOD samples to validate hyperparameters and using the same distribution to evaluate OOD detection results producing overestimated performance estimations 32 To avoid unrealistic access OOD samples and overestimated performance, Lee et al. [19] proposed to validate hyperparameters using adversarial samples. However, this requires the generation of adversarial examples. Moreover, this procedure requires the determination of hyperparameters (e.g.. maximum adversarial perturbation) typically unknown when dealing with novel datasets. Similar arguments hold for solutions based on adversarial training [8] [7 [ [4] [8], which also result in higher training time. Approaches based
distribbbtttion samples be defined 23 22
19
18
25
27
29
30
31
32 on the generation of adversarial examples the use of adversarial training may also have limited scalability when dealing with large images such as those presented in the ImageNet 2
33
Submitted to Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Table 1: Out-of-distributioo detection approaches: special requirements and side effects.
Special Requirement
Side Effect
Approach
Hyperparameter
Outlier
Slow/Inefficient Inference
Classification Accuracy Drop
Tuning
Data
ODIN 2
Required
Not Required
Present
Not Present
Mahalanobis
19
Required
Not Required
|
Present
Not Present
ACET 8
Required
Not Required
Not Present
Present
Outlier Exposure
100
Not Required
Required
Not Present
Not Present
Generalized ODIN
Required
Not Required
Present
Present
Gram Matrices
30
Not Required
Not Required
Present
Not Present
Scaled Cosine
Not Required
Not Required
Not Present
Present
Energy-based 25
Required
Required
Not Present
Not Present
Entropic (Seamless) 27
26
Not Required
Not Required
Not
Not
Iso + Entropic Score
Present
Present
Entropic (Seamless) [ours] Iso + MinDistance Score
Not
Not Required
Not
Not
Required
Present
Present
34
Many solutions make use of the so-called input preprocessing technique introduced in ODIN [23 However, the use of the mentioned technique increases at feast times the inference delay and power consumption (27] since a combination of a first forward pass, backpropagation operation, and second forward pass is required [23
35
36
37
11 3 for single useful inference. Actually, approaches 19.
38 that may be applied directly to pretrained models and altogether avoid training or fine-tuning the
39 model [3 19 30 usually produce inefficient inferences and/or additional computational complexity
40 to perform OOD detection [26), Section IV, DJ. From a practical point of view, this is a drawback, as
41 inferences may be performed thousands or millions of times in the field. Hence, such approaches may 3T and real-world cost-based perspectives.
42
be prohibitive (not sustainable) from environmental
Another harmful common side effect is the so-called classification accuracy drop 33 In such cases, higher OOD detection performance is achieved at the expense of a drop in the classification accuracy compared with models trained using the usual SoftMax loss (i.c., the combination of the SoftMax activation and the cross-entropy loss (24). From a practical perspective, this situation is undesired because the detection of out-of-distribution samples may be rare event. At the same time, the classification is the main aim of the designed system 1
44
46
48
19
Hsu et al. (1]] proposed to use the in-distribution validation set to avoid the need for accessing OOD samples to determine the hyperparameters required by the solution. However, considering that 10 and CIFAR 100 do not have separated sets for validation and testing, the results may also be overestimated because the validation sets used to define the hyperparameters were reused for OOD detection performance estimation. A more realistic OOD detection performance estimation could have been achieved by removing the in-distribution validation set from in-distribution training data. However, this would probably produce an even higher classification accuracy drop. Additionally, the solution proposed in 11 is expensive and not inviromment-friendly as it uses input preprocessing and, consequently, produces slow and energy-inefficien inferences (2
50
51
52
54
56
26 Recently, many OOD
58
detection approaches have used additionallexttra/uttie data [10 25 5 The Gram matrices solution
59
calculates values produced by the model during inference 30 to perform OOD detection.
60 In some cases, an ensemble of classifiers is used 35). For deep ensembles, Lakshminarayanan
81 et al. [17 proposed an ensemble of same-architecture models trained with different random initial
82 weights. Some proposals required model structural changes to tackle OOD detection [37], and
certain trials used uncertainty confidence estimattion/caliraaioon techniques [3] 20 28 16 33)
64 However, Bayesian neural networks used in most of these are usually harder to implement and require
htttps://ww.. http:://www.youuube youtube. . com/vatchi http//aatchhy=nnnnppp KnnKkuugUtta
II this paper, we consider that an approach does not present classification accuracy drop if it always presents a classification accuracy higher or less than percent (1% (%) lower than SoftMax-loss trained models.

---

65
Contributions Our contribution in this paper is threefold: First, in addition to minor changes, we perform what we call an isometrization of the feature-prototype prototype distances used by the Iso loss. We call our modified version of IsoMax the isometric isotropy maximization loss or isometric IsoMax loss (IsoMax, loss). Second, we propose to use the minimum feature- prototype distance as the score much more computational resorces to train. Moreover, computational constraints usually require approximations that compromise the performance, which also affected by the prior distribution used 117. For example, MC-dropout uses pretrained models with dropout activated during the test 68 time. An average of many inferences is used to perform a single decision [6]
66
67
69
The entropic out-of-distribution detection approach, which is composed of the IsoMax loss for training and the entropic score for OOD detection, avoids all mentioned special requirements and side effects 127. Indeed, no hyperparameter tuning is required because the entropic scale is global constant
70
71
72
kept equal to ten for all combinations of datasets and models. Even if we call the entropic scale hypprpaaaameter, the IsoMax does not involve hyperparameter tuning, as the same constant value of entropic scale is used in all situations. This is possible because Macêdo et al. experimentally showed
73
74
75 in 127 Fig. 3] and in [26] Section IV, A] that the OOD detection performance presents a well-behaved
76 dependence on the entropic scale regardless of the dataset and model. No additional/exxraaouttier
77 data are necessary. Models trained using IsoMax loss produce inferences as fast and energy- -nfrgy-efficient
78 as the inferences produced by SoftMax- lallaa---os---rnnne networks. The OOD detection requires only a speedy entropy calculation. Finally, no classification accuracy drop is observed.
80
81
82
83
84
to perform OOD detection. Considering that the minimum feature- feature-prototype distance is calculated to perform the classification, the OOD detection task presents essentially zero computational cost because we simply reuse this value as the score to perform OOD detection. Third, in addition to experimental evidence, we provide insights into why combination of training using the isometric distances provided by IsoMax, and performing OOD detection using the minimum distance scores produces substantial performance increase in OOD detection compared to IsoMax combined with the entropic score. Our approach keeps the solution seamless (i.e., it avoids the previously mentioned special requirements and side effects) while significantly increasing the OOD detection performance. Similar to IsoMax loss, IsoMaxz works as a SoftMax loss drop-in replacement, as no procedures other than regular neural network training are required.
85
86
86
90
92
2
Isometric Distances and Minimum Distance Score
Isometric Distances Consider an input æe applied to a neural network that performs parametrized 96 transformation fo) Moreover, consider p's be the learnable prototype associated with the class j.
95
97 Additionally, let the expression [x) (c')- represent the nonsquared Euclidean distance between (z) and p'g. Finally, consider p learnable prototype associated with the correct class for the 99 input x. Hence, we write the IsoMax loss [27] for a batch of N examples using the equation below:
N log
exp(-Exllfo 1%
LisoMax
(I)
N
Σ Lexp(-E.lIfel r'all)
100
In the above equation, the F,, represents the entropic scale. From Equation T we observe that the distances from IsoMax loss are given by the expression D= [e -p'sl During inference, probabilities calculated based on these distances are used to produce the negative entropy, which serves as a score to perform OOD detection. However, as the features fa( are unnormalized, examples with low norms are unjustifiably favored to be considered OOD examples since they tend
101
102
103
104
105
to produce high entropy. Additionally, as the weights p', are unnormalized, examples from classes that present prototypes with low norms are unjustifiably favored to considered OOD examples for the same reason.
106
107
108
(z)]l- Additionally, we propose to replace p's with its normalized version given by p's pp// p//ll. The expression ||// represents the 2-norm of a given vector
Hence, we propose to replace fo(x) with its normalized version given by
109
3

---

Table 2: Classification accuracy of models trained using MoftMax, IsoMax, and IsoMaxy losses. In addition to avoiding classification accuracy drop compared with Soft.Ma--los loss- and Iso trained networks, 1soMaxy-loss-trannee models show higher OOD detection performance (Table 3).
Train Accuracy (%)
Test Accuracy (%) [t]
Model
Data
SoftMax Loss IsoMax Loss / IsoMax, Loss
CIFARIO CIFARI00
99.9/99.9) 99.9
95.4 95.2 95.2
DenseNctBC100
99.9 99.0 99.9
77.5/ 77.5 76.8
SVHN
96.9 9..6/9..6/99 99.1
96.6 96.6 96.6
CIFAR 10 CIFARIO0 SVHN
99.9 99.9
94.5/ 94.6 94.6
ResNerl 10
99.5 99.9 99.8
72.7 / 74.1 73.9
99.8 / 99.9 99.5
96.7 / 96.9 96.9
111
However, while the distances in the original IsoMax loss may vary from zero infinity, the distance between two normalized vectors is always equal to or lower than two. To avoid this unjustifiable and unreasonable restriction, we introduce the distance scale dg, which scalar learnable parameter. Naturally, we require the distance scale to always be positive by taking its absolute value [d,]-
112
118
114
115
The feature normalization makes the solution isometric regardless of the norm of the features produced by the examples. The distance scale is class independent, as is a single scalar value regularly learnable during training. The weight normalization and the class independence of the distance scale make the solution isometric regarding all classes. Hence, the proposed distance is isometric because produces an isometric treatment of all features, prototypes, and classes. Therefore, we can write the expression for the isometric distances used by the IsoMaxy loss as:
116
117
118
119
120
Dj |d.] fe( P'll
121
Returning to Equation
① we can write the expression for the IsoMax, loss as follows:
LisoMaxx
1
Σ log
N
exp(- [d,l fo(x)) p/1
(3)
N
Σ Exxp(-E, |dsl llfo(z)) p'll)
Applying the entropy maximization trick (i.e., the removal of the entropic score E, for inference) 27
123
we can write the expression for the IsoMax, loss probabilities used during inference for performing
OOD detection when using the entropic score (27
PisoMaxz (z)))) exp(- [dal fo(x) 1l
(4)
Σ exp(- [d,l fo( -p',l)
125
Different from IsoMax loss where the prototypes are initialized to zero vector, we initialized all prototypes using a normal distribution with a mean zero and standard deviation of one. This approach is necessary because we normalize the prototypes when using IsoMaxz loss. The distance scale is initialized to one. We add no hyperparameters to the solution.
126
128
129
Minimum Distance Score Motivated by the desired characteristics of the isometric distances used IsoMaxz, we propose to use what we call the minimum distance as the score for performing OOD detection, Naturally, the minimum distance score for the IsoMaxz is given by:
130
SMinDistance mijn [lFo((z)) -721
[1fo(())-p'a))
(5)

---

Table 3: Fair comparison of seamless approaches: No hyperparameter tuning, addi- tional/extra/outier data, no classification accuracy drop, and no slow/inefficien inferences. SoftMax+ES means training using SoftMax loss and performing OOD detection using the entropic score (ES). IsoMax+ES means training using IsoMax loss and performing OOD detection using the entropic score (ES). MsDM means training using IsoMaxz loss and performing OOD detection using minimum distance score (MDS). The best results are in bold (0.5% tolerance).
Out-of- Co-oo--Ditributionn Detection: Seamless Approaches.
Data (training)
OOD (unseen)
Model
TNR@ @TRY5 (%) [t]
AUROC (%) [t]
SoftMax+ES IsoMax+ES IsoMaxz+MDS (ours)
SVHN
33.2/ 97.2
86.9 96.6 99.5
CIFARIO
TinyImageNet
59.8/ 92.5
94.2/97.8 98.6
LSUN
69.5/94.5 95.3
95.9 / 99.1
SVHN
249//33.4 78.6
81.9 88.6 96.5
DenseNetBC100
CIFARIO0
TinyImageNet
23.7 49.1 85.6
78.8 92.6 97.6
LSUN
23.4/63.0/ 83.4
77.9 /9.77 97.4
CIFARIO
83.7 /9.11 /95.3
96.9 99.55 99.1
SVHN
TinylmageNet
90.0/97.0/ 98.3
98.1 / 99.1 99.77
LSUN
88.4 96.8 97.88
97.8/ 99.77
SVHN
37.8 / 73.0/ 83,6
89.6 / 95.1 97.3
CIFARIO
TinylmageNet
43.7 73.7 75.5
90.6 / 95.9 96.0
LSUN
52.1 82.8 86.3
92.8 / 96.9 97.7
SVHN
15.4 18.7 30.7
67.5 84.7 85.8
ResNetl 10
CIFAR100
TinyImageNet
18.8 26.3 42.9
73.5 84.5 87.9
LSUN
21.3/30.22 46.9
76.4 / 87.1 89.4
CIFARIO
68.6 80.4 72.0
91.7 / 93.3
SVHN
TinyImageNet
71.7 84.4 83.1
93.1 / 95.8 96.2
LSUN
69.1 / 80.4/ 76.3
91.8 / 94.3 94.3
132 In the previous equation, |d,| was removed because it is scale factor that does not change after the training completed. The minimum distance is computed to perform the classification, as the predicted class is the one that presents the lowest powestfeature prototype distance. Therefore, when using this score, the OOD detection presents essentially zero latency and computational cost, as we simply reuse the minimum distance already calculated.
133
134
135
3 Experiments
137
138
To allow standardized comparison, we used the datasets, training procedures, and metrics that were established in Hendrycks & Gimpel [9] and adopted in many subsequent OOD detection papers
140
We did not compare to approaches that produce classification accuracy drop (e.g., this is substantial limitation from practical perspective [ The code reproduce the results is available as supplementary material.
23
g
34
142
We trained many 100-layer DenseNetBCss with growth rate A 12 (i.e., 0.8M parameters) [12]. 10-leyer ResNets 3 and 34-layer ResNets 7 on CIFARIO [55, CIFARI00 15), and SVHN 29 datasets with SoftMax, IsoMax, and IsoMax losses using the same procedures (e.g., initial learning rate, learning rate schedule, weight decay) presented Lee et al. [1)]
144
146
147
We used SGD with the Nesterov moment equal to 0.9 during 300 epochs with batch size of 64, and an initial learning rate of 0.1 with learning rate decay rate equal ten applied the epoch number 150, 200, and 250. The weight decay was 0.0001. We did not use dropout. We used computer with
148
150
CPU Intel 17-4790K, 4.00GHZ, x64, octa-core, 32Gb RAM, and GPU Nvidia GTX 1080 Ti.

---

Table 4: Unfair comparison with approaches that use input preprocessing and produce slow/mneffccient inferences in addition requiring validation using adversarial examples. ODIN and Mahalanobis were applied to models trained using SoftMax loss. These approaches present at least four times slower and less power efficient inferences [27). as they use input prepro- cessing. Their hyperparameters were validated using adversarial examples. IsoMax (ours) means training using IsoMax, loss and performing OOD detection using minimum distance score (MDS). The best results are in bold (0.5% tolerance).
Comparison with approaches that use input preprocessing and adversarial validation.
Data (training)
OOD (unseen)
Model
AUROC (%) [t]
DTACC (%) [t]
ODIN Mahalanobis MsoMaxz+MDSS (ours)
SVHN
92.8 / 97.6 99.5
86.5 / 92.6 96.3
CIFARIO
TinylmageNet
97.2 / 98.8 98.6
92.1 95.0 / 93.9
LSUN
98.5 99.2 99.1
94.3 / 96.2 99.22
DenseNeiBC100
SVHN
88.2 91.8/ 96.5
80.7 / 84,6 90.0
CIFARI00
TinyImageNet
85.3 /7.0 97.6
77.2/ 91.8 91.6
LSUN
85.7 /9..9 97.4
77.3 / 93.8 / 90.88
SVHN
86.5 /95.5 98.2
77.8 / 89.1 93.0
CIFARIO
TinylmageNet
93.9 /99.0 94.8
86.0 / 95.4 88.5
LSUN
93.7 199.5 96.6
85.8 / 97.2 91.0
ResNet34
SVHN
72.0 84.4 88.3
67.7 / 76.5 82.6
CIFARI00
TinylmageNet
83.6 / 87.9 90.5
75.9 84.6 84.4
L.SN
81.9 82.3 88.3
74.6/ / 79.7 82.6
Table 5: Unfair comparison of outlier exposure-enhanced SoftMax loss with IsoMax loss and IsoMaxy loss without using extra data. SoftMax means training using SoftMax loss hanced during training by using outlier exposure [1O], which requires the collection of outlier data, and performing OOD detection using the entropic
(ES). We used the same outlier data used in 10. In each case, we collected the same amount of outlier data as the number training examples present in the training set used train SoftMax Despite being possible 26), the IsoMax loss and IsoMax loss were not enhanced with outlier exposure to keep the solution seamless. IsoMax+ES means training using IsoMax loss and performing OOD detection using the entropic score (ES). IsoMax (ours) means training using IsoMax, loss and performing OOD detection using minimum distance score (MDS). The values of the performance metrics TNR @TP995 and AUROC were averaged over all disti---isttiiutton. The best values are in bold (0.5% tolerance).
Comparison IsoMax loss variants without using extra data outlier exposure-enhanced SoftMax loss.
Data (training)
Model
TNR@ (%) [t] SoftMax 4ES IsoMax+ES IsoMax; (ours)
AUROC (5)) [+]
CIFARI0
93.8 / 84.1 95.0
98.5 97.3 99.1
DenseNetBC100
CIFAR100
23.0 45.1 82.5
80.5 91.9 97.0
CIFAR10
92.6/ 76.5 81.8
98.0 96.0 97.0
ResNetl10 10
CIFAR100
36.1 25.11 440.2
83.2/ 85.5 87.7

---

DISTRIDUTION-CCTTAR NODDL=DenseNNeL LOSS=bbaMex
out-at-distnn ou2-af-distmbuttio:: SVHN
out-ar-distre aistataiistibuttionn linylimgeetet
out-of-distibutoon LSUN
1.25
1.5
1,0
1.0
0.50
0:25
0.00
0.0
0.0
Minimum Distance
Minimum Distance
Minimum Distance
distribution
(a)
IN DISTRIBUTIONNCIFARI OODEL=DenseNNe. LOSS=lsuMakr out-of-ditribution TinyliageNeet
out-of-distribution: SVHN
out-of-distributiin: LSUN
1.25
1.5
1.00
1.9
0.75
0.50
0.00
0.0
4
G
Minimum Distance
Minimum Distance
Minimum Distance
in distribbuuee butionbtioon
out of distribution
(b)
Figure 1: (a) The no isometric distances used by the IsoMax loss make detecting out-of-distribution examples difficult using the minimum distance score. Consequently. the minimum distance score is not competitive with the entropic score in this case. (b) The isometric distances used by the IsoMaxz loss make detecting out-of-distribution examples easy using the minimum distance score. Consequently, the minimum distance score usually overcomes the entropic score in this situation.
We used resized images from the datasets TinylmageNet 2 and the Large-scale Scene UNderstand- ing dataset (LSUN) [6 following Lee et al. [19] to create out-of-distributee samples. We added these out-of-distree distiiiiibtttion images the validation sets presented in the CIFARIO, CIFAR100, and SVHN to form the test sets and evaluate the OOD detection performance.
152
154
We evaluated the OOD detection performance using the true negative rate at 95% true positive rate (TNR @ TPR95), the area under the receiver operating characteristic curve (AUROC), and the detection accuracy (DTACC), which corresponds to the maximum classification probability over all possible thresholds 8:
156
158
min ( Pmn (o (x) < 8) P is from Pio) + Pout (o (x) 8) P (x is from Pout) },
159
where (xx) is the OOD detection score. It is assumed that both positive and negative samples have equal probability of being in the test set, i.e., P (x is from P.a) P (xx from Pout). All the mentioned metrics follow the calculation procedures specified in Lee et al. 19
181
Results and Discussion
162
Classification Accuracy Table2presentt the classification accuracy results. It shows that IsoMaxy loss does not present classification accuracy drop compared to SoftMax loss or IsoMax loss for all datasets and models. We observe that the IsoMax loss variants present more than percent (% I) better accuracy than the SoftMax loss when using ResNetl 10 on the CIFAR 100 dataset.
184
186
Out-of-Distribution Detection We report the results using the entropic score for SoftMax loss (S)ttMax+ESS, outlier exposure-enhanced SoftMax loss (SoftMax +ES), and IsoMax loss (Iso- Max+ES) because it always overcame the maximum probability score and minimum distance score in
168

---

170
these cases. For IsoMaxz. we report the values using the minimum distance score (MsDM)x as it usually overeame the maximum probability and the entropic score in this situation.
171
172
The Table summarizes the results of the fair OOD detection comparison. In the mentioned table, all approaches are accurate (no classification accuracy drop), fast and power-efficient (inferences are performed without input preprocessing), and no validation is required to define hyperparameters. Additionally, no additionnallextta/oullier data are needed. In most cases, IsoMaxz+MDS overcomes IsOMa++ES performance, regardless of the model, dataset, and cuu-of-distribution.
173
174
175
178
The minimum distance score produces high OOD detection performance when combined with the IsoMaxr, which evidences that the isometrization of the distances indeed work in this case. However, the same minimum distance score produced low OOD detection performance when combined with the original IsoMax loss. The Fig. I provides an explanation for this fact.
178
180
181
Table summariizes the results of an unfair OOD detection comparison, as the methods present differ- ent requirements and produce distinet side effects. ODIN [33 and the Mahalanobis [19] approaches require adversarial samples to be generated to validate hyperparameters for each combination of dataset and model. Moreover, these approaches use imput preprocessing. which makes inferences at least four times slower and at least four times less energy-efficient. Validation using adversarial examples may be a cumbersome procedure to be performed from scratch on novel datasets, as hyper- parameters such as optimal adversarial perturbations may be unknown in such cases. MsoMax does not present these special requirements and does not produce the mentioned side effects.
182
184
186
188
189
Nevertheless, IsoMax MDS provides higher performance than ODIN. Usually, this occurs by large margin. In addition to the changes between the entropy maximization trick and temperature calibrations present in (27 [6].. we emphasize that training with entropic scale affects the learning of all weights while changing the temperature during inference affects only the last layer. Hence, the fact that the proposed solution overcomes ODIN by safe margin is additional evidence that the entropy maximization trick often produces much higher OOD detection performance than temperature cali- bration, even when the latter is combined with input preprocessing. Besides, the entropy maxinization trick does not require access to validation data to tune the temperature. In addition to being seamless and avoiding the Mahalanobis approach drawbacks, MMDS usually overcomes it in terms of AUROC and produces similar performance when considering the DTTACC
190
192
194
195
196
198
199
Table 5 unfairly compares the performance of the proposed approach with the outlier exposure solution. Similar to IsoMax variants, the outlier exposure approach does not require hyperparameters tunig and produces efficient inferences. However, it requires collecting outlier data, while our approach does not. is important to emphasize that outlier exposure may also be combined with IsoMax loss variants to increase the OOD detection performance further [26]. Nevertheless, in the mentioned table, we preferred to present the IsoMax loss variants without outlier exposure to show that the outlier exposure ennaanced SoftMax loss usually present lower OOD detection than IsoMax even without using outlier exposure.
200
202
204
206
207
5
Conclusion
208
In this paper, we improved the IsoMax loss by replacing its original distance with what we call the isometric distance. Additionally, we proposed a zero computational cost minimum distance score. The experiments showed that these modifications produce higher OOD detection performance while keeping desired benefits of IsoMax loss (absence of hyperparameters to tune, no reliance on additiooalleettrr/outiier data, fast and power-efficient inference, and no classification accuracy drop).
209
210
211
212
213
Similar to IsoMax loss, after training using the proposed IsoMaxz loss, we may apply inference- approaches (e.g., Gram matrices, outlier exposure, energy -ased) to the pretrained model to eventually increase even more the overall OOD detection performance. Therefore, instead of competitors, the OOD detection approaches that may be applied pretrained models are actually complementary to our approach (27 26 Hence. there is no drawback in training a model using IsoMax, loss instead of SoftMax loss or IsoMax loss, regardless of planning to subsequently use an inference-based OOD detection approach to increase the OOD detection performance further.
215
216
217
218
219
In future works, considering its simplicity, we plan to verify whether our approach scales satisfactorily
221
to lcale-scale image datasets such as ImageNet. We also intend to verify the performance of our solution using text datasets.
8

---

References
223
224
[1] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow,, I. J.,
Madry, A., and Kurakin, A. On evaluating adversarial robustness. CoRR, abs/1902.06705. 2019.
226
227
[2] Deng, J., Dong, W., Socher, R., L., Li, K., and Li, F. ImageNet: A large-scale hicrarchical image database. Computer Vision and Pattern Recognition, 2009.
[3] DeVries, T. and Taylor, G. W. Learning confidence for out-of-distribution detection in neural networks. CoRR, abs/ 180.00465, 2018.
230
231
[4] DeVries, T. and Taylor, G. W. Leveraging uncertainty estimates for predicting segmentation quality, CoRR, abs/ htd//807.000 005000 2018.
[5] Dhamija, A. R.., Günther, M., and Boult, T. E. Reducing network agnostophobia. Neural Information Processing Systems, 2018.
234
235
[6] Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, 2016.
[7] He, K., Zhang. X., Ren, S., and Sun, J. Identity mappings in deep residual networks. European Conference on Computer Vision, 2016.
238
239
[8] Hein, M., Andriushchenko, M., and Bitterwolf, J. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. Computer Vision and Pattern Recognition, 2018.
241
242
99] Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and dit-of-iistribution examples in neural networks. International Conference on Learning Representationss, 2017.
[10] Hendrycks, D., Mazeika, M., and Dietterich, T. Deep anomaly detection with outlier exposure. International Conference on Learning Representations, 2019.
245
246
[11] Hsu, Y.-C., Shen, Y., Jin, H., and Kira, Z. Generalized ODIN: Detecting out-of-distribution image without learning from out-of-distribution data. Computer Vision and Pattern Recognition, 2020.
248
249
[12] Huang, G., Liu, Z., Maaten, L. v. d., and Weinberger, K. Q. Densely connected convolutional networks. Computer Vision and Pattern Recognition, 2017.
[13] Kendall, A. and Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision? Neural Information Processing Systems, 2017.
252
253
[14] Kliger, M. and Fleishman, S. Novelty detection with GAN. CoRR, abs/1802.10060, 2018
254
[15] Krizhevsky, A. Learning multiple layers of features from tiny images. Science Department, University of Toronto, 2009.
[16] Kuleshov, V., Fenner, N., and Ermon, S. Accurate uncertainties for deep learning using calibrated regression. International Conference on Machine Learning, 2018
257
258
[17] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Neural Information Processing Systems, 2017.
[18] Lee, K., Lee, H., Lee, K., and Shin, J. Training confidence-calibratee classifiers for detecting out-of-distribution samples. International Conference on Learning Representations, 2018
281
262
[19] Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. Neural Information Processing Systems, 2018.
[20] Leibig, C., Allken, V., Ayhan, M. S., Berens, P., and Wahl, S. Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7, 2017.
265
286 [21] Li, D., Chen, D., Goh, J., and Ng. S. Anomaly detection with generative adversarial networks for multivariate time series. CoRR,
2018.

---

268
[22] Liang, S., Li. Y., and Srikant, R. Principled detection of out-of- off-distribution examples in neural networks. CoRR, abs/1706.02690, 2017.
269
270
[23]
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. International Conference Learning Representations, 2018.
[24]
Liu, W., Wen, Y., Yu, Z.., and Yang, M. Large-margin softmax loss for convolutional neural networks. International Conference on Machine Learning, 2016.
273
274
[25] Liu, W. Wang, X., Owens, J. D., and Li, Y. Energy-based out-of-distribution detection. Neural
275
Information Processing Systems, 2020.
278
[26] Macêdo, D., Ren, I., Zanchettin, C., Oliveira, I., and Ludermir, B. Entropic out-of- distribution detection: Seamless detection of unknown examples. CoRR, abs/2006.04005, 2021. http://wwww..pp///////./oooooo 04005
277
278
URI.
279
[27] Macêdo, D., Ren, T. I., Zanchettin, C., Oliveira, I., and Ludermir, B. Entropic out- of-distribution detection. Accepted for publication in The International Joint Conference Neural Networks (ICNNN,, 2021. URL batppss:///axxi org.ooog//a....... htggabbs19908 1abbb//990.05699
281
282
[88] Malinin, A. and Gales, M. Predictive uncertainty estimation via prior networks. Neural
283
Information Processing Systems, 2018.
[29] Netzer, Y. and Wang. T. Reading digits in natural images with unsupervised feature learning. Neural Information Processing Systems, 2011.
285
[30] Sastry, C. S. and Oore, S. Detecting cat-of-distribution examples with gram matrices. Interna- tional Conference on Machine Learning. 2020.
287
286
[3]] Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, Green Al. Communications of the ACM, 63(12):54-63 63, 2020.
[32] Shafaei, A., Schmidt, M., and Little, J. less biased evaluation of outtof-distribution sample detectors. Brilish Machine Vision Conference, 2019.
281
292
[33] Subramanya, A., Srinivas, S., and Babu, R. V. Confidence estimation in deep neural networks via density modelling. CoRR, abs/1707.07011 2017.
293
294
[34] Techapanurak, E.,, Suganuma, M., and Okatani, T. Hyperparameter--free disti--distiibutiin detection using cosine similarity. Asian Conference on Computer Vision (ACCV), November 2020.
295
296
297
[35] Vyas, A., Jammalamadaka, N., Zhu, X., Das, D., Kaul, B., and Willke, T. L. dustrf-diittibutiin detection using ensemble of self supervised lcave-out classifiers. European Conference Computer Vision, 2018.
299
300
[36] Yu, F., Zhang, Y., Song, S., Seff, A., and Xiao, J. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.0336, 2015.
302
[37] Yu, Q. and Aizawa, K. Unsupervised out-of- distriibbit detection by maximum classifier discrepancy. International Conference on Computer Vision, 2019.
303
10

---

Checklist
304
305
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] All claims are demonstrated using argumentation and substantial experiments.
307
309
(b) Did you describe the limitations of your work? [Yes] Please, see the last paragraph of the conclusion.
310
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Actu- ally, our approach is much more energy-effciien and environment-riendlly than most competing approaches (see the third and the fifth paragraphs of the introduction).
312
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
315
316
2. If you are including theoretical results...
317
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
318
(b) Did you include complete proofs of all theoretical results? [N/A]
319
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]
321
322
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes]
323
324
(c) Did you report error bars (e.g.. with respect to the random seed after running experi-
ments multiple times)? [N/A] We applied tolerance to indicate the best approaches.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
327
328
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
329
(a) If your work uses existing assets, did you eite the creators? [Yes]
330
(b) Did you mention the license of the assets? [Yes]
331
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
332
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
333
334
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
336
5. If you used crowdsourcing or conducted research with human subjects...
337
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
339
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
340
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
342
11


Reviews:
[{"review_id": "Zv8vesmKjan", "reviewer": "Reviewer_J9aE", "summary": "The work proposes a method to improve the entropic out-of-distribution detection method which comprises the IsoMax loss for training and the entropic score for out-of-distribution detection. The IsoMax method defines the probability (also called entropic score, ES) based on the softmax of the distance from the feature map f(x) to the learnable prototype associated with each class \\mu_j, i.e. p(y=k|x) = exp(-|f(x)-\\mu_k|)/\\sum_j exp(-|f(x)-\\mu_j|). The authors observed that since the feature map f(x) are unnormalized, examples with low norms are unjustifiably favored to be considered OOD examples. So they propose to use the normalized feature map f’(x)=f(x)/|f(x)| to fix it. The method is called IsoMax_I. They also propose to use the minimum distance score min_j |f’(x)-\\mu'_j| as the OOD score instead of using the maximum probability max p(y|x).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality: the work proposes a simple but effective fix to the original IsoMax loss.\n\n- Quality: the paper is written well and easy to follow. There are a few comments:\n1. The experiments compare the performance among the regular softmax loss, IsoMax+ES and IsoMax_I+MDS (the proposed). It would be better to show the performance of IsoMax+MDS and IsoMax_I+ES, so that the contribution made by the two proposed components, IsoMax_I and MDS, can be disentangled. Similarly, in Figure 1, it would be great to show similar histograms for IsoMax + MDS and IsoMaxI+ES. \n2. The proposed IsoMax_I loss involves a hyper-parameter called distance scale d_s. How do you tune d_s? Do you need extra validation data for tuning?\n\n- Clarity: in general the paper's text, figures and tables are presented clearly. There are a few comments: \n1. The authors mentioned that Mahalanobis method (Table 1) requires hyperparameter tuning and requires adversarial samples (line 183). In my understanding, the vanilla Mahalanobis method fits class conditional Gaussian distributions based on the feature map, which does not involve hyperparameters or adversarial samples.\n2. In Table 5, it is unknown which OOD datasets are the results for.\n\n- Significance: the results are important. \n\n\n***Update: Thank the authors for your response to my questions and your efforts on providing more results. Overall the method is simple and effective. I increase my score from 5 to 6. Please include the updated figures and results into the final version. Thank you. ***", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "R6TyBGWil38", "reviewer": "Reviewer_gbXF", "summary": "The authors propose some marginal modifications to the IsoMax loss (i.e. normalising the logits and the prototypes, multiplying the distance between the logits and the prototypes by a non-negative scalar, using a different uncertainty score based on the distance from the prototypes). \n\nPositive aspects:\n- the formulation makes sense, and the distance score is simple but reasonable\n\nUnclear points:\n- why is it necessary to initialize the prototypes randomly? Other papers like [1] show one can select the prototypes to make the training procedure stable.\n- why should multiplying by a fixed scalar |d_s| (learned only at training time, but not depending on the input) have any effect on the performance?\nWeaknesses:\n- Novelty is marginal\n- Experiments are performed with only one seed (need to repeat the experiments with at least 5 different seeds) \n- Does not compare against baselines with similar concepts [1, 2], and makes unfair comparisons with methods that leverage more information/have accuracy drops; recent literature has shown there are many ways to preserve accuracy and improve OOD detection [3,4,5] that would make fair comparison baselines\n- Uncertainty estimation is more than just out of distribution detection: it would be good for the authors to show calibration performance, and evaluate accuracy and calibration on corrupted datasets (e.g. CIFAR-10/100-C) [6].\n- The clustering properties of the proposed method are not analysed in depth\n- Given the few and statistically unreliable experiments presented, at least showing the method could scale to ImageNet would have been interesting\n\n[1] Class Anchor Clustering: a Loss for Distance-based Open Set Recognition, Dimity Miller, Niko Sünderhauf, Michael Milford, Feras Dayoub\n[2] Uncertainty Estimation Using a Single Deep Deterministic Neural Network, Joost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal\n[3]Training independent subnetworks for robust prediction, Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, Dustin Tran\n[4] BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning, Yeming Wen, Dustin Tran, Jimmy Ba\n[5] Fast Predictive Uncertainty for Classification with Bayesian Deep Networks, Marius Hobbhahn, Agustinus Kristiadi, Philipp Hennig\n[6] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, Dan Hendrycks, Thomas Dietterich", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the paper is incremental and does not introduce particularly innovative concepts or methodologies. It just slightly modifies the IsoMax technique. \n\nClarity: the paper is clearly written and organised. The extensive presence of allegedly unfair comparison does not make much sense to me, especially considering the experiments have been performed without averaging over (at least) 5 seeds, and many times the differences in these cases marginal or the method is underperforming. There is also no reason to break the presentation of the result for one type of experiments in so many tables. \n\nSignificance: The paper does not show statistically significant results, and the results seem to be mixed in many cases, also based on which metric is used. The choice of the baselines is questionable and neglects most of the recent literature (I cited just a few of the most recent techniques). \n\nQuality: the paper is well written, but the experiments are not statistically significant nor extensive enough for acceptance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "I4M0gV-AYod", "reviewer": "Reviewer_LMfd", "summary": "This work improves \"IsoMax loss\" by using normalized feature vectors (and a learned scalar multiplier) when training a CNN for classification. The paper situation the new loss function in the Out-of-Distribution literature an focuses on fast inference time without needing input preprocessing.  The results generally show the IsoMax_I loss out performs similar methods.", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe main proposal of this work is to improve IsoMax loss by normalizing it's prediction vectors and multiplying them by a single learned parameter. This is a very minor change over IsoMax. Various distance metrics have been used for OOD, and the proposed S_{MinDistance} is very straight-forward as it's the exact metric being optimized. \n\nClarity: This paper well describes the OOD problem and in what situations the new proposed loss function is suitable for use.  The title is disingenuous as the method is a change to the final layer of a neural net, while the loss function is still LogLoss (or CrossEntropy ).\n\nQuality: The evaluation procedure is generally okay for the proposed method. There is a significant amount of text/format overlap with the prior work IsoMax Loss.\n\nUnfortunately, comparison to state-of-the-art is lacking. Specifically, Generalized-Odin [11] often outperforms the proposed method even without input preprocessing (see table 7). Additionally, Generalized-Odin is also a similar change to the final layer of a neural net. \n\nIt's not clear in what situation IsoMax_I should be used instead of Generalized-Odin.\n\nPost rebuttal update:\n\nWhile I appreciate the authors very detailed response, my primary issues have not been addressed: novelty and comparison to SOTA. The proposed work (a change to the final layer of a CNN) is not of enough novelty to be of interest to the broader community. Some valid critiques against my recommended SOTA (generalized-odin) were given, but failed to address the primary difference in OOD score (using a \"fair\" comparison). Disappointingly, the results discussed in the comments are cherry-picked by architecture. I've run G-odin myself and get better results than reported in this paper (without input preprocessing/hyper-param tuning). Lastly, there's significantly more datasets used in OOD papers in the last year or 2, and it's not clear if cifar10/cifar100 results generalize to harder datasets.\n\nFor these reasons I will keep my initial score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is missing the broader impact section, but there is discussion throughout on reducing energy use.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "AQAd2om9keQ", "reviewer": "Reviewer_M1RX", "summary": "This paper performs isometrization of the distances in the IsoMax loss and replaces entropic score with the minimum distance score for  entropic out-of-distribution detection solution.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is simple and show effective in experiments.\nThe authors also provide insights into why their methods work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the overall method is simple and incremental, the novelty is limited.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The extensive discussion here seems to have resulted in some substantial improvements to your paper, but there remain some concerns about the thoroughness of the experimental results (in particular with respect to the choice of architecture and of datasets) and clarity of the discussion. Some reviewers complain that the changes in your proposal are too simple; I want to emphasize that I do _not_ think this is a good reason to reject a paper (and indeed have fought to accept "too simple" papers in the past), but that "simple" methods do tend to come with a higher bar for clarity and thoroughness of the presentation. The changes brought up in this discussion were also of somewhat significant scope. Thus, it seems best if you integrate the results of the discussion here into a revision to be reviewed at another venue.
Decision: Reject