Improving Entropic Out-of-Distribution Detection using Isometric Distances and the Minimum Distance Score
Year: 2021

Abstract:
Current out-of-distribution detection approaches usually present special require1 ments (e.g., collecting outlier data and hyperparameter validation) and produce 2 side effects (classification accuracy drop and slow/inefficient inferences). Recently, 3 entropic out-of-distribution detection has been proposed as a seamless approach 4 (i.e., a solution that avoids all the previously mentioned drawbacks). The entropic 5 out-of-distribution detection solution comprises the IsoMax loss for training and 6 the entropic score for out-of-distribution detection. The IsoMax loss works as a 7 SoftMax loss drop-in replacement because swapping the SoftMax loss with the 8 IsoMax loss requires no changes in the model’s architecture or training proce9 dures/hyperparameters. In this paper, we propose to perform what we call an 10 isometrization of the distances used in the IsoMax loss. Additionally, we propose 11 to replace the entropic score with the minimum distance score. Our experiments 12 showed that these simple modifications increase out-of-distribution detection per13 formance while keeping the solution seamless. 14

Current out-of-distribution detection approaches usually present special require-1 ments (e.g., collecting outlier data and hyperparameter validation) and produce2 side effects (classification accuracy drop and slow/inefficient inferences). Recently,3 entropic out-of-distribution detection has been proposed as a seamless approach4 (i.e., a solution that avoids all the previously mentioned drawbacks). The entropic5 out-of-distribution detection solution comprises the IsoMax loss for training and6 the entropic score for out-of-distribution detection. The IsoMax loss works as a7 SoftMax loss drop-in replacement because swapping the SoftMax loss with the8 IsoMax loss requires no changes in the model’s architecture or training proce-9 dures/hyperparameters. In this paper, we propose to perform what we call an10 isometrization of the distances used in the IsoMax loss. Additionally, we propose11 to replace the entropic score with the minimum distance score. Our experiments12 showed that these simple modifications increase out-of-distribution detection per-13 formance while keeping the solution seamless.14
1 Introduction15
Neural networks have been used in classification tasks in many real-world applications [4]. In such16 cases, the system usually needs to be able to identify whether a given input belongs to any of the17 classes on which it was trained. Hendrycks & Gimpel [9] called this capability out-of-distribution18 (OOD) detection and proposed datasets and metrics to allow standardized performance evaluation19 and comparison. However, current OOD detection solutions still present limitations (e.g., special20 requirements and side effects) that prevent a more general use of OOD detection capabilities in21 practical real-world applications [27] (Table 1).22
First, OOD detection solutions commonly present hyperparameters that usually presume access to23 out-of-distribution samples to be defined [23, 22, 19, 18, 3]. A consequence of presuming access to24 OOD samples to validate hyperparameters and using the same distribution to evaluate OOD detection25 results is producing overestimated performance estimations [32]. To avoid unrealistic access to OOD26 samples and overestimated performance, Lee et al. [19] proposed to validate hyperparameters using27 adversarial samples. However, this requires the generation of adversarial examples. Moreover, this28 procedure requires the determination of hyperparameters (e.g., maximum adversarial perturbation)29 typically unknown when dealing with novel datasets. Similar arguments hold for solutions based on30 adversarial training [8, 17, 21, 14, 18], which also result in higher training time. Approaches based31 on the generation of adversarial examples or the use of adversarial training may also have limited32 scalability when dealing with large images such as those presented in the ImageNet [2].33
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
Many solutions make use of the so-called input preprocessing technique introduced in ODIN [23].34 However, the use of the mentioned technique increases at least four times the inference delay and35 power consumption [27] since a combination of a first forward pass, backpropagation operation, and36 second forward pass is required [23, 19, 11, 3] for a single useful inference. Actually, approaches37 that may be applied directly to pretrained models and altogether avoid training or fine-tuning the38 model [23, 19, 30] usually produce inefficient inferences and/or additional computational complexity39 to perform OOD detection [26, Section IV, D]. From a practical point of view, this is a drawback, as40 inferences may be performed thousands or millions of times in the field. Hence, such approaches may41 be prohibitive (not sustainable) from environmental [31]1 and real-world cost-based perspectives.42
Another harmful common side effect is the so-called classification accuracy drop2 [34, 11]. In such43 cases, higher OOD detection performance is achieved at the expense of a drop in the classification44 accuracy compared with models trained using the usual SoftMax loss (i.e., the combination of the45 SoftMax activation and the cross-entropy loss [24]). From a practical perspective, this situation is46 undesired because the detection of out-of-distribution samples may be a rare event. At the same time,47 the classification is the main aim of the designed system [1].48
Hsu et al. [11] proposed to use the in-distribution validation set to avoid the need for accessing49 OOD samples to determine the hyperparameters required by the solution. However, considering that50 CIFAR10 and CIFAR100 do not have separated sets for validation and testing, the results may also be51 overestimated because the validation sets used to define the hyperparameters were reused for OOD52 detection performance estimation. A more realistic OOD detection performance estimation could53 have been achieved by removing the in-distribution validation set from in-distribution training data.54 However, this would probably produce an even higher classification accuracy drop. Additionally, the55 solution proposed in [11] is expensive and not environment-friendly, as it uses input preprocessing56 and, consequently, produces slow and energy-inefficient inferences [27, 26]. Recently, many OOD57 detection approaches have used additional/extra/outlier data [10, 25, 5]. The Gram matrices solution58 calculates values produced by the model during inference [30] to perform OOD detection.59
In some cases, an ensemble of classifiers is used [35]. For deep ensembles, Lakshminarayanan60 et al. [17] proposed an ensemble of same-architecture models trained with different random initial61 weights. Some proposals required model structural changes to tackle OOD detection [37], and62 certain trials used uncertainty or confidence estimation/calibration techniques [13, 20, 28, 16, 33].63 However, Bayesian neural networks used in most of these are usually harder to implement and require64
1https://www.youtube.com/watch?v=KnOpWgUCtaM 2In this paper, we consider that an approach does not present classification accuracy drop if it always presents
a classification accuracy higher or less than one percent (1%) lower than SoftMax-loss-trained models.
much more computational resorces to train. Moreover, computational constraints usually require65 approximations that compromise the performance, which is also affected by the prior distribution66 used [17]. For example, MC-dropout uses pretrained models with dropout activated during the test67 time. An average of many inferences is used to perform a single decision [6].68
The entropic out-of-distribution detection approach, which is composed of the IsoMax loss for training69 and the entropic score for OOD detection, avoids all mentioned special requirements and side effects70 [27]. Indeed, no hyperparameter tuning is required because the entropic scale is a global constant71 kept equal to ten for all combinations of datasets and models. Even if we call the entropic scale a72 “hyperparameter”, the IsoMax does not involve hyperparameter tuning, as the same constant value of73 entropic scale is used in all situations. This is possible because Macêdo et al. experimentally showed74 in [27, Fig. 3] and in [26, Section IV, A] that the OOD detection performance presents a well-behaved75 dependence on the entropic scale regardless of the dataset and model. No additional/extra/outlier76 data are necessary. Models trained using IsoMax loss produce inferences as fast and energy-efficient77 as the inferences produced by SoftMax-loss-trained networks. The OOD detection requires only a78 speedy entropy calculation. Finally, no classification accuracy drop is observed.79
Contributions Our contribution in this paper is threefold: First, in addition to minor changes, we80 perform what we call an isometrization of the feature-prototype distances used by the IsoMax loss.81 We call our modified version of IsoMax the isometric isotropy maximization loss or isometric IsoMax82 loss (IsoMaxI loss). Second, we propose to use the minimum feature-prototype distance as the score83 to perform OOD detection. Considering that the minimum feature-prototype distance is calculated84 to perform the classification, the OOD detection task presents essentially zero computational cost85 because we simply reuse this value as the score to perform OOD detection. Third, in addition to86 experimental evidence, we provide insights into why a combination of training using the isometric87 distances provided by IsoMaxI and performing OOD detection using the minimum distance scores88 produces a substantial performance increase in OOD detection compared to IsoMax combined with89 the entropic score. Our approach keeps the solution seamless (i.e., it avoids the previously mentioned90 special requirements and side effects) while significantly increasing the OOD detection performance.91 Similar to IsoMax loss, IsoMaxI works as a SoftMax loss drop-in replacement, as no procedures92 other than regular neural network training are required.93
2 Isometric Distances and Minimum Distance Score94
Isometric Distances Consider an input x applied to a neural network that performs a parametrized95 transformation fθ(x). Moreover, consider p j φ be the learnable prototype associated with the class j.96 Additionally, let the expression ‖fθ(x)−pjφ‖ represent the nonsquared Euclidean distance between97 fθ(x) and p j φ. Finally, consider p k φ as a learnable prototype associated with the correct class for the98 input x. Hence, we write the IsoMax loss [27] for a batch of N examples using the equation below:99
LIsoMax = − 1
N N∑ k=1 log  exp(−Es‖fθ(x)−pkφ‖)∑ j exp(−Es‖fθ(x)−pjφ‖)  (1) In the above equation, the Es represents the entropic scale. From Equation (1), we observe that100 the distances from IsoMax loss are given by the expression D= ‖fθ(x)−pjφ‖. During inference,101 probabilities calculated based on these distances are used to produce the negative entropy, which102 serves as a score to perform OOD detection. However, as the features fθ(x) are unnormalized,103 examples with low norms are unjustifiably favored to be considered OOD examples since they tend104 to produce high entropy. Additionally, as the weights pjφ are unnormalized, examples from classes105 that present prototypes with low norms are unjustifiably favored to be considered OOD examples for106 the same reason.107
Hence, we propose to replace fθ(x) with its normalized version given by f̂θ(x)=fθ(x)/‖fθ(x)‖.108 Additionally, we propose to replace pjφ with its normalized version given by p̂ j φ=p j φ/‖p j φ‖. The109 expression ‖v‖ represents the 2-norm of a given vector v.110
However, while the distances in the original IsoMax loss may vary from zero to infinity, the distance111 between two normalized vectors is always equal to or lower than two. To avoid this unjustifiable and112 unreasonable restriction, we introduce the distance scale ds, which is a scalar learnable parameter.113 Naturally, we require the distance scale to always be positive by taking its absolute value |ds|.114 The feature normalization makes the solution isometric regardless of the norm of the features produced115 by the examples. The distance scale is class independent, as it is a single scalar value regularly116 learnable during training. The weight normalization and the class independence of the distance scale117 make the solution isometric regarding all classes. Hence, the proposed distance is isometric because118 it produces an isometric treatment of all features, prototypes, and classes. Therefore, we can write119 the expression for the isometric distances used by the IsoMaxI loss as:120
DI = |ds| ‖f̂θ(x)−p̂jφ‖ (2)
Returning to Equation (1), we can write the expression for the IsoMaxI loss as follows:121
LIsoMaxI = − 1
N N∑ k=1 log  exp(−Es |ds| ‖f̂θ(x)−p̂kφ‖)∑ j exp(−Es |ds| ‖f̂θ(x)−p̂jφ‖)  (3) Applying the entropy maximization trick (i.e., the removal of the entropic scoreEs for inference) [27],122 we can write the expression for the IsoMaxI loss probabilities used during inference for performing123 OOD detection when using the entropic score [27]:124
PIsoMaxI (y(i)|x) = exp(− |ds| ‖f̂θ(x)−p̂iφ‖)∑ j exp(− |ds| ‖f̂θ(x)−p̂jφ‖) (4)
Different from IsoMax loss where the prototypes are initialized to a zero vector, we initialized all125 prototypes using a normal distribution with a mean of zero and standard deviation of one. This126 approach is necessary because we normalize the prototypes when using IsoMaxI loss. The distance127 scale is initialized to one. We add no hyperparameters to the solution.128
Minimum Distance Score Motivated by the desired characteristics of the isometric distances used129 in IsoMaxI , we propose to use what we call the minimum distance as the score for performing OOD130 detection. Naturally, the minimum distance score for the IsoMaxI is given by:131
SMinDistance=min j
( ‖f̂θ(x)−p̂jφ‖ ) (5)
In the previous equation, |ds| was removed because it is a scale factor that does not change after132 the training is completed. The minimum distance is computed to perform the classification, as the133 predicted class is the one that presents the lowest feature-prototype distance. Therefore, when using134 this score, the OOD detection presents essentially zero latency and computational cost, as we simply135 reuse the minimum distance already calculated.136
3 Experiments137
To allow standardized comparison, we used the datasets, training procedures, and metrics that were138 established in Hendrycks & Gimpel [9] and adopted in many subsequent OOD detection papers139 [23, 19, 8]. We did not compare to approaches that produce classification accuracy drop (e.g.,140 [34, 11]), as this is a substantial limitation from a practical perspective [1]. The code to reproduce the141 results is available as supplementary material.142
We trained many 100-layer DenseNetBCs with growth rate k= 12 (i.e., 0.8M parameters) [12],143 110-layer ResNets [7]3, and 34-layer ResNets [7]4 on CIFAR10 [15], CIFAR100 [15], and SVHN144 [29] datasets with SoftMax, IsoMax, and IsoMaxI losses using the same procedures (e.g., initial145 learning rate, learning rate schedule, weight decay) presented in Lee et al. [19].146
We used SGD with the Nesterov moment equal to 0.9 during 300 epochs with a batch size of 64, and147 an initial learning rate of 0.1 with a learning rate decay rate equal to ten applied in the epoch number148 150, 200, and 250. The weight decay was 0.0001. We did not use dropout. We used a computer with149 CPU Intel i7-4790K, 4.00GHz, x64, octa-core, 32Gb RAM, and a GPU Nvidia GTX 1080 Ti.150
3https://github.com/akamaster/pytorch_resnet_cifar10 4https://github.com/pokaxpoka/deep_Mahalanobis_detector
We used resized images from the datasets TinyImageNet [2]5 and the Large-scale Scene UNderstand-151 ing dataset (LSUN) [36]5 following Lee et al. [19] to create out-of-distribution samples. We added152 these out-of-distribution images to the validation sets presented in the CIFAR10, CIFAR100, and153 SVHN to form the test sets and evaluate the OOD detection performance.154
We evaluated the OOD detection performance using the true negative rate at 95% true positive155 rate (TNR@TPR95), the area under the receiver operating characteristic curve (AUROC), and the156 detection accuracy (DTACC), which corresponds to the maximum classification probability over all157 possible thresholds δ:158
1−min δ
{ Pin (o (x) ≤ δ)P (x is from Pin) + Pout (o (x) > δ)P (x is from Pout) } ,
where o(x) is the OOD detection score. It is assumed that both positive and negative samples have159 an equal probability of being in the test set, i.e., P (x is from Pin) = P (x is from Pout). All the160 mentioned metrics follow the calculation procedures specified in Lee et al. [19].161
4 Results and Discussion162
Classification Accuracy Table 2 presents the classification accuracy results. It shows that IsoMaxI163 loss does not present classification accuracy drop compared to SoftMax loss or IsoMax loss for all164 datasets and models. We observe that the IsoMax loss variants present more than one percent (%1)165 better accuracy than the SoftMax loss when using ResNet110 on the CIFAR100 dataset.166
Out-of-Distribution Detection We report the results using the entropic score for SoftMax loss167 (SoftMax+ES), outlier exposure-enhanced SoftMax loss (SoftMaxOE+ES), and IsoMax loss (Iso-168 Max+ES) because it always overcame the maximum probability score and minimum distance score in169
5https://github.com/facebookresearch/odin
these cases. For IsoMaxI , we report the values using the minimum distance score (IsoMaxI+MDS),170 as it usually overcame the maximum probability and the entropic score in this situation.171
The Table 3 summarizes the results of the fair OOD detection comparison. In the mentioned table,172 all approaches are accurate (no classification accuracy drop), fast and power-efficient (inferences173 are performed without input preprocessing), and no validation is required to define hyperparameters.174 Additionally, no additional/extra/outlier data are needed. In most cases, IsoMaxI+MDS overcomes175 IsoMax+ES performance, regardless of the model, dataset, and out-of-distribution.176
The minimum distance score produces high OOD detection performance when combined with the177 IsoMaxI , which evidences that the isometrization of the distances indeed work in this case. However,178 the same minimum distance score produced low OOD detection performance when combined with179 the original IsoMax loss. The Fig. 1 provides an explanation for this fact.180
Table 4 summarizes the results of an unfair OOD detection comparison, as the methods present differ-181 ent requirements and produce distinct side effects. ODIN [23] and the Mahalanobis [19] approaches182 require adversarial samples to be generated to validate hyperparameters for each combination of183 dataset and model. Moreover, these approaches use input preprocessing, which makes inferences184 at least four times slower and at least four times less energy-efficient. Validation using adversarial185 examples may be a cumbersome procedure to be performed from scratch on novel datasets, as hyper-186 parameters such as optimal adversarial perturbations may be unknown in such cases. IsoMaxI+MDS187 does not present these special requirements and does not produce the mentioned side effects.188
Nevertheless, IsoMaxI+MDS provides higher performance than ODIN. Usually, this occurs by a189 large margin. In addition to the changes between the entropy maximization trick and temperature190 calibrations present in [27, 26], we emphasize that training with entropic scale affects the learning of191 all weights while changing the temperature during inference affects only the last layer. Hence, the fact192 that the proposed solution overcomes ODIN by a safe margin is additional evidence that the entropy193 maximization trick often produces much higher OOD detection performance than temperature cali-194 bration, even when the latter is combined with input preprocessing. Besides, the entropy maximization195 trick does not require access to validation data to tune the temperature. In addition to being seamless196 and avoiding the Mahalanobis approach drawbacks, IsoMaxI+MDS usually overcomes it in terms of197 AUROC and produces similar performance when considering the DTACC.198
Table 5 unfairly compares the performance of the proposed approach with the outlier exposure199 solution. Similar to IsoMax variants, the outlier exposure approach does not require hyperparameters200 tuning and produces efficient inferences. However, it requires collecting outlier data, while our201 approach does not. It is important to emphasize that outlier exposure may also be combined with202 IsoMax loss variants to increase the OOD detection performance further [26]. Nevertheless, in203 the mentioned table, we preferred to present the IsoMax loss variants without outlier exposure to204 show that the outlier exposure-enhanced SoftMax loss usually present lower OOD detection than205 IsoMaxI+MDS even without using outlier exposure.206
5 Conclusion207
In this paper, we improved the IsoMax loss by replacing its original distance with what we call208 the isometric distance. Additionally, we proposed a zero computational cost minimum distance209 score. The experiments showed that these modifications produce higher OOD detection performance210 while keeping desired benefits of IsoMax loss (absence of hyperparameters to tune, no reliance on211 additional/extra/outlier data, fast and power-efficient inference, and no classification accuracy drop).212
Similar to IsoMax loss, after training using the proposed IsoMaxI loss, we may apply inference-based213 approaches (e.g., Gram matrices, outlier exposure, energy-based) to the pretrained model to eventually214 increase even more the overall OOD detection performance. Therefore, instead of competitors, the215 OOD detection approaches that may be applied to pretrained models are actually complementary to216 our approach [27, 26]. Hence, there is no drawback in training a model using IsoMaxI loss instead217 of SoftMax loss or IsoMax loss, regardless of planning to subsequently use an inference-based OOD218 detection approach to increase the OOD detection performance further.219
In future works, considering its simplicity, we plan to verify whether our approach scales satisfactorily220 to large-scale image datasets such as ImageNet. We also intend to verify the performance of our221 solution using text datasets.222
References223 [1] Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I. J.,224 Madry, A., and Kurakin, A. On evaluating adversarial robustness. CoRR, abs/1902.06705,225 2019.226
[2] Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F. ImageNet: A large-scale hierarchical227 image database. Computer Vision and Pattern Recognition, 2009.228
[3] DeVries, T. and Taylor, G. W. Learning confidence for out-of-distribution detection in neural229 networks. CoRR, abs/1802.04865, 2018.230
[4] DeVries, T. and Taylor, G. W. Leveraging uncertainty estimates for predicting segmentation231 quality. CoRR, abs/1807.00502, 2018.232
[5] Dhamija, A. R., Günther, M., and Boult, T. E. Reducing network agnostophobia. Neural233 Information Processing Systems, 2018.234
[6] Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model235 uncertainty in deep learning. International Conference on Machine Learning, 2016.236
[7] He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. European237 Conference on Computer Vision, 2016.238
[8] Hein, M., Andriushchenko, M., and Bitterwolf, J. Why ReLU networks yield high-confidence239 predictions far away from the training data and how to mitigate the problem. Computer Vision240 and Pattern Recognition, 2018.241
[9] Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution242 examples in neural networks. International Conference on Learning Representations, 2017.243
[10] Hendrycks, D., Mazeika, M., and Dietterich, T. Deep anomaly detection with outlier exposure.244 International Conference on Learning Representations, 2019.245
[11] Hsu, Y.-C., Shen, Y., Jin, H., and Kira, Z. Generalized ODIN: Detecting out-of-distribution246 image without learning from out-of-distribution data. Computer Vision and Pattern Recognition,247 2020.248
[12] Huang, G., Liu, Z., Maaten, L. v. d., and Weinberger, K. Q. Densely connected convolutional249 networks. Computer Vision and Pattern Recognition, 2017.250
[13] Kendall, A. and Gal, Y. What uncertainties do we need in bayesian deep learning for computer251 vision? Neural Information Processing Systems, 2017.252
[14] Kliger, M. and Fleishman, S. Novelty detection with GAN. CoRR, abs/1802.10560, 2018.253
[15] Krizhevsky, A. Learning multiple layers of features from tiny images. Science Department,254 University of Toronto, 2009.255
[16] Kuleshov, V., Fenner, N., and Ermon, S. Accurate uncertainties for deep learning using256 calibrated regression. International Conference on Machine Learning, 2018.257
[17] Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty258 estimation using deep ensembles. Neural Information Processing Systems, 2017.259
[18] Lee, K., Lee, H., Lee, K., and Shin, J. Training confidence-calibrated classifiers for detecting260 out-of-distribution samples. International Conference on Learning Representations, 2018.261
[19] Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-262 distribution samples and adversarial attacks. Neural Information Processing Systems, 2018.263
[20] Leibig, C., Allken, V., Ayhan, M. S., Berens, P., and Wahl, S. Leveraging uncertainty information264 from deep neural networks for disease detection. Scientific Reports, 7, 2017.265
[21] Li, D., Chen, D., Goh, J., and Ng, S. Anomaly detection with generative adversarial networks266 for multivariate time series. CoRR, abs/1809.04758, 2018.267
[22] Liang, S., Li, Y., and Srikant, R. Principled detection of out-of-distribution examples in neural268 networks. CoRR, abs/1706.02690, 2017.269
[23] Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection270 in neural networks. International Conference on Learning Representations, 2018.271
[24] Liu, W., Wen, Y., Yu, Z., and Yang, M. Large-margin softmax loss for convolutional neural272 networks. International Conference on Machine Learning, 2016.273
[25] Liu, W., Wang, X., Owens, J. D., and Li, Y. Energy-based out-of-distribution detection. Neural274 Information Processing Systems, 2020.275
[26] Macêdo, D., Ren, T. I., Zanchettin, C., Oliveira, A. L. I., and Ludermir, T. B. Entropic out-of-276 distribution detection: Seamless detection of unknown examples. CoRR, abs/2006.04005, 2021.277 URL https://arxiv.org/abs/2006.04005.278
[27] Macêdo, D., Ren, T. I., Zanchettin, C., Oliveira, A. L. I., and Ludermir, T. B. Entropic out-279 of-distribution detection. Accepted for publication in The International Joint Conference on280 Neural Networks (IJCNN), 2021. URL https://arxiv.org/abs/1908.05569.281
[28] Malinin, A. and Gales, M. Predictive uncertainty estimation via prior networks. Neural282 Information Processing Systems, 2018.283
[29] Netzer, Y. and Wang, T. Reading digits in natural images with unsupervised feature learning.284 Neural Information Processing Systems, 2011.285
[30] Sastry, C. S. and Oore, S. Detecting out-of-distribution examples with gram matrices. Interna-286 tional Conference on Machine Learning, 2020.287
[31] Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green AI. Communications of the ACM,288 63(12):54–63, 2020.289
[32] Shafaei, A., Schmidt, M., and Little, J. J. A less biased evaluation of out-of-distribution sample290 detectors. British Machine Vision Conference, 2019.291
[33] Subramanya, A., Srinivas, S., and Babu, R. V. Confidence estimation in deep neural networks292 via density modelling. CoRR, abs/1707.07013, 2017.293
[34] Techapanurak, E., Suganuma, M., and Okatani, T. Hyperparameter-free out-of-distribution294 detection using cosine similarity. Asian Conference on Computer Vision (ACCV), November295 2020.296
[35] Vyas, A., Jammalamadaka, N., Zhu, X., Das, D., Kaul, B., and Willke, T. L. Out-of-distribution297 detection using an ensemble of self supervised leave-out classifiers. European Conference on298 Computer Vision, 2018.299
[36] Yu, F., Zhang, Y., Song, S., Seff, A., and Xiao, J. LSUN: construction of a large-scale image300 dataset using deep learning with humans in the loop. CoRR, abs/1506.03365, 2015.301
[37] Yu, Q. and Aizawa, K. Unsupervised out-of-distribution detection by maximum classifier302 discrepancy. International Conference on Computer Vision, 2019.303
Checklist304
1. For all authors...305 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s306 contributions and scope? [Yes] All claims are demonstrated using argumentation and307 substantial experiments.308
(b) Did you describe the limitations of your work? [Yes] Please, see the last paragraph of309 the conclusion.310
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Actu-311 ally, our approach is much more energy-efficient and environment-friendly than most312 competing approaches (see the third and the fifth paragraphs of the introduction).313
(d) Have you read the ethics review guidelines and ensured that your paper conforms to314 them? [Yes]315
2. If you are including theoretical results...316 (a) Did you state the full set of assumptions of all theoretical results? [N/A]317 (b) Did you include complete proofs of all theoretical results? [N/A]318
3. If you ran experiments...319 (a) Did you include the code, data, and instructions needed to reproduce the main experi-320 mental results (either in the supplemental material or as a URL)? [Yes]321 (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they322 were chosen)? [Yes]323 (c) Did you report error bars (e.g., with respect to the random seed after running experi-324 ments multiple times)? [N/A] We applied a tolerance to indicate the best approaches.325 (d) Did you include the total amount of compute and the type of resources used (e.g., type326 of GPUs, internal cluster, or cloud provider)? [Yes]327 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...328
(a) If your work uses existing assets, did you cite the creators? [Yes]329 (b) Did you mention the license of the assets? [Yes]330 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]331 (d) Did you discuss whether and how consent was obtained from people whose data you’re332 using/curating? [N/A]333 (e) Did you discuss whether the data you are using/curating contains personally identifiable334 information or offensive content? [N/A]335 5. If you used crowdsourcing or conducted research with human subjects...336
(a) Did you include the full text of instructions given to participants and screenshots, if337 applicable? [N/A]338
(b) Did you describe any potential participant risks, with links to Institutional Review339 Board (IRB) approvals, if applicable? [N/A]340
(c) Did you include the estimated hourly wage paid to participants and the total amount341 spent on participant compensation? [N/A]342

Review summaries:
The work proposes a method to improve the entropic out-of-distribution detection method which comprises the IsoMax loss for training and the entropic score for out-of-distribution detection. The IsoMax method defines the probability (also called entropic score, ES) based on the softmax of the distance from the feature map f(x) to the learnable prototype associated with each class \mu_j, i.e. p(y=k|x) = exp(-|f(x)-\mu_k|)/\sum_j exp(-|f(x)-\mu_j|). The authors observed that since the feature map f(x) are unnormalized, examples with low norms are unjustifiably favored to be considered OOD examples. So they propose to use the normalized feature map f’(x)=f(x)/|f(x)| to fix it. The method is called IsoMax_I. They also propose to use the minimum distance score min_j |f’(x)-\mu'_j| as the OOD score instead of using the maximum probability max p(y|x).

This work improves "IsoMax loss" by using normalized feature vectors (and a learned scalar multiplier) when training a CNN for classification. The paper situation the new loss function in the Out-of-Distribution literature an focuses on fast inference time without needing input preprocessing.  The results generally show the IsoMax_I loss out performs similar methods.

The authors propose some marginal modifications to the IsoMax loss (i.e. normalising the logits and the prototypes, multiplying the distance between the logits and the prototypes by a non-negative scalar, using a different uncertainty score based on the distance from the prototypes).   Positive aspects: - the formulation makes sense, and the distance score is simple but reasonable  Unclear points: - why is it necessary to initialize the prototypes randomly? Other papers like [1] show one can select the prototypes to make the training procedure stable. - why should multiplying by a fixed scalar |d_s| (learned only at training time, but not depending on the input) have any effect on the performance? Weaknesses: - Novelty is marginal - Experiments are performed with only one seed (need to repeat the experiments with at least 5 different seeds)  - Does not compare against baselines with similar concepts [1, 2], and makes unfair comparisons with methods that leverage more information/have accuracy drops; recent literature has shown there are many ways to preserve accuracy and improve OOD detection [3,4,5] that would make fair comparison baselines - Uncertainty estimation is more than just out of distribution detection: it would be good for the authors to show calibration performance, and evaluate accuracy and calibration on corrupted datasets (e.g. CIFAR-10/100-C) [6]. - The clustering properties of the proposed method are not analysed in depth - Given the few and statistically unreliable experiments presented, at least showing the method could scale to ImageNet would have been interesting  [1] Class Anchor Clustering: a Loss for Distance-based Open Set Recognition, Dimity Miller, Niko Sünderhauf, Michael Milford, Feras Dayoub [2] Uncertainty Estimation Using a Single Deep Deterministic Neural Network, Joost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal [3]Training independent subnetworks for robust prediction, Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, Dustin Tran [4] BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning, Yeming Wen, Dustin Tran, Jimmy Ba [5] Fast Predictive Uncertainty for Classification with Bayesian Deep Networks, Marius Hobbhahn, Agustinus Kristiadi, Philipp Hennig [6] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, Dan Hendrycks, Thomas Dietterich