Intra-class Mixup for Out-of-Distribution Detection
Authors: Deepak Ravikumar
Sangamesh Kodge
Isha Garg
Kaushik Roy
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep neural networks have found widespread adoption in solving image recognition and natural language processing tasks. However, they make confident mispredictions when presented with data that does not belong to the training distribution, i.e. out-of-distribution (OoD) samples. Inter-class mixup has been shown to improve model calibration aiding OoD detection. However, we show that both empirical risk minimization and inter-class mixup create large angular spread in latent representation. This reduces the separability of in-distribution data from OoD data. In this paper we propose intra-class mixup supplemented with angular margin to improve OoD detection. Angular margin is the angle between the decision boundary normal and sample representation. We show that intra-class mixup forces the network to learn representations with low angular spread in the latent space. This improves the separability of OoD from in-distribution examples. Our approach when applied to various existing OoD detection techniques shows an improvement of 4.68% and 6.38%  in AUROC  performance over empirical risk minimization and inter-class mixup, respectively. Further, our approach aided with angular margin improves AUROC performance by 7.36% and 9.10%  over empirical risk minimization and inter-class mixup, respectively.

Under review as a conference paper at ICLR 2022
INTRA-CLASS MIXUP FOR OUT-OF--IITTIBUTIIO DETECTION
Anonymous authors Paper under double-bling review
ABSTRACT
Deep neural networks have found widespread adoption in solving image recognition and natural language processing tasks. However, they make confident mispredic- tions when presented with data that does not belong to the training distribution, i.e. out-of dit-o--dittibuttion (OoD) samples. Inter-class mixup has been shown to improve model calibration aiding OoD detection. However, we show that both empirical risk minimization and inter-class mixup create large angular spread in latent representa- tion. This reduces the separability of in-distribution data from OoD data. In this paper propose intra-class mixup supplemented with angular margin to improve OoD detection. Angular margin is the angle between the decision boundary normal and sample representation. We show that intra-class mixup forces the network learn representations with low angular spread in the latent space. This improves the separability of OoD from in-distribution examples. Our approach when applied to various existing OoD detection techniques shows an improvement of 4.68% % and 6.08 % in AUROC performance over empirical risk minimization and inter-class mixup. respectively. Further, our approach aided with angular margin improves AUROC performance by 7.36% % and 9.10 % over empirical risk minimization and inter-class mixup, respectively.
INTRODUCTION
Deep Learning has been employed by many state of the art machine learning models to effectively solve image recogmition (Krizhevsky al., 2012 LeCun al., 2015) and natural language processing (Andor et al. 2016) tasks. Despite their effectiveness, recent research has shown the existence of inputs (Goodfellow et al. 20156 Nguyen et al., 2015) that lead these networks to make confident mispredictions. Two c classes of such inputs have emerged in literature. The first class, adversarial examples (Bruna et al., 2014 Goodfellow et al. 2015a| Kurakin et al. (017). are specifically crafted inputs with the intent of fooling deep neural nets. The second class dubbed Out-of-Distribution (OoD) (Nguyen et al. 2015) examples are examples that do not belong to the underlying true distribution that the training dataset is drawn from. High confidence of the networks on such OoD examples makes it difficult to identify false classifications and poses a challenge to their deployment in safety critical scenarios (Amodei al. 2016), from medical diagnosis self driving applications.
To address this problem several methods have been proposed which aim to identify samples. In the context of deep learning, OoD detection methods be broadly classified into two categories, generative and discriminative methods, Generative methods use various techniques such as Gaussian Discriminant Analysis (Lee al., 2018b), Generative Adversarial Networks (GANs) (Deecke et al., 2018 Lee et al. 2018a] Ren et al. 2211 or more generic generative methods (Wang al. 2017) model the underlying distribution and separate in-distribution samples from OoD samples. On the other hand, discriminative methods (Hendrycks & Gimpel| 2017] Hendryyks et al. 2019 Hssu et al., 20200 Liang al., 20188] [iu et al. (020) detect OoD samples based on measures Hendrycks
such as softmax probability (Hendrycks & Gimpel]2011 2017Heeyy et al.) 2019), energy scores (Liu ctt al. 2020) or other metrics computed from the trained network. Most of these metrics are proxy for the distance of the sample representation from the decision boundary (/2 margin in Figure Ia). Further, we show that existing techniques such as empirical risk minimization (ERM) and inter-class mixup (Zhang et al.) 2018) result in networks learning representations that have large spread. This large spread creates broad overlapping representations for in-distribution and OoD data, making hard to separate in-distribution data from OoD data. Additionally, the use of metrics that proxies

---

Under review as a conference paper at ICLR 2022
for distance from decision boundary have disadvantage when OoD points have similar margin as in-distribution data. For example, consider point "(b) OoD" in Figure where the OoD data point is far from the class representation and has similar l2 margin as in-distribution data; this OoD point will have similar OoD metric scores as in-distribution data. Using angular margin would help separate OoD data in such case. To account for these factors, we propose intra-class mixup supplemented with angular margin (refer to Figure Tay to improve OoD detection where angular margin is the angle between the normal to the decision boundary and the sample representation.
Aripder Mtrgin
dist
Out
Latert Space
(a) In-Distrbuutio
10
magiin
L2 margin
6-
Dosision bouncary
10
50
60
70
80
Angular margin degrees
(a) Visualizing angular margin 0 and margin.
(h) Intra-class mixup.
17.5
Out
15.0
Out-dist
maagg
10.0
7.5
5.0
2.5
75
70
Angular margin degrees
Angular marqin degrees
(c) Inter-class mixup.
(d) ERM.
Figure Visualizing improved OoD separability of intra-class mixup trained ResNet18 model. SVHN was the in in-distribution dataset and TinylmageNet was used OoD dataset. For all the representations un-mixed inputs were used.
We show that the proposed intra-class mixup forces the network to learn latent representations with low variance (i.e. small spread), thus improving in-distribution and OoD separability. This is visualized in Figure [ which plots the angular margin vs. l2 margin for in-distribution and OoD data for ERM, inter-class mixup and intra-class mixup. Further, show that intra-class mixup trained, angular margin augmented OoD detector achieves on average 7.36% and 9.10% improvement in AUROC performance over ERM and inter-class mixup, respectively.
In summary the contributions of this paper are as follows:
We propose intra-class mixup to improve OoD detection. We show that intra-class mixup learnt latent space representations have lower angular spread when compared to ERM and inter-class mixup, thus, improving separability of OoD examples. We attribute this to the reduced input variance resulting from intra-class mixup.
We show existing discriminiative OoD detection methods that proxies for distance from the decision boundary benefit from intra-class mixup when compared to ERM and inter-class mixup.
We leverage better angular separability to improve OoD detection. We show that sup- plementing angular margin with existing OoD detection scores improves OoD detection performance various existing techniques.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
In the context of deep-learning, the approaches to detect OoD examples can be broadly classified into two categories discriminative and generative methods. Under discriminative methods recent works (Hendryeks & Gimpel, ( Hendrycks et al.] 2019 Hsu et al. 2020 Liang et al. 2018) have leveraged softmax probability scores to detect anomalous examples. The authors of Hendrycks & Cimpel (2017) presented their results on using softmax scores to detect OoD examples on several machine learning tasks. The idea utilizing softmax scores was further extended in Henndyycks et al. (2019) by modifying the standard loss function used to train the classifier. The network was trained on an in-distribution dataset as well as an additional auxiliary or outlier dataset. The authors claim that such outlier exposure enables the detector generalize and detect unseen anomalies. Further, the authors also observed characteristic properties of the auxiliary dataset which help improve detection performance. Another approach that leverages softmax scores are the ODIN (Hsu et al.) 2020 Liang et al. 2018) techniques, which showed improvement to OoD detection without having to explicitly tune the out-of-distribution dataset which the authors argue generally hard to define priori. The authors proposed decomposing the confidence score and modifying the input pre- prepprcessing method. Other recent work under the discriminative techniques proposed the use of energy score (Liu et al., 2020) instead of softmax scores arguing that softmax based approaches suffer from overconfident posterior distributions for out-of-distribution samples. They claim that energy scores are theoretically aligned with the probability density of the inputs and is therefore less likely to result in overconfident predictions. Further, in a non deep learning context angular distance based approaches (Kriegel et al. 2008) have been successfully used in literature to avoid the curse of dimensionality.
Generative methods attempt to model the underlying true distribution and hence, identify samples outside the modelled distribution. The authors of Lee et al.] (2018b) proposed technique that modelled a class conditional Gaussian distribution with respect to features of the deep network and used a Mahalanobis distance based confidence score caleulated from the Gaussian model to detect OoD examples. In [n et al.] (201 Sa) the authors proposed the use of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to generate OoD samples to improve OoD detector training. They proposed two additional loss terms that were added to the original loss functioo. One of the proposed terms forced the classifier to be less confident on OoD samples while the other helped generate more effective OoD examples. Another work that investigated a generative model based approach for OoD detection was [en (2019) which proposed a likelihood ratio method which corrected for background statistics and achieved competitive performance classification tasks. Approaches under both categories have focused on networks trained using Empirical Risk Minimization (ERM) which has been observed to result in undesirable behaviors such as memorization (Zhang et al., 2016)) and sensitivity outside the training examples (Bruna et al., 2014). Inter-class mixup (Zhang al.. 2018) has been shown to be effective against such undesirable behaviours (Zhang et al. 2018: T'hllasiaasann et al. 2019) and has been shown to improve calibration and OoD performance (Thulasidasan et al. 2019). However, we show that inter-class mixup and ERM trained networks learn representations with large variance (i.e. large spread), impacting their OoD detection performance. To counter this effect, in this paper we propose intra-class mixup aided with angular margin for improving OoD detection.
vision and sequence
3 BACKGROUND
In-distribution. The training and testing samples used for machine learning task are drawn from some underlying distribution, also called the true distribution. The samples from this underlying true distribution are called in-distribution samples and are represented by D.n in this work. However, in most practical learning scenarios the true distribution unknown a priori and hence, the training distribution, Dtrain is used proxy for Din.
Out-of-distributinn A sample is said to be -ut-of-distribution (OoD) if does not belong to the underlying true distribution D.n. and is denoted by Dout- Since we do not have access to the true underlying distribution, for our experiments we use uniform noise, Gaussian noise and samples from other datasets for similar tasks as a proxy for distt---istiibuttinn samples or Dout. This is common
practice in literature (Hendrycks & Gimpel 2017 Hendrycks et al., 2019 Hsu et al. 2020 Lee et al. 2018b [iang et al. 2018).
3

---

Under review as a conference paper at ICLR 2022
Empirical Risk Minimization (ERM). The objective of learning algorithms, when the true underly- ing distribution is known, is to learn an optimal mapping f(.) from the imput x to an output y. where (at y) ~ Din- This objective is satisfied by learning function f(.) that minimizes a loss function [x)) y) over the entire distribution. We can compute the expectation of the loss function as shown in Equation T This is know expected risk.
Rexpected Dr, [z[())),
As in most realistic scenarios, we do not have access to the true data distribution, and hence, it is common to use available empirical data as proposed in Vapnik (1998) to approximate the expected risk. This form of approximation, presented in Equation is know as Empirical Risk Minimization (ERM).
N: Σ c(f(x.),
Rempirical
Ne
where (Is; vi) Dtrain and Ne is the number of training samples in Dtrain.
METHODOLOGY
4.1 ANGULAR MARGIN
In this subsection we define angular margin. Angular margin 0 can be described by the following equations:
W
(3)
W
ainp
0 = arccos
n
(4)
ainpp
where / is the weight vector of the final (classifier) layer of the deep neural net (DNN) for the predicted class ÿ. n is the unit vector orthogonal to the decision boundary"] Ginp the latent space vector representation for the un-mixed input (i.e. activations of the penultimate layer of the DNN). Further, we ignore the bias term when calculating the angular margin therefore W does not include the bias term.
4.2 INTRA-CLASS MIXUP
In this subsection we explore the properties of intra-class mixup trained deep neural nets and how they can be leveraged to improve OoD detection performance. Inter-class mixup (Zhang al. (018) a technique that helps improve the generalization of a machine learning classifier. Mixup trains neural net convex combinations of examples and their corresponding soft labels. Consider pair of training input, label tuples (æ; yi) and (æx, Vý). Inter-class mixup obtains a new augmented sample (æ, for training the classifier described by Equation
= - x + + (1- x)
(5)
= (1-
where, À E [0, 1] represents the mixing parameter. the same fashion as inter-class mixup (Zhang al. 2018 X Beta(o, a) and for experiments forces the model to learn smooth interpolations between the samples of the training dataset. The proposed technique of intra-class mixup imposes an additional constraint on Equation such that y Vj, forcing the training sample to be a convex combination of inputs within the same class. be shown that the additional constraint on ý reduces the class conditional input variance (see Appendix A.I.
Such augmentation scheme
'The unit vector is the decision boundary normal in case of hinary classification. In case of a multi-class classification, the interpretation depends the training loss used.
4

---

Under review as a conference paper at ICLR 2022
[AP = (1 (1-1)² xx
x
Since 12 + +1111- - 1)² 1
(6)
x < x
where x is the variance of the input after mixup and ox is the variance of the input prior to mixup.
Therefore intra-class mixup results in reduction of class conditional variance at the input. We observe that deep neural nets trained on intra-class mixup learn to represent even unmixed images with lower angular spread (i.e. reduction of the class conditional representation variance) compared to ERM or inter-class mixup. The angular spread refers to the standard deviation of the angular margin for given dataset. This reduced angular spread increases the angular margin between in-distribution and OoD samples improving OoD detection performance.
4.3 SUPPLEMENTING ANGULAR MARGIN
In this subsection describe how angular margin scores can be supplemented with various Ool) scores to improve detection performance. We observe from Figure [ that traditional scores that use the l2 margin can separate OoD data from in-distribution data. From Figure [ this can be interpreted as higher values along the y-axis are more likely to be OoD. But using only 12 ignores information in the angular margin that can be used separate in disistribution samples from OoD samples. From Figure [
we see that using angular margin improves separability (using two axes instead of one). This is also observed our experimental results (refer to Table and 4p. Therefore we propose supplementing OoD score Jg is given by
J, = cos (0)
where is the OoD score from detector of choice, 0 is the angular margin of the test sample described in Equation 4 The OoD score J is different for different detection schemes. For MSP (Hendryeks & Gimpel. 2017) J would be the softmax probability of the predicted class while J would be the energy score (Liu et al. 2020) when using energy score detector, and so on. To complete our analysis we also use "Vanilla cos (0)" which is when J. = cos (0). The following section details various experiments to evaluate the performance of the proposed method.
EXPERIMENTS
We perform experiments which aim understand and show the role of intra-class mixup in improving OoD detection performance. The experiments' goals are summarized below
To show intra-class mixup trained networks learn representations that have lower angular spread and improved angular separability (Section 5.1)
To show intra-class mixup improves OoD detection on methods that use distance proxies (Section 5.2)
To show that supplemental use of angular margin improves OoD detection performance (Section 5.3
5.1 SEPARABILITY
In this section we provide evidence for improved OoD separability resulting from intra-class mixup. To show that intra-class mixup improves OoD separability we evaluate the angular margin for in- distribution and OoD examples. The improvement in separability resulting from intra-class mixup can be observed using separability metric S given by
(0. )
S
(8)
0. + -
5

---

Under review as a conference paper at ICLR 2022
where 8, and 8, are the angular margins for OoD data and in-distribution data respectively, d, and of are the angular scatter/spread (i.e. standard deviation) of the angular margin. The metric the angular version of the Fisher's criterion (Fisher et al. 1936
Average OoD
Average In-Dist.
Angular Separability
Angular Margin (in rad)
Angular Margin (in rad)
(0. 0:) 0 + o7
In-Dist
8 + d
02
ERM
Inter
Intra
ERM
Inter
Intra
ERM
Inter
Intra
1,4262
1.2992
1.4534
1.2901
1.1977
1.3065
CIFAR- 10
+
+
+
1.9469
1.7991
2.8342
0.0587
0.0585
0.0478
0.0779
0.0480
0.0730
1.4481
1,2991
1.4819
1.3817
1.1942
1.4012
CIFAR-100
+
0.4053
0.8211
0.7355
0.0474
0.0553
0.0426
0.0929
0.1017
0.0839
1,1810
1,2351
1.2626
0.8815
1,0501 +
0.9748
SVHN
+
1
3.2375
2.7753
4.0507
0.1174
0.0937
0.0952
0.1180
0.0596
0.1067
1.1344
1,2990
1.2322
1.0931
1,2952
1.1955
TinelmageNet
+
+
+
+
+
0.0467
0.0020
0.0774
0.1008
0.0444
0.0654
0.1623
0.0736
0.1 146
Table Separability of intra-class mixup, inter-class mixup and ERM trained models (averaged over 5 different seeds and OoD datasets) on various datasets. Intra-class mixup has better separability in almost all cases.
Table T shows the angular separability of empirical risk minimization (ERM), inter-class mixup (Zhang et al. 2018) and the proposed intra-class mixup. From Table [ we clearly see improved OoD separability from intra-class mixup. For each in-distribution dataset in Table the numbers show the average angular margin and angular spread computed over 5 differently seeded ResNet18 networks evaluated on the test set of SVHN (Netzer al. 2011). CIFAR- 10, CIFAR-100 (Krizhevsky et al. 2009), Places365 (Zhou et al. 2017). Textures (Cimpoi et al. 2014), LSUN (Yu et al. 2015). TinylmageNet (Li et al.). Gaussian Noise and Uniform Noise OoD datasets.
5.2 INTRA-CLASS MIXUP FOR OoD DETECTION
We have seen in the previous subsection that intra-class mixup improves OoD separability. In this subsection we provide evidence for improved OoD detection performance of intra-class mixup trained DNNs. We present the results of applying intra-class mixup on few unsupervised methods that use proxies for distance from the decision boundary such as Maximum Softmax Probability (MSP) detection (Hendrycks & Gimpel (017). ODIN (Liang et al. (018)) and Energy score detection (Liu ctt al. 2020). To evaluate the performance of the detectors we use variety metrics which are briefly described in the following section.
Metries. The performance of binary classification algorithm can evaluated using Recall or True Positive Rate (TPR), False Positive Rate (FPR), Precision variety of other metries. Recall (TPR), FPR and Precision are defined by equations (9). (10p and (II) respectively.
TP
Recall True Positive Rate (TPR)
(9)
TP + FN FP
False Positive Rate (FPR)
(10)
FP +
TP
Precision
(11)
TP+
where, TP is True Positive, FP is False Positive, TN is True Negative and FN is False Negative.
Metrics commonly used in literature (Hendrycks Gimpel| 2017: 2017 Hendrycks et al. 2019 Lee et al.
2018h) evaluate the performance of Ool detectors are:

---

Under review as a conference paper at ICLR 2022
Datasct
Accuracy
ERM
Inter
Intra
CIFAR-10
92.90 = 0.27
94.32 = 0.29
93,90 = 0.34
CIFAR-100
71.62 + 0.18
76.51 + 0.28
72.28 + 0.48
SVHN
95.69 + 0.16
96.42 + 0.04
95.91 0.11
TinyImageNet
32.80 + 0.17
34.04 + 0.68
34.63 + 0.30
Table Baseline accuracies (mean std. averaged over 5 seeds) of ERM, inter-class and intra-class mixup trained models on various datasets.
The Receiver Operating Characteristic (ROC), the plot of the True Positive Rate (TPR) against the False Positive Rate (FPR). The area under the ROC is called Area Under the Receiver Operating Characteristic (AUROC). AUROC of I denotes an ideal detection scheme, since the ideal detection algorithm results in false positive and false negative samples.
AUROC:
The Precision Recall Characteristic (PRC), the plot of the Precision against the Recall. The area under the PR is called AUPRC and should similarly be for a ideal detection scheme.
AUPRC:
FPR at TPR 95% (FPR95): denotes the FPR when the TPR is 95%. Lower value of FPR at TPR of 95% indicates better classifier.
To evaluate the effect of intra-class mixup existing OoD detection techniques. trained 5 differently seeded models each with ERM, inter-class mixup and intra-class mixup. We trained ResNet18 (He et al. (016)) networks CIFAR-10, CIFAR-100, SVHN and TinyImageNet datasets till convergence (baseline accuracies are presented Table 2). The training procedure used the SGD optimizer with momentum of 0.9 and weight decay of 5 10-4 The training used a 90%-10% training-validation split with the initial learning rate set to 10-2 and it was scaled down by a factor of 10 60% and 80% completion using a learning rate scheduler.
is common practice in literature (Hendrycks & Gimpel| 2017 Lee et al. 2018b Liang et al. 2018 Ren al. 2019) use data from distributions other than the training set OoD data. We follow the same approach. For example, if network is trained on CIFAR-10, datasets other than CIFAR-10 (SVHN, Places365, Textures, LSUN, TinylmageNet, Gaussian Noise and Uniform Noise) are used as OoD data. (able(((whiee) is a condensed version of Table 5 6 and 7 reports AUROC, AUPR and FPR95 results of applying MSP (Maximum Softmax Probability Detector (Hendrycks & Gimpel| 2017)), ODIN (Liang et al. 2018) and Energy Score (Liu et al. 2020) ERM, inter-class mixup and intra-class mixup trained models. The detection performance (AUROC, AUPR and FPR95) was obtained by aggregating results from 5 differently seeded models and OoD datasets. We used Gaussian Noise, Uniform Noise, SVHN, CIFAR-10, LIFAR-100, Places365, Textures, LSUN and TinylmageNet as OoD datasets, excluding the corresponding in-distribution dataset.
We also compare the effect of using only cos (f) as the OoD score, referred in Table as to "Vanilla cos (9)". Using only cos (f) ignores information from the l2 margin, thus when existing techniques are provided with angular information, we observe improved performance. Table 3 demonstrates that using "Vanilla cos (0)" performs better than MSP, ODIN and Energy score based detectors. Further, we observe adding angular information to existing techniques enhances their performance.
in Table 4
On average we observe an improvement of 4.68%. 3.98% and 9.97 % on AUROC, AUPR and FPR95 metrics respectively on unsupervised OoD detection schemes with the use of intra-class mixup over empirical risk minimization. Further, we observe an improvement of 6.08%, and 21.63% % on AUROC and FPR95 metrics over inter-class mixup and near identical AUPR performance when compared to inter-class mixup.
5.3 SUPPLEMENTING ANGULAR MARGIN
In this subsection we present the performance results when the OoD detection score is supplemented angular margin. We show that the use of the cosine of the angular margin the Ool detection score
7

---

Under review as a conference paper at ICLR 2022
AUROC
ALPR
EPR55
Toch
In-Dist
Intr
CR-10
0.823
0,054
0.854 1 0.022
0,864
0.X7
0.750 0.111
0,899 :: 0.084
08199 0,113
0.608 1t 0.111
0.504 0.244
0,303 :: 0.096
CF-100
0.064
0.188
6,773
0,050
0.734 0.020
0.600
0.199
0.701
0.162
0.651 + 0.120
0.130
0.583
0.170
0.608 +1 109
SVEEN
0.923
0.007
0,888
0.022
0,236
0.761 0.109
0.783
0.106
0.719
0.126
0.266
0.044
0,485
0.197
0.358
0,095
TIN
0,551
0.110
0.537 + 0.83
0,581
0.120
0.527 0.199
0.514 + 0,190
0.560 +: 0.206
0.826
0.066
0.825 + 0.050
0,509 +: 0.187
0.741
0.768
0.150
0.772
0.672
0.714 0.191
0.184
0.224
0.595
0.227
0,515 0.228
CI-10
0.897 0.017
0,772 A. 0.100
0,911
0.031
0.852 0,076
0,783 d 0,150
0,878 :: 0,083
0.317 0:086
0.594 1. 0.335
0,259 1: 0.116
0.763
0.125
0.79%
0,06
0.377
0.742
0.172
0.156
0.621 0.347
0.515 0.216
SVHN
0.836 0.187
0,868
0.060
0.855
0.123
0.693
0.216
0.806
0.118
0.700
0,188
0.518 0.210
0.606 0,115
0,480
0,160
0.642 0.134
0,553 + 0.041
0.680
0.145
0.614
0.220
0.53 + 0.181
0.688 0.342
0:229
0.793
0.082
0.680 0.264
0.776
0.729
0.160
0,814
0.135
0.704
0.203
0.716
0.198
0.739
0,200
0.544 0.236
0,658 0,264
0,484 0.348
C3-10
0,850 0.036
0,763 + 0.164
0.983 00832
0.818
0.096
0812 + 0.116
01842 +: 0.087
0:922 0.006
0.688 + 0.344
0.276 + 0.09
0.721
0.177
0.261
SVIEN
0,914 a: 0,014
0,881 à 0.054
0.881
0.035
0.764 0.104
0,819 : 0,065
0,700 a: 0,130
0.369 0,072
0,611 11 0:00.25
0,449 0,100
0.097
0.533 : 0.044
0.636
0.135
0.396
0.521
0.179
0.517 = 0.228
0.803
0.725 0.180
Aug.
0.753 e: 0.179
0.730 &: 0.163
0.792
0.132
0.650
0.197
0.705 :: 0.189
0,700 :: 0.187
0.530 0,218
01674 :: 0.240
0,501 :: 0.214
CF-10
0.712 +: 0.150
0,873 #: 0.061
0.930 = 0,022
0.836
0.082
0.868 1: 0,070
0.882 :: 0,066
0.296 0,083
0.512 d: 0.262
0.319 1: 0.117
0.966
0.796
0.051
0.756
0.053
0.638
0.193
0.718
0,163
0.694
0,170
0.146
0.501 d: 0.179
0,499 0.194
Vaiila
SVHN
0.939
0.038
0.937 + 0.000
0.058
0.023
0.825
0.084
0.844
+ 0.081
0.862 + 0.080
0.196 0.080
0:277 + 0.188
0.182 + 0.002
0.564
0.401
0.101
0.118
0.550
0.489
0.180
0.545
0.202
0.088
0.827
0.073
0.773 0.106
Asg.
0.779 A: 0.174 0,774 à 0.184
0,814
0.163
0.707
0.198
0,732 A 0,202
0.747 :: 0.196
0.469
0.256
0.529 à. 0.264
0.411 1: 0.278
Table Various unsupervised distance proxy based OoD detection techniques on ERM, inter-class mixup and intra-class mixup trained models (averaged over 5 different seeds and OoD datasets) on various datasets (mean + std). Expanded version of this table available Appendix A.2 CF-10 is CIFAR-10, CF-100 is CIFAR-100. TIN is TinylmageNet.
improves performance. The supplemented score is obtained by adding the OoD score the cosine of the angular margin as described in Equation 7
AUROC 1
AUPR
FPR95
Technique
In Dataset
Intra
Cas(!)
Intra
Intra Cus(f)
Intra Cos(B)
CIFAR-10
0,884 = 0.037
0.915 : 0,020
0,819 = 0.113
0.853 + 0,089
0.303 = 0.096
0.107
0.030
CIFAR-100
0.724 + 0.020
0.747 : 0.020
0.651 = 0.170
0.666 + 0.168
0.608 = 0.109
0.557 = 0.127
MSP
SVIIN
0,900 1. 0.036
0.939 1. 0.023
0.719 1 0.126
0.786 . 0.105
0.338
0.095
0.177
0.063
TiyyInggNNet
0.581 ... 0.120
0.583 1 0.121
0.560
0.206
0.558 1 0.205
0.809
0.087
0.799
0.090
Average
0.772 0.146
0.796 + 0.157
0.714 = 0.191
0.716 + 0.187
0.514
0.228
0.441
0.271
CIFAR-10
0.921 1: 0.031
0.935 1 0.027
0.878 :: 0.083
0.900 +: 0.072
0.259 =: 0.116
0.229 : 0.118
CIFAR-100 100
0.798 0.065
0.828 + 0.087
0.721 + 0.172
0.764 + 0.184
0.515 0.216
0.462
ODIN
SVHN
0.855 + 0.122
0.933 + 0.092
0.709 + 0. 188
0.840 + 0.184
0.480 + 0.160
0.221 + 0.228
TinyImageNet
0.680 = 0.145
0.637 : 0.150
0.648 =: 0.242
0.629 + 0.237
0.680 = 0.264
0.712 + 0.221
Average
0,814 # 0.135
0,838 : 0.150
0.739 = 0.200
0.784 + 0.206
0.483 = 0.248
0.406 0.293
CIFAR-10
0.903 # 0.032
0.919 + 0.017
0.842 = 0.087
0.863 + 0.071
0.276 = 0.099
0.238 + 0.094
CIFAR-100 100
0.748 0.048
0.763 + 0.043
0.656 = 0.1
0.700 + 0.173
0.553 0.154
0.532 0.166
Energy Score
SVHN
0.881 11: 0.035
0.927 1 0.030
0.703 :: 0.130
0.798 1: 0.102
0.449 = 0.100
0.279 : 0.098
TinyImageNet
0.635
0.135
0.610 0.125
0.597 1: 0.228
0.570 0.214
0.725
0.180
0.745 0.143
Average
0.792 + 0.132
0.805 + 0.147
0.700 + 0. 081
0.725 + 0.189
0.501 + 0.213
0.449 + 0.242
Table 4: Various unsupervised distance proxy OoD detection techniques on compared with and with out angular margin (averaged over 5 different seeds and OoD datasets, expanded version of this table is available in Appendix A.2)) on various datasets (mean + std).
8

---

Under review as a conference paper at ICLR 2022
1.2
1.0
0.741
768
0.772
0.736
0.776
0.739
0.814
0.838
0.755
0.730
0.792
0.80
0.779
0.774
0.814
0.8
AUROC
0.6
Empirical Risk Minimization
0.2
interclaass Mixup
Mixup (Ours)
intraa class Mixup + Coste) (Ours)
0.0
MSP
ODIN
Energy Score
Vanilla
Figure 2: Comparison of average AUROC performance (averaged over in-dist and OoD datasets and 5 seeds) of methods that use distance proxies improved with the use of angular margin and intra-class mixup. Please note that "Vanilla cos" by definition cannot use the additional angular margin term, hence its not plotted.
Figure 2 and Table 4 compare the OoD detection performance of methods that use distance proxies when used with intra-class mixup aided with angular margin. Table reports mean and standard deviation obtained by averaging 5 different seeds for each in-distribution dataset and averaged over various OoD datasets as used for Table 3 We observe that on average the use of angular margin with intra-class mixup improves OoD detection performance (AUROC) by 7.36 % over empirical risk minimization and 9.10% over inter-class mixup.
CONCLUSION
There have been numerous approaches to tackle the problem of OoD detection. In this paper we explore the use of intra-class mixup to train OoD detectors. We show that intra-class mixup reduces the angular spread of the learnt latent space representation, improving the angular separability of in-distribution and out-of-distribution data. We observe that the use of intra-class mixup detection techniques that use distance proxies improves AUROC performance by 4.68% and 6.08 % over empirical risk minimization and inter-class mixup trained models respectively. Further, the use of the cosine of the angular margin to supplement detection scores improves AUROC performance by 7.36% over empirical risk minimization and 9.10% over inter-class mixup trained networks. Our findings reveal that intra-class mixup can be an effective tool for training OoD detectors.
7 IMPACT STATEMENT
The research presented in this paper focuses on improving the reliability of deep learning techniques in detecting anomalies, reducing the barriers the deployment of deep learning in safety critical applications such as medical diagnosis, self driving cars etc. Hence promises a positive impact on the society. However, conjunction with the technique proposed in the paper to ensure reliable and secure deployment in safety critical applications. This paper does utilize human-derived data and does not utilize any datasets that have been discredited by the creators.
must be aware of the limitations of the specific techniques applied in
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016.

---

Under review as a conference paper at ICLR 2022
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized ceasittion-basee neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2442-2452, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1..18663/v1/116 1231.. URL URL https: http://www.wwww...rgogg/o http://ww.orrgaahooogooogo P16-122
Joan Bruna, Christian Szegedy, Ilya Sutskever, Ian Goodfellow, Wojciech Zaremba, Rob Fergus, and Dumitru Erhan. Intriguing properties of neural networks. International Conference on Learning Representation (ICLR), Poster, 2014.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2014.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference machine learning and knowledge discovery in databuses, pp. 3-17. Springer, 2018.
Ronald Aylmer Fisher et al. 138: The use of multiple measurements in taxonomic problems. 1936.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference Learning Representation (ICLR), Poster, 2015a.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributin ex- amples in neural networks. Proceedings of International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representattions, 2019.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-diitribution data. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Hans-Peter Kriegel, Matthias Schubert, and Arthur Zimek. Angle- outlier detection in high- dimensional data. In Proceedings of the 14th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pp. 444-452, 2008
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems 25, pp. 1097-1 105. Curran Associates, Inc., 2012.
4024- at with dehb- coneoluooovll neural wtw.rraawwwwtoo..............
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. International Conference on Learning Representation (ICLR), Workshop Track, 2017. URL
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 444, May 2015, ISSN 1476-4687, doi: 10.1038/nature 15339, URL /ttp::/:///oo.oo/110000033
naturel 14339
10

---

Under review as a conference paper at ICLR 2022
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrate classifiers for detecting -istriibbittito samples. In International Conference on Learning Representations. 2018a. URL /tllattsss//ppprrreeeee
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing
Systems 3/, pp. 7167-7177. Curran Associates, Inc., 201 8b. URLbttpp http:::://papersniss..oo /
1aper/7947-a-5000 aat.a
ftaaeiifdwddddaaodddaeeeoooe for detecting out-of
Fei-Fei Li, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge.
Shiyu Liang, Yixuan Li. and R. Srikant. Enhancing the reliability of out-of-distribute image
detection in neural networks. In International Conference on Learning Representations, 2018.
URL ottp::/
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy- baaegy out-of-distribution detection, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learn- ing and Unsupervised Feature Learning 2011, 2011. htRlitttpp//lll....aaffoodd housenumbers.pdt.
Moosenumbers/nipssUU
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427-436, 2015.
doi: 10. 1111109/CVRR.201722966666
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14707-14718, 2019.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neu- ral networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: http:eee/ppooeeiing.geepp.epo/ep
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
William Wang, Angelina Wang, Aviv Tamar, Xi Chen, and Pieter Abbeel. Safer classification by synthesis. arXiv preprint 17 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03350. 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL https:
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
11

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 VARIANCE OF INPUT DUE TO INTRA-CLASS MIXUP
Consider two random variables Xo, 11 X where X is a class conditional distribution for class y with mean μ.x and variance xx. Then the variance of X agxx X1 where ag A and
41 = 1 À is given by
o = =Va((X)= = (x (x-1+) )
2 = E' (Ei= aix; - )
o E [Eioo -xxi) }
8 F.
[1 Ek=0 Ck (xx
- wxx) (Ei (x - ,x )
2/ E
E] Σk axaz (Xn - ux) 1x)]
x Ej=o Et= E E(xx - xx)(Xx px)
o Ek=0
ai + 2aga Cov (x. X1)
The Cov(Xo, X1) 0 because Xo and X1 are drawn independently, therefore the equation is reduces to
xx [22 (1 2)²] o%
Since X2 + (1 11 -2)² 1
o < x
A.2 EXPANDED RESULTS FOR VARIOUS DETECTION SCHEMES
12

---

Under review as a conference paper at ICLR 2022
Masream Setrr Probatilty Dender
Oul Dasss
AUIIC
ALBE
11859
BBM
BRM
Inse
BBM
Ossous Nexc
0,592
01223
69191
0L2230
0.5012
2.00615
0.973
05:03
0.3915
0.120-00::1
03151
Unifenniie Neise
0 00303
0 9462 +-0 0.0033
09653
50000
0
00413
9959
0.0063
0,009
0.000
03503
0,029
0.153 +.00000
0.100 00547
CIFAX-10
IL5445 1 100089
03473 1-10344
0.074
147318 2 10035
0.767 1 0.0047
0.438 1:0002
0L5344
100143
03484
0-044-11:00 110004
1L0009 2 140775
03456 2 01005
03RC3
103000 2 11:244
0,600 1:0033
00407
Awsegg
00598
05507
+-00419
0,8665
0
06009=0.005
0.5067:400
2445
0,905 OWNG
Furocze Navio
003965
009955
085616
010055
0.7109
1990
515617 (00181
0.791 0.0075
06711 + 6.199
01337
000744
06:000
0/900
1.000
Luiten Nekes
160027
0.464
1.3000
0.71991
31906
IL5899
000955
DANNS 1 01M77
I13793
100018
0.3131
0:0233
SVEN
0.3425
100273
0.7460
0.0032
0,7001
51005
ILE337
002173
0.842
08000 2: 0/211
06:23 a 010486
0.013:20000066
1.912 01570
CHRMR-00
Solaso
0.543 00033
0.7454
0.0000
00001
0,5015 000446
0.5000
0:0004
03001 0.006
07322 0.007
0.0011
07566
LAUN
0,000 - 00051
07417-+00000 0.0007
07118
50015
0.3591 0/966
0010 + 0.0066
0,3266 + 00031
0:304 0.0099
0.7123+-000
17003-00.0000
00031 000099
Thybbagsher
117011 : 109233
1-1.00
0/7006 6: 3JMCM
IUBEES 22 WJ0222
0-500-241001
b00ps 2: 0AMIS
M8992 : 000955
0.7005 1:01043
16720 CUROS
Awaaye
Casesas Net
IJ215 = 100441
08316
0.906
IU7730 2 160295
0.600 2.0.3014
DASTM 2: 0.1415
IL2335 : (12550
0.018 4:04400
1/36 2 CLIMS
Xciee
00001
0.199
0.244
CIFAR110)
0.337
00032
0::000+000000
09085
30001
0,7993 000367
8672
0.0000
0.872 + 8.085
02552
0.025
0.3283 0-0044
0,200 00112
SWEN
"Sorim
04966
01099
08811
0.005
09019
05651 00151
0.991
0.001
D/WS 04799
03733 00090
0.7700 0 1176
D/017
LSUN
0L2289
109351
0.953
110010
09126
IL9024 100133
0.7100 4.003
013334 (00300
03332.1:00110
0,303 = 00037
0.9554
0.005
2/M05 00339
0L9071 000366
0,8132
000017
05000 :: 010M
0.338 0.2221
0.3234 0
02200 00146
0.231 0.0377
0.955
0.00/3
00034
0.064
0.007
02436 0.255
0.3307.00.1358
Awraae
04735 4 00074
05850
00003
090015
30000
117610 - 00108
05334
+-0 1865
67790-00770 17776
07665 - 605557
0.4953.4 0 1967
03377 0094
Ousace Noto
0.590 0.1004
811255
00065
0..113
(1636 6:
0.334:00000
0.7842
06334
Uritien Noku
0.337 - 00095
0 33981-1 0:131
0.3135
3.333 - (.07))
0.110 5 0.005
0,3817 + 00000)
B9364 - 000755
0.7806 1030
DR0 00170
CIRAR-100
0.3734 = 109234
0.2000:1...00
0.9104
2/MB5
IU5491 2 100035
03543 0:00
0,5644 2: 03044)
0.5713:2 0.0035
0.800-1-00004
09922 CURG1
Tolaso
0,5535 000778
0.00143
0,9115 04005
0.8586::00054
1SUN
0.897 00051
0,513
+00.063
3/007 0.9000
0.2399 0/0541
0.6655
+.000
0,300 - 0.004
0385) - 0.0081
0/60 0/007
365
05117 109315
0,600
0A137
118328 060014
0.200
(.006)
01915
01331 102944
08121 1.0000
0.271
Averas
0.3510 048104
0.3322
5.00002
2.11
(45272
(1.99)
0.312.20.1900
0,MM5
0,300
0,223
0:00
00000 01871
Alerage
017455 J (1771
0.7100 (113558
0.7725
5.3461
115333 (1:415
0.7122
0.1911
[15945 (0.3341
0.5916 1.222
0:5185 0.2385
Table 5: Expanded results for Maximum Softmax Probability Detector.
CIDEN
IBM
Traer
BRM
Iaser
BBM
Iaker
bara
0.8634 00985
0 0.9566= 0.0500
0.938
30005
0,739
00363
0.955 + 0.0600
00455
02800
0.0055
0.1214
09081 00175
SVIN
04337
(717
1 010600
09001
0.9000 0 0.0068
CIRAX-10
Totans
0,993 = 00002
0.045
0
0.825
00266
6400 0.0081
0.078
00088
03350 = 0.836
0.8312 0.005
0.796
00437
1.5N
0415)
0.013
0.003
07955
00025
0 0.000
0.205 D0053
Thybragetee
0L9940
140133
03231 1.003
00000
IL8096
0.2244 0.007
01440
0 0.00011000003
Plaa335
0.9997
00091
0.5429
0.0335
00006
20000
IL9942 000225
0.9065 0.001
00072 8 00000
0.334 : 0.230
0.7875 0.009
03335 DUR06
Awarg:
0.999 00120
0.5388
027764
0.7535
03:67
Garwen Nake
0.537 - 00759
0 39555
+ 0.0005
09065 3.07L
0.563 00333
0.9070 + 0.006
09777 + 0.777
05571 - 007788
0.0005 (0006.+-0000)
01305 0/661
SVIEN
03354:12 100249
0.7256 0.2033
0:1477 0: 51047
IL5490 2 100122
00009 2: 01D01
0.5133 0.339
03005
03323
LSUN
0:333
00099
0.673
0.000
0.0055
3/004
0.385 000933
0.5544 00.545500005
0.676 + 0.006
0.386 = 00027
0:614---00045
06631
00113
03635 00033
2.00022000062
50006
0.000
0 00992
0.005
00000 0.7778
019972 -4 0,0000
0
00100
-0005
07364
0,997
08838-+000044
0/904 0/099
03977 00001
8
0.909
I1553 01126
0.116
0:2015
0/2000
9:724
08002
(1:50
0,6006 1.33
0:588 0216
Casera Nome
(L9229
100257
03318
3.13540
0J0AD
IL7453
0.7085
0/7922
:: 0.13C
I14425
(L:01
0:2501.::03115
0.2158
Unifum Noioe
04323
00992
0.6659
0.1668
0.500
0.1762
0,2320 2 00232
0.5044
0.1992
03811 =: 0.150
03438 = 02224
0:3006
0/754
02206
CIFAR.000
03216
0.9217
0,339 00145
0.0144
01444 0.075
03903 02017
0.5903 +.0005
SVEN
TSommm
04335 60383
000033 05491
0,899 30000
0.5088
00131
07534 5 01033
1.0 00058
03744 00535
0.861
0.006
D696 0.155
02231 = 10000
05002 1.0013
09276
20117
(U5422 2 000142
03553
03431 :: 04215
03353 0,2500
0.5511
03414 2: 04005
Haaa355
05055 0.0022
0.9559
09985 00121
00536
0,5415 0.155
0.3399 (43671
08481 0.0000
0,8448
5.1225
(U9933 22 0.2162
00000
1..1144
0,000 6: 0.182
0L::90 = 0L2003
0.600 0:141
0-000 0.1001
00991
0.5022
00937
04003
0U3558
0.6544::00300 2351
011854
Uribern Neke
0.737 60700
0 68336
+ 1433
0.5900
1665
0.5821 - 0/348
0.0066
0.007 + 0.06
07536
00065
0 1116
6677
01434
SVIIN
0.751
0.511 1.3376
0.0191
50125
IL9018 015355
0.3322
0.004
DMMI1 019154
113337
(00244
0.3341
05611
Thy EraeeNee
CUPR-100
0.016 109334
0A142
2JM02
IL3925
000337
0.5461
0.004
02940 0/002
01354 0.2031
0.38612:0.002
50576
0,55% = 00067
0.5466 0.2255
2.005 0.500
044266
000064
0.3666
0.011
04915
:: 00152
03:45
0.0922
0.0040
00140 00000
0.003
0.5673+40.000
04751 1 10003
03607
(000181
04306
310771
118348 0033)
03337 1 01009
baps: 1 0/006
08373 1 000955
03/13 0.0011
Aeeaag
01734 (+153)
0.719 (1 10.1605
0.117 31577
(17090 0.334
03199 6 1919
0.7901 i. 8199
(5543
013355
0.008 1.036113
0/3095 03/79
Table 6: Expanded results for ODIN.
13

---

Under review as a conference paper at ICLR 2022
Easp Sone
OuB Duzeet
EXM
BRM
BBM
0.066:200
Uxitirn Yoku
0.779 00095
0 9615 10019
040.9
3000
1175%
0909
0.000
04067
0000
0.1000 0
D119 DUKSE
CHAR-1I 10
Tolat>
0,554) = 00114
0.7251 0.0000
05742 30009
017699 2 00132
0.642 0.005
0.748
520
0.312
00541
0.883
0:004
04500 DANS7
LSUN
07747
TaylvagyNer
0.899
00075
0.0066
0.875 00071
00086
0009
08386
0.000
00030
103399
0.7100
000221
0.001
D/W0
01006
11393
000235
03439.1 0.00
00225
Ancrac
0.399
0.0.042
20019
002357
0.116
0.1115
0/421
03670
03222
00052
0:349:200:444
Cuserzn Natae
01313
100966
09955 110019
0.0126
518115
IL3857
0.844
0.135
118290
100331
0.3430 1:111
1280 DUMIE
Noloe
0.303 00373
0.259
20991
IL5654 000464
0.1005
0.1933
0.566
04744
0L3133 0.023
SVEN
0.779 = 00206
0.554
0,0330
0.766
2/020
0.5414 00155
0.743 0.0024
0609 =: 00157
00291 0,5026
0 0.2065 0:0455
04757 = 0/0406
LSUN
03095 1 100057
0:886
006661100000
10000
077M6
IL3754 B/0073
03550 4.000
00775 1 04854
00:35 00000
03167 1.000
D6662 CURCS
Plao355
03559 = 00040
0.600 0.0006
0.766
30064
(U9029 = 00022)
000000
00M4 = 50005
06873
0.035
0.868 0.00
DA374 2 0X022
07200
016665
0,6400::000 2609
0,535
Oasaas Notec
03130
00220
08192 00.171
0.406
20000
0,775 =2 000464
0.7466 2.24662202111
00000 0.1000
0.3248 = 0.059
0.5600
04002 i: 0.2555
Unitern Neke
09007
0 0.166
00504
0.6964::00330
CUXR-100
0433 D0077
09018 00001
09132
30001
0.000
0.09 0006
0331
00090
0.5899 0.111
SNIN
01992 10909
03816 1.1094
0/809
21046
IL5855 1U2267
0.7631 03660
016-7 DA211
I15322
00301
03126 0:061
DA217
03220
0.0010
2/210
02367
0.7556
0.0033
0.903
0.443 0.522
0.3311:00:312
Teyhnagehkt
03283 = 00067
0.900 0.011
09197
20117
0.8266 00155
08842 0.0056
0.99% :: 04004
02579 00351
0.533 0:16276
032100 00475
Awerage
I4340 - 140140
03840-110059
0J808 30000
IL7643 IL0944
GB395 41644
1.006 0.13
I6303 - (0033
0-03331-0.1338
DARM 1. CLINES
Lattern Nous
0.3300 : 100733
045377
5:1766
IL3938 162294
0.4730 0.1057
1VG115 04822
01575 11 000345
0.881 02881:1.0.165 0.33
177M 1. 0.122
Taylssackt
CIRAR-000
0.575 = 00044
0.5322
0.006
0,9158
5/002
0,5500 =2 0/031
0.5387 00000
03991 0.025
0.775
0.0942
0.5945
0.0011
09965 00029
TSotess
05530
0.5366 0.0337
0.5074
0.0079
03598
0.5886
1SUN
(0008 03810
05488
10015
IL3613 00366
0.117 (10006
04008 0.360
01135 (0005)
0.8099
0.001
00223 109335
0.992
9:00276
IU8297 010099
0.844
0.0029
0/015 :: 018/66
I18312 000035
03441
D8109 DJRMS
Anara:
00971
03338
0.0445
21330
0.3236 0L1961
0.511
0.9774
0.2277
0.0225
0.3543
0.201:000000
(13545
101722
0.7300 1.1.632
0.7018 0.511
IL5788 KL1900
0.1163 0.1889
DVVPN 0.1972
I15334 11.11:4
0.6741 0.344
D.000 0,2338
Table 7: Expanded results for Energy Score.
14

---

Under review as a conference paper at ICLR 2022
Maximum Softmax Probability Detector
AUROC
AUPR
FPR95
In-Datuset
OnD Dataset
Intra
Intra Cou())
Intra
Intra Cou(P)
Intra
Inira + Cos(8)
Goussian Noise
0,8191 0.0697
0.9207 0.0195
0,7116 =: 0,0969
08191 0.0463
0,3168 :: 0.0862
0.1070 :: 0.0294
0.0698
0.0894
SVHN
0,9175 = 0.0105
0.9209 :: 0.0101
0,9474 = 0.0071
0.4866
0.0068
0,2135 #: 0.0230
0.2023 :: 0.0344
Textures
0.8674 0.0064
0.9049
0.0098
0.7407
0.0084
0.7806
0.0116
0,4066 0.0330
0,2613 0.0472
CIFAR-10
LSUN
0.8924 = 0.0048
0.9166
0.0033
0.6447 =: 0.0069
0,6952
0.0044
0.3075 #: 0.0090
0.2381 :: 0.0174
Tiny InageeeeNe
0,8743 0,0014
0,8885
0.034
0,8409 4: 0,0023
08597 0.0027
0,3862 0.0141
0.3964 +: 0.0256
0.3299 0.0246
Average
0,8445 + 0,0370
0.9100 +: 0.0197
0.8194 =: 0.1126
08532 0.0886
0.3034 =: 0.0963
0.2321 + 0.1030
Gaussian
0.7191
0.1088
0.0756
0.6208
0.1057
0.6893
0.0951
0.1099
0.3616
0.0481
Uniform Noise
0.7160 1: 0.0911
0.7453
0.0772
0,6050 4: 0.0981
0.6239
0.0891
0.4393 1: 0.0741
0.3772 1: 0.0528
SVHN
0,7501
0,0355
0,7659
0.0992
0.8560 + 0.0211
0,595
0,0799
0.5912 + 0,0570
0.5370 0.0398
0.6879
0.0031
0.0046
0.5205
0.6385
0.0068
CIFAR-100
ISUN
0.7138
0.0018
0.7212
0.0118
0.3746 + 0.0031
0.0038 0.3844
0.6931 + 0.0059
0.6876 + 0.0041
TmageeageNet
0,7494
0.0032
0.7660
0.0029
0.7069
0.0042
0.7238 0.0041
0.6393 0.0037
0,6166 0.0082
Places365
0.7326
0.0029
0.7434
0.0035
0.8885 1: 0.0018
0.8938 0.0020
0.6720 4: 0.0035
0,6782 à: 0.0096
Avenae
0,7242 + 0,0203
0.7473 0.0202
0.6507 + 0.1702
0.6699 0. 1679
0.6083 0,1000
0.5566 + 0.1270
0.6567
0.1995
Uniform Noise
0,8330 = 0.0759
0,8903 1: 0.0401
0.6171 :: 0.1388
0.6862 0.1053
0.1974 4:: 0.1963
0.3064 1: 0.0951
CIFAR-100
0,9285
0.0031
0.9567
0.0015
0.8072 0.0085
0,8648 0.0053
0.2549 0.0112
0.1 0.0081
Textures
0.9019 0.0142
0.9319
0.0093
0.6611 =: 0.0299
0.7248
0.0256
0.1037 :: 0.0827
0.2299 :: 0.044
SVHN
LSUN
0.9176 0.0063
0.0023 0.9521
0,5419 + 0.0200
0,6445
0.0172
0.2964 0.0263
0.1401 + 0.0070
Tiny TaaylmagoNet
0.0091
0.1234 0.0051
Places
0.9266
0.0052
0.9560
0.0022
0.9325 + 0.0047
0.965
0.0025
0.2607 + 0.0020
0.1323 + 0.0056
Avenge
0,9003 0.0360
0.9391
0.0230
0.7194 0.1261
0.7861 0.100
0.3377 0.0948
0.1768 0.0632
Goussian Noise
0.7089 =: 0.1240
0.7103
0.1186
0.6584 :: 0.1297
0,6510 0.1217
0.6331 ::: 0.1384
0.6200 1: 0.1347
Unifom Noise
0,3133
0.1290
0.3151
0.1270
0.3841 :: 0.0419
0.3840 0.0409
0,8869 1:: 0.0472
0,8792 0.0479
+ 0.0262
0.7066
CIFAR- 100
0.5934 = 0.0033
0.5937 1: 0.0031
0.5644 =: 0.0040
0.5634 0.0037
0.8622 :: 0.0031
0,8629 :: 0.0035
ImageNet
Textures
0,5610
0.0109
0.5651
0.0106
0.3913 0.0093
0.3898 0.0089
0.8809 0.0058
0.8645 0.0034
LSUN
0.5892
0.0047
0.5907
0.0052
0.2690 :: 0.0044
0.2671 0.0041
0.8426 :: 0.0057
0.8374 :: 0.0063
Places365
0.6137 = 0.0029
0.6152 à: 0.0033
0.8316 =: 0.0013
0.8304
0.0014
0.8271 4: 0.0025
0,8210 :: 0.0022
Average
0.5809
0.1201
0.16
0.2060
0.0869
0.7988
0.0905
Average
0.7725 = 0.1461
0.7962 0. 1566
0.6875 =: 0.1843
0.7158 i: 0.1874
0.5146 0.2285
0.4411 +: 0.2710
Table Expanded results for Maximum Softmax Probability detector comparing Intra-class mixup and Intra-class mixup with angular margin.
15

---

Under review as a conference paper at ICLR 2022
ODIN
AUROC
AUPR
FPR95
In-Datuset
OoD Dataset
Intra
Intra Cow(d)
Intra
Intra Cou(P)
Intra
Inira + Cos(8)
Goussian Noise
0,8898
0.0665
0,9435 0.05K5
0,8038 :: 0,1183
0.910 0.0764
0.2091 0,1046
0.1194 1: 0.0705
0.0287
0.0367
SVHN
0,95%8 + 0.0116
0.9572 :: 0.0153
0,9801 : 0.0057
09785 0.0033
0.149 :: 0.0388
0.1560 :: 0.0524
Textures
0.8927
0.0128
0.9098
0.0102
0.8078
0.0188
0,8336
0.0145
0.3799
0.0437
0.3296
0.0407
CIFAR-10
LSUN
0.9243 0.005
0.9309
0.0047
0.7552 =: 0.0123
0,7715 0.0121
0.2745 #: 0.0233
0.2541 :: 0.0180
Tiny Imageeee
0,9010 0.0068
0,9036
0.0076
0.8867 :: 0.0074
0,898 0.0079
0,3857 0.0300
0.3750
0.0308
0.3329
0.0060
Average
0.9214 = 0,0308
0.934 :: 0.0269
0,8780 =: 0.0832
0,9996 0.0719
0.2587 :: 0.1162
0.2291 + 0.1178
Gaussian
03365
0.0374
0,8727 0.0757
0,0226
0.1305
0.0661
0.0298
0.0179
Uniform Noise
0,8070 1 0.1008
0.9357
0.0335
0.6811 =: 0.1122
0,8518 0.0701
0.2741 1: 0.1211
0.1055 1: 0.0479
SVHN
0,8177
0.0247
08835
0.0188
0,8859 + 0.0201
08781 0.0175
0,4685 + 0.0336
04492 0.0292
0.6369 0.0209
CIFAR-100
ISUN
0.7625
0.0024
0.7478
0.004)
0,4376 + 0.0036
0,4138 0.0063
0.6831 + 0.0013
0.683 + 0.0078
ImageeageNet
0,7790
0.0012
0.7804
0.0008
0.7376 1: 0.0016
0.7393 0.0006
0.6370 0.0050
0.6343 0.0076
Places365
0.7778
0.0028
0.7658
0.0043
0.9104 =: 0.0015
0,9048 0.0022
0.6624 4: 0.0059
0.6788 1: 0.0091
Avenae
0.7979 0.0681
0,8284
0.0874
0,7210 + 0.1724
0.7646 0.1843
0,5148 0.2166
0.4618 + 0.2598
0.8902
0.2158
Uniform Noise
0.5601 = 0.1762
0.7152 1: 0.0749
0.3311 :: 0.1563
0.1173 0.0785
0.7954 4: 0.2306
0.6907 1 0.1181
CIFAR-100
0,9264 0.0098
0.9815
0.0016
0.8444
0.0178
0.9531 0.0037
0.3807 0.0491
0.0784
0.0078
Textures
0,8539
0.0330
0.9167
0.0139
0.6394 ::: 0.05
0.7835 0.0252
0.6161 :: 0.1253
0,4404 0.0766
SVHN
LSUN
0.908 0.0180
0.9819 + 0.0011
0.6110
0.0497
0,8683 0.0112
0.4339 0.0754
0.0781 + 0.0091
Tiyy TayylmagNNet
0.0798 0.0074
Places365
0.9168
0.0160
0.9794
0.0019
0.9355 + 0.0121
0.9311 0.0016
0.3873 + 0.0966
0.0881 + 0.0089
Avenge
0,8548 0.1225
0,9331
0.0916
0.7094
0.1882
0,8406 0.1836
0,4800 0,1601
0.2208 0.2281
Goussian Noise
0.9772 0.0219
0.9162
0.0364
0.9737 :: 0.0263
0.9386 0.04-8
0.0943 1: 0.0854
0.2069 1: 0.0996
Unifom Noise
0,5562 0.1663
0.4426
0.1669
0.5037 :: 0.0968
0,4370
0.0609
0.6776 0,1424
0,7706 1: 0.0794
0.0054
CIFAR- 100
0.6162 = 0.0032
0.6059 1: 0.0018
0,5940 :: 0.0032
0.5830 0.0030
0.8576 1: 0.0048
0.8621 :: 0.0035
ImageNet
Textures
0.5700 0.0098
0.5789
0.0065
0.4415
0.0132
0.4387 0.0094
0.9140 0.0050
0.8922 0.0054
LSUN
0.6045
0.0052
0.6225
0.0076
0.2803 :: 0.0051
0.2896
0.0050
0.8350 :: 0.0061
0,8032 :: 0.0062
Places365
0.6216 # 0.0030
0.6158 à: 0.0050
0.8344 =: 0.0016
0,8302 0.0025
0.8232
0.0041
0.8217 +: 0.0026
Average
0.6805
0.99
0.2419
0.6291 0.2370
0.7116 0.2210
Average
0.1177 = 0.1347
0,8382 0.15
0.7390 =: 0.1996
0.7835 0.2061
0.4835 :: 0.2479
0.4058 :: 0.2934
Table Expanded results for ODIN comparing Intra-class mixup and Intra-class mixup with angular margin.
16

---

Under review as a conference paper at ICLR 2022
Energy Score
FTR95
In-Dataset
Oal Dataset
Intra
Intra + Cos(P)
Intra
Intra Cost!)
Intra
Intra Cos(P)
Gaussian Noise
0.8390
0.0910
0.9076 0.0591
0.7318 + 0.1090
0,8164
0.0887
0.2829 0.1262
0.1654
0.0900
Uniform Noise
0.9249
0.0102
0.9172
0.0299
0.8322 1: 0.0862
0,5710 0.0696
0,1192 1: 0.05M
0.0864 1: 0.0416
SVHN
0.9266 0.0111
0.9272 0.0116
0.9422 = 0.0088
0,9432 0.0096
0.1839 # 0.0294
0.1812 + 0.0286
0,7548
0.0152
0.0170
0.0437
0.3380
CIFAR-10
LSUN
0.9346 0.0052
0.9356
0.0054
0.7747 = 0.0169
0.7740 0.0159
0.2343 # 0.0136
0.2273 + 0.0162
TinllmugeNet
0,9057 0.0044
0.9060
0,0039
0.8898 =: 0.0049
0,8898
0.0043
0.3584
0.0197
0.3581 1: 0.0233
Places365
0.9693
0.3128
Average
0.9029 0.0319
0.9195 a: 0.0168
0.8421 = 0.0870
0,8635 0.0711
0.2762 #: 0.0987
0.2384 + 0.0938
Gaussian
0,8128
0.6856 0.0920
0.0879
0.0914
0.2450 0.0788
Uniform Noise
0.6778 = 0,0991
0,7511 :: 0.0811
0,3565 = 0.0740
0.5951 + 0.0717
0.4136 :: 0.1134
0.3513 :: 0.0946
SVHN
0.7686
0.0230
0.7724 0.0240
0.8269 0.0137
0,8312
0.0146
0.4757 0.0466
0,4729 0.0481
Textures
0.6860 0.0087
0.7078
0.0068
0,4850 :: 0.0068
0,5022
0.0061
0.7586 =: 0.0162
0.7043 :: 0.0102
CIFAR-100
LSUN
0.7295
0.0038
0.7278 à: 0.0040
0.3175 =: 0.0054
0 4: 0.0054
0.6662 4: 0.0075
0.6719 d: 0.0072
Tiny ImageNet
0.7942
0.7594
0.0088
0.6171 0.0050
Places365
0.7665 = 0.0041
0,7644 à: 0.0041
0.9043 =:: 0.0025
0.9037 0.0026
0.6574 4: 0.0082
0.6643 4: 0.0087
Avenage
0,7479
0.0481
0,7630
0.0428
0.6565 + 0.1769
0.6698 0.1735
0,5535 +: 0,1337
0.5324 +: 0.1688
Gaussian Noise
0.8406
0.0894
0.9078
0.050)
0,6500 =: 0.1680
0.7647 0.1187
0.4952 0.2378
0.3170 +: 0.1596
Uniform Noise
0.8251 + 0.0875
0,8685 à: 0.0614
0.6260 =: 0.1636
0,6893 0.12
0.5352 4: 0.2273
0.4348 d: 0.1837
CIFAR
0.7858
0.0208
0.0459
0.1918 0.0143
Textures
0.8639 + 0.0245
0.9126 + 0.0158
0.6281 + 0.0447
0.7269
0.0141
0.6217 + 0.1184
0.3978 + 0.0896
SVHN
LSUN
0.8928
0,0210
0.9423 0.0092
0.5053 4: 0.0525
0,6789 0.0357
0.4416 0.0824
0,2412 0.0371
ImyliaggNee
0.9197
0.0048
0.7994
0.0244
00107
0.3210
0.0475
0.1729
Places365
0.9111 = 0.0164
0.9513 a: 0.0072
0.9235 =: 0.0134
0,9590 1: 0.0058
0.3668 4: 0.0740
0.1980 :: 0.0310
Avemge
0,880
0.0350
0,9273
0.030)
0,7026 0.1300
0.7975
0.1015
0,1003
0.2791 0.0976
Gaussian Noise
0.8491 + 0.1009
0.7804 + 1048
0.7936 + 0.1324
0,7042
0.1267
0.3730 + 0.1908
0.4802 + 0.1487
Uniform Noise
0,4347
0.1476
0.3757 0,1311
0,4313 :: 0.0622
0.1043
0.0469
0.7789 0,1012
0,8095 0.0831
0.8127
0,8950 0.0267
0,0286
0.0585
0.5699
CIFAR- 100
0.5958 = 0.0032
0.5929 1 0.0029
0,5691 :: 0.0028
0.5631 1: 0.0032
0.8663 1: 0.0029
0,8642 1: 0.0011
Imageegee
0.5874
0,1200
0.0131
0.8494 0.0083
ISUN
0.5598
0.0069
0.5736
0.0066
0.2461 + 0.0048
0,2525 0.0044
0.8457 + 0.0085
0.8335 + 0.0084
Plave3365
0,6087 0.0037
0.6105
0.0039
0.8265 :: 0.0016
0,8257 0.0017
0.8189 0.0015
0,8113 0.0010
Avemge
0.1250
0.5974
0.2277
0.7254
Average
0.7918 0.1318
0.8049 +: 0,1471
0.6996 =: 0.1872
0,7253 0.1889
0,5010 # 0.2135
0.4488 # 0.2420
Table 10: Expanded results for Energy Score comparing Intra-class mixup and Intra-class mixup with angular margin.
In-Dix
AUTRE
COBAR-10
01870 =
0,8765
0.8478
000544
6,8145 0U002
0LM232
0.3386
B.6734 0.1512
0,5555 0,0092
0L6792 0,0000
Table 11: CIFAR-10 (In-Dist) vs CIFAR-100 (OoD) detection results for Maximum Softmax Proba- bility detector.
17

---

Under review as a conference paper at ICLR 2022
Maximum Softmax Probubility Detector with Supplemental Angular Margin trained crained with FRM
AUROC
AUPPR
FPR95
In-Datnset
Oal) Dataset
ERM
ERM - coe (0)
ERM
ERM + cos (9)
ERM
ERM ove (9)
Goussian Noise
0,6991
0.2232
0,8516
0.0843
0.6151 + 0.2250
0,7479 0.1313
0.6103 : 0,3818
02192
0.1015
0.2029
0.2015
SVHN
0.8111 0.0901
0.891 0.0249
0.9008
0.0402
0,9300 0.0155
0.6102 0.2288
0.2339 + 0.0399
0.0058
0.0091
0.1196
0.2940
CIFAR-10
LSUN
0.8598 0.0177
0.9050
0.0041
0.6262 # 0.0203
0,6821 0.0117
0.6217 # 0.1507
0.3098 # 0.0216
Tin
0,834 0.0143
0,8778
0.0041
0.8169 4: 0.0078
0.8471 0.0044
0,7070 0,1248
0.4295 0.0077
0.3935
Average
0.234 = 0.0538
0,8876 1: 0.0157
0,7803 =: 0.1113
0.2006 0.0907
0.6079 1: 0.1
0.2974 : 0.0812
0.2596
0.0665
0.0181
0.0242
0.6752 0.0729
Uniform Noise
0.7941 = 0.0627
0.8030
0.0547
0.6889 = 0.0805
0.6894 0.0709
0.3759 #: 0.0618
0.3441 # 0.0441
SVHN
0,7426
0,0273
0,7686
0.0187
0.8507 :: 0.0173
0,597 0.0144
0.6120 0.0456
0.5155 0.0188
0.0079
CIFAR-100
LSUN
0.7020 = 0.0061
0,7101 à: 0.0048
0.3591 =: 0.0061
0,3666 0.0058
0.7094 4: 0.0059
0,7182 0.0063
Tiny ImggeNe
0.7382
0.0026
0.0023
0.6900 0.0042
0.0025
0.6239 0.0047
Places365
0.7201
0.0040
0.7327
0.0035
0.8818 =: 0.0022
0.8881 0.0020
0.6992 4: 0.0066
0.7068 :: 0.0036
Avenage
0.6645
0.1680
0.6955
0. 1336
0.6204 =: 0,1990
0.6343 0.1932
0.6577 0,1304
0.6044 +: 0.1233
Gaussian Noise
09218
0.0141
0.9376
0.0102
0.7770 +: 0.02M8
0.0088 0.0263
0.0560
0.1884 0.0338
Uniform Noise
0.9247 = 0.0061
0.9391 à: 0.0038
0.7845 =: 0.0061
0,8097 0.0065
0.2494 d: 0.0335
0.1863 +: 0.0186
CIFAR-100
0,9267
0.0033
0.7993 0.0067
0.0257
0.1919
Textures
0.9066
0.0059
0,9250
0.0053
0.6651 =: 0.0154
0.7068
0.0150
0.3723 1: 0.0364
0.2685 1: 0,0310
SVHN
L.UN
0,9258
0.0051
0.9409
0.0033
0.5614 + 0.0133
0.6188 0.0136
0.2534 +: 0.0260
0.3912 0.0155
TnylmageNet
0.930
0.0022
0.0066
0.0221
0.1738 0.0121
Places365
0.9281 = 0.0047
0.9442 à: 0.0030
0.9324 =: 0.0034
0,9460
0.0026
0.2436 4: 0.0256
0.1781 +: 0.0138
Avenge
0,9235
0.0074
0.0063
0,7610
0.10W8
0.7927
0.0442
0.1969 0.0299
Gaussian Noise
0,5506
0.1104
0.5598 0 1041
0,4911 + 0.0665
0.4941
0.0638
0.7374 + 0.1040
0.7174 + 0.1011
Uniform Noise
0,2977
0.0686
0.3028
0.0708
0.3726 1: 0.0220
0.3743
0.0230
0.9064 0.0255
0.8966 0.0242
0.6747
0.7956
0.0237
0.7025
CIFAR 100
0.5774 = 0.0034
0,5775 1: 0.0035
0,5191 =: 0.0038
0.0040 0.5496
0.8718 1: 0.0035
0,8730 1: 0.0032
ThyyIaggeNe
Textures
0.5555
0.0078
0.5609 0.0078
0.3801
0.0099
0.3818
0.8702 0.0056
0.8580 0.0046
LSUN
0.5897
0.005
0.5901
0.0048
0.2709 =: 0.0011
0,2697 0.0042
0.8451 :: 0.001
0,8437 0.0051
Places365
0.6117 + 0.0045
0.6134 + 0.0037
0.8309 + 0.0014
0,8308 0.0015
0.8334 + 0.0042
0.8288 + 0.0041
Avemge
0.99
0.0658
Average
0.7406 1 0.1771
0.7693 1 0.1761
0.6722 :: 0.1915
0.6939 0.19-18
0.5895 1: 0.2243
0.1789 1: 0.2600
Table 12: Expanded results for Maximum Softmax Probability Detector with Supplemental Angular Margin trained with ERM.
18

---

Under review as a conference paper at ICLR 2022
Maximum Sofhmax Prohability Desector with Supplemental Angular Margin triindd with Inter-class Mixup
AUROC
AUPP
FPR95
In-Datnset
Oal) Dataset
Inter
Inter + cos (P)
Inter
Inter tos (0)
Inter
Inter coe (0)
Goussian Noise
0,9170
0.0171
0.9573 0,0148
0.9073 + 0.0816
0,9260
0.0777
0,1290 : 0.1128
0.3077 1: 0.1152
0,9462
0,9059 0.0621
0.0960
SVHN
0.8539 0.0386
0,8523 + 0.0408
0.9263 + 0.0167
0.9245 0.0179
0.6285 + 0.1922
0.6131 + 0.1681
0,8172
0,7677 0.0147
0.0117
0.0782
0.7583 0.0652
CIFAR-10
LSUN
0.8888 0.0063
0.9064 0.0061
0.7074 + 0.0111
0.7446
0.0122
0.5010 + 0.0439
0.4077 # 0.0451
TinglmogeNet
0,8484 0.0016
0.8441 0.0021
0.8456 4: 0.0052
0,8496
0.0038
0.6929 0.0187
0,7481 0.0022
0.6163 0.0425
Average
0,8337 = 0.0419
0,8895 1: 0.0445
0,8387 :: 0.0837
0,7702 0.0756
0.5037 :: 0.2445
0.4862 :: 0.2502
0,8916
0,9012 0.0502
0.7963 0.0875
0.0810
0,2005
0.1832 0.0712
Uniform Noise
0.7464 = 0.2002
0,7559
0.1663
0.6705 = 0.1818
0.6708 0.1714
0.3751 #: 0.2218
0.3527 :: 0.2003
SVHN
0,7460 0,0302
0,7569 0.0315
0.8423 :: 0.0150
0,488 0.0199
0.6137 0.0796
0.5830 0.0761
0.5502 + 0.0047
CIFAR-100
LSUN
0,7417 = 0.0037
0.7479 à: 0.0034
0.4014 =: 0.0076
0.4065 0.0078
0.7162 =: 0.0130
0.6871 +: 0.0113
Tiny ImageNe
0.7857
0.0022
0.0033
0.0133
0.6047 0.0109
Places365
0.7560 = 0.0019
0.7645 a: 0.0016
0.8989 =: 0.0011
0.9026 0.0011
0.7035 4: 0.0043
0.6716 =: 0.0037
Avenage
0,7733
0.0503
0,7828
0.050
0,7006 =: 0.1619
0,7079 0.1619
0.5549
0,1786
0.5265 0.1732
Gaussian Noise
08198
0.1368
0.0841
0.6935 + 0.2014
0.7639 0.162U
0.6185 0.4020
0.4099 0.2984
Uniform Noise
0.7706 =: 0.1228
0.8136 à: 0. 1033
0.6385 =: 0.1586
0.6726 0.1441
0.7598 4: 0.3313
0.6637 :: 0.2931
CIFAR-100
0,340
0.0071
0.0444
0,8672 0.0108
0.3282 0.0841
0.2184 0.0385
Textures
0,8851
0.0253
0.9136
0.0181
0.7391 d: 0.0417
0,7728
0.0336
0.7703 1:: 0.1736
0.5329 1: 0.1445
SVHN
L.UN
0,353
0.0110
0.9529
0.0016
0,7183 +: 0.0313
0,7612 0,0241
0.3332
0,1233
0.1886
0.0421
ThyllageNet
0.9354
0.0662
0.8732
0.0176
0.0133
0.3254 0.10044
0.2036
0.0478
Places365
0.9355
0.0133
0.9521 1: 0.0083
0.9540 =:: 0.0087
0.0059 0,9642
0.3307 4: 0.138
0.1968 1: 0.0607
Avenge
0,8800
0.0624
0.0477
0.7W34 0.1065
0,8188
0,4952
0.1967
0.1787
Gaussian Noise
0,4041 + 0.1878
0.4063 + 0.1324
0,4355 + 0.1123
0.4176
0.0562
0.7842 + 0.1274
0.7800 + 0.0999
Uniform Noise
0,3944
0.12M
0.3678
0,1272
0.1101 :: 0.0188
0.0469 0,4008
0.7805 0.1021
0,7911
0.1043
0.6042
0.0750
0.0607
0.0843
0.7622
CIFAR 100
0.5680 =: 0.0098
0.5671 1: 0.0094
0,5122 =: 0.0095
0.5393 0.0096
0.8819 =: 0.0034
0.8818 1: 0.0023
ThyyIaggeNe
Textures
0.5435
0.0143
0.5195
0.0093
0.3756
0.0091
0.3506
0.0062
0.8786 0.0094
0.8793
0.0089
LSUN
0.5727 1: 0.0263
0.5750
0.0195
0.2615 1: 0.0186
0,2598 0.0112
0.8639 :: 0.0185
0.8581 1: 0.0188
Places365
0,6032
0.0106
0.6040 + 0.0148
0.8270 + 0.0061
0,2588 0.0088
0.8424 + 0.0070
0.8377
0.0088
Avemge
0.1901
Average
0.7680 1 0.1588
0.7693 1: 0.1761
0.71 :: 0.1914
0.7235 0.1977
0.5895 1: 0.2213
0.5462 :: 0.2500
Table 13: Expanded results for Maximum Softmax Probability Detector with Supplemental Angular Margin trained with Inter-class Mixup.
19


Reviews:
[{"review_id": "pHbdQ7sY93F", "reviewer": "Reviewer_WTWp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Overall the paper is clearly written, although the experiments and results could be improved. \n- Good coverage of the related work\n- Very relevant topic\n\nWeaknesses\n- This work is presented as an alternative to standard empirical risk minimization (ERM) or the setup of mixup (Zhang et al, ICLR 2018), where ERM is replaced by vicinal risk minimization (VRM, Chappelle et al, NIPS 2000). What learning strategy is used here, in opposition to these two works, is never stated. \n- The experimental setup is not clear, which limits reproducibility and makes difficult to interpret the obtained results. What exactly is considered as in-distribution and what samples are OoD and how are they generated? (see detailed comments)\n- The proposed comparisons against ERM and mixup are not necessarily relevant. ERM is the standard approach to train a model and mixup proposes a data augmentation strategy to make a model more robust to adversarial samples. This work, instead focuses on the detection of OOD samples. As such, it should compare itself with methods addressing the same problem and not directly against ERM and mixup (table 1).\n\nDetailed comments\n- Please acknowledge previous work on angle-based outlier detection [1], as it closely relates to this work.\n- The angular margin is estimated w.r.t the decision boundary (see eqs. 3 and 4). Therefore, there is an error in the illustration in Fig. 1a.\n- Differently from mixup, in this work \\lambda does not follow a Beta distribution. Moreover, no details are provided on how it is chosen. Please comment.\n- If the label should not change, second line of Eq. 2 could be omitted. Otherwise, there is no guarantee that \\hat{y} will have the same value. This would only hold when \\lambda=1 or 0 and y_i=y_j=1 or 0.\n- Eq 5 implies that new data is being generated, i.e. data augmentation. Is this the case? What happends with the original samples? I suppose they are undesired since they have high variance.\n- Could the authors motivate why the angular margin needs to be used coupled with another metric (eq. 7) and not on its own?\n- The experimental setup is not clear. The paper misses to clearly establish what is an OoD sample/set on each of the experiments. In which way Gaussian and uniform noise are used for this purpose?\n- At inference time, when is a sample considered OOD?\n- The AUROC is not a good measure in OOD detection problems, since usually the majority class dominates. AUPRC should be favored. Interestingly, it is mixup that fairs best in that scenario, despite not being a method designed for OOD detection. Please comment \n\nMinor\n- There are typos in the plots in figure 1b,c,d\n- Table 2 is unnecessary and may be omitted\n\nReferences\n[1] H.-P. Kriegel, M. S hubert, and A. Zimek. Angle-based outlier detection in high-dimensional data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, pages 444–452. ACM, 2008.\n", "summary_of_the_paper": "Inspired from inter-class mixup (Zhang et al, ICLR 2018), where data augmentation is used to train models more robust to adversarial samples, this work proposes intra-class mixup to reduce the variance of in-distribution samples, i.e. the training set, and improve the capacity of a trained model to detect out-of-distribution samples at inference time. The key difference between the two works is that the here proposed does the mixup within samples belonging to the same class. \n\nIn addition to this, the work propose to use the angular margin, i.e. the angle between the normal of the decision boundary of a neural net (obtained from the weights of the last layer) and an unmixed sample, to detect OOD samples. The cosine of such angle shall be coupled with an OOD method to perform OOD detection. ", "summary_of_the_review": "This work proposes intra-class mixup coupled with an angular mesaure for OOD detection. The idea of an angle-based measure has been explored in the past (not mentioned in the work) and it is interesting to propose it in the context of deep nets.\n\nThe work, however, misses some key elements in the presentation of the method and has a weak experimental setup.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jOC-AjjE8rY", "reviewer": "Reviewer_DAih", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A strong point of the paper is that they evaluate their method in many different settings, i.e. on 3 OOD methods on 4 different in-distribution. They also report AUROC, AUPR and FPR95, which is good. They do not fit special hyperparameters for each specific out-distribution, making their comparison fair in my opinion. Furthermore they test their OOD detection performance on many different out-distributions. \n\nThat being said, one OOD detection task that they should definitely add is CIFAR10 vs. CIFAR100 and vice-versa, as this task is known to be particularly difficult.\n\nOverall, the paper's use of the angular margin is not well-motivated. Especially the creation of a new OOD score by simply adding the angular margin to other OOD scores is completely ad-hoc. On top of this, it should be noted that their angular margin score depends on arbitrary bias values in the last layer. If one adds some bias shift $b$ to every bias in the last layer then it shifts the origin and thus modifies the angular margin. However, the resulting network's loss would be identical to the original and thus there is a-priori no reason that training should favor one network over the other. All their OOD performance measures would also remain invariant except for the angular margin score. There should at least be a discussion about why despite all this, the angular margin score is still helpful.\n\nI am not sure about the strength of their baselines. Comparing for example with the results from [1], they used the same architecture but their simple MSP model beats even the best results in Table 9 (on all but one of the datasets and metrics that both report). This doesn't even mention the fact that even if this wasn't the case the paper would still only be comparing their results to weak baselines. Basically, the authors argue that one can improve all three of the baselines that they study using their technique. But even the improved version's performance is worse than, for example outlier exposure [Hendrycks et. al. 2019] or Mahalanobis distance [Lee et. al 2018], so why use the weak techniques in the first place?\n\nOne point of confusion for me was that the standard deviations in Table 1 were seemingly computed across the dataset, whereas in all other tables they were computed across different random seeds. I really think this could be clarified explicitly in the Table's caption (especially given that the authors had lots of space left).\n\nI was also confused how exactly the standard deviations of the averages were computed in Table 4, Table 5 and in the appendix. Are all runs aggregated and then the standard deviation across all of them is computed? Or is the average computed for each seed and then the standard deviation of that is reported?\n\nMore issues:\n- In Table 1 the caption reads \"Intra-class mixup has consistently has better separability\" (small typo here!), even though it clearly isn't true for CIFAR-10. \n- Table 2 is fairly useless to the paper and appears to have been added to pad out the length.\n- In Table 3, we see that Inter generally performs better in terms of accuracy (the gap being quite large on CIFAR-100). This should at least be acknowledged and discussed by the authors. Arguably, this issue is strong enough as an argument against using their method just by itself, so it cannot be glossed over.\n- In Table 4, there are 6 instances of wrong boldfacing, making the authors' results look better than they are.\n\nMinor comments:\n- The citation style is very hard to read. Presumably the authors used \\citet (instead of \\citep) even for citations that are not directly referenced in the sentence.\n- \"Angular\" is misspelled in all axis labels in Figure 1.\n- All Table captions should be above the tables, not below.\n- All equations are missing punctuation.\n- Wenn formatting $\\cos$ or $\\arccos$ in mathmode one should use \\cos and \\arccos (instead of $cos$ and $arccos$). Similarly, I would recommend to use \\mathrm for worded subscripts, i.e. $R_\\mathrm{expected}$ instead of $R_{expected}$.\n- Several figure and table captions end without punctuation. \n\n[1] Towards neural networks that provably know when they don't know, Alexander Meinke, Matthias Hein, at ICLR20", "summary_of_the_paper": "The authors propose to use intra-class mixup (i.e. linearly interpolating images from the same class) as a form of data augmentation. They further define the angular margin and propose to add it to out-of-distribution detection scores in order to improve the performance. They evaluate their method's OOD performance on three different OOD scores and show improvements.", "summary_of_the_review": "I recommend to reject this paper. The theoretical contribution is marginal and mostly unmotivated and the empirical results only compare to relatively weak baselines. On top of this, the paper clearly lacks polish. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "egeltn0cMO", "reviewer": "Reviewer_Fzqo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nUsing intra-class mixup to increase angular separability between in-distribution and OoD and hence OoD performance.\n\nAdding angular margin to the OoD scores improves OoD performance.\n\nWeaknesses:\n\nThe improvement in performance is not significant.\n\nTable 3: the purpose and analysis of Table 3 could be further discussed.   The \"baseline accuracy\" on In-distribution and/or OoD data?  It seems to be in-distribution data.  Accuracy of Intra-mixup is lower in 3 out of 4 datasets, any suggesions on reasons?\n\nEquation 8: theta_o and theta_i -- are they averages of all the samples in the in-distribution and OoD datasets?\n", "summary_of_the_paper": "The authors propose adding angular distance in the OoD score and additional training samples via intra-class mixup.  Angular margin/distance is defined as the angle between the weight vector from a class and an instance (ie. arccos of the dot product of the unit weight vector and the unit vector for a instance.  Intra-class mixup generates samples by interpolating between two samples of the same class.   Angular spread is the standard deviation of the angular margin for a given dataset.  Intra-class mixup reduces angular spread in in-distribution data and increases angular spread between in-distribution and OoD data.   Angular separability is defined as the squared difference of in-distribution angular margin and OoD angular margin, normalized by the sum of their standard deviation.\n\nWith four datasets, they show that intra-class mixup increases angular separability in 3 datasets.  With four datasets and 3 different OoD techniques, intra-mixup is more accurate in AUROC and FPR95, but not AUPR.   Adding the cosine of angular margin in the OoD scores generally improves performance over using OoD scores alone.\n\n\n\n", "summary_of_the_review": "The ideas of using intra-class mixup to increase angular separability and adding angular margin to the OoD scores are interesting.  \n\nHowever, the presentation can be improved.\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RFnsgvJNs74", "reviewer": "Reviewer_xKKx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\nThe paper is well written and easy to follow. The method of intra-class mixup makes sense to me. The results (including both tables and figures) are clearly presented.\n\n\nWeakness:\n\nThe results show that adding Cos(\\theta) on top of the model with intra-class mixup is helpful. I wonder if a regular model without intra-class mixup can also get improvement by adding Cos(\\theta) to the OOD metric. \n\nThe method has been compared with simple baselines MSP, ODIN, and energy score. It would be great if the method can provide supplementary improvement for other recently developed methods such as Bayesian NNs or Gaussian Process.\n\nThe method is evaluated mostly on far-OOD benchmarks (see Table 4 and 6, 7, 8). I wonder if the proposed method can work well for near-OOD benchmarks such as CIFAR-10 vs CIFAR-100. \n", "summary_of_the_paper": "The paper proposes to use intra-class mixup to train OOD detectors. Adding intra-class mixup, the separability between in-distribution and out-of-distribution data is improved. The methods are evaluated in multiple OOD benchmark datasets, showing improvement of 4-6% over the methods without intra-class mixup. \n\nThe paper also interestingly shows that the cosine of the angular margin is also a useful measure for OOD detection. It can be added to other regular OOD measures to further improve the performance. \n\n", "summary_of_the_review": "The paper proposes an interesting idea of using intra-class mixup to improve separability between in- and out-of-distribution data. The paper also shows that cos(\\theta) can be a useful measurement for OOD detection, and can be added to other OOD measurements like MSP and ODIN. It would be great if more results can be provided", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use intra-class mixup supplemented with angular margin to improve OOD detection. 

Strengths:
+ Simple idea
+ Experiments on multiple datasets (although mostly focused on image benchmarks)

Weaknesses:
- Justification for the idea could be improved. It'd be nice to understand when we expect this to (not) work.
- Differences from prior work "Angle-based outlier detection in high-dimensional data" could be better explained.

While the paper has some interesting contributions, the reviewers and I feel that the current version falls short of the acceptance threshold. I encourage the authors to revise and resubmit to a different venue.
Decision: Reject