How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?
Authors: Yifei Ming
Yiyou Sun
Ousmane Dia
Yixuan Li
Venue: ICLR 2023 poster (2023)

Abstract: Out-of-distribution (OOD) detection is a critical task for reliable machine learning. Recent advances in representation learning give rise to distance-based OOD detection, where testing samples are detected as OOD if they are relatively far away from the centroids or prototypes of in-distribution (ID) classes. However, prior methods directly take off-the-shelf contrastive losses that suffice for classifying ID samples, but are not optimally designed when test inputs contain OOD samples. In this work, we propose CIDER, a novel representation learning framework that exploits hyperspherical embeddings for OOD detection. CIDER jointly optimizes two losses to promote strong ID-OOD separability: a dispersion loss that promotes large angular distances among different class prototypes, and a compactness loss that encourages samples to be close to their class prototypes. We analyze and establish the unexplored relationship between OOD detection performance and the embedding properties in the hyperspherical space, and demonstrate the importance of dispersion and compactness. CIDER establishes superior performance, outperforming the latest rival by 13.33% in FPR95. Code is available at https://github.com/deeplearning-wisc/cider.

Published as a conference paper at ICLR 2023
HoW TO EXPLOIT HYPERSPHERICAL EMBEDDINGS FOR DUT-OF-DISTRIBUTION DETECTION?
Yifei Ming', Yiyou Sun', Ousmane Dia², Yixuan Lia
Department of Computer Sciences, University of Wisconsin-Madison" Meta?
[alvinming, sunyi you, sharonli]@cs wtt.ccsswiiicece... ousamdia@meta.coom
ABSTRACT
Out-of df-tiiiiiiiitttionnn (OOD) detection is critical task for reliable machine learn- ing. Recent advances in representation learning give rise to distance-based OOD detection, where testing samples are detected OOD if they are relatively far away from the centroids or prototypes of in-distribution (ID) classes. However, prior methods directly take off-the ff-thh--sheff contrastive losses that suffice for classi- fying ID samples, but are not optimally designed when test inputs contain OOD samples. In this work, we propose CIDER, novel representation learning frame- work that exploits hyperspherical embeddings for OOD detection. CIDER jointly optimizes two losses promote strong ID-OOD separability: dispersion loss that promotes large angular distances among different class prototypes, and a com- pactness loss that encourages samples be close their class prototypes. We analyze and establish the unexplored relationship between OOD detection perfor- mance and the embedding properties in the hyperspherical space, and demonstrate the importance of dispersion and compactness. CIDER establishes superior per- formance, outperforming the latest rival by 13.33% in FPR95. Code available at wllhtps:: /tttt:::/gtttie com dwwppeeeppeeeeeeeeee www.m//deeeeaanngggwaeeee www.://rriiiiiisccciiiid......
INTRODUCTION
When deploying machine learning models the open world, it is important to ensure the reliabil- ity of the model in the presence of out-of-distribution (OOD) aoputs-samples from an unknown distribution that the network has not been exposed to during training. and therefore should not be predicted with high confidence at test time. We desire models that are not only accurate when the input is drawn from the known distribution, but are also aware of the unknowns outside the training categories. This gives rise the task of OOD detection, where the goal is to determine whether an input is in-distribution (ID) or not.
A plethora of OOD detection algorithms have been developed recently, among which lestance-based methods demonstrated promise (Lee et al. 2018 Xing et al., 2020). These approaches circumvent the shortcoming of using the model's confidence score for OOD detection, which can be abnor- mally high on OOD samples (Nguyen et al. 2015) and hence not distinguishable from ID data. Distance-based methods leverage feature embeddings extracted from model, and operate under the assumption that the test OOD samples are relatively far away from the clusters of ID data.
Arguably, the efficacy of distance-based approaches can depend largely on the quality of feature embeddings. Recent works including SSD+ (Sehwag et al.) 2021) and KNN+ (Sun et al.) 2022) directly employ off-the-shelf contrastive losses for OOD detection. In particular, these works use the supervised contrastive loss (SupCon) (Khosla et al.) 2020) for learning the embeddings, which are then used for OOD detection with either parametric Mahalanobis distance (Lee et al., 2018 Sehwagg al. 2)21) or non-parametric KNN distance (Sun et al. 2022). However, existing training objectives produce embeddings that suffice for classifying ID samples, but remain sub-optimal for OOD detection. For example, when trained distance between ID and OOD data is only 29.86 degrees in the embedding space, which is too small for effective ID-OOD separation. This raises the important question:
CIFAR-10 using SupCon loss, the average angular

---

Published as a conference paper at ICLR 2023
How to exploit representation learning methods that maximally benefit OOD detection?
In this work, we propose CIDER, Compactness and DispErsion Regularized learning framework designed for OOD detection. Our method is motivated by the desirable properties of hyperspherical embeddings, which can be naturally modeled by the von Mises-Fisher (vMF) distribution. vMF is a classical and important distribution in directional statistics (Mardia et al. 2000), is analogous to spherical Gaussian distributions for features with unit norms. Our key idea is to design an end- to-end trainable loss function that enables optimizing hyperspherical embeddings into a mixture of vMF distributions, which satisfy two properties simultaneously: (1) each sample has higher proba- bility assigned to the correct class in comparison to incorrect classes, and (2) different classes are far apart from each other. To formalize our idea, CIDER introduces two losses: a dispersion loss that promotes large angular distances among different class prototypes, along with a compactness loss that encourages samples to be close to their class prototypes. These two terms are complementary to shape hyperspherical embeddings for both OOD detection and ID classification purposes. Unlike previous contrastive loss, CIDER explicitly formalizes the latent representations as vMF distribu- tions, thereby providing a direct theoretical interpretation of hyperspherical embeddings.
In particular, we show that promoting large inter-class dispersion is key to strong OOD detec- tion performance, which has not been explored in previous literature. Previous methods including SSD+ directly use sheflf SupCon loss, which produces embeddings that lack sufficient inter- class dispersion needed for OOD detection. CIDER mitigates the issue by explicitly optimizing for large inter-class margins and leads to more desirable hyperspherical embeddings. Noticeably, when trained CIFAR-10, CIDER displays relative 42.36% improvement of ID-OOD separabil- ity compared to SupCon. We further show that CIDER'S 's strong representation can benefit different distance-based OOD scores, outperforming recent competitive methods SSD+ (Sehwag al.. 2021 and KNN+ (Sun et al. 2022 by significant margin. Our key results and contributions are:
1. We propose CIDER, novel representation learning framework designed for OOD detec- tion. Compared to the latest rival (Sun et al. (2022), CIDER produces superior embeddings that lead to 13.33% error reduction (in FPR95) on the challenging CIFAR- benchmark.
2. We are the first to establish the unexplored relationship between OOD detection perfor- mance and the embedding quality in the hyperspherical space, and provide measurements based on the notion of compactness and dispersion. This allows future research to quantify the embedding in the hyperspherical space for effective OOD detection.
3. We offer new insights on the design of representation learning for OOD detection. We also conduct extensive ablations to understand the efficacy and behavior of CIDER, which remains effective and competitive under various settings, including the ImageNet dataset.
2 PRELIMINARIES
We consider multi-class classification, where X' denotes the input space and y (,, C} denotes the ID labels. The training set Din = {xx, y:) | is drawn i.i.d. from Pxy Let Px denote the marginal distribution over X, which is called the in-distribution (ID).
nut-of-distribution detection. OOD detection can be viewed as a binary classification problem. At test time, the goal of OOD detection is to decide whether sample x E X' is from Px (ID) not (OOD). In practice, OOD is often defined by a distribution that simulates unknowns encountered during deployment, such samples from irrelevant distribution whose label set has no intersec- tion with y and therefore should not be predicted by the model. Mathematically, let Deod denote an OOD test set where the label space yood y 0. The decision can be made via a level set estimation: (x(x) = 1xS(x) 2 }}, where samples with higher scores S(x) are classified as ID and vice versa. The threshold À is typically chosen so that a high fraction of ID data (e.g. 95%) correctly classified.
Hyperspherical embeddings. A hypersphere is topological space that is homeomorphic to standard n-sphere, which is the set of points in (n 11 dimeennsioaa Euclidean space that are located at constant distance from the center. When the sphere has unit radius, it is called the unit hyper- sphere. Formally, an n-dimensional unit-hypersphere STh := {z E 1B*1 [|z|| = 1}. Geometrically, hyperspherical embeddings lie on the surface of hypersphere.
2

---

Published as a conference paper at ICLR 2023
Penultimate Layer
Hyperspherical Embedding
Class prototype
Augmented
Projection
Embedoing an instance
inputs
cestance-to-poootyyee ettraction (from Compactness loss feoe Prototype to prototype dispersion (from Dispersion loss Cax)
Figure 1: Overview of our compactness and dispersion regularized (CIDER) learning framework for OOD detection. We jointly optimize two complementary terms encourage desirable properties of the embedding space: (1) dispersion loss encourage larger angular distances among different class prototypes, and compactness loss to encourage samples to close to their class prototypes.
METHOD
Overview. Our framework CIDER is illustrated in Figure I The general architecture consists of two components: (1) a deep neural network encoder X' + R that maps the augmented input X to high dimensional feature embedding / (x) (often referred to as the penultimate layer features); (2) projection head h R + Rd that maps the high dimensional embedding f(x) to a lower dimensional feature representation 2 == hx)()). The loss is applied to the normalized feature embedding z e/|/2|2. The normalized embeddings are also referred as hyperspherical embeddings, since they are on unit hypersphere. Our goal is to shape the hyperspherical embedding space so that the learned embeddings can be mostly effective for distinguishing ID vs. OOD data.
3.1 MODEL HYPERSPHERICAL EMBEDDINGS
The hyperspherical embeddings can be naturally modeled by the von Mises-Fisher (vMF) distribu- tion, a classical and important distribution in directional statistics (Mardia et al. 22000). In particular, vMF is analogous to spherical Gaussian distributions for features with unit norms ( 2 1). The probability density function for a unit vector z E IRd in class e is defined as:
pa(z; Pe. R) = Za(k) exp (Fi14 2),
(1)
where pr is the class prototype with unit norm, x 2 indicates the tightness of the distribution around the mean direction te.. and Z²(k) is the normalization factor. The larger value of x. the stronger the distribution is concentrated the mean direction. In the extreme case of k 0, the sample points are distributed uniformly the hypersphere.
Under this probability model, an embedding vector z assigned to class e with the following nor- malized probability:
Za (k) exp (kity 2)
P(y = c|z; (1,11175))
E/=1 Ze (K) exp (1ipeT72
exp (m! z/T)
(3)
Σラ exp (13. Z/T)
where x 1. Next, we outline our proposed training method that promotes class-conditional vMF distributions for OOD detection.
3.2 HOw TO OPTIMIZE HYPERSPHERICAL EMBEDDINGS?
Training objective. Our key idea to design a trainable loss function that enables optimizing hyperspherical embeddings into mixture of vMF distributions, which satisfy properties simul- tancously: (1) each sample has a higher probability assigned to the correct class comparison to incorrect classes, and (2) different classes far apart from each other.
3

---

Published as a conference paper at ICLR 2023
To achieve we can perform maximum likelihood estimation (MLE) on the training data:
argmax II pluslz (Kis (ogs))))=))
(4)
where i is the index of the embedding and N is the size of the training set. By taking the negative log-likelihodd, the objective function is equivalent to minimizing the following loss:
exp (z, Mcto//T
1 N
Ceomp
log
(5)
Ej=1 exp (z. p2/T)*
where c(i) denotes the class index of sample x,, and is the temperature parameter. We term this compactness loss, since it encourages samples to be closely aligned with its class prototype.
To promote property 2
propose the dispersion loss, optimizing large angular distances among
different class prototypes:
dis = C Σ log C-1
Σ 1=1
1 + ijeu us/T.
(6)
Dog nD!
While prototypes with larger pairwise angular distances may not impact ID classification accuracy, they cru- cial for OOD detection, as we will show later in Sec-
Embeddiin of 000 sample Embeddiing ID sample Class protstype
Fox (20D)
tion 4.2 Since the embeddings of OOD samples lie in- between ID clusters, optimizing large inter-class margin benefits OOD detection. The importance of inter-class dispersion can also be explained in Figure Z where sam- ples in the fox class (OOD) are semantically close to cat (ID) and dog (ID). A larger angular distance (i.e. smaller cosine similarity) between ID classes cat and dog in the hyperspherical space improves the separabil- ity from fox, and allows for more effective detection. We investigate this phenomenon quantitatively in Section
dispersion
Truck HIDI
Cat UD!
Intra-class compactness
Scene (DOD)
Formally, our training objective CIDER (compactness and dispersion regularized learning) is:
Illustration of desirable hyper- spherical embeddings for OOD detection. As OOD samples lie between ID clusters, optimizing large angular distance among ID clusters benefits OOD detection.
Figure 2:
LCIDER = Lais AcCcomps
where X is the co-efficient modulating the relative impor- tance two losses. These two terms are complementary to shape hyperspherical embeddings for both ID classification and OOD detection.
During training, an important step is to estimate the class pro- use the mean vector of all training samples for each class and update it frequently during training. Despite its simplicity, this method incurs heavy computational toll and causes undesirable training latency. stead, the class-conditional prototypes can be effectively updated in an aioonentillmoovin--vveeee manner (EMA) (Li ct al. y020):
Prototype estimation and update.
totype fee for each class E {1, (I.,,...CC) One canonical way to estimate the prototypes
M Normalize(apu. + (1- a))), Vc E (1,
(8)
where the prototype μe for class is updated during training as the moving average of all embed- dings with label and z denotes the normalized embedding of samples of class We ablate the effect of prototype update factor a Section C
The pseudo-code for our method in Appendix A
Remark 1: Differences w.r.f. SSD+. We highlight three fundamental differences w.e.f. SSD+, in terms of training objective, test-time OOD detection, and theoretical interpretation. (1) At train- ing time, SSD+ directly uses off-the-shelf SupCon loss (Khosla et al. (2020), which produces em- beddings that lack sufficient inter-class dispersion needed for OOD detection. For example, when trained on CIFAR- 10 using SupCon loss, the average angular distance between ID and OOD data is only 29.86 degrees in the embedding space. In contrast, CIDER enforces the inter-class disper- sion by explicitly maximizing the angular distances among different ID class prototypes. As
4

---

Published as a conference paper at ICLR 2023
will show in Section 4.2 compared to SSD+, due to the explicit inter-class dispersion. (2) At test time, SSD+ uses the Ma-
CIDER displays relative 42.36% improvement of ID-OOD separability
halanobis distance <Lee et al. 2018). which imposes strong Gaussian distribution assumption hyperspherical embeddings. In contrast, CIDER alleviates this assumption with a non-parametric distance score. (3) Lastly, CTDER explicitly models the latent representations vMF distributions, providing direct and clear geometrical interpretation of hyperspherical embeddings.
Remark 2: Differences w.r.f. Proxy-basee methods. Our work also bears significant differences W.F. proxy-based metric learning methods. (1) Our primary task is OOD detection, whereas deep metric learning is commonly used for face verification and image retrieval tasks; (2) Prior methods such as ProxyAnchor (Kim et al.] (2020) lack explicit peootyyp-to-rrooyyye
dispersion, which we show is crucial for OOD detection. Moreover, ProxyAncho r initializes the proxies randomly and updates through gradients, while we estimate prototypes directly from sample embeddings using EMA. We provide experimental comparisons next.
EXPERIMENTS
4.1 COMMON SETUP
Datasets and training details. Following the common benchmarks in the literature, con- sider CIFAR-10 and CIFAR-100 (Krizhevsky et al.] 2009) as in-distribution datasets. For OOD test datasets, we use a suite of natural image datasets including SVEN (Netzer et al. 2011). P Places36 65 (Zhou et al. 2017). Textures (Cimpoi et al. 2014b), LSUN (Yu et al. 2015). and iSUN (u et al. 2015). In our main experiments, we use ResNet-18 18 as the backbone for CIFAR- 10 and ResNet-34 for 100. We train the model using stochastic gradient descent with momen- tum 0.9, and weight decay 10-4. To demonstrate the simplicity and effectiveness of CIDER, we adopt the same hyperparameters as in SSD+ (Sehwag et al. (021)) with the SupCon loss: the initial learning rate is 0.5 with cosine scheduling. the batch size is 512, and the training time is 500 epochs. We choose the default weight A = 2, so that the value of different loss terms are similar upon model initialization. Following the literature (Khosla et al. 2020), we use the embedding dimension of 128 for the projection head. The temperature т is 0. We adjust the prototype update factor o, batch size, temperature, loss weight, prototype update factor, and model architecture in our ablation study (Appendix C). We report the ID classification results for SSD+ and CIDER following the common linear evaluation protocol (Khosla et al.) (020). where linear classifier is trained on top of the nor- malized penultimate layer Teatures. More experimental details are provided in Appendix Code and data is released publicly for reproducible research.
OOD detection scores. During test time, we employ a distance-based method for OOD detection. An input x is considered OOD if it is relatively far from the ID data in the embedding space. By default, we adopt a simple non- -on-aaamettiic KNN distance (Sun et al. 2022), which does not impose any distributional assumption on the feature space. Here the distance is the cosine similarity with respect to the k-th nearest neighbor, which is equivalent to the (negated) Euclidean distance as all features have unit norms. In the ablation study, we also consider the commonly used Mahalanobis score (Lee et al. 2018) for a fair comparison with SSD+ (Sehwag et al., 202T).
Evaluation metrics. We report the following metries: (1) the false positive rate (FPR95) of OOD samples when the true positive rate of ID samples at 95%, (2) the area under the receiver operating characteristic curve (AUROC), and (3) ID classification accuracy (ID ACC).
4.2 MAIN RESULTS AND ANALYSIS
CIDER outperforms competitive approaches. T a wide range of competitive meth- ods for OOD detection. All methods are trained on ResNet-34 using CIFAR-100, without assuming access to auxiliary outlier datasets. For clarity, we divide the methods into two categories: trained with and without contrastive losses. For pre-trained model-based scores such as MSP (Hendrycks
& Gimpel (007)), ODIN (Liang et al.) (2018), Ma Ma lanobis s (Lee et al. 2018), and Energy (Liu ct al. | (2200), the model is trained with the softmax cross-entropy (CE) loss. GODIN (Hsu et al. 2020)
trained using the DeConf-C loss, while Logi NNorm (Wei et al. 2022) modifies CE loss via logit
normalization. For methods involving contrastive losses, we consider ProxyAnchor (Kim et al.
2020), SimCLR (Winkens et al. 2020), CSI (Tack et al. 2020), SSD+ (Sehwag et al. 2021). } and KNN+ (Sun et al. (022).. Both SSD+ + and KNN+ use the SupCon loss in training. We use the same network structure and embedding dimension, while varying the training objective.
5

---

Published as a conference paper at ICLR 2023
Table 1: OOD detection performance for CIFAR-100 (ID) with ResNet-34. Training with CIDER significantly improves OOD detection performance.
Method
SVHN
Places365
OOD Dataset
LSUN
ISUN
Texture
Average
AUROC
FPR.
AUROCI
FPR.
AUROC1
FPR.
AUROCT
FPR!
AUROC!
FPR
AUROC1
Without
Contrastive
Learning
MSP
78.89
79.80
84.58
74.21
83.47
75,28
84.61
74.51
86.51
72.53
83.12
75.27
ODIN
84.88
79.16
Mahalanobis
$7.09
80:62
84.63
73.89
84.15
79.43
83.18
78.83
61.72
84.87
80.15
79.53
Energy
66.91
85.25
81.41
76.37
59.77
86.69
66.52
84.49
79. 19.01
79.96
70.72
82.55
GODIN
LogitNomn
59.60
90.74
80.25
78.58
81:177
82.99
84.19
80:77
78.35
With
Contrastive Learning
ProxyAnchor
87.21
82.43
70.10
79.84
37.19
91.68
70.01
84.96
65.64
84,99
66.03
84.78
+ SimCLR
24.82
94.45
71.48%
56.40
66.52
X3.82
63.74
82.01
84.15
CSI
44.53
92.65
79.08
76.27
75.58
83.78
76.62
84.98
61.61
86.47
67.48
$418.3
SSD+
31.19
94.19
77.74
79,90
79.39
85.18
80.85
84.08
66.63
86.18
67.16
$5,90
KNN+
CIDER 23.09
95.16
79.63
73.43
16.16
96.33
71,68
82.98
43.87
90.42
46.89
87.67
As shown in Table OOD detection performance is significantly improved with CIDER. Three trends can be observed: (1) Compared to SSD+ and KNN+ CIDER explicitly optimizes for inter- class dispersion and leads more desirable embeddings. Moreover, CIDER alleviates the class- conditional Gaussian assumptions for OOD detection. Instead, a simple non-parametric distance- based score suffices. Specifically, CIDER outperforms the competitive methods SSD+ by 20.3% and KNN+ by 13.3% in FPR95; (2) While CSI (Tack et al., 2020) relies sophisticated data aug- mentations and ensembles in testing, CIDER only uses the default data augmentations and thus is simpler in practice. Performance wise, CIDER reduces the average FPR95 by 20.6% compared to CSI; (3) Lastly, as a result of the improved embedding quality, CIDER improves the ID accuracy by 0.76% compared training with the CE loss (Table We provide results on the less challenging task (CIFAR-10 as ID) in Appendix D where CIDER'S s strong performance remains.
CIDER benefits different distance-based scores. We show that CIDER's strong representation can benefit different distance-based OOD scores. We consider the Mahalanobis score (denoted as Maha) due to its commonality, and to ensure fair comparison with SSD+ under the same OOD score. The results are shown in Table 2 Under both KNN (non-parametric) and Maha (parametric) scores, CIDER consistently improves the OOD detection performance compared training with SupCon. For example, CIDER with KNN significant reduces FPR95 by 13.33% compared to SupCon+KNN. Moreover, compared to SSD+ (SupCon+Maile CiDER+Maha reduces FPR95 by 22.77%. This further highlights the improved representation quality and generality of CIDER.
Table 2: Ablation on OOD detection score. Results are FPR95 on CIFAR-100 (ID) with ResNet-34.
We evaluate both Mahalanobis and KNN score (K 300).
OOD Dataset
Method
AVG FPR95
SVHN
Places365
LSUN
iSUN
Texture
SupCon+Maha (SSD+)
31.19
77.74
79.39
80.85
66.63
67.16
CIDER+Mahl
16.68
80.34
11.07
73.82
40.06
44,39
SupCon+KNN (KNN+)
39.23
80.74
48.99
74.99
57.15
60.22
CIDER+KNN
23.09
79.63
16.16
71.68
43.87
46.89
Inter-class dispersion is key to strong OOD detection. Here we examine the effects of loss components on OOD detection. As shown in Table we have the following observations: (1) For ID classification, training with Ccomp alone leads to an accuracy of 75.19%, similar to the ID accuracy of SSD+ (1%)..)) This suggests that promoting intra-class compactness and a moderate level of inter-class dispersion (as result of sample-to-protott
negative pairs in Leomp) are sufficient to discriminate different ID classes; (2) For OOD detection, further inter-class dispersion critical, which is explicitly encouraged through the dispersion loss Lais. As result, adding Cais improved the average AUROC by 2%. However, promoting inter-class dispersion via Ldis alone without Lcomp is not sufficient for neither ID classification
OOD detection. Our ablation suggests that Lais and Loomp work synergistically to improve the hyperspherical embeddings that are desirable for both ID classification and OOD detection.

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study on loss component. Results (in AUROC) are based on CIFAR-100 trained with ResNet-34. Training with only Ceomp suffices for ID classification. Inter-class dispersion induced by dis is key OOD detection.
Loss Components
AUROCT
ID ACC1
Leoup
Ldis
Places365
LSUN
ISUN
Texture
SVHN
AVG
79.63
85.75
84.45
87.21
91.33
85.67
75.19
V
54, 76
69.81
54.99
44.26
46.48
54.06
2.03
V
73.43
96.33
82.98
90.42
95.16
87.67
75.35
CIFAR-10 prototype)
CIFAR-100
Bhard CCD samples)
CIFAR-10
ID dlass prototype)
CIFAR-10 (D)
CIFAR-100 haadd (OO)
CIFAR-10 (D)
(b) ID & OOD of CE (top) and CIDER (down)
(a) ID embeddings of CE (left) and CIDER (right)
Figure 3: (a): UMAP (MeInnes et al. 2018 visualization of the features when the model trained with CE vs. CIDER for CIFAR-IO (ID). (b): CIDER makes OOD samples more separable from ID compared to CE (c.f. T
4.3 CHARACTERIZING AND UNDERSTANDING EMBEDDING QUALITY
CIDER learns distinguishable representations. We visualize the learned feature embeddings in Figure 3 using UMAP (Melnnes et al.] 2018), where the colors encode different class labels. A salient observation that embeddings obtained with CTDER enjoy much better compactness com- pared to embeddings trained with the CE loss (3a). Moreover, the classes are distributed more uniformly in the space, highlighting the efficacy of the dispersion loss.
CIDER improves inter-class dispersion and intra-class compactness. Beyond visualization, also quantitatively measure embedding quality. propose two measurements: inter-class dis- persion and intra-class compactness:
C
Dispersion (1)
Σ
Σ
m #1).
C
C
-
1=1
1=1
C
Compactness((PP..l
Σ
Σ
x M1 {v: = }}
C j=1
where D = (x, y. and z; the normalized embedding of x; for all S < N. Dispersion is measured by the average cosine similarity among pair-wise class prototypes. The compactness can interpreted as the average cosine similarity between each feature embedding and its corre- sponding class prototype.
To make the measurements more interpretable, we convert cosine similarities to angular degrees. Hence, higher inter-class dispersion (in degrees) indicates more separability among class proto- types, which desirable. Similarly, lower intra-class compactness (in degrees) is better. The results shown in Table based the CIFAR-10 test set. Compared SSD+ (with SupCon loss), CIDER significantly improves the inter-class dispersion by 12.03 degrees. Different from SupCon, CIDER explicitly optimizes the inter-class dispersion, which especially benefits OOD detection.
CIDER improves ID-OOD separability. Next, we quantitatively measure how the feature embed- ding quality affects the ID-OOD separability. We introduce separability score, which measures
7

---

Published as a conference paper at ICLR 2023
Table 4: Compactness and dispersion of CIFAR-I0 feature embedding, along with the separability
W.E. each OOD test set. We convert cosine similarity to angular degrees for better readability.
Training Loss
Dispersion (ID)f
Compactness (ID)
ID-OOD Separability (in degree)
(in degree)
(in degree)
CIFAR-100
LSUN
ISUN
Texture
SVHN
AVG
Eosss-Entopp (SupCon loss)
67.
24.53
7.11
14.57
13.70
13.76
11.08
12.04
28.55
CIDER (ours)
87.53
21.35
31.41
48.37
41.54
39.60
51,65
42.51
on average how close the embedding of a sample from the OOD test set is to the closest II) class prototype, compared that of an ID sample. The traditional notion of "OOD being far away from ID classes" is now translated to "OOD being somewhere between ID clusters on the hypersphere". higher separability score indicates that the OOD test easier be detected. Formally, define the separability measurement as:
Separability
Σ XE Devd
max X Mi JE[C]
Σ
max Ex JE[C]
Mij
D test
test
x'
K'EDI.
where Disd is the OOD test dataset and Zx denotes the normalized embedding of sample Table shows that CIDER leads to higher separability and consequently superior OOD detection perfor- mance (c.f. Table Averaging across 5 OOD test datasets, our method displays relative 42.36% improvement of ID-OOD separability compared to SupCon. This further verifies the effectiveness of CIDER for improving OOD detection.
4.4 ADDITIONAL ABLATIONS AND ANALYSIS
CIDER is competitive large-scale datasets. To fur- ther examine the performance CIDER on real-world tasks, evaluate the performance of CIDER the more challenging large-scale benchmarks. Specifically, we use
100
CIDER
AUROI
99
ImageNet-100 ID, a subset of ImageNet (Deng et al.
2009) consisting of 100 randomly sampled classes. For
OOD test datasets, we use the same ones in (Huang &
Li (021). including subsets of INATURALIST (Van Horn ct al. 2018), SUN (Xiao et al. 2010), PLACES (Zhou al. 2017) and TEXTURE (Cimpoi et al. 2014a). For each OOD dataset, the categories do not overlap with the dataset. For computational efficiency, trained ResNet-34 with CIDER and SupCon losses for 10 epochs with initial learning rate of 0.01. For each loss, update the weights of the last residual block and the nonlinear projection head, while freezing the parameters in the first three residual blocks. At time, use the same detection score (KNN) evaluate representation quality. The performance (in AUROC) is shown in F (more results are in Appendix D). We can see that CIDER remains very competitive on all the OOD test sets where CIDER consistently outperforms SupCon. This further verifies the benefits of explicitly promoting inter-class dispersion and intra- class compactness.
95
SUN
Places366
Texturas
Naturalist
Datazet
Figure 4:
Fine-tuning pre-trained
fine-tune pre-
ResNet-34 on ImageNet-100 100 (ID).
Ablation studies on weight scale, prototype update factor, learning rate, temperature, batch size, and architecture. We provide comprehensive ablation studies to understand the impact of various factors Appendix C For example, as shown in Figure 5] CIDER consistently outperforms SupCon across different batch sizes.
SupCon
AUROC
RELATED WORKS
80
512
Balch Size
Figure OOD Detection perfor-
detection. The majority works in the OOD detection literature focus the supervised setting,
mance across different batch sizes.
8

---

Published as a conference paper at ICLR 2023
where the goal is to derive binary ID-OOD classifier along with classification model for the in-distribution data. Compared to generative model-based based methods (Kirichenko et al. 2020, Nalis- nick et al. 2019 Ren et al.] 2019 [Serrà al., 2020; Xiao al., 2020), OOD detection based on supervised discriminative models typically yield more competitive performance. For deep neural networks-based methods, most OOD detection methods derive confidence scores either based on the output (Bendale & Boult 2016 Hendrycks & Gimpel 2017: Hsu et al., 20200] Huang & Li, Liang al. 20118 Liu al. 2020; Sun et al. 2021 Ming al.) 2022b), gradient informa-
2021
tion Huang et al. 2021), or the feature embeddings (Lee et al. 2018 Sastry & Oore, 2020) Tack
al. 2020 Zhou et al.) 202T; [Sewwag et al.) 2021 Sun al. [002]] Ming et al., [202aa] Du al.,
2022
Our method can be categorized as a distance- baseeeeee--ddde-ee OOD detection method by exploiting the
hyperspherical embedding space.
Contrastive representation learning. Contrastive representation learning (van den Oord et al,, 2019) aims to learn discriminative embedding where positive samples aligned while negative ones are dispersed. It has demonstrated remarkable success for visual representation learning in unsupervised (Chen et al. 2020a bj He et al. 2020 Robinson et al.] 202I), semi-supervised (Assran et al. 2021). and supervised settings (Khosla et al.. (020). Recently, Li et al. (2021b) propose a prototype-based contrastive loss for unsupervised learning where prototypes are generated via clustering algorithm, while our method is supervised where prototypes are updated based on labels.
Li et al. (2021a) incorporate a prototype-based loss to tackle data noise. Waang & Isola (2020) analyze the asymptotic behavior of contrastive losses theoretically, while Waang& & Liu (20214
empirically investigate the properties of contrastive losses for classification. Recently, [Bucci et al. (2022) investigates contrastive learning for open-set domain adaptation. However, none of the works tocus on OOD detection. We aim to fill the gap and facilitate the design and understanding of contrastive losses for OOD detection.
Representation learning for OOD detection. Self-supervieed learning has been shown to improve OOD detection. Prior works (Sehwag et al.] [20T]] Winkens et al.) (020) verify the effectiveness of directly applying the off off-the-shelf multi-view contrastive losses such as SupCon and SimCLR for OOD detection. CSI (Tack et al. (020) investigates the type of data augmentations that are particularly beneficial for OOD detection. Different from prior works, we focus on hyperspherical embeddings and propose to explicitly encourage the desirable properties for OOD detection, and thus alleviate the dependence on specific data augmentations or self-supervision. Moreover, CIDER explicitly models the latent representations metrical interpretation of hyperspherical embeddings. Closest to our work, [D et al. (2022) recently explores shaping representations into vMF distributions for object-level OOD detection. However, they do not consider the inter-class dispersion loss, which we show is crucial to achieve strong OOD detection performance.
vMF distributions, providing a direct and clear geo-
Deep metric learning. Learning a desirable embedding is a fundamental goal in the deep metric learning community. Various losses have been proposed for face verification (Degg et al.
2019
Liu et al., 2017, Wang et al. 2018 person re-identification (Chen et al. 2017 Xiao et al.
2017
and image retrieval (Kim et al.] 2019 Movshovitz- Attias et al. 2017 Oh Song et al. 2016 Teh
et al. 2020). However, none of the works focus on desirable embeddings for OOD detection. The difference between CIDER and proxy-based methods has been discussed in Remark 2.
CONCLUSION AND OUTLOOK
In this work, we propose CIDER, a novel representation learning framework that exploits hyper- spherical embeddings for OOD detection. CIDER jointly optimizes the dispersion and compactness losses to promote strong ID-OOD separability. We show that CIDER achieves superior performance on common OOD benchmarks, including large-scale OOD detection tasks. Moreover, we introduce new measurements to quantify the hyperspherical embedding, and establish the relationship with OOD detection performance. We conduct extensive ablations understand the efficacy and behav- ior CIDER under various settings and hyperparameters. We hope our work can inspire future methods of exploiting hyperspherical representations for OOD detection.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We gratefully acknowledge the support of AFOSR Young Investigator Award under No. FA9550- 23-1-0184; Philanthropic Fund from SFF; and faculty research awards from Google, Meta, and Amazon. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements either ex- pressed or implied, of the sponsors.
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically pre- dicting view assignments with support samples. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 8443 8452-,45,2222 2021.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016.
Silvia Bucci, Francesco Cappio Borlino, Barbara Caputo, and Tatiana Tommasi. Distance-based hyperspherical classification for multi-source open-set domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1119-1128, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:200..05709 2020a.
Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: a deep quadru- plet network for person re-identification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 403 412, 2017.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297,, 2020b.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014a.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613, 2014b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-469,, 2019.
Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detect- ing out-of nut-oo-distribution objects. Advances in Neural Information Processing Systems, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision 2020.
and Pattem Recognition (CVPR), pp.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of distri-iist examples in neural networks. In 5th International Conference Learning Representations, ICLR 2017,
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from oft-of- distribbutto data. In Proceedings of the IFFF/CVF Conference Computer Vision and Pattern Recognition, pp. 10951-10960. 2020.
10

---

Published as a conference paper at ICLR 2023
Rui Huang and Yixuan Li. Towards scaling out-of-distribund detection for large semantic space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. In Advances in Neural Information Processing Systems, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673, 2020.
Sungyeon Kim, Minkyo Seo, Ivan Laptev, Minsu Cho, and Suha Kwak. Deep metric learning beyond binary supervision. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2288-2297, 2019.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. /EEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in Neural Information Processing Systems, 33, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting nut-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, pp. 7167-7177, 2018.
Junnan Li, Caiming Xiong, and Steven Hoi. Mopro: Webly supervised learning with momentum prototypes. In International Conference Learning Representations, 2020.
Junnan Li, Caiming Xiong, and Steven C.H. Hoi. Learning from noisy data with robust represen- tation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9995,9494, 2021a.
Junnan Li. Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021b.
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-diitrribttio image detection in neural networks. In 6th International Conference Learning Representations, ICLR 2018, 2018
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. Advances in Neural Information Processing Systems, 2020.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Kanti V Mardia, Peter E Jupp, and KV Mardia. Directional statisties, volume 2. Wiley Online Library, 2000.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022a.
Yifei Ming, Ying Fan, and Yixuan Li. POEM: In Proceedings of the 39th International Conference on Machine Learning, pp. 15650-15665, 2022b.
detection with posterior sampling.
Yair Movshovitz-Atian Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In Proceedings of the IEEE International Conference Computer Vision, pp. 360-368, 2017.
11

---

Published as a conference paper at ICLR 2023
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? International Conference on Learning Representations, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 427-436, 2015.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004-4012, 2016.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Joshua David Robinson, Ching-Yao Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
Chandramouli Shama Sastry and Sageev Oore. Detecting matrices. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
examples with Gram
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-basee generative models. In Inter- national Conference Learning Representations, 2020.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activa- tions. In Advances in Neural Information Processing Systems, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICML, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Novelty detection via contrastive
Eu Wern Teh, Terrance VeVries, and Graham W Taylor. Proxynca*+: Revisiting and revitalizing proxy neighborhood component analysis. In European Conference on Computer Vision, pp. 448- 464. Springer, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3769-8778 2018.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2495-2504, 2021a.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2405-2504, 2021b.
12

---

Published as a conference paper at ICLR 2023
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2265-5274 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9929.9933. PMLR, 2020.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng. Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. 2022.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive detection. arXiv preprint arXiv:2007.05566, 2020.
training for improved
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition (CVPR), pp. 3485-3492. IEEE, 2010.
Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. Joint detection and identi- fication feature learning for person search. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3415-3424, 2017.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in Neural Information Processing Systems, 33, 2020.
Chen Xing. Sercan Arik, Zizhao Zhang, and Tomas Pfister. Distance- based learning from errors for confidence calibration. In International Conference on Learning Representations, 2020.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint 15XY::5504.0655.. 2015.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv 2015.
preprint 100.:vv::000000
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6); 1452-144, 2017.
Zhi Zhou, -he Guo, Zhanzhan Cheng, Yu-Feng Li, and Shiliang Pu. Step: -uttof-distribution detection in the presence of limited in-distribution labeled data. In Advances in Neural Informa- tion Processing Systems, 2021.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A ALGORITHM DETAILS AND DISCUSSIONS
The training scheme of compactness and dispersion regularized (CIDER) learning framework shown in Algorithm [
We jointly optimize: (1) a compactness loss to encourage samples to be close to their class prototypes, and (2) dispersion loss encourage larger angular distances among different class prototypes.
Algorithm 1: Pseudo-code of CIDER.
Input: Training dataset D, neural network encoder f. projection head h, classifier class prototypes (1 j < C). weights of loss terms λa, and λc temperature r for epoch = 1,2..... do for iter 1, ........d
sample a mini batch B = (x,, 3:
obtain augmented batch B = (8., 2 by applying two random augmentations to E B Vi (1, for x; E B do
// obtain normalized embedding ((f(&.)) /|21
// update class-
He == Normalizeaaua + (1 a)z.), 4 (1.2,...,C)
11 calculate compactness loss
Lomp Eis log
ex((x (ac)+)
Eg (x((*] "xp) my/+]
1/ calculate dispersion loss
Laia 2/2 EE log 1 E 10 # ifee
w//r us/
10
// calculate overall loss
11
== Lais + Achcomp
// update the network wei weights
update the weights in the encoder and the projection head
Remark 1: The prototype update rule. The class prototypes are only updated by exponential moving average (EMA). Since prototypes are not learnable parameters, the gradients of the dispersion loss have no direct impact on their updates. EMA--ttyl techniques have been used in prior works Li et al. (020), and can rigorously interpreted from clustering- Maximization (EM) perspective. Alternatively, the prototypes can also be updated via gradients without EMA. We provide an ablation study of CIDER based on different prototype update rules in Appendix C
Expectation-
Remark 2: CIDER vs. Wang & Isola (2020). The notion of alignment and uniformity for con- trastive losses were proposed Wang Isola (220)) for the unsupervised setting where both metrics are based on individual samples. CIDER a contrastive loss designed for the supervised setting. In particular, the uniform loss WWan & Isola (2020) is defined based on randomly sampled pairs of data and promotes an mistance-to-instane uniformity on the hypersphere. The notion of uniformity fundamentally different from CIDER, which promotes
dispersion.
Remark 3: CIDER vs. Cross Entropy. When a model is trained with the cross-entropy (CE) loss. the weight matrix of the last fully connected layer can be interpreted as the of class pro- totypes. However, CE is suboptimal for OOD detection for two main reasons: (I) CE does not explicitly optimize for the intra-class compactness and inter-class dispersion in the feature space. As consequence, the embeddings obtained by CE loss display insufficient compactness and dis- persion (Table 4 Compared to CE, CIDER is more structured by exploiting the hyperspherical embeddings and explicitly optimizing towards the desirable properties for OOD detection and ID classification. (2) the feature space obtained by CE loss Euclidean instead of hyperspherical. As shown in Tack et al. (2020), ID data tend have larger norm than OOD data. As result, the Euclidean distance between ID features can be larger than the distance from OOD to ID data. A re- cent work (Sun et al. 2022 verified that Euclidean embedding without feature normalization leads
14

---

Published as a conference paper at ICLR 2023
to suboptimal OOD detection performance. Instead, CIDER is designed to optimize hyperspherical embeddings, which benefit OOD detection.
Remark 4: On the measurement of embedding quality. In Section 4.3 we provide measurements of embedding quality (inter-class dispersion and intra-class compactness) via prototypes. In practice, replace the class prototypes in the dispersion and compactness metries be one random sample (or the average of a random subset) from the corresponding class. For example, on CIFAR- 10, as the embeddings of CIDER are compact, we observe that the two metrics give similar results (e.g., the ID-OOD Separability is 42.5 with prototype-base metrics vs. 38.3 with instance-based metries). We choose to use prototypes because (1) the definition directly maps our loss function design, and (2) prototypes are calculated as the averaged feature for each class, which helps to mitigate the sampling bias.
B EXPERIMENTAL DETAILS
Software and hardware. All methods are implemented in Pytorch 1.10. We run all the experiments on NVIDIA GeForce RTX-2080TI GPU for small to medium batch size and on NVIDIA A100 GPU for large batch size and larger network encoder.
Architecture. As shown in Figure 1 the overall architecture of CIDER consists of a projection head h top of deep neural network encoder f. Following common practice and fair comparison with prior works (Sehwag et al. (022). (Kholla et al. 2020). we fix the output dimension of the projection head be 128. We use two-layer non-linear projection head for CIFAR-10 and CIFAR- 100 in Sun et al. 2022).
Training. For methods based pre-trained models such as MSP (Hendrycks & Gimpel ODIN (Liang et al., 2018), Mahalanobis (Lee et al. 2018), and Energy (Liu et (0220) we fol- low the configurations in Sun et al. (2022) for CIFAR-10 10 and train with the cross-entropy loss for 100 epochs. The initial learning rate 0.1 and decays by a factor of 10 at epochs 50, 75, and 90 respectively. For the more challenging dataset CIFAR-100,
2017
train 200 epochs. We use stochastic gradient descent with momentum 0.9, and weight decay 10 10-4 For fair comparison, methods involv- ing contrastive learning (Winkens al. 2020 (Tack et al. 2020 (Schwag et al. 202I are trained for 500 epochs on CIFAR-IO and CIFAR- 100. For CIDER, we adopt the same key hyperparameters for contrastive losses such as initial learning rate (0.5), temperature (0.1), and batch size (512) as SSD+ (Sehwag et al., (021) in main experiments to demonstrate the effectiveness and simplicity of CIDER. For the prototype update factor a, we set the default value as 0.95 for simplicity. We observed that 0.95 on CIFAR-10 with ResNet-18 18 and = 0.5 on CIFAR-100 with ResNet-34 provide stronger performance.
OOD detection score. By default, we use the non-parametric KNN score Sun et al. (2022). We use
a larger K- 300 for CIFAR-100 and smaller K 100 for CIFAR- 10 for simplicity. Adjusting K E [10,20, 50, 100, 200, 300, 500} yields similar performance. In practice, K can be tuned using a validation method (Sun et al. 2022) to further improve the performance.
C ADDITIONAL ABLATION STUDIES
CIDER is effective under various batch sizes. Figure 6a and 6b also indicate that CIDER re- mains competitive under different batch size configurations compared to SupCon. To explain this, the standard SupCon loss requires instance-to-instancce distance measurement, whereas compact- ness loss reduces the complexity instance-lo-protttypee. The class-conditional prototypes are up- dated during training, which capture the average statistics of each class and alleviate the dependency on the batch size. This leads to an overall memory-efficiett solution for OOD detection.
Ablation on the loss weights. In the main results (Table I. demonstrate the effectiveness of CIDER where the loss weight A, is simply set to balance the initial scale between the Cais and feomp- In fact, CIDER can be further improved by adjusting Ag- As shown in Figure 7a the
15

---

Published as a conference paper at ICLR 2023
SupCan CIDER
Supdan CIDER
AUROC
85
84
82
286
512
1024
206
512
Batch Size
Batch Size
(a) FPR95 under different batch sizes
(b) AUROC under different batch sizes
Figure 6: Ablation on CIDER v.s. SupCon loss under different batch sizes. The results are averaged across the 5 OOD test sets based on ResNet-34. CIDER outperforms SupCon across different batch sizes, suggesting the effectiveness explicitly facilitating prototype-wis dispersion.
70
30
FPRO5
FPR85
40
20
0.25
0.5
1.0
2.0
8.0
10.0
0.1
0.2
0.5
0.8
0.9
0.95
0.90
(a) weight Ae
(b) Mov-avg
Figure 7: Ablation on (a) weight A of the compactness loss; (b) prototype update discount factor a. The results are based on CIFAR-100 (ID) averaged over 5 OOD test sets.
performance of CIDER is relatively stable for moderate adjustments of Àe (e.g. 0.5 2), with the best performance at around X₂ E [1, This indicates CIDER provides simple and effective solution for improving OOD detection, without much need for hyperparameter tuning on loss
scale.
Adjusting prototype update factor a improves CIDER. We show Figure 7b the performance by varying the moving-average discount factor in Eq. g We can observe that the detection per- formance (averaged over test sets) is still competitive across wide range of a. particular, for CIFAR- 100, 0.5 results in the best performance with average FPR95 of 46.89% under KNN score. For CIFAR-10, we observe that a larger a (e.g. 0.95 to 0.99) results in stronger performance.
Ablation on the learning rate. Prior works (Khosla et al. 2020 Sehwag et al. 2021) use a default initial learning rate (Ir) of 0.5 train contrastive losses, which is also the default setting of CIDER. We further investigate the impact of the initial learning rate on OOD detection. As shown in Fig- ure 8al a relatively higher initial Ir is indeed desirable for competitive performance while too small Ir (e.g. 0.1) would lead to performance degradation.
Small temperature leads to better performance. Figure 8b demonstrates the detection perfor- mance as we vary the temperature parameter т. We observe that the OOD detection performance is desirable at relatively small temperature. Complementary to our finding, relatively small temper- ature is shown to be desirable for ID classification (Khosla et al., 2020 Wang & Liu 2021b) which penalizes hard negative samples with larger gradients and leads to separable features.
Ablation on network capacity. We verify the effectiveness of CIDER under networks with other architectures such as ResNet-50 for CIFAR-100. The results are shown in Figure 9 The trend is similar to what we observed with ResNet-34. Specifically, result of the improved representa- tion. training with CIDER improves the FPR95 for various test sets compared to training with the SupCon loss.
16

---

Published as a conference paper at ICLR 2023
60
EPRS5
EPR35
40
20
0.1
0.2
0.5 0.6
0.8
0.05
0.1
0.3
0.4
0.5
0.7
0.9
init Ir
temperature
Figure 8: Ablation on (a) initial learning rate and (b) temperature. The results are based on CIFAR-100 (ID)
averaged over 5 OOD test sets.
Ablation gradient-based prototype update. We examine the effect of updating prototypes via gradients. Compared to CIDER with EMA, CIDER with learnable prototypes (LP) can be more sensitive initialization. We report the average performance of CIDER (EMA) and CIDER (LP) across 3 independent runs for CIFAR-10 in 3 All training and evaluation configurations (e.g., learning rate and batch size) are the same. We can see that CIDER with EMA improves the average FPR95 by 5.08% with smaller standard deviation. Therefore, we empirically verify that updating prototypes via EMA a better option with stronger training stability in practice.
Table 5: Ablation on prototype update rules. OOD detection performance for ResNet-18 trained on CIFAR-10 with EMA-style updates denoted as CIDER (EMA) vs. learnable prototypes denoted as CIDER (LP). CIDER with EMA demonstrates strong OOD detection performance. Results are averaged over 3 independent runs.
Testure
Average
AUROUT
AUROCT
AUROUT
AUROCT
AUROU
CIDER (LP) 2.19
09 56
25.13
91.599
98. 16
36 473h
162551
97.38
17.65
CIDER (BMA)
3.01
09.30 20 26.00
4.10
15.01
07. 07.10
18. 18.1g
07
12,07
97. 07.50
Stability CIDER. To verify that CIDER consistently provides strong performance, we train with 3 independent seeds for each ID dataset. Table 6 shows the OOD detection performance of CIDER with R8sNet-18 trained on CIFAR-10 and ResNet-34 trained on CIFAR-100. Comparing Table and Table 7 we
that CIDER yields consistently strong performance. Code and checkpoints are provided in
com
Table Ablation on stability. OOD detection performance CIDER for CIFAR-10 and CIFAR- 100. Results are averaged over 3 independent runs.
Datowrl
SVIN
Plasee65
0OD Dstrwet LSUN
Asurage
AUROCT
AUROCT
AUROCT
AUROC:
AUROCT
AUROC:
CIRAR
10.0
0J
14.31
16.34
15.50 15.
97. 48.4
12.194944
97 19
1257 11.31
17 (113
CIPAR-10
M3 37
US, Up SN
21
22 (Ag
W6,01
S0.71>
48.0G
90.25"
87 1y
D RESULTS ON LARGE-SCALE DATASETS
In recent years, there has been a paradigm shift towards fine-tuning pre-trained models, as opposed to training from scratch. Given this trend, important to explore whether CIDER remains fective based on pre-trained models. Specifically, we fine-tune ImageNet pre-trained ResNet-34 on ImageNet- 100 with CIDER and SupCon losses for 10 epochs. For each loss, we update the weights of the last residual block and the nonlinear projection head, while freezing the parameters in the first three residual blocks. At test time, we use the same detection score (KNN) to evaluate representation quality. FPR95 and AUROC for each OOD test set shown in Figure 10a and [0b] respectively. The results suggest that CIDER remains very competitive, which highlight the benefits of promoting inter-class dispersion and intra-class compactness.
17

---

Published as a conference paper at ICLR 2023
SupCon CIDER
1PX80
80
Places365
Texture
LSUN
SVHN
Figure 9: Ablation architecture. Results based on ResNet-55..
100
SupCon
CDER
FPR05
AUROD
69
SUN
Placos665
Textures
INtturalst
Placwe
Teclures
Dalaset
Delisel
(a) FPR95 for different OOD test sets
(b) AUROC for different OOD test sets
Figure 10: OOD detection performance of fine-tuning with CIDER v.s. SupCon for ImageNet-100 (ID). With the same detection score (KNN), CIDER consistently outperforms SupCon across all OOD test datasets.
E RESULTS ON CIFAR-10
In the main paper. mainly focus the challenging task CIFAR-100. In this section, additionally evaluate on CIFAR-I0, commonly used benchmark in literature. For methods in- volving contrastive losses, we the same network encoder and embedding dimension, while only varying the training objective. The Mahalanobis score is used for OOD detection SSD+ (Sehwag et al. 202T), CE+SimCLR (Winkens al., 2020), SupCon, and CIDER. As CIFAR- 10 is much less challenging compared to CIFAR-100, recent methods with contrastive losses yield similarly strong performance. For methods trained with cross-entropy loss,
the publicly available check- points in Sun et al. (2022) for better consistency. The results are shown in Table 7 Similar trends also hold
describe in Section 4.2 (I) CIDER achieves superior OOD detection performance in CIFAR-10 as result of better inter-class dispersion and intra-class compactness. For example, compared the Mahalanobis baseline (Lee al., 2018). CIDER reduces the FPR95 by 24.99% averaged over 5 diverse test sets; (2) Although the ID classification accuracy of CIDER is similar to another proxy-based loss P ProxyAnchor r (Table (8, CIDER significantly improves the OOD detec- tion performance by 21.00 21.05% in FPR95 due to the addition of explicit inter-class dispersion which we show is critical for OOD detection in Section 4.3 The significant improvements highlight the importance of representation learning for OOD detection.
F ID CLASSIFICATION ACCURACY
The ID classification accuracy CIFAR-10 10 and CIFAR-100 can be seen in Table and Table g where for contrastive losses such as KNN+, SSD+, and CIDER, we follow the common practice as in Khosla et al. (2020) and use linear probe normalized features.
18

---

Published as a conference paper at ICLR 2023
OOD Dataset
Method
SVHN
Placest68
LSUN
ISUN
Texture
Average
FPR
AUROCT
FPR.
AUROCT
FPR.
AUROCT
EPR
AUROCT
FPR.
AUROCT
FPR
AUROCT
Without
Contrastive Learning
MSP
59,66
91.25
62.46
88.64
45.21
93.80
54,57
92.12
66.45
88.50
57.67
90.86
Energy
91.22
42.77
91.02
10.19
98.015
27.52
55.23
ODIN
53.78
91.30
13.40
90.98
10.95
97.93
28.14
95.51
55.59
89.47
38.43
93.01
GODIN
18.72
55.25
85,50
11.52
97.12
30.02
94.02
33.58
92.20
29.55
92.97
With Contrastive Learning
CE + SimCLR
6.98
99.22
$4.39
86.70
61.53
85.60
59.62
86.78
16.77
96.56
40.46
90.97
SSD+ 2.47
99.51
22.05
95.57
10.56
97.83
28.44
95.67
9.27
96.35
14.56
97.38
Prahy-nchor
39.27
94.55
43.46
92.06
21.04
97.02
23.53
96.56
42,70
93.16
34.00
94.67
KNN+
CIDER 2.89
99.72
23.88
94.09
5.45
99.01
20.21
96.64
12.33
96.85
12.95
97.26
Table 7: Results CIFAR-10. OOD detection performance for ResNet-18 18 trained CIFAR-
10 with and without contrastive loss, CIDER achieves strong OOD detection performance and ID classification accuracy (Table 8).
Table 8: ID classification accuracy on CIFAR-10 (%)
Method
ID ACC
w.O. contrastive loss
MSP
94.21
ODIN
94.21
GODIN
93.64
Energy
94.21
Mahalanobis
94.21
contrastive loss
CF SimCI.
93.12
SSD+
94.53
Proxy Anchor
94.21
KNN+
94.53
CIDER
94.58
Table 9: ID classification accuracy CIFAR-100 100 (%)
Method
ID ACC
W.0. contrastive loss
MSP
74.59
ODIN
74.59
GODIN
74.92
Energy
74.59
Mahalanobis
74.59
contrastive loss
CE SimCLR
73.54
SSD+
75.11
ProxyAnchor
74.21
KNN+
75.11
CIDER
75.35
19


Reviews:
[{"review_id": "stNg3yT935v", "reviewer": "Reviewer_U6bR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the out-of-distribution detection task and proposes a new method called CIDER, which trains the model by optimizing a dispersion loss and a compactness loss. The compactness loss encourages samples to be close to their class prototypes while the dispersion loss encourages large angular distance among different class prototypes. This paper also investigate EMA for updating prototype parameters. Experiments demonstrate the effectiveness of CIDER.", "summary_of_the_review": "This paper makes a detailed analysis to OOD detection problem and proposes many metrics (dispersion, compactness and separability) for evaluating OOD performance, which is impressive. Experiment results are thorough, which is helpful for understanding CIDER and its effectiveness compared to previous methods. I just list some points that can be further improved. \n\n1. Updating class prototypes during training seems a little weird. In Algorithm 1, the class prototype $\\mu_c$ is updated using EMA first (line 8) and gradient descent w.r.t. Eq. (7) latter (line 12). Why adopting such a two-stage scheme and what if updating prototypes by optimizing Eq. (7) directly without EMA?\\\n\n2. Figure 3: It will be better to compare with contrastive loss and show OOD embeddings, just like Figure 1 in KNN+ paper.", "strength_and_weaknesses": "Strength:\n1. The idea is well motivated and writing is good\n2. Experiment evaluation is thorough\n\nWeakness:\n1. Training scheme needs further analysis and explanation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jxatJnyzvC", "reviewer": "Reviewer_FAUX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method (CIDER) for representation learning and applies it for out-of-distribution detection. Given an $L_2$ normalized embedding (hyperspherical embeddings) from a deep network, the method computes prototypes, as moving averages of the embeddings corresponding to each class. The method is optimized using two losses. The first one (compactness loss) minimizes the distance between a sample and the prototype corresponding to its class while maximizing the distance to the other ones. The second loss (dispersionloss loss) maximizes the distance between all pairs of class prototypes. The representations learned by this method are then used to detect out-of-distribution samples using KNN distance.\n\n", "summary_of_the_review": "This paper proposes a sound method to obtain representations useful for OOD detection. Some further explanations and investigations into the benefits of the proposed method are still needed. ", "strength_and_weaknesses": "**Strong points**:\n\nS1. The method obtains good results for OOD detection compared to recent methods.\n\nS2. The method and investigations are sound. \n\nS3. Paper is mainly well-written.\n\n**Weak Points**:\n\nW1. The relation between the proposed losses and other common losses (cross-entropy and contrastive losses) should be better explained and investigated more. For example, the goals presented in paragraph Training objective (Sec. 3.2) is also attained by cross entropy loss, why should the proposed method behave better? The last weight matrix of a deep model could be interpreted as the set of class prototypes ( with cross-entropy loss thus being similar to the compactness loss)  what are the advantages of the proposed approach of using moving averages as propotypes. The Alignmentand Uniformity metrics of Wang and Isola 2020 share similar goals as the proposed losses and should be discussed also.\n\nW2. The paper proposes some experimental investigations into the benefits of the method, and they are appreciated, but they should be expanded and better explained. \n\n W2.1. While it is perfectly sound to optimize the embeddings using the class prototypes, it is not clear why the measurements of embedding quality (inter-class dispersion and intra-class compactness from Sec 4.3) should take the prototypes into account. More fundamental metrics should be defined based on similarity between embeddings of samples $z_i$ and $z_j$ of different classes (for dispersion) or the same class (for Compactness) (similar to Wang and Isola 2020). Similarly, the separability metric could be designed based on average similarities between pairs of embeddings. Using the prototypes is just a proxy of these, more fundamental metrics. So why should be metrics be defined based on prototypes? \n\n W2.2 How are prototypes defined for Cross-Entropy SSD+ methods shown in Table 4? Are they computed by eq.8 during training? For the Cross-entropy method, are the embeddings $L_2$ normalized during training and when computing the metrics? If they are normalized only for the metrics, does it make sense to use them in this new hyperspherical space if they were not trained in this space?\n\n \n\nW3. How are the results in Table 1 /  Table 6 computed? Are the other methods trained with original code or are the original results presented? Table 6 shows worse results for all other methods than a similar table (Table 1) from (Sun et al. 2022). Why? \n\nW4. How many seeds were used when computing the results? It would be better to use multiple seeds and present results with mean and standard deviations.\n\nW5. It should be noted that this method needs ID class labels, having a disadvantage as opposed to contrastive methods that are designed to work both with and without labels.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fRcyB89GKZ9", "reviewer": "Reviewer_Fpyq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to detecting out-of-distribution (OOD) samples using hyperspherical embeddings. Specifically, two loss functions: dispersion loss and compactness loss are proposed to increase the inter-class distance and decrease the intra-class distance between samples, respectively. The approach is evaluated on benchmark datasets and various ablation studies are conducted to justify various components of the approach. The results are impressive.", "summary_of_the_review": "Even though the idea of maximizing the inter-class distance and minimizing the intra-class distance is explored in various contexts, applying this with hyperspherical embeddings for OOD detection is novel. The approach is shown to be effective by extensive experiments. Please address the comments in the weaknesses section.", "strength_and_weaknesses": "Strengths:\n\n1. Exploiting hyperspherical embeddings for OOD detection is an interesting and effective idea.\n\n2. The proposed dispersion loss is shown to be effective for OOD detection and improving ID classification accuracy.\n\n3. Extensive experiments are conducted to evaluate various aspects of the approach.\n\na. Proposed approach can be combined with existing parametric (Mahalanobis) and non-parametric (KNN+) approaches (Table 2).\nb. The dispersion loss improves both OOD detection and ID classification accuracy (Table 3).\nc. The proposed approach improves class separation as expected.\nd. Effective for hard OOD samples (CIFAR 10 vs CIFAR 100)\n\nWeaknesses:\n\n1. Authors claim that the OOD samples lie between ID classes. How to validate this claim beyond the visualizations? In some case, OOD sample can be close to ID clusters as well. How does the proposed approach handle such situations? Please refer to the following \nKaur, R., Jha, S., Roy, A., Park, S., Sokolsky, O., & Lee, I. Detecting oods as datapoints with high uncertainty. ICML Workshop on Uncertainty and Robustness in Deep Learning, 2021.\n\n2. Is the proposed loss functions particularly effective while considering the hyperspherical embeddings or can be used with standard euclidean embedding as well?\n\n3. How do various values of the parameter $\\lambda$ in Eq.7 affect the OOD detection and ID classification performance?\n\n4. Does the approach require retraining the backbone encoder network with pre-trained networks by just learning the final projection head as shown in Fig. 1?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vr2QFwRBWU0", "reviewer": "Reviewer_Vej6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new out-of-distribution detection method that is based on supervised contrastive learning in a hyperspherical space.  The core component of the proposed method is a novel regularizer that encourages the separation of class centroids. The resulting hyperspherical space is highly effective in distance-based OOD detection methods.", "summary_of_the_review": "This paper presents solid results on improving OOD detection on hyperspherical representation space.", "strength_and_weaknesses": "## Strength\n\n* The proposed method is simple and significant. This finding seems to be generalizable to multiple different applications.\n* The paper is overall clearly written with meticulously described details.\n* The paper provides extensive empirical study with highly convincing results.\n\n## Weaknesses & Questions\n\n* One thing that is not very clear is how exactly the class prototypes ($\\mu$'s) are updated. The reason of the confusion is that there are two equations that can update the class. Both the dispersion loss (Eq.(6)) and the exponential moving average (Eq.(8)) govern the update of the prototypes. Looking at Algorithm 1 in the Appendix, it seems like both equations are applied sequentially. Please correct me if I am wrong.\n    * If it is correct that the two equations are both applied to update the prototypes, I wonder why we need Eq.(8). Wouldn't the embedded vectors $z$ eventually cluster around the corresponding prototype by simply enforcing the compactness loss, Eq.(5)? I see that Figure 6(b) shows the effect of this moving average update on the final performance but do not understand why this even affects the quality of the final representations.\n* The biggest weakness of this paper is that the finding of the paper is mostly empirical. It would be nicer to have a deeper interpretation regarding the dispersion loss, which is the key contribution of the paper, so that we can have a distilled insight on representation learning and OOD generalization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster