Learning with Mixture of Prototypes for Out-of-Distribution Detection
Authors: Haodong Lu
Dong Gong
Shuo Wang
Jason Xue
Lina Yao
Kristen Moore
Venue: ICLR 2024 poster (2024)
Keywords: out-of-distribution detection
prototypical learning
hyperspherical embedding
Primary area: societal considerations including fairness, safety, privacy

Abstract: Out-of-distribution (OOD) detection aims to detect testing samples far away from the in-distribution (ID) training data, which is crucial for the safe deployment of machine learning models in the real world. Distance-based OOD detection methods have emerged with enhanced deep representation learning. They identify unseen OOD samples by measuring their distances from ID class centroids or prototypes. However, existing approaches learn the representation relying on oversimplified data assumptions, e.g. modeling ID data of each class with one centroid class prototype or using loss functions not designed for OOD detection, which overlook the natural diversities within the data. Naively enforcing data samples of each class to be compact around only one prototype leads to inadequate modeling of realistic data and limited performance. To tackle these issues, we propose PrototypicAl Learning with a Mixture of prototypes (PALM) that models each class with multiple prototypes to capture the sample diversities, which learns more faithful and compact samples embeddings for enhanching OOD detection. Our method automatically identifies and dynamically updates prototypes, assigning each sample to a subset of prototypes via reciprocal neighbor soft assignment weights. To learn embeddings with multiple prototypes, PALM optimizes a maximum likelihood estimation (MLE) loss to encourage the sample embeddings to compact around the associated prototypes, as well as a contrastive loss on all prototypes to enhance intra-class compactness and inter-class discrimination at the prototype level. Compared to previous methods with prototypes, the proposed mixture prototype modeling of PALM promotes the representations of each ID class to be more compact and separable from others and the unseen OOD samples, resulting in more reliable OOD detection. Moreover, the automatic estimation of prototypes enables our approach to be extended to the challenging OOD detection task with unlabelled ID data. Extensive experiments demonstrate the superiority of PALM over previous methods, achieving state-of-the-art average AUROC performance of 93.82 on the challenging CIFAR-100 benchmark.

Published as a conference paper at ICLR 2024
LEARNING WITH MIXTURE OF PROTOTYPES FOR OUT- OF- OF-DISTRIBUTION DETECTION
Haodong Lu', Dong ;ong': Shuo Wang". Jason Xue?, Lina Yao Kristen Moore? University of New South Wales, CCIRO's Data6l [haodong. lu, dong @ong) edu. au, (shuo. wang, jason. xue, lina Lin,,yao, kristen. m
ABSTRACT
Out-of-distribution (OOD) detection aims to detect testing samples far away from the in-distribution (ID) training data, which is crucial for the safe deployment machine learning models in the real world. Distance-basee OOD detection methods have emerged with enhanced deep representation learning. They iden- tify unseen OOD samples by measuring their distances from ID class centroids prototypes. However, existing approaches learn the representation relying on oversimplified data assumptions, e.g., modeling ID data of each class with one centroid class prototype using loss functions designed for OOD detection, which overlook the natural diversities within the data. Naively enforcing data samples of each class to be compact around only one prototype leads to inadequate modeling of realistic data and limited performance. To tackle these issues, we pro- pose PrototypicAl Learning with Mixture of prototypes (PALM) which models each class with multiple prototypes to capture the sample diversities, and learns more faithful and compact samples embeddings to enhance OOD detection. Our method automatically identifies and dynamically updates prototypes, assigning each sample to a subset prototypes via reciprocal neighbor soft assignment weights. To learn embeddings with multiple prototypes, PALM optimizes max- imum likelihood estimation (MLE) loss to encourage the sample embeddings to be compact around the associated prototypes, as well as contrastive loss on all prototypes to enhance intra-class compactness and inter-class discrimination at the prototype level. Compared to previous methods with prototypes, the proposed mixture prototype modeling of PALM promotes the representations of each ID class to be more compact and separable from others and the unseen OOD samples, resulting in more reliable OOD detection. Moreover, the automatic estimation of prototypes enables our approach be extended to the challenging OOD detection task with unlabelled ID data. Extensive experiments demonstrate the superiority of PALM over previous methods, achieving state-of-the-ae average AUROC perfor- mance of 93.82 on the challenging CIFAR- 1-0 benchmark. Code is available at httpss//gghuub..oo/////f0024///
INTRODUCTION
Deep learning (DL) plays crucial role in many real-worke applications, such as autonomous driving (Huang et al. 2020), medical diagnosis (Zimmerer al. 2022), and cyber-security (Nguyen et al. 2022b. When deployed in realistic open-world scenarios (Drummond & Shearer 2006). deep neural networks (DNNs) trained on datasets that adhere to closed-world assumptions (He et al. (015). commonly known as in in-distribution (ID) data, tend to struggle when faced with testing samples that significantly deviate from the training distribution, referred to as outrof-distribution (OOD) data. A trustworthy learning system should be aware of OOD samples instead of naively assuming all input to be ID. recent years, there has been significant research focus on the OOD detection task (Drummond & Shearer 2006), aiming accurately distinguish between OOD and ID inputs. This critical endeavor helps to ensure the secure and reliable deployment of DNN models.
Since OOD samples unseen during training, the key challenge is to obtain model and an associated OOD detection criterion, based on only ID samples. Various OOD detection methods
*D. Gong is the corresponding author. This work was partially supported by ARC DECRA Fellowship (DE23010159) awarded D. Gong. H. Lu affiliated with CSTRO Data61 through PhD scholarship.

---

Published as a conference paper at ICLR 2024
have been developed recently, including confidence score based methods (Hendrycks & Gimpel
2016 Lee et al. [0188] Liang et al., 2018 Liu et al. [2200] Wang et al. 2021 Sun et al. [021] Huang
al.... 202 Wang al. [2022), density- based methods (Kingma & Dhariwal [2018] Du & Mordatch
2019 Grathwohl et al. 2020, Ren et al. 2019 Xiao et al. 2020 Cai & Li (202)) and distance-
methods (Tack et al. 2020 ao al. 2023 Sun et al. 2022 Du et al. 2022a Ming et al. |023 Lee
cct al. 2018 Sehwag et al. 2021). J. Promising distance-basee methods leverage the capability of DNNs
to extract feature embeddings and identify OOD samples by measuring the distances between the
embeddings and the centroids or prototypes of ID classes (Lee et al. 2018 Sehwag et al. [02T Ming
al., [023] Tao al. 2023). They are effective in many scenarios, compared with other methods
that overconfidently misclassify OOD data as ID (Hendrycks & Gimpel| 2016 Liang et al. 2018). or
suffer from difficulty training generative models (Ren et al. 2019; Grathwohl et al. 2020).
Distance-based OOD detection methods aim to learn informative feature embeddings and utilize distance metrics, such as Mahalanobis (Lee et al. 2018 [Sehwag et al. 202T) or KNN distance (Sun)
et al. 2022), during testing to identify OOD samples. Recent advances in distance-based methods
(Tack et al. 2020; Sehwag et al. 202T)) use off-the-shelf contrastive loss (Chen et al.] 2020 Khosla
et al. (2020) to shape the embedding space, which is designed for classification and does not take OOD
data into consideration. On top of that, MMng et al.] (2023); Du et al. (2022a) shape the hyperspherical
embedding space (Wang & Isola Wong & Liu 2021 Du et al. 2022aa Ming al., (202)) of
in-distribution (ID) data with class-conditional von Mises-Fisher (vMF) distributions (Mardia et al.
2000), enforcing ID samples of the same class to be compactly embedded around the prototype of its class. However, in Sec. 4.2 we show that naively modeling all samples of each class with only one single prototype leads to restricted modeling capability, where diverse patterns within each class cannot be well represented, leading to the confusion of ID samples and the OOD samples seen in testing. This lack of comprehensive representation further diminishes the compactness surrounding each prototype, thereby resulting in diminished performance.
In this paper, we propose a novel distance-based OOD detection method, called PrototypicAl Learning with Mixture of prototypes (PALM) to learn high-quality hyperspherical embeddings of the data. To capture the natural diversities within each class, we model the hyperspherical embedding space (Schwag et al. 2021 Sun et al. 2022 Du et al. 2022a Ming et al. (023) of each class by a mixture vMF distributions with multiple prototypes, where each prototype represents a subset of thee most similar data samples. Instead of naively enforcing data samples of each class to be compact around a single prototype (Du et al.] [222aa] Ming et al., [023] Tao et al.; [202)), we encourage a more compact cluster around each one of the multiple prototypes, inherently encourage better ID-OOD discrimination. Specifically, we automatically estimate the reciprocal assignment weights between data samples and prototypes, and dynamically update the prototypes guided by these weights. To ensure each sample a high probability of assignment to the corresponding vMF distribution conditioned on each prototype, we apply a MLE loss to minimize the distance between each sample embedding and the prototype, similar to (Ming et al., 2023 Li et al. 2021: Caron et al.) (020)). Furthermore, we propose a contrastive loss on the prototypes to further enhance the intra-class compactness at the prototype/cluster level and simultaneously encourage inter-class discrimination. Our main contributions are summarized as follows:
We propose a novel bassedce-based OOD detection method, i.e., PALM, which regularizes the representation learning in hyperspherical embedding space. Unlike previous methods with oversimplified assumptions, we use more realistic modeling with a mixture of prototypes to formulate and shape the embedding space, leading to better ID-OOD discrimination.
In PALM, we propose a prototypical learning framework to learn the mixture prototypes automatically, Samples are softly assigned to prototypes using specifically designed methods. PALM uses a MLF loss between samples and prototypes, as well as contrastive loss on all prototypes to enhance intra-class compactness and inter-class discrimination.
Extensive experiments and in-depth analyses show the effectiveness of PALM OOD detection. In addition to the standard labelled setting, the automatic prototype learning enables PALM to be easily extended to unsupervised OOD detection with promising results.
2 RELATED WORK
Out-of-distribution detection. The problem of OOD detection was first posed by Nguyen et al. (2015), to address the limitation that DNN models were found to tend to generate overconfident results. This problem has drawn increasing research interest ever since. To overcome this issue,
2

---

Published as a conference paper at ICLR 2024
various OOD detection methods have been proposed like score-based methods (Hendrycks & Gimpel
2016
Lee et al.
2018
Liang et al., 20188 Liu et al. [2200] Wang et al. 2021 Sun et al. 2021] |
Huang
ct al.
2021
Wang et al. 2022), generative- based methods (Ryu et al. [018 Kong & Ramanan
2021
and distance-based methods (Tack et al. 2020 Tao et al. |023]] Sun et al. 2022) Du et al.
20022a
Ming et al. 2023 Lee et al., 2018 Sehwag et al. 202T). bcore-based methods derive scoring
functions for model outputs in output space including contidence-based score (Hendrycks & Gimpel
2016
2022 WWi al. (022), classification score from learned discriminator (Kong& & Ramanan 2021 energy-based score (Liu et al. 2020 Wang
Liang al., 2018 Sun al. [021 WWan et al.
ct al. 2021) and gradient- based score (Huang et al. 2021). Building upon the idea that OOD samples
should be far away from known ID data centroids or prototypes in the embedding space, recent
distance-based methods have shown promising results by computing distances to the nearest training
sample (Tack et al. 2020) or nearest prototype (Tao et al. 2023), KNN distance (Sun et al.] [022] Du
ctt al. 2022a Ming et al. 2023) and Mahalanobis distance (Lee et al. 2018 Sehwag et al. (022))
In addition to that, only few methods (Tack et al., 2020 Sun et | 2022) explore the potential of
developing OOD detection models using completely unlabeled training data. In this work, we extend
our framework to be able to work with unlabeled training data.
Representation learning for OOD detection. Building on the recent success of contrastive represen-
tation learning methods such as SimCLR (Chen al. 2020)p and SupCon (Khosla et al. 2020), recent distance-based methods (Tack et al. 2020; /Sehwag et al. [2021] [Sun et al., (022) have demonstrated the successful application of these methods to OOD detection, despite their training objectives not being specifically designed for this task. Notably, features obtained from SupCon (Khosla et al. 2020) have been used to compute distance metries, such as the Mahalanobis distance (Sehwag et al. 1021) and KNN distance (Sun et al., 2022), for OOD detection. These approaches outperform previous distance-base based methods that derive features trained from standard cross entropy (Lee et al.. 2018). Methods like VOS (Du et al.) (202bb and NPOS (Tao et al.] (023) have taken different approach by synthesizing OOD data samples to regularize the model's 's decision boundary between ID and OOD data. Recent works (Du et al. 2022at Ming et al.] (023) that model the data as vMF distribution (Mardia et al. 2000) provide simple and clear interpretation of the hyperspherical embed- dings. Specifically, Ming al. (2023) propose a regularization strategy to ensure that all samples compactly located around their class's 's corresponding single prototype.
Contrastive learning and prototypical learning. Contrastive representation learning methods treat each sample as an individual class, and bring together multiple views of the same input sample while pushing away from other samples. This effectively enhances the discriminative properties of the learned representations, enabling these approaches to take the lead in learning powerful feature
representations in unsupervised (Wu et al.) 2018 Oord et al. 2018 He et al.) 2020 Chen al. 2020 Robinson et al. (2021), semi-supervised (Assran et al.) 2021) and supervised settings (Khosla et al. 2020). The fundamental properties and effectiveness of contrastive loss in hyperspherical space have been investigated in studies (Wang & Isola 2020 Wang & Liu 2021). Other methods learn feature representations by modeling the relations of samples to cluster centroids (Caron et al., 2018) prototypes (Snell et al. 2017). On top of contrastive learning, Li et al. (2021) integrate prototypical learning that additionally contrasts between samples and prototypes obtained through offline clustering algorithms. The introduced prototypes benefit the representation ability but require all training samples for cluster assignments, leading to training instability due to label permutations (Xie et al. 2022). Unlike the existing prototypical learning methods focusing on basic classification tasks with generic designs, the proposed method uses a novel and specifically designed mixture of prototypes model for OOD detection.
METHOD
Let X' and yod denote the input and label space of the ID training data given to a machine learning model, respectively. For example, X' = RD and ydd {1....,C) C} are the input image space and the label space for multi-label image classification. A machine learning method can access the ID training dataset Did {(x;, }a)) drawn i.i.d. from the joint distribution Pxxx", and assumes the same distribution (e.g., same label space) in training and testing under a closed-world setting.
The aim of OOD detection is identify whether testing sample x E X is from ID not (i.e., OOD). In typical scenario of classification (Hendrycks & Gimpel 2016), the OOD data are from unknown classes y & ydd, i.e., )rd n yood 0. By letting Ph denote the marginal distribution of Pxxx' X, an input is identified OOD according to pid (x) < a, where threshold is a level
3

---

Published as a conference paper at ICLR 2024
C2
PP
Irpuz
Proseyss
71
U50 W3
155
M55 W5
Pajjeco
Auagment
FFALY
Fieed Farword
Uppaato
Uiu gasiliee)
Data Sample
Phutorgpa
Assggreentt Weagens
Figure 1: Overview of our proposed framework of prototypical learning with mixture of prototypes (PALM). We regularize the embedding representation space via the proposed mixture of prototype modeling. We propose (1) optimizing MLE loss to encourage the sample embeddings to be compact around their associated prototypes, and (2) minimizing prototype contrastive loss to regularize the model the prototype level. We visualize the calculation of assignment weights on the right.
set parameter determined by the false ID detection rate (e.g., 0.05) MMing al., 2023 Chen et al. 2017). Note that OOD samples are not seen/available during training, and the ID training data are labelled. In the unsupervised/uulabelle
OOD/outlier detection (Sehwag et al. 2021 Yang et al.. 2021 setting, the model can only access unlabelled ID data Did
{x+].
3.1 OVERVIEW OF THE PROPOSED METHOD
The proposed method PALM consists of three main components, as shown in Fig. I A DNN encoder used extract feature embeddings h E R from the input x with h fo(x). To regularize the representation learning, a projector 9 _ - RD followed by normalization is used to project the high-dimensional embedding h to lower-dimentional hyperspherical embedding z via z' 9o (h) and z' /l|z'll2. Relying on the mixture modeling of the hyperspherical embeddings, the proposed prototypical learning module with the associated losses is used to shape the embedding space and regularize the learning of fo and Pg. which will produce discriminative embeddings for OOD detection. Distance metrics (Sehwag et al, [021] [Sun et al. 2022) will he applied to embeddings h determine the OOD detection scores, as other distanceerepresentationnn bbased methods (Tack Tao et al. 2023 Du et al.) (022a))
fo X x-II** F
et al. 2020
3.2
MODELING HYPERSPHERICA, EMBEDDINGS WITH MIXTURE MODELS
We formulate the embedding space with hyperspherical model, considering the its benefits in represen- tation learning mentioned above (Wang& & Isola 2020 Khosla et al. 2020). The projected embeddings
z lying on the unit sphere (||z)
L) can be naturally modeled using the von Mises Fiscer (vMF) distribution (Mardia et al.) 2000, Wang & Isola 2020). Generally, the whole embedding space can be modeled as mixture of vMF distributions, where each is defined by a mean pæ: and concentration
parameter 6: PD (z; Pk: k) = (D( exp (KD, z), where p& E I is the the k-th prototype with unit norm, x 2 represents the tightness around the mean, and (D( is the normalization factor.
Prior works use single class- caassicondiional vMF distribution with one prototype to represent the samples within a specific class (Du et al. 2022a Ming al., (003).. In the whole embedding space, each sample is assigned to and pushed to the single prototype of its class. However, uniformly enforcing all samples to be close to one prototype may be unrealistic for complex data, leading to insufficient representative capability, non-compact embeddings and confusions between ID and OOD, as visual- ized in Fig. 5 We thus propose to model each class with a mixture of vMF distributions by multiple prototypes. For each class c, we define K prototypes P {Pā K corresponding
mixture
of vMF, Each z; is assigned to the prototypes via assignment weights w. E R The probability density for a sample z, in class e is defined as a mixture model:
K
plz; w.. P (R.))
wf, ZD(k) exp(xpp exx)xx(( T 22),
where w denotes the k-th element of wf. We define the same number of prototypes, i.e., same K, for each class. In PALM, the assignment weights are decided according to the adjacency relationship between the samples and the estimated prototypes, as discussed the following. Given the probability model in Eq. (I). an embedding z; is assigned to a class e with the following normalized probability:
ply: c/z:3 (w:. Pd p),n) = ΣK wf, exp(pi zz/T)
(2)
D5 EK whw exp(p's 2./T)
4

---

Published as a conference paper at ICLR 2024
where 7 = 1/KK analogous to the temperature parameter in our MLE loss.
3.3 PROTOTYPICAL LEARNING WITH MIXTURE OF PROTOTYPES
Relying on the consicooniii mixture vMF distributions of different classes, we can regularize the representation learning in the hyperspherical embedding space with the prototypes as anchors. Sharing a similar motivation to prior works, we optimize the derived end-to-end trainable objectives to 1) let each sample be assigned to the correct class with a higher probability compared to the incorrect classes, and 2) learn more informative representations for discriminating different classes, as well as ID and OOD samples. In PALM, we learn the (randomly initialized) prototypes and the assignment weights dynamically, and conduct the prototypical learning via optimizing the objectives relying on the mixture prototypes derived from the vMF mixture distributions.
Training objectives. Given the training data, we can perform maximum likelihood estimation (MLE) k== 15=1). where
according to Eq. by solving the problem max8, o II 1 p(y = clz,, {w?, {p n k=
z; is the hyperspherical embedding obtained with fo and Iø- By taking the negative log-likelhhodd the optimization problem can be equivalently rewritten as:
Ek with exp (P% 2,/T)
LMLE
log
K EC I== w exp (pÉ, ///T))"
where y' represents the class index of the sample x,, and is the temperature parameter. This MLE loss encourages samples to be close to the proper prototypes belonging to their own classes. Instead of forcing all samples in one class to be compact around one prototype as in (Ming al. [203]] Du ict al.) 2022a). PALM assigns each sample to specific prototypes according to the assignment weights. The mixture prototypes enable diverse samples to be better represented by specific prototypes.
The MLE loss in Eq. (3) only encourages the compactness between samples and the prototypes. To further shape the embedding space, we encourage intra-class compactness and inter-class discrimina- tion at the prototype level. To do this, we propose the prototype contrastive loss, which relies on the class information of the prototypes an implicit supervision signal (Khosla et al. 2020 Sehwag et al. 202T)
ΣK_ 1= c''=1 Σ 1(K" + k, c' # c) exp(p; /T/
1(W # k) explpi PT/TT)
Cproto-contra
log
(4)
CK
where 1() is an indicator function for avoiding contrasting between the same prototype. As discussed in the following, the prototypes are updated with the exponential moving average (EMA) technique. Thus the loss in Eq. (4) is mainly used to regularize the sample embeddings through the connection between samples and the prototypes.
The overall training objective PALM can be formally defined as:
LPALM = LMLE + X peroto-conira> contra>
where A > 0 the weight to control the balance between these two loss functions.
Soft prototype assignment with reciprocal neighbor relationships. The assignment weights are mainly used for calculating the MLE loss in Eq. (3) and updating the prototypes as discussed in the following, which are both in the training process with label information available. In the class-conditional mixture modeling, for each sample, we consider the assignment within its classes. Given a sample and the mixture prototypes of its class, we assign it to the prototypes relying on the adjacency information. Specifically, we first obtain the assignment weights by considering the global information with online soft cluster assignment method in (Caron al. 2020). Given K prototypes pe of class e and a batch of B embeddings Z, having B samples 2 in class e, we can obtain the assignment weights packed in matrix
PeT Jdiag(v),
W = diag(u) exp(
(6)
where u e K and v E R. are nonnegative renormalization vectors calculated for obtaining the solution (Cuturi 2013), and diag() denotes the diagonal matrix of vector. They can be obtained
5

---

Published as a conference paper at ICLR 2024
through efficient matrix multiplications using the iterative Sinkhorn-Know algorithm (Cuturi 2013). and we show the efficiency of this approximation in Appendix C.I in detail.
Assignment Pruning. Eq. (6) assigns each sample to the prototypes with soft assignment weights. Some prototypes far away from a sample can also be assigned with small weights. Although every prototype can be neighbor of sample described by weight (reflecting the distance), prototype may not be a proper neighbor of a sample, considering it mainly describes another cluster of samples very close to it. We consider further refining the assignments obtained via Eq. (6). considering the potential mutual neighbor relationship from a local perspective. For simplicity, instead of producing complicated mutual neighbor analyses, we conduct pruning on the assignment weights of each sample via top-K K shrinkage operation: ff,x == 1 [w > /] w where 3 is the K-th largest assignment weight, and 1 [wf,, > .9] E {0, 1} is an indicator function. We studied how the hyperparameter K for the top-K selection will influence the performance in Sec. 4.3 as shown in Fig. 4(a)
Prototype updating. As mentioned previously, to ensure optimal performance during the optimiza- tion of network parameters, we implemented the widely-used exponential moving average (EMA) (Li al., 2020) technique update the prototype values. This enables model parameters (fo, hg) and prototypes P to be updated asynchronously. The EMA technique allows us to iteratively update the prototype values while maintaining consistent assignment, which helps to smooth out training. This approach is aimed at avoiding sub-optimal solutions that may arise during training due to fluctuations in the prototype values. Given a batch of B data samples, we formally denote the update rule as:
B
pL == Norma 1 zzzepp + +(1 - Σ 1v: = (2)),
where 1() is an indicator function that ensures only selecting samples belonging to the same class for updating. Subsequently, we renormalize prototypes to the unit sphere for future optimization, ensuring that the distances between prototypes and embeddings remain meaningful and interpretable.
Extension to unsupervised OOD detection. In PALM, the prototypes and the assignments in the class conditiool mixture model can be automatically learned and estimated. Benefiting from this learning paradigm, we can extend PALM to the unsupervised OOD detection without any class labels available during training, with minor adjustments. Specifically, we do not use the label information and release the supervised learning version of prototype contrastive loss into an unsupervised learning version. Even with such simple modifications, PALM can perform well on unsupervised OOD setting, showing the potential. More details are left in Appendix A.4
4 EXPERIMENTS
Datasets and training details. We use the standard CIFAR- and CIFAR-10 dataset (Krizhevsky et al.) 2009) as our ID training dataset, and report OOD detection performance on series of natural image datasets including SVHN (Netzer et al. 2011 Places365 (Zhou et al.) 2017). LSUN (Yu et al. 2015), i SUN (Xu et al. 2015), and Textures (Cimpoi et al. 2014). We further assess the performance of PALM on Near- OOD detection benchmarks (Tack et al. 2020 Sun ct al. 2022). as outlined in Appendix C In our main experiments, we use ResNet-34 as our backbone model for the CIFAR- 100 ID training dataset and ResNet- 18 for CIFAR-10, along with two-layer ML. projector that projects to a 1 22--dimensional unit sphere following (Sehwag et al.) [002]] Sun
al. 2022 Ming et al. 2023 Tao et al. (023). We train the model using stochastic gradient descent with momentum 0.9 and weight decay 10 for 500 epochs. We employ the same initial learning rate of 0.5 with cosine learning rate scheduling (Loshchilov & Hutter 2017, Misra & Maaten 2020).
By default, we maintain 6 prototypes for each class and select 5 closest prototypes during the pruning phase. To ensure consistent assignment between iterations, we use large momentum of 0.999 as the default value for prototype update. For unsupervised OOD detection, none of the class labels for ID training samples are available, presenting a considerably more challenging benchmark. More experimental details are provided in Appendix B and more experiment results Appendix
OOD detection scoring function. Given that our approach is designed to learn compact representa- tions, we select the widely-used distance-based OOD detection method of Mahalanobis score (Lee) et al. 2018 Sehwag et al.) 2021). In line with standard procedure (Sehwag et al., 2021 Sun et al. 2022 Ming et al. 2023), we leverage the feature embeddings from the penultimate layer for distance metric computation. For completeness, we also compare our method using the recently proposed distance metric of KNN (Sun et al. 2022).

---

Published as a conference paper at ICLR 2024
Table I: OOD detection performance methods trained labeled CIFAR-100 ID dataset using backbone network of ResNet-34. J means smaller values are better and 1 means larger values are better. Bold numbers indicate superior results.
OOD Datusets
Average
Methods
SVHN
Places365
LSUN
ISUN
Textures
FPR
AUROCT
FPR,
AUROCT
FPR.
AUROCT
FPR.
AUROCT
FPR.
AUROCT
FPR
AUROCT
MSP
78.89
79.80
84.38
74.21
83.47
75.28
84.61
74.51
86.51
72.53
83.57
75.27
Vim
73.42
84.62
85.34
69.34
86.96
69.74
85.35
73.16
74.56
76.23
81.13
74.62
ODIN
70.16
84.88
82.16
75.19
76.36
80.1
79.54
79.16
85.22 25
75.23
78.70
78.91
Energy
66.91
85,25
81.41
76.37
59.77
$6.69
66.52
84.49
79.01
79.96
70.72
82.55
VOS
45.24
76.85
78.63
73.61
86.32
87.31
CSI
44,53
92.65
79.08
76.27
75.58
83.78
76.62
84,98
61.61
86.47
67.48
84.83
SSD+
31.19
77.74
79.90
79.39
84.08
67.16
KNN+
39.23
92.78
80.74
77.58
48. 48.99
89.30
74.99
82.69
57.15
88.35
60.22
86.14
NPOS
10.62
97.49
67.96
78.81
91.35
32.01
CIDER
12,55
97,83
79.93
74.87
30.24
92.79
45.97
88.94
35.55
92.26
40.85
89.34
PALM (ours)
3.29
84.72
98.01
94.64
28.02
Table 2: Unsupervised OOD detection performance on methods trained unlabeled CIFAR-I00 as ID dataset using backbone network of ResNet-34.
OOD Datasets
Methods
SVHN
Places365
LSUN
ISUN
Textures
Average
FPR
AUROC1
[PRC
AUROCT
FPR.
AUROCC
FPR]
AUROC1
IPRR
AUROCI
FPR
AUROCT
SinCLR+KNN
61.21
84.92
81.46
72.97
69.65
77.77
83.35
70.39
78.49
76.75
74.83
76.56
SSD
60.13
84.47
75.58
71.91
SinCLR
52.63
91.52
77.51
76.01
31.28
94.05
90.90
66.51
70.07
81.81
64.48
81.98
CST
14.47
97.14
86.23
66.93
34.12
94.21
87.79
80.15
45.16
92.13
53.55
86.11
PALM (ours)
13.86
97.53
85.63
69.46
21.28
95.95
53.43
89.06
42.62
88.33
43.37
88.07
Evaluation metries. To demonstrate the effectiveness of PALM, we report three commonly used evaluation metrics: (1) the false positive rate (FPR) of OOD samples when the true positive rate of ID samples is at 95%, (2) the area under the receiver operating characteristic curve (AUROC), and (3) ID classification accuracy (ID ACC).
4.1 MAIN RESULTS
PALM outperforms previous supervised methods by a large margin. Table shows the exper- imental results based on the standard setting of using CIFAR-100 as ID data and other datasets as unseen OOD data. For a fair comparison, all results are obtained using ResNet-34 trained on the ID CIFAR-100 dataset without access to any auxiliary outlier/OOD datasets. We compare our method with recent competitive methods, including MSP (Hendryeks & Gimpel| 2017). Vim (Wang et al. 2022). ODIN (Liang et al.] 2018), Energy (Liu et 2022)). VOS (Du et al. 2022b), CSI (Tack et al.
2020 ). SSD+ Sehwag et al. 2021). KNN+ (Sun et al. 2022
NPOS Tao et al. 2023) and CIDER
(Ming et al. 2023).
Compared to previous distance-based methods such as SSD+ (Sehwag et al., 202T) and KNN+ (Sun)
et al. 2022). which employ contrastive loss designed for classification tasks, PALM outperforms them based on the regularization designed for OOD detection. All method performs not well for the Place365 OOD dataset due its input heing confusing with the II) data. NPOS achieves the best FPR on Textures as the OOD data, since directly generates OOD samples to boost training. By modeling the embedding space with a mixture of prototypes, PALM achieves notable 12.83% reduction in average FPR compared to the most related work CIDER, which also models dependencies between input samples and prototypes. PALM outperforms the most recent work of NPOS, without the need to generate artificial OOD samples. Moreover, PALM achieves new state-of-ther thte-ht-rtttt art level of AUROC
performance. We also report the results on OpenOOD benchmark (Yang et al. 2022
Zhang et al.
2023 in Table in Appendix C
PALM outperforms competitive unsupervised approaches. As demonstrated in Table our method in the unlabeled setting outperforms previous approaches, including SimCLR (Chen et al.
2020 + KNN (Sun et al. 2022), SimCLR (Chen et al., 2020 Tack et al.) 2020). CSI (Tack et al. 2020), and SSD (Sehwag al. 2021b. Although our primary contribution does not specifically
target this unlabeled setting, we still observe a significant performance boost, surpassing CSI on most of the datasets. It is worth noting that CSI utilizes an ensemble of different transformations and incurs 4 times the computational cost during both training and testing. Remarkably, our unsupervised method achieves on par superior performance compared to our supervised prior works NPOS and
7

---

Published as a conference paper at ICLR 2024
CIDER various test datasets under the same training budget without any label information. More experiment results and discussions in Appendix C.2
4.2 DISCUSSIONS
PALM learns more compact cluster around each prototype. Comparing to previous distance-based methods that encourage samples of each class to be close to each other (Tack et al., 2020; See hwag et al., 2021 Sun et al. 2022) or its single pro- totype (Ming et al, 2023), PALM considers more realistic embedding space where samples are enforced to be close to its most similar prototype of its class and learn more compact embedding space. In Fig 2 (top), we evaluate the embedding quality of PALM by calculating the cosine similarity of ID samples to their nearest prototype. We observe more compact distribution with a higher number of similar samples and significantly fewer dissimilar samples. Notably, CIDER struggles with the representative ability of pro- totypes, where some II samples even exhibit near-zero similarity. We also quantify the number of far ID sam- ples, whose similarity is below 0.8, in Fig. 2 (bottom). To provide a comprehensive analysis, we additionally assess the compactness measurements of (Ming al., 2023) in Fig. 2 (bottom), where it measures the av- erage cosine similarity of input samples to its closest prototype.
CIDER PALM
Density
600
01.1
0.0
0.2
0.4
0.6
D.8
1.0
Cosine similarity
Compactness | (in degree)
Number of far ID samples (%) J.
Methods
CIDER PALM 24.21
31.08
25.79
15.71
Figure 2: Analysis of embedding quality for CIFAR-100 (ID) using CIDER and PALM. We examine the distance distribution (top), and evaluate compactness and the propor- tion of far ID samples (bottom).
Breals
Dredty
12
04
00
28
02
24 06
85 12
82 24 43 0.
(a) SSD+
(b) KNN+
(c) CIDER
(d) PALM (ours)
Figure 3: UMAP (Melnnes et al. 2018) visualization of the first 20 subclasses of ID (CIFAR-100) and all OOD (iSUN) samples plotted to the same embedding space for methods including (a) SSD+ (Sehwag et 202T (b) KNN+ (Sun al. 2022) (c) CIDER (Ming et al., 2023) and (d) PALM. The
scores are obtaned by scaling the distance metrics used by each method to [0, I for visulization. We
measure the area of overlapping sections between ID and OOD scores, as shown in Fig. S
PALM enhances the separability between ID and OOD. We visualize the embedding distribution of ID (CIFAR-100) and OOD (iSUN) samples (central cluster in pink) using UMAP (McInnes et al. 2018 in Fig. 3 In contrast to previous works that suffer from correlations between OOD samples and
numerous ID clusters (Figs. 3(a Bb BC). PALM exhibits only weak correlation with a single ID cluster (Fig. 3(d)) Notably, PALM further enhances the separability between ID and OOD samples
by promoting more noticeable distance between ID clusters less correlated to OOD samples.
Additionally, we estimate the distance density distribution of ID and OOD samples for each UMAP visualization in Fig. 3 and quantitatively measure the of overlapping sections in Fig. 5 Our method significantly promotes a large separation between ID and OOD samples, suggesting a more effective OOD detection performance.
4.3 ABLATION STUDIES
PALM generalizes well to various dastance-based OOD detection scoring functions. To demonstrate the effectiveness our proposed representation learning framework for OOD de- fection, perform ablation studies the choice of distance metrics in Table 3 We tested
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation on distance metrics selection for OOD detection. We evaluate two widely used
distance metrics of KNN distance (K=300) and Mahalanobis distance.
Metries
Methods
SVHN
Places366
OOD Datasets
LSUN
ISUN
Textures
Average
FPRL
AUROC1
FPRL
AUROC'
FPP..
AUROCT
FPR
AUROC1
FIR
AUROCI
FPR1
AUROCT
KNN+
39.23
92.78
90.74
77.58
18.99
89.30
74.99
82.69
57.15
88.35
60.22
86.
KNN
NPOS
CIDER
10.62
12.55
97.53
97.49
67:96
79.93
74.87
78.81
30.24
20.61
92.79
92.61
35.94
45.97
88.94
88.94
24.92
35.55
92.25
91,35
40.85
32.01
39.84
89.34
(ours)
61.87
14.60
30.76
92.06
90.84
92.09
Mahalanobis
SSD+
31.19
94.19
77.74
79.90
79:39
85.18
80.85
84.08
66.63
86.18
67.16
85.91
PALM (ours)
3.29
99.23
64.66
84.72
9.86
98.01
28.71
94.64
33.56
92.49
28.02
93,82
AUTOC
AUROC
5
Natrorr of toph sndigumenta
11stasr
Number procotypos clase
Difased
(a) Top-K assignment #.
(b) Assignment rules.
(c) Number of prototypes.
(d) Prototype updating.
Figure 4: Ablation studies on (a) pruning selection, (b) soft vs. hard assignments, (c) number of prototypes for each class and (d) prototype update procedure.
our method on two commonly used distance metrics, KNN distance (Sun et al. 2022, Ming
al. 2023 Tao et al. 2023) and Mahalanobis distance (Lee et al., 2018 Sehwag et al.
2021)
Our method achieves superior performance on these two widely- distance metrics.
PALM improves large-scale OOD detection. We demonstrate the performance of PALM on a more challenged large-scale benchmark in
Amas of interwction (9))
Fig. 6
We consider ImageNet-100 (Tian et al.] 2020) as our ID
5 1 to 5 g
training dataset. Following (Huang & Li 2021 Ming et al.) (023), we
evaluate our OOD detection performance on test datasets SUN (Xiao et al.)
2010), Places (Zhou et al., (017)). Textures (Cimpoi et al. 2014)
and iNaturall ist Van Horn et al. 2018). More experiment details are left in Appendix B and experiment results in Appendix C.1
Patbort
Figure Area of over- lapping sections between ID and OOD distance densities in percentage. Smaller numbers indicate superior results.
Pruning less relevant assignments benefits the learning process. Fig. 4[a] demonstrates the efffcctiveness of our introduced pruning procedure by top-K selection. We observe a significant improvement in performance when selecting more assignments, up to 5 out of the 6 available prototypes.
Soft assignment helps model find optimal solutions. In Fig. 4(b] we compare the performance between soft assignments and hard assignments. Soft assignments help preventing stuck in sub-optimal local minima.
Ablation number of prototypes. We demonstrate the effects of varying the number of prototypes in Fig. 44C)) and observe an increase in
Eakil
ABRIC
performance the number of prototypes increases. We find that defining
only one prototype for each class, results in worse performance.
EMA provides more consistent cluster assignment. As widely dis- cussed in the literature (He et al. 20200; Grill et al.) (2020; Ming et al. 2023), an asynchronous update between the model parameters of the two views or between model parameters and prototypes is fundamental to the success of recent representation learning methods. Here we also examine the effectiveness of the EMA update used for our prototypes in Fig, 4(d)
50/
Tester Aet Wala Aureey
Dutuxet
Figure 6: OOD detection performance on meth- ods by fine-tuning pre- trained ResNet-50 mod- els ImageNet-100..
CONCLUSION
In this work, we propose PALM, novel prototypical learning framework with a mixture of prototypes that learns hyperspherical embeddings for OOD detection. By considering the complex underlying structure of data distributions, PALM model the embedding space by mixture of multiple prototypes conditioned on each class, encouraging a more compact data distribution and demonstrating superior scale method
OOD detection performance. Moreover, impose two extensions where to large-scale datasets and unsupervised OOD detection. The limitation is that PALM needs to be manually assign the number of prototypes hyperparameter, which is left future work.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In International Conference on Learning Representations. 2020. httttps://openrevl.eet net/ forum? ret/oorrmm..-Hys ]yBFFr
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by con-paaamttriaaly predicting view assignments with support samples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8443-8452, 2021.
Mu Cai and Yixuan Li. dittii-disttibuttion detection via reegulecy generative models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5521-5530, 2023.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-999 2-9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Yen-Chi Chen, Christopher Genovese, and Larry Wasserman. Density level sets: Asymptotics, inference, and visualization. Journal of the American Statistical Association, 112(520):1688 1686-1666, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Nick Drummond and Rob Shearer. The open world assumption. eSI workshop: the closed world of databases meets the open world of the semantic web, volume 15, pp. 1, 2006.
Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out- out-of-distribution objects. In Advances in Neural Information Processing Systems, 2022a.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. Towards unknown-aware learning with virtual outlier synthesis. In International Conference on Learning Representations, 2022b. URL https: https://cpenreviie net http://oummrdddddeeeme
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based mod- els. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus Bucc E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL htt.:////2.2.a.....a..aaaaaaaaa3 Pappp
reurips....pp..... www.rriss.ccpppeeee wwtpcpprppppppeeeeeeee fi lesl/epprr
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 1714-1714. 2019.
Will Grathwohl, Kuan-Chieh Wang. Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly energy based model and you should treat it like one. In International Conference on Learning Representations, 2020. URL https net. ottpeeeeeeeeeeoooooooooooooooooo
10

---

Published as a conference paper at ICLR 2024
Jean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.021366 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of out-of--istribution examples in neural networks. In International Conference on Learning Representations, 2017.
URL https:://// /
net forum wow.mmmdd-nkggatt
Rui Huang and Yixuan Li. Mos: Towards scaling citto--distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8710-8719, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances Neural Information Processing Systems, 34:677-689, 2021.
Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp. Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100270, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances neural information processing systems, 33:18661-18673, 2020.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 813-822, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributio samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009.000 007955 2020.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsu-
pervised representations. International Conference on Learning Representations, 2021. URL https://cpenrevia
net /ttp//ooommiidnyyyypssssss.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image
detection in neural networks. In International Conference on Learning Representations, 2018.
URL httpth:::://peeeee//aas
net wow.mmmmmddd1kkII http:://pppaeeeeeeeeeeese/eoooot
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of--isttinutioo detection.
Advances in neural information processing systems, 33:21 04::1464-2111 21475, 2020.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In
International Conference on Learning Representations, 2017. URL
het:forrmmio==kgeeee
11

---

Published as a conference paper at ICLR 2024
Kanti V Mardia, Peter E Jupp, and KV Mardia. Directional statistics, volume 2. Wiley Online Library, 2000.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embed- dings for out-of-distribttion detection? In The Eleventh International Conference on Learning Representations, 2023.
Ishan Misra and Laurens der Maaten. Self-supervised learning of gretext-invariant representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6707- 6717, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Andre T Nguyen, Fred Lu, Gary Lopez Munoz, Edward Raff, Charles Nicholas, and James Holt. Out of distribution data detection using dropout bayesian neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7877-7885, 2022.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Jie Ren, Peter J Liu, Emily Fertig. Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for diitt--dittiibutiin detection. Advances in neural information processing systems, 32, 2019.
Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanic Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021. URL
net -
Seonghan Ryu, Sangjun Koo, Hwanjo Yu, and Gary Geunbae Lee. Out-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 714-718, 2018
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, Advances in neural information processing systems, 30, 2017.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 00827-20846 PMLR. 2022.
Jihoon Tack, Sangwoo Mo. Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11839-11883 11852, 2020.
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://openre//
net fore//orumm20 iidiidd-i Jadk-JKkkEzqqpp
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCo 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
12

---

Published as a conference paper at ICLR 2024
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 87788, 2018.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 2495-2504, 2021.
Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4921 1921-4930, 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939. PMLR, 2020.
Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based open-world uncertainty modeling for confidence calibration. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9302-9311, 2021.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631-23644. PMLR, 2022.
Zhirong Wu, Yuanjun Xiong. Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in neural information processing systems, 33:20685-20696, 2020.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Delving into inter- image invariance for unsupervised visual representations. International Journal of Computer Vision, 130(12):299433113 2022.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsoureing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized dittr--distribution detection: A survey. arXiv preprint 1xXiv:2110.113:: 2021.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li. Yiyou Sun, al. Openood: Benchmarking generalized out-of- distribution detection. Advances in Neural Information Processing Systems, 35:32598-3261 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 0336,, 2015.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, al. Openood v1. 5: Enhanced benchmark for out-of-distrinn nut-o--distribution detection. arXiv preprint arXiv:2306.0999 09300, 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralb. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence,
2017.
13

---

Published as a conference paper at ICLR 2024
Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmen- tation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 2582-2593, 2022.
David Zimmerer, Peter M Full, Fabian Isensee, Paul Jäger, Tim Adler, Jens Petersen, Gregor Köhler, Tobias Ross, Annika Reinke, Antanas Kascenas, et al. Mood 2020: A public benchmark for dittr--distribution detection and localization on medical images. IEEE Transactions on Medical Imaging, (1110)2728822333 2022.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A MORE DETAILS OF THE PROPOSED METHOD
A.1 ALGORITHM DETAILS
We present the training scheme of the proposed method, Prototypical Learning with Mixture of prototypes (PALM), in Algorithm | The algorithm maintains and updates prototypes, conducts the soft prototype assignment for the samples, optimizing the MLE loss and the prototype contrastive loss. Algorithm [ summarizess the main operations in the training process and omits some calculation steps, e.g., data augmentation.
In step 8 of Algorithm[] [] the prototype assignment weights are calculated by considering the global information on the relationship between sample embeddings and the prototypes. The weights are then further refined, considering the mutual neighborhood relationship in step 9. By pruning the weights to keep top-K neighbor prototypes for each sample, the operation makes sure the samples and the assigned prototypes are not too far away from each other. Intuitively. the samples the prototypes in the EMA updating process. The prototypes work as "anchors" to pack-propagate the gradients from optimizing the objectives update the NN parameters, where the mixture of prototypes and the hyperspherical modeling regularize the NN representation learning. At the end of each iteration, the prototypes are detached from gradient caleulation for the next round of updates.
"attached" to
Algorithm 1: Prototypical Learning with Mixture of prototypes (PALM)
Input:
Training dataset Di with C' classes, neural network encoder fe. projection head gø. loss weight, other hyperparameters, egg., number of prototypes K for each class. Initialize the NN parameters;
Initialize the class-conditional prototypes () for each class c;
// No grad. on initializes prototypes while training stopping conditions are not achieved do
/ mini-batch iterations in each epoch: for iter 2, do
Sample mini-batch of B input samples (x), Obtain projected embeddings for each input: x = (x,))
B): Normalize the embeddings to the unit sphere via z, /2/22 E {l,..., B);
// Soft prototype assignment
in Step 8-9:
Compute soft assignment weights for each set of the class-conditional mixture of prototypes {pk 1k 1 1, .... C, relying on Eq. (6):
Obtain the assignment weights via assignment weights pruning via Sec. 3.3 Prototype updating via EMA Eq. 7 (based on assignment weights);
10
// Optimizing training ob objectives:
// Grad. back pack-propagating to NN parameters through the prototypes.
11
Calculate the objectives with Lassgn and Ceroto-comaa in Eq. (3) and Eq. (4
respectively,
and optimizing for updating network parameters;
12
Detach the updated prototypes from gradient calculation (for next round of updating):
A.2 MORE DETAILS ON PROTOTYPE ASSIGNMENT AND UPDATING
During training, we assign each sample to the prototypes with soft assignment weights. In Sec. 3.3 of the main paper, introduce how calculate the assignment weights considering the global infor- mation and then refine the weights via pruning operation considering the local mutual neighborhood relationship. We put more details about the assignment weights calculation and prototype updating in the following.
Estimation of the soft assignment weights. Recall that we have K prototypes p ) for each class where 3 E DR In mini-batch training, obtain batch of B samples and the
15

---

Published as a conference paper at ICLR 2024
As introduced in Sec. 3.2, each sample/embedding is assigned to prototypes in its corresponding class with weights w R Instead of assigning each sample one prototype with a hard assignment, assignment weights to approach more smooth modeling. Experiments in Sec. 4.3 the benefits of soft assignments in Fig.
corresponding projected embeddings { with R?, as shown in Algorithm 1
soft
[bb In each iteration of the mini-batch training, the prototype assignment aims to obtain the weights w for each sample.
To avoid the trivial solution where all samples may be assigned to the sample prototype (Caron et al, 20200] Gong et al., 2019; Zhoo et al., 2022) in the online updating process, we encourage the samples in a batch (belonging to the same class) be assigned to diverse prototypes (for the corresponding class), as the previous methods. As briefly introduced in the main paper, we define Z E RR to represent the concatenated embeddings of the class c, i.e., Z = [21, Ziy BB /wi.... : B
P DxK with P [pf...., PG, ]. and w KxB" for W where B denotes the number of samples belonging in to class c in the batch. To encourage the diverse prototype assignment, we optimize W (for each class c) by solving the following problem to maximize the similarity of the assigned prototypes and the samples (Caron et al. 2020):
max WEEW
Tr(W" P Z²) <R(W"),
(8)
where R-)) is the entropy function as the regularize on W i.e., --E wi, log w9, with wf, to denote the i, j.j element in the matrix W is the regularization weight, Wi is the defined domain of W Following (Caron et al., 2020), W is defined as transportation polytope (Asano al. 2020), in the format of
1 K I W 1x - In).
= = (W E K W1, =
where 1 and y denote the vector of ones in dimension of K and B, respectively. can be handled efficiently in Problem Eq. (8) Please find more details of the definition of the transportation polytope in (Caron et al., 2020 Asano al., 2020 Cuturi 2013). )) As introduced in the main paper, the soft assignment weights can be obtained by solving the problem in Eq. g as the following normalized exponential matrix (Cuturi 2013):
P Z
W = diag(u) exp(
)diag(v),
6
where u and v are renormalization vectors in R K and RB obtained from a few steps of Sinkhorn- Knopp iterations (Cuturi 2013 Caron et al., 2020). As shown in Lemma in (Cuturi 2013). the solution of W with the transportation polytope constraint in Eq. (9) is unique and can be obtained via the form of Eq. (6). and v are two non-negative vectors for obtaining the solution, which can be computed with Shinkhorn's 's fixed point iteration (u, v) <- (1/(KHv), 1/BHu)), where we use H to denote xxp(P"]Zz) in Eq. (6) With further simplification, u and v can be obtained with few iterations, as discussed in Algorithm in (Cuturi 2013). Please find the proof and more details in (Cuturi. 2013). diag(1) and diag(v) are the diagonal matrix of the two vectors u and v. Through solving the problem in Eq. (8). the assignment weights are obtained by considering the similarity between embeddings and prototypes (as reflected by Pe Zz) and the relationships between the prototypes and all the samples in a global viewpoint. In practice, the calculations for different classes can be conducted simultaneously and in parallel.
A.3 MORE STEPS FOR OBTAINING EQ. 2
The probability density for a z; in class is defined as the mixture model in Eq. ①I in the main paper:
K
p(2; wf, (pā x) K
Σ
Wf exp(<ph 2z,,,
where wfee denotes the k-th element of wę. We define the same number of K prototypes for each class. The assignment weights are decided according to the adjacency relationship between the samples and the estimated prototypes, in PALM. Given the probability model in Eq. (I) an embedding z, is assigned to class with the following normalized probability:
Σk wf, Z (K) exp(Epp 2i)
p(y: = clz,, (W.. (pf x) k
(10)
E9 ΣK ZD(x) exp(Mpp 2;)
16

---

Published as a conference paper at ICLR 2024
where 7 = 1/K i is analogous to the temperature parameter in MLE loss. By eliminating Zp(k).
we can obtain Eq. (2) in the main paper as:
Ex= "ho exp(pX 2,/T) ply = clzi, {wi, (pks k)== 11=1))
Ej=1 E Skml wº, exp(p), 2:/T)
A.4 EXTENDING PALM TO UNSUPERVISED OOD/OUTLIER DETECTION
In this work, we mainly study the OOD detection tasks with the labeled datasets with the class label for each sample. Different from previous methods (Ming et al. 2023 Du et al.) 2022a) assigned single prototype to each class, we automatically maintain and update a mixture of multiple prototypes
for each class. Benefiting from PALM's ability to learn multiple prototypes for set of samples automatically, we study the potential of the proposed techniques to handle the OOD/outlier detection with unlahelled ID training data.
On the unlabelled data, we can define K prototypes P = {p= for the entire training dataset without label information and conduct the similar prototype assignment process between all samples and all prototypes. The contrastive loss can be applied to encourage each sample embedding z to be close to the prototypes based on the assignment weights in the form of
K
Wie exp (p ZTTT)
E(zi, w.) = - log
(11)
k=1 Σ
E== exp Z
K
Without label information, Eq. (11 encourages the sample embeddings to be close to the prototypes based on the weights. Note that the underlying model of assignment weights here is slightly different from the weights in the class- conditsondiio mixture model under the supervised setting. The model also has the potential to consider the relationships between prototypes relying the distance to produce latent class-level clustering. The prototype updating and assignment can still be conducted as the standard PALM.
We produce a simple implementation to adapt the proposed method on the unsupervised setting and conduct experiments validate the potential, shown in Appendix C.2 Due to the lack of strong supervision information, we utilize data augmentation in implementation, relying on the swapped assignment in (Caron et al.; (220)). By letting ž, denote the embedding of the augmented version of the sample and w; denote the corresponding assignment weights obtained for 2,, the loss function based on Eq. [1] can be written as:
N
1
Cussuer
(L(z), w:) + ((Z,,,,)),
(12)
N
2
which is similar to the framework of SwAV for self-supervieed learning (Caron et al. 2020). With the experimental results in Appendix C.2
show a hint of the potential of the proposed techniques for unsupervised OOD detection and will study more detailed design of that in the future works.
B MORE IMPLEMENTATION DETAILS
Software and hardware. We perform all experiments on an NVIDIA GeForce RTX-3090 GPU using Pytorch.
Training details for experiments on CIFAR benchmarks. We utilize ResNet-34 as our backbone model for the CIFAR- 100 ID training dataset, and ResNet-18 for CIFAR- along with a two-layer MLP projector that projects to 1 128-dimensional unit sphere following (Sehwag et al.) [021] Sun
et al. 2022 Ming et al.) 2023 Tao et al. 2023). We apply stochastic gradient descent with momentum of 0.9 and weight decay of 10 for 500 epochs. We adopt the same initial learning rate of 0.5 with cosine learning rate scheduling [oohchilov Hutter 2017: Misra & Maaten] 2020). and
use the temperature of 0.1 and 0.5 for the MLE loss and prototype contrastive loss, respectively. We set loss weight as 1 to balance the proposed two loss components for simplicity. Following (Caron et al. 2020), we use value of 0.05 and 3 Sinkhorn-noop iterations to obtain our soft
17

---

Published as a conference paper at ICLR 2024
Table OOD detection performance on methods trained on labeled CIFAR-I0 as ID dataset using backbone network of ResNet-18.
OOD Datasets
Methods
SVHN
Places365
LSUN
iSUN
Textures
Average
FPR
AUROCT
FPR.
AUROCT
FPR
AUROCT
FPR
AUROCT
FPR.
AUROC
FPR
AUROCT
MSP
59.66
91.25
62.46
88.64
51.93
92.73
54.57
92.
66.45
88.50
59.01
90.65
ODIN
20.91
94.82
33.17
94.65
91.56
Vim
24.95
95.36
63.04
86.57
7.26
98.53
33.17
94.65
56.40
86,21
36.96
92.26
Energy
27.52
92.67
VOS
15.69
96.37
37.95
91.78
27.64
93.82
30.42
94,87
32.68
93.68
28.88
94,10
CSI
37.38
94.69
38,31
93.04
10.63
97.93
10.36
98.01
28.85
94.87
25.11
95.71
KNN+
2.70
99.61
23,05
94.88
7.89
98.01
24.56
96.21
10.11
97.43
13.66
97.23
SSD+
2.47
99.51
22,05
95.57
10,56
97.83
28.44
95.67
9.27
98.35
14.56
97.39
NPOS
4.96
97.15
17.61
91.29
3.94
97.67
13.69
95.01
7.64
94.92
9.57
95.21
CIDER
2.89
99.72
23.88
94.09
5.75
99.01
20.21
96.64
12.33
96.85
13.01
97.26
PALM
0.34
99.91
28.81
94.80
L.I1
99.65
34.07
95.17
10.48
98.29
14.96
97.57
online assignment weights. By default, we keep 6 prototypes for each class and select the 5 closest prototypes during the pruning phase. To secure consistent assignment weights across iterations, set large momentum a of 0.999 as the default value for prototype updating.
Details for compared methods. We follow the standard settings in previous methods (Sehwag et al. 2021; Sun et al., 2022 Ming et a 2023 Tao et al. 2023) to produce the results of the compared state-of-the-an art methods. Details are in the following. For methods that derive OOD detection scores from models trained using standard cross-entropy (i.e., MSP (Hendrycks & Gimpel 2017), Vim (Wang et al. 2022), ODIN (Liang et al. 2018). Energy (Liu et | (022)), we train the model with initial learning rate of 0.1 and decays by a factor of 10 at epochs 50, 75, and 90 respectively for 200 epochs on CIFAR- 100 dataset. We use stochastic gradient descent with momentum 0.9 and weight decay 10-4. 4 For most recent powerful methods, including VOS (Du et al. 2022b). CSI (Tack et al. 2020), SSD+ (Sehwag et al. 2021), ) KNN+ (Sun et al., 2022), NPOS (Tao al. 2023), and CIDER (Ming et al.] 2023), we train the model with batch size of 512, using stochastic gradient descent with the same initial learning rate of 0.5 Khoolla et al. 2020 Sehwag et al. 2021 Ming et al. (0023) with
cosine learning rate scheduling (Loshchilov & Hutter 2017: Misra & Maaten 2020).
Details for experiments on ImageNet- 100. We use ResNet-50 for ImageNet- dataset and project the embeddings to 128- 228-dimensional unit sphere with two-layer MLP projector. Following the commonly used setting in OOD detection (Ming et al.] (003). we produce fine- fine-tuning the last residual block and the projection layer pre-trained ResNet backbone for 10 epochs, with learning rate of 0.001.
C ADDITIONAL EXPERIMENTAL RESULTS
OOD Detection Performance on CIFAR-10. In Table I we present the efficacy of PALM against recent competitive OOD detection methods on the challenging CIFAR-100 benchmark. Complemen- tary comparisons on the less challenging CIFAR- 10 benchmark are illustrated in Table[C] The results indicate that, while recent competitive methods yield comparable performance on this benchmark, PALM continues to achieve results that are on par or even surpass on these OOD datasets. This further demonstrates the efficacy and adaptability of our proposed framework.
PALM performs superior on Near-OOD detection. We evaluate the performance of Near-OOD detection following (Tack et al. 2020 Sun et al.) 2022). utilizing ID dataset CIFAR-100 and OOD datasets including -SUN-FIX. ImageNet-F IX, ImageNet-RRSTTT our work is primarily evaluated on the Far-OOD scenario, PAIM nonetheless exhibits state-of-the-art performance these more challenging benchmarks even without explicit considering Near-OOD scenarios. Specifically, PALM yields an improvement of 13.48% in the average FPR and achieves an 81.76 average AUROC score. Importantly, PALM substantially reduces the FPR on ImageNet-R from 56.89 to 27.02, effectively halving the false positives.
and CTFAR-10. 10. While
PALM outperforms previous methods using OpenOOD benchmark. In Table 6 extend the evaluation of our method using the OpenOOD benchmark Yang et al. )0 Zhang et al. (0023). Our findings indicate that PALM achieves superior performance on both CIFAR- 10 and CIFA 1IFAR-100
18

---

Published as a conference paper at ICLR 2024
Table Near-OOD detection performance on methods trained on labeled CIFAR- 100 as ID dataset using backbone network of ResNet-34.
OOD Datasets
Methods
LSUN-F
ImageNet-F
ImageNet-R
CIFAR-10
Average
FPR
AUROC
FPR.
AUROCT
FPR.
AUROCT
FPR.
AUROC
FPR.
AUROCT
MSP
88.24
69.21
86.33
70.74
86.32
72.88
88.06
76.30
87.24
72.28
Energy
87.17
72.20
78.99
76.40
80/93
80.60
86.47
70.50
84.21
74.93
SSD+
83.36
76.63
76.73
79.78
83.67
81.09
85.16
73.70
82.23
77.80
KNN+
84,96
75.37
75.52
79.95
68.49
84.91
84.12
75.91
78.27
79.04
CIDER
90.94
70.31
78.83
77.53
56.89
87.62
84.87
73.30
77.88
77.19
PALM
77.15
77.24
66.19
82.51
27.02
95.03
87.25
72.28
64.40
81.76
Table 6: OOD detection performance on methods using OpenOOD benchmark.
0OD Datasets
Methods
CIFAR-10
CIFAR 100
Near-OOD
Far-OOD
Near-OOD
Far-OOD
MSP
88.03
90.73
80.27
77.76
ODIN
82.87
87.96
79.90
79.28
Vim
88.68
93.48
74.98
81.70
Energy
87.58
91.21
80.91
79.77
OpenGAN
53.71
54.61
65.98
67.88
VOS
87.70
90.83
80.93
81.32
CSI
89.51
92.00
71.45
66.31
kNN
90.64
92.96
80.18
82.40
NPOS
89.78
94.07
78.35
82.29
CIDER
90.71
94.71
73.10
80.49
PALM
92.96
98.09
76.89
92.97
datasets compared to previous methods. This further highlight the efficacy and robustness of our approach using the OpenOOD framework.
More detailed results for the experiments on large-scale OOD detection task. In Fig. 6 in the main paper, evaluate the OOD detection performance large-scale dataset ImageNess 100 (Tian et al. (020). Here we provide more detailed performance in Table 7 We fine-tune the last residual block and projection layer of pre-trained ResNet-50 model, while freezing the parameters in the first three residual blocks, following the setting in the previous work (Ming et al., 2023). We compare our approach with recent powerful methods including SSD+ (Sehwag et al. 2021), KNN+ (Sun et al. 2022) and CIDER (Ming et al., 2023). PALM maintains an outstanding performance compared to previous methods, suggesting the effectiveness of our proposed mixture prototypes framework.
ID classification accuracy. We demonstrate the ID classification accuracy in Fig. 7 Fig. shows 2021 & KNN+
the ID classification accuaracy of standard cross- cross-entropy (CE), SSD+ (Schwag et al.
(Sun et al. (022) (use the same training objective of SupCon (Khosla et al. 2020)) NPOS Tao et al. 2023), CIDER (Ming et al. 2023), and PALM. Commonly used linear probing (Khosla et al. 2020
Table 7: OOD detection performance on methods fine-tuning on ImageNet- using pre-trained ResNet-50 models.
OOD Datasets
Average
Methods
SUN
Places
Textures
iNatualist
FPR.
AUROC1
FPR.
AUROC1
FPR.
AUROCT
FPR..
AUROCT
FPR.
AUROC1
SSD+
93.06
34.38
26.49
94.84
38.19
90.96
32.35
92.60
KNN+
41.85
92.25
44.41
26.60
94.22
38.54
94,15
37.85
92,72
CIDER
42.26
92.84
42.81
91.39
19.31
95.44 35.44
45.49
92.83
37.47
93.12
PALM
42.37
93.20
41.22
91.95
17.02
96.16
32.08
95.14
33.17
94.11
19

---

Published as a conference paper at ICLR 2024
70
20-
032
1001 KNN6
XP1S
PANM
Method
Figure 7: ID classification accuracy of methods trained on CIFAR-100.
Table 8: OOD detection performance comparisons methods trained on labeled CIFAR-100 as ID dataset using ResNet-34 over independent runs. The experiment results for CIDER are obtained
from Ming al. 2023
Datmetts
Averuge
AUROCT
AUROC1
AUROCT
AUROCC
ADRRCC1
25,737
22.01-3111
96.0 10
16
83 20
#1 H.90tem
87 (7.00=110
8.314
09.2
66.51-1.1
63.00-144
10.03-300
97,83-90
90,10-4.0
9481437
35.40
92.21 46.1
$1.3/4534
98 98.02-1.0
used obtain linear classification accuracy for SupCon, CIDER, and PALM. While producing effective OOD detection, PALM can perform well for ID classification with better or competitive results than other methods, showing the effectiveness of the representation learning.
Stability of PALM. To validate the stability of PALM on producing state-of-the-art performance, we train PALM using distinet random seeds and present the average and standard deviation of FPR and AUROC in Table 8 Together with Table in the main paper, it is shown that PALM consistently achieves wate-of-the-aa performance with small standard deviation.
C.1 ADDITIONAL ABLATION STUDIES
Prototype contrastive loss improves OOD detection capability. We demonstrate the efficacy of the introduced prototype-level contrastive loss proto-ccontra in Fig. [(a)aand Table 9 Even without the application of L-pott-contra: our method PALM exhibits state-of-the ofit-----te-attttt performance. Moreover, by inte- grating our proposed Cproto-contra that encourage intra-class compactness and inter-class discrimination at the prototype level, PALM is elevated to a new level of promising performance.
1000
97.
07.5
93.0
07.5
90.5
AUROC
00.0
87.5
SVIEN
Phoselto
ISUN
82.5
Thstures
SVIIN
Planid35
LSUN
I8UN
Teccure
Aweryy
0.38
0.055
0.530
19995
Momemtonn
(a)
Cororo-contm
(b) Momentum of EMA
Figure 8: Ablation studies on (a) prototype contrastive loss Lproto-contm (b) the momentum update parameter a EMA update procedure.
20

---

Published as a conference paper at ICLR 2024
PALM with different momentum hyperparameters for prototype updating. In Fig. 8(b) analyze the influence of maintaining consistent prototype assignment weights. Our observation confirms that utilizzing higher momentum to ensure less changes in the prototypes across iterations, thus achieving more consistent assignment weights, leads to improved performance. The results also show that PALM can perform well consistently with large enough momentum. Moreover, the momentum parameter a mainly controls the updating strength in the iteration (like a learning rate), which does not influence too much the training behaviours on different data. The experiments show that the influence of a is small. A properly set hyperparameter can work well for many datasets. During the updating of prototypes, the sample-prototype assignment weights are highly relevant to the class/data characteristics. We update with the robust cluster assignment method in Eq. and use the assignment pruning method to alleviate the influence of unexpected points prototype updating.
Table Ablations on loss weight
Table 10: Ablations on temperature T
Loss Weight À
FPR
AUROCT
Temperature
FPR|
AUROC1
0
33.77
92.42
0.05
49.70
87.51
0.1
33.04
92.11
0.1 (default)
28.02
93.82
0.5
27.65
93.34
0.2
52.88
84.74
1 (default)
28.02
93.82
0.3
76.82
75.18
Ablation on temperature parameter. In alignment with conventional contrastive learning methods (Chen et al. 2020 Khosla et al. 2020), the temperature parameter 7 plays crucial role in the training process, serving as a hyperparameter to regulate the compactness of each cluster. adeptly weights different examples, and selecting an apt temperature is instrumental in aiding the model to learn from hard negatives effectively. Through empirical examination, we empirically found that a temperature of 0.1 yields optimal results.
Ablation on initialization of prototypes. Table T2, we explore the impact of different prototype initialization strategies. Our analysis compares approaches: initialization with random initializa- tion using normal distribution, and random initialization with a uniform distribution, the latter being default setting. The two random initialization variants demonstrate relatively similar performance, showcasing the efficacy of PALM when given proper initialization. This finding further demonstrates the robustness of our approach in updating the prototypes using EMA, showcasing its effectiveness and high-quality prototypes regardless of the initial state of the prototypes.
Computational efficiency of PALM. In Table 11 demonstrate the training speed of our method when training on CIFAR- 100 using ResNet-34 with single GeForce RTX3090 GPU, we are compara- ble as popular contrastive learning method SupCon (Khosla al. 2020) with negligible overhead. The additional computations of our method are mainly caused by calculating the soft assignment weight using the efficient Sinkhorn- KinhhornKnoop approximation, which takes only 32 milliseconds each time.
In terms of memory, the additional parameters introduced by the multi multi- poaltiipo design are only the prototypes vectors, and the assignment weights for each batch are temporally calculated during training. Under the default setting with 6 prototypes for each of 100 classes (i.e., 600 prototypes in total), it introduces 1.2% of additional parameters compared to the single prototype method. Specifically, based on ResNet-34 with 21,60M, CIDER introduces one prototype per class, increasing the parameters to 21.65M, while our method is 21.91M. The additional computation caused by ours is restricted.
Table 11: Training time for different methods.
Training Time (seconds per epoch)
Methods
CE
12.34
SupCon 19.23
CIDER
23.65
PALM
19.87
21

---

Published as a conference paper at ICLR 2024
Table 12: Ablations the different ways of initializing the prototypes.
Thitasels
of Protetypes
Places365
Average
FPR]
AUROCT
FPX.
AUROC
FPR]
AUROCT
FPR.
AUROC
FPR.
AUROCT
FPR]
AUROCT
Normal Distribution
2.70
99.39
06.04
$1.33
10.06
97.77
32.97
94.26
41.05
91.08
30.69
92.77
Uniferm Distribution (default)
3.29
99.23
64.66
84.72
9.86
98.01
28.71
94.64
33.55
92.49
28.02
93.82
100
PCL
PALM
AUROC
60
FPR
40
Aveege
Plaxs366
Datases
Datasel
(a) FPR for different methods
(b) AUROC for different methods
Figure 9: OOD detection performance on methods trained on unlabeled CIFAR-100 as ID dataset using backbone network of ResNet-34. Smaller FPR values and larger AUROC values indicate superior results.
C.2 MORE EXPERIMENTS AND DISCUSSIONS ON UNSUPERVISED OOD DETECTION
Comparisons to previous prototypical contrastive learning approaches on unsupervised OOD/Outlier detection. In addition to the performance metrics presented in Table 2 conduct comparative study between our unsupervised extension and two powerful prototypical contrastive learning frameworks PCL (Li al. (02I) and SwAV (Caron et al. 2020) on OOD detection per- formance. All methods are trained on unlabeled CIFAR- 100 dataset using ResNet-34, identical to our default experiment settings. For fair comparison, we do not employ the multi-crop technique suggested in (Caron et al. following (Sehwag et al.
2020), we use default data augmentation techniques for all experiments
Ming et al.) (023) and our default settings.
Compared to PCL (Li et al.
2021), which generates prototype assignments using an offline clustering algorithm that may lead to training instability, our method PALM efficiently assigns prototypes stable online procedure. In comparison with SwAV (Caron et al. 2020). we opt to update our prototypes using EMA technique instead of gradient backpropagation, resulting in a more consistent assignment and aiding in the avoidance of sub-optimal solutions. While PCL (Li et al. 1021 and SWAV (Caron et al. 2020) provide similar performance terms of FPR and AUROC, PALM achieve average FPR compared PCL and outstanding average AUROC score of 88.07. The results show the effectiveness of the proposed techniques, rendering promising potential for unsupervised OOD detection.
29.35 % improvement
EMA technique also henefits unsupervised OOD detection. In Fig. 4()) we find that employing EMA technique for prototypes updating considerably benefits the proposed approach PALM for supervised OOD detection. We examine the efficacy of EMA technique for our unsupervised OOD detection extension in Fig. TO, We find that the employed EMA technique remarkably improves our unsupervised extension, achieving state-of-thh-ar OOD detection performance.
D MORE DISCUSSIONS ON LIMITATIONS
As discussed in the main paper (in Sec. 5), PALM faces notable limitation regarding the manual selection of the number of prototypes as a hyperparameter. And we simply assign the same number of prototypes to all classes of samples. Although experiments demonstrated in Fig. A(c)ssho that
22

---

Published as a conference paper at ICLR 2024
Leuranbee Protaypes
ISAA
50
AUROC
40
20-
Plaos00G
IBUN
Testurs
LSUN
SVIEX
Avtrage
Dataset
Figure 10: Ablation study the efficacy of EMA technique for unsupervised extension.
PALM can perform well with different numbers prototypes on the benchmark dataset, it still may limit the applicability of it in more complex scenarios, e.g., the datasets in which different classes require different numbers of prototypes. It may also influence the further extension on the unsupervised setting. In the future, we will consider tackling this limitation by relying parametric Bayesian approaches. Moreover, the potential of the proposed method can be further explored by conducting the learning of the probabilistic mixture model.
23


Reviews:
[{"review_id": "q52aQiGfd5", "reviewer": "Reviewer_qjVi", "summary": "This paper presents an outlier detection model, which assumes each class has multiple centroids in feature space, instead of one centroid that many existing models assume. A model called PALM is proposed that minimizes the MLE loss and prototype contrastive loss. The prototype centroid is updated during the model training. Experiment results demonstrate that the proposed model outperforms baselines in OOD and unsupervised OOD tasks.", "questions": "Please refer to the above section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, the novelty of this work is clearly presented, understandable and consistent with the intuition. Experiment comparison is comprehensive.", "weaknesses": "There have been some OOD detection benchmark datasets, such as Openood: Benchmarking generalized out-of-distribution detection. Advances in Neural Information Processing Systems 2022, 35, 32598-32611. Most of the datasets used in experiments are based on standard benchmark datasets. How are these datasets, such as CIFAR are used for OOD in this work?\n\nOne of the reasons that the proposed model outperforms the compared baselines is it better estimates the class or sample distribution due to the multi-centroid assumption. How about the comparison with generative based models, such as GAN based model?\nReference\nOut-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 714-718).\n\nSome technical details are not clearly described. For example, how to have diag(u) and diag(v) in Eqn. (6). Appendix C.1 is not about the detail of this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2jfJ0eAQsc", "reviewer": "Reviewer_wPGj", "summary": "The paper introduces the PrototypicAl Learning with a Mixture of prototypes (PALM) for Out-of-distribution (OOD) detection in machine learning. Unlike traditional methods that oversimplify data assumptions, PALM models multiple prototypes per class for accurate representations. Experimental results highlight PALM's effectiveness, especially on the CIFAR-100 benchmark.", "questions": "The experimental results from tables 1, 2, and 3 all indicate that PALM does not achieve optimal performance on the texture dataset as it does in other tasks. I am quite curious as to why this phenomenon occurs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The paper provides extensive experimental settings, particularly in the ablation studies.\n-The proposed method introduces an automatic prototype learning framework that incorporates a mixture of prototypes to represent hyperspherical embeddings, effectively capturing the natural diversities within each class.\n-The proposed method achieves a significantly improved performance.", "weaknesses": "-I have some concerns about scalability. The introduction of multiple prototypes and their dynamic updating could lead to scalability issues, especially when handling very large datasets or a vast number of classes.\n-The effectiveness of PALM is highly dependent on the quality of the prototypes. If the prototypes do not accurately represent the underlying data distribution, the model may face challenges in OOD detection.\n-In terms of computational cost, PALM might demand additional computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxlJAP3lNX", "reviewer": "Reviewer_dG1q", "summary": "This paper proposes PALM (PrototypicAl Learning with a Mixture of Gaussian) to address the out-of-distribution detection problem via a distance-based method underlied by a mixture of prototypes. This work is a direct extension of the compactness and dispersion regularized loss (CIDER) by Ming et al. to a probabilistic setting. Although CIDER, and other distance-based OOD detection methods have shown strong results against OOD benchmarks, they indeed tend to make stringent assumptions in their formulation . For instance, CIDER models all samples of each class with only one single prototype, and requires data samples of each class to be compact around a single prototype. These oversimplifying assumptions make the methods quite limiting. This paper addresses some of those limitations to formulate and shape the embedding space.", "questions": "It'd be great if the authors could address the limitations that I raised above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and mathematically sound. The method PALM that is proposed nicely extends an already strong OOD detection method called CIDER. PALM is extensively analyzed via a thorough ablation study, and extensively evaluated against multiple OOD benchmark datasets and methods. PALM boats strong ID-OOD discrimination in almost all of the experiments by outperforming previous supervised and unsupervised methods by a large margin.", "weaknesses": "1. PALM, like its predecessor CIDER, heavily relies on the hyperspherical representation of the learned embeddings to formulate and shape the embedding space. In CIDER, this representation was crucial in achieving strong ID-OOD separability and ID classification. However, since PALM is a mixture of Gaussian, that assumption that the embeddings need to be normalized to unit-norm or need to lie in a hyper spherical space may not necessarily be needed. I wonder if this assumption could be lifted so as to make PALM more general.\n\n2. The second limitation pertains to how the prototypes are updated. While in small data regimes, one can expect the observations to behave similarly in nature, (i.e.; samples of the same class appear somewhat similar throughout the training process), in large data regimes, it's very likely to have samples of the same class behaving differently. In other words, even within the ID distribution, there could be outliers that could affect the ID classification. As a result, maintaining one value of $\\alpha$ throughout the training process for the prototypes update seem quite limited to me. One suggestion to the authors is to make $\\alpha$ adaptive by integrating its learning as part of the whole training process.\n\n3. It is unclear from the manuscript how the mixture proportions $\\omega_{i, k}$ are learned. As the authors may know, GMMs are very sensitive to the initialization of the mixture proportions. A suboptimal initialization may lead to the model getting stuck in a local minimum.\n\n4. It has sounded quite unintuitive to me why the likelihood of a sample would be expressed as a mixture of prototypes given the strong ID separability flavor the authors would want to endow to their ID detection method. This concern is somewhat assuaged by the assignment pruning, but I think more justification is needed as to why one would want to enforce separability and yet mix the prototypes.\n\n5. By having K dedicated prototypes for each class, PALM significantly increases the memory footprint unlike its predecessor CIDER and other distance-based OOD detection methods. This needs to be addressed in the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gPXK6TX8Tp", "reviewer": "Reviewer_ddCD", "summary": "This paper propose a new method that enhances OOD detection by using multiple prototypes per class to capture data diversities, achieving leading results on the CIFAR-100 benchmark.\n* PALM introduces a distance-based OOD detection using hyperspherical embedding space and a mixture of prototypes for superior differentiation.\n* The system automatically learns prototypes, employing both MLE and contrastive losses to enhance class distinction.\n* Empirical studies showcase PALM's effectiveness in both supervised and unsupervised OOD detection scenarios.", "questions": "Please see the content in Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The manuscript is articulately presented with clear and coherent language.\n* The core idea is presented in a lucid manner, ensuring ease of comprehension for readers.\n* The experiments provide compelling evidence for the effectiveness of the proposed approach, e.g., 10-point gain on Places365 (Table 1).", "weaknesses": "Major Points:\n\n1. **Hyperparameter Settings:** While it's commendable that ablation studies were performed for the introduced hyperparameters like $K$ (number of prototypes) and $\\alpha$ (momentum), how do we choose these hyperparameters for different models or datasets? Is there a potential performance dip with varying settings?\n\n2. **ID Accuracy Impact:** What's the influence of PALM on ID classification accuracy?\n\n3. **Intra-class Heterogeneity:** Fig. 5(c) shows varying prototype effects across datasets. Is this due to different levels of intra-class variability? Do authors have any insights or experiments on why this happens? \n\n4. **Technical Contribution:** The paper's core idea, i.e., prototypical learning with a mixture of prototypes, seems like a straightforward and easy extension of CIDER. I acknowledge the improvements shown in experiments and welcome further discussion on this.\n\nMinor Points:\n\n1. Figure 5 is presented before Figure 4.\n\n2. Figures 4 & 6 are too small to recognize when printed. Consider resizing for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)