Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection
Authors: Yingwen Wu
Ruiji Yu
Xinwen Cheng
Zhengbao He
Xiaolin Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Out-of-Distribution Detection
Feature Separation
Neural Collapse
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In the open world, detecting out-of-distribution (OOD) data, whose labels are disjoint with those of in-distribution (ID) samples, is important for reliable deep neural networks (DNNs). To achieve better detection performance, one type of approach proposes to fine-tune the model with auxiliary OOD datasets to amplify the difference between ID and OOD data through a separation loss defined on model outputs. However, none of these studies consider enlarging the feature disparity, which should be more effective compared to outputs. The main difficulty lies in the diversity of OOD samples, which makes it hard to describe their feature distribution, let alone design losses to separate them from ID features. In this paper, we neatly fence off the problem based on an aggregation property of ID features named Neural Collapse (NC). NC means that the penultimate features of ID samples within a class are nearly identical to the last layer weight of the corresponding class. Based on this property, we propose a simple but effective loss called Separation Loss, which binds the features of OOD data in a subspace orthogonal to the principal subspace of ID features formed by NC. In this way, the features of ID and OOD samples are separated by different dimensions. By optimizing the feature separation loss rather than purely enlarging output differences, our detection achieves SOTA performance on CIFAR10, CIFAR100 and ImageNet benchmarks without any additional data augmentation or sampling, demonstrating the importance of feature separation in OOD detection. Code is available
at https://github.com/Wuyingwen/Pursuing-Feature-Separation-for-OOD-Detection.

Published as a conference paper at ICLR 2025
PURSUING FEATURE SEPARATION BASED ON NEURAL COLLAPSE FOR OUT-OF-DISTRIBUTT
DETECTION
Yingwen Wu, Ruiji Yu, Xinwen Cheng, Zhengbao He, Xiaolin Huang"
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University (yingwen_ wu, yj1938, xinwencheng, Istefanie, xiaolinhuang) @sjtu.edu.cn
ABSTRACT
In the open world, detecting out-of-distribution (OOD) data, whose labels are disjoint with those of in-distribution (ID) samples, is important for reliable deep neural networks (DNNs). To achieve better detection performance, one type of approach proposes to fine-tune the model with auxiliary OOD datasets to amplify the difference between II) and OOD data through a separation loss defined on model outputs. However, none of these studies consider enlarging the feature disparity, which should be more effective compared to outputs. The main difficulty lies in the diversity of OOD samples, which makes it hard to describe their feature distribution, let alone design losses to separate them from ID features. In this paper, we neatly fence off the problem based on aggregation property of ID features named Neural Collapse (NC). NC means that the penultimate features of ID) samples within a class are nearly identical to the last layer weight of the corresponding class. Based on this property, we propose simple but effective loss called Separation Loss, which binds the features of OOD data in subspace orthogonal the principal subspace of ID features formed by NC. In this way. the features of ID and OOD samples are separated by different dimensions. By optimizing the feature separation loss rather than purely enlarging output differences, our detection achieves SOTA performance on CIFAR10, CIFAR 100 and ImageNet benchmarks without any additional data augmentation sampling, demonstrating the importance of feature separation in OOD detection. Code is available at https://github..cm www.//wwngwww///ussuuiiiii Soattr-SSppraatoofoo---ooo
INTRODUCTION
In the open world, deep neural networks (DNNs) encounter diverse range of input images, including in-distribution (ID) data that shares the same distribution as the training data, and out-of-distribttion (OOD) data, which has labels that are disjoint from those of the ID cases. Facing the complex input environment, reliable network system must not only provide accurate predictions for ID data but also recognize unseen OOD data. This necessity gives rise to the critical problem of OOD detection (Cao et al. 2007 Liu et al. 202T). which has garnered significant attention in recent years, particularly in safety- applications.
rich line of studies detect OOD samples by exploring the differences between ID and OOD data in terms of model outputs (Hendrycks & Gimpel 2016 Liu al.) (202). features (Sun et al. 202 1 Zhu ct al. 2022b Sun et al. 2022b), or gradients (Huang et al.] [222 Wu et al. 2023). However, it has been observed that models trained solely on ID data
make over-confident predictions on OOD
data, and the features of OOD data intermingle with those of ID features (Hendrycks & Gimpel 2016
Sun et al. 2022b). To develop more effective detection algorithms, category of works focuse the utilization of auxiliary OOD datasets, which significantly improves detection performance on unseen OOD data. One classical method, called Outlier Exposure (OF, (Hendrycks al. (2018)). employs cross-entropy loss between the outputs of OOD data and uniformly distributed labels to fine-tune the model. Additionally, Energy method (Liu et al. 2020) proposes using the energy function as its training loss and designs an energy gap between ID and OOD data. Building on these proposed losses, recent works have concentrated on improving the quality of auxiliary OOD datasets through
"Corresponding author

---

Published as a conference paper at ICLR 2025
Pasture Represestation
Visualization of Festires
ID Dats
W2
Backbone
Original feature.
Our mathod
CNN
*
W1
0O0 Data
Bacikon
Feature of ID data of class
* Pesture OOD data
Subapace sparned by Wr
where OOD Feetures locale
Festure data
Weight layer
Figure 1: Overview of our method. An example of a well-trained binary classification network, where wa denotes the i-th weight of the last fully connected layer. The features of ID samples within a class are nearly identical to the weight of the corresponding class, which is called Neural Collapse. Based on this property, we propose to constrain OOD features on dimensions orthogonal to FC weight subspace to explicitly separate the feature manifolds between ID) and OOD data.
data augmentation Wang et al. 2024
2024 or data sampling (Ming al. 2022a 2023 Zheng et al.
Chen et al.) 2021: Diang et al. 2023) to achieve better detection performance.
Existing losses designed for auxiliary OOD data primarily focus on increasing the output discrepancy between ID) and OOD samples (Hendrycks et al. 2018] Liu et al. 2020). However, NONE of these approaches consider enhancing the separability in the feature space. Insights from knowledge distillation (Gou et al.) (02I) and contrastive learning (Le Khac et al.) (220 have demonstrated that optimizing compactness or dispersion in the feature space is equally or even more important than enforcing similar constraints in the output space. Furthermore, previous detection score function design has shown the importance of employing feature information (Sun et al. [021] 2022b Zhu et al. 2022b), which can greatly improve detection performance. Therefore, when tackling the fine-tuning problem using auxiliary OOD data, we propose that it is crucial to separate the features between ID and OOD data, rather than merely enlarging their output differences.
Designing an effective feature separation loss for ID and OOD data is inherently challenging due to the diversity of OOD samples that belong various categories. This diversity results in a dispersion of their features and difficulty in describing their feature distribution. Consequently, common feature separation losses, such as maximizing the distance between the average features of different classes (Ming et al. (2022b) or increasing the Kullback-Leibler divergence between ID and OOD distributions (Kullback) 1997). ) are not suitable in our cases. Despite the intricate distribution of OOD features posing significant obstacle, in this paper, we derive solutions from the properties of ID features.
recent observation named Neural Collapse (Papyan et al. 2020) gives us an inspiration, which reveals that the penultimate features of ID samples within class are nearly identical to the last fully connected (FC) layer's weights of the corresponding class. Conversely, the features of OOD samples scattered haphazardly throughout the feature space. A direct illustration! can be seen in Figure [ Leveraging the property of ID features, we propose to constrain the features of OOD data on dimensions orthogonal to the subspace (denoted as W) spanned by FC weights. The dimension of W equals to the number of ID categories, while the overall feature space dimension is significantly larger. Consequently, there are numerous redundant dimensions available for OOD features, indicating the fcasibility of our method. To pursue this orthogonality, we introduce a loss function named Separation Loss (ref. Eq. B). which calculates the absolute value of cosine similarity between OOD features and the weights of the final FC layer. By optimizing this simple yet effective loss to zero, we ensure that OOD features are distributed in entirely different dimensions from ID features, thereby enhancing their separability. Our approach utilizes the NC property of ID features, allowing us to avoid modeling OOD feature distributions while effectively segregating ID and OOD features. The overall method can be widely applied as stronger baseline compared to OE (Hendrycks et 2018), and seamlessly integrated with other approaches like ATOM et al.) 2021). POEM(Ming et al.) 2022a), etc (ang et al. 2024 2023) by replacing the OE loss with separation loss.
For binary classification, NC. indicates that the angle between uy and w2 should he 180". But in order to show the general case of higher dimensions, we depict angle of 90" in the figure.
2

---

Published as a conference paper at ICLR 2025
We conduct extensive experiments over representative OOD detection setups, achieving the SOTA performance without any data augmentation or sampling algorithms (Ming et al. 2022a| Wang et al. 2023) on CIFARIO (Krizhevsky et al.) 2009b), CIFAR 100 (Krizhevsky et al.] 2009b), and ImageNet (Deng et al., 2009a benchmarks. For example, on the CIFARTOD benchmark, by using our feature separation loss, achieve the average FPR95 of 29.55 29.58% and AUROC of 94.01 %, outperforming the traditional OE (Hendrycks et al. 2018) method by 8.19% on FPR95. The contribution of our paper summarized as follows:
We are the first propose the concept of feature separation when using auxiliary OOD data to fine-tune models, while previous works pay more attention to the output separation, providing new insights into the design of OOD data loss functions.
To overcome the difficulty caused by OOD data diversity, we propose a feature separation loss based on the neural collapse property of ID features, which constrains OOD features to lie in dimensions where ID features are scarcely distributed.
Our SOTA detection performance on representative OOD detection settings verify the effectiveness of our feature separation loss, implying that our loss can be stronger baseline for future researches.
2 RELATED WORK
Post-hoc Detection. Given a model that is only trained by ID data, post-hoe detection approaches design score functions based on it to distinguish ID and OOD data. One type method named density-
based (Lee et al. 2018 Kobyzev et al. 2020 Zisselman & Tamar 2020) Kingma & Dhariwal 20018] Jiang et al. 2021 Choi et al. 2018) to exphcitly model the ID data with some probabilistic models and flag test data in low-density regions as OOD samples. More popular approaches are to derive
confidence score based model outputs (Hendrycks & Gimpell [ et al.. [017]] Liu et al.
2020), features (Sun et al. [
| Zhu et al.) 2022b] [Snn et al. [002b) Lee et al., 20188] Ndiour et al.
2020
Cook et al.] 2020 Ndiour et al. 2200 Cook et al. [020 Wang et al. [2022]
gradients (Huang
et al. 2021 Wu et al.) 2023 Lee et al. [2023] Lust & Condurache| (0200[Sun et al. 2022a Igoe et al.
2022
based method KNN Sun et al. al. (2022b) employs the Euclidean distance to the k-th nearest neighborhood of training data as a measurement to detect OOD data. Different from KNN which designs detection metric based on fixed feature representations, our approach explicitly enlarges the feature distance between ID and OOD data through optimization.
For example, the recent
Contrastive Learning based Detection. Different from post-hoe methods based on vanilla-trained models, such methods generally apply contrastive losses defined on ID data in the model training process to obtain better feature representations for OOD detection. For example, KNN+ (Sun et al., 2022b) utilizes the SupCon loss (Khosla et al. 2020), which encourages alignment of features within class and dispersion of features of different classes, to train a network to obtain greater diffcrentiation between ID and OOD samples. Besides, CSI (Tack al. 2020) contrasts original samples with their distributionalll-shitfee augmentations to improve detection performance. Recent advancements, such as CIDER (Ming et al., 20226), combine a compactness loss cluster samples near their class prototypes and a dispersion loss to maximize angular distances between different class prototypes, providing more direct and clearer geometric interpretation for the disparity between ID and OOD samples.
Auxiliary OOD Data based Detection. With access to part of OOD data, previous works design training algorithms to utilize auxiliary OOD data for OOD detection. One type method is to propose unsupervised training loss functions (Hendrycks et al. 2018 Liu al. 2020: Bai et af. 2023). such as the (Hendrycks et al. 2018), to fine-tune the model. Based on the proposed losses, another type is to select OOD data close to the decision boundary (Ming et al.] 2022a) or conduct data augmentation through adversarial attack (Chen et al.] 2021 Wang et al. (024) and model perturbations (Wang et al. 2023) in the training process, which can tight the boundary so that pushing unseen OOD data far away from it. In general, using auxiliary OOD data in the training process can significantly improve detection performance, achieving better results compared with other detection approaches.
divergence between OOD output probability and uniformly distributed label
3

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 PRELIMINARY
OOD Detection Problem. The framework for OOD detection is outlined as follows. We consider a classification problem involving C' classes, where X' represents the input space and y denotes the label space. The joint data distribution over X' x y referred to as Dxy. Let fo : X + y be a model trained on samples drawn independently and identically distributed (i.i.d.) from Dxy with parameters Then, the distribution of ID data is the marginal distribution of Dxy over X, denoted as Din- Conversely, the distribution of OOD data is represented as Dmut. whose label set does not intersect with y. The primary objective of OOD detection is to determine whether a test input x originates from Dan Dott. Typically, this decision is made using score function S as follows:
ID
if Stx f) 2 A if S f) A
Gx
OOD
where À is a threshold. Samples with scores higher than À are classified as ID data. The threshold is usually set based on ID data to ensure that a high fraction of ID data (e.g.. 95%) correctly identified as ID samples.
Finetune Model with Auxiliary OOD Data. In this paper, we consider the task of using auxiliary OOD data to fine-tune the model (Hendrycks et al., 2018 Liu et al.) 2020; Ming et al. 2202aa Wang cct al. 2024). which can effectively enlarge the discrepancy between ID and unseen OOD data. Let's denote the auxiliary OOD dataset as D which is a subset of real OOD datasets but has different distributions from the test OOD datasets in the experiments for fair comparison. One classical method is the Outlier Exposure (OE, (Hendrycks al. (0188), which designs outlier exposure loss that calculates the cross-entropy function between OOD outputs and uniformly distributed labels. The equation is follows:
Log (x) === Σ log fz(),
-
1
(1)
where fz) denotes the j-th element of the model output f(z). The final training objective of OE is to simultaneously minimize cross-entropy loss on IID data and outlier exposure loss on OOD data, which can be formalized as:
min (.. LOE(E, v) + XB out (or(z)
where À is apper-parameter. This optimization problem is regarded as a basic setting in auxiliary OOD data approaches. Most of subsequent methods adopt the same or similar loss functions that encourage ID) and OOD data to differ in the output space. For example, POEM (Ming et al., (202aa) designs a data sampling algorithm for efficient training, and DAL (Wang et al. 20224 employs adversarial features to calculate the OE loss to minimize the generalization gap between auxiliary and real unseen OOD data.
3.2 MOTIVATION
Previous works have focused on increasing the discrepancy between ID and OOD data in the output space, while in this paper, we propose to explicitly enlarge the disparity of their features. Intuitively, separating features of ID and OOD data should be beneficial to OOD detection compared to solely augmenting the output differences. Existing feature separation functions in other fields, such as the dispersion loss that enlarges the distance between the average features of different classes (Ming et al. 20226; Khosla et al. [2020), are not suitable for diverse OOD data since their features are dispersed instead of clustering around the mean. To design separation loss that can handle the complicated distribution of OOD features, we delve into the property of ID) features. A recent observation named NC (Papyan et al.) 2020) gives us a new insight, which reveals that the penultimate features of ID samples within class are nearly identical to the last layer weight of the corresponding class. This intriguing property has stimulated many fields of research, including low-dimensional characteristics of ID features (Garrod & Keating 2024 Rangamani et al. 2023) and model generalization analysis (Kothapalli 2022; Hui et 2022). Particularly, several works employ the principal component spaces identified by NC design detection score functions (Liu & Qin 2023 Zhang et al.! 2024b
4

---

Published as a conference paper at ICLR 2025
Den
(a) Vanilla model
(b) OE- trained model
(c) Our model
Cies-3 Clooo2 outier
Ousika
cles outeer
10.0
5.0
0.0
0
2
5.0
7.5
(d) Vanilla model
(e) OF-trained model
(f) Our model
Figure 2: Visualization of features projected into the two-dimensional space consisted of wi and w2 (ref. Figure 2) and the biree-dimensional space consisted of W1, u2 and the principal eigenvector of OOD features on CIFARIO benchmark. The Class-1 and Class-2 represent features of test ID samples of class-l and class-2, and the Outlier means features of test unseen OOD data, i.e. SVHN. It can be observed that the feature separability between ID and OOD data gradually increases from left (Vanilla model) to right (Our model).
Haas et al. 2022 Ammar et al., 2023), which demonstrates the large potential of NC applied in OOD detection. We conduct empirical experiments on CIFAR 10 to validate the NC property, as Figure 2(a) shows, where we plot the feature distribution of test ID samples under weight basis space (For accurate visualization, select two class samples in CIFARIO since the principal component dimension their features equals to the number of their classes, f.e. two, according to NC property. More visualization results of samples of other classes be seen in Appendix (A.9). Furthermore, with the comparison of OOD features vanilla and OE-trained model (Figure ((a) vs. Higure2(bb),
discover that although the outlier exposure loss only optimizes the output of training OOD samples, it implicitly changes the distribution of test unseen OOD features, making them more clustered and far way from test ID features. However, from the 3D visualization of features in Figure 2(c) it can be observed that the features of test unseen OOD data almost lie in the same subspace as ID features, without taking advantage of the new dimension (z-axis) further widen the ID-OOD difference. Based on the above observations, we then design feature separation loss without modeling OOD feature distributions follows.
3.3 FEATURE SEPARATION LOSS
Our key idea is confine the features of OOD data dimensions where ID features are sparsely distributed. Considering that the principal subspace of ID features C-dimensional, as determined by the NC property, while the overall feature space has a significantly larger dimensionality, there exist ample redundant dimensions that can accommodate OOD features. In pursuit of our goal, a straightforward condition arises: 2" w, 0, = 1, 2, ...: C, where z denotes the normalized feature of OOD data, and denotes the normalized fully connected layer weight for class i. According this condition, we devise Separation Loss for OOD data, which computes the average absolute value of the cosine similarity between and u.. The specific equation is as follows:
C
Lsep
Σ
1
T
wi
C
Through minimizing the Lsep loss, the OOD feature 2 tends to be distributed in the dimensions that are orthogonal to w,, = 2, .... C. Figures 2(c) and 2( illustrate the features of our model fine-tuned using Sep- As observed, the features of outlier samples are indeed distributed different dimensions from wi resulting in larger discrepancy between ID and OOD features. Except for
5

---

Published as a conference paper at ICLR 2025
the Lsep loss, also propose an assistant loss that encourages ID features (denote 2ID) within class to align closely with the FC weight of their corresponding class (denote as wy). We term this loss function as Lein it promotes neural collapse phenomenon (Papyan et al. 2020 enabling ID
features within a class more clustered. The formulation as follows:
Lelu = -
Our empirical experiment in Sec 4.4 indicates that adding Lcl in the training loss can further improve the detection performance. Combining the above two losses, the final optimization problem can be formulated as:
min (e. (LCE + a<LCca) Enw (ALOR B /
(5)
where a, X and / are previous works (Hendrycks et al. 2018) and set 1.0 and 1.0 for simplicity.
In our experiments, we use the common setting À = 0.5 in
3.4 OVERALL FRAMEWORK
Train. Based on the optimization problem outlined in Eq. 5 follows. Our training process consisted of two stage: Stage I-training with cross-entropy loss to ensure the occurrence of NC phenomenon; Stage II-training with all losses including cross-entropy. outlier exposure, and our proposed separation and clustering losses to enlarge the discrepancy between ID and OOD data. Notably, when well-trained model is used the initial parameter, Stage I is not necessary and thus can be omitted. More detailed strategy choices and their impact on our performance can be seen in Appendix A.5
we detail the training procedure as
Test. After fine-tuning the model with our loss, we propose a new score function to detect OOD samples. Since our method simultaneously optimizes the outputs and features of OOD data, more proper score function is the sum of the traditional MSP (Hendrycks Gimpel| (016) and the average cosine similarity between features and wi. Mathematically, it can be expressed as:
S(z, f) max
1 ΣS e C
el
+
Σ1
T
(6)
where y denotes the model output f (Ix). Our experiments in Appendix A.8 compare the performance of using our score function with only using the MSP score. The result indicates that our method also performs well under MSP score, but slightly better under our proposed score function.
4 EXPERIMENTS
In this section, we first conduct experiments on CIFARIO, CIFARIO0 and ImageNet benchmarks to validate the superiority of our method in 1 Then, we consider variety of model architectures to further verify the effectiveness of our method in S.c Subsequently, we study the hyper-parametts sensitivity of our method in Sec 4..3 and explore the contribution of each loss part in Sec 4.4 In the last part, we discuss the loss design based on different distance metrics, numerical degree of feature separation and requirement on feature dimension in Scc In our Appendix, we report more experimental results, including combining our loss with Energy-OE (Liu et al.) (0200) in Appendix A.I detailed results on different networks in Appendix A.2 performance on hard OOD detection
settings proposed in CSI (Tack et al., 2020) in Appendix [A.3] generalization to class-imbar imbalanced
datasets in Appendix A..4] training strategy choice and impact in Appendix [A.3 feature distance
anchoring on class mean in Appendix A.6 fluctuation of performance in Appendix A.7 influence of
different score functions Appendix A.8. and more visualization results Appendix A.9 To begin
with, introduce experiment setups follows.
OOD Datasets. For CIFAR benchmarks, we randomly choose 300K samples from the 80 Million Tiny Images (Toralba et al. 2008) our auxiliary OOD dataset. And we adopt five routinely used datasets as the test OOD datasets, including SVHN (Netzer et al. 2011), LSUN Yu et al. 2013). iSUN Xu al.) 2015), Texture (Cimpoi al.) 2014) and Places365 (Zhou et al.) 2017). which have non-overlapping categories w.z. CIFAR datasets. For ImageNet benchmark, we use a validation subset of ImageNet-21ky, dataset auxiliary OOD dataset. And we adopt four commonly-used

---

Published as a conference paper at ICLR 2025
OOD datasets for evaluation, including iNaturalist (Van Horn et al. 2018), SUN (Xiao et al. 2010). Places (Zhou et al. 2017) and Textures (Cimpoi et al. 2014).
Pre-training Setups. For CIFAR benchmarks, employ Wide ResNet-40-2 Zagoruyko & Ko- moddkis 2016) trained for 200 epochs, with batch size 128, init learning rate 0.1, momentum 0.9, weight decay 0.0005, and cosine schedule. For ImageNet benchmarks, we directly use the pre-trained ResNet50 (He et al. 2016) model in Pytorch as the baseline network.
Fine-tuning Setups. For both CIFARIO and CIFARI00 benchmarks, we adopt the model parameter of the 99th epoch in the pre-training process as our initial network parameters, and then add auxiliary OOD data train the model for 50 epochs with IID batch size 128, OOD batch size 256, initial learning rate 0.07, momentum 0.9, weight decay 0.0005 and cosine schedule. This setting is aligned with experiments in DAL (Wang et al., 2024). Since the initial model is not sufficiently converged. we add our proposed Lsep and Lctu into the training loss after 25th epoch of the whole fine-tuning stage. For ImageNet benchmark, we use the pre-trained model in Pytorch as initial network, and then fine-tune the model for 5 epochs with ID/OOD batch size 64, initial learning rate 1e-4, momentum 0.9, weight decay 0.0005 and cosine schedule.
Compared Methods. We compare our method with post-hoc approaches, contrastive learning based methods, and auxiliary OOD data based methods. The post-hoc methods include MSP (Hendrycks Gimpel 2016). Energy (Liu et al. 2020), Maha (Lee al. 2018). and KNN (Sun et al. (2022b). The contrastive learning based methods include CSI (Tack et al. 2020), CIDER (Ming et al. 20222b) and
KNN+ (Sun al. 2022b). The auxiliary OOD data based methods include OE (Hendrycks et al., 2018), Energy-OE (Liu et al. 2020). POEM (Ming et al. 2022a),, and DAL (Wang et al. 2024). For OE and Energy-OE, we adopt the same training setting as ours, since we have discovered that their recommended setting in the original paper performs much worse than our setting. For other methods, we adopt their suggested setups but unify the backbones for fairness.
Evaluation Metrics. We report two classical metrics in this paper: 1) FPR95: the false positive rate of OOD samples when the true positive rate of ID samples is at 95%. 2) AUROC: the area under the receiver operating characteristic curve. A lower FPR95 and higher AUROC indicate better detection performance.
4.1 MAIN RESULTS
The main results are shown in Table and Table 2 where we report the FPR95 and AUROC across the considered real OOD datasets P
Compared to methods based on vanilla or contrastive learning models, whose training datasets only contain ID samples, incorporating auxiliary OOD data into the training process significantly reduce the FPR95 and improve the AUROC, indicating that this direction is valuable to explore. Compared to the classical OE approach (Hendrycks et al.. 2018). our method reduces the average FPR95 by 0.87% on CIFARIO 10, 8.30% on CIFAR 100, and 2.93% on ImageNet, just by adding our Separation and Cluster losses into the training procedure. This result demonstrates the effectiveness of our proposed losses. In addition to the OE approach, we also compare our method with other advanced works, including the classical work that studies data sampling strategies (POEM, (Ming et al. (022a)) and the adversarial feature augmentation work that aims to mitigate the impact of OOD distribution discrepancy (DAL, (Wang et al.) (024)). It worth noticing that our method does not employ any data augmentation or selection algorithms, while exhibiting superior performances on CIFAR and ImageNet benchmarks. On CIFAR 100, our method outperforms the best baseline DAL by 2.00% on FPR95 and 1.18% on AUROC. Notably, the performance of DAL method the Places dataset of ImageNet benchmark is vastly different from their reported result in the original DAL paper. The reason stems from our use of a curated subset (Huang & Li 202T) from Places365, which contains some Near-OOD classes semantically similar to ImageNet categories (e.g., hayfield vs. hay, cornfield vs. corn). In contrast, the DAL paper likely evaluated on randomly sampled Far-OOD data from the complete 10 million size Places365 dataset. Based on the outstanding performance of our method, we suggest that the feature separation loss, which is simple yet effective, can be used as basic training function like OE loss (Hendrycks et al. 2018) in the further works.
The symbol in the table means the results are cited from DAL Wang al. 2024
7

---

Published as a conference paper at ICLR 2025
Table 1: Results on ImageNet- 1k benchmark with auxiliary OOD data. The best result is in bold.
Far-ODD Dataseds
Near-OOD Datasets
Model
Method
INutcralist
Textures
SUN
Places
Average
ID Acc!
TPR95]
AUROCT
FFR995
AUROC
FPR055
AUROC
[1995]
AUROC
AUROC1
Of Handeyckess et
2018
48
$8.91
58.50
52.78
61.40
K3.09
70.
50.78
92.65
K3.89
76.04
RoiNe150
Dive waang Gratn
47.92
89.12
57.91
53,00
61.20
53.23
70.55
50.79
52.39
54.04
75.94
Oure
43.01
90.17
53.35
83:45
60.11
83.56
63.45
$1.31
56.73
$4.62
76,10
Of Haudrycke al DXI
2018
41.96
00:49
52.25
55.97
65.61
$2.30
70.20
50.93
57.51
$4.92
50.05
VI-B-16
40.52
90:92
50.94
56.30
65.07
82.
70.17
80.96
56.67
85 12
50.
OfFe
40,10
91.06
51.70
56.13
63.58
82.25
70.12
$1.07
56.88
85.13
80.29
Table 2: Results on CIFARIO and CIFAR100 benchmarks with WideResNeet40-22 model. The best result bold.
Far-OOD Dntanets
Method
SVIE
LSUN
ISUN
Testures
Pluco365
Averuue
FPR951
AUROCT
FPR95)
AURUC
FPRMA
AUROCT
EPR9
AUROCT
FPRSS.
AUROCC
IPRI5]
AUROCT
CIEAR
With vanill rraming
Hlndryrk Cinred
32.55
58.
90.82
44.76
27.38
91.97
Tick
17.37
6.75
98.45
12.55
577 95.95
94.70
40.00
92.05
96,17
CIUER
6.26
98-25
2h
U5
43.20
91.94
16
KNN
3.28
99.33
2.24
98.90
17.63
97.63
10.57
97.92
30.63
94.98
12.97
97.32
Winh auciliary 0OD dota
UI Hendrycs el
701
1.40
99.54
0.85
99.64
2.20
99. 26
2.80
99.26
9.55
97.99
5.35
99.62
0.75
99.50
0.90
95.95
1.50
99. 22
2.75
98.92
9.05
97.33
2.99
99.79
POEX
25.66
95.43
94.97
76.44
1.58
99. 04.64
20.62
95.73
53.39
85.38
39.24
91.10
DA
0.75
99.28
0.75
99.62
0.70
99. 33
2.35
98.99
8.90
97. 10
2.69
98.56
65
98.93
With vanilla trainin
MS Hendyys das Ginee 2016
74.79
79.64
54.72
86.45
93.85
$6.02
88.76
68.48
83.24
71.95
79.07
72.55
70.18
87.15
17.15
97.05
91.37
65. 50
$4.77
76.72
78.91
75.77
62.75
50:44
Mah
77.73
73.00
98.46
63.44
47.74
85, 76
54.93
$2.53
97.22
54.11
75.22
73.37
71.86
63.31
78.89
70.09
79.69
72.30
30.05
71.33
75:13
contralive
Toaning
77.11
KNN
ausliary OOD
0 Mandyocee di B
38.70
92.90
18.30
96.67
36.5 55
92. 50
43.05
91.00
52.45
87.85
37.27
92.31
Enc
13.75
96.04
34.00
94-82
65.75
83. 52
45.70
90.09
53.50
80.05
42.34
91.65
POEM
Vi Minrees
45.41
90.70
3,01
99.24
15,60
93.79
51.37
$3.85
84.13
73.93
40.5
68.57
DA
16:45
10
17.00
96.52
36,95
90.68
38.40
91.72
48.55
85.91
31.47
91.82
17.95
96.32
12.50
97.64
27.00
90.85
41.20
91.37
48.20
90.64
29.47
94.00
4.2 DIFFERENT ARCHITECTURES
To further verify the effectiveness of our method, we evaluate and compare performance with other approaches on more network architectures, including ResNer18 (He et al. 2016) and DenseNet121 (Huang al. 2017). The results are shown in Table 3 where our method exhibits consistently superior performance across various architectures on CIFAR 10 and CIFAR 100 benchmarks. For instance, we reduce the FPR95 by 4.55% compared to DAL (Wang al. 2024) with ResNet18 architecture on CIFAR100 benchmark. Detailed results can be seen in Appendix A.2
Table 3: Results on different network architectures on CIFARIO and CIFARI00 benchmarks. We
report the average FPR95/AUROC across five OOD datasets. The best result is in bold.
CIFAR-10 ResNet18
CIFAR-100 ResNet18
Method
WRN-10-2
DenseNet-121
WRN-40-2
DenscNet121
OFfendrycks et al.
2018
3.36/99.02
6.35/97.35
10.79/97.54
37.77/92.21
$6.96/90.19
62.08/86.77
DaL ot al.
2024
2.69/98.86 86
3.61/98.20
9.75/97.71
31.47/92.82
54.89/90.95
61.25/87 .66
Ours
2.49/98.93
3.52/98.75
8.90/97.74
29.47/94.00
50.34/90.90
59.13/88.45
4.3
SENSITIVITY
In this section, we study the influence of coefficient a and / in Eq. 5q.55mm on the detection performance. Specifically, we evaluate our method on CIFARIO benchmark with a E {0.11 0.5, 1.0, 2.0) and E {0.1, 0.5, 1.0, 2.0). Experiment results shown in Table 4 Notably, our approach is not sensitive to the choice of
Furthermore, we discover that using a = 0.1 and B = 0.1 can achieve better performance of our method than our previous report.
8

---

Published as a conference paper at ICLR 2025
Table 4: Influence of loss coefficient and B. We report the average HPR9S/AUROC across five OOD datasets on CIFAR1O benchmark. The best result is in bold, and the result for the parameter used in main experiment is underlined.
CIFAR1O
3 == 0.1
5 = 0.5
B == 1.0
8 3= 2.0
0.1
2.21/99.13
2.41/99.03
2.40/99.12
2.60/98.97
a
2,45/98.91
2.52/98.90
2.51/99.07
2.43/99.08
a 1.0
2,49/98.77
2.34/98.92
2,49/98.93
2.44/99.01
a 2.0
2.46/98.73
2.51/98.24
2.36/98.38
2.67/98.72
4.4 ABLATION STUDY
Considering our training objective loss contains four parts: LCE, LCln, LoE, and Lsep, we explore the contribution of each part to the final detection performance in this section. The cross-entropy loss LCF is used for ensuring ID accuracy, thus we skip when discussing the detection performance. The rest three parts, one (Lor) is for output discrepancy and the other two (Lei and Lsep) is for feature separation. We firstly evaluate the performance of purely using cross-entropy loss (LCF) and sepposure
outlier exposure loss (LoE), namely OE method (Hendrycks et al. 2018). And then discuss three situations: 1) adding Lci: 2) adding Lsep: 3) adding Lclu and Seep The results are shown in Table 5 Comparing the 1o.1 and No.3, it shows that feature separation loss significantly improves detection performance compared to OE method. Additionally, only using cluster loss damages the performance but integrating with separation loss achieve the best result. The underlying reason is that the cluster loss only controls the property of ID features, but has negligible effect on enlarging the discrepancy between ID and OOD features when purely using it.
Table 5: Performance of our method under different training losses. The cross-entropy loss is used by
default in all cases.
CIFARIO
CIFARI00
No.
Training Loss
FPR95
AUROC1
FPR95
AUROCT
Log
3.36
99.02
37.77
92.21
Log+ LCa
3.62
98.96
39.91
91.22
Logt Tisep
2.65
99.00
33.30
93.42
Logutcntt
2.49
98.93
29.47
94.00
4.5 DISCUSSION
Cosine Similarity vs. Euclidean Distance. Our designed loss is calculated based on cosine similarity, which then induces dimensionality separation between ID and OOD features. this part, we compare with intuitive loss design, that is, maximizing the Euclidean distance between OOD features and weights of the last FC layer, with our orthogonality-ben based loss Lsep to illustrate the importance of utilizing redundant dimensions enlarge feature discrepancy. Since maximizing the Fuclidean distance will cause its value to approach infinity, we instead use T=-w;l| for OOD features and
|| wyll for ID features the training loss in the Euclidean distance setting, and then minimize this loss to fine-tune the model. The comparison results are presented in Table where the cosine similarity loss significantly outperforms Fuclidean distance loss. The underlying
may be that our separation loss utilizes new dimensions to separate ID and OOD features. When faced with unseen OOD data, the feature variations tend fall on the new dimension, resulting in minimal changes on the output.
Table 6: Comparison between using the Euclidean distance and cosine similarity (ours) as the training loss to separate ID-OOD features.
Methed
FPRS5L
SVHN
AUROC1
FPRO5I
LSUN
AUROC
FPR95L
ISUN
AUROC!
FPR951
Textures
AUROC
FPR95L
Places365
AUROCT
FPR95,
Average
AUROCT
CIFAR-10
Fuclidean
Ours
0.40
99.28
0.60
99.58
1.60
99.25
2.45
98.83
7.40
97.00
2.49
98.93
CIFAR- 100
Euclidean
42.35 27.00
36.95
Ours
17.95
96.52
12.50
97.64
93.85
41.70
91.37
48.20
90.64
29.47
94.00

---

Published as a conference paper at ICLR 2025
Feature Separation Degree. To validate the effectiveness of our proposed loss, evaluate the feature separation degree between ID and OOD data. Leveraging the NC property of ID features, we design metrics based on the fully connected layer weights: (1) Euclidean distance between features and the predicted class weight, (2) cosine similarity between features and the predicted class weight, and (3) reconstruction exhibits significantly larger differences under these metrics compared to vanilla and OE-trained models, where OOD refers to unseen test OOD data. Additional experiments using class mean vectors for distance measurement are provided in Appendix A.6
the subspace spanned by FC weights. As shown in Table 7, our model
Table 7: Feature separation degree of different methods measured by three metrics. The higher difference (Diff) means better discrepancy between ID and OOD features.
Ruclidean Distance
Cosine Similarity
Reconsiruction Error
Method
0OD
Diff
00D
Difft
ID
0OD
Difft
Vanilla
0.80
1.12
0.32
0.69
0.47
0.22
0.19
0.32
0.13
OE
0.32
0.83
Ours
0.69
1,16
0.47
0.75
3c-3
0.75
0.43
0.86
0.43
Requirement Feature Dimension Our method leverages redundant dimensions in the feature space to enhance the discrepancy between ID and OOD features, requiring the feature dimension exceed the output dimension. This condition typically satisfied in modern CNNs, given the class counts of common datasets (ranging from tens to thousands). However, we also evaluate scenarios where feature dimensions are smaller than output dimensions. Experiments on CIFAR 100 using WideResNeet44-11 (feature dimension 64) demonstrate that, while less pronounced than in high- dimensional feature cases, our loss still marginally outperforms the basic OE loss (see Table 8). Future work could explore applying our loss intermediate layers, where NC phenomenon also occurs Rangamani et al. (203)) : Parker et al. (2023), ) or inserting additional linear layers to increase feature dimensions.
Table 8: Results on WideResNeet40-1 model (feature dimension is 64) on CIFAR100 benchmark. We report FPR95/AUROC.
Method
FPRUS
SVIIN
AUROC
FPRU5
LSUN
AUROCT
FPRIS)
ISUN
AUROC!
FPRU5.
Testures
AUROC1
FPRUSL
Places165
AUROC!
FPRU5.
Average
AUROCT
OB Heeddyyks et 2018
40,60
92.19
20.30
96.13
55.70
86.81
55.90
85.44
59.55
$4.65
46.41
$9.04
Olzss
57.05
92.27
20.30
96.08
56.63
K5.17
45.00
87.37
59.30
K4.54
44.26
89,19
LIMITATION
The main limitation lies on two aspects: one is the requirement feature dimension, which we have discussed; the other is our dependency on NC property. Although for modern CNNs and language models, NC is relatively common phenomenon Zhu al. (2024); Dang et al. (003) that happens when networks convergence on training dataset. But there will still be cases where the NC property may fail Hui al. (2022), like imbalanced data distribution (we discussed this case in Appendix A.4). Our method essentially utilize the low-dimensional property of ID features, which induces from Neural Collapse. Therefore, when NC fails, is important to verify whether the low dimension of the feature exists and how to determine the low dimensional subspace, which is an important and interesting direction for our future work.
6 CONCLUSION
In this paper, we propose a novel training loss to enhance feature discrepancy between ID and OOD data during model fine-tuning with auxiliary OOD datasets. Leveraging the Neural Collapse property of ID features- penultimate features of ID samples converge to their class weights- introduce a separation loss that constrains OOD features to dimensions orthogonal to the principal subspace of ID features formed by NC. Extensive experiments demonstrate saute-of-the-ar perfor- mance on CIFAR-10 10, CIFAR- 100, and ImageNet benchmarks. Our work provides a foundation for further research on feature separation in OOD detection using auxiliary OOD data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their insightful comments.
The research leading to these results has received funding from National Key Research Development- Project (2023YFF1 104202), National Natural Science Foundation of China (62376155), Shanghai Municipal Science and Technology Research Program Major Project (2021SHZDZX0102),
REFERENCES
Mouin Ben Ammar, Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi. Neco: Neural collapse based out-o--distribution detection. arXiv preprint arXiv:2310.0683 06823, 2023.
Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both -iito--dittiittionn generalization and detection. In International Conference on Machine Learning, pp. 1454-1455 PMLR, 2023.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with tabel-distribution-awaa margin loss. Advances in neural information processing systems, 32, 2019.
Cao, CW Huang, DYT Hui, and JP Cohen. benchmark of medical out of distribution detection. arxiv 2020. arXiv preprint ftf::2007.042500 2007.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pp. 430-445. Springer, 2021.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Matthew Cook, Alina Zare, and Paul Gader. Outlier detection through null space analysis of neural networks. arXiv preprint urXiv:2007.01265 2020.
Hien Dang, Tho Tran, Stanley Osher, Hung Tran-The, Nhat Ho, and Tan Nguyen. Neural collapse in deep linear networks: from balanced imbalanced data. In Proceedings of the 40th International Conference on Machine Learning, pp. 6873-6947, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009a.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009b.
Connall Garrod and Jonathan P Keating, Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model. arXiv preprint urXiv:2404.061066 2024.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A
survey, International Journal of Computer Vision, 12966):189--1199 2021.
Jarrod Haas, William Yolland, and Bernhard Rabus. Linking neural collapse and 12 normalization with improved -ittof-distribution detection deep neural networks. arXiv preprint arXiv:2209.08378 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02116,, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, arXiv preprint arXiv:18122 04606, 2018.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Rui Huang and Yixuan Li. Mos: Towards scaling cut-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8710-8719, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances Neural Information Processing Systems, 34:677-689, 2021.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprini arXiv:2202.08384, 2022.
Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprini (iti::205510439 2022.
Dihong Jiang, Sun Sun, and Yaoliang Yu. Revisiting flow generative models for out-of-distribution detection. In International Conference on Learning Representations, 2021.
Wenyu Jiang, Hao Cheng, Mingcai Chen, Chongjun Wang, and Hongxin Wei. Dos: Diverse outlier sampling for out-of-distribution detection. arXiv preprint arXiv:2306.02000 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:16661-18673, 2020.
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43 (11):3964-397)) 2020.
Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.0404 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009a.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009b.
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Phuc H Le-Khac, Graham Healy, and Alan Smeaton. Contrastive representation learning: A framework and review. Ieee Access, 8:193907-193934, 2020.
Jinsol Lee, Charlie Lehman, Mohit Prabhushankar, and Ghassan AIRegib. Probing the purview of neural networks via gradient analysis. IEEE Access, 11:32716-32732. 2023.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems. 31, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of outtof--ditrributin image detection in neural networks. arXiv preprint arXiv: /rXiv:/706.026900 2017.
12

---

Published as a conference paper at ICLR 2025
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards cutt-f-distribution generalization: survey. arXiv preprint arXiv:2108. 132222108..324,,, 2021.
Litian Liu and Yao Qin. Detecting diittiiiitiiittionn through the lens of neural collapse. arXiv preprint drX1v:231..0149, 2023.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based oft--o---issttte of-distribution detection. Advances in neural information processing systems, 33:21464-21475, 2020.
Julia Lust and Alexandru Paul Condurache. Gran: An efficient gradient-norm based detector for adversarial and misclassified examples. arXiv preprint arXiv:2004.09119, 2020.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In International Conference on Machine Learning, pp. 15650-15665. PMLR, 2022a.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? arXiv preprint arXiv:2203.04453 2022b.
Ibrahima Ndiour, Nilesh Ahuja, and Omesh Tickoo. but-of-distribund detection with subspace techniques and probabilistic modeling of features. arXiv preprint arXiv:2012.04250, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. Granada, Spain, 2011.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers of classification neural networks. arXiv preprint arXiv:2308.0277 02760000022...... 2023.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference Machine Learning, pp. 297445. PMLR, 2023.
Jingbo Sun, Li Yang, Jiaxin Zhang, Frank Liu, Mahantesh Halappanavar, Deliang Fan, and Yu Cao. Gradient- based novelty detection boosted by self-supervised binary classification. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8370-8377, 2022a.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection Advances in Neural Information Processing Systems, 34: 144-157, 2021.
detectoooo with rectified activations.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributioo detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022b.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances neural information processing systems, 33:1 18:11889-11 1152,, 2020.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 10(11))195--1900 2008
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8778 2018.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: out-of-disttibuttin with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921 4921-4930, 2022.
13

---

Published as a conference paper at ICLR 2025
Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. arXiv:2303.05053, 2023.
detection with implicit outlier transformation. arXiv preprint
Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning augment distributions for out-of-distribution detection. Advances in Neural Information Processing Systems, 36, 2024.
Yingwen Wu, Tao Li, Xinwen Cheng, Jie Yang, and Xiaolin Huang. Low-dimensional gradient helps out-of-distribution detection. arXiv preprint 13.Xi:23110171060 2023.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-34850 3492. IEEE, 2010.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755. 2015.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506. 03365, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 160.00660..14444 2016.
Enhao Zhang, Chaohua Li, Chuanxing Geng, and Songean Chen. All-around neural collapse for imbalanced classification. arXiv preprint arXiv:2408.0720 2024a.
Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. Epa: Neural collapse inspired robust out-of-distribution detector. arXiv preprint arXiv:2401.017100 2024b.
Haotian Zheng, Qizhou Wang, Zhen Fang, Xiaobo Xia, Feng Liu, Tongliang Liu, and Bo Han. Duttof-dist detection learning with unreliable distrf-distribution sources. Advances in Neural Information Processing Systems, 36, 2024.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): (4:0::15221110 1464, 2017.
Didi Zhu, Zexi Li, Min Zhang. Junkun Yuan, Jiashuo Liu, Kun Kuang, and Chao Wu. Neural collapse anchored prompt tuning for generalizable vision- languan models. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4631-4640, 2024.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6908-6917, 2022a.
Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting out-of-distribution detection with typical features. arXiv preprint arX1v:2210.04200, 2022b.
Ev Zisselman and Aviv Tamar. Deep residual flow for out distribution detection. Proceedings of the IEEF/CVF Conference on Computer Vision and Patern Recognition, pp. 13994-199119 14003, 2020.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 COMBINATION WITH OTHER OUTPUT-BASER LOSS
In our main paper, utilize feature separation loss based OE method (Hendrycks et al. 2018 since it is the most classical approach. In this section, we also combine our feature separation loss with another output-based loss demonstrate our wide availability. We adopt Energy-OE approach (Liu et al. 2020) as our basic loss, which is commonly-use output-based loss. The mathematical formula follows:
min LCR + ALnnrrgy
f
Lenergy E(ee (max(0, E(win) - Min ))²
(8)
+ R (max(0, Mout. ((
where E(x)= = log E, Celite/T is the energy function, and X, Min- and Mout are hyper- parameters. We adopt the recommended setting in Energy-OE (Liu et al. (2020) to set the parameters and finetune model. Combining our feature separation loss with the basic method, obtain training objective loss as follows:
min LCE + XLenergy + aLoia + 3
(9)
In our experiments, we still set = 1.0 and 3 = 1.0 for consistency with previous setting. The results are shown in Table g where method significantly reduces the FPR95 by 6.35% CIFAR100 benchmark compared to the basic Energy- approach, convincingly demonstrating our wide availability and effectiveness.
Table Combination with Energy- loss on CIFARIO and CIFAR100 benchmarks. The best result is in bold.
Mathod
EPROS)
SVIN
AUROC1
FPROS,
LSUN
AUROCCT
FPROS)
ISUN
AUROC
FPROS]
Toxtuces
AUROCT
EPROS]
Places368
AURROCT
FPROS,
Averae
AUROCC
CIFAR 10
Energ-OF Lin al 20207
0.90
2.75
98.90
2.09
Olus
1.20
99.33
0.65
99.14
1.75
99.33
2.20
99.08
7.55
97.88
2,67
98.95
CIUAR
Eneryy - DIE.
30200
17.75
96.94
$4.00
94.82
60.75
67.30
45
90.00
53.50
89
42.34
91.65
11.05
97.65
21.35
96.40
$2.95
88.63
42.65
91.14
51.95
88.81
38.99
92.53
A.2 DETAILED RESULTS ON DIFFERENT ARCIITECTURES
We report the detailed results with ResNet18 and DenseNet121 architectures on CIFAR10 and CIFAR 100 benchmarks in Table I0
Table 10: Detailed Results with ResNet18 and DenseNet121 architectures. The best result is in bold.
Model
Method
SVII
AUROCT
LSUN
AUROCT
EPRUS
ISUN
AUROCH
SPROS]
Textures
AUROCT
EPR95L
Places365
FPRUS
AUROCT
CIFAR-10
4.20
03.58
7.05
11.70
ResNe118
DenceNee121
97.41
98.76
16.30
CIPAI-T000
OF
55.75
92.95
35.45
93.96
70 10
87.35
65.00
88.38
58.50
88.20
56.96
96.19
ReNNe115
DAL
51.55
93.40
32.35
94.65
68.75
88.90
63.50
89.52
$7.50
85.31
$4.89
96:95
Ours
36.70
93.29
34.50
94.12
66.00
88.05 68
$7.35
90.01
56.75
88.39
50.34
90.90
O6
59.40
90.35
48.70
90.15
70.65
83.28
68.90
36
64.75
81.65
62.00
86.76
DenseNet121
87.88
401
85.08
63.00
A.3 HARD OOD DETECTION
In addition to testing on the regular OOD datasets, we further consider three hard OOD datasets proposed in Tack et al. 2020), which considered more difficult to distinguish from ID samples.
15

---

Published as a conference paper at ICLR 2025
Following the same setting in (Tack et al., 2020. Sun et al. 2022b Wang et al. 2024). we evaluate our detection performance on LSUN-FiL Fix Yu et al. 2015). ImageNet-Resize Resize (Deng et al. 2009b) and CIFARI00 (Krizhevsky et al., 2009a) with CIFAR to as the ID dataset. Specifie results are shown in Table 11 As we can see, our method shows comparable performance with DAL (Wang et al. 2024) three hard OOD datasets, outperforming the baseline OE method by 2.55% on FPR95 the ImageNet-Resize dataset.
Table 11: Hard OOD detection on CIFARIO benchmark.
Near-OOD Dataset
Method
LSUN-Fix
ImageNct-Resize
CIFAR-100
Tiny-ImageNet
FPR95
AUROCT
FPR95
AUROCT
FPROS
AUROC
FPR95}
AUROCT
With contrastive learning
CSI
39.79
93.63
37.47
93.93
45.
87.64
CIDER
8,98
98.56
43.45
93.82
55.84
90.0
KNN+*
95.75
30.52
94.85
40.00
With auxillary OOD data
OE
1.00
99.53
7,20
98.48
25.05
94.86
19.55
91.49
DAL
0.65
99.59
26.00
94.35
20.75
92.18
Ours
0.75
99.07
4.65
98.42
24.60
94.69
17.65
92.48
A.4 GENERALIZATION TO CLASS- IMBALANCD DATASETS
Since method is based on the assumption of the NC property, we recognize that the phenomenon may not always hold in real-world scenarios. Therefore, in this section, we evaluate our performance on imbalanced data distribution to explore the generalizability of our approach.
On imbalanced data distribution, the NC property declines to "Minority Collapse" Zhang et al. (2024a). where the classifiers for minority classes destroys the geometric structure of NC, but does not break the low- -iw-dimensional property of features of ID data. Following the settings in Cao al. (201 9); Zhu et al. (022a). we create an imbalanced version of CIFARIO dataset, denote as CIFARIO-LI Specifically, we consider a long-tailed imbalance with ratio (denote the ratio between sample sizes of the most frequent and least frequent class). Based on above imbalanced data, we firstly pretrain a WideResNNet400. model on CIFAR 10-LT, and then finetune the model using auxiliary OOD datasets. The training setting is the same as our previous experiments. The final detection performance of our method and OE is shown in Table 12 which validates our effectiveness under imbalanced data condition.
squeezed into
direction. This phenomenon
Table 12: Results imbalanced CIFARIO data. We report the FPR95/AUROC.
Malhal
FFROSL
SVHN
AUROC-
FPRO5.
LSUN
AUROCC
FPR95)
ISUN
AUROCC
Testures FFROS,
AUROCC
FPROSL
Plaore665
AUROCT
FPROS,
Averaee
AURRCC
OF Hemdryyckss et al 2018
15.15
96 56
12.00
97.25
19:20
96.63
18.05
95.15
29.75
03.63
18.93
95.04
Chix
14.05
98
13.05
96.93
19.55
93.38
24.05
95.63
24.95
94.07
17.73
96.08
A.5 TRAINING STRATEGY CHOICE AND IMPACT
In this section, we discuss the choice of our training strategy and its impact on our performance. Our training process is consisted of two stage: Stage I-training with cross-entropy loss to ensure the occurrence of NC phenomenon; Stage Il-training with all losses including cross-entropy, outlier exposure, and our proposed separation and clustering losses to enlarge the discrepancy between II) and OOD data. When a well-trained model is used as the initial parameter, Stage is not necessary. In practice, for the sake of an unified and simple framework, we can always firstly train with CE loss, and when the accuracy remains unchanged, switch to Stage II. But in our experiment, to align with experimental settings of other methods, we fixed the training epoch at 5, which is a relatively small epoch, thus we directly use Stage II on the ImageNet benchmark.
To further investigate the impact of training strategy on our performance, we conduct experiments
16

---

Published as a conference paper at ICLR 2025
on CIFARIO benchmark (where the initial model parameter is half-trained), with different training epochs of Stage I. Specifically, denote the training epoch of Stage I as "Ep-CE", training epoch of choose different Ep-CE ranging from [0, 10, 20. 25, 30, 40, 50] with fixed Ep-All equalling to 25. When Ep-CE equals to zero, it corresponds to directly train the model using Stage II. Experiment results shown in Table 13] where method is not sensitive the strategy choice, but CE loss fine-tuning does enhance performance for half-trained models.
Stage II "Ep-All",
Table 13: Influence of training strategy on CIFARIO benchmark. FPR95/AUROC across five OOD datasets.
We report the average
Ep-CE + Ep-All 0+50
10+25
20+25
25+25
30+25
40+25
50+25
FPR95/AUROC: ID Acc 94.43
2.53/98.53
2.56/99.066 95.33
2.56/99.09 95.52
2.49/98.95 93 95.53
2.46/99.16 95.56
2.01/99.11 95.64
2.53/98.97 95.35
A.6 FEATURE DISTANCE ANCHORING ON CLASS MEAN
Since our training loss explicitly optimizes the cosine similarity between FC weights and features, the distance metric we used in Table is align with objective in some degree. Therefore, for more fair comparison, we evaluate the feature distance based on the class mean vectors instead of FC weights in this section. The result is shown in Table 14 The absolute value is different from results calculated based on model weights, but in relative comparison, our method still shows better feature separation between ID and OOD data.
Table 14: Feature Distance based on class mean vectors. we report the average Euclidean distance and cosine similarity on the whole dataset.
Euclidean Distance
Cosine Similarity
Method
0OD
Difft
ID
OOD
Difft
Vanilla
2.31
3.08
0.77
0.94
0.81
0.13
OE
1.46
4.91
3.45
0.94
0.49
0.45
Ours
1.47
6.18
4.71
0.98
0.31
0.67
A.7 FLUCTUATION IN DETECTION PERFORMANCE
We have observed considerable fluctuations in the performance of the DAL method (Wang et al. 2024) under repeated experiments with identical settings. Therefore, evaluate the mean and variance of performance after repeating the same experiment five times. The results, shown in Table
15 indicate that our approach exhibits greater stability, particularly on the CIFAR100 benchmark. Notably, even the OE method shows the FPR95 variance of 1.137% on CIFARI00, whereas our method maintains a variance only 0.027%.
Table 15: Fluctuation in detection performance of different methods.
CIFARIO
CIFAR 100
Method
FPR95.J
AUROC1
FPR95.
AUROCT
OF Hendrycks al. (2018)
3.22 10.0017
99.07 10.0014
36.29.1.137
11.137
92.31 10.01
DAL Wang et al. (2024)
2.87+0.0234
98. 98.82+0.008 0027
30.4442.216
93.07+0
Ours
2.49 00.0007
98.92+0.0083
29.50+0.027
93.98+0.014
A.8 DIFFERENT SCORE FUNCTIONS
Since our proposed score function in Eq detect OOD samples while other auxiliary OOD data based methods only employ MSP score (Hendrycks & Gimpel, 2016). in this part, we also evaluate model using MSP score for fair comparison. The results in Table 16 demonstrate
17

---

Published as a conference paper at ICLR 2025
that approach also achieves commendable performance under the MSP score, with only slight decline compared to using the proposed score function.
Table 16: Performance of adopting different score functions in our method.
CIFARIO
CIFAR100
Method
Score Function
FPR954
AUROC1
FPR95.
AUROC1
MSP Eq. 6
2.77 2.49
98.76
93.27 29.96 29.47
Ours
98.93
94.00
A.9 VISUALIZATION
In Figure we visualize the random two class samples of CIFAR1O dataset and the test unseen OOD samples of SVHN dataset. The low-dimensional visualization is calculated by linear projection into the subspace spanned by W1, W2 and principal components of OOD features, where w; is the corresponding model parameters of the last fully connected layer. Specifically, denote features as z E Raxd, the reduction matrix as M E Rdx3 where n the sample number and d the feature dimension, then the coordinate in 3D space is computed as: 2an 2.M, 23D nx
Using the above linear projection operation, we obtain the coordinate for both ID and OOD samples, then visualize them in 2D and 3D space. In the following. samples and the test unseen OOD data to visualize their features in 2D and 3D space. In these figures, we can discover that, for all samples of the classes in CIFARIO 10, ID features rarely distribute on redundant dimensions, in contrast, OOD features almost locate on redundant dimensions while little component on model weight dimensions.
respectively choose two class of ID
Clere
Clss
Cass
Cleos
Cless-1
Cass-1
Curberr
Cudlaer
(a) Vanilla model
(b) OE-trained model
(c) Our model
Cless:
Class-1
Class-1
Outtier
Outlier
Outlier
10
-3
15
10.0
5.0
2.5
22 0
(d) Vanilla model
OE-trained model
(f) Our model
Figure 3: ID sample: Class-0 and Class- 1, OOD sample: SVHN
18

---

Published as a conference paper at ICLR 2025
Class
Class-
Cles-2
Gass2
Outler
Outle
(a) Vanilla model
(b) OE-trained model
(c) Our model
Class-2
Class-3
Class-3
Outlier
-
5
2
Class-3
Outlier
-4
6
(d) Vanilla model
(e) OE-trained model
(l) Our model
Figure 4: IDD sample: Class-2 2 and Class-3, OOD sample: SVHN
Clea-1 Clas
Cheo-4
Outbe
Orter
(a) Vanilla model
(b) OE-trained model
(c) Our model
Class Class 5
Class-4
Clacs-4
Class
Class-5
Outlier
Outlier
Outlier
110
-1
6
-5
6
-2
-2
0
-
(d) Vanilla model
OE-trained model
(f) Our model
Figure 5: ID sample: Class and Class-5, OOD sample: SVHN
19

---

Published as a conference paper at ICLR 2025
Clas 6 Casso Oater
Clsss
Cass B
Outler
OuBer
(a) Vanilla model
(b) OE-trained model
(c) Our model
Cless-6
Class-7
Class Outlier
Outtier
5
10
15
8
6
-2.
1
22
Glass
5.0
7.5
0
2
4
-4
Outtier
Class-7
7.5 10.0
-2
(d) Vanilla model
(e) OE-trained model
(l) Our model
Figure ID sample: Class-6 6 and Class-7, OOD sample: SVHN
Outlx
(a) Vanilla model
OE-trained model
(c) Our model
Clas-88
Cless-8
Class-8
Clsss99
Class-9
Class-9
Outtier
Outlier
Outlier
0.0
0.5
2.5
-.5
7.5
7.0
12.5
22
-
0
2
0
2
7
0
(d) Vanilla model
(e) OE-trained model
(f) Our model
Figure sample: Class-8 and Class-9, OOD sample: SVHN
20


Reviews:
[{"review_id": "tq1c3m4jI4", "reviewer": "Reviewer_J1wr", "summary": "The paper proposes an OOD detection approach that enhances feature separation. The proposed method is based on Neural Collapse (NC), which suggests that features of ID samples collapse to similar values within a class. Motivated by NC, the authors introduce Separation Loss to encourage OOD features to occupy subspaces orthogonal to the principal subspace of ID features, effectively separating ID and OOD samples at the feature level.", "questions": "Please check questions alongside the weaknesses.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-motivated in its use of Neural Collapse (NC) to improve OOD detection, effectively highlighting NC’s natural feature clustering property as a robust basis for separating ID and OOD data. \n- The incorporation of Neural Collapse as a theoretical foundation adds substantial depth to the methodology, offering a compelling explanation for the effectiveness of feature-level separation in enhancing OOD detection. \n- The proposed approach is conceptually straightforward and easy to implement.", "weaknesses": "- In Section 3.4, the paper discusses two alternative training strategies: one that directly incorporates all loss terms and another that initially trains with cross-entropy (CE) loss before introducing the proposed losses. However, guidance on when to choose between these strategies in general cases is lacking. Additional discussion on how this choice could affect the model’s performance or usability would be beneficial.\n- The proposed method mainly includes two loss terms, i.e., the Sep loss and Clu loss, upon the OE loss. As discussed in the paper (e.g., the abstract), the Sep loss reflects the main motivation of neural collapse (NC). The ablation studies in Table 5 show that these two terms perform with different significances. The behaviours and importance of these two loss teams are unclear. \n- Although the proposed method performs well on ImageNet-1k (Table 1), its performance is less consistent on other benchmarks, as seen in Table 2 and particularly Table 3, where broader comparisons are provided. Further explanation or analysis regarding this variability would be valuable to understand the method’s limitations. \n- More analysis of the learned representation space, such as visualizations of the embedding space, would provide insight into how well the method achieves feature separation and could strengthen the evaluation of its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9S3c8dX3zK", "reviewer": "Reviewer_xHA8", "summary": "The paper proposes a novel strategy for out-of-distribution detection. The approach is based on Neural Collapse and employs this property to avoid modeling OOD feature distributions while effectively segregating ID and OOD features.", "questions": "- The proposed method shares fundamental similarities with KNN, as both approaches classify data based on distance calculations in feature space, identifying ID data through closer distances and OOD data through farther distances. Could the authors elaborate on the key distinctions between their method and the KNN approach?\n- Given the use of Wide ResNet-40-2 architecture for CIFAR-10 and CIFAR-100 datasets, does the method have specific requirements regarding network feature dimensions? A detailed analysis of this aspect would be valuable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation behind this method is reasonable and worth exploring.", "weaknesses": "- While Figure 2 demonstrates the distribution changes of the OOD training set during training, which is expected, it would be more insightful to see the distribution patterns of OOD data during the testing phase.\n- The improvement in AUROC metrics compared to existing methods is relatively modest.\n- Although the authors claim to follow DAL's experimental setup on ImageNet (lines 321, 332), there are discrepancies between the metrics reported in Table 1 (lines 374, 375) and those in Table 16 of the DAL paper (\"Learning to Augment Distributions for Out-of-Distribution Detection\").\n- The experimental comparison on ImageNet is limited to only two methods, whereas the DAL paper compared five methods in their Table 16, suggesting insufficient comparative analysis.\n\np.s. There appears to be an inconsistency in decimal places for the average AUROC of Energy between lines 381 and 382.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vTCa7rQZ6I", "reviewer": "Reviewer_sGxj", "summary": "This paper focuses on feature-space separation rather than output-space discrepancies. The authors propose a novel approach leveraging Neural Collapse, which describes how ID samples’ penultimate features converge near their respective class weights. The core contribution is the Separation Loss, which aligns OOD features orthogonally to ID feature subspaces. Extensive experiments across standard benchmarks demonstrate that the proposed method achieves state-of-the-art performance in OOD detection.", "questions": "1. The improvements on the ImageNet benchmark are relatively marginal compared to the CIFAR benchmarks. Could you provide a more detailed discussion to clarify the potential reasons for this difference?\n\n2. As noted in line 469, `When faced with unseen OOD data, the feature variations tend to fall on the new dimension, resulting in minimal changes on the output.` Would these orthogonal dimensions become exhausted when dealing with a large-scale ID dataset? Additional justification on this point would be helpful, no further empirical results are needed.\n\n3. In Table 9, the calculation relies on FC weights, which aligns with the paper’s motivation since the optimization is specifically designed to achieve these results. Would using empirical class mean features instead of FC weights yield different outcomes? This approach could also further align with the NC motivation, given that the penultimate features of ID samples within a class are nearly identical to the last layer weights.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a separation loss based on the Neural Collapse phenomenon to enhance feature-space disparity between ID and OOD samples. This approach builds on strong analytical foundations from NC, addressing key challenges specific to OOD detection.\n\n2. The experiments are comprehensive, covering multiple datasets and architectures, with ablation studies that effectively address most concerns.\n\n3. The proposed method is straightforward and clear. The proposed method is broadly applicable to any OOD detection method based on Outlier Exposure.", "weaknesses": "I hope the authors could address my questions as follows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "he8tBQB8y9", "reviewer": "Reviewer_bNBL", "summary": "The paper introduces a novel approach for enhancing out-of-distribution (OOD) detection in deep neural networks (DNNs) by improving outlier exposure training. This approach leverages the Neural Collapse (NC) phenomenon where features within each class collapse to a point forming an ETF simplex with other classes to improve the separability of in-distribution (ID) and OOD data. The authors propose a new loss function, \"Separation Loss,\" designed to maximize the orthogonality between the feature representations of ID and OOD data. The orthogonality is maximized by minimizing the cosine similarity with the last layer FC layer weights for OOD data and by aligning the representation of ID data with the predicted class weight from the last layer. This method achieves state-of-the-art (SOTA) performance on CIFAR10, CIFAR100, and ImageNet benchmarks without requiring additional data augmentation or sampling techniques.", "questions": "1- I would suggest testing your method using OpenOOD framework[1], as you can test Near vs Far OOD beside testing transformer based architectures.\n\n2- The manuscript discusses the application of your method under the assumption of the Neural Collapse (NC) property. However, it is not clear how these methods might perform in scenarios where NC does not hold, such as in imbalanced datasets[2]. Could the authors investigate and elaborate on the potential impact on the results if the NC property is weakened or absent due to dataset characteristics like imbalance? Additionally, I recommend including experiments with imbalanced datasets to provide empirical evidence of the method's robustness under these conditions. This would enhance the paper's contribution by addressing the generalizability of the NC phenomenon across different data distributions.\n\n\nBibliography:\n\n1. Zhang J, Yang J, Wang P, Wang H, Lin Y, Zhang H, Sun Y, Du X, Zhou K, Zhang W, Li Y. Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.09301. 2023 Jun 15.\n\n2. Zhang E, Li C, Geng C, Chen S. All-around Neural Collapse for Imbalanced Classification. arXiv preprint arXiv:2408.07253. 2024 Aug 14.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The concept of applying the NC phenomenon to directly manipulate feature spaces for OOD detection is innovative. Unlike traditional methods that focus on output discrepancies, this paper focuses on feature separability, which is a fresh perspective in the field.\n\n- The empirical validation is robust, covering several benchmarks and comparing against a comprehensive suite of methods. The novel loss function, grounded in theoretical insights from NC, is shown to effectively enhance feature separation.\n\n- The paper is well-organized and articulately written, with clear explanations of the NC phenomenon, the motivation for the new loss function, and the experimental setup. The visualizations effectively illustrate how feature separation is achieved, aiding in understanding.\n\n- The approach addresses a critical gap in OOD detection by improving model reliability in real-world applications, where ID and OOD data may not be clearly distinguishable through outputs alone. This has significant implications for deploying DNNs in safety-critical domains.", "weaknesses": "Dependency on NC Property: The effectiveness of the method is heavily dependent on the presence of the NC phenomenon, which may not occur in all network architectures or training configurations. (The NC property usually fail on the testing set [1])\n\nThe authors claim that their method works well across different architectures while only testing ResNet and WideResnet and DenseNet. It would be more valuable to test your method against transformer based architecture like ViT or Swin as their features differ due to the global attention mechanism and patch based representations. This experiment will validate the method’s generalizability to different architectures.\n\nIn table 2, categorizing the OOD datasets into Near and Far OOD would provide useful insight in the performance of the separation loss, as it might be working on Far-OOD better than Near-OOD which are more entangled in the representation space.\n\nBibliography:\n1. Hui L, Belkin M, Nakkiran P. arXiv.org. 2022 [cited 2023 Dec 10]. Limitations of Neural Collapse for Understanding Generalization in Deep Learning. Available from: https://arxiv.org/abs/2202.08384v1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)