Detecting Out-of-Distribution through the Lens of Neural Collapse

## Abstract

Abstract Out-of-Distribution (OOD) detection is essential for safe deployment; however, existing detectors exhibit generalization discrepancies and cost concerns.
To address this, we propose a highly versatile and efficient OOD detector inspired by the trend of Neural Collapse on practical models, without requiring complete collapse.
By analyzing this trend, we discover that features of in-distribution (ID) samples cluster closer to the weight vectors compared to features of OOD samples.
Additionally, we reveal that ID features tend to expand in space to structure a simplex Equiangular Tight Framework, which explains the prevalent observation that ID features reside further from the origin than OOD features.
Taking both insights from Neural Collapse into consideration, our OOD detector utilizes feature proximity to weight vectors and further complements this perspective by using feature norms to filter OOD samples.
Extensive experiments on off-the-shelf models demonstrate the efficiency and effectiveness of our OOD detector across diverse classification tasks and model architectures,
mitigating generalization discrepancies and improving overall performance.

## 1 Introduction

Machine learning models deployed in practice will inevitably encounter samples that deviate from the training distribution.
As a classifier cannot make meaningful predictions on test samples that belong to classes unseen during training, it is important to actively detect and handle Out-of-Distribution (OOD) samples.
Considering the diverse and oftentimes time-critical application scenarios, an OOD detector should be computationally efficient and can effectively generalize across various scenarios.

In this work, we focus on *post-hoc* methods, which address OOD detection independently of the training process.
One line of prior work designs OOD scores over model output space (Djurisic et al., ; Hendrycks et al., ; Liang et al., ; Liu et al., ; Sun et al., ; Sun & Li, )
and another line of work focuses on the feature space, where OOD samples are observed to deviate from the clusters of ID samples (Lee et al., ; Mahalanobis, ; Sun et al., ; Tack et al., ).
While existing research has made strides in OOD detection, they still face two major challenges: 1) maintaining detection effectiveness across different scenarios, and 2) ensuring computational efficiency for real-world deployment.
For example, both output space and feature space methods suffer from performance discrepancy across different classification tasks, as shown in Table 1 (a).
Specifically, strong algorithms on CIFAR-10 (Krizhevsky et al., ) OOD benchmarks perform suboptimally on ImageNet (Deng et al., ) OOD benchmarks, and vice versa.
No existing algorithm can simultaneously rank in the top three across two benchmarks, leading to sub-optimal average performance as shown in Table 1 (b).
Such discrepancy is also observed across different architectures, as shown in Table 2.
In addition, feature space methods, which rely on auxiliary models, raise efficiency concerns.
For example, Lee et al. () learns a Gaussian mixture model from training features and detects OOD based on Mahalanobis distance Mahalanobis ();
Sun et al. () records the training features and measures OOD-ness based on the k-th nearest neighbor distance to the training features.
As shown in Liu & Qin (), such reliance on auxiliary models introduces additional cost, posing challenges for time-critical applications.

Figure: Figure 1: Illustration of our framework inspired by Neural Collapse. Left: On the penultimate layer, ID samples cluster near their predicted class weight vectors (marked by stars) while OOD samples reside separated, as shown by UMAP. Middle: ID and OOD samples are separated by $\mathtt{pScore}$ (Equation 6), which measures feature proximity to weight vectors. Also, ID samples tend to be further from the origin, illustrated with $\mathtt{L1}$ norms. Right: ID samples cluster near a simplex Equiangular Tight Framework, illustrated with black arrows denoting weight vectors. We detect OOD by thresholding on $\mathtt{pScore}$, selecting blue-shaded hypercones centered at weight vectors, with OOD samples outside these areas. We also filter OOD samples characterized by smaller feature norms. Left & Middle present a CIFAR-10 ResNet-18 classifier with OOD set SVHN. While Neural Collapse does not completely converge (Left), the ID/OOD relationship inspired by the trend remains valid (Middle). Right depicts our scheme on a three-class classifier with 2D penultimate space.
Refer to caption: x1.png

To this end, we aim to develop an efficient and versatile OOD detector by focusing on the penultimate layer, i.e., the layer before the linear classification head.
We take insights from Neural Collapse (Papyan et al., ), which characterizes the interplay between the linear classification head and the penultimate layer features in training.
Neural Collapse is observed across diverse architectures and classification tasks (see Appendix E).
While the complete collapse requires strict conditions like prolonged training, we leverage its early-stage *trend* observed in (He & Su, ) to study practical models.
The effectiveness of prior methods utilizing Neural Collapse in OOD detection Zhang et al. (); Ammar et al. () further supports the prevalence of such trend in practical models.

Particularly, we revisit the observation that ID features tend to form clusters while OOD features reside apart.
While this observation is well-established in prior literature Lee et al. (); Sun et al. (); Tack et al. (), the underlying mechanism remains largely unexplained.
Separately, Neural Collapse reveals that features of each class gradually converge toward a single point during training.
We suggest that the clustering behavior observed in *off-the-shelf* models can reflect the trend of Neural Collapse.
Inspired by this, we leverage the landscape of Neural Collapse to study:

*Where do features of ID samples form clusters?*

To address the question, we first demonstrate that as a deterministic effect of Neural Collapse, features of training samples will converge towards *the weight vectors of the predicted class*.
Additionally, Neural Collapse reveals that training features also converge towards a simplex Equiangular Tight Framework (ETF) (Equation 1). The spatial structure of an ETF, illustrated in Figure 1 Right, corresponds to the maximum separation in space achievable by equiangular vectors, requiring the features to reside sufficiently far from the origin.

Figure: Figure 2: Features of ID samples tend to cluster closer to the predicted class weight vectors, indicated by higher average cosine similarity (Equation 5) than OOD. This observation, inspired by the trend of Neural Collapse, emerges early in the training of this CIFAR-10 ResNet-18 classifier, with OOD set SVHN, without requiring convergence.
Refer to caption: extracted/5924189/evolution.png

The complete convergence landscape of Neural Collapse sheds light on the geometric structure of ID clusters on practical models.
Specifically, for ID test samples, drawn from the same distribution as training samples, we anticipate a similar *trend* of clustering behavior towards the weight vectors and towards an ETF.
Conversely, OOD samples do not undergo the same training process, which enables the model to align features with weight vectors and to expand features to accommodate the spatial structure of ETF in Neural Collapse.
Therefore, we do not expect the model to effectively align the weight vectors learned from ID features with unseen OOD features.
Nor do we anticipate the model to posit OOD features far from the origin to structure an ETF.
To validate our hypotheses, we trace a model’s training stages in Figure 2.
We observe that ID samples consistently cluster closer to the weight vectors than OOD samples.
This observation emerges early during training, without requiring complete convergence of Neural Collapse.
Our observation is reinforced in the UMAP (McInnes et al., ) visualization on an *off-the-shelf* CIFAR-10 classifier with ResNet-18 backbone in Figure 1 Left.
Here, ID features do not completely collapse into weight vectors.
Nevertheless, ID features cluster near predicted class weight vectors (marked by stars), whereas OOD features are distant.
Combining our observation with (Zhu et al., ), which show the weight vectors form an ETF, we conclude that ID features are driven to structure the ETF during training, whereas OOD features lack the incentive to expand in space to form an ETF.
Note that the lack of incentive for OOD features to expand explains the well-established observation (Tack et al., ; Huang et al., ; Sun et al., ) that OOD features tend to reside closer to the origin, offering an alternative to model confidence view in Park et al. ().

Based on our understanding, we design an efficient and versatile OOD detector.
We first leverage feature proximity to the weight vectors to characterize ID clustering, bypassing auxiliary models and reducing the computational cost.
Specifically, we define an angle-based proximity score as the norm of the projection of the weight vector of the predicted class onto the sample feature.
As shown in Figure 1 Middle, our proximity score can effectively separate ID/OOD.
A higher score indicates closer proximity and a lower chance of OOD-ness.
Geometrically, thresholding on the score selects hyper-cones centered at the weight vector, as illustrated in Figure 1 Right.
Notably, our proximity score effectively incorporates class-specific information and brings in performance benefits as well as efficiency gain.
Complementing the proximity score’s contingency on ID clustering, we also consider feature distance to the origin.
Specifically, ID features tend to reside further from the origin as they expand in space to form an ETF, whereas OOD features tend to reside near the origin, as illustrated by Figure 1 Right.
Using the L1 norm as an example metric for distance to the origin, we observe that ID features can be separated from OOD features, as supported by Figure 1 Middle.
Combining both aspects, we propose Neural Collapse Inspired OOD Detector ($\mathtt{NCI}$).

Notably, prior methods, e.g., KNN Sun et al. (), focus on ID clustering but do not explicitly consider feature distance to the origin.
Such approaches fall short in scenarios like ImageNet benchmarks but yield superior performance in CIFAR-10 benchmarks in Table 1(a).
Conversely, methods such as Energy Liu et al. (), Energy-based ASH Djurisic et al. (), and, Energy-based Scale Xu et al. () inherently utilize feature distance to the origin by considering log-sum-exp of logits, yet largely overlook ID clustering.
These approaches excel in scenarios like ImageNet, but perform sub-optimally in others, e.g., CIFAR-10.
Through the lens of Neural Collapse, we explain, connect, and complete prior methods under a holistic view, resulting in reduced latency and generalization discrepancies.

We summarize our main contributions below:

- •
Understanding and Observation:
By analyzing ID clustering through the trend of Neural Collapse, we novelly establish the significance of weight vectors in the clusters.
We also explain the observation that ID features tend to be farther from the origin from a spatial structure perspective.
Our understanding and observation do not depend on complete complete Neural Collapse convergence.
- •
OOD Detector:
We leverage feature proximity to the weight vectors of predicted classes for OOD detection, integrating class-specific information.
Complementary to feature clustering, we propose to detect OOD samples by thresholding the feature distance to the origin.
- •
Experimental Analysis:
We evaluate $\mathtt{NCI}$ across diverse classification tasks (CIFAR-10, CIFAR-100, ImageNet) and model architectures (ResNet, DenseNet Huang et al. (), ViT Dosovitskiy et al. (), Swin Liu et al. ()).
Rather than focusing on *individual* benchmarks, $\mathtt{NCI}$ reduces the generalization discrepancies and improves the *overall* effectiveness.
In addition, $\mathtt{NCI}$ matches the latency of vanilla *softmax-confidence* detector.

#### Remark on Convergence of Neural Collapse & NCI Effectiveness.

*Complete* convergence of Neural Collapse for ID samples often requires strict conditions unmet in practice.
However, NCI does not depend on convergence; instead, it leverages the *trend* of Neural Collapse, which we empirically validate on practical models.
Additionally, the effectiveness of prior methods utilizing Neural Collapse in OOD detection (Zhang et al., ; Ammar et al., ) further supports the prevalence of the trend of Neural Collapse in practical models.
We extensively validated the effectiveness of NCI on practical models without convergence requirement.

#### Remark on NCI Performance.

NCI does not focus on individual benchmarks.
While NCI may not achieve the best performance on every benchmark, existing detectors exhibit larger generalization discrepancies.
NCI mitigates the discrepancies, achieving the best *overall* performance across all benchmarks. Additionally, NCI incurs minimal latency and enhances computational efficiency.

## 2 Problem Setting

We consider a data space $\mathcal{X}$, a class set $\mathcal{C}$, and a classifier $f:\mathcal{X}\rightarrow\mathcal{C}$, which is trained on samples *i.i.d.* drawn from joint distribution $\mathbb{P}_{\mathcal{X}\mathcal{C}}$.
We denote the marginal distribution of $\mathbb{P}_{\mathcal{X}\mathcal{C}}$ on $\mathcal{X}$ as $\mathbb{P}^{in}$.
And samples drawn from $\mathbb{P}^{in}$ are In-Distribution (ID) samples.
In practice, the classifier $f$ may encounter $\bm{x}\in\mathcal{X}$ yet is not drawn from $\mathbb{P}^{in}$.
We say such samples are Out-of-Distribution (OOD).

In this work, we focus on detecting OOD samples from *classes unseen during training*, for which the classifiers cannot make meaningful predictions.
The OOD detector $D:\mathcal{X}\rightarrow\{\text{ID},\text{OOD}\}$ is commonly constructed as: $D(\bm{x})=\begin{cases}\text{ID}&\text{if }s(\bm{x})\geq\tau\\
\text{OOD}&\text{if }s(\bm{x})<\tau\end{cases}$, where $s:\mathcal{X}\rightarrow\mathbb{R}$ is a score function of design and $\tau$ is the threshold.
Considering the diverse application scenarios, an ideal OOD detector should be efficient and generalizable.
In this work, we leverage insights from Neural Collapse to achieve reduced computational costs and minimize generalization discrepancies.

## 3 OOD Detection through the Lens of Neural Collapse

In this section, we re-examine the observation in Lee et al. (); Sun et al. () that ID features tend to form clusters while OOD features deviate from the clusters.
We suggest understanding the clustering phenomenon can reflect the trend of the Neural Collapse (Papyan et al., ), which does not necessitate complete Neural Collapse convergence.
Leveraging the landscape revealed by Neural Collapse, we examine:

*Where do features of ID samples form clusters?*

Through analytical and empirical study, we hypothesize and validate with pre-trained models that
(1) ID features tend to cluster closer to the weight vectors compared to OOD features;
(2) ID clusters tend to reside further from the origin, as necessitated by their spatial structure.
From our understanding, we develop a *post-hoc* OOD detector with enhanced efficiency and effectiveness.

### 3.1 Neural Collapse: Convergence of Training Features

Neural Collapse, first observed in Papyan et al. (), occurs on the penultimate layer across canonical classification settings.
To formally introduce the concept, we use $\bm{h}_{i,c}$ to denote the penultimate layer feature of the $i_{th}$ training sample with ground truth / predicted label $c$, Neural Collapse is framed in relation to

- •
the feature global mean, $\bm{\mu}_{G}=\mathrm{Ave}_{i,c}\bm{h}_{i,c}$, where $\mathrm{Ave}$ is the average operation;
- •
the feature class means, $\bm{\mu}_{c}=\mathrm{Ave}_{i}\bm{h}_{i,c},\ \forall c\in\mathcal{C}$;
- •
the within-class covariance, $\bm{\Sigma}_{W}=\mathrm{Ave}_{i,c}(\bm{h}_{i,c}-\bm{\mu}_{c})(\bm{h}_{i,c}-\bm
{\mu}_{c})^{T}$;
- •
the between-class covariance,
$\bm{\Sigma}_{B}=\mathrm{Ave}_{c}(\bm{\mu}_{c}-\bm{\mu}_{G})(\bm{\mu}_{c}-\bm{
\mu}_{G})^{T}$;
- •
the linear classification head, i.e. the last layer of the NN, $\arg\max_{c\in\mathcal{C}}\bm{w}_{c}^{T}\bm{h}+b_{c}$, where $\bm{w}_{c}$ and $b_{c}$ are parameters corresponding to class $c$.

Neural Collapse comprises four inter-related
limiting behaviors:

(NC1) Within-class variability collapse: $\bm{\Sigma}_{W}\rightarrow\bm{0}$

(NC2) Convergence to a simplex Equiangular Tight Frame (ETF):

$$ $\displaystyle|\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2}-\|\bm{\mu}_{c^{\prime}}-\bm{ \mu}_{G}\|_{2}|\rightarrow 0,\ \forall\ c,\ c^{\prime}$ (1) $\displaystyle\frac{(\bm{\mu}_{c}-\bm{\mu}_{G})^{T}(\bm{\mu}_{c^{\prime}}-\bm{ \mu}_{G})}{\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2}\|\bm{\mu}_{c^{\prime}}-\bm{\mu}_{ G}\|_{2}}\rightarrow\frac{|\mathcal{C}|}{|\mathcal{C}|-1}\delta_{c,c^{\prime}} -\frac{1}{|\mathcal{C}|-1}$ $$

where $\delta_{c,c^{\prime}}$ is the Kronecker delta symbol.

(NC3) Convergence to self-duality:

$$ $\frac{\bm{w}_{c}}{\|\bm{w}_{c}\|_{2}}-\frac{\bm{\mu}_{c}-\bm{\mu}_{G}}{\|\bm{ \mu}_{c}-\bm{\mu}_{G}\|_{2}}\rightarrow\bm{0}$ $$

(NC4) Simplification to nearest class center:

$$ $\displaystyle\arg\max_{c\in\mathcal{C}}\bm{w}_{c}^{T}\bm{h}+b_{c}\rightarrow arg \min_{c\in\mathcal{C}}\|\bm{h}-\bm{\mu}_{c}\|_{2}$ $$

We first remark on (NC2) that an ETF achieves the maximum separation possible for globally centered equiangular vectors Papyan et al. () and extends in space, as visualized in Figure 1 Right. Since training features converge towards an ETF, they need to have sufficient norms to accommodate the spatial arrangement.

We next build on (NC1) and (NC3) to demonstrate in the following that training features converge towards the weight vectors of the linear classification head, up to a scaling factor.

###### Theorem 3.1 .

(NC1) and (NC3) imply that for any sample $i$ and its predicted class $c$, we have

$$ $(\bm{h}_{i,c}-\bm{\mu}_{G})\rightarrow\lambda\bm{w}_{c}$ (2) $$

in the Terminal Phase of Training, where $\displaystyle\lambda=\frac{\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2}}{\|\bm{w}_{c}\|_{
2}}$.

###### Proof.

Considering that $(\bm{h}_{i,c}-\bm{\mu}_{c})(\bm{h}_{i,c}-\bm{\mu}_{c})^{T}$ is positive semi-definite for any $i$ and $c$.
$\bm{\Sigma}_{W}\rightarrow\bm{0}$ thus implies $(\bm{h}_{i,c}-\bm{\mu}_{c})(\bm{h}_{i,c}-\bm{\mu}_{c})^{T}\rightarrow\bm{0}$ and $\bm{h}_{i,c}-\bm{\mu}_{c}\rightarrow\bm{0},\ \forall i,c$.
With algebraic manipulations, we have

$$ $\frac{\bm{h}_{i,c}-\bm{\mu_{G}}}{\|\bm{\mu}_{c}-\bm{\mu_{G}}\|_{2}}-\frac{\bm{ \mu}_{c}-\bm{\mu_{G}}}{\|\bm{\mu}_{c}-\bm{\mu_{G}}\|_{2}}\rightarrow\bm{0},\ \forall i,c$ (3) $$

Applying the triangle inequality, we have

$$ $\displaystyle|\frac{\bm{h}_{i,c}-\bm{\mu_{G}}}{\|\bm{\mu}_{c}-\bm{\mu_{G}}\|_{ 2}}-\frac{\bm{w}_{c}}{\|\bm{w}_{c}\|_{2}}|\leq|\frac{\bm{h}_{i,c}-\bm{\mu_{G}} }{\|\bm{\mu}_{c}-\bm{\mu_{G}}\|_{2}}-\frac{\bm{\mu}_{c}-\bm{\mu_{G}}}{\|\bm{ \mu}_{c}-\bm{\mu_{G}}\|_{2}}|+|\frac{\bm{w}_{c}}{\|\bm{w}_{c}\|_{2}}-\frac{\bm {\mu}_{c}-\bm{\mu}_{G}}{\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2}}|.$ (4) $$

Since both terms on the RHS converge to $\bm{0}$, as demonstrated by equation 3 and (NC3), it follows that the LHS also converges to $\bm{0}$.
∎

###### Theorem 3.1 .

###### Proof.

### 3.2 Trend of Neural Collapse and Geometric Structure of the ID Clusters

While the complete collapse occurs during the Terminal Phase of Training (TPT) where training error vanishes and the training loss is trained towards zero, it is observed in He & Su () that the trend of Neural Collapse establishes in the early stages of training.
We thus suggest that the clustering behavior of ID features observed in *off-the-shelf* models can reflect a trend of Neural Collapse, corresponding to the within-class variability collapse (NC1).
Such a trend does not necessitate a complete convergence and the prevalence of the trend in practical models is supported by the effectiveness of prior OOD detector which leverages Neural Collapse (Zhang et al., ; Ammar et al., ).
In light of this, we leverage the landscape of Neural Collapse revealed in Theorem 3.1 and (NC2) to examine the geometry of ID feature clusters.

Since ID test samples are drawn from the same distribution as the training samples, we anticipate a similar pattern in their features.
Specifically, we expect ID features to cluster towards the weight vectors of their predicted class during training.
Additionally, we expect ID features to reside near a simplex Equiangular Tight Frame (ETF), thereby acquiring sufficient norm. Conversely, OOD samples are unseen during training and do not undergo the process of iterative adjustment, which drives the Neural Collapse phenomenon. Thus we expect the model to be less effective in aligning the OOD samples with weight vectors, placing OOD further from the weight vectors than ID features.
Meanwhile, we do not expect the model
to effectively align the OOD samples with an ETF.

In Figure 2, we validate our hypothesis across the training process of a CIFAR-10 classifier with ResNet-18 backbone.
In Figure 2, we compute over the ID set (CIFAR-10) and OOD set (SVHN) the average cosine similarity between the centered feature $\bm{h}_{i}-\bm{\mu}_{G}$ and the weight vector $\bm{w}_{c}$ of the predicted class $c$, i.e.,

$$ $Avg_{i}\ \ \frac{(\bm{h}_{i}-\bm{\mu}_{G})\cdot\bm{w}_{c}}{\|\bm{h}_{i}-\bm{ \mu}_{G}\|_{2}\|\bm{w}_{c}\|_{2}}$ (5) $$

We observe that ID features have higher similarity scores and cluster closer to the weight vectors than OOD features. This relative relationship emerges early in training, without requiring full convergence.
We further reinforce our observation in Figure 1 Left where we visualize ID features, OOD features, and weight vectors of a CIFAR-10 classifier with UMAP(McInnes et al., ).
ID features are color-coded to align with the weight vectors (marked by stars) of their predicted classes, revealing a distinct clustering pattern near the weight vectors.
Conversely, OOD features reside further away.
While ID features don’t fully collapse onto weight vectors, showing incomplete Neural Collapse, the emerging trend still holds, and the ID/OOD relationship remains valid.

Additionally, we combine our observation with (Zhu et al., ), showing that the weight vectors form an ETF during training.
Our observed proximity to the weight vectors thus also validates the clustering of ID features near an ETF and the divergence of OOD from this structure.
The lack of structure and incentives to extend in space explains the relatively smaller norm of OOD features.

### 3.3 Out-of-Distribution Detection

Based on our understanding, we design an efficient and versatile OOD detector.
Specifically, we propose to detect OOD based on feature proximity to the weight vectors of the predicted class.
For the proximity metric, we avoid Euclidean-based metrics as they require estimating the scaling factor $\lambda$ in Equation 2.
This estimation tends to be imprecise for general classifiers which may cease training prior to convergence, resulting in suboptimal performance of Euclidean-based metrics shown in Appendix B.
Instead, we design an angle-based metric, adjusted for class-wise difference.
Specifically, we propose to quantify the proximity as the norm of projection of the weight vector $\bm{w}_{c}$ onto the centered feature $\bm{h}-\bm{\mu}_{G}$, where $c$ corresponds to the predicted class, i.e.,

$$ $\mathtt{pScore}=cos(\bm{w}_{c},\bm{h}-\bm{\mu}_{G})\|\bm{w}_{c}\|_{2},$ (6) $$

where $cos(\bm{w}_{c},\bm{h}-\bm{\mu}_{G})=\frac{(\bm{h}-\bm{\mu}_{G})\cdot\bm{w}_{c}
}{\|\bm{h}-\bm{\mu}_{G}\|_{2}\|\bm{w}_{c}\|_{2}}$.
A higher $\mathtt{pScore}$ indicates closer proximity to the weight vector and thus a lower chance of OOD-ness.
Geometrically, thresholding on $\mathtt{pScore}$ selects infinite hyper-cones centered at the weight vectors, as illustrated in Figure 1 Right.
Within the same predicted class, $\mathtt{pScore}$ is proportional to the cosine similarity.
Across different classes, $\mathtt{pScore}$ adapts to class-wise difference by selecting wider hyper-cones for classes with larger weight vectors, which tend to have larger decision regions.
As shown in Appendix B, our $\mathtt{pScore}$ with class-wise adjustment outperforms vanilla cosine similarity.
Notably, our $\mathtt{pScore}$ incorporates class-specific information into characterizing ID clustering by using the weight vectors of the predicted class.
This brings in additional gain in detection effectiveness, as we shall see in Section 4.

While $\mathtt{pScore}$ enhances efficiency and effectiveness, its performance is intrinsically contingent on the strength of ID clustering. Such contingency, widely exhibited by clustering-based methods Lee et al. (); Sun et al. (); Tack et al. (), poses challenges on classifiers with less pronounced ID clustering, such as ImageNet ResNet-50 in Section 4.1.
To mitigate such discrepancy,
we complement $\mathtt{pScore}$ by considering the distance of ID clusters to the origin.
Specifically, we enhance our proximity score by incorporating feature norms to filter out OOD near the origin, as illustrated in Figure 1 *Right*.
Taking $\mathtt{L1}$ norm as an example, we define our detection score as $\mathtt{pScore}+\alpha\|\bm{h}\|_{1}$, where $\alpha$ controls the filtering strength and can be selected from a validation set as detailed in Section 4.
We refer readers to Section 4.3 for the effect of different orders of $p$-norm. Thresholding on the detection score, we have Neural Collapse Inspired OOD Detector ($\mathtt{NCI}$):
A lower score indicates a higher chance of OOD-ness.

$\mathtt{NCI}$ has $O(P)$ complexity, where $P$ is the penultimate layer dimension.
The complexity theoretically ensures computational scalability of $\mathtt{NCI}$ on large models.
Empirically, $\mathtt{NCI}$ maintains inference latency comparable to the vanilla *softmax-confidence* detector, as we shall see in Section 4.

## 4 Experiments

Table: Table 1: $\mathtt{NCI}$ reduces discrepencies and improves overall performance on CIFAR-10 and ImageNet benchmarks with minimal latency. CIFAR-10 uses ResNet-18 and ImageNet uses ResNet-50.

Table: Table 2: $\mathtt{NCI}$ reduces discrepencies and improves overall performance on ImageNet benchmarks across ViT B/16 and Swin v2 classifiers. Bold marks best, underline 2nd

In this section, we extensively evaluate $\mathtt{NCI}$ across classification tasks: CIFAR-10, CIFAR-100 (see App. D), ImageNet, as well as model architectures: ResNet, DenseNet (see App. D), ViT, Swin.
We compare $\mathtt{NCI}$ against *thirteen* baseline methods. While $\mathtt{NCI}$ may not achieve the best performance on individual benchmarks, it mitigates the exisitng generalization discrepancies and achieves the best overall performance with minimal latency.
Following the OpenOOD benchmark Zhang et al. (), we evaluate on *six* OOD sets for CIFAR-10 and CIFAR-100 classifiers and *five* for ImageNet classifiers.
Performance is evaluated using two widely recognized metrics: the False Positive Rate at 95% True Positive Rate (FPR95) and the Area Under the Receiver Operating Characteristic Curve (AUROC).
Lower FPR95 and higher AUROC values indicate better performance.
We also report the per-image inference latency (in milliseconds) evaluated on a Tesla T4 GPU.
In our experiments, other than the ablation study in Section 4.3, we use the $L1$-norm as the filtering term and select the filtering strength $\alpha$ from $\{10^{-4},10^{-3},10^{-2},10^{-1}\}$ based on a validation set generated per pixel from Gaussian $N(0,1)$, following Sun et al. (); Sun & Li ().
For detailed setups, please see Appendix A.
Our method and all baselines are *post-hoc* methods, while all models used are *off-the-shelf* and do *not* require complete Neural Collapse Convergence.

### 4.1 Mitigating Discrepencies across Classification Tasks

We first assess the performance of $\mathtt{NCI}$ and baselines across CIFAR-10 and ImageNet classification tasks.
The two tasks provide an ideal test bed for evaluating versatility, as they drastically differ in input resolution, number of classes, and classification accuracy.
We use ResNets from OpenOOD Zhang et al. (): ResNet-18 for CIFAR-10 (95.06% accuracy) and ResNet-50 for ImageNet (76.65% accuracy).
Based on validation results, we set the filter strength $\alpha$ of the $L1$-norm to $10^{-2}$ for CIFAR-10 experiments and $10^{-3}$ for ImageNet experiments.

Datasets
For CIFAR-10 experiments, We follow the OpenOOD split of ID test set and evaluate on the OpenOOD benchmarks, including CIFAR-100 Krizhevsky et al. (), Tiny ImageNet Le & Yang (), MNIST Deng (), SVHN Netzer et al. (), Texture (Cimpoi et al., ), and Places365 (Zhou et al., ).
For ImageNet experiments, we follow the OpenOOD split of ID test set and evaluate on the OpenOOD benchmarks, including SSB-hard Vaze et al. (), NINCO Bitterwolf et al. (), iNaturalist (Van Horn et al., ), Texture (Cimpoi et al., ), and OpenImage-O Wang et al. ().

Baselines
In Table 1(a), we compare our method with *thirteen* baselines.
Some baselines focus more on the CIFAR-10 Benchmark while others focus more focused on the Imagenet Benchmark.
Therefore, we categorize the baselines, besides the vanilla confidence-based $\mathtt{MSP}$ (Hendrycks & Gimpel, ), into two groups: the “CIFAR-10 Strong” baselines, including $\mathtt{ODIN}$ (Liang et al., ), $\mathtt{Energy}$ (Liu et al., ), $\mathtt{Mahalanobis}$ (Lee et al., ), $\mathtt{KNN}$(Sun et al., ), $\mathtt{ViM}$ (Wang et al., ), and $\mathtt{fDBD}$ Liu & Qin ();
the “ImageNet Strong” baselines, including $\mathtt{GradNorm}$ (Huang et al., ),
$\mathtt{NECO}$ Ammar et al. (),
$\mathtt{React}$ (Sun et al., ), $\mathtt{Dice}$ (Sun & Li, ),
$\mathtt{ASH}$ Djurisic et al. (),
$\mathtt{Scale}$ Xu et al. ().
See Appendix C for details of the baselines.

Performance
Table 1(a) shows that $\mathtt{NCI}$ consistently ranks *top-three* across benchmarks, whereas baselines exhibit greater variability.
To assess overall performance, we averaged AUROC across benchmarks, which are of a similar range.
Table 1(b) highlights that NCI improves *overall* performance compared to strong baselines on individual benchmarks.
Further, $\mathtt{NCI}$ is as efficient as $\mathtt{MSP}$, as shown in Table 1(b)(^1^11Running time of $\mathtt{KNN}$ on ImageNet are copied from Table 4 in Sun et al. ().), which enhances efficiency compared to strong baselines.
This aligns with the analysis in Section 3 and Appendix C.
We highlight the following pairs of comparison:

- •
$\mathtt{NCI}$ v.s. $\mathtt{NCI}$ w/o filter:
On the CIFAR-10 classifier, strong ID clustering allows our method to rank top-3 without filtering.
Conversely, on the ImageNet ResNet-50, weaker ID clustering (see Appendix E) makes norm-based filtering crucial for reducing generalization discrepancy.
Complete Neural Collapse occurs on neither model while $\mathtt{NCI}$ remains effective.
- •
$\mathtt{NCI}$ v.s. $\mathtt{KNN}$: Compared to $\mathtt{KNN}$, $\mathtt{NCI}$ significantly reduces the latency (Table 1(b)).
Notably, without filtering, our hyperparameter-free score outperforms KNN with tuned parameters on most benchmarks (Table 1(a), Table 2(a) & Table 8), highlighting the benefit of using class-specific information.
- •
$\mathtt{NCI}$ v.s. $\mathtt{ASH}$ / $\mathtt{Scale}$:
Compared to both, $\mathtt{NCI}$ delivers competitive performance on ImageNet and *significantly* improves CIFAR-10, enhancing *overall* performance ( Table 1(b)).
Also, $\mathtt{ASH}$ and $\mathtt{Scale}$ introduce in a small delay on the ImageNet benchmark due to activation sorting, with larger activation dimensions likely widening the latency gap on larger models.
- •
$\mathtt{NCI}$ v.s. $\mathtt{NECO}$:
$\mathtt{NECO}$ (Ammar et al., ) is motivated by Neural Collapse.
Like $\mathtt{NCI}$ with filtering, $\mathtt{NECO}$ uses max-logit and incorporates distance to the origin.
However, $\mathtt{NECO}$ exclusively analyzes features, requiring expensive matrix multiplication and leading to higher inference latency (Table 1(b)).
Conversely, $\mathtt{NCI}$ explores the interplay between features *and* the classification head, integrating class-specific information to improve both efficiency and effectiveness.

### 4.2 Mitigating Discrepancies across Architectures

Next, we study two transformer-based models: ViT B/16 Dosovitskiy et al. () and Swin-v2 Liu et al. (), both finetuned on ImageNet, achieving an accuracy of 81.14% and 82.94% respectively.
We follow the setup of the OpenOOD ImageNet Benchmark in Section 4.1.
Based on validation results, we set the filter strength $\alpha$ of the $L1$ norm to $10^{-3}$ for both classifiers.
In Table 2, we observe strong baselines suffer on Swin v2, echoing the observations in Ammar et al. ().
Conversely, our $\mathtt{NCI}$, even without filtering, improves baseline performance on Swin v2.
Filtering further enhances the performance, leading to improved *overall* performance (Table 2(b)).

Table: Table 3: NCI improves the overall performance, averaged across Table 1(a), Table 2(a) & Table 8.

We further aggregate in Table 2 with experiments on ResNet (Table 1) and DenseNet (Table 8) and report the average AUROC in Table 3.
$\mathtt{NCI}$ significantly boosts the overall performance.

### 4.3 Ablation on the Filtering Effect

In Table 4, we assess different orders of $p$-norm as the filtering term, compared to the $L1$ norm used so far.
To ensure a fair comparison, we report
the best performance from the filter strengths $\{10^{-4},10^{-3},10^{-2},10^{-1}\}$.
The rest of the setup follows the ImageNet benchmarks in Section 4.1.
As shown in Table  4, filtering with $L1$ norm achieves the best performance across OOD datasets, aligning with prior observations Huang et al. (); Park et al. ().
Meanwhile, we observe that in rare scenarios, e.g., a ResNet-18 on CIFAR-10, the $\mathtt{L1}$ norm cannot effectively characterize OOD’s proximity to the origin, leading to no extra performance gain compared to simply thresholding on $\mathtt{pScore}$.
In these cases, our algorithm benefits from its ability to automatically select a low filter strength based on validation results, effectively disregarding the filtering term.

Table: Table 4: Ablation on filtering norm on ImageNet OpenOOD Benchmark with ResNet-50 backbone. AUROC score is reported (higher is better). Bold denotes the best result. Filtering with $\mathtt{L1}$ norm outperforms alternative choice of norms across OOD datasets.

We also test the sensitivity of $\mathtt{NCI}$ to filtering strength $\alpha$ in Table add.
As shown on the ImageNet ResNet50 benchmark, performance remains stable for $\alpha$ values within the same scale.
Given this insensitivity, we select hyperparameters from four scales $\{10^{-4},10^{-3},10^{-2},10^{-1}\}$ without extensive finetuning in this work.

Table: Table 5: Sensitivity of $\mathtt{NCI}$ to filtering strength. Average AUROC on ImageNet ResNet-50 Benchmark reported. Performance remains stable within the same scale.

We further apply $\mathtt{L1}$-norm based filtering to $\mathtt{KNN}$ to see if this perspective can mitigate the discrepancy of clustering-based methods in general.
In Table 6 (^2^22 Note that we report our run of KNN here to ensure a fair evaluation of the filtering effect. Our results are very similar to the OpenOOD results reported in Table 1(a), with only marginal differences.), we report the the best performance of $\mathtt{KNN}$ from filter strengths $\{10^{-4},10^{-3},10^{-2},10^{-1}\}$.
We observe a significant performance gain from adding the filter, which further validates our understanding of ID clustering landscape from Neural Collapse.
Note that our method outperforms the standalone $L1$ norm as well as $\mathtt{KNN}$, before and after filtering.

Table: Table 6: Effectiveness of our filtering scheme on $\mathtt{KNN}$. Performance gain validates our understanding of ID clustering landscape. $\mathtt{NCI}$ outperforms $\mathtt{KNN}$ and standalone $L1$ norm. AUROC reported (higher is better). Bold highlights the best result.

## 5 Related Work

OOD Detection
Extensive research has focused on OOD detection algorithms.
One line of work is post-hoc and builds upon pre-trained models.
For example, Hendrycks et al. (); Liang et al. (); Liu et al. (); Sun et al. (); Sun & Li (); Liu & Qin (); Xu et al. () design OOD score over the output space of a classifier.
Meanwhile, Lee et al. () and Sun et al. () measure OOD-ness from the perspective of ID clustering in *feature* space.
Our work extends the observation that ID features tend to cluster from the perspective of Neural Collapse.
While existing work is more focused are certain classification tasks than others, our proposed OOD detector is tested to be highly versatile.

Others (Sharifi et al., ; Patil et al., ; Zhu et al., ) explore the regularization of OOD detection in training.
For example, DeVries & Taylor (); Hsu et al. () propose OOD-specific architecture whereas Huang & Li (); Wei et al. () design OOD-specific training loss. In particular, Tack et al. () brings attention to representation learning for OOD detection and proposes an OOD-specific contrastive learning scheme.
Our work does not belong to this school of thought and is not restricted to specific training schemes or architecture.

Neural Collapse
Neural Collapse was first observed in Papyan et al. ().
During Neural Collapse, the penultimate layer features collapse to class means, the class means and the classifier collapses to a simplex equiangular tight framework, and the classifier simplifies to adopt the nearest class-mean decision rule.
Further work provides theoretical justification for the emergence of Neural Collapse (Han et al., ; Mixon et al., ; Zhou et al., ; Zhu et al., ).
In addition, Zhu et al. () derives an efficient training algorithm drawing inspiration from Neural Collapse.
Our concurrent work Ammar et al. () also leverages insights from Neural Collapse for OOD detection.
However, they tackle from the subspace perspective and largely overlook class-specific information revealed by Neural Collapse, which is essential for our work.

## 6 Conclusion

This work leverages insights from Neural Collapse to propose a novel OOD detector.
Specifically, we study the phenomenon that ID features tend to form clusters whereas OOD features reside far away.
Inspired by the trend of Neural Collapse prevalent on practical models, we hypothesize and validate that ID features tend to cluster near weight vectors.
We also explain why ID features tend to reside further from the origin and complement our method from this perspective.
Experiments show the effectiveness of our method on practical models without requiring the complete convergence of Neural Collapse.
Further, our method improves the overall performance with minimal latency across diverse benchmarks.
We hope our work can inspire future work to explore the interplay between features and weight vectors for OOD detection and other research problems such as calibration and adversarial robustness.

## Appendix A Implementation Details

### A.1 CIFAR-10

ResNet-18
For visualization in Fig. 1Left, Middle, we use a CIFAR-10 classifier of ResNet-18 backbone trained with cross-entropy loss.
The classifier is trained for 100 epochs, with the initial learning rate 0.1 decaying to 0.01, 0.001, and 0.0001 at epochs 50, 75, and 90 respectively.
For experiments in Table 1(a), we use the pre-trained model provided by the OpenOOD benchmark.
And we refer readers to Zhang et al. () for their training recipe.

DenseNet-101
For experiments on CIFAR-10 Benchmark presented in Table 8, we evaluate a CIFAR-10 classifier of DenseNet-101 backbone.
The classifier is trained following the setups in Huang et al. () with depth $L=100$ and growth rate $k=12$.

### A.2 CIFAR-100

DenseNet-101
For experiments on the CIFAR-100 Benchmark presented in Table 8, we evaluate a CIFAR-100 classifier of the DenseNet-101 backbone.
The classifier is trained following the setups in Huang et al. () with depth $L=100$ and growth rate $k=12$.

### A.3 ImageNet

ResNet-50
For evaluation on ImageNet Benchmark in Table 1(a), we use the default ResNet-50 model trained with cross-entropy loss provided by Pytorch.
Training recipe can be found at
[https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/](https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/)

ViT B/16
In Table 2, we use the PyTorch implementation and pre-trained checkpoint of ViT B/16, available [https://github.com/lukemelas/PyTorch-Pretrained-ViT/tree/master](https://github.com/lukemelas/PyTorch-Pretrained-ViT/tree/master).

Swin v2
In Table 2, we use the $\mathtt{timm}$ Wightman () implementation of Swin v2 as well as their pre-trained checkpoint ’swinv2_base_window8_256’.

## Appendix B Alternatives Proximity Metrics

In this section, we validate that under alternative similarity metrics, ID features also reside closer to weight vectors and empirically compare the metrics.
In addition to our proposed $\mathtt{pScore}$, we consider two standard similarity metrics, cosine similarity and Euclidean distance.
For cosine similarity, we evaluate

$$ $\mathtt{cosScore}=\frac{(\bm{h}-\bm{\mu}_{G})\cdot\bm{w}_{c}}{\|\bm{h}-\bm{\mu }_{G}\|_{2}\|\bm{w}_{c}\|_{2}}.$ (7) $$

As for Euclidean distance, we first estimate the scaling factor in Theorem 3.1 by
$\displaystyle\tilde{\lambda}_{c}=\frac{\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2}}{\|
\bm{w}_{c}\|_{2}}.$
Based on the estimation, we measure the distance between the centered feature $\bm{h}-\bm{\mu}_{G}$ and the scaled weight vector corresponding to the predicted class $c$ as

$$ $\mathtt{distScore}=-\|(\bm{h}-\bm{\mu}_{G})-\tilde{\lambda}_{c}\bm{w}_{c}\|_{2 }.\vspace{-2mm}$ (8) $$

Same as $\mathtt{pScore}$, the larger $\mathtt{cosScore}$ or $\mathtt{distScore}$ is, the closer the feature is to the weight vector.

We evaluate in Table 7 OOD detection performance using standalone $\mathtt{pScore}$, $\mathtt{cosScore}$, and $\mathtt{distScore}$ as scoring function respectively.
The experiments are evaluated with AUROC under the same ImageNet setup as in Section 4.1.
We observe in Table 7, that across OOD datasets, all three scores achieve an AUROC score $>50$, indicating that ID features reside closer to weight vectors compared to OOD under either metric.

Furthermore, we observe that $\mathtt{pScore}$ outperforms both $\mathtt{cosScore}$ and $\mathtt{distScore}$.
Comparing the performance of $\mathtt{pScore}$ and $\mathtt{cosScore}$, the superior performance of $\mathtt{pScore}$ implies that ID features corresponding to the classes with larger $\bm{w}_{c}$ are less compact.
This is in line with the decision rule of the classifier that classes with larger $\bm{w}_{c}$ have larger decision regions.
As for comparison against Euclidean distance based $\mathtt{distScore}$, $\mathtt{pScore}$ eliminates the need to estimate the scaling factor, which can be error-prone before convergence, potentially leading to performance degradation.

Table: Table 7: Ablation on proximity scores. AUROC score is reported (higher is better). ID features are closer to weight vectors than OOD features (AUROC $>$ 50) under all metrics. Across OOD datasets, our proposed $\mathtt{pScore}$ can better separate ID an OOD features than $\mathtt{distScore}$ and $\mathtt{cosScore}$.

## Appendix C Baseline Methods

We provide an overview of our baseline methods in this session.
We follow our notation in Section 3.
In the following, a lower detection score indicates OOD-ness.

MSP
Hendrycks & Gimpel () proposes to detect OOD based on the maximum softmax probability.
Given the penultimate feature $\bm{h}$ for a given test sample $\bm{x}$, the detection score of MSP can be represented as:

$$ $\frac{\exp{(\bm{w}_{c}^{T}\bm{h}}+b_{c})}{\sum_{c^{\prime}\in\mathcal{C}}\exp{ (\bm{w}_{c^{\prime}}^{T}\bm{h}}+b_{c^{\prime}})},$ (9) $$

where $c$ is the predicted class for $\bm{x}$.

ODIN
Liang et al. () proposes to amplify ID the OOD separation on top of MSP through temperature scaling and adversarial perturbation.
Given a sample $\bm{x}$, ODIN constructs a noisy sample $\bm{x^{\prime}}$ from $\bm{x}$.
Denote the penultimate feature of the noisy sample $\bm{x^{\prime}}$ as $\bm{h^{\prime}}$, ODIN assigns OOD score following:

$$ $\frac{\exp{((\bm{w}_{c}^{T}\bm{h^{\prime}}}+b_{c})/T)}{\sum_{c^{\prime}\in \mathcal{C}}\exp{((\bm{w}_{c}^{\prime T}\bm{h^{\prime}}}+b_{c^{\prime}})/T)},$ (10) $$

where $c$ is the predicted class for the perturbed sample and $T$ is the temperature.
In our implementation, we set the noise magnitude as 0.0014 and the temperature as 1000.

Energy
Liu et al. () designs an energy-based score function over the logit output.
Given a test sample $\bm{x}$ as well as its penultimate layer feature $\bm{h}$, the energy based detection score can be represented as:

$$ $-\log\sum_{c^{\prime}\in\mathcal{C}}\exp{(\bm{w}_{c^{\prime}}^{T}\bm{h}}+b_{c^ {\prime}}).$ (11) $$

ReAct
Sun et al. () builds upon the energy score proposed in Liu et al. () and regularizes the score by truncating the penultimate layer estimation.
We set the truncation threshold at $90$ percentile in our experiments.

Dice
Sun & Li () builds upon the energy score proposed in Liu et al. ().
Leveraging the observation that units and weights are used sparsely in ID inference, Sun & Li () proposes to select and compute the energy score over a selected subset of weights based on their importance.
We set a threshold at $90$ percentile for CIFAR experiments and $70$ percentile for ImageNet experiments following Sun & Li ().

ASH Djurisic et al. () builds upon the energy score proposed in Liu et al. ().
Prior to the Energy score, ASH sorts each feature to find the top-k elements, scales up the top-k elements, and sets the rest to zero.
We note that in addition to the cost of Energy, ASH introduces a sorting cost of $O(P\log k)$, where $P$ is the penultimate layer dimension.

Scale Xu et al. () builds upon the energy score proposed in Liu et al. ().
Prior to the Energy score, Scale sorts each feature to find the top-k elements and based on the statistics, scales all elements in the feature.
We note that in addition to the cost of Energy, Scale also introduces a sorting cost of $O(P\log k)$, where $P$ is the penultimate layer dimension.

Mahalanobis
On the feature space, Lee et al. () models the ID feature distribution as multivariate Gaussian and designs a Mahalanobis distance-based score:

$$ $\max_{c}-(\bm{e_{x}}-\hat{\bm{\mu}_{c}})^{T}\hat{\Sigma}^{-1}(\bm{e_{x}}-\hat{ \bm{\mu}_{c}}),$ (12) $$

where $\bm{e_{x}}$ is the feature embedding of $\bm{x}$ in a specific layer, $\hat{\mu_{c}}$ is the feature mean for class $c$ estimated on the training set, and $\hat{\Sigma}$ is the covariance matrix estimated over all classes on the training set.

On top of the basic score, Lee et al. () also proposes two techniques to enhance the OOD detection performance.
The first is to inject noise into samples.
The second is to learn a logistic regressor to combine scores across layers.
We tune the noise magnitude and learn the logistic regressor on an adversarial constructed OOD dataset.
The selected noise magnitude is 0.005 in both our ResNet and DenseNet experiments.

KNN Chen et al. () proposes to detect OOD based on the k-th nearest neighbor distance between the normalized embedding of the test sample $\bm{z_{x}}/|\bm{z_{x}}|$ and the normalized training embeddings on the penultimate space.
Chen et al. () also observes that contrastive learning helps in improving OOD detection effectiveness.

GradNorm
Huang et al. () extracts information from the gradient space to detect OOD samples.
Specifically, Huang et al. () defines the OOD score function as the $\mathtt{L1}$ norm of the gradient of the weight matrix with respect to the KL divergence between the softmax prediction for $\bm{x}$ and the uniform distribution.

$$ $\|\frac{\partial D_{KL}(\bm{u}\|softmax{f(\bm{x})})}{\partial\bm{W}}\|_{1}.$ (13) $$

ViM
Wang et al. () proposes to integrate class-specific information into feature space information by adding energy score to the feature norm in the residual space of the training feature matrix.
The detection score is designed to be:

$$ $\alpha\sqrt{\bm{h}^{T}\bm{R}\bm{R}\bm{h}},$ (14) $$

where $\bm{R}\in R^{P\times(P-D)}$ correspond to the residual after subtracting the $D-$dimensional principle space.
In the preparation stage, ViM requires evaluating the residual/null space from the training data, which is computationally expensive given the data volume.
During inference, large matrix multiplication is required, resulting in a computational complexity of $O((P-D)^{2})$.

NECO is inspired by the ETF structure of Neural Collapse to utilize feature subspace for OOD detection.
The detection score is designed to be

$$ $\text{MaxLogit}\times\frac{\sqrt{\bm{h}^{T}\bm{P}\bm{P}\bm{h}}}{\sqrt{\bm{h}^{ T}\bm{h}}},$ (15) $$

where $\bm{P}\in R^{P\times d}$ correspond to the $d-$dimensional principle space.
In the preparation stage, NECO requires evaluating the residual/null space from the training data, which is computationally expensive given the data volume.
During inference, large matrix multiplication is required, resulting in a computational complexity of $O((d)^{2}+P)$.

fDBD
Liu & Qin () proposes to detect OOD based on estimated feature distance to decision boundaries of class $c\in\mathcal{C}$ besides its predicted class $f(\bm{x})$:

$$ $\tilde{D}_{f}(\bm{h},c)=\frac{|(\bm{w}_{f(\bm{x})}-\bm{w}_{c})^{T}\bm{h}+(b_{f (\bm{x})}-b_{c})|}{\left\lVert\bm{w}_{f(\bm{x})}-\bm{w}_{c}\right\rVert_{2}}, \vspace{-3mm}$ (16) $$

The detection score is designed as

$$ $\frac{1}{|\mathcal{C}|-1}\sum_{\begin{subarray}{c}c\in\mathcal{C}\end{subarray },\ c\neq f(\bm{x})}\frac{\tilde{D}_{f}(\bm{h},c)}{\|\bm{h}-\bm{\mu}_{train}\| _{2}}.$ (17) $$

fDBD has time complexity $O(|\mathcal{C}|+P)$, where $|\mathcal{C}|$ is the number of training classes and $P$ is the penultimate layer dimension.

## Appendix D Evaluation on DenseNet

In addition to evaluation on ResNet and transformer-based model in Section 4, we report the performance of our $\mathtt{NCI}$ along with the baselines under AUROC and FPR95 across OpenOOD benchmarks in Table 8.

Table: Table 8: Our OOD detectors achieves high AUROC and low FPR95 across CIFAR-10 and CIFAR-100 OOD benchmark on DenseNet. $\uparrow$ indicates that larger values are better and vice versa. Bold highlight the best results and underline denotes the 2nd and 3rd best results. We note that for DenseNet CIFAR-10 and CIFAR-100 classifiers, the discrepancy among existing methods is not as severe as in the examples presented in the main paper. Nevertheless, our $\mathtt{NCI}$ achieves state-of-the-art performance or improves upon existing methods, enhancing overall performance on average.

## Appendix E The Prevalence of Neural Collapse across Canonical Classification Tasks

The phenomenon of Neural Collapse, as established in the seminal work by Papyan et al. Papyan et al. () and corroborated by subsequent studies Han et al. (); Mixon et al. (); Zhou et al. (); Zhu et al. (), widely exists across canonical classification datasets and model architectures.
The prevalent occurrence of Neural Collapse forms a robust foundation for the design of our versatile OOD detectors.
To this end, we review the empirical evidence of Neural Collapse across different datasets and model architectures in Figure 3, Figure 4, Figure 5,
Figure 6,
and Figure 7.
Comparing CIFAR-10 and ImageNet behaviors with ResNet backbone in Figure 7, we note that the clustering of CIFAR-10 is more prominent than Imagenet, as indicated by a higher ratio of between-class variance to within-class covariance.
Note that the figures and captions are sourced from Papyan et al. ().
The definition and notation follow Section 3.

Figure: Figure 3: (ref. Figure 2 in Papyan et al. ()) Train class means become equinorm. In each array cell, the vertical axis shows the coefficient of variation of the centered class-mean norms as well as the network classifiers norms. In particular, the blue lines show $\text{Std}_{c}(\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2})/\text{Avg}(\|\bm{\mu}-\bm{ \mu}_{G}\|_{2})$ where $\{\bm{\mu}_{c}\}$ are the class means of the last-layer activations of the training data and $\bm{\mu}_{G}$ is the corresponding train global mean; the orange lines show$\text{Std}_{c}(\|\bm{w}_{c}\|_{2})/\text{Avg}(\|\bm{w}_{c}\|_{2})$ where $\{\bm{w}_{c}\}$ is the last-layer classifier of the $c$ th class. As training progresses, the coefficients of variation of both class means and classifiers decrease.
Refer to caption: extracted/5924189/nc_fig02.jpg

Figure: Figure 4: (ref. Figure 3 in Papyan et al. ()) Classifiers and train class means approach equiangularity. In each array cell, the vertical axis shows the SD of the cosines between pairs of centered class means and classifiers across all distinct pairs of classes $c$ and $c^{\prime}$. Mathematically, denote $\cos_{\mu}(c,c^{\prime})=<\bm{\mu}_{c}-\bm{\mu}_{G},\bm{\mu}_{c}^{\prime}-\bm{ \mu}_{G}>/\|\bm{\mu}_{c}-\bm{\mu}_{G}\|_{2}\|\bm{\mu}_{c}^{\prime}-\bm{\mu}_{G }\|_{2}$ and $\cos_{w}(c,c^{\prime})=<\bm{w}_{c},\bm{w}_{c}^{\prime}>/\|\bm{w}_{c}\|_{2}\| \bm{w}_{c}^{\prime}\|_{2}$, where $\{\bm{w}_{c}\}_{c=1}^{C},\{\bm{\mu}_{c}\}_{c=1}^{C}$, and $\bm{\mu}_{G}$ are as in Figure 3. We measure $\text{Std}_{c,c^{\prime}}(\cos_{\mu}(c,c^{\prime}))$ (orange) and $\text{Std}_{c,c^{\prime}}(\cos_{w}(c,c^{\prime}))$. As training progresses, the SDs of the cosines approach zero, indicating equiangularity.
Refer to caption: extracted/5924189/nc_fig03.jpg

Figure: Figure 5: (ref. Figure 4 in Papyan et al. ()) Classifiers and train class means approach maximal-angle equiangularity. We plot in the vertical axis of each cell the quantities $\text{Avg}_{c,c^{\prime}}|\cos_{\mu}(c,c^{\prime})+1/(C-1)|$ (blue) and $\text{Avg}_{c,c^{\prime}}|\cos_{w}(c,c^{\prime})+1/(C-1)|$ (orange), where $\cos_{\mu}(c,c^{\prime})$ and $\cos_{w}(c,c^{\prime})$ are as in Figure 4. As training progresses, the convergence of these values to zero implies that all cosines converge to $-1/(C-1)$. This corresponds to the maximum separation possible for globally centered, equiangular vectors.
Refer to caption: extracted/5924189/nc_fig04.jpg

Figure: Figure 6: (ref. Figure 5 in Papyan et al. ()) Classifier converges to train class means. The formatting and technical details are as described in Section 3. In the vertical axis of each cell, we measure the distance between the classifiers and the centered class means, both rescaled to unit norm. Mathematically, denote $\tilde{\bm{M}}=\dot{\bm{M}}/\|\dot{\bm{M}}\|_{F}$ where $\dot{\bm{M}}=[\bm{\mu}_{c}-\bm{\mu}_{G},c=1,....,C]\in\mathtt{R}^{P\times C}$ is the matrix whose columns consist of the centered train class means; denote $\tilde{\bm{W}}=\bm{W}/\|\bm{W}\|_{F}$ where $\bm{W}\in\mathtt{R}^{C\times P}$ is the last-layer classifier of the network. We plot the quantity $\|\tilde{\bm{W}}^{T}-\tilde{\bm{M}}\|_{F}^{2}$ on the vertical axis. This value decreases as a function of training, indicating that the network classifier and the centered-means matrices become proportional to each other (self-duality).
Refer to caption: extracted/5924189/nc_fig05.jpg

Figure: Figure 7: (ref. Figure 6 in Papyan et al. ()) Training within-class variation collapses. In each array cell, the vertical axis (log scaled) shows the magnitude of the between-class covariance compared with the within-class covariance of the train activations. Mathematically, this is represented by $\text{Tr}(\Sigma_{W}\Sigma_{B}^{+}/C)$ where $\text{Tr}(\cdot)$ s the trace operator, $\Sigma_{W}$ is the within-class covariance of the last-layer activations of the training data, $\Sigma_{B}$ is the corresponding between-class covariance, $C$ is the total number of classes, and $[\cdot]^{+}$ is Moore–Penrose pseudoinverse. This value decreases as a function of training—indicating collapse of within-class variation.
Refer to caption: extracted/5924189/nc_fig06.jpg

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: 1. The paper's strength lies in its extensive experimentation across different network backbones, including ResNet, ViT, and Swin. This variety showcases the versatility of the NCI detector and its robustness in various model architectures.

2. The NCI detector is noted for its efficiency, with the paper claiming it matches the latency of a vanilla softmax-confidence detector. This is a significant advantage for real-world deployment where computational resources and time are critical.

3. The paper provides a view of OOD detection by connecting and completing prior methods under the lens of Neural Collapse, offering a unified perspective on a problem often tackled in isolation.
summary: The paper presents an approach to Out-of-Distribution (OOD) detection in machine learning models, leveraging the concept of Neural Collapse. The authors argue that In-Distribution (ID) samples' features cluster closer to the weight vectors of their predicted classes compared to OOD samples. They propose a detector that uses feature proximity to weight vectors and feature norms to filter OOD samples. The method is evaluated on various classification tasks and model architectures, demonstrating efficiency and effectiveness without requiring complete Neural Collapse convergence.
weaknesses: 1. The observation that ID samples form clusters near the weight vectors while OOD samples reside separately is not new but considered basic knowledge in the field. The paper does not offer substantial novelty in this aspect, which may reduce its impact. 

2. Besides, the conclusion drawn from Figure 2 that "early low cosine similarity equals no need for Neural Collapse convergence" is weak. The logical connection between early training observations and Neural Collapse is not well-established, and this represents a significant weakness in the paper's argumentation.

3. The paper could benefit from a more rigorous theoretical foundation to support its claims, especially concerning the relationship between the geometric structure of ID samples and **OOD** samples.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: Please refer to the weakness.
soundness: 3
strengths: 1. The motivation for designing an OOD detector from the perspective of neural collapse is technically sound. The proposed method, which leverages feature norms to filter OOD samples, is both simple and effective.

2. The paper is well-written, clear, and easy to follow.

3. The authors present empirical evidence showing that their straightforward method performs comparably to existing state-of-the-art OOD detection techniques across various benchmarks.
summary: The paper focuses on Out-of-Distribution (OOD) detection, which is important for the safe deployment of AI models. The authors propose a OOD detector inspired by the concept of Neural Collapse. They identify that in-distribution (ID) features cluster closer to their corresponding weight vectors, while OOD features are more dispersed. The proposed OOD detector leverages these insights by focusing on the proximity of features to weight vectors and utilizing feature norms for filtering OOD samples. Extensive experiments demonstrate the detector's efficiency and effectiveness, highlighting its ability to reduce generalization discrepancies while matching the latency of conventional score-based OOD detectors.
weaknesses: 1. The observation that ID samples consistently cluster closer to the weight vectors than OOD samples is well-established in the literature [1,2,3,4] and is referred to as the familiarity hypothesis [5]. Designing OOD detectors based on feature norms has also been thoroughly explored [2,3,4]. As a result, the novelty of this paper appears to be marginal and limited.

2. The neural collapse view has also been explored for OOD detection in [6], and the comparison between this paper and those high related work [1-6] is missing.

3. It has been revealed that existing OOD detection methods is harmful for detecting other prediction failures of a model like misclassified ID data upon real-world application [7, 8, 9], and the simple softmax response baseline as the overall best performing method. Thus, the verification and discussion of the proposed method on detecting misclassified ID data is important.

Reference

[1] CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances. NeurIPS 2020.

[2] Understanding the Feature Norm for Out-of-Distribution Detection. ICCV 2023.

[3] Block Selection Method for Using Feature Norm in Out-of-distribution Detection. CVPR 2023.

[4] Feature Space Singularity for Out-of-Distribution Detection. Arxiv 2020.

[5] The familiarity hypothesis: Explaining the behavior of deep open set methods. Pattern Recognition 2022.

[6] NECO: NEural Collapse Based Out-of-distribution detection. ICLR 2024.

[7] ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms. ICLR 2024.

[8] OpenMix: Exploring Outlier Samples for Misclassification Detection. CVPR 2023.

[9] A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification. ICLR 2023.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: How do the authors choose “CIFAR-10 Strong”/ “ImageNet Strong”? They are the top methods (e.g., top-5) in OpenOOD? Why weren’t other methods in OpenOOD included for comparison?
soundness: 2
strengths: 1. The motivation is sound. The perspective of neural collapse is novel.
2. The proposed NCI method is both simple and efficient for OOD detection. 
3. The authors conduct experiments on multiple architectures (including ResNet-18, ResNet-50, Vit B/16, and Swin v2), showing the effectiveness of NCI.
summary: This paper investigates OOD detection problem through the lens of neural collapse. The authors reveal that ID features would converge to class vectors during training, while OOD features tend to cluster near the origin. Based on this observation, they introduce the NCI method, which quantifies the proximity between feature vectors and class vectors. Experiments conducted on the ImageNet and CIFAR-10 benchmarks further validate the effectiveness of NCI.
weaknesses: 1. Although NCI ranks in the top three on two benchmarks, the improvements over existing SoTA methods are marginal. Previous leading methods on single benchmarks exhibit significantly larger gains compared to NCI. This raises questions about the effectiveness.

2. While the introduction of a filtering score to enhance the separation between ID and OOD samples is reasonable, it primarily serves as a mitigation strategy for misclassifying OOD samples. A more thorough analysis of success and failure cases using NCI should be warranted, as many models may not be well-trained and could violate the neural convergence law.

3. The observation that ID features exhibit greater similarity to class vectors than OOD features is not surprising, given that OOD samples often originate from unknown classes. Thus, analyzing the effectiveness of NCI in scenarios where ID and OOD classes are similar or identical is essential. Additionally, the authors should clearly explain the distinctions between the ID and OOD data used in experiments.

4. Some claims made in the paper are overly broad. For instance, the statement, "Through the lens of Neural Collapse, we explain, connect, and complete prior methods under a holistic view" in Introduction, requires more substantiation. Given the current scope of the method discussion, it seems not possible to extend the neural collapse theory to most prior methods.

5. The organization of this paper should be improved. The introduction appears somewhat redundant with Section 3. Is it necessary to discuss the question, “Where do features of ID samples form clusters?” in both sections?

6. It would be beneficial for the authors to explore the relationship between Neuron Collapse and NCI performance, particularly in light of existing metrics for collapse quantification.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See most of the questions on the weakness.
soundness: 3
strengths: - The method seems effective for OOD detection and is lightweight.
- Although already explored in previous works, the theoretical inspiration from Neural collapse is interesting.
summary: The authors propose an OOD scoring function, "NCI," inspired by the concept of neural collapse. Neural collapse is a phenomenon observed toward the end of training, primarily in the penultimate layer of DNNs, where features of in-distribution (ID) data align closely with the network's weights. Building on previous applications of neural collapse in OOD detection, the authors introduce a metric called pscore, defined as the angle between the ID feature and the network weights, scaled by the norm of the weights. Given that pscore is closely related to the clustering capacity of the DNN, they augment it with an additional term, the norm of the penultimate feature representation, to develop NCI as a robust OOD score.
weaknesses: **1/ Proof Issue in Theorem 3.1**
There may be a minor issue in the proof of Theorem 3.1 related to the alignment between class clusters and the weight vector. Specifically, it seems wrong  that \Sigma_{W} ->0 necessarily implies  (h_{i,c} - \mu_c)(h_{i,c} - \mu_c)^T \to 0(hi,c​−μc​)(hi,c​−μc​)T->0. Revisiting the proof could clarify this aspect.

**2/ UMAP Visualizations**
While UMAP provides visually appealing feature visualizations, relying on it for interpretive arguments is limited because UMAP is non-linear and does not preserve global structure or accurately reflect true density and cluster separation. To strengthen this analysis, could the authors replicate the visualizations using PCA? Additionally, it would be valuable to apply this analysis across a diverse set of DNNs and ID vs. OOD datasets, such as:
- ResNet-18: CIFAR-10 vs. CIFAR-100; ImageNet vs. ImageNet-O, Textures, iNaturalist, SUN, and Places365
- ResNet-50: CIFAR-10 vs. SVHN, CIFAR-100; ImageNet vs. ImageNet-O, Textures, iNaturalist, SUN, and Places365
- ViT: CIFAR-10 vs. SVHN, CIFAR-100; ImageNet vs. ImageNet-O, Textures, iNaturalist, SUN, and Places365

 Q/ Could the authors explain why they chose UMAP over other visualization techniques, and whether they considered alternatives like PCA.?

**3/ Hyperparameter Transparency**
For reproducibility, it would be helpful if the authors provided the hyperparameter choices for all comparison methods to ensure transparency.

Q/ could the authors include a table in the appendix and in the answer listing all the hyperparameters for each method (also their model)?

**4/ Further Collapse for Improved NCI Performance**
If NCI leverages neural collapse (NC) to detect OODs, would further convergence to NC enhance its performance? Specifically, would training the model further to achieve stronger NC improve the efficacy of NCI?

Q/ could the authors save the pscore and NCI and the NC values (1 to 4) of a DNN during the training to see how it behave for most of the DNNs in 2/ for a long training.

**5/ Transformer Model Results**
The results for transformer models are incomplete, with some methods missing. Is there a specific reason for this? A complete table with transformer model results should be included.

**6/ OpenOOD Benchmark and Covariate Shift**
The OpenOOD benchmark includes additional datasets that assess covariate shifts. Could the authors explain why these experiments were excluded? It would be insightful to test NCI’s performance under covariate shift settings and include additional results for this scenario.