GradPCA: Leveraging NTK Alignment for Reliable Out-of-Distribution Detection
Authors: Mariia Seleznova
Hung-Hsu Chou
Claudio Mayrink Verdun
Gitta Kutyniok
Venue: ICLR 2026 Poster (2026)
Keywords: Out-of-Distribution (OOD) detection
Neural Tangent Kernel (NTK)
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: GradPCA is a spectral OOD detector based on NTK alignment

Abstract: We introduce GradPCA, an Out-of-Distribution (OOD) detection method that exploits the low-rank structure of neural network gradients induced by Neural Tangent Kernel (NTK) alignment. GradPCA applies Principal Component Analysis (PCA) to gradient class-means, achieving more consistent performance than existing methods across standard image classification benchmarks. We provide a theoretical perspective on spectral OOD detection in neural networks to support GradPCA, highlighting feature-space properties that enable effective detection and naturally emerge from NTK alignment. Our analysis further reveals that feature quality—particularly the use of pretrained versus non-pretrained representations—plays a crucial role in determining which detectors will succeed. Extensive experiments validate the strong performance of GradPCA, and our theoretical framework offers guidance for designing more principled spectral OOD detectors.

Published as a conference paper at ICLR 2026
GRADPCA: LEVERAGING NTK ALIGNMENT FOR RELIABLE
DETECTION
Mariia Seleznova' Hung-Hsu Chou²
Claudio Mayrink Verdun
Gitta Kutyniok' 1,45,66 14,556
Harvard University 4 University of Tromso DD-R-Gemman Aerospace Center Munich Center for Machine Learning
Buudwig-Maximilinn Universität München
University of Pittsburgh
ABSTRACT
We introduce GradPCA, an exploits the low-rank structure of neural network gradients induced by Neural Tangent Kernel (NTK) alignment. GradPCA applies Principal Component Analy- achieving more consistent performance than existing methods across standard image classification benchmarks. We provide a theoretical perspective on spectral OOD detection neural networks to support GradPCA, highlighting feature-space properties that enable effective detection and naturally emerge from NTK alignment. Our analysis further reveals that feature quality-particularty the use of pretrained versus non-pretrained representa- tions-plays crucial role in determining which detectors will succeed. Extensive experiments validate the strong performance of GradPCA, and our theoretical framework offers guidance for designing more principled spectral OOD detectors.
(OOD) detection method that
sis (PCA) to gradient
INTRODUCTION
In modern deep learning. models often produce confident but incorrect predictions when presented with inputs outside their training distribution (Goodfellow et al.) 2015 Kurakin et al., 2017 Nguyen jet al. 2015). coatr--Distribution (OOD) detection provides a mechanism for models to know when they don't i know, enabling them to reject inputs beyond their domain of competence. As deep learning systems become integrated into critical decision-making processes, robust OOD detection is essential for ensuring safety and effective human oversight (Amodei et al. 2016).
Although designed to improve the reliability of deep learning, OOD detection methods have them- selves proven unreliable (Tajwar al. 2021 Szyc al. 2023). Their performance often hinges subtle assumptions about the model, data, and training procedure, with little guidance as to when these assumptions hold. As result, OOD detection effectiveness is assessed through empirical validation alone, making the performance difficult to predict and dependent on ad hoc tuning. In response to these challenges, we propose new OOD detector that combines classical spectral analysis with recent insights into deep learning theory. offering principled and interpretable approach.
Our main contributions are as follows:
GradPCA: Our method leverages the low-rank structure of gradients in well-trained neural
networks (NNs), induced by the Neural Tangent Kernel (NTK) alignment phenomenon (Atanasov et 2022, Baratin et al. 2021 Seleznova & Kutyniok 2022] Shan & Bordelon 2021). Under NTK alignment, gradients of in-distribution (ID) inputs concentrate within stable, low-dimensional subspaces spanned by class-specific directions. GradPCA performs PCA gradient class-means to efficiently model this subspace and flag inputs whose gradients fall outside it. GradPCA is the first OOD detector to exploit NTK alignment, and principled design ensures robust performance across realistic detection scenarios.
Revisiting spectral OOD detection: To explain GradPCA's empirical success, we present theoretical framework for spectral OOD detection neural networks (NNs). Our framework extends the principles of classical and kemnel PCA (Hotelling) 1933) Scchlkopf et al.) 1998) and allows to derive one-sided, per-sample OOD certificates for spectral detectors theoretical guarantee rarely found in the predominantly empirical OOD detection literature.
"Correspondence Mariia Seleznova (sel ez@math. de).

---

Published as a conference paper at ICLR 2026
Feature quality matters for OOD detection: We show that the performance of OOD detectors critically depends on the quality of feature representations specifically. whether they come from general purpose (pretrained) task-specific (non-pretrained) models. To demonstrate this, we evaluate existing OOI) detectors on a carefully designed benchmark comprising pretrained and non-pretrained models with matched ID accuracy. Notably, methods that detect abnormal model behavior (e.g., confidence-based scores) often worsen with improved feature quality, while approaches that exploit geometric regularities in feature space improve. This underexplored factor helps reconcile inconsistencies in prior works and highlights the need account for feature quality when designing future OOD detectors.
Empirical validation and open-source implementation: We evaluate GradPCA exclu- sively on publicly available models and community-released datasets, avoiding manual subset selection or ad hoc model training. This principled setup mitigates common evaluation biases and supports fair comparison across diverse settings. We benchmark against widely used competitive baselines spanning diverse methodological approaches. including recent gradient- based detectors, and find that GradPCA delivers the most consistent performance, achieving near state-of-the afrt results across all benchmarks. All code and experimental configurations are available our GitHub repository
Notation and outline. We denote vectors and matrices by bold lowercase and uppercase letters, respectively (e.g., for a vector and F for matrix). Bold numerals such as 0 and represent vectors of zeros and ones. The indicator function (z) equals x E A and 0 otherwise. We denote the in- distribution and out-distribution probability density functions by dpaa and dpoood. with corresponding measures Hid and Paud- The support of (Ma is defined as supp( = {x ddsa(z) > 0}.
We present the OOD detection problem setup in Section 2
GradPCA and key empirical insights in
Section 3 theory in Section 4 experimental setup in Section 5 and conclude with Section 6
2 OOD DETECTION: PROBLEM SETUP AND METHODS
Before presenting our method, we formulate the OOD detection problem and introduce our catego- rization of existing OOD detection approaches.
Problem formulation. The goal of OOD detection to construct a detector D + {0, 1} that
returns (zz) = if ar ~ Pid and (xx) = 1 if x Pood. for arbitrary Food- However, this problem is fundamentally ill-posed: no detector is universally optimal across all possible Pood (see Appendix [B To make the problem well-posed we reformulate it as support recovery: identifying the support of Hid, denoted supp(Hia). In this setting, the optimal detector is D := supp())())) which flags any input outside the support of Usa OOD. We adopt this support-based perspective throughout and refer to any x & supp()aa) as guaranteed OOD RODD-independem of Hood- This view aligns with practice: an input is effectively ID only if the model can correctly classify it, reinforcing that OOD detection should focus on the boundaries of reliable model behavior rather than distributional likelihoods.
Taxonomy of OOD detectors. The vast body of work on OOD detection makes it challenging to precisely categorize existing approaches. However, for the purposes of this paper, we find helpful to broadly classify existing methods into two categories based on their main assumptions:
These methods assume that ID data occupies a compact or structured region in featuue space. A sample is flagged OOD if it deviates significantly from this "regular" region. Classical approaches such as density estimation and (kernel) PCA fall into this category, do neural OOD detection methods such as Mahalanobis (Lee) et al. 2018) and KNN (Sun et al. 2022b). Our method, GradPCA, also belongs to this class.
Regularity
These methods instead focus on identifying aaanoormaity""pptteee behaviors that characteristic of OOD inputs. Rather than relying on proximity to ID samples in any well-defined space, they exploit cues such as predictive uncertainty (c.g.. ODIN (Liang et al., 2018), Energy (Liu et al. 2020)) or atypical activation patterns (e.g.. ReAct (Sun et af.] 2021), GAIA (Chen et al. 2023)).
Abnormality-
Our experiments reveal that each class of methods performs best under different conditions (see Section 3.3 In addition to this taxonomy, we include a detailed review of related work in Appendix
2

---

Published as a conference paper at ICLR 2026
Spectral methods: PCA and Kernel PCA. We focus spectral OOD detectors subset of the regularity-based category since GradPCA belongs to this class. canonical example of spectral methods is kernel PCA, which operates in two steps:
1. Choose feature map h X' + H. that embeds the data into suitable Hilbert space H.
2. Perform standard PCA in the space H.
The challenge lies step (I): selecting an appropriate h such that and OOD samples become separable in the induced space. Classical kernel PCA specifies a kernel function rather than an explicit feature map, but this merely shifts the difficulty to kernel design, retaining the full complexity of representation selection. Prior work on kernel PCA for OOD detection largely relied generic or hoc kernels applied to network activations (Guan et al. 2023: Fang et al. 2024b) Hoffmann 2007, Xiao al.) 2013). Several recent works have also applied variants of spectral methods to gradients (Wu et al. [222a]] [Behpour et al.) (023). GradPCA adds to this line of works by employing the Neural Tangent Kernel (NTK)-a a task- and model-specific kernel-and introducing
efficient procedure for performing PCA in gradient space. In the next section, we formalize GradPCA and
show that can be viewed as an approximate kernel PCA method using the NTK kernel.
GRADPCA
The core idea GradPCA is to apply classical PCA in the space of neural network's 's gradients. While conceptually simple, this approach is not directly feasible for modern NNs due to the prohibitive size of the gradient covariance matrix. Consider NN with output function f : - and parameters E R.". For dataset {( y4 j== the empirical covariance matrix of gradients is given by:
FF E RRXX, where Fúj := V ww (x;).
(1)
In PCA, the standard approach for handling high-dimensional features is compute the eigende- composition of the smaller dual matrix F E NNN, However, in modern deep networks, both P (number of parameters) and N (dataset size) are typically too large for this be practical. To address this, GradPCA exploits the inherent low-rank structure of the gradients induced by NTK alignment.
3.1 NTK ALIGNMENT
A central observation in our approach is that the dual matrix F F NxN introduced above is exactly the empirical Neural Tangent Kernel (NTK) evaluated on the dataset:
= F. where O. == (w,)( faz)))
(2)
This implies that PCA in gradient space hinges on spectral analysis of the NTK- kernel that captures correlations between input samples in network's 's training dynamics.
A growing line of recent rescarch shows that, dur- ing training, the empirical NTTK of well-performing neural networks progressively aligns with the struc- ture of the learning task phenomenon termed NTK alignment (Atanasov et al. 2022 Baratin et al. 2202 Shan & Bordelon 2021). This alignment reshapes the kernel spectrum in task-relevant manner and
ResNet20 MNIST
DenseNet40 CIFARI0
VGGI CIFARI0
associated with stable training and strong gener-
alization (Chen et al.) 2020 Atanasov et al.) 2022 Seleznova al. 2023). This echoes classical kernel methods theory, where kernel-target alignment is
Figure 1: Block-diagonal NTK structure. Heatmaps show the NTK matrix on ID data subset. Lighter colors indicate larger entries. Diagonal blocks correspond to samples from the same class. A detailed description is pro- vided in Appendix E.
known predictor of generalization (Cristianini et al.
2001 Gönen & Alpaydın 2011 Canatar al. [2021).
In classification problems, NTK alignment manifests
approximate block-diagone structure in the NTK matrix: inputs from the same class exhibit strong correlations, while cross-class interactions are weak (Seleznova et al.) (023). This behavior is deeply related to the principle of local elasticity (He & Su
during training, each
3

---

Published as a conference paper at ICLR 2026
input primarily influences inputs belonging to the same class. As result, gradients concentrate in a diwddimnnsional subspace spanned by elass-specifer directions, inducing a pronounced low-rank structure. This structure is visualized for several neural network classifiers in Figure [ More formally, for a classification problem with C' balanced classes, each containing m == N/C samples, the NTK takes the following form:
θ = G G G = g
Σ J, &
(3)
where @ denotes the Kronecker product. The leading term is rank C (typically C < N, P), and É captures deviations from the perfect alignment. When alignment is strong, || S for some small e. and the NTK spectrum is dominated by the block-structured term. This approximation underpins our method, and we analyze its robustness with respect to the residual term in Section 4.2
3.2 ALGORITHM
The final observation underlying GradPCA is that the eigendecompositionn of the block-structured matrix G G @ 1m1m depends solely on the class-mean gradients gg ......... As result, there no need to store or process the full dataset (or even a large batch) to approximate the principal components of the gradient covariance. It suffices to compute- gradient vectors, which greatly reduces the computational cost. The resulting method reduces to performing PCA on modestly-sized matrix and is summarized in Algorithm
approximate the C' class-mean
Algorithm 1: GRADPCA
Input Test input x, output function f X ID dataset D, threshold 6, number of PCs k Detector decision D(x) E {0, 1}
Output:
Offline Stage (Training)
Compute class-mean gradients g² ....., g and global mean g:
Form centered matrix G := Ig' g, ,,...,g gC g]:
Compute eigendecompositionn of
G G R
Cxc via θ VEVT:
Compute PCs of S GG R PxP via U GV 1/2, Construct projection matrix P = U2 U onto the span of PCs of
Online Stage (Inference)
Compute centered gradient g g(x) = f=f(x)-B
Compute score s(z) == |P 1/ll=)//l[==))]
10
return (xx) = [s,s)
Remark 3.1 (Computation of PCs). Since E CCxC and S E PxP share the same nonzero eigenvalues, PCA can be performed in the smaller C-dimensional space. We compute the eigende- composition
θ = VEVT
where V [1 (......, Tc] and Σ diag 11,.... ac) with or 2 2 oc 2 0, and keep the top i eigenpairs by setting V. = [u 2x and Σg diag(ø 1o.., Rx). The lifted PCs and the associated projection matrix are then given by
U = GV
P:= U
(5)
The resulting columns of U are precisely the top & principcal components of S.
Remark 3.2 (Score function). We use the fraction of the centered gradient norm preserved by the principal subspace as the OOD detector's score s(x) := IPP ||gx))] where ID inputs typically yield larger values. A classical PCA-based detector would instead use the reconstruction error (gr))) Pg(x)]]. However, recent work (Guan et al. 2023) has shown that the angle between a vector and its projection often more informative for OOD detection than the magnitude of the residual. Our score is precisely angle-based measure, since s(æ) cos(<(gt (gg(2), Pg(=))). We provide a formal justification for why scores of this type, as well as spectral methods more broadly,
principled and effective for OOD detection in Section 4
4

---

Published as a conference paper at ICLR 2026
Remark 3.3 (Aggregation), An important practical consideration is that method is formulated for scalar-valued output functions, whereas standard classifiers produce vector-valuee outputs RR As result, GradPCA requires an aggregation step combine information across output dimensions. In the default version of our implementation, we use the maximum of the network's 's logits as the scalar output, i.e., (x) max.c(L...... (x)), where (1)), { (1,...,. C) the logits input
Remark 3.4 (Using subset of parameters.). To further reduce computational cost, we compute gradients with respect only subset of parameters parametess--partcullllyy useful when the number of classes C' or parameters P is large. Importantly, GradPCA is flexible: it can operate on arbitrary parameter subsets and is not tied to any specific layer or parameter group. Moreover, our ablations in Appendix show that different parameter subsets may be optimal for different models, likely reflecting which layers carry the most information for OOD detection. Additional implementation details are provided in Section an and Appendix
3.3 GRADPCA PERFORMANCE: CONSISTENCY AND EFFECTS OF FEATURE QUALITY
Before detailing our experimental setup Section 5 we first highlight the main empirical insights.
Consistency. A central observation in our experiments that OOD detectors' performance can be highly inconsistent even across models with the same architecture and ID dataset-aa an issue noted prior work (Tajwar et al. 2021 Szyc et al., 2023). To address this, our evaluation emphasizes consistency by comparing methods across diverse benchmarks designed to reveal both strengths and weaknesses of each approach. As shown in Figure GradPCA achieves the highest average performance among competitive baselines, ranking within the top three methods in virtually every setting. This demonstrates stability of GradPCA's performance, observed for many baselines. We attribute this robustness to the principled foundation of our approach: since NTK alignment is pervasive in well-trained networks, and is tied to strong performance (Chen al. 2020 Shan & Bordelon 2021 Seleznova al. 2023), GradPCA is expected generalize well across a broad range of realistic settings. Beyond consistency with respect to architectural and dataset choices, prior work has shown that OOD detection also be surprisingly sensitive the random seed used during training (Fang et al. (024a). We examine this factor in Appendix E.3 and find that GradPCA remains stable across multiple training runs with different seeds.
contrast to the high variability
11:11
$3.80
72.99
74.72
67.0
00.34
17.15
11.74
09:21
95.14
43. 19
$3.01
$4.25
51.71
53.75
$8.0
$22-49
9102
$1.12
48.51
90.71
92.11
63.97
141.2
40.55
16/21
90.55
90.4
90.43
K0/06
$0.41
06.95
98.00
6G-27
06.43
16:22
GALL-A
Max Judia
Figure 2: Performance comparison of OOD detection methods across multiple settings. For each method, the left bar shows the stacked average AUC scores 1 on 6 benchmarks described Section (in order from bottom to top): (1) CIFAR-10 BiT-M (pretrained, Table 4 (2) CIFAR-10 TIMM (Table (3) CIFAR- 100 BiT-M (pretrained, Table Th. (4) CIFAR-100 TIMM (Table (5) ImageNet BiT-M (pretrained, Table 2). (6) ImageNet BiT-S (Table 2.. The middle bar shows the stacked average FPR95 scores I for each method. The right bar shows the stacked runtime per sample estimates (Table The methods ordered left to right by the average AUC score.
Effects feature quality. While prior work has shown that OOD detection performance can vary significantly due to subtle factors such as minor architecture details, random seeds, or dataset splits (Szyc et al. 2023). there is little guidance on how to choose or adapt methods accordingly. We advance this discussion identifying feature mmality-heete (pretrained) or task specifcc (non-po pretrrrttai)ee) models as key factor for OOD detection performance. We further explain its role through categorization of OOD detectors into regularity-basse and
features come from general- ccurpoo-purpose
5

---

Published as a conference paper at ICLR 2026
abnormality-basto -bseed (see Section 2). Regularity-base based methods perform best with general purposs features of pretrained models, while abnormality- bbssdd methods are more effective when models are trained from scratch, likely because general-purpos purpose features suppress the irregularities these methods aim to detect. Our benchmarks are designed to reflect this: for each II) dataset, we evaluate on two models with similar architectures and II) accuracy but differing feature quality-on pretrained on large general dataset and fine-tuned on the ID dataset, the other trained directly on the ID methods (GradPCA, KNN (Sun al. 2022b), Mahalanobis (Lee et al. (0118) tend to excel on pretrained models, while abnormality- based methods (GAIA (Chen et al. 2023). ODIN (Liang al., 2018). Energy (Liu al., (2200) are closer to state-of-the-art in the non-pretrained settings. This observation offers practical guidance: regularity- based methods are preferable when strong, pretrained features are available, while abnormality- methods may be more suitable in lower-quality or non-pretrained regimes.
dataset. Our results (Figure 2 show that regularity-
Computational cost. GradPCA achieves competitive inference- time efficiency through its paral- lelized implementation and support for batch evaluation, performing on par with fast logits-based methods such as MSP and ODIN on CIFAR (see Figure 2 As with other regularity- methods, incurs two additional computational costs: memory usage (requiring storage of O(C) vectors) and training phase. On modern hardware, GradPCA remains practical on ImageNet, processing over 100 samples per second in our setup (see Appendix D). and provides favorable trade-offs in applications prioritizing robustness and interpretability.
4 SPECTRAL OOD DETECTION FOR NEURAL NETWORKS
Spectral methods for OOD detection apply PCA to representations defined by a feature map h: X - H. which embeds inputs into a Hilbert space where ID data exhibit meaningful spectral structure. The effectiveness of such methods depends critically on the choice of h. In this section, we formalize the principles underlying spectral OOD detection.
4.1 SUFFICIENT CONDITION FOR SPECTRAL OOD DETECTION
We begin with a simple yet broadly applicable result showing how the structure of the covariance matrix can be leveraged for OOD detection. Specifically, the following theorem (proven Ap- pendix B.2) establishes that any point with a component outside the range of the covariance matrix is guaranteed to be OOD. This yields one-sided, per-sample OOD certificates for spectral detectors.
Theorem 4.1 (Sufficient condition for spectral OOD detection). Let X ~ Pia and h X - TR. be any function I.2 (1sa). Consider the covariance matrix
Shh) = E[M(X)((X))] 1.
(6)
Let Ph(xx) be the orthogonal projection of h(x) onto the range of S(h). For any x E X, if [[Ph(x)]2 < [[(x)]]2 and h is continuous at then x is OOD.
Although this result resembles the elementary linear algebra fact that ranges of A and AAT equal, which is used in some classical PCA works (Schölkopf et al.) 1998 Hastic et al. 2009). our version introduces a probabilistic perspective that, our knowledge, has not been formally stated in the OOD detection literature. In our view, this provides foundational motivation for the following detector:
s(z) = [Ph(x)]2
Da(z) = (sxx)
|1(x)]]2
where s: X + [0, 1] is score function and 8 E (0, 1] is a threshold. We next show how this connects the practical use of PCA for OOD detection.
4.2 CONNECTION TO PCA AND GRADPCA
We now show how Theorem 4.1 yields a practical OOD certificate for PCA-based detectors when the empirical covariance matrix approximates a low-rank population covariance S(h).
Theorem 4.2 (Robust OOD certificate for PCA). Consider PCA applied to a matrix S RPxP (e.g., estimated from a noisy sample). Let h and (h) be as in Theorem 4.1 and assume the following:
[h) $11 S e, rSnk)S(A)) =C,
8

---

Published as a conference paper at ICLR 2026
i.e. S approximates rank-C covariance matrix S(h). Let Pe denote the orthogonal projector onto the top eigenvectors of S, and let Ae be the C-th largest eigenvalue of (h)). Then, for any input t E X, the following condition is sufficient to guarantee that x is OOD:
2e
(PCA(x) = [Poh(x)12 [|(x)]]2
SPC) (x) <
Àc c
The proof, based on standard projection bounds and the Davis-Kahhn theorem (Davis & Kahan 1970). is provided in Appendix [B.] This result characterizes the robustness of spectral OOD detection to perturbations in the covariance structure- structure-arisingg arrisng from noise, saatee effects, or deviations from an ideal low-rank model (e.g., the residual term & in equation 3. This framework aligns naturally with GradPCA, where we approximate the empirical (potentially full-rank) gradient covariance using a low-rank surrogate constructed from class-mean gradient vectors.
4.3 NECESSARY CONDITION FOR SPECTRAL OOD DETECTION
The results discussed so far provide only sufficient, but not necessary. conditions for identifying a point E X' OOD, Indeed, depending the choice of h, Theorem 4. may allow to identify all OOD points or, in the worst case, none at all. We illustrate this by the following two examples.
Example 4.3 (Best case). Let h X R. be an indicator function of pia support, defined as h(x) = v 1 sapp(pu)( for some fixed nonzero vector v E IR.. Then, I E X is OOD if and only if [[Ph(z)]l < [1(x)]l2, and detector Dú, (defined in equation 7 identifies all OOD points.
Example 4.4 (Worst case). Let h X - TR.. be a function such that rank = P. Then Ph(x) h(x) for all x E X, and no OOD points can be detected by DT.
As illustrated by the worst-case example, the following condition is necessary for spectral OOD detection to be effective. A formal proof is provided in Appendix [B.5]
Theorem 4.5 (Necessary condition for spectral OOD detection). Consider the same setting as in Theorem and let Dr be the spectral OOD detector defined in equation 7 with an arbitrary threshold 1]. Then, the following condition is necessary for Dp, to be effective (i.e., have non-zero sensitivity):
rank(S(h)) < dim ((h(z) EX))
(10)
In other words, the image of the OOD data must not lie entirely within the image of the ID data. The ideal setting occurs when the feature map h embeds all ID points into a low-rank subspace while mapping OOD points outside of it-precisely the structure exhibited in the best-case example. In the absence of any knowledge about the distribution or the structure of h, it is generally intractable to formulate necessary and sufficient conditions for a given point x E X' to be OOD.
4.4 CHOICE OF FEATURE MAP FOR SPECTRAL OOD DETECTION IN NEURAL NETWORKS
Based on the previous discussion, an effective feature map h for spectral OOD detection should satisfy two key principles consistent with standard PCA and kemeel PCA intuition: (1) The image of ID data (h(z).
supp(pia) should concentrate in low-rank subspace; (2) the image of OOD data (h(z): x & supp(μa))} should fall outside this subspace. We now evaluate the following feature spaces commonly used OOD detection methods against these criteria: logits, hidden activations (hidden-layee features), and gradients.
Logits: In classification tasks, the logits are a vector in RC, where C is the number of classes. Since logits are trained to approximate one-hot vectors of class labels, the image of ID data (h(x) E supp(/aa) spans the whole C As a result, the logits space lacks the low-rank structure necessary for effective spectral OOD detection. Consistent with this, competitive logits- based OOD detectors (c.g., ODIN (Liang et al. 2018), Energy (Liu et al. 2020)) do not rely on spectral methods.
The spectral structure of hidden-layer features depends heavily the choice of layer. When maps to the penultimate layer features, the Neural Collapse (NC) phenomenon (Papyan) et al. 2020) shows that the ID data {h(x) of rank C, much smaller than the ambient dimension. However, this structure degrades in earlier
Hidden activations:
supp(p indeed often concentrates in subspace
layers (Parker et al. 2023). Several seuuarity-based OOD detectors (e.g. Mahalanobis (Lee et al.
7

---

Published as a conference paper at ICLR 2026
operate in the activations space, suggesting that it often retains sufficient structure in practice. In addition, a recent work proposed an OOD detector explicitly based on the NC phenomenon (Liu & Qin 2025). That said, it remains unclear whether OOD data {h(x) a & supp(sna)} consistently deviates from the ID subspace. In fact, results from the adversarial examples literature (Carlini & Wagner (017) shows that OOD inputs can be crafted to effectively mimic ID features in hidden layers, which limits effectiveness of spectral detectors in hidden activations space.
2018) KNN (Sun et al. 2022b))
Gradients: As discussed in Section 3 the NTK alignment phenomenon implies that the ID gradients (h(z) : x E suрр(ru)) approximately lie
diweemmeniionll subspace of rank C. Empirically, this low-rank structure is robust across architectures and datasets (Baratin et al. [021] Atanasov et al.
2022 Shan & Bordelon 2021, Seleznova et al. 2023). Notably, NTK alignment has been observed to happen more often than NC (Seleznova et al.) 2023). 3) and is not limited to the final stage of training (Fort et al.) 2020). Unlike forward-pass features, gradients belong to a much higher-dimensional space, making the aligned subspace significamtly harder to mimic-e.g. e.g. through adversarial examples. This high ambient dimensionality amplifies spectral separation, reducing the likelihood that OOD samples project strongly onto the principal components of the ID-induced subspace.
5 EXPERIMENTS
In this section, we outline the technical setup of our experiments. We evaluate on three ID datasets: CIFAR-10 [ CIFAR-100 (Table and ImageNet- Ik [Table Table Aggregated results across the main benchmarks are summarized in Figure 2 To ensure reproducibility and control for variability, we use only publicly available datasets and models. Benchmarks for each dataset include at least two models to assess the impact of representations quality (sce Section 3.3).
Implementation of GradPCA. Our implementation of GradPCA follows Algorithm I with two key extensions: (1) a configurable aggregation scheme over output heads, and (2) the option to compute gradients with respect to subset of trainable parameters, improving scalability. To truncate the spectrum, we use a configurable threshold (default: 0.99), denoting the fraction of the trace retained by the principal components. The default variant, referred to GradPCA, operates on gradients with respect to the last hidden layer parameters, averages logits across output dimensions before PCA application, and computes gradient class-means sequentially over the training Mata-making it scalable to large datasets. Computational costs are described in Appendix D
In addition, we implement several variants of GradPCA: (1) GradPCA+DICE, which incorporates
the DICE method (Sun & Lij (022) by sparsifying gradient class-means means before PCA; (2) GradPCA-
Batch, which computes the empirical NTK on a batch instead of relying on class-means; (3) GradPCA-Ver which omits output aggregation and computes GradPCA separately for each of the C' output heads, producing C' scores that can be aggregated post hoc. While these variants can be advantageous in certain scenarios, our main evaluation (Figure 2 focuses on the base GradPCA variant due to its efficiency and lack of hyperparameter tuning enabling fair comparison across models and datasets. We include several GradPCA variants in Tables I and 2 and extensive ablations in Appendix E
We release implementation of GradPCA in JAX (Bradbury et al. 2018).
Baselines. We compare GradPCA against a range of established methods that have demonstrated effectiveness across diverse benchmarks. In particular, we include regularity- bassd methods such as Mahalanobis (Lee et al. 2018) and Deep k-Nearest Neighbors (KNN) (Sun et al. (022b),, which methodologically aligned with GradPCA and serve as natural points of comparison. We also incorporated a range of confidence-based scores (MSP (Hendrycks & Gimpel 2017), ODIN (Liang al. 2018). Energy (Liu et al., 20200), as well as DICE (Sun & Li| (022). which leverages sparsity, and ReAct (Sun et 2021). which relies on abnormal activation patterns. Collectively, these baselines span broad spectrum of approaches commonly found in the literature. Since our method operates in the gradient space, we include comparisons with three saute-of-the-ee gradient-
OOD detectors: GAIA (Chen et al. 2023), GradOrth (Behpour et al.) 2023). and the Projected Gradients method of Wu et al. (2024a). We additionally compare against two recent PCA-based approaches: the Revisited PCA method of Guan et al. (023) and Kernel PCA (CoRP) (Fang et al. 2024b). Finally, we compare against the recent Neural Collapse Inspired (NCI) detector (Liu & Qin 2025), which similarly exploits low-rank structure (in penultimate features rather than gradients). Implementation and experimental setup details for all baselines are provided in Appendix
8

---

Published as a conference paper at ICLR 2026
5.1
CIFAR BENCHMARK
SYIN
Fbrn
LSCSE
BUN
Melaal
Labek
0137
22.55
70.31
25.35
$4.33
37700
55.5
11.1
Bunny
69.76
67.4
43.03
$1.05
07:09
0100
30.0
31.00
40.15
01
50.43
11
23.34
4.10
12.00
71
Boka
3657
03.07
11.24
13.76
$109
$3.39
13.33
1515
1030
Cradeen
25294
1250
16.33
36.33
28.05
$5.00
$3.80
$2.45
15.35
2.80
18.33
31.79
12.90
Peo (hah
16.76
8166
15:0
11.44
81:3
16.0
KarwiiCA SCaad)
11.9
9650
$3.96
3511
95.35
GUaRNA
-708
98.30
25.04
90.30
8.38
86.3
76.00
$1:34
3.41
9924
2780
Lakeb
SYHE
Fbun
13E34
LSCN
BEN
Arstag
11:20
03.49
08:11
26.34
1111
2013
2110
1211
13
Punny
1315
06.34
50.13
0:13
11.33
35.0
23.34
5100
65%
13
8:51
33.31
15.13
45.71
KNN
10.00
2015
06.37
1000
9032
25.07
64.5
03.05
$2.0
12.35
13
3057
4331
9652
7035
$3.66
05.35
15.0
3657
17.15
0145
$7.00
0534
13.50
Peo Dheh
11.0
9.39
11
770
1131
0113
11:11
1710
15.3
(Caa))
31.33
28
0131
03:00
61:11
01355
1125
0.38
Table I: CIFAR-100. For each architecture and evaluation metric (FPR95, AUROC), the best- performing method is shown in bold, and the second and third best methods are underlined. The Labels column indicates whether the method requires access to ID data labels during training (V).
Datasets: For CIFAR-I0 and CIFAR-100) 100, we use the standard training splits during the training
phase and the corresponding test splits as ID data during the detection phase. We evaluate against
a range of commonly used OOD datasets: SVHN (Netzer et al.) 2011), Places (Zhou et al. (017)) LSUN (Yu et al., 50)5a), iSUN (Xu et al. 2015), and Textures (Cimpoi et al. 2014). SVHN (test split), Places (validation), and Textures (full dataset) are taken directly from the predefined splits in the TensorFlow Datasets collection (TFD). For LSUN and iSUN, we rely on curated subsets for OOD
detection benchmarks introduced in the ODIN paper (Liang al. 2018).
Models: We consider two publicly available models: (1) ResNetV2-50 from Big Transfer (BiT)
repository (Beyer et al. 2022) pretrained on ImageNet-21k (BiT-M) and fine-tuned on CIFAR, (2)
ResNet-34 trained directly on CIFAR from PyTorch Image Models (TIMM) (Wightman 2019)
Results: As shown in Table D PCA-based methods (including GradPCA. CoRP, and Revisited PCA) as well as other regularity-base based approaches such as Mahalanobis and KNN, achieve strong performance on CIFAR- 100 when using general-purpose (pretrained) features. However, as discussed in Section 3.3) the landscape shifts substantially in the task-specific (non-pretrained) setting: here, GAIA method excels, while confidence- also approach state-of-the-ar performance. In contrast, most regularity-based methods degrade noticeably in this scenario, a decline not observed among abnormality-baan based approaches. An additional observation, also visible in Figure 2 is that methods that excel under certain conditions may fail entirely in others due violations their core assumptions. Examples in this benchmark include ReAct and GAIA-Z. each underperforming in one of the settings.
the abnormality-
detectors (e.g., ODIN, MSP)
5.2
IMAGENET BENCHMARK
JUN
Astrag
Trskem
14:0
01:4
6144
1122
14.72
2517
0234
14.05
12.7
$4.30
13:00
4114
13.0
7434
01/2
1572
61.5
0918
11.0
12:1
08.11
15:2
12.00
03:55
95.0
16.00
02:0
11:05
17.00
0760
2110
18.00
12.03
12.4
12.00
2110
35.00
3.38
8100
3113
1148
1220
0134
13.8
1210
18
11.0
13.0
14
8385
1375
04.30
18.0
1255
1010
1370
150
24.05
342
71/8
2133
43.1
25.8
2..0
514
12
211
21
12
2.30
312
$1.0
1220
1741
Table 2: ImageNet. For each architecture and evaluation metric (FPR95, AUROC), the best- performing method is shown in bold, and the second and third best methods are underlined.

---

Published as a conference paper at ICLR 2026
Datasets: We use the full ImageNet- 1k training split in the training phase. While subset could be used, our method is scalable enough to handle the entire dataset efficiently. For the detection phase, we treat ImageNetV2 (Recht et al. 2019) as the ID dataset. As OOD datasets, use curated subsets of Places (Zhou et al. 2017). SUN (Xiao et al. 2010), and iNaturalist (Van Horn et al., 2018), released by the MOS paper (Huang & Li 2021) to ensure minimal category overlap with ImageNet. This control is especially important given that these datasets contain many overlapping classes. We additionally include the full Textures dataset (Cimpoi et al. 2014) as an OOD source.
Models: We consider two BiT models (Beyer et al. 2022); : (1) ResNetV2-101 pretrained on ImageNet- 21k (BiT-M) and fine-tuned on ImageNet- Ik, (2) ResNetV2-50 trained directly on ImageNet- Ik
(BiT-S). In addition, we include a benchmark for (3) Vision Transformer (ViT-B/16) (Dosovitskiy
et al.
202T) pretrained on ImageNet-21K and fine-tuned on ImageNet- 1k in Appendix F.2
Results: Consistent with trends observed on CIFAR, GradPCA achieves competitive performance on ImageNet-rankinn first in the pretrained setting and third in the non- con-pretrained setting. where DICE performs best. GradPCA also ranks first on the ViT-B/16 benchmark (Appendix, Table (6) Most other regularity-based methods lag behind the state of the art on ImageNet benchmarks, with the notable exception of GradORTH, which performs strongly on the BiT-S model but underperforms on the BiT-M model. As with CIFAR, method rankings vary substantially between the two models. Notably, the pretrained ImageNet BiT benchmark is substantially more challenging than the non- pretrained variant for all detectors, despite the models having similar architectures and ID accuracy. This underscores the sensitivity of OOD detection to subtle differences in evaluation setup. We report results for GradPCA+DICE (p = 0.8) in Tables 2 and along with ablations and additional GradPCA variants in Appendix E While certain tuned variants yield modest gains in specific cases, their performance is inconsisteni across settings; for this reason, we focus on the default tuning-free GradPCA in our main discussion.
CONCLUSIONS AND BROAD IMPACT
This paper introduces GradPCA, an OOD detection method based on the low-rank structure of gradients in well-trained NNs, induced by NTK alignment. Our approach bridges recent advances in OOD detection with classical PCA-based intuition, clarifying when spectral methods are applicable in modern deep learning. In addition, our theoretical analysis and the observation about feature quality as a critical factor for OOD detectors' performance offer practical guidance for selecting appropriate detectors based on the training regime. Such guidance remains rare in the OOD detection literature. Finally, our work establishes a novel connection between the NTK- a common topic in deep learning and OOD detection, a largely empirical field with limited theoretical grounding.
Limitations. GradPCA is built on the assumption that the NTK structure effectively separates ID and OOD points in gradient space. While this may not hold universally, the assumption is both explicit and empirically Importantly, this assumption has been found to hold in wide range of practical settings, including publicly available, well-trained models commonly used in OOD evaluation. A second limitation memory scalability with respect to the number of classes: GradPCA stores up to O(C) gradient vectors, which can be costly for large C. This requirement is common to many regularity- based approaches and can be mitigated through (1) retained variance threshold e (see Appendix (E.7) and (2) parameter subset selection. Our method shares two other limitations with Mahalanobis and prototype- based detectors (see Appendix A). First, it requires ID labels during training, though this can be mitigated using ppeudo-labels clustering (Sehwag et al.) (202T). Second, it is inherently tied to classification tasks, where NTK alignment induces the low-rank structure our method relies on.
many heuristic or implicit assumptions used in prior works.
Future work. By establishing a connection between NTK alignment and OOD detection, our work enables future advances in NTK theory to directly inform and improve methods like GradPCA. As the study of feature learning dynamies and NTK structure remains an active area of theoretical research, we anticipate that forthcoming insights in this domain could be leveraged to design more principled and effective OOD detectors. On the other hand, our results may motivate future works on the relationship between NTK alignment and generalization. On the practical side, several directions remain open, including scalability to very large models and extensions beyond classification setting.
10

---

Published as a conference paper at ICLR 2026
Reproducibility statement. We provide full details of our experimental setup in the main text (Section 5) and supplementary materials (Appendix C). and include anonymized source code in the supplementary submission. The code will be made publicly available upon acceptance. All experiments are conducted using publicly available models and datasets. Where applicable, we use official community-releaser implementations of baseline methods. Dataset splits follow standard OOD benchmarks, and we provide references or instructions for obtaining each one. All theoretical results are stated with explicit assumptions and are accompanied by complete proof' in Appendix
M. Seleznova acknowledges travel funding from the LMU Postdoc Support Fund for presenting this work at ICLR. G. Kutyniok acknowledged support from the gAIn project, which is funded by the Bavarian Ministry of Science and the Arts (StMWK Bayern) and the Saxon Ministry for Science, Culture and Tourism (SMWK Sachsen).
Acknowledgements.
REFERENCES
TensorFlow Datasets, collection of ready-to-use datasets. Jt:Ds::/ww..ensooff.ooo datasets
Dario Amodei, Chris Olah, Jacob Steinhardr, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR, abs/1606.06565, 2016. URL http:://rxii..ooe
abs/1606. 06565
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan, Neural networks as kernel learners: The silent alignment effect. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL htttps://opeereeee/// net / Eorum ? id 1Nvf
Aristide Baratin, Thomas George, César Laurent, R. Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pp. 2269-2277. PMLR, 2021. URL (ttp:ttt:/ppooceeiii..
mlr
html
Sima Behpour, Thang Long Doan, Xin Li, Wenbin He, Liang Gou, and Liu Ren. Gradorth: A simple yet efficient out-of-distribution detection with orthogonal projection of gradients. Advances in Neural Information Processing Systems, 36:38206-38238 2023.
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10925-10934, 2022.
James Bradbury. Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://githun com/jax-mman jax-m1/jax
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model align- ment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):9914, 2021.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 3-14, 2017.
Jinggang Chen, Junjie Li, Xiaoyang Qu, Jianzong Wang. Jiguang Wan, and Jing Xiao. Gaia: Delving into gradient-based attribution abnormality for distr--disttitttion detection. Advances Neural Information Processing Systems, 36.79946-79958, 2023.
Shuxiao Chen, Hangfeng He, and Weijie Su. Label-aware neural tangent kernl:: Toward better generalization and local elasticity. Advances in Neural Information Processing Systems, 33: 15847-15858, 2020.
11

---

Published as a conference paper at ICLR 2026
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2014.
Nello Cristianini, John Shawe- Shawe-Taylor Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
Eduardo Dadalto. Detectors: python library for generalized diitt-ddittiittiion detection, 5 2023.
URL http::tpp://t hattps://github... com/
http:///hh....cccceeaaeaeoooo/// edadaltocg/detettttt
Chandler Davis and W. M. Kahan. The rotation of cigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1-46, 1970. doi: 10.1137/0707001.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
Xuefeng Du Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't 't know by virtual outlier synthesis. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URUhtttss
Forum??d=7T7d65uu
Mostafa ElAraby, Sabyasachi Sahoo, Yann Pequignot, Paul Novello, and Liam Paull. Grood: Gradient-aware distt--distiibution detection. Transactions on Machine Learning Research, 2025.
Kun Fang, Qinghua Tao, Xiaolin Huang, and Jie Yang. Revisiting deep ensemble for out-of- distribution detection: loss landscape perspective. International Journal of Computer Vision, 132(12):1077-612 2024a.
Kun Fang, Qinghua Tao, Kexin Lv, Mingzhen He, Xiaolin Huang, and JIE YANG. Kernel PCA for Out-of-Distribttion Detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances Neural Information Processing Systems, 33:5850-5861, 2020.
Mehmet Gönen and Ethem Alpaydın. Multiple kernel learning algorithms. The Journal of Machine Learning Research, 12:2211 1222211-2268, 2011.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL httpttt::///aaeae
org/ abs / 1412. 6572
Xiaoyuan Guan, Zhouwu Liu, Wei-Shi Zheng, Yuren Zhou, and Ruixuan Wang. Revisit pea- based technique for -uito--distiibutio detection. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 19382.19382. IEEE, 2023, doi: 10.1 10090CCV551000.222..117777 URL https::/ org/ 1 1111010CV1C110 2023. 01780
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Hangfeng He and Weijie J. Su. The local elasticity of neural networks. In 8th International Conference on Learning Representattons, ICLR 2020, Addis Ababa, Ethiopia, April 26 2020. OpenReview.net, 2020. URI. httthttss:://pee open ew net./ met./forrm forum? d=HIxMYANt
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2024. URL
http::/gitte thwbb./.tttt....mm com/ goog le/ flax
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Distrriution Examples in Neural Networks. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Heiko Hoffmann. Kemnl pea for novelty detection. Pattern Recognition, 40(3): 2007.
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology. 24(6):417-441, 1933. doi: 10. 1100//0007125..
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance gradients for detecting distributional shifts in the wild. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.). Advances Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual, pp. 677-6897 689, 2021. URL 6c6 http:///222...01000000oo./aaaaa.
www.:eddiiinn.ueeneooeeeeee neurips cttsscccpaaaee pappp/202////////////// Abstract .tt.ractttmmml
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8589, 2018.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 491-507. Springer, 2020.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL /tttps://openreviiii net /ert/forum? Hit.uummndd-GGGGRcoo
Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Backpropagated gradient representations for anomaly detection. In Proceedings of the European Conference on Com- puter Vision (ECCV), 2020.
Jinsol Lee and Ghassan AlRegib. Gradients measure of uncertainty in neural networks. In 2020 IEFE International Conference on Image Processing (ICIP), pp. 2416-2420, 2420. IEEE, 2020.
Jinsol Lee, Mohit Prabhushankar, and Ghassan AlRegib. Gradient- adversarial and out-of- distribution detection. arXiv preprint arXiv:2206.08255. 2022.
Kimin Lec, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Yixia Li. Boya Xiong, Guanhua Chen, and Yun Chen. Setar: selective low-rank approximation. Advances Neural Information Processing Systems, 37: 72840-72871, 2024.
detection with
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Litian Liu and Yao Qin. Detecting out-of-distribution through the lens of neural collapse. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15424-15433, 2025.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out oft--of-dittrbutiie detection. Advances in neural information processing systems, 33:21 21475,,,,, 2020.
13

---

Published as a conference paper at ICLR 2026
Yizhang Lou, Chris E Mingard, and Soufiane Hayou. Feature learning and signal propagation in deep neural networks. In International Conference on Machine Learning, pp. 12248-14282. PMLR, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and insupervised feature learning, pp. 4. Granada, 2011.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 427 436. IEEE Computer Society, 2015. doi: 1A..1199VVRR..15.222888...
URL oro./
1109/CVPR. 7298640
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Liam Parker, Emre Onal, Anton Stengel, and Jake Intrater. Neural collapse in the intermediate hidden layers classification neural networks. arXiv preprint urXiv:2308.02766 2023.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pp. 3389-5400. PMLR, 2019.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan Likelihood ratios for out-o--distribution detection. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Alché-Bus Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14680- 14691, 2019. URL /tt. http://pweeeeea......eeeeaaaaaaa ctpppp//ppepprrrsosooop//ppepe ccppaprr///ppp
htmm
Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. Journal of the ACM (JACM), 54(4):21-es, 2007.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and distr--iistii detection: Solutions and future challenges. Transactions Machine Learning Research, 2022.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pp. 8491-8501. P PMLR, 2020.
Bernhard Schölkopf, Alexander Smola, and Klaus-Rohert Müller. Nonlinear component analysis as a kernel eigenvalue problem. In Neural Computation, volume 10, pp. 1299-1319, 1998.
Vikash Sehwag, Mung Chiang, and Prateek Mittal, Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the infinite-wide limit: Effects of depth and initialization. In International Conference on Machine Learning, ICML 2022, 17- 23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 19522-19560. PMLR, 2022. URL v1 62. se www.leenoaa2a....
ngs. ml rrrsss/
Mariia Seleznova, Dana Weitzner, Raja Giryes. Gitta Kutyniok, and Hung Hsu Chou. Neural (tangent kernel) collapse. In A. Oh. T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 16240-16270. Curran Asso- ciates, Inc., 2023. URL. MtR..tttpp::/pprceeeddiii 2D02/FELee3347ccceee3a222222222
neur ps cc wttcc/pppr__ffeee//eeeeeee firrrr_fii_ wwt.ffflesspppppee Paper Conference oer.reeeeeppppprrooreeeeeeeeee pdf
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprint 105. 2021.
14

---

Published as a conference paper at ICLR 2026
Sina Sharifi, Taha Entesari, Bardia Safaci, Vishal M Patel, and Mahyar Fazlyab. Gradient-regulatee out-of-distribution detection. In European Conference on Computer Vision, pp. Springer, 2024.
G. W. Stewart and Ji-Guang Sun. Matrix Perturbation Theory, Academic Press, Boston, 1990.
Jingbo Sun, Li Yang, Jiaxin Zhang, Frank Liu, Mahantesh Halappanavar, Deliang Fan, and Yu Cao. Gradient-basee novelty detection boosted by self-supervised binary classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8370-8377, 2022a.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In European conference computer vision, pp. 691-708. Springer, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-diitriibttion detection with rectified activations.
Advances in neural information processing systems, 34:14441577 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distributtion detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022b.
Kamil Szye, Tomasz Walkowiak, and Henryk Maciejewski. Why out-of-distribution detection experiments are not subill experimental details muddle the ood detector rankings. In Uncertainty in Artificial Intelligence, pp. 1078-2084 PMLR, 2023.
Fahim Tajwar, Ananya Kumar, Sang Michael Xie, and Percy Liang. No true state-of-the-art? ood detection methods are inconsistent across datasets. arXiv preprint arXiv:2109.055544 2021.
Huy Tran. Ood machine learning: Detection, robustness, and generalization. https://g Out
ccom homm/huytrane rans cessed: 2025-03-30.
2023. Ac-
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1690097000 PMLR, 2020. URL mlr. wwwpppsssssllll www.rtaammrroortrrammmaammm html
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard. Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Ross Wightman. Pytorch image models.
https://gitiub.ccm www.:/wwww.........w........
pytorch- imagmmmaeaaaa. mmage-modelss 2019.
Yingwen Wu, Tao Li, Xinwen Cheng, Jie Yang, and Xiaolin Huang. Low-dimensional gradient helps Out-of-distribution detection. IEEE Transactions on Puttern Analysis and Muchine Intelligence, 2024a.
Yingwen Wu, Ruiji Yu, Xinwen Cheng, Zhengbao He, and Xiaolin Huang. Pursuing feature separation based on neural collapse for dfito--diitributiin detection. arXiv preprint arXiv:2405.17816 2024b.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yingchao Xiao, Huangang Wang, Wenli Xu, and Junwu Zhou. LI norm based kpea for novelty detection. Pattern Recognition, 46(1):389-36) 2013.
Pingmei Xu, Krista Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint 15X.w:1504.06755, 2015.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of oft-of distrribbbttin detection: A survey. arXiv preprint a 10.1 2021.
15

---

Published as a conference paper at ICLR 2026
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1306.0333 03365, 2015a.
Yi Yu, Tengyao Wang, and Richard Samworth. A useful variant of the davis-kahan theorem for statisticians, Biometrika, 102(2):315-333, 2015b.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, al. Openood v1. 5: Enhanced benchmark for out-of- distribution detection. arXiv preprint arXiv:2306.09301 2023.
Zihan Zhang and Xiang Xiang. Decoupling maxlogit for out-of-distrinee of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3388-3397 3397, 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine 2017.
intelligence,
16

---

Published as a conference paper at ICLR 2026
Supplementary Material
This supplementary material is organized as follows:
Section A provides detailed discussion of related work, expanding on the brief overview given in the main paper.
Section presents additional theoretical results and complete proofs the main propositions and theorems introduced in the paper.
Section C describes the implementation details of GradPCA algorithm and the baseline methods used in our experiments.
Section discusses the computational cost considerations, including a breakdown of the components involved, empirical runtime comparisons, and an estimate of total compute usage.
Section E reports additional experimental results, including CIFAR-10 benchmark and set ablation studies for GradPCA variants.
Use of LLMs. Large language models (LLMs) were used to assist with editing and polishing the writing of this paper. All substantive ciriibutioosiiicuudii theoretical results, empirical findings, and experimental design were developed entirely by the authors without the aid of LLMs.
A RELATED WORKS
The field of out-of-distribttion detection has witnessed rapid growth in recent years, with over 15,000 publications since 2021 referencing either "OOD detection" and "ont-of-distribution detection" This surge in research activity has made increasingly challenging maintain a comprehensive view of methodological advances and theoretical developments. Still, this section complements the overview provided in Section by offering a broader survey of related work. We organize existing OOD detection methods according to our proposed taxonomy of regularity- and abnormalit- based approaches, which provides a unified lens for understanding a wide range of techniques in the literature.
A.1 SURVEYS AND BACKGROUND ON OOD DETECTION
Out-of-distribution detection has become a central topic in machine learning, particularly in the context of safety and robustness. For broad overview of the field, Tran (203) maintain extensive and regularly updated repository of OOD detection research. The foundational work by HHendrycks) Gimpel| (017) introduced one of the first widely adopted baselines for OOD detection. More
recent surveys, such as those by Yang et al. (2021) and (aalhii et al. (2022), provide systematic categorizations of the growing variety of techniques developed in this rapidly evolving area.
A.2 REGULARITY--ASSE BASED OOD DETECTION
Many OOD detection methods rely on the structured nature of in-distribution (ID) data in given feature space. These regularity-based methods model the geometry or statistics of ID samples-using representations, decision boundaries, or statistical criteria-to flag inputs that deviate from this structure. Due the nature this class, regularity- methods usually require a training stage to learn the properties of the ID data. While Section provides unified overview, here expand on specific approaches.
Proximity- -ased approaches. A wide range of methods rely the distance of test points to certain learned regular region in the feature space. The Mahalanobis detector (Lee et al. 2018 fits class-conditional Gaussians and scores inputs by their distance to the closest class mean. KNN-based detectors (Sun et al. 2022b) operate in the penultimate-lan space, identifying OOD inputs by their distance to nearest neighbors on the ID feature manifold.
Prototype-based detectors (van Amersfoort et al. 2020; Du et al. 2022) compute per-class feature centers--roootyy vectors, similar to the class means used in GradPCA or Mahalanobis- and
17

---

Published as a conference paper at ICLR 2026
measure similarity between a test imput and the nearest prototype. Since these methods often include additional regularization (e.g., energy shaping or uncertainty constraints) to improve ID-OOD separation, they can be viewed as hybrids that combine regularity-based modeling with abnormality- oriented scoring techniques.
Gram-matrix consistency (Sastry & Oore 2020) and likelihood-ratio scoring (Ren al. 2019) offer alternative strategies for exploiting structural regularities in ID data. In particular, the re elaioonship between the activation Gram matrix and OOD detection explored in (Sastry & Oore (020)) resonates with our theoretical discussion about the covariance matrix, which provides principled guarantees for spectral detection.
Spectral approaches. Spectral methods constitute key class of regularity- -based OOD detectors. canonical example kernel PCA, introduced by SSchökkopf et al. (1998) and later applied to novelty detection Hoffmannn (2007), who measured novelty via the squared distance to learned principal subspace in feature space.
More recently, Guan al. 2024 revisited PCA-based OOD detection, identifying the limitations of
conventional PCA and proposing regularized reconstruction error, further enhanced by combining it with energy-based scoring. Building on this and (Sun et al. 2022b), Fang et al. (2024b) introduced a kernel PCA detector using generic nonlinear kernels (e.g.. RBF, plynomial) to improve separation between ID and OOD imputs within the principal component subspace.
Other recent methods apply spectral principles in novel, non-covariance--arn based ways. Li et al. (2024) proposed SeTAR, a training-free, post-hoc detector that applies spectral analysis directly to pretrained model weights, arguing that the low-rank directions preserve ID-relevant structure while suppressing OOD-sensitive responses. Ww al. (202b) took a complementary approach, enforcing neural collapse during training to align ID features with a principal subspace and constrain OOD features to lie in the orthogonal complement. This hybrid method combines regularity modeling with supervised feature separation.
Gradient-based approaches. While the majority of gradient-based OOD detectors align more closely with the abnormality-based category, a few recent methods employ spectral regularity explic- itly in the gradient space. In particular, GradOrth (Behpour et al. (2023) operates in loss-gra doss-gradient space and applics SVD to last-layer activations to estimate principal directions and tranfer them to the gradient space. In contrast, Wu et al. (2024)) project input gradients onto class-meat directions, avoiding SVD altogether for efficiency.
Although these methods bear superficial similarity GradPCA in that they spectral reasoning and gradient space representations, they lack the theoretical grounding of our approach. In GradOrth, it is unclear why the SVD of activation covariances should align with meaningful directions in the gradient space-there there is no formal justification for this connection. In contrast, GradPCA performs SVD on the empirical NTK, which is the true dual of the gradient covariance matrix and thus guaranteed to share its spectrum. Moreover, GradOrth's 's reliance on large batches to estimate this structure limits its scalability to settings with many classes.
The approach Wu et al. 2024a)), while more scalable, makes a strong assumption that class- class-mean gradients themselves represent principal directions. However. these means are generally neither orthogonal nor aligned with the dominant eigenvectors of the gradient covariance. GradPCA addresses this by using class means as compressed basis to approximate the NTK's 's leading eigenspace, rather than assuming they are eigenvectors themselves.
Another regularity-based method leveraging gradients was introduced by Sun et al. (002aa), who use class-conditional gradient distributions and Mahalnnobis-distance score for OOD detection. A related recent approach is GROOD [ElAraby et al. 2025), which computes gradients with respect to synthetic OOD prototype and uses nearest-neighbor distances in gradient space the detection score. In contrast to GradPCA, these methods do not use spectral decomposition and thus do not fall3 within the spectral framework developed in this paper.
18

---

Published as a conference paper at ICLR 2026
A.3 ABNORMALITY-BASE OOD DETECTION
A complementary line of research focuses on detecting OOD inputs by monitoring deviations in model behavior, rather than modeling the structure of ID data. As discussed in Section 2 we refer to these as abnormality- indicators such as uncertainty, confidence degradation, or anomalous activation patterns. Unlike regularity- based approaches, which characterize the ID data structure in a well-defined feature space, these methods rely on behavioral cues that arise when the model encounters OOD samples samples-giving rise to a broad range of detection strategies.
methods. They evaluate model responses to unfamiliar inputs by tracking
Confidence-basee approaches. Many abnormality- -baed OOD detection methods define scoring functions the model's output logits to estimate prediction confidence. These scores- such as softmax maximum, energy, or temperature-sealed outputs- do not correspond to distances in any well-defined feature space, as they lack a reference point or metric structure. Moreover, these methods are post-hoc: they do not estimate or rely on the structure of ID data, but rather assume that OOD samples induce abnormal confidence patterns. For these reasons, confidence-based methods are best categorized as abnormality-basse
foundational method Maximum Softmax Probability (MSP) (Hendrycks & Gimpel 2017)
which uses the model's 's top predicted class probability as an OOD score. ODIN (Liang et al. 2018) improves upon MSP by applying temperature scaling and gradient- based input perturbations to enhance confidence separation between II) and OOD samples. Energy-basedd -baedd detection (Liu et al. 2020) generalizes these ideas by scoring inputs based on the log-sum-exp of logits.
Several methods further refine confidence signals. ReAct (Sun al. 2021 clips abnormally high activations in the penultimate layer to reduce overconfidence of OOD inputs, aiming to improve the scores like MSP ODIN. DML (Zhang & Xiang, 2023) decouples the max-logit score into cosine similarity and norm components, learning a weighted combination to better separate ID and OOD examples. Finally, DICE (Sun & Li (022) identifies a sparse, high-coofidence subnetwork by pruning low-magnitude weights, aiming to sharpen the model's output distribution and improve OOD) discrimination.
Gradient-based approaches. A growing number of abnormality -base methods leverage gradient information to detect OOD samples by assuming that such inputs trigger distinct unstable model responses. These approaches do not model nn-distribution structure explicitly but instead define OOD based on deviations in gradient norms, patterns, or attribution behavior.
Lee & AlRegib (020) introduced the feattrre-etrractionn hypothesis, proposing that the norm of the loss gradient can indicate how well the model understands given input. This and follow up work (Lee et al.] 2022) showed that OOD or corrupted samples often yield larger or more erratic gradients, making gradient magnitude useful proxy for abnormality. GradNorm (Huang al.. 2021) builds on this idea by computing the f--nrmm of gradients from the KL divergence between the model output and uniform distribution, observing that ID inputs typically produce stronger gradient signals. K Kwon et al. (2020) proposed a Fisher score for anomaly detection in autoencoders, interpreting reconstruction gradients as features and arguing that anomalous inputs require more complex updates.
Other methods analyze gradient-derived attribution maps. particular, GAIA (Chen et al. 2023) detects OOD inputs by analyzing disruptions in gradient- attribution maps such as saliency and integrated gradients. These are inpul
with respect to the input, not model parameters as in GradPCA. GAIA method is post-hoc and model-agnostic, relying irregularities in explanation behavior rather than ID modeling.
Some approaches incorporate gradient behavior during training. Sharifi et al. (hariii et al. 2024 introduce a regularization scheme to enforce locally consistent OOD scores, combining it with an energy-based sampling strategy to expose the model to diverse outlier patterns.
A.4 NEURAL TANGENT KERNEL ALIGNMENT
GradPCA is inspired by insights from the Neural Tangent Kernel (NTK) framework, originally introduced by Jacot et al. (Jacot et al. 2018). While classical NTK theory considers the infinite-wide width limit where the kernel remains fixed during training, recent work has shifted toward studying the
19

---

Published as a conference paper at ICLR 2026
empirical NTK and its dynamic alignment to the target function termed NTK alignment (Baratin ce al. 2021 Shan & Bordelon 2021; Atanasov et al. 2022).
NTK alignment has been linked improved generalization (Chen et al. 2020) and builds carlier work in kernel methods, where alignment between kernel and target label structures was formally studied (Cristianini al., 2001] Gönen & Alpaydın 2011). Empirical studies have shown that
this alignment emerges early in training (Fort et al.) 2020). with recent works identifying multi-
phase alignment dynamics (Shan Bordelon [02 I Atanasov al. (022), implicit regularization
effects (Baratin et al., 2021), and block-structured NTKs aligned with class semantics (Seleznova &
Kutyniok| Seleznova al. 2023).
For GradPCA, this block structure is particularly important: it enables an efficient approximation of the NTK using only class-mean gradients. This insight underpins the core design of GradPCA, linking NTK alignment theory with practical OOD detection mechanism.
B ADDITIONAL THEORY AND PROOFS
B.1 SUPPORT-BASDE OOD DETECTION
In this section, we provide further motivation for the support-bar based formulation of the OOD detection problem introduced in Section 2
No free lunch for OOD detection, Most the OOD detection literature adopts a probabilistic view, where detector D X' + {0, 1} must decide whether sample X' is more likely to be drawn from the in-distribution pia or some out-distribution Food- However, Mood is unknown in practice. Typically, we only have access samples from Pid, and aim construct detector that performs well across a range of possible, unseen Mooo distributions (e.g., different OOD datasets).
This objective is fundamentally ill-posed: without assumptions on Mood- detector can guarantee reliable performance. The following result formalizes this observation.
Theorem B.1. free lunch) For any OOD detector D and fixed in-distribution pia, there exists an out-iistribution pood such that the performance of D is no better than random guessing.
Proof. Let D =x + (0. to be an OOD detector. Let Au = fo E X A, fI E X: D(x) 1}. Denote Y random variable such that Y 1 X Mood and Y if X ~ Wid- Then we have the following for the detector error probability:
D(x) = 0} and
EID(X) + YJ P[A, O|P[Y = 0| P[Ao|Y 1] 1].
(11)
Moreover, noticing that P[A1 Y 0] (A1) and P[A.|Y 1] (Ao). and assuming that the samples are sampled from in- and -out-distribution with equal probability, i.c. P[Y = 0] = P[Y 1] 1/2, we can write:
EID(X) 1 (mna(AI) + Mood (Ao) = マ (1 + (ia(Ai) - (A1)).
(12)
2
If and D are fixed but Mooo is arbitrary, one can always construct distribution Food such that Food (A)) S (A1), causing the detector to perform no better than random guessing.
Likelihood support. In the probabilistic view of OOD detection, the optimal detector for a given pair (pia, Muma) naturally depends on likelihoods under both distributions. Specifically, as follows from the proof of Theorem B.I. the set of inputs flagged as OOD by the ideal detector should satisfy:
A arg arg (ia Mood( (A1,,
(13)
which yields the likelihood- based optimal detector:
D'(x): = {o; [aa(x) 2 Mood (z))
1, (ia(x) < Mana(1)
(14)
However, this formulation leads to a conceptual issue: the classification of a fixed input x E as ID or OOD may vary depending the choice of poods even though (4a()) remains constant. This
20

---

Published as a conference paper at ICLR 2026
counterintuitive in practical OOD detection, where the goal is determine whether model can reliably predict the label of z. Since the choice of Pood does not influence the model's prediction for it should not determine whether is considered OOD.
In contrast, the support-based perspective avoids this issue by providing an optimal detector that independent of Mooo Here, define 14a as the distribution over inputs the model correctly classify, and any input outside supp(/Ha) is considered beyond the model's competence. This aligns better with the practical objective of OOD detection; identifying inputs where the model is likely to fail.
B.2 SUFFICIENT CONDITION FOR SPECTRAL OOD DETECTION (THEOREM 4.1
Theorem (Sufficient condition for spectral OOD detection). Let X Pia and h X - Rp be any function from 1,2 (1sa). Consider the covariance malrix
(hh == E[x(X)((X)))
(15)
Let Ph((x)) be the orthogonal projection of h(x) onto the range of S(h). For any x E X, if
[[Ph(x)]22 < [[(x))]2 and h is continuous then x is OOD.
Proof. Since the space of matrix is perpendicular to its kernel, v := h(z) Ph(x) is in the kernel of S(h), i.e. S(h)v = O. Hence
(M
|| T M1 = T
-= SSh)o 0.
(16)
Now we will prove the result by contradiction. Suppose x E supp(pia). Note that since
v h()) [1<=)) 1Ph(=)) 0,
(17)
by the continuity of h at x. there exists some ball (xr such that o "h(x')) 2 M()) on B. Since x E supp(uaa), Hia (B. (x)) ) > 0. Thus we have
f
|| ToTTllls (1)2)(m)))
(v h(=)()))² dppa)a)) 2 1 (B.)) (v (oT((z))² >0.
(18)
(x)
However, this contradicts equation
16 Thus x & supp(pia) and hence is OOD.
Remark B.2. Theorem 4.1 has the following properties:
1. Distribution-free: The result makes no assumptions on either the in-distribution Paa or the out-distribution Hands making it completely distribution-Fi free.
2. Deterministic condition: By adopting the support-based perspective, the theorem provides deterministic condition that guarantees a point x e X' is out-of-distributi rather than relying on probbbrrbbbylitty bounds.
3. Flexibility in feature map choice: tinuous function h E (Iia), which may represent logits, activations, gradients, other feature maps useful for OOD detection. This generality allows the result to cover wide range of existing spectral methods.
The theorem applies to any (almost everywhere) con-
B.3 CONNECTION TO PCA (THEOREM 4.2
Before proving Theorem 4.2 we restate simplified version of the Davis-Kahan sin θ theorem Davis Kahan 1970), which plays central role in our analysis:
Theorem B.3 (Davis Kahan sin θ theorem). Let A and B = A E be self-adjoint matrices in PxP, with eigenvalues A 2 let V E Pxk and V E Pxk contain the top-k eigenvectors of A and B, respectively. Define the spectral gap:
Ap and X ... > Åp, respectively. Fix k (1...., P), and
inf A : (I, (-00,A+1))
(19)
21

---

Published as a conference paper at ICLR 2026
Then the following bound holds:
||
(spa(()) (pan(p)) -
(20)
A
In particular, for the corresponding orthogonal projections Pk = VV and Pk = VVT we have:
2| A
||Px
Pxll2 v
(21)
generalization of this result available (Stewart & Sun 1990); the version presented here follows the formulation in (Yu et al. 2015b). We now proceed to the proof of our main result.
Theorem (Robust OOD certificate for PCA). Consider PCA applied to malrix $ E RxPx (e.g.. estimated from a noisy sample). Let h and (h) be as in Theorem 4.1 and assume the following:
[h)h) $11 S e, rank(S(h))> C.
(22)
i.e., S approximates rank-C covariance matrix S(h). Let Pe denote the orthogonal projector onto the top eigenvectors of S, and let Xe be the C-th largest eigenvalue of ((h). Then, for any input E X. the following condition is sufficient to guarantee that is OOD:
SPCA(x) = [Pch(x))12
(PCA(F) <1
(23)
Ac -
|x(x)]2
Proof. We prove the result by applying the Davis-Kahan theorem to matrices S(h) E PxP and E Re PxP, Suppose the eigenvalues of (symmetrie positive defimite) S(h) and $ are, respectively
A d2 ... Ap and Å I A2 S .... dp. From the low-rank condition, have that
Ac+1 = = 0.
To show the relationship between the score functions corresponding to S(h) and S, we notice the following:
(s(z) - SPCA (z)) = [1)(()) -
(1)
- I Pollz.
(24)
|
where the inequality follows from the observation that projection matrices symmetric and idem- potent:
P= PT = P2
(25)
Now we notice that the spectral gap defined in equation 19 is given by:
A = inf 1| A [Ac A1), A (c+1) d sct1 Ac e,
(26)
since AC+1 zero, and ||S(h) - $112 & e by assumption. Therefore, applying the Kahis theorem, get:
2r
PP- Poll2 S
(27)
Ào -
Together with equation 24, this completes the proof, this implies the following for the score functions:
2x
(s(x) (xc)(x)
(28)
Ac -
Finally, from Theorem 4.1 we know that the following condition is sufficient to guarantee that X is OOD (i.e., x & supp(pa))
(a)) < 1, s(x) = IPh())
(29)
where P is the projection on the range of S(h). Moreover, by equation 28
we have:
2e
s(x) < SPCA (x) +
(30)
Therefore, the sufficient condition in terms of SPCA (x) is given by
2c
2e
(PC + λc
# SPCA(T) < =1-
(31)
do
n
22

---

Published as a conference paper at ICLR 2026
B.4 SAMPLE COMPLEXITY OF COVARIANCE MATRIX
Before proving the main result, we first state the following concentration bound for empirical covariance estimation:
Theorem B,4 (Rudelson- Versohyin (Rudelson & Vershynin 2007
Thm. 3.1)). Let Y be a random
vector in Rd satisfying || M almost surely and | Y
I 1. For independent copies
Yi...., YN set
N 5
log N
I' = N YYy,
E[ 8 Yr
a CM
1=1
where C is universal. Then
(i) (Expectation) I 1 one has E | 2/22221 S
(ii) (Tail) For every E (0, 1), Pr N ll > t) S 2 eexpl-cf/po where c > 0 universal.
Now are ready to state and prove the main result of this section:
Theorem B.5 (Sample complexity of covariance matrix), Assume [|/(x)] & for all a E and let
1 N S(h) = N 1=1
Σ ((Z:)((Z))T
(32)
be the approximation of Sh) from sample of size N. Denote SPCA and śPCA the PCA score
functions for S(h) and S(h) respectively (as defined in Eq. equation 9 probability at least 8 we have the following:
Then for any 5 E (0, 1) with
2E
2CR2
log(N/5)
(SPCA(x) (SPCA())) S
A A-E' E
(33)
A
N
where C > 0 is a universal constant, and A = Ac AC+1 the spectral gap of S(h).
Proof. We start by defining rescaled feature maps h(x)) k(w))
vA where A1 is the largest eigenvalue of Additionally,
S(h). This rescaling ensures
[[E[h @ Al]l2 [|(( 1, as required by Theorem B.4
ft. almost surely. Then, applying Theorem B.4 with M have [l /x; for any parameter & E (0, 1):
obtain that
Vx."
PN -C M²
Pr
Σ (ax)) s M(x.) EiT ii)
>
< 2exp
N
where is universal constant.
Setting / C"
logí N/6)
for an appropriate constant C², we get:
N
N
log (N((N))
Pr
Mxx) & - > C
S
N
1=1
Rescaling back to the original features, we have:
[i(hh) S(h))l2 A
Σ Ma.) h(x)) E(h sh)
SC'RVAI
log(n/0)
7
Since A < ||||| S R² (by our assumption that [[h(z)]l2 I R), we can further simplify this to:
[og(N/6)
[S(h) - S(1)l2 S CR2
N
for a universal constant C.
The remainder of the argument follows analogously to the proof of Theorem 4.2
23

---

Published as a conference paper at ICLR 2026
B.5 NECESSARY CONDITION FOR SPECTRAL OOD DETECTION (THEOREM 4.5)
Theorem (Necessary condition for spectral OOD detection). Consider the same setting as in The-
orem 4.1 and the spectral OOD detector Dr. defined in Eq. equation with arbitrary threshold
5 E (0, 1. Then the following condition is necessary for the efficiency (Le., non-zero sensitivity) of
Da:
rank(S(h)) < ((im(((x)::)))) : x)))
(34)
Proof. As per Theorem 4.1 the range of (hh) has to be included into the full image of the feature
map span( (a())) 2)) C R.P. we have
ra))(((())) & dim((f(a)) (((()) x E x))).
(35)
In case of equality, we have ||Ph(z)] [|b(x)]] for all X. Therefore, a detector of the form
(x(x)) == (0.0) (s(x), s(x) = [PP(<>)]?
(36)
||x())]1
does not identify single OOD point and, thus, has sensitivity (true positives rate) equal to zero. Therefore, by contradiction, strict inequality is necessary for effectiveness of Dn.
C IMPLEMENTATION DETAILS
In this section, we describe the implementation of GradPCA and all baselines, along with details about the pretrained models used in our experimental setup.
C.I GRADPCA IMPLEMENTATION
We implement GradPCA custom class compatible with JAX transformations, leveraging the struct module from the Flax library (Heek et al. 2024]. We chose JAX (Bradbury et al. 2018) (with Flax as the deep learning framework) for two main reasons: (1) its automatic differentiation system provides a clean, expressive interface for working with neural network gradients, which central our method; and (2) its just-in-time (JIT) compilation enables competitive runtime performance with minimal low-level optimization.
The GradPCA class includes several configurable parameters that control which variant of the method is executed:
Method: Specifies how principal directions computed. We support three options:
1. Block structure (default): Computes directions using class-mean gradients, as described in the main text.
Batch: Computes gradients over batch and estimates directions via SVD of the dual matrix.
3. GradOrth: Implements the GradOrth method from recent literature (Behpour et al. 2023) (also included in the baselines) using SVD of the covariance matrix of the last-layer activations instead of the NTK for estimating principal components. for consistency with the original GradOrth paper, this variant uses loss gradients (with uniform vector as label) rather than output gradients.
Parameter subset: Specifies which subset of model parameters to differentiate with respect to (as list of keys in the parameter dictionary). In main experiments, we use only the parameters from the last hidden layer. Additionally, we include ablations for CIFAR-I0 (see Appendix E.
Spectral threshold 6: Used to truncate the spectrum of size C at the smallest K such that E >// Ee A 2 e. The default = 0.99 typically retains the full or nearly full spectrum on moderate-class datasets such as CIFAR.
Aggregation: A boolean flag controlling whether to aggregate gradients across all C' output heads (via summing) before PCA (when True), or to compute separate scores for each head (when F Palse). While the non-aggregated version allows finer analysis, it is computationally demanding and practical only for small-class datasets like CIFAR- 10. Our main benchmarks use the aggregated version.
24

---

Published as a conference paper at ICLR 2026
Sparsification (DICE): An option to sparsify gradients before PCA, roughly following the DICE method (Sun & Li 2022). Sparsification based on parameter magnitudes (not gradient values) and requires specitying sparsity level p. which determines the fraction of parameters removed. The method applies sparsification independently to each parameter matrix.
The GradPCA variants discussed in the main text correspond to the following configurations: the default GradPCA uses the block structure method with 0.99, the parameter subset restricted to the last hidden layer (classifier) parameters, and aggregation enabled. GradPCA-Bated uses the batch method with c = 0.97. GradPCA-Vec disables aggregation. GradPCADDICE enables the sparsification option with the default sparsity parameter p 0.8. In the following paragraphs, describe the high-level logic of the training and inference procedures for GradPCA
Training (offline) phase. Training is initiated upon instantiating the GradPCA class and executed over the input dataset. At the end of training, the computed principal component vectors are stored as attributes of the class for use during inference. The procedure requires a training state (i.e., model and parameter values) and GradPCA configuration parameters. The input dataset must be provided as a function that returns a per-class data loader, allowing compatibility with varied loading pipelines. by sequentially evaluating gradients over batches
The training loop computes gradient of configurable size. This implementation is fully scalable to large datasets and has computational requirements comparable to single training epoch. Once the gradient Mass-means computed, the remaining steps follow directly from Algorithm [
Inference (online) phase. During inference, the principal components- -stoedd as attributes of the GradPCA instance- are used to compute scores by projecting the input gradients onto these components. Since inference speed is critical for OOD detection, efficient parallelization plays key role. The optimal strategy depends on factors such as batch size, the size of the parameter subset, the number of principal components, and hardware capabilities. In practice, it may be more efficient to parallelize over either the batch dimension or the principal components. Our implementation supports both options, though for consistency, we use parallelization over principal components in our runtime evaluations (Appendix D)
C.2 BASELINES IMPLEMENTATION
For most baselines exxcludns-excludingg gradient- based methods GAIA and GradOrth-a we use implementa- tions from the Detectors library (Dadalto (003). This ensures consistent, efficient, and reproducible comparisons. To emphasize fairness in our main evaluations, we apply each baseline with the same hyperparameters across all benchmarks (matching our use of single GradPCA variant without tuning). Below, we detail the specific settings used for each method.
Max logits. parameters.
This baseline uses the maximum logit value as the OOD score and requires no additional
ODIN. We set the softmax temperature to 1000, the default in the Detectors library.
Energy.
The softmax temperature is fixed at 1.0, following the library's 's default.
DICE. We the sparsity parameter to p- 0.7, which specifies the fraction of weights removed from the last layer. This value is the default in Detectors and follows the recommendation from the original DICE paper (Sun & Li 2022), particularly for ImageNet. As there is no principled strategy for selecting this parameter, we adopt the known optimal setting for ImageNet across all experiments. In most cases, this baseline performs reasonably well. However, we observe that it completely fails on the 10 BiT-M model, resulting in an AUC of just 35% while other logit-based methods succeed. This illustrates a failure mode where the core assumption of DICE that pruning small weights at a given sparsity level improves ID-OOD separation- not hold.
Mahalanobis.
We extract features from the penultimate layer and estimate the conditinnii means and covariance matrix from samples (using mean aggregation and sample covariance, in
25

---

Published as a conference paper at ICLR 2026
Detectors). Since this method is computationally intensive, the full training data for CIFAR, but limit ImageNet training to a report average performance along with standard deviations over three independent runs.
subset. For the ImageNet benchmark (Table 2). we
We use penultimate-layer features and evaluate based on the 10 nearest neighbors. Because KNN inference time scales poorly with dataset size, we subsample 10,000 points for training in all settings to maintain acceptable runtime-ever for CIFAR. report average performance along with standard deviations over three independent runs in the main benchmarks for this baseline.
KNN.
ReAct. We apply ReAct to penultimate-layer features, using a clipping threshold of 0.9 the default value in Detectors. As with Mahalanobis, we use the full training split for CIFAR and a 50,000-sample subset for ImageNet. We report average performance along with standard deviations over three independent runs for ImageNet. Similar to DICE, this baseline performs well on several benchmarks but fails completely (AUC 50%) on the CIFAR- 10 BiT-M model (where DICE also fails) and the CIFAR-100 ResNet-34 model. This highlights the inconsistency of ReAct's core assumption- least under a fixed hyperparameter setup across evaluation.
GAIA. GAIA is not included in the Detectors library, but the authors provide open-source code. While their implementation is architecture-dependenn they include support for ResNet-34 and BiT models, which enables direct comparison using their original setup.
GradOrth. GradOrth (Behpour al. 2023) a recent gradient- based spectral method that differs from GradPCA in two key ways: (1) it uses gradients of the loss function rather than the model output, and (2) it derives dominant directions via SVD on the last-layer activations, rather than NTK structure. Since no official implementation was released, we reimplemented GradOrth within the GradPCA framework based on the paper's description.
Revisited PCA. This baseline applies PCA to the penultimate-layer features and computes regularized OOD score following Guan et al. 2023). We set the retained variance threshold to 0.9, as recommended in the original paper.
Projected Gradients. Following Wu et al. (2024a), this method projects gradients onto a low- dimensional subspace spanned by class-mean gradients. An auxiliary
of BatchNorm layer followed by a single linear layer-is then trained on these projected gradients, and OOD detection is performed using the Energy score applied this auxiliary network. As suggested in the original work, we train the auxiliary network for 3 epochs, using the full ID dataset for CIFAR benchmarks and 10% of the ID data for the ImageNet benchmark.
Kernel PCA (CoRP). CoRP applies Kernel PCA using cosine Gaussian kernel approximated through random Fourier features, with feature map @(z)
where the cosine normalization is given by ocus(2) (x) 2/ || la (Fang et al. 2024b). This method involves several hyperparameters, including the number of random features M, the Gaussian kernel parameter 7, and the variance retention threshold Because performance is highly sensitive to 7, we evaluate E 0.5, 1.0, 3.0 and select 7 = 0.5, which yields the best results on ImageNet. Consistent with the original paper, we M
4096 and set e 0.9.
NCL. The Neural Collapse Inspired (NCI) detector (Liu & Qin 2025) computes a score by projecting the centered penultimate layer feature of an input onto the last- last-layer weight vector corresponding to the model's 's predicted class on this input. We use the basic NCI variant, which omits the additional filtering step based on the penultimate layer feature norm, and therefore hyperparameter- free.
C.3 MODELS
We use two classes of models in our experiments: ResNetV2 models from the Big Transfer (BiT) repository (Kolesnikov et al. 2020: Beyer et al. 2022),, and ResNet34 models from the Py PoTcch Image Models (TIMM) library (Wightman] 2019). Since GradPCA is implemented in JAX and most baselines are implemented in PyTorch, our comparison setup relies on translating shared models between the two frameworks.
26

---

Published as a conference paper at ICLR 2026
R2 (BiT). BiT models are publicly available and widely used in the OOD literature. These are ResNetV2 architectures pretrained either on ImageNet- 1k (BiT-S) or on ImageNet-2IK (BiT-M), with additional BiT-M variants fine-tuned for smaller datasets such CIFAR (after pretraining). We use these released models in our experiments.
BiT models supported in both JAX and PyTorch. We the official implementations for both frameworks, enabling a direct comparison between GradPCA and baselines. We confirm that the Torch and JAX versions of these models yield nearly identical accuracy, ensuring consistent evaluation.
Res.ee33 (TIMM). The TIMM library provides a wide range pretrained models, including ones for smaller datasets like CIFAR, while many popular frameworks offer only ImageNet-pretrained versions. We select the ResNet34 model to align with the setup of the GAIA method (Chen et al. 2023), which reports strong results on this architecture. However, we note that their reported performance is based on a custom custom-traines trained version of ResNet34, which we cannot replicate using the standard TIMM model.
TIMM models are not directly available in JAX, so we implement a custom conversion. Despite matching the architecture exactly, direct translation leads to a noticeable accuracy drop-liket) due to numerical inconsistencies and subtle differences in layer implementations between Py'Torch and Flax. For instance, the converted model achieves 93% accuracy on CIFAR-10 (vs. 95% PyTorch) and 72% on CIFAR-100 (vs. 78%).
To ensure fairness, we fine-tune the converted JAX models for 10 epochs using float32 precision and small learning rate, correcting for conversion artifacts without altering the weights significantly. We then convert the fine-tuned model back to Py Porch, which restores the original performance, This strategy guarantees consistent model behavior across frameworks. We release all model checkpoints (both JAX and PyTorch) with our code.
D COMPUTATIONAL COSTS
In this section, we break down the computational performance of GradPCA and compare it with baseline methods. The total cost involves two key components: the training (offline) phase and the inference (online) phase. While online efficiency is critical for real-time deployment, the offline phase must also be scalable enough to run on standard hardware. We discuss each phase in turn below, and also review memory requirements and provide an estimate of the total compute time for the project.
Inference (online) phase. The inference-time performance of GradPCA and baselines is sum- marized in Table 3 GradPCA highly efficient on CIFAR, evaluating over 2,000 samples per second-well well within the range for real-time deployment. However, like all methods, its performance degrades on ImageNet, which presents significantly greater computational challenges.
CIFAR-10
CIFAR-100
ImageNet- Ik
Methods
Training
V2-55
Ne134
ResNeiV2-50
ResNet54
RosNetV2-101
Res NeIV2-0
Max logiss
2825 s/s
50TB 375
2572
13317
425
127 s/s
s/s
5565
2825
475
ODIN
2817 s/s
5060 s/s
2502
5436 s/s
456 s/s
473 s/s
Energy
2815 s/s
5750 s/%
26488
5123 4/s
458 s/6
476 s/%
DICE
2815 s/s
5731
2703
5141
457
476
Mahalanobis
2539 s/4
4015
2108
353
376 a/s
KNN
443
1462 s/s
449 s/s
1476
89 c/s
80 g/s
ReAct
2809 s/s
5413
2838
5117
450 s/s
184 s/s
GAIA-A
247
129
245
GAIA-Z
317
489 s/s
307 s/s
172 s/s
309 s/s
350 a/s
GradPCA
2654 samples/s
2440 samples/s
1974 samplesis
2311 samples/s
103 samples/s
128 samples/s
Table 3: Runtime of OOD detection methods evaluated on NVIDIA RTX A6000. Each method allows batch-evaluation, the results in the table show the average runtime per sample when evaluating batches of size 128. The time includes standardized data loading routine (TFDS loader).
Our implementation parallelizes GradPCA over the principal components, making its runtime rel- atively insensitive the number of classes components, provided sufficient GPU memory is
27

---

Published as a conference paper at ICLR 2026
available. The negligible difference in runtime between CIFAR-10 10 and CIFAR-100 supports this. The slowdown on ImageNet is primarily due to the increased cost of a single projection, resulting from the larger parameter matrix in the last hidden layer. This cost could potentially be reduced by further limiting the subset of parameters used in gradient computation.
Training (offline) phase. The offline phase of GradPCA involves computing average class Ja- cobians over an input dataset (or subset thereof). Jacobians are evaluated batch-wise wise, with configurable batch size. As result, the training time scales approximately with the dataset size divided by the batch size and is comparable to the time required for a single training epoch.
In our experiments, using the ResNerV2-50 model and an NVIDIA RTX A6000 GPU, we observed the following training times over the full datasets: approximately minutes for CIFAR-10, 10 minutes for CIFAR-100, and 90 minutes for ImageNet-Ik.
Memory requirements. Like other regularity-based methods, GradPCA incurs memory overhead for storing the principal component vectors. The number of components retained depends on the spectral threshold e, but we can estimate the maximumm memory requirement as O(CP), where C is the number of classes and P is the number of parameters per class-specife vector.
In our setup using the ResNetV2-50 model (float32 precision, accounting for the fact that scales with C' in the final layer), this corresponds to the following worst-case memory usage: approximately 7.8 MB for CIFAR-10, 78 MB for CIFAR-100, and 7.5 GB for ImageNet. While the worst-case cost on ImageNet is relatively high, the actual memory usage in most experiments is much lower. This is because the default 0.99 threshold typically retains fewer than half of the principal components.
Total computational cost. The main computational cost of our project comes from training on the ImageNet benchmarks, including ablation studies. Specifically, we conducted 15 ImageNet each for ResNetV2-50 and ResNetV2-1111, totaling approximately 45 hours of training time on an NVIDIA RTX A6000 GPU, with under 5 hours needed for evaluation. In comparison, the CIFAR experiments required significantly less compute, amounting to fewer than 10 GPU hours in total. Overall, we estimate the total computational cost of our results be under 60 GPU hours on an NVIDIA RTX A6000.
E ADDITIONAL EXPERIMENTS
This section presents additional experimental results included in the main paper, specifically the CIFAR-10 benchmark and ablation studies on GradPCA variants.
E. CIFAR-10 BENCIMARK
The setup of the CIFAR-10 10 benchmark is identical to that of CIFAR-100, deseribed in Section 5 As shown in Table similar to the CIFAR-100 results, regularity-based methods achieve state-of-the- performance the pretrained model, while abnormality based methods perform better the non-pretrained model, with GAIA achieving the best results.
Due the smaller number of classes, this benchmark is significantly more computationally accessible than the others. This makes it feasible to apply GradPCA- and GradPCA-Batch, as well as to perform GradPCA on a much larger subset of parameters beyond just the penultimate leyur-resulting in improved performance. While these variants may be impractical for benchmarks with a larger number of classes, we recommend their use on smaller benchmarks to fully leverage the capabilities of GradPCA.
E.1.1 NEAR-OO SETTING
To further assess GradPCA's robustness under more challenging conditions, we complement the experiments standard OOD detection datasets in Table 4 with OOD benchmark in Table 5 using CIFAR-10 10 as the ID dataset and CIFAR- 100 as the OOD dataset. As expected, GradPCA's performance decreases slightly in this more difficult setting, similar to the behavior observed for the
28

---

Published as a conference paper at ICLR 2026
SVII
Flave
LSEIVE
LAUNT
ISEN
TINTRINN
AVErRIN
AUSKK
ATBEX
TIMES
AUNIC
PEROALE
SUBOUTT
AUPOET
Stor
21.41
03:12
04:41
5:3
23:1
07.43
10.00
01.36
31.4
97.31
18:15
10.13
14.11
26. 11
Taurg
03.11
1333
05.53
03:33
06.90
11:35
-8.07
91.00
63.86
78.05
30
07.03
12:81
14
6/0
08:13
144
996
6.0
08.55
17:34
07.01
16:34
07.13
0.82
06-07
8.1
19.00 18:13
3okd
71.34
28
7135
13:33
15005
45.35
99350
19:43
7126
03
41/2
33:00
GALL-2
41:33
07
18.14
90.31
26.30
$2.53
18:34
01.7
15.35
13.23
77:09
$1.16
75:14
Revisked PCA
23.00
08.20
$2.35
36.36
$3.35
2338
94.00
04.53
34.55
Pey. Creah
15:34
10.10
36.32
1.3
$5.24
13.01
$3.77
26.13
16.76
11:3
16.00
15.34
20.33
14.33
1616
54.33
H036
17:31
14.10
9671
13.4
277
1134
CraAPPCA (Mtck
2,31
23.31
1806
97:35
581
95.
3580
16,00
1387
18.40
0931
1045
16.
For -
10:60
67 0/
16.77
41335
1714
9675
11:00
43.
11.
90.17
11T
66:3
$5355
NVHN
Raa
LSEN4
ESUN-
IENN
Baltro
Avtrage
AUROCC1
AUBOCC
ALROCC
AUBOC
AEROCT
AUBOCT
Mas light
42.5
23.45
04.35
25,05
31.02
MEP
37.47
5-45
$7.5
20.6
$1:11
01.0
59:53
$2.60
(7:33
60.55
Caugg
21.13
10.13
4112
21.37
$4.85
21.6
123
2021
4113
10.00
13.55
1638
BocNee 34 (TTTNN
600
5763
M432
201
5156
10.4
40.00
4518
91.65
316
9131
13.08
1043
11:07
00.65
3533
11.11
03.00
1158
65.
6.00
08.79
11
137
110m
47.0
1136
97.1
18
96.11
1337
DTTE
Ronseal PEA
24.05
04.00
15.11
3090
$8.05
$3.13
18.3
05.9
43.3
50.11
1333
92A
Nareel PCA (CORP)
00.10
11.30
03.01
56.3
0437
13.51
9::11
4343
05.15
12:35
0318
33.55
5-5g
3151
3.31
91.0
03:37
0339
93.81
3353
0355
GratITAA (Mack.3)
2741
SLAT
85.13
1106
1771
1T11
0772
1911
16.55
415
553
$4.34
39:44
9126
4310
3333
12.87
25.33
21.7
35.35
19:11
4633
$3.32
Table 4: CIFAR-10. For each architecture and evaluation metric (FPR95, AUROC), the best- performing method is shown in bold, and the second and third best methods are underlined.
baselines. However, no critical degradation is observed, confirming that this near-OOD scenario does not compromise the method.
Model
Metric PPRGS
MaxLogits
MSP
ODIN
Energy
DICE
Mahalanobis
KNN
ReAdt
GAIA-A
GAIA-Z
GradPCA
51,95
95.07
29.39
90.02
25.67
BIT-M
AUROC
81.19
85.71
81.20
78.77
48.24
93.56
96.00
44.21
88.72
53.02
93.68
TIMM
PPR93
53.73
62.71
53.77
3182
55.71
69.19
34.04
57.87
35.12
31.78
53,86
AUROC
81.43
83.90
81.43
81.47
27.78
85.37
89.36
76.08
86992
84.63
86.32
Table 5: CIFAR-10 as ID vs. CIFAR-100 as OOD. Near-OOD detection benchmark. For each architecture and evaluation metric (FPR95, AUROC), the best-performing method is shown in bold, and the second and third best methods are underlined.
E.2 VIT-B/16 IMAGENET BENCHMARK
Since many recent OOD benchmarks include Vision Transformer (ViT) models Dosovitskiy et al. (202T), we have evaluated GradPCA on ViT-B/16 in addition to our main benchmarks. We used a publicly available checkpoint pretrained on ImageNet-21k and fine-tuned on ImageNet- 1k from the Vision Transformer repository.
As shown below, GradPCA remains effective and competitive on this architecture. We note that prior work has observed that OOD detection performance is generally comparable between ResNetV2 (BiT) and ViT backbones (Zhang et al., (003). Since GAIA is not directly applicable to ViT models and GradOrth does not provide computationally feasible implementation (see Appendix C.2) we did not include these methods in the following table.
E.3 CONSISTENCY WITH RESPECT TO RANDOM SEED
In addition to the inconsistencies in OOD detection performance arising from architecture and training setup, discussed in the main text, recent work has shown that OOD performance can also vary substantially with changes to the random seed governing training stochasticity (Fang et al. 2024a). To assess the impact of this factor on GradPCA, we trained a VGG16 model on CIFAR-IO 10 using ten different random seeds, yielding models with nearly identical accuracy but converging to different local minima. We then evaluated GradPCA on each of these models. The results, presented in Table 7 show that performance varies only mildly across seeds (within roughly a 3% AUC range).
29

---

Published as a conference paper at ICLR 2026
INaturalist
Textures
Averuge
FPR95
AUROC
FFR995
AUROCT
FDRS5 | AUROC
FFR95
AUROC
FPR95
AUROC
Max logtis
61.92
K3-10
55. 17
55.12
11:37
07.54
$2.50
86.87
45.75
29:41
MSP
71.23
79.32
66.59
80.92
22.05
94.92
62.07
50.00
55.59
83.79
ODIN
13.27
VIT-B/16
Energy
89
11.29
12.90
57.37
87.80
90.22
DICF
41.81
90.60
95.73
86.96
41.47
Mahalanobiis
67.60
K1.76
61.12
85.68
4.75
99.00
54.3 14.30
86.62
46.94
88.20
KNN
69.02
58.23
62.08
86 06
12.48
97.63
50.30
87.57
48.47
88.37
ReAct
36.72
55. 64.84
42.86
89. 29
14.90
97.29
45.69
87.65
41.29
90.22
GradPCA
30.86
89.02
12.80
91:37
3.74
99.16
59.94
86.21
39.84
91,42
Table 6: ViT-B/16 on ImageNet-lkk. For each architecture and evaluation metric (FPR95, AUROC), the best-performing method is shown in bold, and the second and third best methods are underlined.
No drastic fluctuations are observed, indicating that GradPCA exhibits stable and consistent behavior under changes in training randomness.
$7.535
$0.273
$6.612
$7.827
48.508
02.049
-9.078
91.285
52.257
90.543
65.518
87.062
54.920
$9.520
10
63,678
87:551
56.30 66.361
83.348
58:023
89.334
62:550
85.629
61.311
86.626
67.827
83.686
63.380
$6.200
60 -1155
92:240
52.653
35:251
43950
92.246
55.
13:258
85.811
56.352
84.844
60.301
87.054
58.000
55.070
1891
19.188
50.613
72.045
81.252
56.577
90.945
04:052
194
06.
85.471
22.887
54.387
64.291
Scals
56.362
S6.349
90.678
$9.593
18000
34.343
20000
64,019
92.491
67.53
34.056
45
179
61.071
56.935
42000
92.200 2001
700
48
91
59
34V
63:
59.63
100000
37.307
93.337
64.232
85.927
43.109
92.43
59.856
85.059
62. 466
87.044
72.177
62.442
$6,496
68.207
Table 7: CIFAR-10. Ablation over random seeds.
E.4 ABLATION: PARAMETERS SUBSET SELECTION
A key property of GradPCA, which contributes its generality, is that it can be applied any subset of network's 's parameters. Prior work on NTK aligmment has shown that alignment strength varies considerably across layers (Lou et al.) 2022: Baratin et al.) 202T): early layers typically exhibit weaker alignment, whereas later layers show stronger alignment, with the maximum often occurring intermediate layers. This suggests that applying GradPCA to intermediate layers may be particularly effective.
We therefore conduct ablations in which GradPCA is applied to different parameter subsets of the BiT-M and TIMM models CIFARI0 10 and CIFAR 100. Table reports the results for CIFARIO, where GradPCA is applied separately to the parameters of each residual block ("blocks 1-4"). As shown, intermediate blocks indeed yield strong performance, often exceeding that of the default version. For BiT-M, the best results occur at block 4, while for TIMM they occur at block 3. The performance degrades substantially for the earlier layers (blocks 1 1-2), indicating that their gradients are less informative for OOD detection, likely due to weak alignment.
Results for CIFAR100 are presented in Table 9 To manage computational cost, each block is further divided into "units", each containing two convolutional layers. Similar to CIFAR 10, the best performance is achieved using intermediate layers (block 3, unit for BiT-M and block 4, unit 2 for TIMM).
Although the most informative parameters consistently reside in later blocks, the exact choice of the optimal subset does not transfer cleanly across models or datasets, complicating layer selection for new architectures. Our practical recommendation is therefore to rely on empirical evaluation when selecting parameter subsets. We also note that understanding how NTK alignment and gradient representations evolve across layers remains an open problem in deep learning theory, and progress in this direction may further inform optimal layer selection for GradPCA.
E.5 ABLATION: TRAINING DATA FRACTION
Since GradPCA relies on approximating gradient class means using ID data, is natural to ask how sensitive the method is to the amount of available training data, and whether computation can be reduced by using only a subset. To investigate this, we performed an ablation in which GradPCA was trained varying fractions of the ImageNet training set using the BiT-S model, and we evaluated the
30

---

Published as a conference paper at ICLR 2026
SVIIN
Planes
LSU.e
E-SUNer
SUN
Textures
Asrrage
ATTROC
FPRUS
AUROC
FIRRY
ATTRCC
AUROC
AUTROC
TERUT
AUROR
ATTRIC
GrodICA (block 4)
5:11
99.01
10.96
99.25
$6.49
96.45
0.62
$9.57
30.43
96.00
GrodICA (block 3)
8,10
08.45
21.54
55.T3
9.57
98.45
60:11
58.51
55.57
85.05
0.36
99.53
35.71
05.08
GrodICA (blocck)) 3)
58.96
07.00
5.30
98.53
47.76
91.15
85.12
72.10
$5.08
73.59
0.75
$9.60
47.66
87.06
GeadiPCA (block- 4)
48.46
M2.41
50.90
90.58
16.23 30
01.20
58.58
90.00
52.31
01.75
64:00
50141
55.06
ME-41
GondICA (black 3)
15.00
3187
11.06
19
06.50
25.75
98.66
95-43
GradICA (black
09.52
21.75
95.10
51.95
95.84
41.35
96.65
47.89
95.86
51.52
90.54
5051
55.14
44.15
Table CIFARIO. Ablation of the parameter subset choice using residual blocks of ResNet50-V2 (BiT-M) and ResNet34 (TIMM).
SVHIN
Plane
ESUINe
ESUINe
ISUUN
Tysteres
Average
ALTROC
FF305
AUTOCC
GratPCA (lbck 6 unit
11:26
04:10
00.41
00:23
40
43.52
92.83
50:32
11.80
62.27
9.08
97.03
45:26
10.U0
GradPCA (Mock
93.05
67
4.67
GradPCA (lack Kosttn
11.50
07:3
48 03800
97.16
90
95.11
78.00
79:62
77 00
78.01
0.76
99.67
40.17
GratCAA [lock 2. umit
30.27
20.75
33.30
05.23
13:21
77:35
1.26
09.53
GraPPAA (1ock Nostt 31
12.07
50
3.11
177
43
97.00
17.
05
34.50 57
43.79
0.05
00.00
15.33
07:
GratPCh (Mack 3 21
30.75
01.91
15.00
70
61.06
2.75
GoaIPAA (lack usit
18.01
02.75
$7.00
57.01
$7.43
10.01
57:30
4b
$7.53
52.57
15123
17.41
GraPPCA 1llck usti
$6.04
79.05
76.74
$0.99
$3.66
79.64
78
76.33 55
75.06
50.25
06.55
81:
GraIPCA (Mock
70.04
20:00
13.80
35.00
91.33
79.61
GndtEA (lbck
GratiCA unit- 41
$4.27
08.96
122
59.35
87.30 79
50.75
03.04
57 26
91.23
02.18
53.00
85.25
$3.02
63.3 63:71
50-40
101
00:27
11.44
09.13
44:31
$1:49
16.30
(Mock
GraPPCA Blick 3 usit 1)
77.31
78.50
96.55 53
56.05
77.75
68.45
97.31
$3.91
96 18
16.98
05.25
73.51
15.03
05.45
Table 9: CIFAR-100. Ablation of the parameter subset choice for GradPCA using individual units in blocks 3 and 4, for BiT-M ResNet50- V2 and ResNet34 (TIMM). All numbers rounded to two decimals.
resulting OOD performance. The results are shown in Table 10 Notably, training with as little as 10% of the ID data yields similar performance to that obtained with the full dataset, demonstrating that GradPCA is robust to this choice and that gradient class means can be reliably estimated even from relatively small data subsets.
Places
SUN
INaturalist
Textures
Average
FPR95
AUROC
FPR95
AUROC
FPRYS
AUROC
FPR95
AUROC
FPR9S
AUROC
10%
60.36
81.649
48.42
87.208
16.32
96.029
17.365
96.316
35.62
90.30
20%
61.93
81.327
50.44
86.853
17.80
95.731
17.827
96.254
37.00
90.04
30%
61.33
81.757
49.21
87.233
17.38
95.835
17,809
96.186
36.43
90.25
40%
81.781
87.345
17.63
95.758
17.738
96.211
90.27
50%
61.4 48
81,650
49.73
87.139
18.54
95.693
17.987
96.148
36.93
90.16
60%
60.90
81.738
49.03
87.142
17.84
95.742
17.489
96.261
36.32
90.22
70%
61.58
81.685
49.53
87.160
17.62
95.849
17.720
96.212
36.61
90.23
80%
61.40 63
81.761
49.51
87.208
18.11
95.758
18.111
96.159
36.84
90.22
90%
61.61
81.565
87.083
17.95
95.766
18.058
96.149
36.87
90.14
Table 10: ImageNet (BiT-S). Ablation of the training set fraction used fit GradPCA.
E.6 ABLATION: RETAINED VARIANCE THRESHOLD
In the default version of GradPCA used throughout all benchmark tables, we set the spectral threshold parameter to E = 0.99. For datasets with a small number of classes-such as CIFAR- 10 10-performance is largely insensitive to this parameter, any 0.90 typically results in selecting all C - principal components. However, for datasets with a large number of classes, the choice of e can have more significant effect. To assess its impact, we conduct ablation studies on ImageNet (Table IT) and CIFAR-100 (Table
We draw two main observations from this ablation:
1. The optimal value of e varies across datasets and model architectures.
2. GradPCA's performance largely stable under reasonable changes to
While tuning may lead to small gains losses (typically within 1-2% in detection accuracy), there is no clear strategy for selecting an optimal value in advance, as its impact appears to depend on the specific task. Based these findings, recommend choosing the largest that remains
31

---

Published as a conference paper at ICLR 2026
Textures
Average
Methods
FPR35
AUROC
FPR55
AUROC
FPRSS
AUROC
FURSS
AUROK
FURSS
AUROC
GradPCA 0.900
85.96
66.47
78.12
74.95
401.88
92.00
49.33
$3.96
63.57
$0.00
BTE
GradPCA
GradPCA 0.97)
82.25
70.63
73.70
78.39
38.56
92.77
53.66
$8.28
62.04
82.52
GradPCA 0.900
83.12
71.55
75.16
78.68
45.84
91.66
60.74
86,49
66.22
82.10
GradPCA ( 0.900
72.39
77.45
61.88
83.74
32.52
92.73
19.28
95.92
46.52
87.46
BITS
GradPCA 0.96)
GradPCA 0.97)
66.08
80.12
55.28
85.66
20.98
95.09
18.38
96.25
40.18
$9.28
GradPCA 0.99)
61.27
81.75
49.35
87.20
17.78
95.80
18.08
96.14
36.62
90.22
Table 11: ImageNet-Ik lk: Ablation of the parameter.
SVHIN
Place
Nor
ISUI
Testume
Aawrag
Gr 630
21122
15.35
11:33
$3.04
13311
13.40
05.97
4.31
192155
01:30
V1170
GroPPCA 0.50
19:20
05.95
28.83
53.71
5.71
98.37
13.43
55. 10
59.00
68.00
3.43
09:25
25.75
93.68
GroPPCA 0.97
17.44
96.35
23.90
03.12
5.55
58.34
$3.00
83.87
57.91
08.10
3.24
97.25
28.26
53.86
5.38
GraPPAA
87.83
63.32
57.31
70.95
55.38
62.90
87.19
GraPPAA 00.0
56
9047
61.07
87.36
01.43
57.83
13,67
86.1 18
72.86
83.70
70.70
85.03
66
$5.45
GrdPPCA L 0.500
61.22
89.10
00.71
87.31
63.45
57.11
73.97
84.25
73.01
83.85
72.30
84.63
07.78
$604
Table 12: CIFAR-100: Ablation of the parameter.
computationally practical, as this parameter directly controls the number of principal components retained-and therefore the memory cost of GradPCA (see the next section).
E.7 REDUCING MEMORY COST
GradPCA is spectral method, and its memory footprint dominated by the number of retained cigenvectors. A principled way to reduce memory usage is to lower the retained variance threshold e, which controls how many top eigenvectors are kept.
Table 13 reports the number of retained cigenvectors (N) and the corresponding AUROC on the ImageNet BiT-M and BiT-S benchmarks, evaluated different values. These results were extracted from the ablation in Appendix E.6
Table 13: AUROC scores and number of retained eigenvectors (N) for GradPCA on ImageNet BiT-M and BiT-S, across varying retained variance thresholds e.
ImageNet BiT-M
ImageNet BiT-S
N
AUROC
N
AUROC
0.99
568
82.10
0.99
432
90.22
0.97
400
82.52
0.97
274
89.28
0.95
322
82.18
0.95
208
88.82
0.90
221
80.60
0.90
132
87.46
These results show that controlling e allows up a 5x reduction in memory with only modest drop in AUROC on ImageNet.
E.8 ABLATION: GRADPCA+DICE SPARSITY PARAMETER
In this section, we present ablation results for the sparsity parameter p used in the GradPCA+DICE variant of our method. GradPCA+DICE applies a mask to sparsify the gradient vectors, retaining only the top (1 p) 100% of entries (by magnitude) within the selected parameter subset. Table [4] shows results on ImageNet- 1k, and Table 12 reports results on CIFAR-I00.
From Tables 15 and 14 we observe that the optimal choice of the sparsity parameter varies significantly across settings. At the same time, sparsification does not lead to substantial performance differences compared to the default GradPCA in most cases. We fix the default value to p 0.8, as it provides the greatest benefit on the BiT-M ImageNet benchmark. This value is also close to the DiCC-recommended setting of p 0.7, which is considered optimal for ImageNet.
32

---

Published as a conference paper at ICLR 2026
Places
SUN
INaturalist
Textures
Averuge
Methods
FPROS
AUROC
RPRV5
AUROC
FRRE5
AUROC
FER9S
AUROC
FPRUS
AUROC
RouNer (BIT-M)
GradPCA
63.12
71.55
75,16
78.68
45.84
91.69
60.74
$6.49
66.22
82.10
GrudPCA (p- 0.1)
82.71
71.70
74.58
78,90
91.88
GradPCA (r = 0.2)
82.17
71.84
73.63
79.20
42. 30
92.22
56.04
87.68
63.54
82.74
GradPCA -0.3)
81.22
72.19
72.03
79.69
39.42
92.62
52.56
58.43
61.31
83.23
GrudPCA (p- (0.4)
72.72
70,66
83.73
GrudPCA (p- (0.5)
79.32
73.20
69.08
80.84
36.54
93.03
47.59
89.46
58.13
84.13
GradPCA (1.60
78.75
73.83
68,16
81.18
36.05
93.25
45.69
89.88
57.16
84.58
GradPCA (0 (1.7)
36.49
05
GradPCA (p= 0.8)
76.97
75.51
64.82
$2.95
37.68
93.10
44.07
90.40
55.88
85.49
GrudPCA 0.99
76.38
76.41
63.81
83.57
39.55
92.87
43,22
90.73
55.74
85.90
GradPCA
61.27
81.75
49.35
87.20
17.78
9/5 30
18.08
96.14 14
36,62
90.22
ResNerV2-SS0 (BFT-S)
GradPCA 0.11
61.51
81.75
49.57
87.21
95.77
18.22
36.80
90.22
GradPCA 6= 0.22
61.88
81.54
49.89
87.04
18.02
95.74
18.00
95.15
36.95
90.12
GradPCA (p= -0.3)
60.16
82.25
48.20
$7.46
16.97
95.99
18.82
95.98
36.04
90:42
GrudPCA -0.4)
61.09
81.89
49,49
87.21
17.27
95.92
18.95
95.97
36.70
90.25
GradPCA 6- -0.6)
62.24
81.54
50.43
86,92
17,80
95. 81
18.98
95.99
37.36
90.06
GradPCA (== -0.6)
62.51
81.41
50.76
96.84
17.97
95.80
19.05
96.01
37.57
90.02
GrudPCA 0..7
63.20
81.07
51,41
86.59
18.40
95.64
19.00
96.04
38.00
84
GradPCA 6- -0.8
10.10
53,16
85.85
19.25
95 48
18.16
96.20
38.91
GradPCA 6 0.9)
67.17
79.07
35.47
85.09
21.61
94.92
17.19
96.42
40.36
88.88
Table 14: ImageNet-Ik, Ablation of the parameter p in GradPCA+DICE.
SVIEN
Placas
ESUNS
ESSUN
ISUNN
Testores
Anerage
AUBIX
GraIPCA
17.2
96.53
25.04
05.56
5:25
51.75
$8.03
51.05
$7.34
3.41
09.24
27.37
DDCCE
0.13
17.67
56.31
25.3
05.30
0.50
08.00 156
65.00 6.6
99.00
55 2.20
$7.90
3.55
09..00
27.43
16:18
09.90
23.21
9.31
185 23
48:33
18.12
3.53
28.74
14:39
GrWIPCA-DDCE
(-- 014)
17.00
56603
27.02
06.03
8.90
19.33 54
48.00 06
90.00
65.23
3.30
69.34
26.71
01.41
GralPCA
17.7%
38:77
GruIPCA-DMCE
56.55
27:34
A0001 (- 11.7)
17:00
16.57
25.45
10.04
6.36
185.00 06
51.75
11:30
59.00 01
87.40
3.45
19.23
27.00
04.01
18:11
15.37
30:22
622
20.13
67:43
80.27
GeuUPCA-DDCE
GralPCA
61.22
62.71
67.11
73.95
73.01
53.85
72.30
54.53
57.78
(00otPCCA-DD00: (y- 11.3)
59.39
$0-18
03191
17.00
51.01
37 (1
25.75
03.00
75.00
83.75
73.99
$2575
57.08
87.67
Geoatr 1o
10.0
20.24
53.99
98.34
155 mm
75.31
$2.08
83.93
56.08
70.82
85.99
GrWIPCA. DBCC
(.40
55.30
50.45
51.88
67.30
57.05
35 50
75.37
$2.58
73.00
$3.35
34.34
$3.65
65.75
85.66
GruIPCA- (-- 0..0
33.36
50.30
51.00
87.13
58.36
76.76
73
$3.68
71.45
53.71
65.75
85.55
52.96
87.31
53.43
87.0
76.00
$4.35
K3234
16.
CDUC (-- 0.55
01:21
$0.10
67.27
87.84
64.81
35 62
70.59
09 95
65.79
18.24
96. 10
56.34
96.66
G
63.27
87.06
72.96
54.35
$4.98
Table 15: CIFAR-100: Ablation of the parameter p in GradPCA+DICE.
E.9 ABLATION: GRADPCA BEADPCA-BATCH BATCH SIZE
SVEN
Places
L.SNNe
LSUN
ISUN
Textucm
Average
Medhals
1191))
11.2
4133
921TC
10:32
1/31
15.77
15.23
15772
"S7TIC
CradlPCA-Belia
20000
0.54
59.70
3.65
.9.17
2.66
09:35
5.21
99.35
6.85
19.00 69
0.00
60.99
3.68
-9.
(990))
1.55
09:24
3.31
99.27
2.26
6.81
99.01
3.30
16.12
59390
199.9
GraeeL -i-chhchhhhh CVV
(2840)
0.52
09.75
3.30
99.30
2.25
59h45
0.48
09.07
3.55
08.36
11.02
99/90
3.00
99.31
GradiPCA, Baw.hBtckk22 (V
10000
21.47
56.04
51.05
90.33
25.13
05.13
45.06
93.85
48.30
02:12
43,02
91.81
40:1
03 13:24
-3910
35.23
03.99
09.57
25.11
42.16
01.23
60.39
12.
13.72
02.47
42.05
GradIPCA-BBaie Bat.h-hcck22
-000
25155
55.01
14
11.21
34.98
05.51
13
99.30
66.11
02 64
43,07
00.51
37
Table 16: CIFAR-10: Ablation of batch size N in GradPCA-Bare
In this section, we present ablations for the batch size, denoted N. used in the GradPCA-Batch variant of our method. Computing principal components over a batch requires storing N gradient vectors, which is not feasible for large parameter spaces or large batch sizes. In addition, performing SVD on the batch NTK matrix of size N x N scales poorly with N, further limiting practicality. As a result, we apply this variant only in the smaller CIFAR-i0 benchmark, with results shown in Table 16 Nevertheless, where computationally feasible, GradPCA- can provide additional performance gains.
E. NTK BLOCK STRUCTURE FIGURES
In this section, we provide full description of the NTK block-structure illustrations shown in Figure 1. Each heatmap visualizes the kernel values Σ O xy)) computed over all network parameters. The kernels are evaluated on a random subset of the training data containing 12 samples per class, i.e., 2; and xy range over 120 inputs in total. The samples are ordered so that examples
33

---

Published as a conference paper at ICLR 2026
from the same class appear consecutively, i..,, #12c:.... 211c++1 correspond to class C {1, C}. Under this ordering, the diagonal blocks represent within-class interactions. All heatmaps are max- normalized, with white indicating the maximum kernel value and black indicating zero. The models included in the figure are: (1) ResNet20 trained on MNIST, (2) DenseNet40) trained on CIFAR 10, and (3) VGG11 I trained
10. Each model was trained for 400 epochs with batch size of
120 using SGD with Nesterov momentum 0.9 and weight decay 5 10 10-4
E.11
GRADPCA UNDER LABEL NOISE
Labelmiss
Tixt Ace
FFRS5
SVHN
ALEOC
F2R95
Plares
AUROC
FF825
LSUNe
AUROC
FPRM
LSU
ALEOO
FFR95
ISLIN
AUROC
FF535
Textures
AURCC
FPRUS
AUROC
10%
59.25
59.43
$2.84
77.08
2016
101.32
71.59
69:11
13.95
75.57
53
76.31
30%
0.63
62.30
14.48
85.43
68.79
93.45
72.01
83.85
70.33
82.41
70,80
$0.90
68.65
83.05
70.84
401
0.53
76.99
140 16
63.33 18
73.68
71.31
90.56
62.30 -0-40
90.49
67.3 15
38. 30
18.77
85.34
66.35
30%
0.41
00.45
59.05
91.95
58.55
90.05
51.74
91.83
59.30
01.44
59.55
$2.96
62.26
89.05
50.41
801
57.42
51.73
Table 17: VGG16 on CIFAR-10 with label noise. GradPCA OOD performance across noise levels.
In this section, we present experiments illustrating how label noise introduced during training affects the performance of GradPCA. Specifically, we train a VGG16 model on CIFAR- R- under ten levels of label noise, ranging from 0% to 90%. The label noise is fixed, i..., labels are randomly corrupted once before training begins and remain unchanged throughout training. The training setup is as described in Appendix E33 Results reported in Table 17] We observe that GradPCA's performance degrades steadily as the noise level increases, mirroring the decline test accuracy.
34


Reviews:
[{"review_id": "C6M0x5iTao", "reviewer": "Reviewer_U4mW", "summary": "The paper proposes GradPCA that applies PCA to class-mean gradients of a trained neural network. The approach is motivated by Neural Tangent Kernel (NTK) alignment, which empirically yields a low-rank, approximately block-diagonal structure in the gradient covariance of well-trained models. GradPCA formalizes this link between NTK alignment and spectral OOD detection, derives sufficient and necessary conditions for detection guarantees, and demonstrates consistent, near–state-of-the-art performance across CIFAR-10/100 and ImageNet under pretrained and non-pretrained settings.", "questions": "Could the method be extended to characterize OOD detection in terms of Type I and II errors, rather than a hard binary boundary? Specifically, how would the approach behave if in-distribution (ID) and OOD supports overlap, as might occur with visually similar classes?\n\nAs a related question, the theoretical assumptions (low-rank structure, small residual \\xi) may not hold when classes are mixed or poorly separated. Have the authors tested how sensitive GradPCA is to such violations?\n\nHow is the kernel chosen? Could alternative kernels (e.g., Fisher, feature-space, or adaptive kernels) improve detection? Is the NTK selection fixed or adaptive to data/model characteristics?\n\nFor the covariance (aka $FF^\\top$) matrix, are gradients centered before computing PCA? Clarification is needed on whether centering affects the eigenstructure and stability of the projection.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, this is a well-written and sophisticated paper with clear motivation, background, and rationale.  The writing is made to be accessible to a general deep-learning audience.\n\nThe paper has a clear theoretical grounding that connects OOD detection to NTK alignment and covariance low-rank structure, offering a mathematically elegant view. The method essentially captures a low-rank representation of the NTK kernel / gradient covariance matrix. Kernel-based reasoning is well-founded; kernel and spectral methods have a strong theoretical pedigree and proven reliability for measuring similarity in DL to model complex latent spaces of features. The approach aligns conceptually with sparse / low-rank matrix representation theory, which is also a mature and robust field.\nExperiments have been conducted on multiple benchmark datasets. Empirically results are generally sound and demonstrate performance suggested by the theory.", "weaknesses": "The choice of NTK, while natural given the theoretical link, is not unique. Other kernels (e.g., Fisher information, feature-space kernels) could also exhibit similar low-rank behavior, so the generality of the method is not fully demonstrated. The paper would benefit from a clearer articulation of why the NTK is the most appropriate or insightful kernel for connecting spectral structure with OOD behavior. In other words, it needs to “ring a bell” by making the NTK–OOD link feel both necessary and intuitively strong.\n\t\nExperimental coverage is confined to image classification. It remains unclear whether the same assumptions and empirical stability extend to other domains such as text, time series, etc. While full experiments are not required, a discussion of applicability and limitations across modalities would strengthen the paper.\n\t\nThe theoretical results rely on simplified assumptions (rank-C covariance matrix, small residual, etc.) that may not strictly hold in real neural networks. What if some of the classes have similar, or even partially overlapping feature semantics? \n\t\nRelated comments are provided in the questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f6WBoG99Ks", "reviewer": "Reviewer_vPae", "summary": "This paper introduces GradPCA, a novel and principled method for Out-of-Distribution (OOD) detection. The core idea is to exploit the low-rank structure of neural network gradients, which the authors connect theoretically to the Neural Tangent Kernel (NTK) alignment phenomenon. The method applies Principal Component Analysis (PCA) to the class-means of the gradients to define a low-dimensional \"in-distribution\" subspace. At inference, inputs whose gradients fall outside this subspace are flagged as OOD.\nThe authors provide a theoretical framework for spectral OOD detection, including a sufficient condition that yields per-sample OOD certificates. Empirically, the paper demonstrates that GradPCA achieves highly consistent and competitive performance across several standard benchmarks (CIFAR-10, CIFAR-100, ImageNet). A significant additional contribution is a systematic analysis showing that OOD detector performance critically depends on feature quality (neural collapse property).", "questions": "The paper mentions that GradPCA defaults to using only the last hidden layer parameters. It would be beneficial to include a more comprehensive ablation study on this choice. How does the method's performance vary when using gradients computed from the output layer as in GradNorm, or from specific intermediate blocks?\n\nTo further strengthen the empirical validation, the authors are encouraged to benchmark GradPCA within the OpenOOD framework. This would enable a standardized comparison and, more importantly, provide valuable insights into the method's performance on challenging near-OOD datasets versus far-OOD datasets.\n\nFor improved readability, especially for black-and-white printing, the authors should reconsider the highlighting scheme in the results tables. The current use of color to highlight the top-3 performers  can be confusing. A simpler, more standard convention, such as bolding the top-performing method and underlining the second, would significantly enhance clarity.\n\nThe paper's analysis of feature quality versus detector performance is a key contribution . To further explore this relationship, could the authors include an ablation study where feature quality is degraded in a more controlled manner? For instance, how do the \"regularity-based\" and \"abnormality-based\" detectors compare when the in-distribution (ID) training dataset is corrupted with increasing levels of label noise or input noise?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Strong Theoretical Contribution: The paper offers a formal framework for spectral OOD detection in NNs. Theorem 4.1, which provides a \"sufficient condition for spectral OOD detection\" , is a strong theoretical result that provides a deterministic, per-sample OOD certificate.\n\n- Excellent Empirical Results & Consistency: GradPCA achieves SOTA or near-SOTA results, but more importantly, it demonstrates the most consistent performance across all benchmarks. This directly addresses a major problem in the OOD field, where methods often fail erratically in different settings.\n\n- Valuable Analysis of Feature Quality: The paper's distinction between \"regularity-based\" and \"abnormality-based\" detectors is insightful. The finding that their effectiveness is tied to whether features are pretrained (general-purpose) or not (task-specific) is a practical and important contribution that helps reconcile inconsistencies in prior work.", "weaknesses": "- Memory Scalability: The primary weakness is that the memory cost scales with the number of classes, $C$. The method stores $O(C)$ gradient vectors, which \"can be costly for large C\" like in ImageNet15. While the paper shows this is manageable (e.g., 7.5GB for ImageNet in the worst case, but often less  ), it could be a barrier for datasets with thousands or tens of thousands of classes.\n\n- Core Assumption: The method's success relies on the assumption that NTK alignment provides a low-rank structure that effectively separates ID and OOD points in the gradient space . The authors note this \"may not hold universally,\" though they rightly argue this assumption is explicit and empirically supported.\n\n- Some missing references and comparisons that similarly leverage the gradient space:\n\nLee, Jinsol, et al. “Gradient-Based Adversarial and Out-of-Distribution Detection.” arXiv:2206.08255, arXiv, 4 July 2022. arXiv.org, http://arxiv.org/abs/2206.08255.\n\nSun, Jingbo, et al. “Gradient-Based Novelty Detection Boosted by Self-Supervised Binary Classification.” arXiv:2112.09815, arXiv, 17 Dec. 2021. arXiv.org, http://arxiv.org/abs/2112.09815.\n\nElAraby, Mostafa, et al. \"GROOD: GRadient-Aware Out-of-Distribution Detection.\" Transactions on Machine Learning Research. https://arxiv.org/abs/2312.14427", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eEap2s2JCz", "reviewer": "Reviewer_LCME", "summary": "The paper uses NTK alignment theory (that neural networks are low-rank subspaces) to effectively detect OOD samples. GradPCA is an OOD detection framework that applies PCA to neural network gradients.The authors use NTK Alignment to make this computationally tractable.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper establishes strong theoretical motivation behind the GradPCA framework along with some reasoning as to why it works and when the OOD detectors can be thought of as reliable, along with complementing empirical results.\n\nThe proposed method is computationally tractable and therefore more practical. O(NP) to O(C).\n\nDecent insights on feature quality explaining when regularity based methods outperform abnormality based methods and vice-versa.\n\nGood empirical results.", "weaknesses": "How does GradPCA compare to the similar recent work [1] which also does PCA on the gradients? Would be good if the authors clarify the major differences and their contributions.\n\n[1] Wu, Yingwen, et al. \"Low-dimensional gradient helps out-of-distribution detection.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).\n\n\nOnly 3 models seem to be used in the experimentation. A greater diverse set might have introduced models that might not exactly follow the NTK alignment theory and cause GradPCA to degrade, and a brief insight into that would have been nice.\n\n\nSensitivity to parameter subset selection is important since the method is primed at exploiting the low-rankness of the neural network. So, just using the final hidden layer might be sub-optimal, and a brief insight on this would be helpful.\n\n\nAblation study as to how to approximate class-means (smaller batch-sizes) would also have been helpful as they are primary in the proposed algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3bCeYkunoV", "reviewer": "Reviewer_qt5N", "summary": "This work proposes a new method for OOD detection by leveraging NTK alignment over a well-trained network, where both analytical discussions and  empirical evidence are both provided.\n\nIn general, the work is easy to follow and the NTK alignment technique is new to the task with different perspectives, even though the PCA technique and its operation on the gradients are not.", "questions": "Please view the Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This method is easy to implement and shows good efficiency (no training). The empirical performance is on average good and partially near the state-of-the art performances. In particular, the involved NTK perspective is note fully investigated in this field, which may bring some new potential inspiring future work.", "weaknesses": "Some weakness (or points to be clarified) as below:\n\n1. what if the labels of training data are not accessible? This method requires the labels/classes of training samples to construct the projection in PCA, but this is not really requested in other methods and might not be feasible in practical settings, especially considering data privacy. Could the authors make some discussions and remedies? Further, it would be good to mark out in the table whether this requirement  is taken in each compared method.\n\n2. Up to section 3.2, why line 9 and line 10 in algorithm 1 can successfully enable OOD detection? In previous context, it mentions separability of features, which is yet still not direct to the exact rationale of such detection scores. \n\nIn line 149-155, the motivation is rather intuitive and based on existing work (He & Su, 2020). Could you also specify the motivation or evidence? Otherwise, it seems that this wok gives a new technique to do OOD with NTK and PCA, without a clear, strong and verifiable motivation/rationale to do so.\n\nThis is very important for the readability of this work and the clarification of its novelty.\n\n3. Why not present discussions with (Guan et al., 2023)  and (Wu et al., 2024a) in the main context, but appendix, and why not present the empirical comparisons in the experiments? These works are highly related. \n\n4. This work presents quite some discussions on the impact of \"feature quality\". It still seems unclear and not specific enough for the reviewer accessing the so-called \"feature quality\" quantitatively. Is it possible to have a clear definition  or metric? Please be more strictly technically precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "llOl918O0A", "reviewer": "Reviewer_zZXM", "summary": "This work takes a Neural Tangent Kernel (NTK) perspective to investigate the Out-of-Distribution (OOD) detection problem. A novel detection method named GradPCA is proposed. In GradPCA, PCA is executed on the class-wise gradients of in-distribution (IND) data to obtain a low-dimensional subspace, and IND and OOD data are expected to exhibit well separability on this low-dimensional gradient subspace. Theoretical results are presented about the sufficient and necessary conditions for identifying a sample as OOD in spectral OOD detection, with discussions on how to select the feature mapping. Extensive empirical results validate the effectiveness of GradPCA. Besides, the relationships between OOD detection and some other issues such as consistency and feature quality are discussed.", "questions": "My questions are the five concerns listed above. I will raise my rating given all the concerns are well addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe NTK perspective and the associated GradPCA method are novel and beneficial to the OOD detection community.  \n2.\tDiscussions on the consistency and feature quality issues are appreciated and can provide new insights for OOD detection.  \n3.\tThe writing is good and easy to follow.   \n4.\tTheories and extensive empirical results are provided.", "weaknesses": "**Major concerns**  \n\n1.\tUnclear descriptions on the algorithm implementation  \n\nSome descriptions in Algorithm 1 are confusing, and thereby related details are suggested to be supplemented.   \n\n1.1\tIn line 4-6 in the offline training stage, what are the detailed executions and differences between line 4 and line 5? I guess that here PCs are obtained through eigendecomposition on the covariance matrix and its dual matrix, and please supplement detailed mathematical equations. But why there are two times of eigendecomposition? Then, given line 4 and line 5, which PCs are adopted to obtain the projection matrix in line 6? Please specify the calculations of the projection matrix clearly.  \n\n1.2\tIn line 9 in the online inference stage, I guess the PCA reconstruction error on gradients of a new sample is set as the detection score. But the definition for the projection matrix is missing, causing confusion on the detection score. Besides, why the norm of the projected gradients is normalized by that of the original gradients? Calculating reconstruction errors does not need the normalization of input norms. Please clarify this issue.\n\n2.\tInsufficient experiments.  \n\nWhile the experiments include several general detection methods, the comparisons lack more directly relevant, subspace-based baselines, as GradPCA itself is based on PCA and model gradients. The suggested baselines have been reviewed in the submission but are missing in empirical comparisons: (i) PCA [1] and Kernel PCA [2] applied to penultimate layer features, and (ii) the low-dimensional gradient subspace method explored in [3].\n\n3.\tExtended discussion on the consistency.  \n\nThe inconsistency issue in OOD detection is rarely explored. The experiments in Figure 1 of this submission are executed across diverse benchmarks. Meanwhile, the inconsistency across multiple independently-trained modes (local optima) is highlighted in [4]. It would be appreciated to evaluate GradPCA on such independent modes to validate its consistency from the model side, which will further strengthen this work.\n\n[1] Revisit pca-based technique for out-of-distribution detection. ICCV 2023.  \n[2] Kernel PCA for out-of-distribution detection. NeurIPS 2024.  \n[3] Low-Dimensional Gradient Helps Out-of-Distribution Detection. TPAMI 2024.  \n[4] Revisiting Deep Ensemble for Out-of-Distribution Detection: A Loss Landscape Perspective. IJCV 2024.  \n\n**Minor concerns**  \n4.\tIn the basic settings, the neural network function $f$ outputs a real number. Please specify the calculations of this outputted real number.  \n5.\tThe GradOrth method demonstrates quite poor detection performance across almost all datasets. It would be helpful for authors to clarify whether the hyper-parameters of GradOrth have been carefully tuned. If so, some explanations into the potential reasons for these results would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)