Diagnosing Generalization Failures from Representational Geometry Markers
Authors: Chi-Ning Chou
Artem Kirsanov
Yao-Yuan Yang
SueYeon Chung
Venue: ICLR 2026 Poster (2026)
Keywords: Representational geometry
Out of distribution generalization
Image classification
Primary area: interpretability and explainable AI
TL;DR: Representational geometric signatures from in-distribution data consistently predict failure in out-of-distribution generalization

Abstract: Generalization—the ability to perform well beyond the training context—is a hallmark of biological and artificial intelligence, yet anticipating unseen failures remains a central challenge. Conventional approaches often take a "bottom-up" mechanistic route by reverse-engineering interpretable features or circuits to build explanatory models. While insightful, these methods often struggle to provide the high-level, predictive signals for anticipating failure in real-world deployment. Here, we propose using a "top-down" approach to studying generalization failures inspired by medical biomarkers: identifying system-level measurements that serve as robust indicators of a model’s future performance. Rather than mapping out detailed internal mechanisms, we systematically design and test network markers to probe structure–function links, identify prognostic indicators, and validate predictions in real-world settings. In image classification, we find that task-relevant geometric properties of in-distribution (ID) object manifolds consistently forecast poor out-of-distribution (OOD) generalization. In particular, reductions in two geometric measures—effective manifold dimensionality and utility—predict weaker OOD performance across diverse architectures, optimizers, and datasets. We apply this finding to transfer learning with ImageNet-pretrained models. We consistently find that the same geometric patterns predict OOD transfer performance more reliably than ID accuracy. This work demonstrates that representational geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and AI interpretability.

Published as a conference paper at ICLR 2026
DIAGNOSING GENERALIZATION FAILURES FROM REPRESENTATIONAL GEOMETRY MARKERS
Chi-Ning Chou Flatiron Institute
Artem Kirsanov Harvard University
Yuo-Yuan Yang Google DeepMind
SueYeon Chung Harvard University
ABSTRACT
Generalization the ability to perform well beyond the training context-is hall- mark of biological and artificial intelligence, yet anticipating unseen failures re- mains a central challenge. Conventional approaches often take "bottom-up" mechanistic route by reverse-engineerrngg interpretable features or circuits to build explanatory models. While insightful, these methods often struggle to provide the high-level, predictive signals for anticipating failure in real-worid deployment. Here, we propose using "top-down" approach to studying generalization failures inspired by medical biomarkers: identifying system-level measurements that as robust indicators of a model's future performance. Rather than mapping out de- tailed internal mechanisms, we systematically design and test network markers probe cruuture-fuucciinn links, identify prognostic indicators, and validate pre- dictions in real-world settings, In image classification, we find that task-relevant geometric properties of in-distribution (ID) object manifolds consistently fore- cast poor nuttof-distribution (OOD) generalization. In particular, reductions two geometric measures effeectvveeeeeeivee manifold dimensionality and utility-predice weaker OOD performance across diverse architectures, optimizers, and datasets. We apply this finding to transfer learning with ImageNet-pretrainee models. We consistently find that the same geometric patterns predict OOD transfer perfor- mance more reliably than ID accuracy. This work demonstrates that representa- tional geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and Al interpretability.
INTRODUCTION
Biomarkers like blood pressure or cholesterol levels are indispensable tools for anticipating health risks before symptoms emerge. Throughout the history of medicine, physicians have of- ten utilized these diagnostic measures effectively before figuring out all the biological details. This pragmatic, top-down approach of correlating biomarkers with outcomes has thus driven medical progress, while simultaneously providing the foundational insights for figuring out causal mecha- nisms. In neuroscience, the same methodology has been fruitful: single-neuron and population-leve signatures have served as useful analysis units, revealing principles of coding and computation often before a full mechanistic understanding (Rigotti et al., 2013; Barak et al., 2013; Mastrogiuseppe & Ostojic, 2018; Stringer et al., 2019).
As deep neural networks (DNNs) become increasingly integrated into critical applications, a sim- ilar challenge arises: how can anticipate their unseen failures? This is particularly important under distribution shifts where training and deployment environments differ (Sagawa et al., 2020a; Liu et al., 2021; Yang al., 2024). Current research often favors bottom-up approaches, such as mechanistic interpretability (MI), which aims to reverse-engineer DNNs by identifying interpretable features (Olah al., 2017: Yun al., 2023; Cunningham al., 2023), functional circuits (Olah al., 2020; Dunefsky et al., 2024), or causal structures (Mueller et al., 2024; Geiger et al., 2025). While
Contact: {cchou, schung) flati
tute. .rg
'The lipid hypothesis. for instance, linked cholesterol to cardiovascular disease risk well before lipid path- ways were mapped. Similarly, selective serotonin reuptake inhibitors treated depression, informed by the sero- tonin hypothesis, decades before serotonin's 's precise role in moad regulation fully understood.

---

Published as a conference paper at ICLR 2026
these MI methods offer granular insights, they may lack identifiability (Méloux et al., 2025), and it remains unclear how they can provide concrete diagnostics on wealdworld models.
Here we propose complementary perspective inspired by the history of medicine: diagnostic, system-level paradigm for understanding neural networks. Rather than attempting to reconstruct their internal mechanisms post-hoc, focus developing task-relevant
cetturementttmarreee for Al models- that serve as reliable indicators of potential failure modes. Our methodology fol- lows three-step cycle (Figure 1): (i) Marker Design: develop task-relevant measures probe which structures in neural networks (e.g., feature vectors, weights) relate to their function and per- formance; (ii) Prognostic Discovery": conduct medium-size experiments across diverse architec- tures and hyperparameters. and identify patterns that serve as prognostic i in in-distribution (ID) properties that can forecast future generalization failures without requiring (OOD) tasks; (iii) Real-world Application: apply these insights to practical settings, such as predicting which pretrained models will transfer more robustly across datasets. We demonstrate this research cycle by using ID measures based on task-relevant representational geometry to diagnose failure in OOD generalization. Our framework points toward diagnostic science of Al models, offering tools to anticipate vulnerabilities and improve robustness in safety-critical domains.
present
any knowledge of the out-of-distribun
Marker Dasign (Section
Prognostic Discovery (Saction
Real-Worid Application (Section 4):
Taskerelevant measures for models to grobe structures that likk to funcieresteehaviore
Idenify progneslic patterns of markers linked fallure madium exper
Diagnose and make predictions unseen falure mode in resl-vorld models
comelalions dumbbell
Imagenet ancurasy
Monfold
StanfordCars ant.rany
dimensian
"preclirical studies
(sonwention) (our progmoetia)
(COD)
Yeot
77.04
11.45
77.38 V
Car
spoet
79.67
9.23
X
performance mesoures
stat sticili messutee
geometric tneessul
compression meniffll dimensior
OCI
Pytorch protrained model with different weights
performanca OOD dalaset
generalizaiion
design treatment, discover phenomens toward better mechanistic understanding....
Figure 1: A diagnostic, system-level paradigm for studying generalization failures in DNNs, with an example image classification. See Section 1.1 for an overview.
1.1 OVERVIEW AND OUR CONTRIBUTIONS
In this work, we apply the proposed diagnostic, system-level paradigm to investigate failure modes of OOD generalization in image classification. Our key finding is that feature overspecializa- qun--uuttfiied by reduced effective dimensionality and utility of object manifolds is a reliable indicator of poor performance under class-level distribution shifts in transfer learning.
Marker design for image classification (Section 2). A central step in our diagnostic framework is selecting and designing markery-scale quantities computed entirely from ID data-that capture aspects of a pretrained model relevant for downstream generalization. In image classification, we focus on penultimate-layer feature vectors, as the final classification decision is obtained by linear readout from this layer. Accordingly, we evaluate a broad family of candidate markers, including: (i) accuracy- and logits-based based quantities, (ii) low-order statistical summaries of representations (e.g., sparsity, covariance structure), and (iii) geometric measures of class-conditioned point-cloud mani- folds-such as participation ratio, within-class spread, neural-collapse metrics (Papyan et al., 2020; Harun et al., 2025), numerical rank (Masarczyk et al., 2023; Harun et al., 2024), and task-relevant geometric measures from the GLUE framework (Chou et al., 2025a).
Prognostic discovery of OOD generalization failures (Section 3). We conducted exploratory experiments to investigate whether these metrics can predict failures in OOD generalization. Specif- ically, we trained a broad class of deep networks on in-distribution (ID) data (e.g., CIFAR-10) and
II medicine, diagnostics identify present conditions, while prognostics forecast future risks. Our frame- work is termed "diagnostic" broadly, with Step 2 specified as "prognostic discovery" to emphasize prediction of OOD failures from ID data.
2

---

Published as a conference paper at ICLR 2026
evaluated their OOD performance on datasets with disjoint classes (e.g., CIFAR- 100; Figure b), Our sweep spanned five architectures (e.g., ResNet, VGG), multiple depths, two optimization al- gorithms (SGD, AdamW), and grid of hyperparameters (learning rate, weight decay). We found that different training hyperparameters can lead to markedly different OOD performance, despite nearly identical ID train and test accuracy. Task-relevant geometric measures of ID object mani- folds correlated far more strongly with OOD performance than conventional performance metrics (e.g., ID accuracy) or statistical measures (e.g. sparsity, covariance) (Figure 3c, top). In particu- lar, reductions in effective dimensionality and utility consistently served as prognostic indicators of OOD failure (Figure 3c, bottom). Together with prior work linking representational geometry to feature learning (Chou et al., 2025b), these findings suggest that overspecialized features undermine generalization performance, echoing previous accounts of shortcut learning (Geirhos al., 2020).
Applications to failure prediction in pretrained models (Section 4). Finally, we applied our prognostic indicators to prretenetpretraineed models from public repositories. In practice, when selecting among multiple pretrained weights of the same architecture, the most common criterion test accuracy. Here, we measured the effective dimensionality and utility of ImageNet object manifolds from 20 architectures available in PyTorch (e.g., RegNet, MobileNet, WideResNet), each released with two weights (vl and v2); by construction, v2 achieves higher ID accuracy. Unlike controlled prognostic studies, these pretrained weights were produced under distinct training recipes, regularization schemes, and preprocessing pipelines, making them a much more heteroge- neous testbed. Nevertheless, consistent with the predictions from medium-scale experiments, models where v exhibited higher manifold dimension and utility than v2 also achieved better OOD) performance under v1 weights- though v1 had lower II) accuracy (Figure 5). This demon- strates that ID representational geometry can serve as an early diagnostic for OOD robustness.
Summary. Our work demonstrates a diagnostic, system-level paradigm that complements conven- tional mechanistic interpretability by focusing on predictive indicators of model failure. Our results highlight how task-relevant geometric measures of ID representations can serve as markers for di- agnosing failures in OOD generalization, even when mechanistic details remain opaque.
1.2 RELATED WORK
Representational geometry and generalization. A growing body of work suggests that proper- ties of internal representations in DNNs can indicate generalization performance. In the standard ID setting, both statistical features of activations-such as sparsity, covariance, and inter-feature cor- relations (Morcos et al., 2018)-and geometric measures of object manifolds (Ansuini et al., 2019; Cohen et al., 2020; Chou et al., 2025b) have been predictive. For example, networks that gener- alize well often exhibit low intrinsic dimensionality in their final-layer representations, and such compactness correlates with test accuracy in image classification (Ansuini al., 2019). related phenomenon is neural collapse (Papyan et al., 2020), where within-class variability of final hidden representations vanishes in the terminal phase of training.
The picture becomes more convoluted under distribution shifts. (Galanti al., 2022) showed that neural collapse can generalize to new data points and classes when trained on sufficiently many classes with lots of samples. By contrast, (Zhu et al., 2023) found that encouraging diversity and decorrelation among features improves OOD performance in image and video classification. Sim- ilarly, in neuroscience. high- high-dimensional yet smooth population codes in mouse visual cortex have been linked to generalization across stimulus conditions (Rigotti et al., 2013; Stringer et al., 2019). These conflicting results call for more systematic study how representational properties nect to OOD generalization, and our findings- that OOD failures correlate with overcompression of object
supporr these results on the advantage of high-dimensional representations. However, most of these approaches rely on generic geometric or statistical descriptors that are not explicitly tied to the computational task, whereas the GLUE measures we employ use the anchor point distribution (Figure 2c and Section B.3) to directly link representational geometry to down- stream linear classification performance.
Distribution shift in ML: detection, transfer, and prior approaches. OOD detection methods aim to distinguish OOD samples from ID samples by examining differences in their feature repre- sentations or logit distributions. These approaches typically operate under labse-rres distri- bution shifts, where the input distribution changes but the class label space remains the same. In
3

---

Published as a conference paper at ICLR 2026
contrast, class-level OOD generalization also called transfer learning- substantially more chal- lenging, since the OOD task contains entirely unseen class labels. Performance in this setting is usually assessed by training a linear probe (often the penultimate-layerr features) of pretrained network. Several works have studied how architectural or representational factors influence this probe-based OOD performance. For example, the Tunnel Effect papers (Masarczyk et al., 2023; Harun et al., 2024) showed that the drop in OOD linear-probe accuracy across layers correlates with drop in the numerical rank of OOD features. Similarly, Neural Collapse-based analyses (Harun et al., 2025) have examined how the extent of collapse (e.g.. the .NC1 metric (Papyan et al., 2020)) relates to OOD generalization performance across layers. Outside image classification, recently Li et al. (2025) used interpretability methods to predict OOD model behavior in language tasks through ID attention patterns.
In this work, we incorporate several of these ideas into our marker design. For OOD detection methods, many algorithms require access to OOD samples in their scoring pipelines, making them incompatible with our ID-only diagnostic setting. However, methods that rely solely logit statis- ties or feature-level summaries can be adapted into scalar markers and included in our evaluation. For the Tunnel Effect and Neural Collapse lines of work, we directly implement their corresponding measures- numerical rank and feature-collapse metrics (e.g., NC1) and compare them against our GLUE-based markers in the prognostic analysis. It is worth noting that both the Tunnel Ef- fect (Masarczyk et al., 2023; Harun et al., 2024) and Neural Collapse (Harun et al., 2025) studies primarily analyze how their measures vary across layers within the same model, rather than across different models or training configurations. Our focus, in contrast, is on comparing models trained under different hyperparameters or initialization regimes, which is the setting relevant for model selection and prognostic prediction.
2 MARKERS FOR IMAGE CLASSIFICATION
Given neural network with parameters 0 and ID dataset DID. define marker as function that maps (0, DID) to scalar value indicative of potential failure modes in OOD gen- eralization. Train and test accuracy are examples of such markers, but they are often non- discriminative (D' Amour et al., 2022), propelling us to open the black box of DNNs and seek measures that are both task-relevant and discriminative.
Among the many ways to peer inside a DNN, we focus on feature embeddings. Concretely, we an- alyze penultimate-layer feature vectors [ M, (e.g.. avepool in ResNet, see Table 3) extracted from the ID data. Each z, R is an N-dimensional feature vector, and in image classification these can be grouped by class: letting P denote the number of classes and MP the number of samples in class μ4, we write { M 1= so that [ = U [ In addition to geometric markers derived from representational manifolds, we also consider conventional ID-only markers inspired by prior OOD-detection methods. These include low-order statistical summaries of penul- timate representations (e.g.. sparsity, covariance structure, pairwise distances and angles) as well as quantities computed directly from the logits distribution (e.g., averaged confidence).
Ti=1-
In the remainder of this section, we first review the conventional statistical and logits-based based markers that serve as baselines in our analysis (Section 2.1), and then introduce task-relevant geometric markers grounded in representational manifold theory (Section 2.2).
2.1 CONVENTIONAL MEASURES
We also examine low-order statistics of penultimate feature vectors. We consider several standard statistics: activation sparsity, off-diagonal covariance magnitude, and mean pairwise distance/angle. Each measure is applied both globally across {z.} and within each class {z," }. These descriptors summarize the distribution of representations but do not capture per-class manifold geometry, moti- vating the measures introduced next. Formal definitions are given in Section B.2.
In addition to feature-level statistics, we incorporate several logit-based markers commonly used in research and adapt them to our ID-on ID-only diagnostic setting, including averaged con- fidence (AUROC) (Hendrycks & Dietterich, 2018), Entropy (Guillory et al., 2021), and Energy (Liu al., 2020). While these methods were originally designed detect OOD inputs- typically
4

---

Published as a conference paper at ICLR 2026
Uniform dietribution
P.D. D
Mer (1+R_
Prabsaa 2 1 doe Cemeena
decrensing Dut
cog
randomy
cat
projecting
manilolde
decreasing Rir
smaller
Anchor point distr istribution
mensioral subspeces
increasing V.
neorghniding
matiiouus without wastefal compression
65
featurniembacdnng space
(prajected dimensionsity)
Figure 2: Object manifolds and task-relevant geometric measures. a, Object manifolds are per-class point clouds in the feature space. b. Critical dimension Narit quantifies the degree of manifold untanalinggxpparabllity in an average-case sense via random projection. c, Anchor point distribution gives higher weight to points that more important for linear classification. 'd, The degree of manifold separation (quantified by critical number of neurons Nom) analytically linked to three task-relevant geometric measures: effective dimension Dent: radius Retl and utility V Venl-
assume access to shifted data-we evaluate them here as scalar markers derived solely from ID logits.
Beyond these statistical and logits-based metrics, prior work has also analyzed representational ge- ometry in neural populations (Chung & Abbott, 2021; Li et al., 2024). Whereas statistical metrics capture overall spread pairwise correlations, geometric descriptors characterize manifold struc- ture such as alignment, curvature, and class-specific variability. A widely used task-agnostic geo- metric marker is the participation ratio (PR), which estimates the intrinsic dimensionality of each class manifold from the spectrum of its covariance matrix. We also include Neural Collapse mea- sures (Papyan et al., 2020; Ammar et al., 2023; Harun et al., 2025) and the numerical rank measure from the Tunnel Effect hypothesis (Masarczyk et al., 2023; Harun et al., 2024). Formal definitions for markers are provided in Appendix B.
2.2 WASK-RELEVANT GEOMETRIC MEASURES
To obtain task-relevant markers of ID representations, we adopt the Geometry Linked to Untangling Efficiency (GLUE) framework (Chou et al., 2025a), which builds on the theory of perceptron capac- ity for points (Gardner & Derrida, 1988) and manifolds (Chung et al., 2018; Wakhloo et al., 2023; Mignacco al., 2025; Chou et al., 2025a) from statistical physics. Similar support vector ma- chine (SVM) theory (Cortes & Vapnik, 1995), where the max-margin classifier can be expressed as linear combination of support vectors, GLUE theory provides analytic connection between the critical number of neurons Neit and the geometry of object manifolds (Figure 2a) through an anchor point distribution over the object manifolds (Figure 2c).
GLUE as average-case analog of SVM. Consider classifying two object manifolds M Hull( [2] [z...... 2/r1) and M Hull( ..., 22,1) such that manifolds remain linearly separable with probability least 0.5 after random projection to an N dimensiimniio subspace (Figure 2b). Manifold capacity a is defined as P/Nort where P the number of manifolds. lower value of Narit (i.e., higher value of a) means that the object manifolds are more separable on average. The key result in GLUE theory is closed-form formula for Non:
}) in NR,, Noii defined as the minimum Nproj
Nori (0, (AEEMM (t)) pro (t)) tl
(t))
where (O0 Tx) is the isotropic Gaussian distribution in N span(.) denotes linear span of set, and proj denotes orthogonal projection. Equation naturally leads to defining anchor points as the maximizers of the inner optimization problem. The anchor point distribution non-uniform
TThis figure is schematic illustration of the non-unin non-uniform, resk-relevant anchor point distribution. The 2D depiction is only intuitive and can be misleading, analogous to how in high dimensions Gaussian mass concentrates on the sphere rather than at the origin.
5

---

Published as a conference paper at ICLR 2026
measure over the manifolds and favors those points that more important for downstream clas- sification (Figure 2c). Hence, GLUE theory can be thought of as an average-cass analog of SVM theory: whereas SVM assesses separability in the best-case scenario by leveraging the full feature space, GLUE evaluates separability under random projections, effectively averaging across many such subspaces, and hence is able capture more complex, heterogeneous, and nuisance structure present in the data (Chou et al., 2025a;b).
By exploiting symmetries in the equation, GLUE theory derives three effective geometric mea- sures efresteffective dimension Dett. effective radius Retf. and effective utility Ver-and reorga- nizes Equation into simple expression (see Section B.3 for details and derivations):
P DeDt
N crit
Lett (1 + Rent
where P is the number of manifolds. Intuitively, Equation 2 shows that Narit decreases (i.e., mani- folds become more separable/untangled) with smaller Dett. smaller Ren, and larger e (Figure 2d). Because the GL.UE theory captures task-relevant structures in neural representations via the anchor point distribution (as opposed to the uniform distribution, i.e., equiprobable sampling of points), recent work (Chou al.. 2025b) has shown that Narit and GLUE measures are much more dis- criminative than conventional measures (c.g., kernel-based methods, weight changes) in the study of feature learning. GLUE also defines additional measures (c.g., center, axis, center-axis alignment) from the anchor point distribution, detailed in Section B.3 and omitted here for brevity. We provide intuitions for the three effective geometric measures Table (see Table for the full version).
Table I: Intuitions for GLUE measures.
Den 2 0
Ret 2 0
Van E [0.
Geometric intuition
Quantify the task-relevant dimensionality of object manifolds.
Quantify the task-relevant spread within each mani-
Quantify the amount of
excessive compression
fold relative to their centers.
untangling manifolds.
Effect on linear separability Example
More separable when Den is small.
More separable when Ret is small.
More separable when Ver is large.
Dell equals dimension of uncor. random spheres
Ret equals the radius un- cor. random spheres
Collapsing manifolds to points yields Vatt - 0.
Interpretation in fcature
Low Dell indicates smaller of fcature modes use.
Low Ret indicates more similar feature usage across examples within class.
Low Danl indicates in- efficient compression of within-class variability.
learning"
Connection to feature learning. We follow a top-down view of feature learning (Chou et al., 2025b), where features understood functionally through their consequences for computation (e.g., enabling linear separability) rather than as specific interpretable axes or neurons. This per- spective emphasizes how representational geometry changes with feature usage without requiring explicit identification of the features themselves. Moreover, by thinking of direction in the rep- resentation space feature (linear representation hypothesis (Park et al., 2024)), the effective geometric measures offer interpretation in feature learning as listed in the table.
3 DISCOVER PROGNOSTICS FOR FAILURE IN OOD GENERALIZATION
We study medium-scall models as a testbed for identifying prognostic indicators of failure modes. Our goal is to detect ID signals that reliably predict how model will behave under distribution witht-t any access to OOD data. This departs from most existing OOD-detection meth- ods, which typically rely information from the shifted distribution. Our diagnostic analysis markers measured solely from ID properties to anticipate vulnerabilities before deployment.
3.1 METHODS
We adopt an experimental design in (Chou et al., 2025b) where DNNs are trained on an ID image dataset, and OOD performance is evaluated on a different dataset with disjoint set of classes.
Training procedure. We trained multiple DNN architectures (c.g., ResNet, VGG) from scratch on CIFAR-10. For each architecture, swept four initial learning rates, four weight decay

---

Published as a conference paper at ICLR 2026
Experimental setup:
Porformanee measures on the ID dataset are weukly prediolive of OOD performaroo
In-distribution (ID) dataset (e g. CIFAR-10)
Out-of-distiluution (000) dataset (e CIFAR-1000
Verying network archiecture, networrk size, optiitzation algonthm, cataset, etc.
airplane
hird
apple
bey
DNNs are pre-trained finettuned on ID data
000 Aosurayy %
Geometric measures the 1D datasel
coll
ddg
cass
dincesur
strongly predictive OOD performance
ID Bffedee TARwed Den
Probing clessifiess trained and evalusted on ODD
cisjontt Image classes
OO0 ROSURREY 9
Figure 3: Prognostic discovery for OOD generalization. a, We consider the image classification problem with an ID dataset and an OOD dataset with disjoint image classes b, We trained DNNs on the ID dataset and evaluated the OOD performance as linear probe accuracy. c, Conventional performance and statistical measures on the ID dataset are weakly predictive of OOD performance, while some task-relevant geometric measures can robustly predict failures in OOD generalization.
values, and three random seeds, using both SGD and AdamW optimizers. In all cases, we ensured that the training accuracy was above 99% and the test accuracy ranged from 88% to 95%.
OOD evaluation via linear probing. To assess the OOD generalization of learned representa- tions, adopt linear probing framework (Alain & Bengio, 2016; Zhu et al., 2023; Chou et al., 2025b). After ID training, the network's s feature extractor was frozen. A new linear classifier was then trained top of these features using the OOD dataset. The test accuracy of this linear probe served as our measure of OOD performance (Figure 3b). See Appendix A for details.
3.2 RESULTS
We find that models trained with distinct hyperparameters can exhibit similar ID accuracy while their OOD performance can differ drastically. This variation, however, is not random; we find that OOD performance can be consistently predicted by geometrie properties of ID representations.
Task-relevant geometric markers are predictive across architectures. First, we trained differ- ent architectures (ResNet, VGG, etc) on CIFAR- 10 and evaluated OOD performance on CIFAR- 100. As summarized in Figure 4, conventional metrics like ID accuracy and statistical measures like sparsity showed weak and inconsistent correlations with OOD performance. In contrast, several ge- ometric measures- measures-parricclanee particularly participation ratio, effective dimension, and effective utility-were were strong predictors and consistently performed well across all architectures.
Findings hold across model sizes, optimizers, and datasets. Next, we tested the generality of our findings by varying model size (R3445c)1/344550,, optimizer (SGD, AdamW), and the OOD dataset (CIFAR-100, ImageNet). The results, shown in Figure 4, remained consistent. Across all these settings, task-relevant geometric signatures of the ID data were systematically predictive of OOD performance, whereas alternative maarkers-including Neural Collapse (Harun et al., 2025), numeri- cal rank (Tunnel Effect (Masarczyk et al., 2023)), and logits-based OOD-detection scores- statistically weaker or less consistently predictive trends across settings, with numerical rank per- forming well in most but a few cases (c.g., VGG-19 using SGD). We suspect this is because the Neural Collapse and Tunnel Effect measures were primarily designed on mathematical intuition rather than task-relevant considerations; as a result, they may not capture the fine-grained structure of complex neural activity patterns across different models training regimes. Conversely, logit- based markers such as AUROC or entropy are task-relevant but appear to discard too much of the rich information in internal representations, limiting their predictive power in this setting. Additional results are provided in Appendix C.
Task-relevant geometric markers from ID training data also show strong trends. While the main figures report results using ID test validation features, we find that the same geometric in- dicators measured directly on the ID training data exhibit similarly strong correlations with OOD performance (see Figure 6). This indicates that the predictive signal is not limited to held-out exam- ples, but is already present in the geometry of the training representations themselves.
ID Test accuracy best predicts OOD performance on corrupted images. We also consider a corrupted version (e.g., adding noise, varying brightness, pixellating, etc.) of the original images as
7

---

Published as a conference paper at ICLR 2026
an OOD dataset (e.g., CIFAR-106 (Hendrycks & Dietterich, 2018)). Since the class labels remain identical to the ID dataset, OOD performance can be measured directly by the trained network, without training additional linear probe. In this setting, ID test accuracy is the strongest predictor of performance on corrupted data (see Section C.3), although we note that it does not always work. We also observe distinct geometric patterns across different cornuption types. These results highlight that the correlation between OOD accuracy and manifold compression (Figure 4) is non-trivial and specific class-level shifts, but does not extend to corruption- shifts where the label space is unchanged. Exploring robustness to corruption thus remains an interesting direction for future work.
Performance/Station /ttatt//ttttttiiii/aaaaaa
Staasstica/Seemmttii /Seometrical Measures on In Distribution Test Data
ResNett8
0.75
0.17
0.30
10:82
0.03
0.41
-0.12
+0.07
0.11
0,49
0.59
0.50
0.80
00.29
0.70
0.465
-0.45
(SGD)
VG009
<0 06
+0 56
+0.35
0.05
0.60
0.70
0.75
<0.
56
0.87
0.61
0.30
0.90
0.96
-0.3
0.94
0.55
0.32
0.39
(SGD)
Mobie/Net
0.07
6 0.89
0.79
0.60
0.77
0.91
0.92
3.35
0.82
0.92
0.50
0.94
0.92
0.45
0.92
0.75
0.15
0.30
0.75
0.18
0.01
0.24
6.85
6.07
0.28
0.28
0.08
0.05
0.08
0.05
-0.37
0.69
-0.19
DenscNoc121
6.00
0.05
0.00
0.13
0.23
0.35
0.38
0.11
030
(930)
6004
8-04
(Adamm))
VGE19
*.*
0.60
0.69
6...
6...
0.79
0.71
0.25
MnDO Coeef cent
(Adamm)
4.6
684
Mohilete!
0.23
0.17
0.06
0.71
0.47
0.05
0.71
0.77
0.91
0.94
+0.0
0.93
0.76
+0.00
EmdeetteeBBC
(Adammy)
0.50
0.51
0.30
0.54
0.25
0.20
0.49
0.46
0.05
0.40
0.59
0.20
0.70
....
0.28
0.02
0.39
0.34
0.02
-0:00
(Admmmy!
DerseNet::::
0.20
0.45
0:30
10.35
0.78
0.76
0.43
0.60
4.27
0.75
0.75
0.59
0.05
00.75
0.94
0.02
0.557
+0.5
(Adamny)
Reshot24
0.49
0:16
0.31
0.13
0.44
0.16
0.15
0.50
0.32
0.63
0.60
0.82
0.83
0.13
0.78
0.81
0.58
0.37
-0
25
Accis
RasNet50
6.52
4460
0.18
0.07
0.04
0.26
0.69
0.53
0.79
-0.25
0.70
VG613
0.16
0.11
-0.50
846
6444
A+AA
8-644
ANAA
A*AA
(5CD)
Reshett8
0.68
0.05
0.76
0.73
0.74
00330
.0.00
0..44
0.75
0.71
0.85
0.85
0.50
0.85
0.91
40.35
0.91
0.54
0.06
0.32
Einggehck Not S0.0
11.00
Thin Accurety
Ter Acturtty
AUROC
Peatun Ch Rance
Teatore Ande
pounat Colleoo 10C11
Effcctioe Dimamiion
Elfectio Beaaii
Eflectow Uiing
Cantir Digg mane
pacs Alian morc
Conter pale Alg
Our
Figure 4: All results on models trained on CIFAR-10. showing correlations between markers (x- axis) and OOD performance across a hyperparameter sweep. Numbers indicate Pearson r; asterisks
denote significance (" S 0.05; p S 0.01: *** : S 0.001; : S 0.0001).
3.3 DIAGNOSING FAILURES IN GENERALIZATION VIA DETECTING SHORTCUT FEATURES
Failures in generalization are often attributed to model specializing in its training regime. A clas- sic example is overfitting, where high training accuracy but low validation accuracy indicates that the model has memorized the training rather than learned transferable patterns. Under distribu- tion shift, however, such straightforward indicators as validation accuracy are absent. Our findings in Section 3.2 suggest using Dett and Left measured on ID object manifolds as prognostics for indi- cating potential failure in OOD datasets with new classes of images.
Failures in OOD generalization are often attributed to reliance on shortcut or spurious fea- tures (Geirhos et al., 2020; Sagawa et al., 2020b; Singla & Feizi, 2021; Yang et al., 2022). A network may correctly classify cows in typical training images, yet fail when cows appear in unusual con- texts such as beaches or mountains, suggesting that background cues like grass had been used as unintended predictors of class identity (Beery et al., 2018). Features such as "grass" correspond to microscopic details, whereas generalization performance is macroscopic outcome. Effective geo- metric measures act as mesoscopic descriptors, bridging how microscopic features are at play and how efficiently they used for macroscopic behavior, such as separability (Chou et al., 2025b) (see also Table 4). Low Dell and Ven indicate that the model relies on a smaller set of features, used inefficiently for separability, agreeing with chorteut-learnine interpretations.
Finally, we remark that although untrained or randomly initialized networks also exhibit very high manifold dimension and poor generalization (Chou al., 2025b) (i.e., lazy learning, Figure 7), our
8

---

Published as a conference paper at ICLR 2026
analysis concerns models with comparable ID validation accuracy- -ie, after meaningful feature learning has taken place. In this regime, larger Dett and een reflect richer relk-atlevant variability, whereas excessive compression signals ovenspecialization to the ID distribution.
4 APPLICATIONS TO PREDICTING PERFORMANCE OF TRANSFER LEARNING
A common scenario in applied machine learning involves selecting a pretrained model from a public repository like Py PyTorch Hub or Hugging Face. For a given architecture, multiple sets of weights are often available, each trained with different optimization recipes, regularization schemes, or data preprocessigg pipelines. The standard heuristic is to choose the model with the highest reported in- distribution (ID) accuracy. However, it is unclear whether this metric reliably predicts performance on other downstream tasks, especially under the distribution shifts inherent in transfer learning.
Here, we apply the prognostic indicators discovered in our exploratory experiments (Section 3) to this practical challenge. Our findings suggest clear guiding principle for model selection: when faced with multiple weights for the same architecture, prefer the model that exhibits higher ef- fective manifold dimensionality (Der) and utility (V (Werr) on its ID data, as this signals a greater potential for robust OOD generalization.
Model
Marker
Acc
OOD Acc (mreasure by linear prube)
ODD
Arch.
Weight
D,
4
ImageNet
Flower
Food
Nat.
Place
Texture
Aircraft
Skin
pred.
scc.
MobilaNott
v1
13:07
0.956
74.04
89.16
89.71
72.79
36.56
44.39
53.28
67.64
41.60
80.29
[V3_Largi)
v2
10.06
0.924
75.27
91.05
85.82
71.84
66
43.70
49.97
66.28
34.86
80.12
RagNet
v1
10.50
0.929
80.06
92.72
88.74
75.86
42.09
45.72
56.13
70,96
41.77
82.65
(16GG)
v2
8.78
0.899
82.72
93.13
85.88
74.95
38.84
48.02
49.43
72.30
28.01
80.49
6/9
RegNet
v1
11.35
0,935
77.04
91.89
88.28
72.36
39.33
44.27
53.96
68.60
40.37
79.45
(1 BGF)
v2
9.19
0.908
79.67
92.43
79.22
68 03
32. 68
43.45
48
67.06
22 99
76.26
8/9
RegNet x (32G5))
v1
v2
10.15
8.81
0.930
0.905
80.62
83.01
93.00
93.38
85 94
88.06
76.00
75. 73
40.93
36. 59
45.72
46.35
54.86
47.7 22
71.67
70.41
38.54
26. 26.42
81 19
82.68
7/9
RegNet Y (16(3-)
v1
80.42
92.81
76.66
45.89
54.62
71.10
83.42
v2
9.29
0.906
62.89
82. 30
85.07
75.99
56
46.72
48.63
71.19
27.26
80.02
7/9
RegNet Y (1]6FF
v1
11.06
92.30
73.64
40.26
80.45
v2
8.60
0.909
80.88
93.13
79.46
69.61
31.67
43.67
40.58
65.11
25.88
77.58
8/9
RegNet Y (400MFF
v1
v2
10.98
9.16
0.914
0.933
75.80
05
91.06
90.55
76 81
63. 61
68.
27 75
34.74
40.77 77
42.68 68
33 52
43.00
63.71
38.32
22 36
75.555 92.92
8/9
RegNet Y (600MF)
v1
11.27
0.937
42
86,25
72.28
38.97
44.34
62.91
69,34
42.29
79.09
8/9
v2
9.42
0.913
78.83
92.44
79.48
68.00
32.28
42.92
43.37
65.80
30.10
77.39
RegNat Y
(BGF)
v2
v1
10.32
8.61
0.903
0.926
81.68
79.34
93.15
82.40
85.10
88,23
75.53
76.39
36.19
42.38
45, 86
48.24
50.18
55.70
70,60
70.09
28.30
44.32
80.99
83.65
6/9
ResNet
(152)
v1
v2
9.14
9.48
0.920
0.910
82.28
78.31
92.68
92.86
81.17
84.73
71.88
71.70
32.23
36.26
44.81
46.02 02
40.62
47.96
67.73
69.82
23.18
35.47
79.19
81.22
7/9
(B_16)
VIT
v1
9.56
8.86
0.956
0.928
85.30
81.87
95.24
95.
97.77 71
99.56
94.04
89
61.09
50 14
50.51
79.98
78.51
76
50.10
43.05
83.58
81.42
7/9
(H 14)
v1
9.89
0.907
0.946
95.71
96. 33
96.73
99. 64
99.64
95.95
95.
59 60
52. 02
53.29
85:48
88.68
77.66
79.73
46.91
84.75
83.98
1/9
v1 9.56
0.947
85.15
95.99
99.58
95.74
58.39
51.37
84.16
79.15
45.48
81.78
16:
v2
96.51
99.43
85.02
68.18
87.20
78.35
81.45
WiceResNat
v1
9.19
0.914
78.47
92. 02
81.14
14
67,
46
32, 46
43.26
42, 34
66.81
27. 64
79.92
8/9
(50_2)
v2
10.59
0.925
81.60
93,20
82.29
72.67
35.41
46.61
47,50
28.03
79.89
Figure 5: Predict OOD transfer performance on ImageNett models via Detf and Vi V For the first block of models, our prognostic indicators predicted that v1 would outperform v2. For the second block of models, our prognostic indicators predicted the other way around.
Experimental procedure. To test this principle, we analyzed 20 popular architectures from Py- Torch's official repository, each released with two sets of weights (vl and v2). By design, the v2 weights achieve higher accuracy on the ID) ImageNet benchmark. However, the specific changes in training procedure are often opaque to the end-user (see Table 5 for key differencess.. This hetero- geneity makes for a challenging and realistic testbed for our diagnostic framework. For each v vl/v2 pair, we first measured the Deff and Vef of their ImageNet object manifolds. We then evaluated their OOD transfer performance on 9 image classification datasets: Flowers 102 (Nilsback & Zisserman, 2008), Stanford Cars (Krause et al., 2013). Places365 (Zhou et al., 2017), Food101 (Bossard al., 2014), OXford-III Pet (Parkhi et al., 2012), etc. For each OOD dataset, we train a linear probe on the training set of the OOD dataset, and report the test accuracy (see Section 3.1 for details). See Appendix D for more experimental details.
Diagnosing transferability through ID effective manifold geometry. Consistent with the hy- pothesis derived from our initial explorations, we found that models with higher Dett and V ofl of-
9

---

Published as a conference paper at ICLR 2026
ten demonstrated stronger OOD transfer performance, when their ID ImageNet accuracy was lower. As shown in Figure 5, across the 20 architectures we examined, our prognostie indicators predicted that v1 would outperform v2 on OOD transfer in 14 cases (despite v2 having higher ID accuracy), that v2 would outperform v1 in 1 case, and yielded no clear verdict for the remainder. Among these 15 models and 9 OOD datasets, our prediction accuracy is 73.02% (92 out of 126). This is much higher than using ID test accuracy as a predictor for OOD performance (37. 3.7. We remark that using some of the other markers (e.g., Neural Collapse. Participation Ratio) also yields non-trivial prediction accuracy in OOD performance. See Section D.4 for more results.
Revealing differences in fine-tuning dynamics. Finally, explored whether these initial fea- ture advantages pensist during full-model fine-tuning. As expected from prior work showing that the benefits of pretraining diminish with longer fine-tuning (Kornblith al., 2019; He al., 2018), both v1 and v2 initializations ultimately converged to a similar performance level. However, we observed a drastic difference in the early fine-tuning stages: models initialized with v1 weights sometimes exhibited faster learning, hinting that their features may provide a more efficient trans- ferable starting point (Figures 22, 23). These results show that test-relevant geometric measures can reveal differences in fine-tuning dynamics, motivating future study on their role in transfer learning.
DISCUSSION
We introduced a diagnostic, system-level paradigm for anticipating generalization failure in neural networks. Instead of reconstructing detailed internal mechanisms, we treated task-relevant geomet- ric markers of ID representations as prognostic indicators. Through discovering prognostic markers in medium-sized experiments, we found that overcompression of object manifold dimension consis- tently predicts failures in OOD generalization. Applied to ImageNet-pretren pretrained models- a far more heterogeneous real-world setting-our prognostic measures predict which models transfer bustly across tasks. Together, these results demonstrate the power of a diagnostic framework for studying generalization failures. This work opens up several future directions.
Theoretical foundations. In Section 3.3, link overcompression of object manifolds overspe- cialization of learned features. Strengthening the theoretical basis of this hypothesis is important. A related question is whether incorrectly classified OOD examples share common traits that can be explained by the overspecialization intuition.
Causal mechanisms and interventions. Geometric indicators could inspire investigation into underlying causal mechanisms and practical interventions, such as geometry-aware regularization, carly-stopping criteria, model selection rules that prioritize robustness alongside accuracy.
Extending the proposed diagnostic research framework. Expanding our proposed analysis framework beyond vision to language, reinforcement learning, or multi-modal models remains an open challenge. A natural starting point to first identify and characterize the relevant failure modes in each domain, and then examine how representational markers correlate with those fail- ures. Another direction is to extend our findings into deployable protocols for diagnosing OOD failures across wider range of models and datasets.
Linking diagnostics to parameter transfer. A future direction is to explore whether insights from our controlled experiments can inform parameter transfer between models of different scales, in Net2Net (Chen et al., 2015). While our focus here is on diagnostics, connecting weight transfer could provide a complementary path for robust initialization.
Parallels with neuroscience. High-dimensional yet structured codes in the brain have been linked to generalization neuroscience studies. Our hypothesis linking manifold compression feature overspecialization may provide framework for interpreting these findings and exploring common principles across biological and artificial systems.
Theoretical work on neural networks has long been shaped by mathematics and physics, with an emphasis on bottom-up mechanistic explanations. We suggest that the history of medicine offers a complementary perspective: effective diagnostics can anticipate risks and guide treatment well before underlying causal mechanisms fully understood. Neural networks, as emergent high- dimensional systems, may likewise benefit from a diagnostic science that anticipates vulnerabilities and guides future mechanistic insight.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Hang Le for the helpful discussion. This work was supported by the Center for Computa- tional Neuroscience at the Flatiron Institute, Simons Foundation. S.C. was partially supported by Sloan Research Fellowship, a Klingenstein-Simons Award, and the Samsung Advanced Institute of Technology project, "Next Generation Deep Learning: From Pattern Recognition to AI." All experi- ments were performed using the Flatiron Institute's highh perforrmnee computing cluster. Yao- Yuan Yang worked in advisory capacity.
CODE AVAILABILITY
All code required reproduce the figures presented is available under MIT License
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.0164, 2016.
Mouin Ben Ammar. Nacim Belkhir, Sebastian Popescu, Antoine Manzanera, and Gianni Franchi. Neco: Neural collapse based out-of-distribution detection. arXiv preprint arXiv:2310.06223, 2023.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Sys- tems, 32, 2019.
Omri Barak, Mattia Rigotti, and Stefano Fusi. The sparseness of mixed selectivity neurons controls the generalization discrimination trade-off. Journal of Neuroscience, 33(9):3844-333 3856, 2013.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint 151 2015.
Chi-Ning Chou, Royoung Kim, Luke Arend, YaonYuan Yang, Brett Mensh, Won Mok Shim, Matthew Perich, and Sue Chung. Geometry linked to untangling efficiency reveals struc- ture and computation in neural populations. bioRxiv, 2025a. doi: 10.1101/204.002266552222
Chi-Ning Chou, Hang Le, Yichen Wang, and Yeon Chung. Feature learning beyond the lazy-rich dichotomy: Insights from representational geometry. In Forty-second International Conference on Machine Learning, 2025b.
Yeon Chung and Larry F Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology, 70:137-144, 2021.
Yuo Chung, Daniel Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 2018.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Patlern Recognition (CVPR), 2014.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 2020.
Corinna Cortes and Vladimir Vapnik. Support networks. Machine learning, 20(3):273-297, 1995.
11

---

Published as a conference paper at ICLR 2026
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models, 2023. URL https: /arxivv. org/abs/ 2309. 08600.
Alexander D' Amour. Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, al. Underspecifica- tion presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226): 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. Advances Neural Information Processing Systems, 37:24375- 24410, 2024.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in trans- fer learning. In International Conference on Learning Representations, 2022. URL https: / httpp/nneview.nee/eeeeeeeeeeeeee wett://ooorm/adSS11111111..
Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 21(1):271, 1988.
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary. Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, et al. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83): 2025.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665- 673, 2020.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF international confer-
computer vision, pp. 1134-1122 1144, 2021.
Md Yousuf Harun, Jhair Gallardo, and Christopher Kanan. Controlling neural collapse enhances out-of-distribution detection and transfer learning. In Forty-second International Conference on Machine Learning. 2025.
Yousuf Harun, Kyungbok Lee, Jhair Gallardo, Giri Krishnan, and Christopher Kanan. What vari- ables affect -ut-of-distribution generalization in pretrained models? Advances in Neural Infor- mation Processing Systems, 37:56479-5633 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training, 2018. URL pettss:/aaxtte htgg//b///81...00/
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2018.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 7744044844444 2017.
Bo Hu, Nesibe Z Temiz, Chi-Ning Chou. Peter Rupprecht, Claire Weissner-Bernart Benjamin Titze, Yeon Chung, and Rainer W Friedrich. Representational learning by optimization of neural manifolds in an olfactory memory network. bioRxiv, pp. 2024-11, 2024.
12

---

Published as a conference paper at ICLR 2026
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 7700-4708, 2017.
Artem Kirsanov, Chi-Ning Chou, Kyunghyun Cho, and SueYeon Chung. The geometry of prompt- ing: Unveiling distinct mechanisms of task adaptation in language models. In Findings of the Association for Computational Linguisties: NAACL 2025, pp. 1855-1888, Association for Com- putational Linguistics, April 2025.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better?, 2019. URL
805. 08974.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-graing graggaanndd categorization. In Proceedings of the IEEE international conference computer vision work- shops, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Michael Kuoch, Chi-Ning Chou, Nikhil Parthasarathy, Joel Dapello. James DiCarlo, Haim Som- polinsky, and Sue Chung. Probing biological and artificial neural networks with task- dependent neural manifolds. Conference on Parsimony and Learning (Proceedings Track), 2024.
Qianyi Li, Ben Sorscher, and Haim Sompolinsky. Representations and generalization in artificial and brain neural networks. Proceedings of the National Academy of Sciences, 121(27):c2311855 2024.
Victoria R Li, Jenny Kaufmann, Martin Wattenberg, David Alvarez-Melis, and Naomi Saphra. Can interpretation predict behavior on unseen data? arXiv preprint arXiv:2507.006445 2025.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- cat-of-diistibution generalization: survey. arXiv preprint arXiv:2108. 13624, 2021.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based aittof-distribution detec- tion. Advances in neural information processing systems, 33:21464-21175, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sue Chung. Emergence of separable manifolds in deep language representations. In Pro- ceedings of the 37th International Conference on Machine Learning, pp. 67 17-333,,,, 2020.
Wojciech Masarezyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Milos, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36:76772-76805, 2023.
Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, 99(3):609-623, 2018.
Maxime Méloux, Silviu Maniu, François Portet, and Maxime Peyrard. Everything, everywhere, all at once: is mechanistic interpretability identifiable? In The Thirteenth International Conference on Learning Representations, 2025.
Francesca Mignacco, Chi-Ning Chou, and Yuo Chung. Nonlinear classification of neural man- ifolds with contextual information. Physical Review E, 111(3))035302, 2025.
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representatiinss 2018.
13

---

Published as a conference paper at ICLR 2026
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayaman, Arnab Sen Sharma, Jiuding Sun, et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv preprint arXiv:2408.014 2024.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp- 722-729. IEEE, 2008.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: https//ddittill.ubb/// /tfp:///luuuuii///eeeuueu-siuula 7/feaaure-visuullzztionn
Christopher Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and
Shan Carter. Zoom in: An introduction to circuits. 2020. URL https://api.
semanticscholi org/ wttg :ttp:/ooooSII1::03333555
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 14652-24666 2020.
Nihaad Paraouty, Justin D Yao, Léo Varnet, Chi-Ning Chou, Yeon Chung, and Dan H Sanes. Sensory cortex plasticity supports auditory social learning. Nature communications, 14(1):5828, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, pp. PMLR, 2024.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and V. Jawahar, Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10428-10436, 2020.
Mattia Rigotti, Omri Barak, Melissa Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585-590 2013.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion, 2020a. URL https: arriv. org/abs 111///911100 08731.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizztion exacerbates spurious correlations. In International Conference Machine Learning, pp. 8346-8356. PMLR, 2020b.
Simonyan and Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
Sahil Singla and Soheil Feizi. Salient imagenet: How discover spurious features in deep learning?
arXiv preprint arXiv:2110.0430) 2021.
Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeon Chung. On the geometry of generalization and memorization in deep neural networks. arXiv preprint arXiv:2105. 14602, 2021.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry of population responses in visual cortex. Nature, 571(7765):361- 365, 2019.
14

---

Published as a conference paper at ICLR 2026
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference Machine Learning, Proceedings of Machine Learning Research. PMLR, 2019.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, a large collection of multi- dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-87778 2018.
Roman Vershynin. /figh-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Albert J Wakhloo, Tamara J Sussman, and Sue Yeon Chung. Linear classification of neural manifolds with correlated variability. Physical Review Letters, 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual
transformations for deep neural networks, 2017. URL https:// httpp:://aaxii.or/g//e.oo abs1161.. 05431.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized outoof-dittiibiition detection: A survey. International Journal of Computer Vision, 132(1))5533--662, 2024.
Yao- Yuan Yang, Chi-Ning Chou, and Kamalika Chaudhuri. Understanding rare spurious correla- tions in neural networks. arXiv preprint (xXiv:2202.05/88 2022.
Justin D Yao, Klavdia Zemlianova, David L Hocker, Cristina Savin, Christine M Constantinople, Yeon Chung, and Dan H Sanes. Transformation of acoustic information sensory deci- sion variables in the parietal cortex. Proceedings of the National Academy of Sciences, 120(2): e2212120120 20, 2023.
Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictio- nary learning: contextualized embedding linear superposition of transformer factors, 2023.
URL https::////// /arxiv. org/ abs /2103. 15949.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
1:X:v:1605.07146, 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Jiachen Zhu, Katrina Evtimova, Yubei Chen, Ravid Shwartz-Ziv, and Yann LeCun. Variance- covariance regularization improves representation learning. arXiv preprint arXiv:2306.1322 2023.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETTINGS
In this section, we provide a complete description of our experimental setup to facilitate repro- ducibility.
A.1 DATASETS
Our study utilized a range of standard image classification datasets, which served different roles: either as in-distribution (ID) training sources or out-of-distribution (OOD) evaluation benchmarks across two distinct experimental settings. Table 2 provides summary of these roles. Below, we describe each dataset and the specific preprocessing pipelines applied.
Table Summary of dataset roles in our experiments.
Experimental Setting
In-Distribution (ID) Dataset
Duttr--Distribution
(OOD) Datasets
Prognostic discovery (Section 3)
CIFAR-I0
CIFAR-100
ImageNet-Ie 1k (resized 32x32)
Transfer Learning Applications (Section 4)
ImageNet- lk
Flowers102 102
Stanford Cars Places365
Oxford-IIIT Pets Food- Food-101
iNaturalist 2018
DTD
FGVC-Aireraft HAM10000
Datasets for prognostic discovery. In our controlled medium-scale experiments, trained mod- els from scratch on a single ID dataset and evaluated their generalization to two different OOD datasets with disjoint classes.
CIFAR-10 (Krizhevsky et al., 2009) served our primary in-distribution (ID) dataset for train- ing. It contains 60,000 color images of 32 x 32 pixels, split into 50,000 training and 10,000 test images across 10 object categories. For training, mean (0.4914, 0.4822, 0.4465) and a standard deviation of (0.2023, 0. 0.1994, 0.2010). We also applied standard data augmentation: padding with 4 pixels on each side, followed by random 32 x 32 crop and random horizontal flip with 50% probability. For evaluating ID test accuracy, augmentation was disabled.
normalized images using per-channel
CIFAR-100 (Krizhevsky et al., 2009) used the primary int-of-diittiibttion (OOD) bench- mark. has the same image format and size as CIFAR-IO but contains 100 distinct object classes with overlap. For OOD evaluation, images were only normalized using the CIFAR-10 statis- tics; no data augmentation was applied to ensure a deterministic evaluation protocol.
second, more challenging OOD benchmark to test generalization under significant domain shift. This dataset contains over 1.2 million high- resolution images from 1,000 categories. To maintain compatibility with our CIFAR-trained mod- els, all ImageNet images were resized to 32 x 32 pixels using bicubic interpolation. They were then normalized using the standard ImageNet per-channel mean (0.485, 0.456, 0.406) and stan- dard deviation (0.229, 0.224, 0.225). No data augmentation applied during evaluation.
ImageNet-1k (Deng al., 2009) was used
Datasets for Transfer Learning Applications. In this setting, we analyzed publicly available models pretrained on ImageNet-Ik Ik and evaluated their transferability three downstream, fine- grained classification tasks.
ImageNet-1k 1kk served as the in-distribution (ID) dataset, as all models we analyzed were pre- trained it. For measuring the ID geometric markers, used the official validation set. Images
16

---

Published as a conference paper at ICLR 2026
were processed according to the standard pipeline for each model: resized to 256 x 256, center- cropped to 224 x 224, and normalized using the standard ImageNet mean and standard deviation.
Flowers102 (Nilsback & Zisserman, 2008) is fine- ane-grained OOD dataset containing 8,189 images of flowers belonging to 102 different categories.
Stanford Cars (Krause et al., 2013) is another ane-grained OOD dataset consisting of 16,185 images of cars, categorized by 196 classes (e.g.. make, model, and year).
Places365 (Zhou et al., 2017) is large-scale scene-centric OOD dataset with over 1.8 million images from 365 scene categories.
Oxford-IIT Pets (Parkhi al., 2012) contains a 37-category pet dataset with roughly 200 images for each class.
Food-101 (Bossard et al., 2014) includes 101,000 images of 101 food dishes (750 training and 250 test images per class). The dataset exhibits large variation in presentation, lighting, and style.
iNaturalist 2018 (Van Horn et al., 2018) consists of over 450,000 training images from more than 8,000 species of plants, animals, and fungi, collected and verified by citizen scientists on the iNaturalist platform. The long-tailed distribution and diverse real-world conditions make this dataset highly challenging for transfer evaluation.
Describable Textures Dataset (DTD) (Cimpoi al., 2014) contains 5,640 texture images amno- tated with 47 describable texture attributes. Images span varied materials, lighting, and scales.
FGVC-Aireraft (Maji et al., 2013) is fine-grained visual classification dataset containing 10,000 images across 100 aircraft variants. Images differ in viewpoint, environment, and model-year variations.
HAM10000 (Tschandl et al., 2018) is dermatology image dataset containing 10,015 dermato- scopic images drawn from seven diagnostic categories (e.g.. melanocytic nevi, melanoma, benign keratosis, vascular lesions). The images exhibit substantial variation acquisition conditions, anatomical location, and lesion appearance, making HAM visually and semantically distinct OOD dataset relative to natural-image pretraining.
For all three OOD datasets in this setting, images were resized to 224 224 pixels using bicubic interpolation and then normalized. During the OOD evaluation via linear probing, no data augmenta- tion was applied. For the full-model one-tuning experiments (see Figure 22), data augmentation was applied during the training phase, which included random horizontal flipping (with 50% probabil- ity) and color jitter. These augmentations were disabled during the evaluation of model checkpoints on the OOD validation subsets.
A.2 MODEL ARCHITECTURES
A.2.1 MODELS FOR PROGNOSTIC DISCOVERY (SECTION 3)
To ensure our findings generalize across different model design philosophies, our exploratory stud- ies included a diverse set of convolutional neural network (CNN) architectures. All models were adapted for CIFAR-scale (32 x 32 pixel) inputs and trained from random initialization, ensuring that their learned representations were not influenced by prior pretraining:
ResNet (He et al., 2016): family of foundational deep residual networks that utilize skip con- nections to enable effective training of very deep models. We used the ResNet-18, ResNet-34, and ResNet-50 variants.
VGG (Simonyan & Zisserman, 2015): Classic deep feedforward networks characterized by their architectural simplicity and sequential stacking of small 3 3x3 3 convolutions. We included VGG-13, VGG-16, and VGG-19, each augmented with batch normalization.
MobileNetV1 (Howard et al., 2017): A lightweight architecture designed for computational effi- ciency through the use of depthwise separable convolutions.
EfficientNet-BO (Tan Le, 2019): A modern, highly efficient model that systematically scales network depth, width, and resolution using compound scaling method.
DenseNet (Huang et al., 2017): An architecture designed to maximize feature reuse and improve gradient flow by connecting each layer to every other subsequent layer within dense blocks.
17

---

Published as a conference paper at ICLR 2026
This selection spans wide architectural landscape, including canonical residual and feedforward designs, modern efficient networks, and architectures with alternative connectivity patterns. This diversity allows to validate that our findings are general property of deep representations, rather than an artifact of a specific model family.
A.2.2 MODELS FOR TRANSFER LEARNING APPLICATIONS (SECTION 4)
For the experiments in Section 4, we shifted from training smaller-scale models from scratch across wide range of hyperparameters to analyzing publicly available, pretrained models to test our di- agnostic framework in a realistic setting. Our primary selection criterion was the availability of two official pretrained weight versions, typically labeled "v1" and "v2", within the PyTorch model repository.
This v1/v2 setup provides unique opportunity for controlled comparison. By design, the v2 weights offer higher in-distribution (ID) accuracy on ImageNet, often due to improved training recipes, data augmentation (e.g., AutoAugment), or regularization (e.g., label smoothing). This allows us to directly test our central hypothesis: whether ID geometric markers can identify cases where higher ID accuracy masks hidden vulnerability, leading to poorer out-of-distribution (OOD) transferabilityy
Our final set of 20 architectures is highly diverse, spanning multiple design generations and prin- ciples. In addition to deeper variants of models used in our control studies (ResNet-50/101/15,, MobileNetV2/V3, EfffcientNet-Bt), our selection also includes:
RegNet (Radosavovic et al., 2020): A family networks (e.g. RegNerY-400MF, RegNetX- 32GF) whose structure is discovered by optimizing data-driven design space, resulting in well- performing models.
ResNeXt (Xie et al., 2017): evolution of ResNet that introduces a cardinality dimension, increasing model capacity by aggregating set of parallel transformations.
Wide ResNet (Zagoruyko & Komodakis, 2016); A variant of ResNet that is wider but shal- lower, demonstrating that width can be more effective dimension for improving performance than depth.
A.3 COMPUTING RESOURCES
All experiments were conducted on NVIDIA HI00 (80GB) or A100 (80GB) GPUs, paired with
128 128-core Rome CPU and 1 TB of RAM. Training each model for 200 epochs required approximately 1-3 hours, depending on the architecture and optimizer. Unless otherwise specified, all experiments single GPU worker. These specifications, together with the full training configurations described in earlier subsections, are provided to facilitate reproducibility.
18

---

Published as a conference paper at ICLR 2026
B DETAILS ON ID MEASURES
In this section, we define the performance, statistical, and geometric measures used in our analy- sis. These are computed on the feature representations extracted from models using the ID training dataset, unless stated otherwise. Our goal is to identify which properties of model's ID represen- tations can serve as reliable indicators its out-of-distribution (OOD) generalization capability.
The measures are grouped into three categories: performance measures that quantify classifica- tion accuracy, statistical measures that summarize low-order distributional properties of features, and geometric measures that characterize the structure of class-specific feature manifolds. A key distinction is that while statistical metrics typically operate on pooled features, our primary geo- metric measures are computed on object manifolds the per-class point clouds in representation space. This allows them to directly capture properties relevant to classification, such as manifold size, shape, and correlation structure in the representational space.
We first describe how feature representations are extracted and then define each measure detail.
B.1 REPRESENTATION EXTRACTION
All representational measures computed feature vectors extracted from the penultimate layer of each network - the final layer before the classification head. This layer captures high-level, task- specialized features that are not yet collapsed into class logits. For convolutional networks, the feature vector is obtained via global average pooling. The exact layers used for each architecture are listed in Table
Table 3: Exact layer names used for extracting feature representations.
Architecture
Layer name in PyTorch module
VGG13
features 34
VGG16
features 43
VGG19
features. 52
ResNet
avgpool
DenseNet121
avg-pool2d
MobileNet
avg-pool2d
EfficientNetBO
adaptive aww.vvevvvg.pooo avl.ee22aoppoo22
RegNet
avgpool
ResNeXt
avgpool
Wide ResNet
avgpool
Given an ID dataset Did and trained network fo, let z, RN denote the N-dimensional feature vector for the i-th input sample in DID. extracted from the layer listed in Table 3. All statistical and geometric measures described in the following subsections are computed from the collection {z, JMM of such feature vectors, where M is the total number of samples in DID-
For measures that require class-specific statistics (e.g., within-class covariance, manifold radius), further partition }z,} by ground-trutt label into {z 12/'' M! for each class μ (1,..., P), where Timl
M the number of samples in class J..
B.2 STATISTICAL METRICS
We compute a of statistical descriptors from the ID feature representations quantify basic structural properties of the learned embedding space. All metrics are computed from the collection of penultimate-laye feature vectors M extracted from the ID dataset (see Table 3).
Activation sparsity. The activation sparsity measures the proportion of non-zero entries across all feature vectors,
sparsity
ΣΣ
1 1[2zl] E),
MN
19

---

Published as a conference paper at ICLR 2026
where N is the feature dimension and 1 is small threshold to account for numerical noise. Higher sparsity indicates more silent units on average across the dataset.
Covariance magnitude. We compute the empirical covariance matrix Σ RXXN over features and take the mean absolute value of its off-diagonal entries,
2 N(N 1)
mean covariance
jck
which reflects the average degree of linear correlation between distinct feature dimensions.
Pairwise distance. We compute the mean Euclidean distance between all pairs feature vectors,
2
mean_distance
12,
Bz 22112,
M(M
providing coarse measure of spread in the representation space.
Pairwise angle. After fg-normalizing each feature vector, compute cosine similarities and con- vert them to angles radians via 8ig = arccos sim,j). The mean pairwise angle reflects the typical directional separation between features.
All statistical metrics are computed on the raw feature vectors without centering unless required by the measure (e.g., covariance).
B.3 GEOMETRIC MEASURES: PARTICIPATION RATIO AND GLUE-BASED TASK-RELEVANT METRICS
Unlike the statistical measures described above, our geometric analysis operates on object mani- poist clouds in feature space containing activations from the same class. This distinction is important: geometric metrics explicitly quantify per-class representational structure, whereas most statistical metrics aggregate across the entire dataset without regard to class boundaries.
Participation ratio (PR). As a conventional baseline for manifold dimensionality, we compute [M the participation ratio (PR) of the penultimate-layer features for each class. Let {2, Ji=1 denote the M feature vectors for the j-th class, and X be the eigenvalues of their covariance matrix. The PR of this class defined as
DRR (EA00 E
which measures the effective number of principal components with substantial variance. In all figures we present the average of PR over all classes, i.e., + Σ M D While PR is widely used, it is task-agnostic and does not incorporate information about class separability.
Neural Collapse measure (NC1). In addition to per-class geometric descriptors, we also include global Neural Collapse-inspired measure that captures the degree of zero-collapse between within- class and between-class structure (Papyan et al., 2020; Harun et al., 2025). Let Ew N denote the pooled within-class covariance and Σg E IR N.N the between-class covariance of the penultimate-laye features (see Section B.2 for definitions), and let P be the number of classes. We first form a truncated pseudo-inverse of Σ by eigendecomposition. Write
Σg = UAUT, A = diag(A)......) AN). A 2 ...
and let Amaa = A1. We retain only eigen-directions with sufficiently large eigenvalues,
I = { X 2 Tmmax
with a small threshold (we use inverse
- in all experiments). and define the truncated pseudo-
Σ/ x u.u.
20

---

Published as a conference paper at ICLR 2026
where u, denotes the i-th column of U. The NC1 (zero-collapse) score is then
NC1 = tn tn(Ew
Smaller values of NC1 indicate stronger collapse of within-class variability relative to the between- class structure. We treat NC1 as geometric marker and compare with the participation ratio, numerical rank, and the GLUUE-bssed task-relevant measures in our prognostic analysis.
Tunnel Effect: numerical rank. Inspired by recent studies on the Tunnel Effect hypothe- sis (Masarczyk et al., 2023; Harun et al., 2024), we also compute the numerical rank of the feature representations. For given class p. let {z] (z)") A
fi=1 denote its feature vectors and let
M
Σn
Ee cw)(z, cc) MF
1
be the corresponding empirical covariance matrix, where C, is the class-mean representation (de- fined above). Let of 2 o > ... denote the singular values of En. Following prior work, the numerical rank of class y is defined
Rank = # (i:00 2 Tof), with 10-3
The reported value is the average over all classes, Ranknum Ene Rank Lower numer- ical rank indicates stronger compression of the class manifold. Prior work has shown that layers exhibiting low rank often display degraded OOD linear-probe accuracy. We include numerical rank as baseline geometric marker for comparison against the task-relevant GLUE-based measures.
B.3.1 TASK-RELEVANT GEOMETRIC MEASURES FROM GLUE
To capture the aspects of representational geometry most relevant for classification, we employ the effective geometric measures introduced in the Geometry Linked Untangling Efficiency (GLUE) framework (Chou et al., 2025a), grounded manifold capacity theory (Chou et al., 2025a; Chung al., 2018). The theory has found wide applications in both neuroscience (Yao al., 2023; Paraouty et al., 2023; Kuoch et al., 2024; Hu et al., 2024) and machine learning (Cohen et al., 2020; Mamou al., 2020; Stephenson et al., 2021: Kirsanov et al., 2025: Chou al., 2025b).
Analogous to support vector machine (SVM) wherr an analytical connection between the max-margin linear classifier and its support vectors is used assess separability in the best-case sense Wivse-GLUE establishes similar analytical connection in average-cass sense, as follows. Rather than analyzing the max max-marga margin classifier directly in the original N W-dimensional feature space R.N,LL considers random projections to d'-dimensional subspace and evaluates whether the representations remain linearly separable. Intuitively, if the data are highly separable in TRA, they will. with high probability, remain separable even after projection to much lower N'. Conversely, the data are barely separable RR,, the probability of maintaining separability will rapidly drop to zero N' decreases.
GLUE
Formally, following the modeling and notation in GLUE, each object manifold modeled as the convex hull of all representations corresponding the p-th class:
M = conv ((
where {2 } is the collection of M feature vectors of the 4-th class. A dichotomy vector {-1, 1) and a collection y C 1 are chosen by the analyst. Common choices are y being the set of all 1-vs-rest dichotomies (e.g.,
-I), (-1,1,-1,..,,--
(-1,-1,-1.... 1)) or 11.
The key quantity in GLUE for measuring the degree of (linear) separability of manifolds is the critical dimension, defined as the smallest N' such that the probability of (linear) separability after projection to a random N "d-dimensiinal subspace at least 0.5:
Navi == min N'.
p(N5)
21

---

Published as a conference paper at ICLR 2026
where
PNN)
Pr E RRN s.t. (", (w.x") 0, Vu, x" E M -RX
By scaling Neri with the numher of manifolds, we define the classification capacity a = P/Nerits which intuitively captures the maximal load network can handle. Larger a corresponds more separable manifolds in the average-case sense.
GLUE theory relates a manifold structure through:
-
(t. V (n. E yn X x
a P
max 20 Vj,
(4)
yuy (O,
Equation 4 can be numerically estimated using a quadratic programming solver (see Algorithm in (Chou et al., 2025a)).
Observe that one can view the optimal solution (y t) for the inner maximization problem as function of y, t. This naturally leads the following definition of anchor point for class μ
s" (y, = Σ, (x,
(y,,,,
and stacking them into a matrix S E PxN and let Sy = diag(y)S, GLUE yields an equivalent form:
a = P.
E yuy t~.((
(S,,))
(5)
where i denotes the pseudoinverse. This parallels SVM theory, where the margin is linked simple function on the support vectors.
Center-axis decomposition of anchor points. For each μ e [P], define the anchor center of the ju-th manifold as:
$ = Ey. (y, t)], and for each (y, t), define the axis component of the f--hh anchor point as:
(y
(y, $
Similar to Sy, we denote Sy.o, Sy,o.Sy.16 (y, E I PxN the matrices containing y"s and y (y, t) on their rows, respectively, i.c. Sy.0 == dyag and (y, ( t) where So and S (y, t) have si and sy t) stacked on their rows.
With these, define:
a(y, t) (Syt) (S,s))
b(y, t) (Sy. t)t)
(Sy. cly,t) t) (y, t)t) (Sy.08J. Sy, (x, (y.t)()))tt)
Note that a = P/Ey,,l((y,) t)).
Effective geometric measures. GLUE further decomposes into three measures:
1 + R
Velt
Deft
where:
Effective dimension:
1 Dett p Ey,(b((, ([(y)))] t)]
Intuitively, Det measures the intrinsic dimensionality of the manifolds while incorporating axis alignment between them. Lower Dett corresponds to more compact, better-aligned manifolds, improving linear separability.
22

---

Published as a conference paper at ICLR 2026
Effective radius:
Ey,lc t)] Rell V Ey,tlbyy -(y,
Intuitively, Ret quantifies the scale of manifold variation relative to its center, incorporating center alignment between classes. Smaller Ret reflects tighter clustering features around class centers, reducing manifold overlap.
Effective utility:
By (o(y, t]]
elf
y [a(ytt)
Intuitively, Ven measures the combined effect of sigmal-to-noise ratio (SNR) on separa- bility. Higher Vott corresponds to manifolds that are both low-dimensional and compact relative to inter-class distances.
For further derivations, illustrations, and examples, see the supplementary materials of (Chou al., 2025a). In all our experiments, for each manifold we subsample to 50 points, conduct GLUE analy- sis on each manifold pair, and apply Gaussianization preprocessing (Wakhloo et al., 2023) to ensure initial linear separability.
Pr = (st.:s))
Pr.
y.t (1( (y, t), t),
VuN
E 11 sy t))|l
Implementation details. In all our experiments, we consider the following specific hyperparame- ter choice for GLUE analysis. We randomly
Intuitions for GLUE measures. The three task-relevant geometric measures- Dett- Rett, and eflt- serve as markers that directly link geometrie properties of object manifolds to classification efficiency. As we show in later sections, they are substantially more predictive OOD performance than conventional measures. Here summarize key properties, examples, and approximations of GLUE measures in Table for intuition building.
For the f-thh manifold, define its anchor center as s := (t)) and the axis-part of the anchor point as (t) := st) - - Intuitively, s/ is the mean representation for the p-class, and sft corresponds to the within-class variation/spread. (:-)) denotes inner product and || ||2 denotes l norm. Formulas for uncorrelated random spheres provide useful mental picture: Det resembles the Gaussian width, equal to the sphere's dimension (Vershynin, 2018); Ren reflects ratio of wilhin-manifold variation to mean response; and Van corresponds to the fraction of error (i.e., inner product with t) attributable to within manifold variation.
"We follow top-down view of feature learning (Chou al., 2025b), where features understood func- tionally through their consequences interpretable axes or neurons. This perspective emphasizes how representational geometry changes with feature usage without requiring explicit identification of the features themselves. Moreover, by thinking of direction in the representation space feature (linear representation hypothesis (Park et al., 2024)), the effective geo- metric measures offer interpretation in feature learning as listed in the table.
computation (c.g., enabling linear separability) rather than specific
23

---

Published as a conference paper at ICLR 2026
Table 4: Intuitions for GLUE measures.
Den 2 0
Rat 2 0
Verl E [0, 1]
Geometric intuition
Quantify the task-relevant dimensionality object manifolds.
Quantify task-relevant spread within each mani-
Quantify amount
excessive compression of untangling manifolds.
fold relative to their centers.
Effect linear separability
More separable when Dett small.
More separable when Ret small.
More separable when Ven large.
Example
Den equals the dimension of uncor. random spheres
Ren equals the radius of un- cor. random spheres
Collapsing manifolds to points yields Van - 0.
Formula for correlated ran- dom spheres
E, E ( (ff
( 1l5
(of E, E
) (a)(()))))
P
E.
IE
Interaction with correla- tions among manifolds
If within manifold varia- tions align along similar directions, Det decreases.
If manifold centers move farther apart, RaF de- creases.
If within manifold varia- tions reduce without im- proving separability, Vat decreases.
Interpretation in feature Iearn- ing"
Low Den indicates a smaller set of feature modes use.
Low Rat indicates more similar feature usage across examples within class.
Low Ven indicates in- efficient compression of within-class variability.
24

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS FOR SECTION 3
C.I IMPLEMENTATION DETAILS
During the initial exploration of how OOD performance varies across a wide range of final model states, we trained all architectures from scratch on CIFAR-10. We used two optimizers: SGD with momentum of 0.9, and Adam (Loshchilov & Hutter, 2019). We ran training for 200 epochs with cosine annealing learning rate schedule, which smoothly decays the learning rate to zero, stabilizing late-stage representation geometry.
For each architecture and optimizer pair, we performed systematic 4 x grid search over the initial learning rate (7)) and weight decay (A). The specific values for each grid, which were tailored to each architecture family based empirical best practices, are detailed in Table 5 and Table 6. This diverse grid was designed to produce models in various training regimes, from under- to over- regularized, allowing to find cases where ID performance is stable while OOD performance varies a key aspect of our analysis.
Table 5: Hyperparameter grid for SGD optimizer.
Architecture
Initial learning rate list
Weight decay list
VGG (13/16/19)
[0.01000, 0.00333, 1,00111.000 0.00037]
[0.0010000, 0.0003333, 1,0001111, 0.0000370]
ResNet (18/34/5))
[1.00000, 0,50000, 0.25000, 0.12500]
[0.0002000, 0,0001000, 0.0000500, 0.0000250]
DenseNet121
|0.05000, 0.01667, 0.00556, 0.00185]
10.0005000, 0.0001667, 0.0000556, 0.0000185]
MobileNet
[0.20000, 0.06667, 0.02222, 0.00741]
[0.0001000, 0.0000333, 0,0000111, 0.0000037]
EfficientNetBO
[0.20000, 0.06667, 0.02222, 0.00741]
[0.0001000, 0.0000333, 0.0000111, 1,000011,0000000377 0.0000037]
Table 6: Hyperparameter grid for Adam W optimizer.
Architecture
Initial learning rate list
Weight decay list
VGG (13/16/19)
[0.02000, 0,00500, 0.00125, 0.00031]
[0.0100000, 0.0033333, 0.0011111, 0]
ResNet (18/34/50)
[0.10000, 0.02500, 0.00625, 0.00156]
[0.0100000, 0.0050000, 0.0025000, 0]
DenseNet121
[0.05000, 0.02500. 0.01250. 0.00625]
[0.0100000, 0.0033333, 0.0011111.
MobileNet
[0.02000, 0.00500, 0.00125, 0.04000]
[0.0100000, 0.0033333, 0.001
EfficientNetBO
[0.10000] 0.05000, 0.01000, 0.00100)
[0.0010000, 0.0003333, 0.0001 111,
C.2 DETAILS FOR FIGURE 4
In this section, we present supplementary figures that provide more detailed view of the main findings stated in Figure 4 from Section 3. Table 7 provides list of content for this subsection.
Quantification of Relationships. We quantify the relationship between ID measures and OOD performance by computing the Pearson correlation coefficient (r)) and its associated p-value via ordinary least-squares linear regression between the measure values and OOD accuracies. For all figures with heatmaps, we annotate each r-value with significance asterisks based on its p-value: < 0.0001 (****), ) < 0.001 (***), p < 0.01 (**), and S 0.05 (*).
Data Splits for Measures. The terms "Test" and "Train" in the figure labels indicate whether the representational measures were computed on the ID test set or the ID training set, respectively.
25

---

Published as a conference paper at ICLR 2026
Table 7: Organization figures in Appendix C.
Figure label
Model set
Optimizer
ID split
OOD dataset
Figure
Five DNNs
SGD
Train
CIFAR-100
Figure 8
Five DNNs
SGD
Test
CIFAR-100
Figure
Five DNNs
AdamW
Train
CIFAR-100
Figure 10
Five DNNs
Adam W
Test
CIFAR-100
Figure 11
Three ResNets Three VGGs
SGD
Train
CIFAR-100
Figure 12
Three ResNets Three VGGs
SGD
Test
CIFAR-100
Figure 13
ResNet18 + VGG19
SGD
Train
ImageNet subset
Figure 14
ResNet18 + VGG19
SGD
Test
ImageNet subset
SnrfomanceStatstiiaaaccco
Measures Distribbutoo Data
ReaWett
0.72
0.27
0.62
0:35
.0.3
0034
0:30
4,22
0.49
0.53
0.73
0:83
0.52
0:74
CLES
-0.35
0.44
5:75
-2.53
0.43
C,85
(903)
0.25
(903)
0.28
4.57
0.85
0.01
0.07
0.01
0.70
0.07
0.35
0.76
2.67
(902)
Densehae121
0.05
1524
0.21
0:33
021
0.02
0.05
0.63
0.53
0.35
0.05
0.67
5.24
0.77
0:72
0.63
0:20
Reclenta
0.50
0.76
0.79
0.42
0.55
0.17
0.67
0
0.56
0.97
0.25
(Azom/V)
1818
Conffrdeet
0.68
2:06
0.06
7:72
(A2om/7))
0.00
0.50
0.51
0.25
0.25
0.89
0.37
0.38
0.54
0.45
0.66
0.60
0.20
01.5
9.26
0.37
0.16
0252
(Abomtii)
0.20
0.45
0.:2
01/44
0.43
10.55
0.35
10.5
4..13
0.25
0:92
-0025
0.41
4.15
0.35
0.48
0.31
0.13
4..45
(503)
(5G2)
.0.16
-0.99
0.81
0.87
-3.75
A0
0.23
4.57
0.56
0.89
0.59
EL
DUE4
(502)
0.43
2.57
0.84
5.59
0.81
22.79
4.42
0.48
4,42
0.00
0
0U52
0.77
D-22
0.90
0.84
6.20
0.39
3.36
022
0.57
4.23
8.43
4.44
Dmrggeliet 9021
Imoggleet 902)
Rais Accomarr
Accuraii
Searity
Ceverianee
Festora Dastance
Restore Anghl
AURO
Brimay
Enarryy
Pertaalation Mososaaaioonno Radio
Weniiee Dimeroion
offeciiee Ratbor
Effeccios US8RY
Caeterr Algionent
Aligratent
Centaio Aigrommett
Figure 6: All results, measures computed on the ID frain
26

---

Published as a conference paper at ICLR 2026
0.73
0.17
0:47
0:25
41.17
-0.13
0:30
+0:03
0.11
0,60
0.50
0.28
0.53
8.32
0:74
2:45
+0:30
-..44
VGS
0.56
-.94
0.05
0.75
0:25
0.51
0.15%
0.01
0.37
0.61
0.43
0:41
0.06
FUOO
0.40
2.85
0.E%
0.29
MobFetee
0.07
-088
0.75
6:50
4.24
0.07
0,52
0.53
0:44
D:0G
SA
C/R
2.8K
0:20
3:25
0100
0.38
0.51
2:31
6.79
0.60
0
0:78
0.6R
D/6S
0.85
0.33
C
5-76
0.01
-3.25
-0.0
21
0.09
0:24
-0.4
0:33
0.2
3.52
0.0%
01
0.51
0:35
D/7
D.M4
0.77
C.77
0.20
0.43
kecuery
002
Figure 7: Five DNN architectures, trained with SGD, measures computed on the ID train set.
0.73
0.17
D:03
0.41
0.12
3:67
0:30
0:62
0.13
0.49
0.50
0.80
0.50
0:29
0.74
0.79
0.66
0:43
V99
0.50
0:95
0:08
0:70
0:73
3.56
0:55
0:85
0.87
0.81
0.30
0.90
0.54
0.27
0.94
0.88
02
0.19
030
0.25
Mobkerer
0.07
0.89
0.77
0:51
0:92
2.35
0,75
60
0.02
0.62
0.50
0.94
0.52
0:45
0.07
0.70
0.15
3.3
0.00
0.19
0.53
0.24
0.64
0.87
2.28
0:60
00
9.25
0.60
0.65
0:68
0.56
0:37
0.89
0.56
0.19
0.42
+0
25
-0.00
Denceleet
0.09
0:
4.00
0.33
0.23
0.52
0.05
62
0.53
0.35
0.85
0:69
0:66
30
0.64
0.56
0.15
0.11
Beit Srouery
000 Acconcy
Accnee
Ascunay
Aco/vo
000
002 Aoursec
Figure 8: Five DNN architectures, trained with SGD, measures computed on the ID rest set.
27

---

Published as a conference paper at ICLR 2026
0.50
0.28
0.76
8.42
0.50
0:51
0:37
+0.67
0.41
0.70
0:28
0.50
+0.30
D:47
-0.74
VSS
0.56
0.41
0.00
0.50
0.53
0.41
0.24
-0.75
4.50
0.53
0/50
0.56
+0.00
D:ST
0:25
0.31
0.2
MobFetee
023
0.12
0.71
0.60
0M
4.44
0.06
-0.34
4.31
0.71
0.77
5:57
D:54
T7
01
0100
0.50
0.51
0.75
0.25
0.03
5.35
0:30
0151
0.01
0.46
0.5
0.13
0.66
2-37
06
0.5
22
0.20
0.45
0.76
0.14
0.04
0.16
0:10
-0.35
4.77
0.75
0
0.94
10
0.41
5.40
000
000
100
y
Figure 9: Five DNN architectures, trained with AdamW, measures computed on the ID train set.
0.50
9.25
0:70
0.41
0.51
0:53
0:37
0:67
0.43
0.70
0.62
0:77
0:50
0:53
0.07
0:94
0.67
0.75
V99
0.56
0.41
0:50
0.50
0.52
3.50
0.24
0.73
0.50
0.53
0.69
9:24
0:79
0:67
0.71
0.14
0.27
0.4
0.0
Mobketee
0.23
0.12
0.71
0.68
0.47
5.06
0.06
0.34
4.35
6:71
0.72
0:83
0.54
8:63
0.83
0:76
0.67
0.04
0.00
0.50
0.51
0.25
0.28
0.49
0.46
0.38
0.54
0.05
0.46
0.50
0.29
6.70
0:
0.02
0.30
34
0:02
-0.00
Denceleet
0.28
0.45
0.76
0.14
0.43
5.60
0.30
0.35
0.27
0:75
73
0.89
0:55
0:73
0.94
0.92
0.67
0.00
Teat Srouery
000
CCO
000
002
000
Figure 10: Five DNN architectures, trained with W, measures computed on the ID rest set.
28

---

Published as a conference paper at ICLR 2026
1.00
RecNett
0:71
0:17
6:82
0:35
10:17
-0:13
0:30
8:82
0.11
0:45
0.50
0:73
0.01
0:37
0.24
0:65
0.55
Reckee3
4.36
0.0
V0013
00.16
0..49
6:85
0.57
0.7g
0.25
0.40
4.23
0:56
50
0:25
78
-9
VCGL5
0.45
0.87
6.86
0.80
0.90
0:42
0:42
0.02
0.05
0.79
0.02
0.03
0,00
-0.0
V0G1
Tast Ancuvss
Righe
EPeciine Bapni
COO AcourIcy
0:0 Acoraiy
200 Aoueecy
020 ACC
000 AccarO
020 Sranyy
000
Figure 11: Three ResNet and three VGG architectures, trained with SGD, measures computed on the ID train
29

---

Published as a conference paper at ICLR 2026
RocNett5
0:17
0.43
4.36
0.44
0.16
0.15
0.50
0.31
4..15
0.32
0.62
0.00
0.52
0.53
0.03
1.75
0.0%
0.50
0.37
0.50
0.52
-0.55
0.26
0.04
0.04
6,50
P3
-.455
025
9.00
VDG13
Wort
0.45
6:44
1840
1800
0.42
0.40
0.39
4444
0.02
0
1114
0.52
VOGLE
10
10 06
0.40
6.75
0.77
0.85
0.97
0.01
1:30
0.00
0.04
0.91
0
0.12
0:19
9.73
-1.0
CONNAR
Clock ElR
with
070
1000
Asurtiy
000 KCO0
Secrcyy
050 Acoieey
Accenoyy
Figure 12: Three ResNet and three VGG architectures, trained with SGD, measures computed on the ID test set.
1.41
0.50
0.50
0.70
0.59
0.44
0.89
0.85
0.70
Sali Acesnry
020 Acoreey
600 ACC
con souutey
000 soouaey
000 Leuurry
000 eocaac
020 actuiee
0700
Figure 13: ResNet18 and VGG19, trained with SGD, evaluated on ImageNet subset OOD, measures computed on the ID train set.
30

---

Published as a conference paper at ICLR 2026
AscNect8
0.60
VOGLE
4.50
6.79
0..07
0444
0:93
0:93
0.04
0.06
0.12
Toer Iea Aocurer
Roge
Effectie Ditiersann
Effectrr Dooy
Gentrr Aage
Centan
sourse
020 Acausey
000 lecaty
020 Acouacy
Joca way
Figure 14: ResNer18 8 and VGG19, trained with SGD, evaluated on ImageNet subset OOD, measures computed on the test set.
31

---

Published as a conference paper at ICLR 2026
C.3
RESULTS ON CORRUPTED IMAGES OOD DATA
Here we provide results on 6 out of 19 corruption methods in CIFAR-IOC.
Corruption: gaussian_
Resher18
0.07
+0.59
0.41
0.22
0.14
+0
0.65
0.60
0.43
0.71
10
Conndaliin Cooffr clant
V0G
0.08
0.01
0.05
0.03
0.01
0.17
0.10
0.01
0.05
0.01
0.5
0.56
0.37
0.58
00
EBlclentNee80
Densehee1121
0.00
0.30
0.03
0.39
0.61
-1.0
Taam Accormty
Non Accorecy
Spenda
Greenence
Festur Distonce
Feators Angle
Effecive Dimertom
Bitectie Radius
Effertee Haary
Partripation
43
Accurncy Aocurey GPXP10C
DCB Acourary
Acnraay OFAR-30C
Acturnyy
Partospation
Figure 15: Corruption type: gaussian noise. Five DNN architectures, trained with SGD, measures computed on the IDD rest set.
Comupttion
Restet11
0:44
0.96
0.14
0.22
-0.76
0.79
-0.58
0..7
-0.35
0.75
10
Corneatton Coothtiont
0.63
-0.97
0.3
Mobileteet
www
Hiiiientieeii
0.35
-0.43
00.00
Dergelee1121
0.00
0.94
-0.43
0 0,88
0.86
0.69
0.55
0. 0.62
4.
0.61
-1.0
Yrans Acconwy
Accomery
Copettance
Festure Destance
Feeture Auppe
Pettraunton Rata
Infectvve Dinenotion
Ereccio Raniaa
Iifuctoe Eabryy
bugg
Etertheuutry
Checthe
OGD ACOUERT CEAR-10C
033 Accunee CRAR-10C
OCD ACOUTICY CRAP-10
030 ACOUNCY GRA--I0
0CD AEW.AEU CRM--8C
Figure 16: Corruption type: fog. Five DNN architectures, trained with SGD, measures computed on the ID test set.
32

---

Published as a conference paper at ICLR 2026
Corruption: brightness
Resher18
0.47
0.58
0.30
0.34
+0.74
0.68
-0.44
-0.56
+0.95
+0
100
Telalalon Couff clani
WGG
0.74
0.99
0.69
0.08
0.72
0.58
0.85
0.37
0.47
0.97
-05
MobileNeet
0.55
0.89
0.84
0.87
0.88
0.44
-0.31
0.93
0.61
0. 0.94
00
ElliiinnNNei0
00.00
DenseNec1121
-0.33
Taaio Sicuaayy
Bost Aicunayy
Spendrs
Govenennc
Frstorm Ditence
Nesturn Angle
Partoipetoon Rana
Efeecive Dimersio
Eltection Raeiue
01:00
Staraity
0C0 Accurasy CIAR-52C
CC0 Accuresy CPA0-302
DCD Acouracy CPAS-10:
035 Acurway CFAS-00C
000 Accunssy GIAR-D0
Mffcaiie Jblice
A/9
6.7 1/5
Accuraty
Aocurayy CEAIMOT
DS0 Accunary CEAS-O0C
Acomary DEAS-O0
000 Accunery
Figure 17: Corruption type: brightness. Five DNN architectures, trained with SGD, measures com- puted on the ID test set.
Corruption: contrast
ResNer11
0.34
0.87
.0.04
0.16
-0.49
0.84
-0.79
-0.85
-0.1
4.90
10
Conel Telation Cooffeiees
VGG
0./4
0.99
0.70
40.69
-0.72
0.57
-0.83
-0.95
0.43
4.95
05
MobileNee-
0.51
0.55
0.82
0.86
0.87
0.42
-0.91
0.90
0.98
0 0.90
00
0.50
-0.5
DanseNor121
Been Accurayy
Bost Accorticy
Specato
Coverrence
Frstiom Distens
Fesmur Angle
Particpetim Rataa
Cleecivt Dinwestoon
Intertion Bedioe
Elfecore Uitler
Sparsity
CCC Accurasy CPNn 52C
CCD Accurasy CPA0--22
DCB Asourdy CRAB-10C
030 Acuaaay CPAB.10C
000 Acturosy CPAC 20C
blfccive sullir
Peaturn Ang
Accurscy Accurscy CHAN190
0C0 Ascurary
Aconary ORAN-10C
Accuracy
Figure 18: Corruption type: contrast. Five DNN architectures, trained with SGD, measures com- puted on the ID test set.
33

---

Published as a conference paper at ICLR 2026
Corruption: pixelate
10
-0:34
Conelstoon Cooffident
VGG
0.65
0.67
-.48
-0.32
-0.35
0.44
-0.31
-0.59
0.35
-0.57
05
Motslelket
027
0.11
6.11
0.04
0.05
0.26
-0.05
-0.12
0.19
4.12
00
Lfficienttett00
+0.4
0.39
0.27
00.77
-0.72
-0.07
-0.53
-0.05
0.36
+0 10
-0.5
4446
Tatos Acrteary
Tape Actenary
Spenads
Femture biraent
Freese bogit
Paticipatan Ratto
Efrectvn dimersinn
Effecino Ramtis
Emeation Daary
000 Accumcy CPR 15c
GCB Acouasy CPA
055 Acuracy CF38.00
000 Accuney CIAA-D0
A/9
Accurdsy Ascuraty CENIMOE
D50 Accurary CEAI-O0C
Acomany DEAR-OC
000 Acsunary GIWA-IDD
Figure 19: Corruption type: pixelate. Five DNN architectures, trained with SGD, measures com- puted on the ID test set.
Comuption: jpeg compression
ResNet11
e*e
**
VGG
0.83
-.*
0.85
-0.49
-0.45
-0477
0.41
-0.56
+0.76
0:30
-0.44
Mobileetet
0.80
6.18
-0.18
-0.77
-5.74
0.05
-0.36
-0.34
8:51
4..0
0.0
[fficientNeei0
0.33
0.55
0.30
0.32
0.36
0.17
00.30
0.01
-0.15
0.03
-0.5
Treis Accormrr
Yest Acceneee
Sporddyy
Covelannce
Dostavee
Festom Anges
Eleesive Disersaton
Edfectin Redlon
Dileciie Hate
Pertiqutiaa
Oooaterrce
CCC Acourscy CPRIL10C
000 Acurary DHAR-10C
000 Accurtcy CPALLEC
CCC Acouscy
000 dccurosy CHAN2D0
Stediio Uif
Offectee
5 St
Acouray
Figure 20: Corruption type: jpeg compression. Five DNN architectures, trained with SGD, measures computed the ID test set.
34

---

Published as a conference paper at ICLR 2026
D DETAILS ON THE APPLICATIONS TO PRETRAINED MODELS
Here we provide implementation details and statistical procedures underlying the pretrained model analysis in Section 4. This section accompanies the full results reported in Figure 21 and Figures 22, 23.
D.1 MODEL SELECTION AND WEIGHTS
We evaluated 20 pretrained architectures available through the PyTorch model zoo, spanning fam- ilies such as RegNet, MobileNet, ResNet/ResNeXt, WideResNett, EfficientNet, and Vision Trans- former (ViT). For each architecture, we included both the v1 and v2 weight releases. The two weight sets differ training recipes and regularization schemes, though exact details are not always disclosed, making them a heterogeneous and realistic testbed. By design, the v2 models typically achieve higher ImageNet top-1 accuracy, while weights often exhibit higher manifold dimen- sionality.
We remark that for the VIT models, we treat INAGENETIK.WWAG TVAGENETIK.SWAG.E2E as v2.
as v1 and
D.2 REPRESENTATION EXTRACTION
For each model, we extracted feature representations from the penultimate layer (see Table 3 for exact layer names). Input images were preprocessed by resizing to 224 x 224 pixels, converted to tensors, and normalized with standard ImageNet statistics. For GLUE analysis, subsampled classes and for each class we subsampled 50 feature vectors, applied Gaussianization preprocess- ing, and computed effective geometric measures (Der, Ran, Verr described in Appendix B. We repeated the above random subsampling for 100 times.
D.3 OOD EVALUATION VIA LINEAR PROBING
To evaluate the OOD generalization of the frozen fcature extractor, we attached a linear classifier to the penultimate feature representation of each pretrained model (see Table 3 for layer details). Crucially, the pretrained backbone weights remained frozen throughout this process; only the pa- rameters of the new classifier were trained. For each OOD dataset, train linear classifiers on the penultimate feature vectors for 50 epochs using the Adam optimizer with an initial learning rate of 0.1 and cross-entropy loss function. In all the results, we report the average linear probe accuracy over 3 repetitions on different random seeds.
D.4 PROGNOSTIC PREDICTION
For each model, after measuring the (Dett, Wett) of v1 and v2 respectively. We use the following criteria to make prognostic prediction: if the Den(z) Detr(y) is greater than the sum of the standard еиrоr of estimating Den(x) and Den(y), plus V ez) - Ven(y) is greater than the sum of the standard eиror of estimating V (x)( and (y) then we predict is going have better OOD performance than y; otherwise we make verdict (here y E [v1, v2
Recall that in Section we applied our prognostic method to 20 ImageNet pretrained perttrNetprrtrriieee models across 9 OOD datasets and achieved prediction accuracy of 73.02% (compared to 37.22% when using ID test accuracy as the marker). Here, we systematically evaluate other markers that showed reasonable performance in Section 3.2. Specifically, consider Dett and Ve before, along with the Neural Collapse metric, numerical rank, average within-class distance, and participation ratio (definitions in Appendix B).
The prediction procedure follows the same criterion described earlier: for each marker, we compare the two weight versions (v1 1 v2) and issue prediction only when the gap between their marker values exceeds the sum the standard errors estimation. We evaluate both individual markers and pairwise combinations.
The results, summarized in Figure 21, show that all these markers substantially outperform ID test accuracy as prognostic indicators of OOD transfer performance.
35

---

Published as a conference paper at ICLR 2026
Remark alternative markers and future directions. As shown in Fig. 21, several alternative markers- or combinations markers also achieve strong prognostic performance, and in some cases perform comparably to or slightly better than the specific pair (Dett, Vett) used in the main analysis. This fully consistent with the broader message of our work: a wide range of manifold- geometry-basse quantities, both within and outside the GLUE family, contain significant predictive signal for OOD transfer performance. A deeper understanding of why different markers succeed on different subsets of architectures and how these markers may complement one another an exciting direction for future investigation.
It is important to emphasize that the goal of the present experiment is not to identify a single "opti- mal" marker, but rather to demonstrate that geometric markers offer a substantial improvement over the conventional practice of using II test accuracy as predictor of OOD performance. Indeed, across all markers and marker-pairs we evaluated, the resulting prediction accuracies (ranging from 62% to 76%) consistently exceed that of ID test accuracy (37.22%) by a factor approximately two. This reinforces the central conclusion that geometry -aased diagnostics provide a robust and broadly effective alternative for prognostic prediction in transfer learning.
100
Prediction
Pairs of Markers
Fockrior
Eftectioe tniltty
23.42
72.22
72.65
26:22
22.22
80
74
72
Nearal Coll
62.67
70:37
73
Ma Mare of Ronk
93:57
62.70
Particpation
61
0.
traetve Eaity
Effectvv Deperonii
Numenica Rank
Hnivie Qlei Boan Distarcy
Effecive sievetion
Friical Calagsu
Bumeniaal Beick
bost anep Disbnsce
Bertoiianton Rudio
Perticaation
Nenural
Figure 21: Prediction accuracy of OOD performance using different markers (or marker combina- tions). a, Using single marker. b, Using a pair of markers.
D.5 FULL MODEL FNNETTNNIGG PROTOCOL
As complementary evaluation, also performed end-to-end fine-tuning Models initialized with either the v1 or v2 pretrained weights, and a new task-specific classifier head was randomly ini- tialized. Unlike the linear probe, all model parameters (both in the backbone and the new classifier) were updated during training.
To simulate a realistic application scenario, we fine-tuned the models on the complete official train- ing splits of Flowers 102 (6, 149 images) and Stanford Cars (8, 144 images). Training was conducted for 50 epochs with a batch size of 64. We used the AdamW optimizer (Loshchilov & Hutter, 2019) with weight decay of 10-6 and cosine annealing learning rate scheduler with an initial learning rate x 10
To monitor the learning dynamics, we evaluated the model's performance on the validation set at 40 checkpoints, spaced logarithmically throughout the training process.
36

---

Published as a conference paper at ICLR 2026
Fine-tuning Flowers102 regnet, x 166
regnet 32gf
regnet Bgf
1.0
v1
0.001
v1 0.098
0.900
000 Test accurac
0.8
v2
0.344
v2
E.0KS
0.6
0.4
0.2
mobilenet_ v3_ v3_lagg
regnet_ __6_off__6
regnet y_00mf
0.343
0- 0.918
0.908
000 Test accuracy
0.8
* D1922
UF 0.257
9 63995
0.6
0.4
0.2
0.0
regnet_y. B_00m 800mf
regnet_y @gnne_y____gff
efficientnet_
1.0
v1.
6- 11.29
O 11.22
D 12.66
00D lest accuricy
0.8
v2
D- 2.37
D B-43
2 12.63
0.6
0.2
0.0
resnext50 32x4d
regnet, y y_16gf
resnet101
1.0
V1
*
0/922
v1 0.00
E.90
000 Test accuracy
0.8
v2
0,909
v2 a 0.995
0.500
0.6
0.4
0.2
0.0
resnet152
mobilenet_v2 v2
resnext101 32x8d
1.0
D 0.900 6
CO13
000 Test accuracy
22.85
6 0.300
4 C913
0.6
0.4
0.2
0.0
wide resnet50,2 2
resnet50
1.0
v1.
2.13
D192
v1 8
0OD lest accuracy
0.8
v2
8 10.60
D 20.36
0.6
0.0
10a
10a
step
step
Figure 22: Fine-tuning dynamics of ImageNet-pretrainet networks on Flowers102 dataset from vl and weights. Insets show ID measures initialization
37

---

Published as a conference paper at ICLR 2026
Fine-tuning StanfordCars
regnet x 3.32gt
regnet_x X. X_
regnet y 8gf
v1 0.001
v1 0,890
v1 0.003
000 Test accuracy
0.0
v2
0.344
v2 a
v2 DuRss
8.23
0.6
0.4
mobilenet large
regnet, x_1_6gf
regnet_y 400mf y_00m
0.343
0.30
000 Test accuracy
x 0/817
0.6
0.4
0.0
regnet_y. yf000m
regnet_y_1_6g
efficientnet.S5
0.8
v1.
6 11.29
v1 B 11.22
v1.
B 33.66
000 Test accursc)
v2
D- 5-31
v2 D 5-43
v2
B- 12.03
0.6
0.2
0.0
resnext50 32x4d
regnet_y_ y_1gf
resnet101
000 Test accuracy
0.8
V1
* 0/922
v1 * 0003
v1 = D.55
v2
0,909
v2 0.90
v2 0.909
0.6
0.4
0.2
resnet152
mobilenet_vv
resnextid1__2 32x8d
D-
000 Test accuracy
0.8
0,900
0013
6 0.30
E322
= 0223
0.6
0.2
0.0
wide_resnette 2es_rrentttt_2
resnet50
0.9
v1.
2-12
0132
v1 D 9-57
GOD Text accurscy
v2
8 10.00
v2
10.35
0.6
0.2
0.0
10
10a
10a
step
step
Figure 23: Fine-tuning dynamics of networks on StanfordCars dataset from
vI and v2 weights. Insets show ID measures at initialization
38


Reviews:
[{"review_id": "v1hXzMPsSH", "reviewer": "Reviewer_qBkT", "summary": "The manuscript attempts to identify markers — statistics of the learned representations — that would help predict OOD failures. Comparing multiple marker candidates across multiple datasets, it concludes that object manifold dimension and utility are prime candidates for this. In a follow-up experiment, it is demonstrated that indeed they serve this purpose well.", "questions": "* Can you suggest what the optimal levels of the candidate markers are? Or an educated procedure on how to operationalize their use in practice?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* Fantastic scientific exposition of the idea, the design, and the results. \n* The found candidates for OOD failure markers are interesting and non-trivial. Thus, it may trigger future research on the mechanism beyond their contribution to identifying OOD failures, leading to a breakthrough in our understanding of the issue.", "weaknesses": "* The manuscript is performing what is called in statistical literature \"a fishing expedition\" for markers. For a fixed set of datasets, had the author tested thousands of markers, they could have reported selectively on the most promising ones, thus finding markers that succeed by chance and do not generalize to other datasets. I don't suspect the authors' ethics -- but they need to take measures against such a mistake. The authors can safeguard against random markers by calculating the number of markers for which the achieved result would have happened by change, showing they are well below it (e.g. \"this would have happened by change in probability 1/200, and we only tested 20 markers\").\n * The main result, that a reduced manifold dimension is associated with poor generalization, is somewhat paradoxical. Previous literature cited in \"related work\" reports a reduction in manifold dimension across layers of a deep network, which is associated with improved classification performance. Indeed, if an object were to collapse to a point, ID performance would peak, and OOD performance would diminish. On the other hand, an uninitialized deep network where the manifold dimension is very high would perform equally bad in ID and OOD tasks. The authors should acknowledge and address this weakness openly.\n * The authors' explanation for why these specific markers are best (section 3.3) seems post-hoc and premature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYIBEaFp7W", "reviewer": "Reviewer_idQw", "summary": "This paper introduces a top down, systems-level framework for understanding neural network failures inspired by similar approaches in biology and medicine. The framework is broken down into three stages: 1) identifying \"biomarkers\" or task-relevant measures of performance, 2) using biomarkers as prognostic tools in \"clinical experiments\" (i.e., to predict how markers identify failures to generalize), and 3) apply prognostic tools in a real-world application such as selecting the best pre-trained model for transfer learning. The \"biomarkers\" used in this paper lean heavily on prior work in representational geometry, specifically GLUE. They do a comprehensive empirical study to show that two measures from GLUE, the effective dimension and effective utility, reliably predict OOD performance better than less discriminative measures such as test accuracy. Finally, they apply their framework to the transfer learning setting they introduce earlier in the paper and show that geometric measures are more useful for predicting the best model to select.", "questions": "The paper's claim's can be broken down into:\n1. A framework for neural network failure diagnosis based on top-down approaches.\n2. A specific instance of (1) for image classification based on GLUE.\n\nQ1. For Claim (1), little time is spent discussing how the framework should apply to settings other than image classification. Figure 1 is vague-enough that it could describe nearly any top down approach. And this is not the only top down approach for OOD performance prediction (see [1-5]). Can you explain what this framework proposes specifically to do in general?\n\nQ2. For Claim (2), the paper focuses on GLUE, and it does a good job of explaining why the measures make sense for OOD prediction. But it isn't clear if this is the only measure that could work, or what properties of this measure make it effective. Based on the text, it seems that being task-relevant and discriminative are important, but then could [2] work? Are there alternatives?\n\nQ3. Related to Q2, stronger evidence for Claim (1) would be to show something outside of GLUE that works and captures the essential properties of an effective biomarker.\n\nQ4. The evaluation in Table 2 is limited to a set of models that where v1 and v2 differ in how much they train on the ID data. Could you test of a more diverse set of pre-trained models where it is less expected that the over-trained version performance worse OOD but better ID?\n\nQ5: [6] seems to have noted the relationship between OOD performance and the measures in this paper; it is probably worth citing as related work.\n\n[1] : https://openaccess.thecvf.com/content/CVPR2023/html/Yu_Distribution_Shift_Inversion_for_Out-of-Distribution_Prediction_CVPR_2023_paper.html\n\n[2] : https://proceedings.mlr.press/v162/yu22i.html\n\n[3] : https://openaccess.thecvf.com/content/ICCV2021/html/Guillory_Predicting_With_Confidence_on_Unseen_Distributions_ICCV_2021_paper.html\n\n[4] : https://aclanthology.org/D19-1222/\n\n[5] : https://proceedings.neurips.cc/paper/2019/hash/8558cb408c1d76621371888657d2eb1d-Abstract.html\n\n[6] : https://openreview.net/pdf?id=1ae108kHk2", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is extremely well-written, clear, and argues its central claims well. Despite relying heavily on prior work in GLUE, and therefore having little space to go over the theory, the authors do a good job of providing the necessary intuition for various concepts.\n2. The empirical evaluation is comprehensive, covering many model architectures, datasets, and hyper-parameter configurations. Not only does this go a long way towards bolstering the authors claims, I believe the existence of this evaluation is useful to the community in and of itself.\n3. Framing neural network interpretability work in the context of medical methodology is novel and potentially insightful.", "weaknesses": "1. While I appreciate the novelty of the medical framing, ultimately, none of the technical aspects of this framing translate into the framework. Instead of providing new insight, this perspective only seemed to confuse me. For example, the connection to \"biomarkers\" is far less important than emphasizing that measures of performance need to be task-relevant *as well as* descriptive of underlying mechanisms. The framing is not a deal-breaker, but I believe the paper would be stronger if it spent more time explaining what is unique about their measures w.r.t. predicting OOD generalization.\n2. Though there is strong evidence in the paper for the GLUE-based framework under image classification, there is little evidence or discussion of how to extend the framework to other settings or measures. Overall, it feels as though this should have been a paper about the applications of GLUE for OOD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iBo3ZzhPv8", "reviewer": "Reviewer_H4Eg", "summary": "The paper studies the question of predicting OOD performance of an image classification model having only the access to the in-distribution data. The authors draw parallels between the methodology applied in medicine to find meaningful biomarkers with significant predictive power about some phenomenon of interest. The parallel is exploited to find a set of analogous quantities for neural networks and their OOD performance based on the geometric structure of the neural network representations and the previously introduced GLUE framework. The paper argues for the approach using several experimental results suggesting the potential of the presented approach.", "questions": "1. How does the method compare to previously observed link between representations rank and ood performance? \n2. What is the complexity of the method? How it depends on the number of samples used for the estimation of the quantities? How sensitive this method is to noise in both data and labels?\n3. What are the weak points of the method? When can we expect this method to be a reliable source of information and when should we expect to break? How do these areas compare to using numerical rank?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The topic is of great importance -- being able to reliably predict a model's OOD performance without having access to the OOD dataset is of great importance.\n2. The presentation of the whole work is clear and interesting, with some room for improvement regarding the technical details (which I'll discuss later). \n3. I particularly appreciate authors' debating current trends which overly focus on studying models using tools developed in mathematics or physics. The idea to draw more inspiration from other fields, e.g. medicine is an important one to keep a healthy level of exploration within the community. That being said, I don't feel that the analogy is particularly helpful in this case (I'll elaborate on this in the weaknesses section).\n4. The claims made by the authors are mostly clearly supported by the provided and clearly presented experiments; I didn't have any feeling of the authors overclaiming their contributions.", "weaknesses": "1. The main issue I see with this work is the lack of comparison to previous works studying the problem. While the authors do reference several papers in related works, they seem to miss the core works that study the same questions. For instance, [1] introduced the Tunnel Effect Hypothesis, showing that the drop of OOD performance is strongly correlated with the numerical rank of representations. Further [2] refined the Tunnel Hypothesis, showing how the Tunnel Effect (and thus OOD performance) depends on multiple factors (e.g. image resolution, augmentation strength, architecture). While in this work, the authors study only 4 learning rates and 4 weight decays. Next [3, 4] study how one can influence the model's OOD performance by regularising for or against Neural Collapse, in particular [3] shows that OOD generalisation works against OOD detection. Finally [5] further confirms the findings from [1, 2, 3] and explains the mechanisms for increased or degraded OOD performance (something that the authors of this work purposefully set aside as they advocate for finding reliable markers first and trying to explain their mechanisms later).\n\n 2. The work is heavily based on the GLUE framework, which is only briefly introduced in the main paper, and crucial aspects such as explanation of the effective dimension, effective radius,and  participation rate, are sent to the appendix -- these are the crucial objects \nstudied in this work and require a proper introduction (and justification) within the main text.\n\n3. Limited scope of the experiments, while the experiments on image classfication are well executed I would expect more experiments focusing on different domains or different architectures especially given the fact that the experiments require training only linear classifier on top of representations, which can be easily collected from pretrained models which are freely available on repositories like HuggingFace or PyTorch.\n\n4. The analogy with medicine \"preclinical\" studies is exaggerated. Testing the validity of the approach on smaller, simpler datasets and applying it on bigger, more complex datasets is what a typical workflow in ML experiments looks like, I don't feel we need to justify it with the medicine-based approach. \n\n5. Overly used footnotes, please use them sparingly it makes it harder to follow the text when one has to jump to footnoote too often.\n\n\n[1] The Tunnel Effect: Building Data Representations in Deep Neural Networks, https://arxiv.org/abs/2305.19753\n\n[2] What Variables Affect Out-of-Distribution Generalization in Pretrained Models?, https://arxiv.org/abs/2405.15018\n\n[3] Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning, https://arxiv.org/pdf/2502.10691\n\n[4] NECO: NEURAL COLLAPSE BASED OUT-OF-DISTRIBUTION DETECTION, https://arxiv.org/pdf/2310.06823 \n\n[5] Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization, https://arxiv.org/abs/2506.01562", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RDF8WCTnS7", "reviewer": "Reviewer_YFoW", "summary": "The paper proposes to predict when models will fail to generalize especially under distribution shifts, even before deploying them. A diagnostic framework is presented that uses measurable properties of the model's internal representations. Through experiments on CIFAR-10/100 and ImageNet, the authors show that simple geometric indicators of in-distribution (ID) features, such as effective manifold dimension (Deff) and effective utility (Psi_eff), can serve as biomarkers for out-of-distribution (OOD) robustness. Applied to ImageNet-pretrained architectures, these metrics are able to identify which model weights will transfer better to new datasets like Flowers, Cars, and Places.", "questions": "How computationally expensive are the GLUE metrics used in this paper?\n\nWhat hyperparameters (if any) that must be set for computing the GLUE metrics?\n\nIn practice, are their any thresholds of when Deff or Psi_eff would indicate \"good\" vs. \"bad\" geometry?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a real gap: predicting generalization failure before deployment.\n\nThe results hold across several architectures, optimizers, and datasets.\n\nAlthough dense, the paper is well-written and the figures are quite helpful.", "weaknesses": "While the methods are well explained, the paper could present a step-by-step practical guide (an actual \"diagnotic kit\", as the authors suggest themselves). This would make the findings more easy for practitioners to apply when designing new architectures.\n\nIt is unclear how this framework generalizes to language or multimodal models. A short discussion about this or a pilot result would considerably strengthen the relevance of these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)