Gauge-invariant representation holonomy
Authors: Vasileios Sevetlidis
George Pavlidis
Venue: ICLR 2026 Poster (2026)
Keywords: Representation learning
Gauge invariance
Holonomy
Geometric deep learning
Robustness
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce representation holonomy, a gauge-invariant statistic that captures the path dependence of learned features, revealing geometric structure in deep networks linked to robustness beyond pointwise similarity measures.

Abstract: Deep networks learn internal representations whose geometry—how features bend, rotate, and evolve—affects both generalization and robustness. Existing similarity measures such as CKA or SVCCA capture pointwise overlap between activation sets, but miss how representations change along input paths. Two models may appear nearly identical under these metrics yet respond very differently to perturbations or adversarial stress. We introduce representation holonomy, a gauge-invariant statistic that measures this path dependence. Conceptually, holonomy quantifies the “twist” accumulated when features are parallel-transported around a small loop in input space: flat representations yield zero holonomy, while nonzero values reveal hidden curvature. Our estimator fixes gauge through global whitening, aligns neighborhoods using shared subspaces and rotation-only Procrustes, and embeds the result back to the full feature space. We prove invariance to orthogonal (and affine, post-whitening) transformations, establish a linear null for affine layers, and show that holonomy vanishes at small radii. Empirically, holonomy scales with loop radius and depth, separates models that appear similar under CKA, and correlates with adversarial and corruption robustness across training regimes. It also tracks training dynamics as features form and stabilize. Together, these results position representation holonomy as a practical and scalable diagnostic for probing the geometric structure of learned representations beyond pointwise similarity.

Published as a conference paper at ICLR 2026
GAUGE-INVARIANY REPRESENTATION HOLONOMY
Vasileios Sevetlidis; & George Pavlidis
Athena Research Center
Kimmeria Campus
Xanthi, GR-67100, Greece
{vasiseve, gpavlid) Bateenarc.
ABSTRACT
Deep networks learn internal representations whose geometry- how features bend, rotate, and evolve-affee affects both generalization and robustness. Existing similarity measures such as CKA or SVCCA capture pointwise overlap between activation sets, but miss how representations change along input paths. Two models may appear nearly identical under these metrics yet respond very differently to perturba- tions or adversarial stress. We introduce representation holonomy, a gauge-invariant statistic that measures this path dependence. Conceptually, holonomy quantifies the "twist" accumulated when features are in imput space: flat representations yield zero holonomy, while nonzero values reveal hidden curvature. Our estimator fixes gauge through global whitening, aligns neighborhoods using shared subspaces and rotation-only Procrustes, and embeds the result back to the full feature space. We prove invariance to orthogonal (and affine, post-whitening) transformations, establish linear null for affine layers, and show that holonomy vanishes at small radii. Empirically, holonomy increases with loop radius, separates models that appear similar under CKA, and correlates with adversarial and corruption robustness. It also tracks training dynamics as features form and stabilize. Together, these results position representation holonomy as a practical and scalable diagnostic for probing the geometric structure of learned representations beyond pointwise similarity.
transported around a small loop
INTRODUCTION
Modern deep networks learn internal representations whose geometry- features orient, align, and evolve-matters matters for generalization and robustness. Yet most standard diagnostics are pointwise: they compare two activation sets a fixed dataset using singular vector canonical correlation CCA (PWCCA), centered kernel alignment (CKA), or representational similarity analysis (RSA) thereby judging subspace overlap while remaining blind to how features move as inputs are varied along natural directions (pose, illumination, texture) (Raghu
analysis (SVCCA), projection-weightee
et al. 2017: Morcos al.) 2018 Kornblith al. 2019, Kriegeskorte et al., 2008). This leaves practical gap: two models can appear highly similar under CKA or CCA, and still behave differently under adversarial corruption stress because their intermediate features rotate differently along input paths.
We address this gap by turning alignment itself into object of study. We view a layer's representa- tion as field over input (or transformation) space and endow it with discrete connection: between nearby inputs we estimate shared principal subspace and compute the optimal special-orthogonal alignment (rotation-only Procrustes) of the two local feature clouds; composing these small rotations around a closed loop yields single orthogonal matrix whose deviation from identity we call repre- sentation holonomy. Nonzero holonomy indicates path-dependent (nonintegrable) transport in the classical sense of connections and their curvature (Ambrose and Singer, 1953). The construction is gauge-invariant by design: global whitening fixes sensible gauge by removing second-order anisotropy; orthogonal reparameterizations of layers leave the statistic unchanged; restricting to
low-rank shared subspace improves stability and cost (Schönemann)
1966; Kabsch 1976 Kessy
et al. 2018 Björck and Golub 1973 Davis and Kahan 1970).
*Corresponding author: VS, ORCID VS: 101-0001--3448-- GP: 1000-000-9999-15544

---

Published as a conference paper at ICLR 2026
Our proposal complements two nearby lines of work rather than competing with them. First, local equivariance tests (e.g.. Lie-derivative "local equivariance error") quantify infinitesimal sensitivity but do not assess global path denendeeende via loop composition (Lenc and Vedaldi, 2013;; Gruver architectures build a connection into the model so that desired transports integrable by design; we instead measure the emergent transport of standard models, providing a diagnostic that travels with existing practice in vision architectures (Bronstein al., 202T] Cohen al. 2019) Schonsheck et al. 2018 Masci al. 2013). In downstream terms, holonomy gives a compact, layer-wise summary of pathwise geometry that (i) is inexpensive
let al. 2022). Second, nauge-/maifoll-eeuivaaan
to compute, (ii) scales to common backbones, and (iii) adds information orthogonal to pointwise similarity, making it a natural candidate to relate feature geometry to robustness (Hendrycks and Dietterich 2019).
Contributions. (1) We propose a practical estimator of representation holonomy that combines global whitening, shared-neighhor subspaces, and rotation-only Procrustes alignment. The estimator is explicitly gauge-invariant and stable in the small-radius limit. (2) We prove formal invariances (orthogonal and, after whitening, affine), establish a linear null showing affine layers yield zero holonomy, and derive small-radius limir where holonomy vanishes linearly with loop radius. A perturbation analysis (Procrustes + Davis- Kea--aaaaannniinnn
provides explicit finite- sample and truncation error bounds (Schönemann) 1966 Björck and Golub 1973 Davis and Kahan 1970 Kessy ett al. 2018). (3) On MNIST/MLP and CIFAR-10/100 with ResNet- 18, we show that holonomy (i) increases with loop radius and depth even when CKA remains high, revealing pathwise geometry beyond pointwise similarity; (ii) rises during training as features form and stabilizes at convergence; and (iii) correlates with adversarial and corruption robustness across training regimes including ERM, label smoothing, mixup, and adversarial training (Kornblith et al. Hendrycks and Dietterich 2019).
Section situates our work among similarity metrics, equivariance diagnostics, and gauge-/manifold equivariant architectures. Section Eutiionnorrma formalizes the discrete connection and holonomy estimator and establishes invariance and small-radius results with finite-sample error bounds. Section 4 reports controlled loops, training dynamics, robustness studies, and ablations (whitening choice, SO vs. O, neighbor sharing. and k/q q sensitivity). Section 6 summarizes limitations and implications, and we release code and seeded configs for full reproducibility.
RELATED WORK
Comparing learned representations across networks and inputs is complicated by the fact that layer activations admit many equivalent parameterizations, i.e., a gauge freedom that allows local changes of basis without altering function. A large body of work therefore develops invarianv or basis-robust comparison tools. CCA-based approaches- SVCCA and PWCCA- compare the subspaces spanned by activations across models or training checkpoints, reducing sensitivity to neuron permutations while still depending on preprocessing choices and data coverage (Raghu et al. 2017, Morcos et al. (2018). Kernel-based Centered Kernel Alignment (linear and nonlinear CKA) has emerged as a simple and reliable alternative with improved stability across architectures, layers, and seeds, and clear links to representational similarity analysis (RSA) from systems neuroscience (Kornblith et al. 2019 Kriegeskorte et al. 2008). Beyond scalar similarities, a complementary line aligns entire representations by explicit linear transports: orthogonal Procrustes (and its det +1 Kabsch variant) yields optimal rotation-only maps between paired activation matrices, while principal angles quantify shared subspaces; classical perturbation theory Daavis-Kahan/Weiin provides finite- sample error control for the estimated subspaces and transports (Schönemann) 1966 Kabsch 1976)
Björck and Golub 1973) Davis and Kahan 1970). Preprocessing is itself a gauge choice: statistically principled whitening schemes such as ZCA-corT justiffy fixed global gauge that removes second- order anisotropy before any local alignment (Kessy et al. (0018). Parallel to these comparison methods, empirical tests of (approximate) equivariance and model equivalence probe how features change under controlled input transformations; early work proposed finite-difference tests for CNNs, and more recent formulations use Lie derivatives to define local equivariance error that is differential and layer-wise (Lene and Vedaldi 2015) Gruver et al. 2022). Geometric deep learning places these observations in coordinate-fre free framework: data domains (c.g., manifolds) come with local frames (gauges), and operations should be gauge-aware gauge- equivrrquiaa CNNs make the connection (in the -ifferential-ggomettii sense) architectural primitive, while manifold convolutions based on
2

---

Published as a conference paper at ICLR 2026
(A) Input space
(B) Feature space
(C) Holonomy
1.0
1.0
feature dim
0.5
feature dim
Input dim
0.0
0.0
0.0
-0.5
-0.5
-0.5
-1.0
-1.0
-1.0
Input dim
feature
feature dim 1
Figure 1: Holonomy as path-dependent feature rotation. (A) A small closed loop Y ,.,,...,,,-11, -1 To= in a 2D input slice. (B) The corresponding features 2; = =(2,) and their local neighbourhoods (zz); for each edge we estimate orthogonal transport Ra,++ that best aligns the two nearby feature clouds. (C) Composing these transports around the loop yields the holonomy H- Rt-1---- Ro. visualised the net rotation of a reference direction by angle 0. Holonomy is invariant to layer-wise gauge changes (global change of feature basis) and measures how much the representation "twists" when inputs follow loop, rather than just how similar activations are at individual points.
parallel transport or geodesic patches move features intrinsically across space (Bronstein et al. 2021 1
Cohen et al. 2019 Schonsheck et al.] 2018 Masci et al. 2015). Relatedly, Riemannian approaches on structured spaces (e.g.. SPD/Grassmann) deploy intrinsic means, transports, and normalizations inside networks, underscoring the usefulness of connection-lic operations on representation manifolds (Huang and Van Gool 2017; Brooks et al. 2019). Finally, robustness benchmarks such as ImageNet- C/P offer downstream behavioral checks; because they include sequences of small perturbations, they are natural testbeds for path-sensitive phenomena in representation geometry (Hendrycks and Dietterich 2019).
This paper adopts the geometric viewpoint but applies it as a diagnostic to standard models rather than as architectural constraint. We model layer-wiee wese representations over data (or transformation) space as sections of a vector bundle and make the alignment rule itself a connection: locally, we estimate shared subspace (with principled whitening as a fixed gauge) and define the transport between nearby inputs by the optimal special-orthogonal map in that subspace; globally, we compose these local transports around closed loops and quantify the resulting holonomy. By construction, our measurement is invariant to per-layer orthogonal reparameterizations (gauge transforms) and robust admissible whitening choices, distinguishing it from scalar, path-agnostic similarities such as CKA/RSA and from single-step Procrustes aligmments (Korbblihh et al. 2019 Kriegeskorte et al., 2008 Schönemann (966)). The Ambrose-Sinee perspective links small-loop holonomy to curvature, yielding concrete predictions that we test empirically; in particular, we show that networks can be locally equivariant (small Lie-derivative error) yet exhibit nontrivial global holonomy that correlates with stability under perturbation sequences, a phenomenon invisible to standard similarity scores (Ambrose and Singer 1953 Gruver et al. 2022 Hendrycks and Dietterich 2019).
REPRESENTATION HOLONOMY
Intuitively, layer's representation assigns each input a feature vector =(x) E RP, we move along small closed loop in input space (for example by composing small transformations), the corresponding features 2 (z) trace out a loop in representation space. Locally, between two nearby points on the loop we can align their feature neighbourhoods by an orthogonal map R1, E SO(p) that best matches the two clouds [ panels A-B). Composing these local transports around the entire loop yields net rotation H Ri-1- R₁ Ro (panel C). If the representation were perfectly "flat" along 7-for instance, if it were globally linear and we controlled for gauge-this -his product would be the identity. Deviations of H from I therefore quantify the path dependence (curvature) of the learned features, and are insensitive to global changes of feature basis.
At high level. the proofs rely on three standard tools: (i) Procrustes alignment in shared low- dimensional subspaces, (ii) matrix perturbation bounds of Davis-Kahan/Wer Kahan/Wedin type for controlling subspace errors, and (iii) finite-sample concentration bounds for covariance and whitening operators.
3

---

Published as a conference paper at ICLR 2026
We collect the technical details in Appendix S.3-S.5. At a given layer, call transform 2(x) H Q2(x) + b, with Q E O(p) (after whitening) and E R", a gauge transformation: an input- independent change of basis in feature space. Two networks related by such transforms internal layers are representation-cquivallee whitening, general invertible affine reparameterisationns) so that holonomy reflects only relative orientation changes induced by input paths rather than arbitrary basis choices.
Our estimator is invariant to these transformations (and, after
Let f E CC be a classifier and let 2 Rd Ie denote a fixed layer's representation (layer index suppressed). For inputs I E X' C IR we write z(x) E IRP. Given a small loop y (ro.... ................................ 2,--1, =1=)) in imput (or transformation) space, we define local linear transport R E SO(p) between the features at successive points and take the holonomy
[1(7)) - -IE
H()) = RL-1 R1 SO(p),
huorm (v)
2/p E [0.1],
(I)
reporting also the eigen-angle multiset (0,) of (7) (eigenvalues e on the unit circle). Con- ceptually, if is C² then first-order linearization suggests R; I + 0c(12:441 =;ll), hence H(7) - Oc(length(z)).
Estimator (used in practice). We pool a set /' of examples, compute features 2 (= their mean μ and covariance Σ, and fix global gauge by whitening ((x) = Σ (2(/2(2(2)) p) (ZCA-corr; any fixed symmetric square root suffices) (Kessy et al. 2018). For edge (Iz -z-1 1), let m, = 2 (ā(m)) + ((x++1)) and choose a shared index set 'n of size as the k-NN of m, in the whitened pool. On these same rows we compute shared soft center at the midpoint Ja
Ejer, j Zj: with weights w x exp(-112, m4l/o,,) and set X, = Y = Z,, P.. Let
X2 Y;
W; diag(w) (") JEI. and B, E RPx4 be the top-g right singular vectors of
In R we solve
orthogonal Procrustes: if U. = B.
W; (Y::)) then R 9))
U.V. E SO()4)
(enforce det 1) (Schönemann
1966 Kabsch 1976). We embed back to RPP by
R
R B + (I B.BBBBB E SO(p),
(2)
compose (T()) = R ... Ro. and report morm 1| || together with eigen-angles of Å.. Indeed, (I - [ ) = O so Rt R. = /; moreover det f2 = del n²!9) = +1. This construction
inexpensive (small SVDs in a shared subspace) and numerically stable.
Structural properties (statements; full proofs in App. F (i) Gauge invariance. If whitened features are reparameterized by any U E O(p), i.e., (x(x) -0: UZ(r), then the shared indices I, are unchanged, R2 U URUT and H = UNU T. Thus ||A!' 1|| = || - /|I and the eigen-angle multiset is identical. (ii) Affine invariance (post-whitening). For any invertible affine map on raw features, 2'(x) A2(x) + b, whitening by the corresponding pool statistics yields 3' Q 3(x) with Q (pp) (because -' 1/ 2 is orthogonal when E' AΣA )) hence the previous item applics. (ii) Linear null. If 2(x) = Bz + is affine and each edge uses shared rows, then X, = Y; for all i and R. = I, so H()) = I. (iv) Orienntation/cycling. Reversing a loop inverts holonomy, H (1-- (-)- so the Frobenius gap is unchanged; cyclic reparameterizations of n leave H unchanged. (v) Normalization. For any H E O(p) with eigen-angles {0,), || I| 2 ( cos 0z) < 4p, hence Inorm E [0, 1] with equality 1 iff all 0, = π. All invariance statements apply to the post-readout features; non-invertible readouts (e.g., JL) are outside the affine-ivvriaace claim.
Small-radius behavior (statement; proof in App. S.2). Assume is C2 with Lipschitz Jacobian on a neighborhood of 7/. the loop 7/r has total length Oc(r), the miidint k-NN has overlap
Pointers to supplement: App. S.O fixes notation; App. S.1 gives full proofs of invariances, nulls, and normalization; App. S.2 proves the small-radius limit; App. S.3 states a Procrustes perturbation lemma; App. S.4 handles subspace truncation; App. S.5 derives per-edge and holonomy error bounds; App. S.6 provides an explicit algorithm and App. S.7-S.8 cover complexity and practical implications.
4

---

Published as a conference paper at ICLR 2026
probability 1 Oc(r) - 0, and the subspace rank q covers the local feature rank. Then for each edge || 1||= = Oc(r) and
H (7+) 1||= Oc(r), hence Anorm (n+) Oc(r).
Intuitively, shared-row centering cancels translations; Lipschitz variation of Jg makes the optimal rotation deviate from by Oc([ - æxll); products of I+Oo(r) along L=0c(1) edges yield an overall Oc(r) deviation.
Estimator stability and error decomposition (statement; full derivation in App. S.S). Under standard sampling assumptions for the neighbor pool sub-Gaussian rows; a spectral gap separating the top-q right-singulat subspace), the per-edge error relative to the population transport R, obeys
11,11 C J
C2
+ TV(I, C4 [[2((141) J2 (x:))
linite sample
index mismatch
curvature
subspace truncation
(4) with I1 -I and constants depending smoothly on local condition numbers; composing over T==0((1) edges yields the holonomy error bound. Here (y{E,) denotes the q-th largest eigenvalue of the population covariance Σ, on the shared rows. The sinite-sample term follows from Procrustes perturbation via singular-subspce angles, the truncation term from Davis- Hiis-aaha//Wedi.. and the curvature/mismaach terms from continuity of J₂ and the shared-midpoint k-NN (Björck and Golub) 1973; Davis and Kahan 1970
Empirically, this decomposition matches the behaviour we observe in the vision experiments. Choos- ing k moderately large and smaller (e.g.. k E {96, 128, 192), q E {32, 64, 96) on MNIST hidden 1) keeps the sinite-sample and truncation terms small: Anorm varies only at the level of a few 10-? across this grid, without qualitative changes. The shared-midpoint k-NN construction effectively controls the for the two endpoints, holonomy increases markedly and becomes unstable at small radii. Finally, in linear or self-loop settings (affine networks, r-0 loops), /norm collapses to the numerical floor (~ 10-8 -010"?, ) indicating that once samile truncation, and mndex-mismatch effects are controlled, the remaining signal is consistent with genuine curvature of the learned representation field.
term TV J:): when we deliberately use disjoint neighbour sets
Per edge, forming the shared g-subspace (thin SVD of a (2k) x p stack) costs Oc(kpg), Procrustes in I& costs Oc(g*), and embedding costs Oc(pr); thus loop costs Oc [kpg + g²)). typically dominated by the subspace SVD. In practice, choose >> q (e.g.. k E [128, 192] with q E [64, 96) for vision layers), keep loop radii small to ensure neighbor overlap, project to SO (not O) to avoid reflection flips (Schönemann) 1966 Kabsch] 1976 Kessy et al. 2018). Per-neighborhood whitening induces stepwise gauge drift; allowing reflections (O(p) introduces x-flips; and using disjoint neighbor sets increases index noise-all create a non-vanishing bias floor as - The combination of global whitening, shared neighbors, SO-only Procrustes, and subspace transport removes this bias and restores the small-radin limit.
fixed global whitening. and
4 EXPERIMENTAL PROTOCOL
We study whether representation holonomy is valid, reliable, and usefull. This section describes datasets, models, training, readout/gauge fixing, loop construction, and the estimator. All figures the same code and seeded configs; details (incl. exact hyperparameters and seripts to regenerate CSVs/plots) are in the Supplement.
Convolutional feature maps are globally averaged (2x2 adaptive pooling only where explicitly stated), flattened, and projected by a fixed orthonormal Johnson-Linndestrraus map to 1024 if needed (only when the rost-readout feature dimension p exceeds 1024). Empirically this changes hnorm negligibly while reducing memory and runtime"] We fix gauge using global featurewise vean-variance standardization from a model-agnostic pool of 2208 representations. Note:
see Appendix S.B for a short numerical check
5

---

Published as a conference paper at ICLR 2026
Table I: Setup a glance
Datasets
MNIST; CIFAR-10/100 (standard splits; MNIST: 10" rot: CIFAR: crop+flip)
Models
MNIST: 2-layer MLP (512); CIFAR: ResNet-18 (3x3 stem; no max-pool)
Training
Adam; MNIST: 5 cp. 2x wd 10 10-";
CIFAR-10: 8 ep, Ir 10 wd 10 CIFAR-10: ep. Ir 10 wwd 5x10
Regimes (C10)
ERM: label smoothing 2=0.1: mixup 0.2: short PGD (step 2/255. ==4/255. steps)
Readout
GAP (2x2 adaptive only where noted): JL to 102-===11::000000000000000 y
Gauge fixing
Global featurewise z-score using model-agnostic pool 2p
Loops
Per test 26: 2D PCA plane from 512 nearest training neighbors (pixcls); 1=12-point circle
Radii
MNIST: {0.01, 0.02, 0.05, 0.10, 0.20}; CIFAR: f0.02, 0.05, 0.10, 0.20)
Estimator
Shared-midpoint k-NN; soft centering, joint q-dim. subspace; SO(g) Procrustes; embed to SO(p)
Defaults
MNIST: (k, q) ( (128, 61); CIFAR 1aye=2: (192, 96); seeds = 5
our theory assumes full-covariance whitening; we empirically compare z-scoring vs. ZCA-cort in the Supplement and find similar outcomes in settings. For each held-out test image zo. we form local 2D PCA plane using its 512 nearest training neighbors (in pixel space) and sample a regular 1=12-point circle of radius Varying Ppoinnt E {,, 8, 12, 16, 24) changes /norm smoothly and by less than 1.2 10-7, 7, with no sign of instability We report results across the radii sets above. For each edge on the loop: (i) find a shared k-NN in whitened space at the edge midpoint; (ii) softly center both point clouds; (iii) learn shared g-dimensional right-singular subspace from the stacked clouds; (iv) solve an (g(g) Procrustes alignment; (v) embed back to I2" as an SO(p) rotation. Composing edges yields H()) and Anorm || -lle with p the post-readout dimension. Unless stated, 2/p
defaults are as above. Reported intervals are as described in section + (Uncertainty and statistical reporting). Unless otherwise specified, we report two-tailed Pearson and Spearman p computed over (regime, seed) pairs (n = 20). Partial correlations II clean" residualize both variables on clean accuracy via OLS and correlate the residuals. Regression coefficients are standardized (z-scored predictors and targets); we report the coefficient/ 33 for holonomy together with its standard error (SE), p-value, and adjusted 12 of the full model (holonomy clean accuracy). For small-radius behavior we fit Anorm = + Br on r E {0.02, 0.05, 0.10} and report nonparametric 95% bootstrap CI for B using 4,000 resamples over the (regime, seed) rows. Error bars in small-radius plots denote standard error of the mean across (regime, seed) at each r.
RESULTS
We first study how holonomy scales with radius and depth, then examine its relationship to robustness across training regimes, and finally assess its stability and invariance properties. Figure 2 plots mean holonomy with 95% confidence intervals as a function of loop radius on MNIST and 10. On MNIST, both hidden layers exhibit clear positive scaling. Fitted slopes are 1.54x 10 for Hidden 1 and 6.10x 10-6 for Hidden 2, with corresponding means at r 0.10 of 6. 10-7 + 5.74x 10-9 (Hidden 1) and 2.86x 10 + 2.64x10 (Hidden 2) The deeper layer consistently exhibits larger holonomy and stronger radius dependence in this setting. On CIFAR- 10, layerl and layer2 show very similar positive dependence on radius; Figure Figure@hottom bottom overlays regime-wise CIs for both regime-ws
layers. Fitted slopes remain positive for both layers (Layer I: 2.52x 10-7; Layer 2: 3.66 10 10-9), with means at r 0 of 6.01x100 (Layer 1) and 4.45x10-7 (Layer 2). Across datasets we thus consistently observe positive dependence on radius. Deeper layers often exhibit larger holonomy (e.g., MNIST, Figure Figure(toop, top), although this trend is not strictly monotone across all architectures, and on CIFAR-10 the first two layers very close in magnitude (Figure bottom).
To probe the small-radius regime more directly, we aggregate CIFAR-IO layer2 across seeds and regimes and fit line over {0.02, 0.05, 0.10) (Figure 3, left). The fitted slope is 1.44x10 with a 95% bootstrap CI of [- x007x1007 4.22x 10-"] ". consistent with near-linear behaviour and the O(r) scaling predicted by Theorem 1.
On CIFAR-10 10 with ResNet- 18 we consider four standard training recipes: (i) empirical risk minimi- Sation (ERM) with cross-entropy loss; (ii) label smoothing (LS) with smoothing coefficient 0.1:
Experiment C (Appendix S.B)

---

Published as a conference paper at ICLR 2026
MNIST/MLP hidden1 holonorny radius
MNIST/MLP hidden2 halonomyy radius
10-5
1.2
1.0
0.6
1.0
0.5
0.2
(a) MNIST Hidden
(h) MNIST Hidden 2
CIFAR-10/ResNet-188 layer1 holonomy vs radius
CIFAR-1D/ResNet-18 layer2 holonomy vs radius
1--
3.0
3.6
6.0
32-
0.025
0.050
0.075
0.100
0.125
0.150
0.175
0.200
0.025
0.075
0,100
0.125 loop radius
0.150
0.175
toop radius
(c) CIFAR-I0 10 layerl
(d) CIFAR-IO layer2
Figure 2: Holonomy vs. radius MNIST and CIFAR-10. Mean +95% CI across seeds (MNIST) and across seeds and training regimes (CIFAR-10) 10). Both datasets exhibit positive dependence on radius; MNIST the deeper layer has larger amplitudes, while CIFAR- 10 the first two layers very similar in magnitude.
Sclf-loop blas mnist/hidden1
Seraliaadaa betiavior Jayee,, CRAN-
5.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
220
Buvee (self-loop.
10-7
Figure 3: Small-radius regime (left) CIFAR-10 (ResNet-18, layer2). Points show mean Inorm over seeds and training regimes; error bars are s.e.m. Slope estimate: 10- (95% CI [-1.07 10-7 (.22x10-7] Self-loop bias (right) near zero (MNIST Hidden 1, r 10 10-1) The bias floor ().0-8).
(iii) mixup with parameter = 0.2; and (iv) short projected-gradient-ded desceettt (PGD) adversarial training with foo bounded perturbations (radius 4/255, step size 2/255, small number of steps). Throughout, we use "adversarial stress" to denote test accuracy under single-step FGSM and multi- step PGD-10 10 attacks with these hyperparameters, and "corruption stress" to denote accuracy under simple lew-level corruptions (Gaussian blur, colour jitter, additive Gaussian noise), instantiated in the spirit of CIFAR- 10-CCsty-e corruptions. The robustness panel and Table 3 report clean, adversarial, and corruption accuracies for these four regimes.
At matched budgets on CIFAR-10 10, holonomy on layer2 systematically varies across ERM. label smoothing, mixup, and short PGD training (Table 3) At r= 0.100 the adversarially trained model exhibits the largest holonomy, followed by ERM, mixup, and label smoothing. A small, single- radius slice of holonomy already associates with standard stressors: across the four regimes
7

---

Published as a conference paper at ICLR 2026
Table CIFAR-I0 (ResNet-18, 1ayer2, radius r 0.1): correlation and regression of robustness
targets against holonomy nurm with clean accuracy as control. Coefficients are standardized.
Target
n
Pearson r
Spearman
Partial r clean
3 (std)
SE
Adj R
fgsm acc
0.805
0.565
0.223
0.080
0.085
0.36
0,950
Pgd10
20
0.809
0.501
0.276
0.051
0.043
0.253
0.987
corr acc
20
-0.785
-0.421
0.027
0.006
0.057
0.913
0.977
Table 3: CIFAR-10 regimes (layer2). Mean h at r=0.10 and held-out accuracies from the
robustness panel.
Regime
haorm r=0.10
Clean Ace. (%)
FGSM Acc. (%)
Corrupt Acc. (%)
ERM
3.46x 10-7
82.37
36.54
57.11
LabelSmooth
3.04x107
81.32
34.81
58.
Mixup
3.19x10-7
74.11
22.51
49.54
AdvPGD
4.74x10-7
12.24
67.85
11.96
Correlations (h vs. cleam//GSSMcorrrp.). R2 -0.96, RJ 0.94, R -0.96.
observe strong correlations between mean holonomy and FGSM/corruption accuracies (r94 and P 0.96), and corresponding inverse relation with clean accuracy (ræ 0.96). Regimes that are more adversarially robust (higher FGSM accuracy) tend to have larger holonomy but lower clean and corruption accuracy, indicating that representation holonomy tracks tradeoffs along the robustness-accuracy frontier at the regime level.
Regime-meant thus show strong descriptive correlations between holonomy and robustness across the four training recipes. However, per-seed analysis conditioning clean accuracy indicates only modest incremental signal: at r = 0.10, partial correlations are r x 0.22..28 for FGSM/PGD-10 and near zero for CIFAR-10-C (Table
To isolate what holonomy adds beyond pointwise comparisons, we aligned MNIST Hidden 1 test activations with an orthogonal Procrustes map and computed linear CKA. Despite very high aligned CKA (0.987), the composed holonomy remains nonzero; the post-alignment Frobenius misfit is 10 8 yet loop composition still accumulates measurable twist. This control shows that near-identical pointwise representations can possess different pathwise geometry, and that holonomy detects those differences.
We pre-registered a sensitivity slice and ablations. At r = 0.10 on MNIST Hidden 1, vary- ing (k, 4) E {96, 128, 192} x {32, 64, 96) changes huoru by only 70-0 end-to-end (SD 2.86x10-7) Increasing the standardization pool from 103 to 8x 10 shifts h by 6.49x10-* (from 10055100-5 4.06x10 9). indicating practical insensitivity to Npoot- Ablations confirm that each
"bias guardrail" matters: switching from SO(p) O(p) (reflections allowed) raises h by 5.37x10 10-7 on average; using per-neighborhood (local) rather than global whitening increases / by 1.59 10 10-7; and, critically, dropping shared-mise mindoint neighbors (separate h-NNs per edge endpoint) catastrophi- cally inflates measured holonomy (e.g., +2.22x 10-1) even with other safeguards on. Finally, using random plane instead a local PCA plane reduces modestly by 1.92x 10 at r = 0.10, contextual- izing our loop construction choice. Varying only the loop discretisation Apoints over {6, 8, 12, 16, 24} at fixed radius on MNIST Hidden yields smooth curve with Anorm in the range 3.5-4.7 x 10-7, further supporting numerical stability of the estimator with respect to loop discretisation.
A near-zero self-loop (r 10-4) produces numerically tiny bias floor on MNIST Hidden 1 (mean 1119x10 10- max 5.04x10 10-8), A complementary small-radius study (Experiment D, Appendix S.B) on a separately trained MNIST MLP at Hidden 1 yields knorm & 3.08 x 10-7 for exact self-loop (r = 0), and for PCA loops with radii r E {10-3 2 x 10 10-3 x 10 10-2 x 10 all values lie in the narrow band Amorm [2.37, 2.46] x 10 (variation 3 x 10 $). Together, these numbers characterise the numerical floor of our estimator in this setting and are consistent with the ((r)
small-radius behaviour predicted by Theorem 1.
8

---

Published as a conference paper at ICLR 2026
1e-7 Poal stability
mnisthiddenn
MNIST//MLF hidden1 PCArplane rendon--llaee
5.5
pic nundem
5.0
4.-
35-
3.0
10
2xx00" 30100%100"
0.075
0.125
0.150
pool size (log)
(a) Pool size stability
(b) Plane ablation (PCA vs. random)
Figure 4: Reliability/stabiiity. Left: hnorm is nearly flat as Nool increases. Right: PCA planes yield
slightly higher, more geometry-awe aware holonomy than random planes.
Replacing nonlincaritics by identity (linear null) drives holonomy to noise level (mean 9.57 x 10- SD 2.22x 10- Gauge invariance holds: post-multplyingg the readout by random orthogonal basis changes h by only 10- on average (MNIST: Ah = 1.17x10 8; CIFAR-10: 1.65 10-*) and leaves the eigen-angle spectrum near-identical (mean L2 discrepancy R 7.1 x on MNIST; = 7.88110- on CIFAR-10). Orientation reversal behaves expected: composing the forward loop with the inverse yields a tiny normalized gap (7.14x10 10 on MNIST; x.53 on CIFAR-10). Across datasets, layers, and training regimes, representation holonomy (i) validly measures a pathwise geometric effect distinct from pointwise similarity, (ii) is reliable under reasonable readout/estimator choices provided bias guardrails are kept, and (iii) is useful, describing adversarial and corruption robustness from a small, fixed-radius probe early in the network. Extended stress tests (PGD-10) 10, CIFAR-10-C, partial correlations) and additional spectra/ablations deferred to the Supplement.
DISCUSSION
Our estimator targets local, gauge-invariant property of the learned representation field: the parallel transport induced by the network when we traverse small input-space loop. At a given layer with feature dimension p. we compose per-edge transports in an estimated g-dimensional subspace (embedded back into SO(p)) and summarise the loop via hnorm = || H || (2v/) and, when needed, the eigen-angle spectrum of H. This statistic is complementary to pointwise similarity measures such as CKA, SVCCA, and PWCCA: those compare unordered sets of activations at fixed inputs, while holonomy probes how features evolve along a path and whether composing local transports around loop yields a non-trivial "twist". In particular, two networks can exhibit near-maxinn maximal aligned CKA yet differ in holonomy, indicating different pathwise geometry despite almost indistinguishable pointwise alignment; our MNIST and CIFAR-10 10 experiments give concrete instances of this "CKh-high but holonomy-differen regime.
Holonomy is not single global distance between models, nor evidence of topological monodromy in data space. It captures curvature-like effects local to the family of loops under consideration, and depends on both loop design (centre, radius, plane) and the feature metric (made explicit by whitening). Our gauge choice (global whitening, shared k-NN at edge midpoints, rotation-only Procrustes) removes arbitrary reparameterisations of feature space, so that hnorm reflects genuine changes in representation orientation along input paths rather than artefacts of the basis.
Empirically, we find holonomy most useful three situations. (i) Early-epoch selection: small- radius hnorm measured early in training already correlates with eventual robustness across regimes, providing cheap, label-free signal for choosing runs or stopping early. (ii) Diagnosing geometry vs. alignment: when pointwise similarity metrics indicate that checkpoints are nearly identical, holonomy can still separate them by sensitivity to small input transports, shedding light on robustness or transfer differences that CKA alone does not explain. (iii) Layer-wise profiling: holonomy radius and depth highlights where the network introduces most path dependence, which can guide where regularise suggest small radii (where hnorm scales roughly linearly). k > q with thared-neighher selection at midpoints, and reporting distributions (medians and IQRs) over loop centres and planes. Stability
function of
where to attach heads in transfer settings. For reliable use, experiments

---

Published as a conference paper at ICLR 2026
MAcerry Kicend gan
6:0
1.30
13:4
Figure 5: A single MNIST digit is translated around small 4-step loop. At conv2, the nearly translation-equivvriant CNN yields an almost closed feature loop and tiny holonomy, while the aliased CNN produces a distorted loop and holonomy about three orders of magnitude larger.
diagnostics such as neighbour-overlap IoU and the fraction of variance captured by help detect pathological settings that inflate variance.
Holonomy also has clear limitations. It is inherently local: it summarises curvature near the sampled loops rather than a global property of the data manifold, and results depend how loops are constructed. PCA planes around a datum provide reasonable default but may not always align with semantic directions, especially off off-manifold. Global whitening assumes a single feature metric; strong conditional anisotropy can bias neighbourhoods and centres. The shared-midpon reduces index noise but may under-represent rare modes, and rotation-only Procrustes deliberately discards scaling and shear, scalar horm will under-report effects dominated by those components. Finally, although the estimator linear-time in pool size and practical at CIFAR/Imagel with compression, very deep models dense grids of radii and planes can still be costly, reporting confidence intervals and wall-clock helps make comparisons transparent.
heuristic
scales
Some extensions augmentation orbits (e.g., small rotations or translations), to domain-shiff curricula, or to generative manifold paths can better align the probe with semantics and reduce off-manifold artefacts; short geodesic rectangles would directly probe commutators of input directions. Second, richer gauges and architectures: per-class per-mode whitening, equivariant layers with structured gauges, and transformers with token- and position-wise gauges all offer sharper tests. Third, an especially natural application is diffusion / score networks, where the learned score field is theoretically curl-free but in practice may deviate from this ideal; holonomy could expose such non-curl-fee structure along generative trajectories.
particularly promising. First, beyond-local loops: constraining loops to
7 CONCLUSION
We introduced representation holonomy gauge-invariant statistic of learned feature fields, to- gether with practical estimator based on shared-neighhour Procrustes transport in low-dimensional subspaces. Theoretical analysis shows that, after whitening, holonomy is invariant affine repa- rameterisations, vanishes on affine maps, and scales linearly with loop radius under mild regularity assumptions, with an explicit error decomposition separating finite-sample, cabspace-truncalte index- mismmach, and curvature contributions. These properties make holonomy a well-defined, local notion of "'uurvature" for layer-wise representations rather than an artefact of arbitrary feature bases. Holonomy is complementary to standard indistinguishable under aligned CKA can still differ in holonomy and in robustness, and training recipes that change robustness also systematically modulate hnorm This supports the view of holon- omy as a diagnostic tool rather than replacement for existing metrics. Its locality and dependence on loop design make it well suited for probing specific hypotheses about representation geometry for example, along augmentation orbits, domain-shiff curricula, or generative paths-whil while its gauge invariance enables meaningful comparisons across checkpoints, architectures, and training regimes.
petpeeettttionssmmmtttiit measures: networks that are almost
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work has been partially supported by the ARGUS EU project (Grant Agreement No. 101132308), funded by the European Union.
REFERENCES
Warren Ambrose and Isadore M Singer. A theorem holonomy. Transactions of the American Mathematical Society, 75(3):428 443, 1953.
Åke Björck and Gene H. Golub. Numerical methods for computing angles between linear subspaces. Mathematics of computation, 27(123):579-559) 1973.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13778, 2021.
Daniel Brooks, Olivier Schwander, Frédéric Barbaresco, Jean- vves Schneider, and Matthieu Cord. Riemannian batch normalization for spd neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning, pages 1321-1338 PMLR, 2019.
Chandler Davis and William Morton Kahan. The rotation of cigenvectors by perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1-46, 1970.
Nate Gruver, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie derivative for measuring learned equivariance. arXiv preprint arXiv:2210.0298. 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12266 2019.
Zhiwu Huang and Lue Van Gool. A riemannian network for spd matrix learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Foundations of Crystallography, 32(5):222-923, 1976.
Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309-314 2018
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519- 3529. PMIR, 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 991 1999, 2015.
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pages 37-45, 2015.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
Maithra Raghu, Justin Gilmer, Jason Yosinski. and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2026
Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Stefan C Schonsheck, Bin Dong, and Rongjie Lai. Parallel transport convolution: A new tool for convolutional neural networks on manifolds. arXiv preprint 1rXiv:180..07857, 2018. 2018.
12


Reviews:
[{"review_id": "Lkg3pGUOuI", "reviewer": "Reviewer_Jj9N", "summary": "The authors define, implement, and test a measure of Holonomy, designed to quantify geometric distortions within network representations, in a way that current metrics do not.  The authors provide mathematical definitions, details of computation/implementation, and empirical demonstrations on a variety of datasets and models. These demonstrate that holonomy differs from other metrics in the literature, confirm expected properties (increases with stimulus loop radius) as well as less-expected properties (increases with network depth, decreases with training, and correlates with average network robustness to noise or adversarial attack), suggesting its usefulness in applications of diagnosing and improving networks.", "questions": "In Figures 1/2, why does measured holonomy not tend to zero as r goes to zero? Also, can the authors rule out the possibility that the increase of holonomy with loop radius is a result of increased numerical error?  Does the trend persist if the number of loop samples is adjusted in proportion to loop radius r?\n\nHave you measured holonomy of a score network used for diffusion generative modeling?  Some literature suggests that  generalization and equivariance to geometric distortions are more robust than for recognition networks (eg, Kadkhodaie et al 2024).  But they are also typically not curl-free (even though an optimal denoiser should be, since it computes a score), and this failure would presumably be exposed by the holonomy measure.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is an important topic.  ML networks are highly redundant in their parameterization and optimization landscapes, and it is\ncritical for the field to develop measures that can characterize fundamental properties for use in analysis/comparison/design.\n\nThe paper is built on a mathematical foundation that is rich, and perhaps not so familiar to a large portion of the NeurIPS community.  The authors have generally done a good job of defining and explaining their construction, and the community will benefit (but see below).\n\nThe authors have also done a careful job of implementing and testing their measure, providing reasonable detail, evaluation of computational cost, verification of expected properties, and demonstration of potential uses.", "weaknesses": "The main weakness, for me, is that after several passes I am still uncertain of exactly why the properties captured by holonomy (distortion of geometry along closed-loop paths) are essential to understanding or comparing network functionality.  Specifically, under what conditions are the properties captured by holonomy, but not captured by current alignment methods (CCA, RSA) or differential analalyses (Lie derivatives, or simply comparison of local Jacobians), essential for evaluating or improving a network?  When is loop composition/path-dependency important (as opposed to simpler tests of invariance or equivariance)?  Perhaps the authors could provide more examples of applications/tasks where this arises?\n\nAnother weakness: Given that derivation and computation are complex and involve many approximate steps, it would be really useful to verify the entire enterprise on a few simple examples for which ground truth is known.  For example, image translation loops represented in a purely convolutional (and thus translation-equivariant) architecture, vs. a representation with known aliasing artifacts (e.g., subsampling in the inner layers).  This could prove interesting also in evaluating boundary-handling artifacts (the other source of non-translation-invariance in CNNs), which are expected to worsen in later layers.  I do see in the Discussion that the authors specify \"beyond-local-loops\" as a future direction, so my example woud need to be computed with very small displacements, perhaps to the point of rendering it unconvincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W0ZFQfSIjY", "reviewer": "Reviewer_rL8F", "summary": "This paper proposes a new representation similarity metric that complements existing metrics like CCA or CKA by taking into account how changes the input space map to changes in the representation space. In particular, the proposed representation holonomy metric is invariant to affine transformations and is zero for small loops in the input space. The estimator for this quantity is shown to be numerically stable and tractable to compute for various architectures of DNNs. Experiments show that holonomy is a valid measure of geometric effects (while measuring something different than other RS metrics), relatively robust to parametric choices, and can be used to predict robustness from small probes.", "questions": "* Does the number of points in the loop impact the accuracy/variance of your estimator? Why did you choose 12 points for your experiments?\n* Do the suggestions for uses of holonomy in section 6 correspond to any concrete experiments, or are these future avenues of study?\n* The caption for Figure 2 states that Layer 2 exhibits larger holonomy, but the data in the plot for Layer 1 appears to be larger in magnitude, which appears to contradict the caption. Is my understanding of the data being displayed in Figure 2 incorrect?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* Holonomy captures properties of a network that aren't measured by other representation similarity metrics. In particular, this metric captures the dynamics of representations as inputs change, which may make it more relevant to studies of properties like robustness than traditional RS metrics.\n* Holonomy can be used in conjunction with existing RS metrics to allow for a significantly deeper understanding of learned representations, as shown in section 5.3.\n* Estimates of error are included with most of the experimental results.\n* The evaluation section provides strong evidence that the theoretical claims made in Section 3 are exhibited in realistic settings.\n* Practical improvements can be made using holonomy as a diagnostic, such as improving methods for early stopping.\n* Limitations are thoroughly discussed in section 6.1.", "weaknesses": "* The theory is somewhat difficult to follow. A reader coming from the representation similarity literature might not have a background in the mathematics that are relevant to this paper. I think that defining terms more clearly and pointing readers to relevant background would be helpful. For example, gauge-invariance is never defined despite being in the title. Given the breadth of the ICLR audience, this type of knowledge cannot be assumed.\n* Similarly, the inclusion of proof sketches would be helpful for readers. What are the important mathematical techniques that are used to establish the properties of representation holonomy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VvaOXpOh5n", "reviewer": "Reviewer_n1HR", "summary": "This paper proposes a new geometric framework called representation holonomy, which quantifies how neural features change along loops in input space using gauge-invariant parallel transport. The authors define a practical estimator combining global whitening, shared-subspace Procrustes alignment, and loop composition, and they prove gauge and affine invariance as well as small-radius asymptotics. Empirical analyses on MNIST and CIFAR suggest that holonomy increases with layer depth and may correlate with robustness indicators such as adversarial and corruption accuracy.\n\nThe concept is theoretically intriguing and potentially valuable for studying the geometry of learned representations. However, both the experimental validation and the presentation of the theory can be much improved for clarity and completeness.", "questions": "1. How sensitive are the results to hyperparameter choices such as k?\n2. Can you provide quantitative comparisons with CKA or SVCCA to highlight the unique information captured by holonomy?\n3. Among the four terms in the error bound (Eq. 4), which dominates in practice?\n4. In your experiments, are there any correlations between task performance (e.g., error) and your measures?\n5. Could schematic illustrations or toy examples be added to visually illustrate the concept of holonomy in representation space?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Conceptual originality: Introducing holonomy—a gauge-invariant geometric notion from differential geometry—into representation analysis is creative and nontrivial.\n* Mathematical rigor: The formal statements (gauge invariance, affine invariance, small-radius behavior) are internally consistent using matrix perturbation tools. Although I have to admit that I didn’t get a chance to go over all the appendices.\n* Potential significance: In principle, this idea could offer new insights for feature-space curvature and robustness that go beyond CKA and related similarity measures.", "weaknesses": "* Incomplete and inconclusive experiments: The reported results (e.g., Figs. 2–3) show overlapping confidence intervals, and several key correlations (Table 2) are weak or include zero within the 95% CI. Without stronger or broader evidence, the empirical support for holonomy as a meaningful metric remains limited.\n* Missing comparisons to existing metrics: Although the paper discusses CKA and CCA extensively, it never provides side-by-side empirical comparisons. Such baselines are crucial to demonstrate what new insights holonomy provides.\n* Lack of sensitivity analysis: The estimator relies on multiple hyperparameters (e.g., k). The paper should quantify how results depend on these choices to establish robustness.\n* Unclear connection to task relevance: Theoretical results (e.g., Eq. 4) include several interacting error terms, but it remains unclear how holonomy relates in a concrete or predictive way to model performance or generalization.\n* Poor illustration of geometric intuition: Given the geometric nature of the work, schematic or pictorial figures showing loops, transports, or curvature effects are essential. The heavy formalism and lack of visual explanation make the paper difficult to follow.\n* Presentation density: The writing style is more reminiscent of a mathematical physics paper than a machine-learning paper, which may alienate part of the ICLR audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWINdHknUj", "reviewer": "Reviewer_ERwd", "summary": "This work presents a geometrically guided diagnostic tool for meaning the path-dependent geometry of learning features in neural networks. The authors present a novel problem formulation and approach that empirically demonstrate that lower holonomy results in greater model robustness. While the contributions are significant the presentation of the paper could be improved in places with improved rationale, and writing clarity. However, the resulting outputs are potentially valuable as a tool for model comparison and diagnostic work for practictioners and researchers.", "questions": "1. You mention: “projected by a fixed orthonormal Johnson–Lindenstrauss map to p⋆=1024 if needed”. What is the the determination if the projection is needed or not?\n\n2. Did you experiment with alternatively chosen loops, if so how much does performance vary? Given this is a noted limitation, is it significantly impactful to the work?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The identified problem of intermediate feature paths during input perturbations is an interesting and rationaled area of study. The authors address a potentially problematic area of neural network diagnostics, and present an interesting solution using geometry as a diagnostic tool, which is a valuable and interesting direction.\n- The mathematical approach is seemingly well constructed and correct, with the representation holonomy being a highly original and novel direction of model analysis, supported by proofs. \n- The results and analysis demonstrates the correlation of holonomy with adversarial robustness, and links model geometry and its model robustness, a new insight that is an interesting tool for early-training model selection and to investigate flatter representation manifolds.\n- Limitations are presented and discussed.", "weaknesses": "**Major:**\n- The problem setting while rationaled in the text could be better supported and demonstrated empirically. For example, what evidence is provided that supports the statement “two models can appear highly similar under CKA or CCA, and still behave differently under adversarial or corruption stress because their intermediate features rotate differently along input paths.”\n- Generally the presentation of the manuscript is poor in places (mainly in its clarity and readability), many of the preliminaries and experimental setup information is assumed by the authors. For example 5.2 is presented with little introduction, and \n\n**Minor:**\n- Acronyms such as SVCCA, CKA, etc need to be defined, while most readers will know such terminology, it is good practice to define such acronyms.\n- The term corruption or adversarial stress could be better defined in the early part of the work to introduce the reader to the concepts. However, this is a minor readability complaint and an opinion. \n- However, much of the presentation could be improved where some preliminary explanation and definitions would strengthen the readability. This could include moving some of the preliminaries from the appendix into the main manuscript as there is space to do so.\n- Code should be included for reproduction and validaiton.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)