Energy Landscape-Aware Vision Transformers: Layerwise Dynamics and Adaptive Task-Specific Training via Hopfield States
Authors: Runze Xia
Richard Jiang
Venue: NeurIPS 2025 poster ()
Keywords: Vision Transformers
Hopfield networks
Computational efficiency
Explainable AI
Primary area: deep_learning

Abstract: Recent advances in Vision Transformers (ViTs) have shown remarkable performance across vision tasks, yet their deep, uniform layer structure introduces significant computational overhead. In this work, we explore the emergent dynamics of ViT layers through the lens of energy-based memory systems, drawing a connection between self-attention and modern Hopfield networks. We introduce a novel metric—Layer Instability Index (LII)—derived from the operational softmax mode and its variability, to quantify the metastability of each Transformer layer over time. Our analysis reveals that certain layers exhibit consistent convergence to attractor-like states, suggesting functional specialisation and early stabilisation. Leveraging this insight, we propose an adaptive training framework that dynamically freezes or skips stable layers based on their energy landscape behavior. Our method reduces training costs while maintaining or improving accuracy. Extensive experiments on ViT-S/B/L on CUB-200-2011, CIFAR-10/100, Food-101, Stanford Dogs, and Beans demonstrate the generality and efficiency of our approach. This work provides new theoretical and practical perspectives for energy-aware optimisation of deep Transformer models.

Energy Landscape-Aware Vision Transformers: Layerwise Dynamics and Adaptive Task- Training via Hopfield States
Runze Xia
Richard Jiang
School of Computing Communications Lancaster University, UK r. xia2@lancaster..c ac. uk
School of Computing & Communications Lancaster University, UK
r. j.jang20lancasterr.. ac. uk
Abstract
Recent advances in Vision Transformers (ViT') have shown remarkable perfor- mance across vision tasks, yet their deep. uniform layer structure introduces signif- icant computational overhead. In this work, explore the emergent dynamics of VIT layers through the lens of energy-based memory systems, drawing connec- tion between self-attention and modern Hopfield networks. We introduce novel metric-Layer Instability Index (Lil)-derived from the operational softmax mode and its variability, quantify the metastability of each Transformer layer over time. Our analysis reveals that certain layers exhibit consistent convergence to attractor- like states, suggesting functional specialisation and early stabilisation. Leveraging this insight, we propose an adaptive training framework that dynamically freezes or skips stable layers based on their energy landscape behavior. Our method reduces training costs while maintaining or improving accuracy. Extensive experiments on ViT-S/B/L on CUB-200-2011. CIFAR-10/100, Food-101 101, Stanford Dogs, and Beans demonstrate the generality and efficiency of our approach. This work pro- vides new theoretical and practical perspectives for energy-aware optimisation deep Transformer models.
Introduction
Vision Transformers (ViTs) have emerged as transformative architecture in computer vision, achieving state-of-the-art results in classification, detection, and segmentation tasks. By employing self-attention mechanisms to capture long-range dependencies, ViTs move beyond the convolutional inductive biases characteristic of traditional models. However, this expressive capacity incurs substantial computational overhead, significantly restricting their applicability in resource-constraine scenarios.
Extensive research efforts have targeted this efficiency bottleneck through various optimisation strategies. Token-level sparsification methods, such as DynamieViT EVIT(23). A- A-VTT(4)), and [TAR(45), dynamically prune redundant patches during inference. Additionally, methods em- ploying end-to-end sparse training, including SViTE(6) for simultancous weight and token sparsity, and DIMAP[T]] for module-aware pruning in hierarchical models, further address computational efficiency. Other adaptive strategies modulate depth or resolution per image, exemplified by cas- cade token resampling(33
(37]. width- and depth-elastic adaptations in DynaBERT(12),, and early-exit mechanisms as demonstrated in LGVIT and PABEE [4]]. Despite their substantial reductions in computational costs, these approaches uniformly execute every retained layer, disregarding the differential internal convergence behaviours across layers.
"Correspondence author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Concurrently, complementary research aims to mitigate training overhead via Parameter-Effieee Fine-Tuning (PEFT). Lightweight modules, such as Adapters [3]] AdaptFormer (5), and Com- pacter[ 16]. introduce compact structures into models. Techniques like LoRA and its exten- sion Hydral.oRA employ low-rank parameter updates, while methods such as CoDA (20), and DAS utilize conditional routing strategies. Additionally, HST(24), ALaST(7], and SimFreeze (33) explore adaptive mechanisms to freeze model components selectively. Yet, despite reducing param- eter updates, these strategies typically introduce additional modules gating networks, lacking theoretically grounded criterion for determining which layers are optimal candidates for freezing or pruning.
Motivated by recent insights connecting Transformer self--ateention modern Hopfield networks, this study revisits VIT behaviour through an associative memory framework. Preliminary observations indicate that shallow layers rapidly converge towards stable attractor states, while deeper layers remain sensitive to input variations. This differential metastability across layers suggests novel efficiency dimension: selectively freezing or bypassing entire layers once their computational "energy" stabilises. To operationalise this insight, we propose Energy Landscape-Aware ViTs (ELA-ViT) incorporating novel metric termed the Layer Instability Index (LII). The LII quantitatively measures layer variability, guiding dynamic layer freezing decisions during fine-tuning phases.
Contributions
Theoretical grounding: We extend the interpretation of self-attention energy minimisation and formally connect VIT layer behavior to metastable dynamics in Hopfield networks.
Metric for metastability: We propose the Layer Instability Index (LII), a principled. data- driven measure that identifies stable versus adaptive layers based on attention distributions.
Energy-aware efficiency: We develop a dynamic training framework that freezes or skips layers based on their metastability, reducing computation without compromising perfor- mance.
Extensive empirical validation: Our method facilitates up to 12.2% reduction in fine-tuning time for ViT-B and yields 6.9% improvement in accuracy for ViT-L.
Our work provides new lens for understanding ViT layer behavior and paves the way for energy- aware optimisation in large-scale Transformer models.
2 Previous Work
VisioneTransformer Efficiency. Large-scale Vision Transformers (ViTs) underpin modern
vision vasion-language systems 28 26 and have even been distilled to mobile form factors (3 4 27
To mitigate their high inference cost, recent work makes computation input-adaptive along the token
or depth axis. Token Seassfiiaatin--yyaaaaa
[30]. EVIT [23), A-ViT [40] drops low-saliency
patches, while resolution-cascade designs refine token grids progressively [26). Depth adaptivity is
handled either by width--dlepth-elastie backbones such as DynaBERT (12 or by inserting internal
early-exit heade--ioneerred in CNNs by BranchyNet (34 and ported to ViT' via LGViT 33) and
PABEE [47]. Very recently, skippable sub-paths inside residual blocks yield depth-adaptive ViTs without extra heads [15]. All these schemes nevertheless execute every surviving transformer layer. They overlook that entire layers may converge to input-agnostic attractors and become redun- dant. Our work is complementary: we act on the layer axis. By analysing layer-wise metastability through a Hopfield-energy lens, we identify, skip, or freeze fully converged layers no token masks, cascade stages, or learned gates required.
Adaptive Fine-Tuning of Transtormers. Parameter-efficient fine-tuning (PEFT) freezes most backbone weights and updates only small subset. Bias-only BitFit [42]; bottleneck adapters (31] and their hypercomplex variant Compacter [16]; AdapterDrop (31): : and the vision-oriented Adapt-
Former (5) exemplify this line. Low-rank adaptation LoRA [14] has been enhanced by asymmetric multi-branch HydraLoRA (35) and orthogonal Householder adapters (8); IA freezes all but per-layer
scaling vectors (25:: LBP-WHT accelerates back-prop with low-rank Hadamard projections (23);
CoDA gates adapters conditionally at inference for further FL.OP savings (20). Budget-learning schemes such as ALaST
7 and SimFreeze
33 attach differentiable gates that allocate layer budgets

---

but introduce extra parameters and tuning overhead. In contrast, we propose parameter-free cri- terion-ton the Layer Instability Index (LII) computed directly from attention scores. Once low-LII layers are identified (after short warm-up), they are frozen, yielding simultaneous savings in parameters and compute without auxiliary modules, gates, or loss terms.
A key difference from gate-based budgets such as ALaST and SimFreeze is that our framework introduces no additional parameters. Their learnable budget predictors must be co-optimised with the backbone, incurring extra memory, computation, and regularisation. LII, derived from Hopfield-style energy dynamics, uses attention scores already present the forward pass, leaving model capacity and training complexity unchanged.
Hopfield Networks and Energy-Bare -Based Models. Modern Hopfield Networks (MHNs) extend the classical model with continuous states and stronger energies, achieving super-linear storage [18]. The seminal work of Ramsauer et al. (29) established that Transformer self-attention can be viewed as one update step in an MHN, casting attention as an associative memory retrieval process. This energy- based perspective has motivated a significant line of theoretical research aimed at understanding the fundamental dynamies of token evolution.
Foundational studies by Geshkovski et al. have employed tools from statistical physics and differential equations to provide rigorous mathematical characterization of these dynamics (10). Subsequent studies, including the work of Bruno et al. [2). have further extended this formalism by employ- ing mean-field PDEs and Wasserstein gradient flows to demonstrate the emergence of metastable clustering in idealized, continuous-time Transformer models in the limit of infinite depth and large token counts. The primary aim of these mathematical studies is formally elucidate the mechanisms through which tokens converge to stable clusters under idealized conditions.
Our work pursues distinct but complementary goal. Rather than extending this formal theory, we aim to operationalize its core insights for a practical application: the efficient fine-tuming of real-world, finite-depth Vision Transformers. We introduce the Layer Instability Index (LIf) not as new mathematical construct for proving convergence, but as a lightweight, computationally efficient proxy for the layer-wise metastability that these theoretical studies describe. Consequently, our primary contribution is the ELA-ViL framework- an adaptive algorithm that leverages these energy dynamics make concrete decisions about freezing layers during training. While the aforementioned theoretical works provide the fundamental "why" behind layer stabilization, our paper provides a practical "how" exploit this phenomenon for tangible gains in computational efficiency.
3 Methodology
Our approach, termed Energy Landscape-Aware Vision Transformers (ELA-ViT), introduces an adaptive fine-tuning framework guided by the energy dynamics of self-attention layers. The entire process, from an initial warm-up phase to an efficient consolidation phase, is illustrated in Figure I
Fin to Metrersiie
Faskkiig Fpervdss
Frarr Dasiiin Wolbe Tatepe)
Liee
Shoum
Liyerd
Thmmmn vartad By up compute the noce Ef sercoo sanales and bocrs with
F'e Tos cnl to bastt dan Bygng fow oe
madian, 24 0.
La medion, Ef medtion
AIl www.ioononnn Aay
LEB Cakaationn auch Lager Lllvaie ant www.kaatttooooooo for nomh Syyer
Figure 1: Overview of the ELA-VIT VIT adaptive fine-tuning pipeline and its ladrgy-landscape motivation. Left: self-attention as Hopfield energy minimisation; the operational mode reflects the width of the layer's low-energy basin (how concentrated or diffuse attention is). Middle: during a warm-up window, we estimate each layer's instability via the MAD of i (LII). Right: layers with LII < r are frozen in one-shot decision, and fine-tuning proceeds only on the remaining adaptive layers.
3

---

At its core, our method leverages novel metric, the Layer Instability Index (LII), to quantify the metastability of each layer. Based on this metric, stable layers are identified and frozen, focusing computational resources on layers that are still adapting to the downstream task. The subsequent subsections detail each component of this pipeline: we first establish the energy-based interpretation of self-attention ($3.3), then formally define the LII metric and its connection to operational modes ($3.2), and finally describe our adaptive maerrfrrezing mechanism ($3.4).
3.1 Energy-based View of Self-Attention
Following Ramsauer et al. [29]. reformulate self atfenttennion energy minimisation process in modern Hopfield network. Each attention head computes an energy function:
1 +
E -Is(
(1)
Let Ise(z) = log E; e*)) denote the log-sum-exp of vector z. Let X RRdx be the key matrix, Ç E Fd the query, and 3 the inverse temperature. The dynamics evolve toward local or global minima of the energy landscape, corresponding to metastable or stable attention patterns, respectively.
3.2 Layer Instability Index (LII)
We quantify how variable the attention patterns are across inputs each layer. For each layer we define the operational mode Ke as the median minimum number of tokens required to accumulate 90% of the attention mass:
0.9 })
V = median, min
m
(E,))
(2)
where af are sorted attention scores from head in layer f. Intuitively, hy reflects the width of the low-energy attention basin layer f: smaller values correspond to more concentrated attention (sharper basins), whereas larger values correspond to more diffuse attention (broader basins).
(f,m)
Warm-up Window, For the first i A iterations the sliding window lefi-truncated: compute the median over the available prefix {O...., If only. Operationally we maintain a circular buffer
of length A and simply update LII, = (buffrnn, where buffer, contains the most recent min(L+1, A) observations of kτ. After / 2 A the buffer is full and the window slides the usual FIFO manner.
To measure variability, we compute the Median Absolute Deviation (MAD) across samples and heads:
LII' median, | mediany (K-AA))
(3)
Here, A denotes the sliding window length, and t indexes over batches. Layers with low LII are considered stable and thus candidates for freezing skipping. For our experiments, set A 20. We empirically observed that the method robust to this choice, with results showing minimal sensitivity values in the range 10-40.
Empirical Evidence for Layer Instability. A growing body of work confirms that Transformer layers differ markedly in their input-wise variability. Zhai et al. observe that attention entropy collapses in early Vision-Triee Transformer blocks while remaining high in the middle, and propose entropy regularisers to stabilise very deep models [43). SmartFRZ automatically freezes layers whose attention variance drops below threshold and reports large training speed-ups with negligible accuracy loss [22) Unified ViT Compression finds that attention maps the topmost ViT blocks become nearly identical, enabling those blocks to be skipped with minimal impact on accuracy [4T]. Conversely, Li al. show that copying the pre-trained attention matrices alone transfers most downstream performance, suggesting these matrices are intrinsically stable across tasks (2)

---

Such results motivate using attention based statistics here the median operational mode ke and its MAD- quantify a layer's "instability".
Connection to Fisher Information. Several pruning frameworks exploit second-order curvature to rank layers or heads: Movement Pruning [32] SAViT's Hessian-block analysis [46). and Kwon et al.'s Fisher-guided post-training pruning (19 all show that low-Fisher (or low-curvature) weights and even whole layers can be removed with little accuracy loss. Dynamic sparsity allocation for Large
Language Models (LLMs) reaches similar conclusion the layer scale [2T]. Sec Sec formalises this intuition by proving that LII upper- bouedd the trace of the wise-wise Fisher matrix, i.c. low-LII layers are Fisher-flat and hence safe freeze.
3.3 LII as Proxy for Energy Gaps
We now tighten the theoretical footing of the Layer Instability Index (LII) by showing that it upper- bounds, up to a laycr-dependent Lipschitz constant, the expected energy gap between the metastable state reached self-attention and the global minimum of the corresponding Hopfield energy.
Hopfield Energy. For a fixed layer and head h. self-attention can be written as modern Hopfield update that minimises
ghh -
1 - log Ei exp(39Th.") C,
where is the query, (1 are the keys, and 3 is the inverse temperature.
Energy Gap. Let k be the operational mode introduced in Eq. (3). Denote by
Fk
a a: sorted softmax scores,
the residual attention mass after the top-k tokens (by definition rı S 0.1 because we choose p 0.9).
Define ex as the energy obtained when the summation in 44) is truncated to the top-k keys only (i.e., the metastable approximation). The per-head energy gap
A k
= gl.h gt.h
5
Information-Geomee View LII. Let F'EE [Vo[(( Vo,((r)] be the Fisher Information Matrix (FIM) of layer Under the exponential-tail assumption of Sec. 3.3 the residual attention mass T& satisfies Tk S e- Using the chain rule for self-attention w.r.t. its softmax scores one obtains
7² E.[rao]] s Ce LII
tr F <
f2
3
with C = --(1---J Details are deferred to App. B. Thus LII is a proxy for the local curvature of the loss in the Fisher-Raa Riemannian metric: layers with small LII are Fisher-flat and can be frozen without harming generalisation, whereas Fisher-steep (high-LII) layers benefit from continued adaptation.
Bounding the Gap via Residual Mass.
Because logE, exp(-) is monotonically increasing,
AgLh = log
+
rk <
rk
I-Th
81-rk
where the last inequality uses log(1 + r) S x. For T 0.1 the right-hand side at most 0.11/B.
Link to & and LII. Our analysis builds on the empirical finding that attention distributions in trained Transformers are typically sparse, concentrating most of their mass on few key tokens
while the remaining scores decay rapidly [36] I7]. Consistent with this behavior, we model the

---

sorted tail weights as decaying exponentially, i.e., at! slate with rate > 0. Then t%: S a/-->//(1 e>) Hence
E I L L 3(---
(6)
Taking logarithms and rearranging yields a Lipschitz relation between is and AE:
ct.h A
s Leye >h
161 k2l,
:= k
Lemma (LII Bounds the Expected Energy Gap). Let fe(k) E., [AE Under assumption 6 the map fe is L-Lipschitz with L S LEY. Consequently,
LII = MAD[R
[x[AE((x)] S L LIF o(1).
Proof Sketch. The full proof is detailed in Appendix A. The key steps are:
Decompose Around the Median: Use the Lipschitz continuity of the energy gap function to relate the gap at any input's operational mode to the gap at the median operational mode.
Connect to LII: Take the expectation over all inputs, which connects the energy gap to the Median Absolute Deviation (MAD), LII metric.
3. Bound the Residual: Show that the remaining term, corresponding to the energy gap at the median mode, negligible and can be absorbed into the o(1) term.
4. Combine: Substitute these bounds yield the final result.
Practical Implications. Lemma justifies using LII as computationally cheap surrogate for the (otherwise expensive) expected energy gap. Low-LII layers are guaranteed to have vanishing [ and are also shown to be Fisher-flat. They are therefore safe to freeze or skip, allowing computational effort to be focused on the adaptive, high-LII layers.
3.4 Adaptive Fine-Tuning via Layer-Aware Freezing
Our adaptive fine-tuning procedure consists of three distinct stages: warm-up, decision, and consoli- dation, each controlled by the Layer Instability Index (LII"), defined previously in Section 3.2 See pseudocode algorithm in the Appendix C.
Stage I: Warm-up (0 < l < T). All layers remain trainable while we accumulate a running estimate of LII over a sliding window of length W:
T = median'_.
median K
Empirically T=3W W with W 2:20 mini-batches suffices for stable estimate.
Stage II: Freeze decision (t T). For each layer l we compute the smoothed estimate LII,. If
LII
Tireze:
the layer is declared stable and its parameters are marked requires_gras False. No further gradients optimizer momentum are stored for that layer, yielding immediate O(ne) memory and FLOP saving.
Stage III: Consolidation (t> T). Training continues the remaining unfrozen layers. Optionally,
re-evaluate LII for the still trainable layers and apply stricter threshold Threze > Theere to capture late-stabilisin layers, but in practice a single decision at = already realises most of the gains.
every K iterations
Adaptive freezing yields benefits across computational efficiency, representation preservation, and focused learning. Specifically, (i) Computational Efficiency: When a stable layer is frozen, its

---

forward activations are omitted from the autograd computation graph, and its backward pass is skipped. This omission results in approximately 2n, fewer multiply-add operations per iteration each from forward and gradient computations. (ii) Representational Integrity: According to Lemma] [ layers with low Local Instability Index (I.II) occupy relatively flat regions in the Hopfield energy landscape. Consequently, additional gradient steps would only minimally perturb these parameters, at most by an amount proportional to (LII), thus preserving their stored associative representations. (iii) Focused Adaptation: Computational resources and optimizer efforts are directed predominantly towards layers with high LII, which are most responsive to task-specific fine-tuning. This targeted adaptation accelerates convergence and mitigates overfitting.
Empirical results show that our adaptive freezing strategy consistently locks 40%-70% of ViT parameters after the first epoch (Figure 2b). This yields a fine-tuning time reduction of up to 12.2% while maintaining or even enhancing validation accuracy (Section 5 with further validation provided in the appendix.
Most existing efficiency methods either learn gates that decide which layers to update (e.g., ALaST trains an auxiliary gating network over the backbone (7)) or introduce extra tunable modules such as adapters and low-rank updates (AdaptFormer (5]. HydraLoRA (35]). Both add parameters, hyperparameters, and optimisation overhead because gate weights or adapter matrices must be trained and maintained. By contrast, our sapfield-guided criterion is entirely parameter-free. The Layer Instability Index (LII) is computed directly from existing attention scores. Once low-LII layers are identified, we simply stop updating them requirngg no gating layers, adapters, or auxiliary losses. This minimalism reduces complexity and memory usage while ensuring efficiency gains accrue directly to the original ViT parameters, preserving interpretability and deployment compatibility.
Experiment Setup
We evaluate the proposed ELT five standard image classification benchmarks- C0O- 2011, Stanford Dogs, NABirds, Beans, and CIFAR-10-using three pretrained ViT backbones (ViT-S/16, ViT-B/16, and ViT-L/16 All images are resized 224 224 and standardized prior to fine-tuning. Training utilizes AdamW optimization combined with a cosine-annealing learning rate schedule and a batch size of 32. Hyperparameters for each backbone are selected to balance convergence speed with regularization: specifically, (17 Awa) = (3 x 10-5 0.05) for ViT-S, (1 x 10 10-500.1) for ViT-B, and (5 x 10 0.2) for VÝT-L.
Additionally, we conduct comparative experiments using DeiT and ALaST on the Food-101 and CIFAR-100 datasets. To deepen our interpretability analysis, we also include experiments us- ing ImageNet-1K K for the Layer Instability Index (LII) assessment. Performance evaluation spans three key metrics-top-1 accuracy, wall-clock fine-tuning time, and peglated-parmmeter ratio-with interpretability examined through layerwise LII heatmaps, elucidating the underlying dynamics responsible for each method's efficiency gains.
All experiments are conducted using Py Torch on single NVIDIA V100 GPUs with 50 GB allocated memory. Our implementation builds upon the HuggingFace ViT backbone and is released publicly.
Metrics. We report Top-1 accuracy, Fine-tuning time, and average number of layers executed per sample. All results are averaged over 3 seeds.
5 Experiment Results
We evaluate ELA-ViT across two dimensions: fine-tuning efficiency and accuracy preservation. To understand how ELA-ViT identifies which layers freeze, Figure 2a reveals two key trends in layer-wise instability patterns. First, the shape of the MAD profile is consistent: layers 3-4 display the lowest instability on all datasets, whereas the high-level layers 7-11 remain volatile, confirming that only subset of the stack needs task speeifp adaptation. Second, the scale of the MAD values
'ViT-L was excluded from our evaluation on the smaller datasets (e.g.. Beans, CIFAR-10) for two primary reasons. First, its high capacity (>300M parameters) mismatched with the limited sample size of these benchmarks, making it highly prone to overfitting without extensive, dataset- specific regularization. Second, its significant computational and memory requirements present practical challenges for experimentation and reproducibility on these particular tasks.
7

---

MAD (CIFAR- 1IN), Food-101) 1011
6
MAD (ImageN CaeaggeNe---SS
Tup- Acccuacy Change (%)
Accuracy Improvemett
Paniind
0-
Soseline
Aucurecy Degradation
Layer Index
Sudeer
1226/44 4ith
CIEAR-10)
Tood-101
Ie-
10
Fine-tuming Time Reduction (%)
(a) MAD of the operational mode & for ViT-B/16
(b) Accuracy-efficieng trade-off of ELA-VIT variants on CUB-200-2012 (ViT-L backbone). Each point responds to different freezing budget: the number in the label denotes the percentage of frozen after the warm-up stage. Moderate freezing (ELA- VIT- 45% and
fine-tuned three benchmarks. Left axis corresponds CIFAR-I00 Food-10 101 right axis ImageNet- Ik
All three curves exhibit the same qualitative U-shape: instability is moderate in the first two layers, reaches minimum around layers 3 4, and increases sharply in the higher-level semantic layers 7-11. Absolute magni- tudes, however, differ: the small-scale datasets show an order-of-magnitudd higher MAD than ImageNet, indi- cating that (i) pre-training on large, diverse data makes early layers more stable, and (ii) task-specific fine- tuning mainly perturbs the upper transformer block.
delivers the best overall balance, cutting fine-tuning time by 10.6% while improving accuracy by +6.7-46.9 pp. Aggressive freezing (ELA-VrT-755) ing (12.2%) but causes -4.4 drop in accuracy. illustrating the diminishing-rehie returns region beyond the Pareto knee.
yields the largest time sav-
Figure 2: (a) ViT-B/16 shows universal "U-shaped" layer-wise instability curve across datasets, (b) ELA-VIT accuracy-time Trade-Off.
seen during pre-training- sub-unit Fisher-flat values (< 6), while CIFAR-100 and Food-101 reach 40-50. This suggests using relative thresholds (Tireeee
is strongly dataset-dependent:
median,LII') rather than fixed absolute constant: layers below the dataset- specific median are safely frozen, whereas the adaptive upper block is left trainable. However, unlike ALaST, which relies learned layer-wise budget requiring continuous optimisation throughout training, our method computes per-layer MAD only during the initial steps (e.g., the first T steps) and freezes layers accordingly. This results in significantly lower overhead: ALaST incurs high cost in the early epochs due to full-model gradient computation combined with budget learning, whereas our method introduces only minor one-time cost 1% additional computation). Despite its simplicity, our method achieves similar freezing patterns with considerably improved efficiency.
5.1 Accuracy Preservation and Improvement
Table I summarises the validation accuracy of three ViT backbones (Small, Base, Large) across five benchmarks when trained (i) with standard full fine-tuning (Baseline) and (ii) with our .II-guided layer freezing (LII). The last column reports the absolute difference.
and Stanford Dogs, LIl-freezing consistently improves accuracy, with gains up to +6.9 pp on ViT-L for CUB. Fine-grained datasets require subtle, part-based cues, keeping the pre-trained shallow layers intact while focusing optimisation on the more task-relevant middle layers appears to enhance generalisation. The benefit also extends to VIT variant backbones: on CIFAR-100. ELA-DeiT shortens fine-tuning from 129.0 min 115.5 min (-10.5% while increasing top-1 accuracy from 86.91% to 87.49%.
For
Improvements scale with capacity: VIT-L gains four of five datasets and ViT-B three, whereas VIT-S sometimes over-regularizes (Beans, CIFAR-10). While freezing layers restricts capacity in small models, larger backbones retain sufficient expressivity even with significant parameter locking.
On BEANS and CIFAR- 10 the VIT-S variant loses about 1-2 pp. Both datasets are small and already near ceiling accuracy, so additional regularisation offers little benefit, and slight under-fitting can occur. Nevertheless, the ViT-B variant regains parity (+0.5 pp on CIFAR-10), showing that the effect is dptaset-specific rather than intrinsic to LIl-freezing.
8

---

Table Validation accuracy (%) for different models and freezing methods across datasets.
Dataset
Model
Baseline
ELA-Vit (Our)
Diff (%)
ViT-S
80.1
82.2
+2.1
ViT-B
77.9
82.2
+4.3
ViT-L
78.9
85.8
+6.9
VIT-S
78.3
82.5
+4.2
Stanford Dogs
ViT-B
85.3
85.9
+0.6
ViT-L
87.4
89.2
+1.8
VIT-S
68.4
69.3
+0,9
NAbirds
ViT-B
88.7
89.3
+0.6
ViT-S
99.3
97.7
-1.6
Beans
ViT-B
98.5
97.7
-0.8
ViT-S
94.5
92.8
-1.7
CIFAR1O
ViT-B
96.4
96.9
+0.5
DeiT-B
86.9
87.5
+0.6
5.2
Comparison with State-of-the-Art Layer-Budgeting
To benchmark against the current state-of-the-art in layer-budget learning, we implemented ALaST (7) across three datasets. We followed the exact same training protocol used for ELA-ViT: a ViT-B/16 backbone, ImageNet-2Ik initialisation, input size 224 batch size 32, AdamW (n7 1 x 10 Awd 0.1). a cosine schedule, and fine-tuning epochs. We kept all ALaST hyperparameters at their published values, including the redge--regullaristtion weight Auadee = 0.05.
Table 2: Performance comparison of ELA-VIT, ALaST, and full fine-tuning baseline the ViT-
B/16 backbone.
Dataset
Method
Top-1 Acc. (%) 1
Time (min) 1
Time A (%)
Full fine-tuning
88.6
297.9
0.0
Food-101
ALaST (75% frozen)
77.0
181.1
-39.2
ELA-VIT (50% frozen)
90.0
283.0
-10.5
Full fine-tuning
83.6
99.9
0.0
CIFAR-100
ALaST (75% frozen)
76.8
80.4
-19.5
ELA-VIT (50% frozen)
84.3
90.2
-9.7
Full fine-tuning
85.3
43.1
0.0
Stanford Dogs
ALaST (75% frozen)
75.7
33.6
-22.0
BLA-VIT (50% frozen)
85.9
37.0
-14.2
The results, presented in Table 2 demonstrate that EL.A provides vastly superior accuracy-to- compute trade-off. Across all three diverse datasets, ELA-VVT not only matches but improves upon the full fine-tuning baseline, achieving gains of +1.36 pp on Food-101, +0.65 pp on CIFAR-100 and a substantial +5.77 pp on Stanford Dogs. In sharp contrast, ALaST's aggressive 75% freezing budget results in performance degradation, sacrificing 6.8 pp. 8.8 pp. and 11.6 pp accuracy on the respective datasets. While ALaST achieves greater time savings, its method of learning budget incurs a catastrophic accuracy cost that makes impractical. ELA-ViT's ViT's LIl-guided approach, by contrast, delivers more modest but highly valuable speed-up of 9-14% while simultancously improving generalization, establishing
far more effective and practical fine-tuning strategy.
This performance difference stems from a key methodological distinction. ALaST learns the layer budget online, every mini-hatch it back-propagates through the entire network plus its auxiliary gating weights. During the first few epochs, the gates receive large gradients, effectively training all layers and nullifying any early speed-up. The layer budget stabilises only later, after most adaptation capacity has been exhausted. ELT by contrast, computes I.Il in one-pass warm-up (a negligible - -1% overhead) and freezes 50% of the parameters from epoch two onwards. This means

---

Table 3: ImageNet- 1k, ViT-B/16. t= higher better, + lower is better.
Method
Frozen
Train. (%)
Top-1 (%) f
Lat. (ms)
lat. (%)
Train time (h)
Full fine-tune tune
100.0
89.71
399.1
0.0
43.45
ELA-ViT-3520 35%
3-6
67.2
89.17
330.8
-17.1
39.59 (-9.0%)
EL-A-VTT-500
2-7
50.9
89.10
341.4
-16.7
38.81 (-10.7%)
EILA-VI-755
0-7,
26.3
87.92
322.7
-12.3
35.09 (-19.2%)
our computational savings materialize immediately, all while preserving the pre-trained, low-energy representations that prove crucial for generalization.
5.3 Scalability and Efficiency on ImageNet-1kk
To validate the scalability and practical utility of our approach, we evaluated ELA- on the large- scale ImageNet- Ik benchmark. As detailed in Table 3 our method demonstrates a highly favorable trade-off. With our balanced A-Vi--0500% configuration, we reduce wall-clock training time by 10.7% and inference latency by 16.7%. while maintaining top-1 accuracy within negligible 0.61 percentage points of the fully fine-tuned baseline. This result confirms that ELA-ViT effectively identifies and freezes stable layers scale, yielding significant computational savings without compromising performance on this challenging benchmark. Furthermore, for applications where efficiency is paramount, the more aggressive ELA-Vit ViT-75% setting nearly doubles the training speed-up to 19.2% for modest drop in accuracy, illustrating the controllable nature of our method's trade-off.
Discussion and Conclusion
We presented Energy Landscape-Award Vision Transformers (ELA-ViT), which harnesses metastable self-attention dynamics via a modern Hopfield view to enable adaptive training. We introduced the Layer Instability Index (LIl) quantify layer convergence. Across VTT-B and DeiT-B on ImageNet- 1k, CIFAR-100, and Tiny ImageNet, low-LII layers reliably settle into attractors. Freezing them during fine-tuning cuts parameter updates by up 52% and speeds training by up to 12.2%.
key insight that not transformer layers contribute equally to learning: shallow layers tend to capture general patterns and stabilize early, whereas deeper layers remain adaptive and task-specific. ELA-ViT provides a principled mechanism for identifying and exploiting this heterogeneity, in contrast to static pruning token-based dynamic methods.
Limitations include using attention mass concentration proxy for representational stability and the need to track attention over multiple steps, which can be challenging in low-resource settings. Future work will explore lightweight instability proxies, extend to multimodal transformers, and investigate theoretical guarantees convergence and generalisation.
Because LII-guided freezing operates the layer level and adds no new parameters, it orthogonal to most parameter-efffiieen fine-tuning schemes. A natural next step is to combine our criterion with conditional adapters such as CoDA 20) or with multi-branch low-rank modules such Hy 33). In such bybrids, low-L.I layers can be frozen, medium-LIL layers can retain lightweight adapters that activate on difficult inputs, and high-LII layers can remain fully trainable. We expect this dual strategy, parameter sharing plus dynamic layer skipping, to push the compute-accurat accuracy Pareto frontier, especially for large ViTs and multi-domain settings.
Overall, ELA-ViT bridges energy-based theory and practical efficiency, offering a principled view of adaptive computation deep transformers and motivating further work energy-aware optimisation and interpretable dynamics.
Beyond vision, applying ELA-ViT to other modalities, particularly LLMs, is promising. LII may identify layers that encode task-agnostic features (e.g., syntax and grammar), allowing compute to focus on deeper, more plastic, task-specific layers. Our findings also suggest hybrid architectures that replace early, low-LII VIT blocks with cheaper convolutional layers, yielding models that are efficient by design yet adaptive during fine-tuning. ELA-VIT offers mechanistic XAI insights and embodies early Agentic Al through energy-driven, self-regulating layer dynamics.
10

---

References
[1] Tolga Bolukbasi et al. "Adaptive neural networks for efficient inference". In: International Conference on Machine Learning. PMLR. 2017, pp. 527-536.
[2] Giuseppe Bruno, Federico Pasqualotto, and Andrea Agazzi. "Emergence of meta-stable clus- tering in mean-field transformer models". In: arXiv preprint arXiv:2410.2322 (2024). [3] Han Cai et al. "Efficientvit: Multi-scale linear attention for high-resolution dense prediction". In: arXiv preprint arXiv: 14756 (2022).
[4] Han Cai et al. "Once-for-all: Train one network and specialize it for efficient deployment". In: arXiv preprint arXiv: 0979/ (2019).
[5]
Shoufa Chen et al. "Adaptformer. Adapting vision transformers for scalable visual recognition".
In: Advances in Neural Information Processing Systems 35 (2022), Pp. 16664-16600
[6]
Tianlong Chen et al. "Chasing Sparsity in Vision Transformers: An End-to-End Exploration". In: Advances Neural Information Processing Systems. Ed. by M. Ranzato et al. Vol. 34. Curran Associates, Inc., 2021, pp. 19974-19988, URL:
neurips
cc/
pdf
[7]
Alessio Devoto "Adaptive Layer Selection for Efficient Vision Transformer Fine-Tuning". In: arXiv preprint arXiv:2408,.00 08670 (2024).
[8]
Wei Dong et al. "Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation" In: arXiv preprint arXiv:2410.2223 10.22522 (2024).
[9] Alexey Dosovitskiy. "An image worth 16x16 words: Transformers for image recognition at scale". In: arXiv preprint arXiv:2010. 1929 (2020).
[10] Borjan Geshkovski al. "A mathematical perspective on transformers". In: Bulletin of the American Mathematical Society 62.3 (2025), pp. 427-479.
[11] Yang He and Joey Tianyi Zhou. Duta-independent Module-aware Pruning for Hierarchical Vision Transformers. 2024.
arXiv: 2404. 13648 [cs.cV]
URL: hoLL:https://aan wtt.aiooi...oa..ao//////... org/ abs/
2404. 13648
[12] Lu Hou et al. "DynaBERT: Dynamic BERT with Adaptive Width and Depth". In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle et al. Vol. 33. Curran Associates, Inc., 2020, pp. 9782 9793. URL: https: httppe://proceeddnngg neurips. cc wt/ppp/p/pepp.... /apppaap....... lamperrffles files/ Paper Pappledddapppp.pppp... .pf
[13] Neil Houlsby al. "Parameter-effice
transfer learning for NLP". In: International comerece on machine learning. PMLR. 2019, pp. 2790-2799.
[14] Edward J Hu et al. "Lora: Low-rank adaptation of large language models." In: ICLR 1.2 (2022),
p. 3.
[15]
Woochul Kang and Hyungseop Lee. "Adaptive Depth Networks with Skippable Sub-Paths". In: Advances in Neural Infomation Processing Systems. Ed. by A. Globerson et al. Vol. 37. Curran Associates, Inc., 2024, pp. 33213-33231, URL:
neurips forrrffleese http:////..aaaaae//ocaaaaaaaaaa0 Conference. pdf
[16] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. "Compacter: Efficient low- rank hypercomplex adapter layers". In: Advances in Neural Information Processing Systems 34 (2021), pp. 1022-1035.
[17] Kyungmin Kim et al. "Rethinking the self-attention in vision transformers" In: Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. 2021, pp. 3071-3075,
[18] Dmitry Krotov and John J Hopfield. "Dense associative memory for pattern recognition". In: Advances in neural information processing systems 29 (2016).
[19] Woosuk Kwon al. "A Fast Post Fost-nrain Pruning Framework for Transformers". In: Ad- vances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 24101-24116. paper files / paper / 2022 file /ttT:///99//000111110000000066a0 Conf Conference.pee
URL: https / / neurips cc/ Paper
[20]
Tao Lei et al. "Conditional adapters: Parameter-efficient transfer Icarning with fast inference". In: Advances in Neural Information Processing Systems 36 (2023), pp. 8152-8172.

---

[21] Lujun Li et al. "Discovering Sparsity Allocation for Layer-wise Pruning of Large Lan- guage Models". In: Advances in Neural Information Processing Systems. Ed. by A. Globerson et al. Vol. 37. Curran Associates, Inc., 2024, pp- 141292-141317. URL. https / proceedings neurips ac paper files paper 2024 file / 0 Paper- Conf erence. ereence. pdff
[22]
Sheng Li et al. SmartFRZ: An Efficient Training Framework using Iitention-Basedd Layer
Freezing. 2024. arXiv: 2401 16720 [cs.LG)
URL: https: a/axiv. org/ //a//////// 2401
16720
[23] Youwei Liang al. "Not all patches are what you need: Expediting vision transformers via token reorganizations". In: arXiv preprint arXiv:2202.04 07800 (2022).
[24]
Weifeng Lin et al. Hierarchical Side-Tuning for Vision Transformers. 2024. arXiv: 2310
05393 [cs.CV CV] URL: hattps://arziii orgz hts//bb//23111 abs/2330.05393
[25] Haokun Liu et al. "Few-shot parameter-efficient fine-tuning is better and cheaper than in- context learning". In: Advances in Neural Information Processing Systems 35 (2022), pp. 1950- 1965.
[26] Xinyu Liu et al. "Efficientvit: Memory efficient vision transformer with cascaded group attention". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023, pp. 14420-14430.
[27] Sachin Mehta and Mohammad Rastegari. "Mobilevit: light-weight, purpos,, and fobile-friendly vision transformer". In: arXiv preprint arXiv:21 /0.02 /78 (2021).
[28] Alec Radford et al. "Learning transferable visual models from natural language supervision". In: International conference on machine learning. PMLR. 2021, pp.
[29] Hubert Ramsauer et al. "Hopfield networks is all you need". In: arXiv preprint arXiv:2008. 02207 (2020). [30] Yongming Rao et al. "Dynamievit: Efficient vision transformers with dynamic token spar- sification". In: Advances in neural information processing systems 34 (2021), pp. 13937- 13949.
[31] Andreas Rücklé et al. "Adapterdrop: On the efficiency of adapters transformers". In: arXiv preprint arXiv:2010.1 1 1918 (2020).
[32] Victor Sanh, Thomas Wolf, and Alexander Rush. "Movement pruning: Adaptive sparsity by fine-tuning". In: Advances in neural information processing systems 33 (2020), pp. 20378- 20389.
[33] Tianyi Shen, Chonghan Lee, and Vijaykrishnan Narayanan. "SimFreeze: Adaptively Freeze Vision Transformer Encoders with Token Similarity". In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 8266-8270.
[34] Surat Teerapittayanon, Bradley McDanel, and Hsiang- Tsung Kung. "Branchynet: Fast infer- ence via early exiting from deep neural networks". In: 2016 23rd international conference pattern recognition (ICPR). IEEE. 2016, pp. 2464-2469.
[35] Chunlin Tian et al. "Hydralora: An asymmetric lora architecture for efficient fine-tuning". In: Advances in Neural Information Processing Systems 37 O. pp. 9565-9584.
[36] Elena Voita et al. "Analyzing multi-head self-attention: Specialized heads do the heavy lifting, (2019).
the rest be pruned". In: arXiv preprint
[37] Yulin Wang et al. "Not All Images are Worth 16x16 Words: Dynamic Transformers
for Efficient Image Recognition". In: Advances in Neural Information Processing Sys- lems. Ed. by M. Ranzato et al. Vol. 34. Curran Associates, Inc., 2021, pp. 11960-11973. paper URL: https: : // proceedings neurips cc c files / paper / 2021 / /ile/
[38] Qiong Wu et al. "Parameter and Computation Efficient Transfer Learning for Vision- Language Pre-trained Models". In: Advances in Neural Information Processing Systems. Ed. by A. Oh et al. Vol. 36. Curran Associates, Inc., 2023, pp. 41054-41050. URL: https: / proceedings neurips cc paper files / paper / 2023 / file / 805544fac2211111116551111111 Paaeeraa---aeaaaaaaa-------- Conf erence. pdf
[39] Guanyu Xu et al. "Lgvit: Dynamic early exiting for accelerating vision transformer". In: Proceedings of the 31st ACM International Conference on Multimedia. 2023, pp. 9103-91 14. [40] Hongxu Yin et al. "A-vit: Adaptive tokens for efficient vision transformer". In: Proceedings of the IEFF/CVF conference on computer vision and pattern recognition. 2022, pp. 10809-10818.
12

---

[41] Shixing Yu et al. Unified Visual Transformer Compression. 2022. arXiv: 2203 08243
[cs LG]] URL: httpp:://aa....r////aaat... 2221/22333.00833
[42] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. "Bitfit: Simple parameter-efficient fine- tuning for transformer-based masked language-modes models". In: arXiv preprint arXiv:2106.10199 (2021).
[43] Shuangfei Zhai et al. "Stabilizing Transformer Training by Preventing Attention Entropy Collapse". In: Proceedings of the 40th International Conference on Machine Learning. Ed. by Andreas Krause et al. Vol. 202. Proceedings of Machine Learning Research. PMLR, 23-29 Jul 2023, pp. 40770 40803. URL: http::/t////rrrceennggs mlr press/v2022 2wt/s///2222222222aa.a zhai23a. html
[44] Xiaohua Zhai et al. "Scaling vision transformers". In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 2022, pp. 12104-12113.
[45] Yuyao Zhang, Lan Wei, and Nikolaos Freris. "Synergistic Patch Pruning for Vision Trans- former: Unifying Intra- & Inter-Layer Patch Importance". In: The Twelfth International Con- ference on Learning Reprexentations. 2024. URL: https://openrevier wwtp:/ee//eeeeeeee fest/forum?dd
C0051g4104
[46] Chuanyang Zheng et al. "SAVIT: Structure-Aware Vision Transformer Pruning via Col- laborative Optimization". In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 9010-9023. URL: https // proceedings neurips ac paper files paperr 2022 file /
3b1 00166688666223366466676122222222
Conf erence. .df
[47]
Wangchunshu Zhou al. "BERT Loses Patience: Fast and Robust Inference with Early Exit".
In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle et al. Vol. 33, Curran Associates, Inc., 2020, pp. 18330-183411 URL: https: ptt.httptss//preeeeennnn neurips
cc httpa////.//////2/22///////11113
pdf
13

---

Appendix
A LII upper-bounds expected energy gap
We prove the following result to formalise the intuition that stable layers sit close to their energy basins. Lemma] states that for every layer l
E AE (z) - Ly LII o((),
where AE (x) is the Hopfield energy global minimum, and L, is finite layer-dependent constant. Thus, the median absolute deviation of the operational mode-the Layer Instability Inder-provee provides a linear upper bound on the expected energy sub-optimality. with the residual term vanishing exponentially in sequence length. Layers with small LII' are therefore already near their optimal attractor and may be safely frozen or skipped without degrading convergence or generalisation.
energy gap between the attention state reached on input x and the
Proof. Fix layer i and write K (x) (x) for the operational mode of input æ. Let
median, K K(x)
(8)
and recall that by Eq. 6 the expected per-head energy gap satisfies
fe(k) = Bh, A (x)) S Lee 7k
(9)
and is I.-Lipschitz with constant L
Step 1: Decompose around the median. Using Lipschitz continuity, we have, for each input x,
(e((())) - fe(h) s LK(()) ks).
(10)
Taking expectation x and rearranging,
Ex [f(((())) S fe(k) L K(x) k).
(11)
Step Median absolute deviation. By definition.
LII [MA[[]] median, [K(x) 1 < E, (x))) kl.
(12)
so the second term
is bounded by LLII'
Step 3: Bounding the residual term. Applying Eq. 66 at k i gives
fe( S Lie-
(13)
Because i is a median of token counts, it grows at least logarithmically with sequence length; hence Lye yh o(1) and can be absorbed into the o(1) term of the lemma.
Step 4: Combine. Substituting these bounds into Eq. yields
Ex [AE (x) L LII + o(1).
(14)
Thus, the desired inequality follows immediately.
B Information-geometic bound: LII upper-bounds the fisher trace
We establish theoretical bound showing that the Layer Instability Index (LII) upper-bounds the trace of the Fisher Information Matrix (FIM) for transformer layers. The FIM trace characterises sensitivity of the loss to parameter updates, providing insights into learning dynamics. Through careful analysis of gradients via softmax logits and exponential tail bounds of attention probabilities, we derive direct relationship between the Fisher trace and the dispersion of operational modes as measured by LII.
14

---

Preliminaries. Let p; softmax( q k;) with inverse temperature 5. Denote the per-example loss by (p) and define the layer-wise FIM
Fd
Bay Vo 00 E ( W Wk.
(15)
Our goal is to bound tr F
Step 1: Gradient of the loss w.r.f. logits.
For the logits 21 = .4' kis
ac
Σ
0L Opj
B(p, - -(),,
(16)
dz
Op, dzi
where y; the "effective" target (one-hot for CE).
Step 2: Fisher trace through the chain rule.
Let J2 := az/00,. Then
tr F Er V
Ex
(17)
min
where Amin is the smallest singular value of Jg (assumed layer-dependent but strictly positive for typical initialisation).
Using T6..
V+Cli== = E (pi 0i)2 S 8 Σ pi
(18)
Step 3: Relate P. to residual mass Let k = (x) be the operational mode and Ts :=
Eise Pi S 0.1 (by definition of p = 0.9). Then
p
Σ p
S
Σ iSk
Ps + maxp: S 0.9 +r
(19)
i>k
iSk
1>
Assuming an exponential tail Pi>a S P&e r(i-k), rk S pp/ - Ce
Step 4: From e-th to LII
Taking expectation over inputs and using Jensen,
E (e [b(2)]
median(A) (k) (1+ (1+7LI'')
(20)
hence
Σdt pt s 0.9 O median(k) (k) (1+
(21)
Step 5: Final bound.
Substituting into 17
(0(0 (1+ CI + Co.e,
tr F
(22)
min
where Ct and Co.e are layer-dependent constants. For practical purposes Co is negligible once LII exceeds 10-2, yielding
tr F S C
(23)
as claimed.
Connection to the 1-Wasserstein distance. Sort the attention vector of layer & and head h at step t, at vi..((). and define its empirical cumulative distribution function (CDF) F((m) Em (t). Because tokens are indexed by their rank, the carth-mover (1- (1-Wasscrstein) distance between two attention snapshots is simply
N
Wi(a()).((1)) = Σ [((m) [m(mm)
(24)
15

---

one-shot pruning of stable layers Let 0.9 and let kt be the minimal m such that (i(m) 2 (operational mode). Then any deviation
1k; ker shifts at least a residual mass (kg pl S positions, so
0.1 across [k; kp token
Wi(a(t),a(t))) - r 1kz - kel S 0.1 |k+ - kerl-
(25)
Taking the median over t' in the sliding window and then the median over t gives
( < 0.1
(A.1)
where Wimood (f) is the median Wasserstein distance between successive attention snapshots of layer & Equation (A.1) shows that **LII controls the earth-move distance between attention distributions low LII implies the layer's attention landscape hardly moves in Wasserstein space and is therefore safe to freeze. Via the Gantorovich-Rubinsie
dual, the same bound controls the difference of all 1-Lipschitz observables of the attention measure, linking the energy and Fisher-flatness views in a common optimal- transport metric.
C Energy landscape-aware fine-tuning
Algorithm details the complete training routine used all experiments. After short warm-up that estimates the Layer Instability Index (LII) for every block, layers whose instability falls below user-defined threshold Theeze are frozen "requires_grad-Pee -grad=Faaae) Fine-tuning then proceeds on the remaining adaptive layers, incurring further LII overhead.
Algorithm 1: Energy Landscape- Aware ViT Fine-Tuuing
Input: pre-trained weights (0); dataset D; freeze threshold Thess: warm-up steps T: window
Warm-up phase:
// estimate layer instability // collect it statistics
for = 0 to 7 do
sample mini-batch (z, y) D;
forward and backward pass; update ((++1) with AdamW;
update the circular buffer size W and compute L.TI for all layers E;
Freeze decision: foreach layer & do
if LTT Tower then freeze(()
// disable gradient updates
Consolidation phase:
// train only adaptive layers // until convergence
for = T to max_ steps do L mini-batch (I, y) D:
forward + backward pass on unfrozen layers only;
The algorithm runs in three stages: (i) Warm-up gathers a robust estimate of each layer's variability via the median absolute deviation of its operational mode k. (ii) Freeze decision is executed once, turning off gradient flow for layers whose LII indicates convergence to low-energy basin. (iii) Consolidation fine-tunes the remaining layers, yielding substantial savings in memory and computation with no extra learnable parameters.
D Online update of the LII circular buffer
-
During warm-up we compute LII, for every layer on the fly. Algorithm shows eight-line Python reference implementation; relies only on layer-indexed deque of fixed capacity W (the sliding window size, default W 20).
At each mini- we compute the layer's operational mode K (Sec. 3.2 and call update_ 1ii The deque acts circular buffer: the newest value is appended, the oldest is popped when the buffer overflows, and both operations are ((1). We then take the median of the window, followed by the median absolute deviation deviation-exactlyy exactly Eq. (2) but restricted to the latest W steps. The result is the
online estimate LII, used in Alg.
16

---

Algorithm 2: Update of layer instability index (LII) buffer
Input: layer ID layer_ide new value k; buffer buf; window size Output: current LII;
buf [layer_idd _id] append (R)
// 1. push newest value
len (layer_____))
then
buf [layer_id] http_iiddppppleeeee popleft()
1/ 2. drop oldest (FIFO)
med median medianbbu (layer_id])
// running median
abs_dev [abs med) for buf (Layer_iidd]
return median (aba_dev);
// 4. current LII;
Note:
One dictionary, one deque per layer (maxlen = W). Complexity: (11) per call, (()) memory per layer.
Cold-start. For & W the deque contains fewer than W elements; the function still returns valid LII based on the available prefix, ensuring that no additional initialisation logic is required.
Efficiency. The routine consumes negligible resources: O(LW) memory for L. layers and O(1)
extra time per iteration, contributing less than % overhead in all experiments (see App. B).
E Code Availability
The source code supporting this study is available httpe://gghbbb.cc/aaaaaaaLLLLLL
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Can be found in the section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Can be found in the section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
Can be found in the section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Can be found in the section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: material.
Additional details and code repository URL are included in the supplemental
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//iips.c//ppi wttpccpppppppppeeeeeeeeeeeeeeeee gtte///guis////ee itt.eeSSubmissioPoolieci.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Can be found in the section 4
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Given that our experiments focus on training and fine-tuning Vision Trans- former models to demonstrate improvements in accuracy and efficiency rather than testing hypotheses, statistical significance through error bars or formal significance testing is not di- rectly applicable. Instead, clearly report model performance metrics, including accuracy and computational improvements, consistently across multiple datasets and settings. These metrics reliably indicate performance differences without requiring traditional statistical significance testing.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Can be found in section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: I confirm the research conform with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No potential societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The research is not high risk research.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used explicitly mentioned and included in the reference.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Not introduced new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Not require human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not require human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLM used for original, or non- non-standard component of the core methods.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre http:///rrrrrrecooofferrre////// cc//oonferences/2225///.. for what should or should be described.
24


Reviews:
[{"review_id": "fdekYOTfgj", "reviewer": "Reviewer_Bcjj", "summary": "The authors present a method that reduces the training time while enhancing accuracy of Vision Transformers by introducing a metric quantifying layer stability over time. By dynamically freezing or skipping stable layers, they are able to avoid redundant or extraneous training, which surprisingly benefits performance as well.", "questions": "Seeing how these methods fare on e.g. Imagenet (I am guessing this would yield even more impressive results) would be extremely interesting. Going further, one might expect some interesting results from experiments on other kinds of attention based models and/or language models. In my opinion adding one or more of these would push this paper above the acceptance threshold.  \n\nI would also like to comment that Figure 1 is very low resolution and looks odd in the paper. This should be fixed.", "limitations": "In my opinion, no. They should perhaps add some comment on the ways in which this could be extended.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "This paper provides an extremely novel and interesting application of the close theoretical relationship  between Hopfield Networks with nontrivial activation functions, or “Dense Associative Memories” and attention based models such as Transformers. Specifically, self-attention can be viewed as one step of an energy minimization of a modern Hopfield network, which in practice is usually sufficient. This work points toward a more naturalistic and dynamic paradigm for training state of the art models, and is thus highly relevant to the broader research community. \n\nThe main novelty of this paper is the “Layer Instability Index” and adaptive training scheme, which yield some very nice improvements in accuracy and training time. Still this paper could be considerably strengthened by exploring performance advantages at scale. While the LII is very smart, there are probably much broader implications for other classes of attention-based model. In my opinion the paper could go much further in exploring these.", "was_revised": true, "final_justification": "I am raising my assessment of quality as it seems some of the results requested were included in some version of the original submission, I'm curious however, why I have not been able to see the appendix referred to in the authors' response.", "rebuttal": "# Response to Comments\n\nWe would like to extend our sincere gratitude to the reviewer for your positive and encouraging feedback on our manuscript. We are delighted that you found our application of Hopfield network theory to be **\"extremely novel and interesting\"** and that you see the potential of our work to inspire a **\"more naturalistic and dynamic paradigm for training.\"** We appreciate the constructive suggestions you provided, particularly regarding the evaluation at scale, and are confident that we can address these points to elevate the paper above the acceptance threshold.\n\n---\n\n## 1. Performance at Scale: ImageNet Results\n\nThe reviewer's suggestion that we explore the performance advantages of our method at a larger scale, specifically on a benchmark like ImageNet, is well-taken. We wholeheartedly agree that demonstrating the effectiveness of our method on large-scale benchmarks is crucial for validating its practical utility. We are pleased to report that we have already conducted these experiments and presented the results in **Appendix E: ImageNet-1k: large-scale validation** (page 16) of our original submission.\n\nOur results, summarized in **Table 3**, confirm that the benefits of our approach hold at this scale. Specifically, our experiments demonstrate that:\n\n1. **ELA-ViT-50%** (freezing half the layers) reduces wall-clock training time by **10.7%** and inference latency by **16.7%**, while maintaining accuracy within **0.61 pp** of the fully fine-tuned baseline.\n\n2. **ELA-ViT-75%** (freezing nine of the twelve layers) achieves a **19.2%** reduction in training time for only a **1.8 pp** drop in accuracy.\n\nThese findings strongly support your intuition that our method would yield impressive results on ImageNet. To ensure that these results are not overlooked, we will make them more prominent in the main body of the revised manuscript. We will relocate the relevant discussion and tables to a more visible section, thereby directly addressing your key point and highlighting the effectiveness of our method at scale. By doing so, we aim to provide a clearer understanding of the scalability of our approach and its potential for practical applications.\n\n---\n\n## 2. Broader Applications and Future Work\n\nYour observation that our method has broader implications and could be extended to other models, such as language models, is insightful and well-founded. We appreciate this comment and agree that the principles behind LII and energy-based layer freezing are not limited to Vision Transformers. The extension to Natural Language Processing is a particularly exciting direction for future research, as it could potentially lead to significant advances in the field.\n\nTo address this suggestion, we will expand our **\"Discussion and Conclusion\"** section in the revised paper. We will add a new paragraph explicitly discussing the potential to adapt ELA-ViT for large language models (LLMs). We hypothesize that since LLMs also exhibit layer-wise functional specialization, our method could similarly identify and freeze stable, foundational layers (e.g., those handling syntax) while focusing training on more task-specific, semantic layers.\n\nFurthermore, we will outline potential future directions for our research, including:\n\n1. **Application of LII to other model components**, such as MLP layers or residual blocks\n2. **Continuous layer freezing** with per-layer annealing schedules\n3. **Integration with budget-aware or curriculum-based training** (e.g., learnable τ thresholds)\n4. **Extension to unsupervised and contrastive training objectives**, where attention convergence may be more volatile\n\n---\n\n## 3. Minor Presentation Issue\n\nThank you for pointing out the issue with Figure 1. We acknowledge that the current version is affected by a rendering artifact in the compiled PDF, which impacts resolution and clarity. If the paper is accepted, we plan to:\n\n1. **Regenerate all plots** at 300+ dpi resolution\n2. **Ensure vector-based exports** (PDF or SVG) are used to retain crispness\n3. **Separate subfigures clearly** with updated labels and font sizes for readability\n\nBy doing so, we aim to improve the overall presentation and clarity of our figures, making it easier for readers to understand and appreciate our results.\n\n---\n\n## Conclusion\n\nWe are grateful to you for your thoughtful feedback and clear path toward improving our manuscript. By highlighting our existing large-scale results and expanding our discussion on future applications, we believe the revised manuscript will fully address your points and make a stronger case for acceptance. We are encouraged by your positive assessment of our core ideas and hope that you will agree that these additions make for a valuable contribution to the field. We look forward to the opportunity to revise and resubmit our manuscript, and we are confident that the changes we will make will significantly strengthen the paper."}, {"review_id": "etUw2YIYEP", "reviewer": "Reviewer_f65B", "summary": "The authors relate the architecture of vision transformers, particularly the self-attention module, to Hopfield networks and, more specifically, to an energy-based model.", "questions": "1. How is the LII metric linked to the energy? Perhaps there is a functional inequality that could clarify this connection.\n\n2 The authors show that LII bounds the energy gap. Could this be formulated more transparently so that a clear inequality is presented? The discussion on pages 3–5 is mathematically fuzzy, and a bit more formalism would be appreciated.\n\n3. Comment: The term “lse operator” is not a standard mathematical description. Although I understand what the authors mean, it is not comparable to something like the sine function. The authors should define this appropriately.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "My main concern with this paper is that it lacks rigorous mathematical results (in fact, there are none), despite the same results having already been established using deep mathematical methods by Philippe Rigollet’s team. Moreover, the metric the authors consider is among those studied in the work “A Mathematical Perspective on Transformers” (see section 6 therein).\n\nTo proceed further, the authors need to clearly discuss how their setup differs from these more mathematical works, as well as from subsequent studies on metastability (see works by Rigollet, and also by Bruno, Agazzi, Pasqualotto). It is not that I require rigorous results per se, but the formalism should be more transparent, and the relevant references should be included.\n\nThat said, I acknowledge the experimental section, which is interesting, and the authors could expand on it further. Doing so would make this a valuable contribution.\n\nI would be open to raising my score if the above comments and the following ones are addressed.", "was_revised": true, "final_justification": "The authors answered all of my questions appropriately, and I better understood the take-away and contributions of the paper.", "rebuttal": "# Response to Comments\n\nWe extend our sincere gratitude to the reviewer for their insightful and constructive feedback. We greatly appreciate the time and effort invested in providing detailed comments, which have helped us reflect critically on the clarity, rigor, and positioning of our manuscript. Below, we respond to each point raised, clarify the theoretical motivations, and outline improvements we will incorporate in the final version.\n\n---\n\n## 1. On Mathematical Rigor and Relation to Prior Work *(Rigollet et al., Bruno, Agazzi, Pasqualotto)*\n\nWe are grateful for your reference to the relevant mathematical literature, particularly the foundational contributions of Rigollet and colleagues. While our work does not aim to replicate the full generality of their rigorous formalism, it seeks to propose a pragmatic and computationally efficient proxy, the **Layer Instability Index (LII)**, which aligns with intuitions from energy landscapes and can be directly derived from existing attention weights without introducing additional computational overhead.\n\nTo clarify, our primary contribution lies in operationalizing theoretical insights for practical and scalable fine-tuning applications. The work of Rigollet et al. offers a profound mathematical framework for understanding the asymptotic clustering behavior of idealized Transformer models, establishing convergence to a single cluster under specific conditions. Their contributions provide a critical foundation for analyzing long-term dynamics in transformer models.\n\nIn contrast, our research pursues a related but distinct objective: leveraging energy-based stability concepts for empirical and practical purposes. The proposed LII is not intended as a novel theoretical construct for proving convergence but as a lightweight, computationally efficient metric to assess layer-wise metastability during the fine-tuning of real-world Vision Transformers. Thus, the core contribution of our paper is the introduction of the **ELA-ViT framework** and the **LII metric**, which dynamically adapts the training process based on stability insights, rather than focusing on long-term clustering behavior.\n\nWe acknowledge the need for greater clarity in distinguishing our work from prior theoretical contributions. To address this, we will revise the final version to include:\n\n1. **Explicit citation and detailed comparison** with Rigollet et al. (e.g., referencing specific sections such as \"A Mathematical Perspective on Transformers,\" Section 6), highlighting the differences in scope and objectives.\n\n2. **Discussion on the limitations** of our approximations compared to their formal results, particularly regarding metastability analysis using rigorous energy gap estimations.\n\nWe believe this will more accurately position our work within the broader theoretical landscape and clarify its intended contribution.\n\n---\n\n## 2. On the Link Between LII and the Energy Gap\n\nThank you for highlighting the need for greater mathematical clarity in establishing the connection between the Layer Instability Index (LII) and the Hopfield energy gap, which is indeed a central motivation of our study. This relationship is articulated in our current manuscript (Section 3.3 and Appendix A), with **Lemma 1** stating:\n\n$$\\mathbb{E}_x[\\Delta \\varepsilon^{ℓ}(x)] \\leq L^ℓ \\cdot \\text{LII}^ℓ + o(1)$$\n\nThis result is derived under assumptions of exponential decay in attention scores and links LII to residual energy gaps through a Lipschitz-bound analysis. This inequality provides the theoretical justification for our method:\n\n1. **The left side**, $\\mathbb{E}_x[\\Delta \\varepsilon^{ℓ}(x)]$, is the expected energy gap for a given layer $ℓ$. This measures how far, on average, the layer's current attention state is from its most stable possible state (the global energy minimum). A small energy gap means the layer has converged and is stable.\n\n2. **The right side** is anchored by our proposed metric, $\\text{LII}^ℓ$ (the Layer Instability Index), which empirically measures the variability of attention patterns across different data samples.\n\n3. **The inequality**, therefore, formally states that a small, empirically measured instability (a low LII) guarantees a small energy gap. This is precisely why we can use LII as a reliable proxy: if a layer's attention patterns are not changing much (low LII), we have a theoretical guarantee that it is already in a low-energy, stable state and can be safely frozen.\n\nWe recognize that the presentation of this result in the current version lacks the desired rigor and visibility, as it is situated in the appendix and described in a manner that may appear insufficiently formal. To address the reviewer's concerns, we will revise the final version in the following ways:\n\n1. **Relocate the formal statement** of Lemma 1, along with a detailed proof sketch, to the main body of the paper (Section 3.3), positioning it as a central theoretical result.\n\n2. **Present Lemma 1** as a clearly formatted theorem or boxed lemma, accompanied by a step-by-step derivation. This will include explicit definitions of all terms, such as the Lipschitz constant $L$, and a clear articulation of the exponential decay assumption on attention score tails that underpins the relationship.\n\n3. **Highlight the relationship** as a standalone inequality, as requested by the reviewer, to ensure transparency and rigor in connecting the empirical metric (LII) to the theoretical energy gap.\n\nThe revisions will make the theoretical underpinnings of LII more explicit and should clarify its role as an interpretable and principled estimator of local energy instability.\n\n---\n\n## 3. On the Term \"lse operator\"\n\nWe appreciate your observation regarding the terminology used to describe the log-sum-exp operation. While the current manuscript expands the abbreviation \"lse\" to \"log-sum-exp operator,\" we acknowledge that we neglected to provide its precise mathematical definition, which is essential for rigorous and unambiguous communication. This was an oversight on our part.\n\nIn the final version, we will introduce the explicit definition and corresponding formula at its initial occurrence as follows:\n\nFor a vector $\\mathbf{x} = (x_1, x_2, \\ldots, x_n) \\in \\mathbb{R}^n$, the **log-sum-exp function** (often abbreviated as LSE) is defined as:\n\n$$\\text{log-sum-exp}(z) = \\log \\sum_i \\exp(z_i)$$\n\nWithin the context of the energy function (as presented in Eq. (1) or (4)),\n\n$$E = -\\mathrm{LSE}(\\beta \\cdot \\mathbf{X}^\\top \\xi) + \\frac{1}{2} \\|\\xi\\|^2 + C$$\n\nthe LSE term encapsulates the soft-maximum alignment between the query and all keys, corresponding to the negative energy in the interpretation of Hopfield dynamics. By incorporating this definition, we aim to align our usage of terminology with established mathematical conventions, thereby enhancing the comprehensibility of our methodological framework.\n\n---\n\n## 4. On Expansion of the Experimental Section\n\nWe appreciate the reviewer's favourable assessment of our experimental results and concur that an expanded discussion would enhance the manuscript. While adhering to the submission guidelines, we outline the following extensions that we are prepared to incorporate into the camera-ready version, contingent upon acceptance:\n\n 1. **Ablation Analyses**\nTo further substantiate the robustness and sensitivity of our approach, we plan to conduct ablation studies on key hyperparameters, including the LII sliding window size and attention mass threshold, thereby providing a more nuanced understanding of their impact on performance.\n\n 2. **In-depth LII Examination**\nWe intend to supplement our analysis with visualizations illustrating the temporal dynamics of LII across layers during training, as well as a more detailed examination of consistently frozen layers across diverse datasets and architectures. This will be grounded in representation learning theory, offering a more comprehensive interpretation of our findings.\n\n\nThese proposed additions directly address the reviewer's suggestions and will significantly enhance the manuscript, subject to acceptance and spatial constraints in the final version.\n\n---\n\n## Conclusion\n\nWe once again express our heartfelt thanks to the reviewer for their deep engagement with our work. We believe the core contributions of our paper: **(1)** the introduction of LII as a practical estimator of layer-wise metastability, **(2)**  an adaptive, parameter-free fine-tuning scheme, offer novel and valuable insights for both theoretical understanding and practical implementation.\n\nWe are fully committed to addressing all concerns raised regarding clarity, positioning, and mathematical rigor. We are confident that the proposed revisions will significantly enhance the quality and impact of our manuscript. We look forward to incorporating these changes and hope that the revised final version will meet the reviewer's expectations."}, {"review_id": "NLC3ErK9pi", "reviewer": "Reviewer_hVop", "summary": "The authors set out to improve the current computational inefficiency of finetuning vision transformers (ViTs) by studying the stability of ViT layers from an energy-based, modern Hopfield network perspective. In particular, they propose the layer instability index (LII), a theoretically justified metric that quantifies the variability of each transformer layer over time based on attention scores. Consistent with previous results, they find that early layers tend to stabilise much faster than later ones. Based on this result, they propose a new, parameter-free fine-tuning method called energy landscape aware ViT (ELA-ViT) that adaptively freezes stable layers based on thier LII. Experiments across different datasets show that ELA-ViT provides significant wall-clock time savings at equal and sometimes greater accuracy.", "questions": "For the experiments on NAbirds, Beans and CIFAR10, why was ViT-L not\ntested?\n\nThe fast energy convergence of early layers could suggest that the ViT is\nlearning simple, CNN-like features. If so, wouldn’t it make sense to have a\nhybrid “ViT-CNN” model with convolutions at early layers and attention at\nlayer layers? Has this been attempted?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\nThe paper is well written and structured.\n\nIt is also well motivated, effectively relating to previous work and identifying a gap in previous attempts at making ViTs more efficient as well as other fine-tuning methods.\n\nThe quantification of the variability of each transformer layer through the LII is theoretically justified.\n\nThe proposed fine-tuning method (ELA-ViT) is, in contrast to prior approaches, parameter-free, is described in detail and seem relatively easy to implement\n\nThe authors seem to perform a comprehensive set of experiments to evaluate ELA-ViT, including a comparison with the current state-of-the-art method for budgeted layer-wise fine-tuning (ALaST)\n\nWeaknesses\n\nWhy is the comparison with the current state-of-the-art for layer-wise budgeting (ALaST) based on a single dataset? While the authors performed a wide range of experiments, one wonders why this comparison is limited to just one dataset, which seems insufficient to claim superiority\n\nIn the derivations, it is not clear where the exponential-tail assumption for the attention weights is coming from and whether it is justified (empirically or theoretically). Because this is not mentioned in the first statement of the result (lines 150-1) one wonders whether there are other important assumptions behind the result. It would be useful if the authors could clarify this\n\nAs the authors acknowledge in the limitations, the LII might not capture all aspects of representational variability in a transformer layer\n\nMinor points:\nFrom first reading of the paper (including the abstract), it is not clear that the method proposed is for fine-tuning as opposed to training.\n\nThe writing of the theoretical section can become dense at times and one wonders whether more details of the derivations could be deferred to the appendix\n\nThe presentation of the figures could be improved, especially given the remaining space left", "was_revised": true, "final_justification": "I thank the authors for their detailed response. The response generally addressed all my questions and concerns. Based on the discussion, I particularly encourage the authors to improve the presentation of the paper, especially the figures and the theoretical section.", "rebuttal": "# Response to Comments\n\nWe wish to extend our sincere gratitude to you for your thoughtful evaluation and constructive suggestions. We are encouraged that you found our work **well-written**, **well-motivated**, and **theoretically grounded**, and that our proposed method, ELA-ViT, is clearly described and easy to implement. \n\nWe believe your comments have significantly strengthened our work. Below, we address each concern in turn.\n\n---\n\n## *Response to Weaknesses*\n\n---\n\n## 1. Comparison with the State-of-the-Art Method ALaST on a Single Dataset\n\nThank you for raising this important point. We acknowledge that comparing ELA-ViT with ALaST on a single dataset may appear insufficient to claim broad superiority. Our primary motivation for choosing Food-101 was that ALaST’s official implementation and reported results are tailored for large-scale, fine-grained classification tasks, making it an ideal stress test for both accuracy and efficiency trade-offs. As reported in **Table 2**, ELA-ViT achieves a substantial accuracy improvement (**+13 pp**) over ALaST while maintaining greater training efficiency:\n\n| Method                  | Top-1 Acc. (%) ↑ | Time (min) ↓ | Time Δ (%) |\n|-------------------------|------------------|--------------|------------|\n| **Full fine-tuning**    | 88.66            | 297.9        | 0.0        |\n| **ALaST (75% frozen)**  | 77.02            | 181.1         | -39.2      |\n| **ELA-ViT (50% frozen)**| 90.02            | 283.0        | -10.5      |\n\nDue to computational constraints and the complexity of tuning ALaST’s additional loss terms and hyperparameters (e.g., λ_budget) across diverse datasets, we limited the comparison in this submission to one large and representative benchmark. That said, we agree that additional comparisons would further strengthen our claims.\n\nTo that end, we are currently running new comparative experiments with ALaST on CIFAR-100 and Stanford Dogs, using the same training protocol applied to ELA-ViT. Should the paper be accepted, we intend to report these results in the camera-ready version and make them publicly available via our repository.\n\nWe also emphasize that, unlike ALaST, ELA-ViT introduces no additional learnable modules or optimization overhead, which enables broader and easier deployment across domains and models.\n\n---\n\n## 2. Clarification on the Exponential-Tail Assumption for Attention Weights\n\nWe appreciate your request for greater clarity. The exponential-tail assumption is based on empirical observations that trained attention maps often exhibit **high concentration of mass on a small subset of tokens**, with the remainder **decaying rapidly**. This behavior is common in Transformer architectures and motivates our assumption.\n\nPrior empirical and theoretical studies on Transformer interpretability have observed that attention weights often become increasingly peaked or sparse in early-to-mid layers, with a small number of tokens receiving most of the attention mass. For example, [Voita et al. (2019)](https://arxiv.org/abs/1905.09418) showed that attention heads specialize in certain linguistic functions and that some heads become redundant or highly selective. In the vision domain, analysis of token importance in Vision Transformers has found that earlier layers tend to focus attention on local structures, often exhibiting \"Convolution-like patterns\" concentrated around image patches [(Kim et al., 2021)](https://openaccess.thecvf.com/content/CVPR2021W/ECV/papers/Kim_Rethinking_the_Self-Attention_in_Vision_Transformers_CVPRW_2021_paper.pdf). This behavior frequently leads to a phenomenon of 'token overfocusing,' where the model relies on a few important tokens, resulting in highly sparse attention distributions [(Guo et al., 2021)](https://openaccess.thecvf.com/content/ICCV2023/papers/Guo_Robustifying_Token_Attention_for_Vision_Transformers_ICCV_2023_paper.pdf).\n\nThat said, we agree the presentation could be clearer. In the revision, we will:\n\n - Explicitly state the assumption and its empirical justification in the main text (not only the appendix),\n - Add an empirical plot showing the sorted attention weights and their exponential fit,\n - Clarify that while the exponential tail is used for bounding purposes, the LII metric itself is non-parametric and does not require fitting a distribution.\n---\n\n## 3. Limitations of LII in Capturing Representational Variability\n\nWe agree, and we appreciate your recognition of our openness in discussing this limitation. While LII is a simplified proxy based on attention concentration, our theoretical formulation connects it to:\n\n - The residual energy gap.\n - The Fisher Information Matrix trace (**Section 3.3**).\n\nThis supports its interpretability as a metastability indicator.\n\n---\n\n## *Response to Minor Points and Questions*\n\n---\n## 1. Clarity on \"Fine-Tuning\"\n\nWe thank you for pointing this out. We will revise both the **abstract** and **introduction** to explicitly state that ELA-ViT is a fine-tuning method from the outset.\n\n## 2. Dense Theoretical Section\n\nWe appreciate your feedback on presentation. We will move technical derivation steps to the appendix, while retaining the core insights and results in the main text to improve accessibility.\n\n## 3. Figure Presentation\n\nWe will revise the figures, especially **Figure 1** by:\n- Increasing font sizes,\n- Enhancing color contrast,\n- Optimizing layout to improve readability.\n\n## 4. Why Was ViT-L Not Tested on NAbirds, Beans, and CIFAR-10?\n\nThank you for the question. We did not use ViT-L on NABirds, Beans, and CIFAR-10 primarily for two reasons:\n\n **Model–data mismatch**: These datasets are relatively small (e.g., Beans: < 10k samples). ViT-L has >300M parameters, which quickly overfit and lead to degraded performance unless aggressive regularization is applied.\n\n **Compute and reproducibility**: ViT-L requires significantly more memory and compute than ViT-S or ViT-B, limiting its usability on modest hardware.\n\nTo clarify, we will:\n\n - Add a footnote in the experimental section explaining this design choice.\n\n - Include a discussion of model-data scaling trade-offs in Section 6.\n\n## 5. Hybrid \"ViT-CNN\" Architectures?\n\nWe greatly appreciate this creative and insightful suggestion. Indeed, the fast stabilization of early ViT layers supports the idea that they may be functionally analogous to CNN filters.\n\nWhile our current work does not explore architectural hybrids, we agree that replacing early ViT blocks with convolutional layers (as seen in CMT, CoAtNet, and ConvFormer) is a promising direction.\n\nIn the final version (if accepted), we intend to:\n\n - A discussion of this idea in Section 6 under “Future Work”,\n\n - Cite recent hybrid architectures that support this line of thought,\n\nNote that our method could potentially skip convolutional stages based on an analogous “instability index” applied to early CNN features.\n\n---\n\n## Conclusion\n\nWe thank you again for your thorough and thoughtful feedback. We believe that our contributions offer novel insights that are both theoretically informed and practically valuable:\n\n1. **Layer Instability Index (LII)**.\n2. **Adaptive, parameter-free freezing mechanism**.\n\nWe are committed to improving the clarity, rigor, and scope of our manuscript in line with your feedback and believe the planned revisions will substantially strengthen the final version."}, {"review_id": "rRgvkEP8y2", "reviewer": "Reviewer_rPQp", "summary": "The work introduces a fine-tuning method which relies on their novel metric, Layer Instability Index (LII), which quantifies metastability of each Transformer layer over time. Analyses, done by the work, reveal certain Transformer layers exhibit consistent convergence to attractor-like states. Using this insight, it enables the author to create an adaptative training framework which dynamically freezes or skips stable layers based on their energy gaps. To derive their technique, through the energy perspective, the authors relied on the theoretical foundation and connection of Dense Associative Memory to the attention mechanism --- making the work quite interesting.", "questions": "# Questions/Suggestions\n\n1. How do you know if LII actually correlates well with the flatness of energy landscape? Perhaps, you can demonstrate this via some sort of toy experiments as I am still unsure if this claim is true.  \n\n2. Although much of your experiments relied on ViTs, perhaps demonstrating your approach with other models too.\n\n3. For figure 1 (a), are you able to show the same plot with more datasets?\n\n4. It is difficult to see Figure 1 (b). I recommend updating it with much larger font size and better colors.", "limitations": "Yes, they discussed this in their text.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. This is a very interesting work that tackles the problem of efficient fine-tuning a model in a systematically way, which is enabled through the perspective of modern Hopfield models and their usage of energy. Specifically, instead of fine-tuning all Transformer layers, or randomly fine-tune a set of these layers, the authors used their LII metric to identify layers with high LII, which indicate the energy landscape of such layers is not yet flatten (or generalized). \n\n2. The approach dynamically decides which layer to train or skip and introduces no additional big operational costs (e.g., additional losses, learnable gating layers, or adapter low-rank approaches). \n\n3. The experimental results illustrate that many of the models, once fine-tuned via the method, exhibit an increase in performance with respect to the original (non-fine tuned) models.\n\nWeaknesses:\n\n1. It is certainly more efficient to fine-tune with this method in contrast to fine-tuning the entire set of Transformer layers. But, depending on the activity of the attention scores, LII may require the training of all Transformer layers and thus, effectively negate its efficiency benefits. \n\n2. The experimental results are focused on image classification. It is unsure whether this method and its performance will perform well with multi-modalities model or even only text-based model.  I posit that models from other modalities will benefit from the method, but it is unclear regarding the efficiency of the method. Moreover, for the ImageNet models, it seems that the models fine-tuned with this method are only comparable to fine-tuning the entire model.\n\n3. Although I understand the idea, it is likely not very intuitive to others. Thus, to me, the paper lacks a main figure which illustrates the entire process in a digestable way to the general audience.", "was_revised": true, "final_justification": "It is not a bad work as it is quite **innovative** actually. I maintained my score of 4 here because I would like the authors to focus on experiments of reliability of LII. See below for my discussion with the authors.", "rebuttal": "# Response to Comments\n\nWe sincerely thank you for the excellent and insightful review. We are very encouraged that you found our work **\"very interesting,\"** appreciated the systematic approach grounded in the energy perspective of modern Hopfield models, and recognized the novelty and parameter-free nature of our method. We have carefully considered all feedback and will revise our manuscript accordingly. Below, we address each of the specific points you raised.\n\n---\n\n## *Regarding Weaknesses*\n---\n## 1. LII may require training all layers, negating efficiency benefits\n\nWe thank you for this insightful point. It is an important theoretical consideration that if all layers remained unstable, our method would offer no efficiency gains. We are grateful for the opportunity to clarify this. \n\nOur work finds that in the **practical and common setting of transfer learning**, this scenario is empirically not observed. Pre-training on large, diverse datasets (like ImageNet-21k) produces robust, general-purpose feature extractors in the early layers of the network. As our empirical results in **Figure 1a** consistently demonstrate across three very different datasets, these early layers reliably and quickly stabilize (exhibiting low LII) when fine-tuned on downstream tasks. This ensures that a significant portion of the network is always available for adaptive freezing, guaranteeing practical efficiency gains.\n\n## 2. Experimental results are focused on image classification\n\nWe thank you for highlighting the performance improvements of our method. We would also like to gently clarify a point from the summary: our method's performance gains are compared against a **standard full fine-tuning baseline**, not the original (non-fine-tuned) models. This is a much stronger and more challenging baseline, which we believe makes the reported accuracy improvements (e.g., **+6.9pp on CUB-200 with ViT-L**, **+4.2pp on Stanford Dogs with ViT-S**) even more significant. We will ensure this is communicated clearly in the revised manuscript.\n\n## 3. ImageNet performance is only \"comparable\"\n\nRegarding the ImageNet performance, we respectfully wish to frame this result as a **success**. Achieving accuracy that is comparable to the fully fine-tuned baseline (e.g., within **0.6 pp**, as shown in **Table 3**) while simultaneously:\n\n - Reducing training time by **over 10%**,\n - Reducing inference latency by **17%**.\n\nThis demonstrates a highly favorable accuracy-to-compute trade-off, which is the core value proposition of our work. We will revise our discussion in **Section 5** to emphasize this crucial point more clearly.\n\n## 4. The paper lacks a main figure illustrating the entire process\n\nThis is an excellent suggestion for improving the paper's clarity. We agree that a high-level diagram would make our method much more accessible to a general audience. We will add a new main figure to the manuscript that provides a clear, step-by-step illustration of the ELA-ViT fine-tuning process:\n\n - **The initial warm-up phase** to collect statistics,\n - **The calculation of LII** for each layer,\n - **The one-shot freeze decision**,\n - **The final, efficient consolidation phase** where only high-LII layers are trained.\n\n---\n\n## *Regarding Questions/Suggestions*\n---\n## 1. How do you know if LII actually correlates well with the flatness of the energy landscape?\n\nThis is a crucial question that gets to the heart of our method's justification. Our primary evidence is **theoretical**, as established in the paper through two key results that directly link LII to landscape flatness:\n\n  ###  **LII bounds the Hopfield energy gap**:\nIn **Lemma 1**, we prove that the expected energy gap, $ΔE_l$, between a layer's current state and its global energy minimum is upper-bounded by its LII:\n\n$$\\mathbb{E}_x[\\Delta \\varepsilon^{ℓ}(x)] \\leq L^ℓ \\cdot \\text{LII}^ℓ + o(1)$$\n\nThis result guarantees that layers with a small LII have a small energy gap, meaning they have already settled into a flat basin of their optimal Hopfield energy landscape.\n\n  ###  **LII bounds the loss landscape curvature**:\nIn **Appendix B**, we further show that LII upper-bounds the trace of the layer's Fisher Information Matrix ($\\text{tr }F^{ℓ}$), a standard measure of the loss landscape's curvature:\n\n$$\\text{tr } F^ℓ \\lesssim C_ℓ \\cdot \\text{LII}^ℓ$$\n\nThis proves that layers with low LII are **\"Fisher-flat,\"** meaning they are in regions where gradients are small and further training offers diminishing returns.\n\n**Together**, these theoretical results provide a firm justification for using LII as a proxy for flatness. To complement this, as you suggested, we will add a new visualization to the appendix that empirically plots LII against the Fisher trace, visually confirming this strong correlation.\n\n## 2. On Generalizing Beyond Vision Transformers\n\nWe appreciate this suggestion to broaden the scope. We would like to highlight that our experiments in **Table 1** already include **DeiT-B**, a popular and efficient ViT variant, which shows that our method generalizes beyond the standard ViT architecture. We will make this point more prominent in our results section. Extending the work to fundamentally different architectures (e.g., CNNs or other non-Transformer models) is an exciting avenue for future research. We agree that the core insights of our work, particularly the metastability of early layers and energy-based freezing, are not limited to ViTs. As noted in Section 6 (Limitations), both the LII metric and the ELA-ViT mechanism are broadly applicable to attention-based models.\n\nWe envision natural extensions to:\n\n - Language models (e.g., BERT, RoBERTa), by tracking token- or head-level instability,\n - Multi-modal models (e.g., CLIP, Flamingo), where attention across modalities may stabilize at different rates.\n\nWhile our current submission focuses on vision due to scope and compute constraints, we are actively running pilot experiments with RoBERTa on GLUE, where early indications suggest similar layerwise stabilization trends.\nIf accepted, we intend to:\n\n - Add a “Future Work” box outlining how LII could generalize across modalities,\n - Clarify in the main text that our framework is architecture-agnostic and grounded in attention dynamics, not modality,\n - Include a brief mention of our ongoing NLP experiments, if they are completed in time and align with NeurIPS camera-ready guidelines.\n\n## 3. For Figure 1(a), are you able to show the same plot with more datasets?\n\nWe appreciate the feedback about Figure 1’s clarity and presentation. **Figure 1a** currently shows the LII profile across three diverse datasets:\n - The medium-scale **CIFAR-100**,\n - The fine-grained **Food-101**,\n - The large-scale **ImageNet-1K**.\n\nWe believe this selection is sufficient to robustly demonstrate the consistent **\"U-shaped\" instability pattern** that is central to our motivation. While we could add more datasets, we feel it would likely clutter the figure without providing significant new insight beyond what is already shown.\n\n## 4. It is difficult to see Figure 1(b)\n\nThank you for pointing this out. We apologize for the poor readability. As also noted by the other reviewer, we have updated this figure (and all others in the paper) with:\n\n -  **Significantly larger font sizes**,\n -  **Improved color contrast**.\n\nWe appreciate the suggestion for a more accessible, high-level illustration. If the paper is accepted, we plan to include a conceptual overview figure near the beginning of the final version to visually summarize the ELA-ViT pipeline. We believe this will improve accessibility and align with your recommendation for a main figure that illustrates the process in a digestible way.\n\n---\n\n## Conclusion\n\nWe thank the reviewer for the generous evaluation and thoughtful suggestions. We have carefully considered all feedback and intend, if the paper is accepted, to:\n - Clarify the connection between LII and energy dynamics,\n - Strengthen generalization claims with preliminary cross-domain findings,\n - Improve figure quality and interpretability,\n - Expand both theoretical and empirical support for key assumptions.\n\nWe believe these improvements will enhance the accessibility, rigor, and overall impact of the work."}]

Author rebuttal:

Meta-review: This submission proposes a measure they call the Layer Instability Index (LII), a measure of the variability in attention patterns in a given layer across input tokens. LII is shown to upper-bound the trace of the Fisher information matrix for the layer and the energy gap in the Hopfield update interpretation of self-attention. When fine-tuning ViTs on some standard benchmark datasets, freezing layers with the lowest LII leads to accuracy improvements during fine-tuning with a modest improvement in wall-clock time.

Reviewers commended the novelty of the proposed approach, its simplicity, and the existence of a theoretical justification. Reviewers f65B and hVop initially had some concerns regarding the details of the theory but now recommend acceptance. Reviewers Bcjj, hVop, and rPQp all asked for additional experimental validation. Reviewer Bcjj asked for ImageNet results, which were present in the supplementary material. I don't think the technique works particularly well here, as noted by reviewer rPQp, but I commend the authors for including the results. Reviewer hVop was concerned that the paper compares vs. SoTA on only one dataset; the authors agreed to add comparisons vs. ALaST on CIFAR-100 and Stanford Dogs to the paper. Finally, Reviewer rPQp asked for experiments beyond image classification. While I agree that such experiments would add to the paper, their absence does not undermine the novelty or claims of the paper.

Overall, this is a borderline paper. Reviewers did not note any flaws in the theory, but they also felt that it was somewhat derivative of previous work. The experimental evaluation is quite interesting if not entirely conclusive—the method convincingly improves over the baseline for small image classification tasks, but not on ImageNet. Overall, despite the paper's minor flaws, I believe the ideas described here are promising enough to be worth communicating to NeurIPS's audience. I thus recommend acceptance
Decision: Accept (poster)