NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks
Authors: Nandan Kumar Jha
Brandon Reagen
Venue: ICLR 2026 Poster (2026)
Keywords: Eigenspectrum
feed-forward networks
training dynamics
latent space geometry
optimizer geometry
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce NerVE, an eigenspectral probe of FFN nonlinearities that quantifies how they restructure latent variance, yielding spectral signatures that track generalization and reveal consistent effects of architecure and optimizer design.

Abstract: We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and regulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their high-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigenspectrum dynamics via four complementary metrics: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality), Eigenvalue Early Enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shifts). Our *key insight* is that FFN nonlinearities reinject variance across eigenmodes, fundamentally governing latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection.
We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dynamics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model's generalization ability and respond predictably to design choices, generalizing beyond transformer to MLP-Mixer architectures,  providing actionable insights for architectural and optimizer choices beyond trial-and-error.

Published as a conference paper at ICLR 2026
NERVE: NONLINEAR EIGENSPECTRUM DYNAMICS IN LLM FEED-FORWARD NETWORKS
Nandan Kumar Jha & Brandon Reagen
New York University
{nj2049, bjr5 @nyu. edu
ABSTRACT
We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and reg- ulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their sigh-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigen- spectrum dynamics via four complementary metries: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality). Figenvalue Early Enrichment (top- heaviness), and -aasennn divergence (distributional shifts). Our key insight is that FFN nonlinearities reinject variance across eigenmodes, fundamentally gov- erning latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection. We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dy- namics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model's ' generalization ability and respond predictably design choices, generalizing beyond transformer to MLP-Mixer architectures, providing actionable insights for architectural and optimizer choices beyond trial-and-erron
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across wide range of natural language tasks, driven in part by advances in transformer-bast architcctures. While much emphasis has been devoted understanding attention mechanisms and token-wise interac- tions, the role of feed-forw forwwfd networks (FFNs), particularly their nonlinear components, remains underexplored, despite FFNs dominating both the parameter budget and computational footprint of transformer-basee models (Geva et al. 2021 de Vries (023).
Despite their apparent simplicity, FFNs perform digh-dime nonlinear transformations that regulate information flow by reorganizing, compressing, and propagating the information extracted by attention modules across layers. Understanding how these transformations evolve and interact with architectural design choices remains a fundamental open question.
One challenge in interpreting FFNs is the absence of systematic and efficient tools for characterizing how latent representations are structured and transformed by nonlinear activations. FFN transforma- tions unfold a high-dimensional latent space which far less accessible for direct visualization and probing compared to multi-head attention. Prior work, Kobayashi et al. (2024) used attention maps to study the input-contexttualizziion effect of FFNs, Balesstrro et al. (2024) characterized FFN geometry through piecewise-affine spline partitions. Neither lens reveals how nonlinearity redistributes variance, nor captures the rich spectral structure inherent in these transformations.
To this end, we introduce NerVE, a unified, online, and memory-efficient framework for analyz- ing FFN latent geometry through the eigenspectrum analysis. NerVE summarizes pre- and post- activation spectra using four ocale-invariant, distribution-awen metrics: spectral entropy (dispersion vs uniformity), participation ratio (effective latent dimensionality), eigenvalue early enrichment (top-heaviness), and Jensen- Jensen-Shannon divergence (distributional shift).

---

Published as a conference paper at ICLR 2026
Spectral Uniformity
Effective Latent Dimennlonality
Top ttp-haaviness
Distributional Shift
037
Natto Pulls
1020
0.3
500
08
ESE,
10K
20C
30K
40K
10kK 22K
3.k
40c
10K
20K
30K
<DK
1:k
23k
3.6
43k
Taking
Thinng
Takkep Slep
Figure 1: NerVE quantifies nonlinear eigenspectrum dynamics in FFNs of GPT-2. FFN nonlinearity (GELU) regulates information flow by reinjecting variance, reactivating (post-activation SE1 and fR and flattening the eigenspectrum, less top-heavy (post-activation BEEL). The IS heatmap shows a depth-localized transition band where redistribution is strongest.
directions
From methodological standpoint, these metrics span broad theoretical range and expose the complementary facets of the cigenspectrum that any single scalar would obscure, thereby enabling continuous tracking of latent geometric dynamics. Spectral Entropy (SE) captures the uniformity of variance distribution (De Domenico & Biamonte (2016), and Participation Ratio (PR) reflects the geometric notion of effective dimensionality, indicating how many directions meaningfully contribute to total variance (Gao et al. 2017). Unlike SE and PR, Eigenvalue Early Enrichment, which quantify the top-heaviness, can distinguish the eigenspectrum utilizing different fractions of the latent space (Marbut et al. 2023). Finally, the Jensen-Shannon (JS) divergence provides an information-theoredt distance measure between two eigenspectra (Lin 1991p, quantifying distributional shifts of variance.
We apply this framework across diverse range of architectural settings, including LayerNorm placements: PreLN, PosILN, and MixLN et al., 2025);; normalization- free variants (Jha & Reagen 2024): FFN weight-geometry constraints (Miyato et al. 2018 Sallmaan & Kingma, 2016),, and hyperspherical constraints (Liu et al. 2017); positional encoding scheme (Su et 2004)): and the optimizer choices, including Adam (Kingma 2015 Loshchilov & Hutter 2019), Muon (Jordan et al.,
2024), Dion (Ahn et al. 2025), Adafactor (Shazeer & Stern 2018), and SGD; see Table
Across settings, clear pattern emerges: FFN nonlinearities do not merely rescale the activations, they actively reinject the variance across eigenmodes and reawakens the inactive directions in high- dimensional latent space. As shown in Figure [ the post-activation spectra in GPT-2 consistently show increases in SE and PR, and decreases in EEE, while JS heatmaps reveal transition bands where redistribution is strongest. This highlight the active role of FFN nonlinearities in regulating information flow and latent geometry that downstream layers further exploit.
Contributions: Our contributions can be summarized follows:
1. Conceptual. We demonstrate that FFN nonlinearities do not simply rescale activations but actively reorganize eigenspectra, reinjecting variance into under-utilized directions. Moreover, optimizer geometry modulates the extent of variance reinjection, altering the role of FFN nonlinearity from repair (recovering spectral collapse) to refinement (stabilizing a well-conditioned spectrum).
2. Framework. We introduce NerVE, lightweight and memory-efficiens methodology for online tracking of FFN eigenspectrum dynamics, using four distribution-awan
scale-invariant metrics. 3. Diagnostic. We show that architectural (normalization layers, activation functions, gating, weight geometry, positional encodings), and optimizer (Adam W. Muon, Dion, Adafactor, SGD) choices imprint distinct spectral signatures in FFNs, which can be used for diagnosing the model behaviors.
4. Empirical. We validate NerVE on GPT-2 and LLaMA models (7 1M to 1.3B) trained from scratch on CodeParrot, Open WebText, FineWeb, and C4 datasets; extend to the anntrranfformr MLP. Mixer (B/16) on CIF,R-100, confirming arssttrcche generality; and perform extensive robustness studies across normalization variants, optimizer family, and token positions.
Table 1: Summary of key NerVE findings per experimental axis.
Experimental Axis
Key Findings
Activation (GELU vs ReLU)
Similar trend, distinct dynamics; GELU explores broader subspace ($3.)
Norm-free models
GELU exhibits spectral inertia; ReLU compensates LayerNorms ($3.2)
FFN weight geometry
Spectral dynamics matter; performance tracks sustained flattening ($3.3
Norm placement (Pre/Post/Mix)
PreLN: best return-on width; PostLN: diminishing spectral returns (33.4
Positional encoding
RoPE prevents mid-to-der spectral collapse (depth utilization 1) ($3.6
Optimizer (AdamW vs Muon)
Repair or refinement; performance follows mid-layer capacity trends
($3.7
Token positions
Position-uniform early on; depth shift capacity to later-tokens (Appendix
Non-trane Non-transformer IMLP-MixeN
Core findings generalize beyond transformer architecture [Appendix[]
LayerNorm vs RMSNorm
Robust: placement and FFN activation choices matter more (Appendıx K
2

---

Published as a conference paper at ICLR 2026
2 NERVE: A PRINCIPLED FRAMEWORK FOR EIGENSPECTRUM ANALYSIS
Notations, Let be the number of layers, d the embedding dimension, D the FFN hidden dimension, B the batch size, S the context length, and E the FFN (pre/post-activation) covariance matrix.
2.1 FORMULATION OF EIGENSPECTRUNE BASED FRAMEWORK
To understand how information is structured and propagated through FFN latent space, we analyze: (I) variance distribution and its impact on effective dimensionality; (2) how nonlinearity within layer reshapes this distribution; (3) how these patterns evolve across layers and training. The NerVE framework (Figure 10 in Appendix [A.T] consists of four main components: i) activation collection, (ii) covariance matrix computation, (iii) eigendecoomposition, and (iiv spectral metrics calculation.
Activation collection. For FFN with (non-gating) architecture (FN = W (Wup +b, +b2, where is the activation function (e.g., ReLU, GELU), we collect PreAct( X) = W Wap + bi and PostAct( X) = (W + b1) the output of the up projection, and input to the down projection (after activation function), respectively. For activation with gating mechanisms (e.g., SwiGLU in LLaMA), the architecture becomes FFN = W Woown (a( gate @ (Wupz), where @ denotes element-wise multiplication, and collect PreAcc((X) = W gate and PostAct(()) = o o gune O (Wup.).
Covariance matrix computation. At the logging step t, for each layer we collect full activation matrices PreAct(X) ((.)0) E NR and PostAct( (otAAc(((((.t)) (1.))) NR where N B x the total number of tokens in the batch. These tensors, originally shaped [B, D]. are flattened to [B x S, D), intentionally discarding sequence order. This allows us to compute an unbiased covariance matrix for all tokens in the batch, treating each token as independent sample in FFN latent space.
Computing covariance with all N tokens a batch, with no sub-sampling ensures exact second- orddr statistics rather than their statistical approximations; thus, spectral analysis captures true statistical properties distribution. For each set of activations, we compute covariance matrix as follows:
Σ (X u p)) E IR DxD, where X E IR Nx are activations and N
X: (I)
N -
E1
This yields two covariance matrices per FFN layer: (L.t) PreAA (X) and ProstAer (X).
(1.1)
Eigendecompusstion For each covariance matrix, we perform eigendecomposition (Ev Av), and sorted the eigenvalues in descending order: λ 2 A2 2 2 Ap 2 O. We define A = Σi A; the total variance, and normalized cigenvalues to create probability distribution ÅΙ A.AA
Spectral metrics computation Next, we compute four scalar metrics from the cigenspectrum of (ll.t) E'reAca (X) and X/ (X), which quantify distinct aspects of the eigenspectral dynamics: Spectral Entropy, Participation Ratio, Eigenvalue Early Enrichment, and the lensen-Shannon divergence.
2.2 EIGENSPECTRUM METRICS FOR ANALYZING HIGHI- DIME-DIMENSIONAL LATENT SPACE
Spectral Entropy (SE) Spectral Entropy quantifies the uniformity of eigenvalue distribution in high-dimensionall latent spaces. Formally, is the Shannon entropy of the normalized eigenvalue distribution derived from a layer's covariance matrix: SE = Σm= log A,
Mathematically, spectral entropy equivalent to the von Neumann entropy (vNE) in quantum information theory, which quantifies the degree of quantum entanglement or mixedness of a quantum state (De Domenico Biamonte| 2016). In quantum mechanics, vNE defined as: SNNE(P) -Tr(pInp)) p). where P denotes density matrix, a positive semidefinite operator with unit trace that encapsulates the probabilistic nature of quantum states (Nikitin al. 2024 Huang et al. 2023).
For FFN, an analogous density matrix is created by normalizing the covariance matrix by its trace: Applying this to pFFN, SE becomes the Shannon (or von
TH where Tr(E)
PFEN
Neumann) entropy of the normalized eigenvalue distribution SE = Dp Å log d,-
Thus, when the eigenspectrum exhibits significant anisotropy (e.g., \1 > Az...., AD). SE approaches zero, indicating a collapsed or low-rank representation. Conversely, when eigenspectrum approach uniformity (Az R Aÿ : Vi, j), SE approaches its theoretical maximum, In()).
3

---

Published as a conference paper at ICLR 2026
Participation Ratio (PR). It measures effective dimensionality of an eigenspeetrum (Hu & Som polinsky 2022) and quantifies how many dimensions significantly hold variance. Formally.
()) A
PR
where 1 PR <
X
Σ
Σ
PR values close to 1 indicate maximal anisotropy (L.c., variance concentrated in single direction), while a value near D indicates uniform variance across all dimensions. While SE depends on the entire distribution shape (including small eigenvalues) and measures the uniformity of distribution, PR focuses on how many directions are active and meaningfully contributing to the total variance.
Early Eigenvalue Enrichment (EEE). It quantifies the top-heaviness of an eigenspectrum by tracking how rapidly the leading principal directions accumulate variance. Specifically, it captures how front-loaded the variance is among the top eigenvalues, by assessing how quickly the cumulative sum surpasses that of uniform spectrum (Marbut et al.) 2023).
Formally, the proportion of variance explained by the top k principal directions defined by the normalized cumulative sum at index as Sj: = x -ー=1 dis and for comparison, the ideal uniform & reference grows linearly as D (see Figure 2). The EEE score is then the average vertical distance between the empirical cumulative curve and This ideal line, normalized by the maximal possible value:
k: ( = =2 Σ D
1 D
D
A
k
EEE
E
D
x
D
k=1
EEE x indicates that most of the variance is concentrated the top few directions, forming a steep eigenvalue spectrum; conversely, EEE x 0 corresponds to nearly uniform spectrum.
We analyze the cumulative variance distribution of various (f8-dimensional latent space, using the EEE metric in Figure 2 The resulting curves shows a spectrum of dimensional utilization, ranging from extreme anisotropy to fully uniform variance. The One dimen- sion spectrum exhibits an EEE of 1.00, where nearly all variance is concentrated in a single dominant principal component, indicative of a highly degenerate latent rep- resentation. As more dimensions begin to carry variance (from 10% to 99%), the EEE value decreases from 0.94 to 0.44, suggesting gradual transition toward more dis- tributed representations. Notably, EEE's nonlinear scaling with dimension count highlights its sensitivity to early eigenvalue dominance, making it a valuable diagnostic for understanding how architectural choices and training dy- namics shape the effective dimensionality of latent spaces,
1.00
eigenspectra
Cumulative Variazce
0.8
0.72
0,20
0.6
Onc dimcauion 10% dimencion
0.4
25% dimensions
50% dimencions
75% dimemiiona
84% dimenxinox
Uniffero
0.0
100
200
300
400
500
600
700
Principal Component
Figure 2: Cumulative variance distri- bution across (68-dimensional latent space. Higher values (shown on curves) indicate top-heavy concentration in few dominant directions, while lower values reflect a more uniform distribution.
Jensen- Shannon Divergence (JS) Unlike the previous metrics, which describe single eigenspectra in isolation, JS provides a principled measure of dissimilarity between two eigenspectrum within a layer. Specifically, quantify the extent of distributional shifts from the pre to post eigenspectrum caused by FFN nonlinearity. For a normalized eigenvalue distributions Pree
p and Chapter 4.6.3)):
{XPTh )
D Post (Apon I where π >>/ Σ}= Xÿ, the IS is defined as (see Amari 2016
JS(P pre
Ppoa) 2 (Pore Pree M) + D (Pposs M)
where M = Pun Punt Pon is the midpoint distribution and DRL. is Kullback-Leibler divergence:
A
DxL (P (1 Σ XP
log
(5)
For numerical stability, we compute IS for FFN as follows:
D
21.0
21post
JS pre
I Prost)
pre log
+
post log
(6)
post
TT +
A pusi
1=1
4

---

Published as a conference paper at ICLR 2026
3 EXPERIMENTAL RESULTS
Models and datasets We evaluate the FFN eigenspectrum of two model families: GPT-2 and LLaMA-style architectures. For GPT-2, we train a 125M parameter model on 2.1B tokens from the CodeParrot dataset, which is created from 20M GitHub Python files and preprocessed using HuggingFacc tokenizer of vocabulary size 50K. For LLaMA-style models, we train in-house variants with 71M and I 130M parameters on the C4 dataset (Raffel al. 2020), tokenized using the T5-base tokenizer with a 32K vocabulary. These LLaMA variants follow the architectural specifications (depth, embedding dimensions, FFN width, positional encoding, and SwiGLU activation) from Li et al. )2025), which adopts downscaling methodology Lialin et al. (2024); Zhao et al. (2024). For experiments with RoPE, we train GPT-2 on OpenWebText dataset, following the architectural settings and training recipe from Loshchilov et al. (2025). To study pprimizer-dependen (AdamW W, Muon, Dion) dynamics, we train GPT-2 350M and 160M variants on Web (Penedo et al. 2024) dataset.
Training setup All experiments are conducted on NVIDIA RTX 3090 GPUs (24 GB). GPT-2 models are trained for 41 K steps with context length 128 on the CodeParrot dataset. For RoPE experiments, GPT-2 is trained on 26B tokens from OpenWebText using 4 GPUs with context length 512. LLaMA- 71M is trained on I.IB tokens for 10K steps, while LLaMA- 30M, LLaMA- and LLaMA-1.3B variants trained on 2.2B tokens for 20K steps. All LLaMA models use context length of 256. For optimizer-speciinn eigenspectrum analysis, we train GPT-2 models with 512 and 1024 context lengths, following the hyperparameter settings from Ahn al. (025))
3.1 FFN NONLINEARITY REINJECT VARIANCE AND FLATTEN THE EIGENSPECTRUM
Variance is reinjected, not merely rescaled Figure contrasts pre- and post-activation spectral dynamics and highlights the role of nonlinearity withing FFN. The PreAct eigenspectrum is highly top-heavy as most variance is concentrated in a few leading directions. This is reflected by lower SE and PR, indicating lower utilization of the latent space. Once the nonlinearity activated, both SE and PR jump upward across training, suggesting that the nonlinearity redistributes variance across more dimensions. effect, the nonlinearity reawakens previously inactive directions, injecting new degrees of freedom into the latent space. This variance reinjection promotes features disentanglement which facilitate more effective downstream processing subsequent layers.
Flattening and reshaping the eigenspectrum The variance redistribution has noticeable impact on the spectrum shape. The EEE values, which quantifies how sharply leading cigenvalues dominate, drops consistently for post-activation, re-affirming that the spectrum is being flattened. Instead of concentrating variance in a small number of dominant modes, the post-activation spectrum spreads variance more evenly. Moreover, the JS heatmaps shows a distributional shift: post-activation eigenspectra are not merely scaled versions of pre- activatii ones but are effectively reordered.
GELU vs. ReLU: Similar Trajectory, Distinet Dynamics GELU (Figure I and ReLU (Figure follow the same qualitative trajectory-varinne reinjection ($SE 1PR), spectral flattening (LEEE), and distributional reordering (fJS)-buw differ in pace and extent. While ReL.U stabilize SE and PR carlier, suggesting a faster reinjection of variance, GELU progresses more gradually yet ultimately pushes post to higher values. This indicates that smoother nonlinearity in GELU enables broader subspace exploration, which correlates with GELU's lower perplexity (Table
These eigen-metries highlight the functional role of nonlinearity: (1) improving the directional usage of FFN latent space (SE 1. PR 1), reflecting an increased participation of multiple latent directions for encoding information; and (2) reducing the dominance of a few principal direction (EFE L). Thus, NerVE provides geometric underpinning of the nonlinear expressivity in transformer models.
3.2 COMPENSATORY ROLE OF FFN NONLINEARITY IN THE ABSENCE OF LAYERNORMS
Removing LayerNorm from transformer architectures eliminates their layerwise re-centering and variance normalization, shifting the burden of statistical regularization entirely onto the attention and FFN sub- blocks. This motivates a central question: Can FFN activation functions compensate for the absence of normalization, and if so, to what extent and through what mechanisms? Our findings reveal that, unlike GELU, Remulfy activations actively compensate the absence for removal of LayerNorms by regulating the FFN latent space variance.
5

---

Published as a conference paper at ICLR 2026
RP.5 0, (est
PR RetB p00---997 (
Redl pre.dd 9831
Rete
750-
500-
LA
66
25 pm
12K
25K
30K
40K
30K
20
30K
4.K
10K
20K
30C
40k
20K
40K
Therin Soop
Thikiing Steze
Takking Stez
Figure 3: Eigenspectrum dynamies illustrate how FFN nonlinearities regulate information flow and reshape the eigenspectrum during training for GPT-2 (ReLU) on CodeParrot, Pre- and post-activation dynamics shown for SE, PR, and EEE, highlighting how nonlinearities reinject variance and alter spectral structure. JS heatmaps (rightmost) capture the layer-wise distributional shift induced by nonlinearity, In-panel titles report Pearson correlations (r) between each metric and evaluation loss.
$48, poat- NernFrea/(6W0)
Spectes Spectet
15.
www.
20
32k
10K
20C
20C
resthatn)
GainaPPartirall Pornttast
past Nee-mnrr (heil))
JS. Nemmrrssllaa
300
Speditl Sendii
200
Liser
1000
10K
204
3:k
10K
201
20K
30K
6
10c
20K
666 past- SarnerrelLLaaklleeee
73445
300
200
1:00
10K
201
3GK
40K
J0K
201
30K
40C
10K
20K
30K
4
10C
20K
36k
46K
Figure 4: Eigenspectrum dynamics for norm-free GPT-2 (125M) models with GELU (top), ReLU (middle), and learnable-slope Leaky ReLU (bottom). Columns show layer-averaged SE (pre vs. post). PR gain (post to pre), post-activation EEE (yellow regions indicate top-heavy distribution), and JS (yellow regions highlight strong redistribution) across layers and training steps. Norm-free GELU exhibits spectral inertia in layers 0 to 5 (EEE + I, - O); whereas, ReLU and Leaky ReLU aggressively reinject variance (PR gain > 200x) and flattening the spectrum (EEE- < 0.3).
Spectral inertia in normalization-free GELU models Normalization-ffre GELU model exhibits spectral inertia in early layers, characterized by EEE & 1 and JS R (see Figure 4. This indicates that the nonlinearity in early FFNs fails reinject variance into the latent space, leaving the cigenspectrum heavily front-loaded. Consequently, variance remains confined to a few dominant subspaces, and there is significant overlap between SPire and SEpps- Thus, nonlinearity in early FFNs does not activate new directions, and information continues to flow through a narrow subspace in subsequent layers. This spectral bottleneck reflects downstream consequence of entropic overload, a critical failure mode observed in normalization-free LLMS (Jha & Reagen 2024). where disproportionate number of attention heads in the early layers stuck in higher-entropy states throughout training, squandering the representation diversity of multi-head attention mechanism. and degrades the performance (higher perplexity, see Table 2
Early FFNs overcompensate to break spectral inertia in normalization-fre ReLU models In contrast with GELU, ReLU and learnable-slope Leaky ReLU variant exhibit strong compensatory behavior when LayerNorms are removed. Specifically, in the first two FFN layers, the post-to-pre Participation Ratio (PR) gain surges by A 20x to 300x (blue curves, Figure 4 indicating an abrupt reinjection of variance into previously underutilized latent directions. Consequently, the post- activation EEEpost remains consistently low (== 0.3-0.5) across layers, indicating that the spectrum becomes flatter and more isotropic, rather than top-heavy. This redistribution is further corroborated by non-overlapping SEpre and SEpost spectrum, and by JS peaks =00488 in the early-layer contour maps, confirming the crucial role of nonlinearity in reshaping eigenspectrum in early FFNs

---

Published as a conference paper at ICLR 2026
400
P5, Weight Norraliaation
514 Ci thick Ae!
iPost Pre) ad
Weight Nomallzation
100
e0r. pest Weight Hermultaation
BEC POSt
Weig Normulication
100-
200-
0.4
114
300-
0.
PR, poot
0.4
1.k
20k
40K
0.6
14K
20K
45K
10K
20K
15K
10C
20C
10K
Traiing Siss
Turnng Step
Tainig SHap
Teiting Sies
300
poic Spectral Homooiization
0.0
14 (Puct Pre) Sipectral Normalization
1.0
446 pact -Specrral Normaliiation
15- Speccrai Nomalization
200
12 Stteoceeeu
-02
-0.4
0.4
1.2K
20k
30K
40K
-0.6
1.k
20K
30K
45K
10K
23K
30k
4D%
10C
20k
30k
45K
Traning Sice
Toriing Step
Turnin Stpp
Teeting Sies
PRupost- hyperapheriaal Normoltaation
000
(Paste Prei Hepampherical Normaliation, 10 post- -yyerspherkaal Nomelization
Hiperrpphrical Norriizatiion
-02-
0.8
4229
3220
1110
200
-04
15K
20K
30K
40K
-0.6
1/k
20K
30K
45k
10K
25K
30C
40c
10C
20K
30k
40K
Yaaain Step
Yaanng Stop
Figure 5: Impact of FFN (parametric) normalization in norm-free GPT-2 with learnable-slope leaky ReLU. Eigenspectrum dynamics are quantified by latent capacity (PR-post), spectral regularization and flattening (ABEE and EEE .post,, and distributional shift (JS). Top bottom: Weight, Spectral, and Hyperspherical Normalization. Each method exhibits distinet JS localization and spectral patterns, showing different influences on FFN internal dynamies.
This aggressive variance injections demonstrate that FFN nonlinearity can partially assume the statistical regularization role of LayerNorm, widening the latent manifold and mitigating spectral bottlenecks. In terms of predictive performance, both ReLU variants reduce the perplexity gap to the LayerNorm baseline by =50% (refer to Table
Table 2: Evaluation perplexity (PPL L) comparison across GPT-2 baseline models (GELU and ReLU), norm-free models (GELU, ReLUU, learnable-slope Leaky ReLUU. Parametric normalization (Weight, Spectral, Hyperspherical) are applied to FFNs of norm-free learable-slope Leaky ReLU models. All models trained on 2.1B tokens from CodeParrot dataset.
Baseline Models
Norm-free Models
Norm-free w/ FFN-Norm
GELU
ReLU
GELU
ReLU
Leaky ReLU
WNorm
SNorm
HNorm
PPL
2.714
2.774
3.223
2.988
3.081
3.041
3.000
3.122
3.3
FEED-FORWARD NETWORKS WEIGHT GEOMETRY AND EIGENSPECTRUM DYNAMICS
Previously, we have seen that how ReLU variants improve the redistribution of top-heavy eigenvalues in the early layers of normalization-fre free LLMs. We now analyze how parametric normalization applied to their FFNS further influence eigenspectrum dynamics. Figure 5 shows the effects of weight, spectral, and hyperspherical normalization applied FFNs.
Parametric normalization alters the localization of distributional shifts across layers Despite being applied only to FFN linear layers, each parametric normalization technique induces distinct learning dynamics, as demonstrated by the layerwise JS divergence in Figure 5 (rightmost column). Specifically, SNorm exhibits highly localized distributional shifts in the mid-to-deeper layers that emerge very early in training. In contrast, WNorm induces distributional shifts in a smaller subset of mid layers that appear very late in training. Meanwhile, HNorm triggers strong shifts in the early layers at the very-beginning of training, which gradually diminish as training progresses.
Spectral normalization achieves superior performance through smooth and sustained spectral flattening By constraining the spectral norm of each FFN weight matrix, SNorm induces early and consistent spectral flattening, reflected in uniformly negative ΛEEE (Post-Pre) values in Figure 5 especially in deeper layers. This yields the lowest EEE_post (=-0.455) among all parametric normal- ization methods, indicating balanced variance distribution across a moderate number of directions (PR.post 200) and improved latent space utilization. In contrast, WNorm shows delayed and highly localized flattening in a few mid layers, while HNorm induces early flattening in shallow layers that vanishes training progresses.
7

---

Published as a conference paper at ICLR 2026
Hyperspherical normalization underperforms due to early overshooting in eigenspectrum
HNorm projects weight vectors onto a unit hypersphere (Loshchilov et al. 2025| Lee et al.. 2025a Wang & Isola 2020), which rapidly expands latent capacity, indicated by sharp increase in PR post (exceeding 600). However, this expansion cause an early-overshooting in AEEE dynamics, and F.F. values remain high across depth, indicating the persistent dominance of few principal directions. Moreover, the JS divergence patterns also reflect the inefficient use of model's 's depth. Hence, the combination of early overshooting, lack of spectral control, and depthwise redundancy leads to HNorm's 's degraded perplexity. While large latent capacity can be beneficial, it must be paired with sustained flattening mechanisms to prevent top-heavy eigenspectrum from re-emerging.
3.4 IMPACT OF LAYERNORM POSITIONING ON THE FFN LATENT SPACE DIMENSIONALITY
PreLN turns width into usable dimensions while PostLN shows diminishing returns at higher width. Across the FFN-width sweep (D=ld-8d) 8d), the normalized PR, which reflects the effective utilization of available latent space, for PreLN is highest and remains nearly flat as D increases. Figure shows the layer-consistent behavior for PreLN, highlighting the conversion of added width into usable dimensions. Thus, PreLN offers the best return-on-width.
The width utilization is lowest for PostLN and decreases with D. revealing growing spectral concentration- concentrated into fewer dominant direc- tions instead of broadening the effective di- mensionality. MixLN is intermediate with medians between PreLN and PostLN and wider layer-to-layer spread, implying less stable inductive bias across depth. Thus, LayerNorm placement governs how width is spent. Refer to Table 7 (Appendix D for raw PR post values, and Figure (Ap- pendix C.I for a detailed discussion on spectral signatures.
capacity is
20335
2443
$214
Hoe Dincrsio
Figure 6: LayerNorm positioning and FFN width sweep: Post-activation participation ratio is normalized by D for PreLN, MixLN, and PostLN configurations. PreLN sustains the highest and most stable utilization of FFN width across the sweep, PostLN incurs diminishing re- turn higher FFN width, MixLN lies in between but with greater layer-to-layer variability.
3.5 EIGENSPECTRAL SIGNATURES PREDICT GENERALIZATION
Practical utility of VE as online monitoring tool and architectural selection proxies. First, to assess whether VerVE metrics serve as online training diagnostics, we correlate SE and PR with validation loss across training checkpoints for each FFN width variants (Table left). Pre-activation correlations exceed |r| Z 0.97 at every width, indicating that spectral metrics track generalization throughout training, and they can be used as nerward-pass-ond activation PR correlation strengthens from |r| = 0.85 at D = 1d to Ir| 2 0.93 at D 2 2d, suggesting that a modest FFN width is required for producing
diagnostic. Notably, the post-
pewerraiiizatiorreeeiieee Table 3: Correlation between eigenspectrum metrics (SF, PR) and generalization. Left (within-run): Pearson r between each metric and validation loss over checkpoints at each FFN width (D=ld-8d). Right (cross-config): Pearson r between final metric values and perplexity across the eight width configurations for each architecture and activation. Higher SE and PR consistently implies lower loss.
spectral signatures.
FEN Width Configuration (GPT=2 GELU)
GPT-2
NommFiee GPTT-2
D-Id
D-24
D-3d
D-4d
D-5d
D-6d
D-7d
D-&d
GELU
ReLU
GeGLU
SwiGLU
GELU
ReL.U
LReLU
SE,pre
.0.88
-0.98
.0.99
-0.99
-0.99
-0.99
.0.99
-0.99
40.99
.0.98
.0.55
.0.97
-0.82
0.03
0.03
SH_post
-084
41.86
-0.87
-0.87
-1.87
-1.00
-1.00
-0.55
-0.92
-0.99
-1.00
PR pro
0.97
-0.98
00.88
-0.99
-0.98
-0.97
0.98
0.97
0.99
-0.98
0.97
-0.7
-0.93
-0.55
-060
PR
.0.85
-0.93
.0.94
-0.94
.0.95
-0.95
0.93
0093
-1.00
-0.97
-.94
-0.99
-0.99
.0.94
.0.99
Second, for crofs-configurationn ranking, we correlate the final values of eigen-metrics against final perplexity across the eight width configurations, for each architecture and activation variants (Table 3 right). Correlations remain strong across for across the configurations (rr) 2 0.85), with one ReLU and LeakyReLU, where pre-activation correlations weaken sharply while post-activation correlations strengthen. This inversion directly reflects how FFN nonlinearity overcompensate to break spectral inertial identified in Section 3.2, making the post-activation spectrum the more informative diagnostic in this regime. Thus, short preliminary runs with Ve metrics can rank architectural configurations without training each to convergence.
notable
8

---

Published as a conference paper at ICLR 2026
3.6 LAYERWISE DYNAMICS FOR POSITIONAL ENCODING: ROPE VS NoPE
RoPE prevents mid-to-deep spectral collapse, improving depth utilization
Figure 7 demonstrate that the NoPE's PR declines in the middle and deeper layers, indicating that representations collapse into narrow subspace and squander model depth. RoPE, on the other hand, sustains higher PR across the mid-to-deeper layers. improving the depth utilization. This ef- fect aligns with recent evidence that inter- mediate layers are disproportionately im- portant (de Llano et al.) 2026 Lad et al. 2025 Ikeda et al. [205 Skean al. 2025); which collapse under NoPE but remain ef- fective under RoPE. These improved spectral utilization in RoPE helps achieves lower evaluation perplexity (15.20 vs 16.78) than NoPE. Figure [3] in Appendix [33 shows the spectral entropy heatmaps reaffirming that RoPE prevents the mid-to-deep spectral collapse characteristic of NoPE.
Rotk jovd tit 16.0
Risl (5,200
Decirpaton sats
Figure 7: Layerwise participation ratio (PR) comparison (RoPE vs NoPE) in GPT-2 models trained from scratch on 26B token form openwebtext dataset with 512 con- text length for 100K steps. RoPE sustains higher PR in the middle and deeper layers, indicating better utiliza- tion of latent space and network's depth.
3.7 OPTIMIZER- -EEPNDEEN ROLE OF FFN NONLINEARITY: REPAIR VS REFINEMENT
To understand the optimizer-dependent role of FFN nonlinearity, we examine eigenspectrum dynamics under three LLM optimizers; AdamW, W, Muon, and Dion; and summarize the observations below.
PK Goins (PactiPra Adaed
$50 (Pact PPa] Adanity
100- Adanw
18- Adaad
2.0
-1
63
Sard Percer
s0-
[rT_pre
poct
20
0.6
255
2K
35
16
36
2X
3K
2.K
2K
36
4K
Yuinin Stop
Tuaning Stop
Traiing Stop
Toeery Stop
PR Gaina (Poutiire Muos
CEE (Post Proi - Maon
ECE- Muon
J5- Muse
0.0
030
-22-
3.2
20
acse
24
3
1K
3
0.4
14
24
3/c
23
14
24
3
Sop
Thainin Stp
Thalting Soup
TuilinSpp Stup
PR Gaino (PottPrel Den
0.0
tee (Pott Prel Dien
tgemabaccten Eary berdmmet
1.0
ent. Dion
0.5
000
10-
12
0130
20
1K
3
2/4
3K
14
2.
36
Tradeny Step
Traknng Stop
Wakling
Figure 8: Optimizer-dependentt FFN eigenspectrum dynamics in GPT2-350M. Rows show AdamW (top), Muon (middle), and Dion (bottom). Adam W has large early PR gains and high JS with relatively high EEE.post, indicating optimizer-induced pre-activation collapse followed by aggressive but incomplete nonlinear repair. Muon shows the smallest PR gains, lowest JS, and lowest EEE_post, and flatter post-spectra. Dion is intermediate and falls between these two regimes, improving over AdamW but not matching Muon's ' pre-/post-spectral behavior. The perplexity ordering (Muon > Dion >
AdamW) W) aligns with post-activation spectral flatness.
Muon minimizes nonlinearity burden by preserving eettvattio-compaaiee eigenspectrum Figure 8 shows that Muon maintains uniformly small (R(Psst/rre) gains and consistently low divergence throughout training. This combination indicates that Muon keeps the pre-activation FFN eigenspectra high-dimensional and near-isotrooic, a property also shown in a recent line of work (Wang et al., 2026; Vasudeva et al., 2026). so the FFN nonlinearity does not need to substantially restructure representations. Dion follows the same trend but less strongly: PR gains are moderate and JS is higher and more uniform across layers, suggesting broader activation-driven reshaping than Muon yet still far less mismatch than AdamW. Thus, geometric updates serve as spectral equalizers, preventing the pre-activation spectrum from drifting into regimes that demand large nonlinear correction.
9

---

Published as a conference paper at ICLR 2026
The early-layer spectral collapse in AdamW forces FEN nonlinearity into repair mode.
In contrast, AdamW exhibits very large PR(Post/Pre) gains concentrated in early layers, indicating optimizer- induced pre-activation collapse, en- ergy concentrating into a small set of dominant eigenmodes followed by strong nonlinear repair (Figure ( However, this repair does not translate into better utilization, as the PR post re- mains lower than Muon and Dion in early and intermediate layers despite their massive PR gains during training. Thus, under AdamW the activation expends capacity primarily to undo collapse rather than refine a healthy spectrum, consistent with AdamW's worse perplexity (see Table
1500
Miret Just Donitral 27.50
3 11250
756
Figure 9: Layerwise PR-pre (top) over training, and final PR_post per-layer (bottom) for Adam W, Muon, and Dion optimizer. Muon maintains the highest _R_pre across almost all layers, Dion is intermediate, and AdamW shows early- layer collapse. Moreover, Muon concentrates the largest effective dimensionality in middle FFNs.
Muon concentrates representational capacity where it matters: the middle FFNs. Figure isolates where effective dimensionality (PRguss) accumulates after training. Muon achieves the highest PRpstt in intermediate FFNs; whereas, Dion inflates PRpsst mainly in early FFNs without yielding the best perplexity; instead, the perplexity ordering follows mid FFNs post trend. This highlights the spectral mechanism for Muon's superiority- well-conditione latent-space usage in the mid-FFNs.
The optimizer- -eepnnde dynamics persist across scales and context lengths (Appendix L. and extend to Adafactor (Appendix L.2) and SGD on tontrrasformer MLP-Mixer (Appendix L.3pp Thus, core findings-nonline nonlinearity remjects variance and flattens spectra- are aptimizer-agnostie while remaining oprimizer-dependen in degree, supporting the view that optimizers induce representational biases which should be used as explicit sources of inductive biases (Pascanu et al. 2025).
4 RELATED WORK
Prior work leverages spectral signals of weights or representations understand model internals. RankMe (Garrido et al. 2023) and Diff-eRank (Wei et al. (202) use spectral-entrope Rank measures to predict downstream accuracy and quantify compression. Bao et al. (2024) established the relation between spectral concentration of QK weight matrix and attention localization, which Lee et al. (2025b) addressed using one-step belief-propagation refinement. Hu al.] (2025) showed that heavy-tail nature of weight ESD is biased by layer aspect ratio and propose fixed-aspeci sub-ESD averaging to debias. Zhang et al. (024) use IS divergence between Hessian spectra of parameter blocks in networks at initialization to suggest optimizer choice. Ruscio et al. (2025) used spectral gap and participation ratio on attention eigenspectra to reveal that attention sinks serve as geometric reference frames anchoring token representations, Poole et al. (2016) used Riemannian geometry with mean-field theory to demonstrate an arder-to-chaod phase transition governed by the nonlinearity's 's derivative in randomly -iittaiii networks. Cowsik et al. (2025) used Lyapunov exponents for signal propagation, treating token evolution as a particle system to predict trainability. Dong et al. (2021) used relative residual norms and path decomposition to demonstrate attention-induced rank collapses.
In contrast, Ner VE directly tracks FFN eigenspectrum dynamics, showing how nonlinearities redis- tribute variance in latent space, offering a diagnostic for architectural and optimizer choices.
5 LIMITATIONS AND CONCLUSION
While NERVE eigen-metrics analyze how FFNs organize variance in LLMs, they do not directly predict downstream task quality. Moreover, computing full eigendecompositions in large dimensions can be costly, often necessitating sampling or approximation (See Appendix G). Despite these constraints, our analysis shows that each metric contributes a distinct and complementary view of high-dimensional usage. Refer to Appendix M for further details limitations of Ne framework.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide the training details, including hyperparameter configurations in the very-beginning of
Section The other implementation details are provided Appendix A including the project page.
LLM USAGE STATEMENT
We utilized Large language models (LLMs) as rewriting tool, mainly for grammar correctness and stylistic refinement, to improve the quality of written text. We also used LLMs for refining the plots, figures, and tables. We did not use LLMS for any intellectual contribution and experimental design.
ACKNOWLEDGMENTS
This research was supported in part by the NSF CAREER Award #2340137 and gift award from Google. We thank the anonymous reviewers for their valuable comments and constructive feedback, which significantly improved the quality of this work.
REFERENCES
Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.0052 055995, 2025.
Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
Randall Balestriero, Romain Cosentino, and Sarath Shekkizhar. Characterizing large language model geometry helps solve toxicity detection and generation. In Forty-first International Conference on Machine Learning (ICML), 2024.
Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention networks localize when QK- eigenspectrum concentrates. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
Valentyn Boreiko, Zhiqi Bu, and Sheng Zha. Towards understanding of orthogonalization in muon. In High-dimensional Learning Dynamics 2025.
Jop Briët and Peter Harremoës. Properties of classical and quantum jensen-shannon divergence. Physical Review A-Atomic, Molecular, and Optical Physics, 2009.
Aditya Cowsik, Tamra Nebabu, Xiaoliang Qi, and Surya Ganguli. Geometric dynamics of signal propagation predict trainability of transformers. Physicul Review E, 2025.
Manlio De Domenico and Jacob Biamonte. Spectral entropies
cnormation-thooretie tools for
complex network comparison. Physical Review X, 2016.
Enrique Queipo de Llano, Alvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael M. Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. Attention sinks and compression valleys in LLMs are two sides of the same coin. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.
Harm de Vries. In the long (context) run: It's not the quadratic attention; it's the lack of long pre-training data, hhttps://www httsps::/ww.aaedeerrrr.....
2023.
Yihe Dong, Jean- can-tBatiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning (ICMI), 2021. 2021.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, 2017.
11

---

Published as a conference paper at ICLR 2026
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. RankMe: Assessing the downstream performance of pretrained self-super representations by their rank. In International conference machine learning (ICML), 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Behrooz Ghorbani, Shankar Krishman, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density, International Conference on Machine Learning (ICML). 2019.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 2011.
Yu Hu and Haim Sompolinsky. The spectrum of covariance matrices of randomly connected recurrent neuronal networks with linear dynamics. PLoS computational biology, 2022.
Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov, and Yaoqing Yang. Eigenspectrum analysis of neural networks without aspect ratio bias. In Forty-second International Conference on Machine Learning (ICML), 2025.
Qiyao Huang, Yingyue Zhang, Zhihong Zhang, and Edwin Hancock. ESSEN: Improving evolution
state estimation for temporal networks using von neumann entropy. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
HuggingFace. chapter7/6
Codeparrot.
Wataru Ikeda, Kazuki Yano, Ryosuke Takahashi, Jaesung Lee, KeigoShibata, and Jun Suzuki. Layerwise importance analysis of feed-forward networks in transformer-based language models. In Second Conference on Language Modeling (COLM), 2025.
Nandan Kumar Jha and Brandon Reagen. ReLU's revival: On the entropic overload in normalization- free large language models. NeurlPS Workshop on Atributing Model Behavior Scale, 2024.
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. URL https://klllejjoddum github. io/posts/muon, 2024.
Federico Arangath Joseph, Jerome Sieber, Melanie Zeilinger, and Carmen Amo Alonso. Lambda- skip connections: the architectural component that prevents rank collapse. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Diederik P Kingma. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui, Analyzing deed-forward blocks in transformers through the lens of attention map. In The Twelfth International Conference Learning Representations (ICLR), 2024.
Vedang Lad, Jin Hwa Lee, Wes Gurnee, and Max Tegmark. Remarkable robustness of LLMs: Stages of inference? In The Thirty-ninth Annual Conference Neural Information Processing Systems (NeurIPS), 2025.
Coreelis Lanczos. An iteration method for the solution of the eigenvalue problem of linear differen- tial and integral operators. Journal of research of the National Bureau of Standards, 1950.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. In International conference on machine learning (ICML), 2025a.
Nakyung Lee, Yeongoon Kim, Minhae Oh, Suhwan Kim, Jin Woo Koo, Hyewon Jo, and Jungwoo Lee. Mitigating attention localization in small scale: Self-attention refinement via ste-step belief propagation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (FMNLP), 2025b.
12

---

Published as a conference paper at ICLR 2026
Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the power of deeper layers by combining pre-LN and Post-LN. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde. and Anna Rumshisky. ReloRA: High- rank training through low-rank updates. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 1991.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for Ilm training, arXiv preprint arXiv:2502.16982. 2025.
Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In Advances in neural information processing systems, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized transformer with representation learning on the hypersphere. In The Thirleenth International Conference Learning Representations (ICLR), 2025.
Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference Machine Learning (ICML), 2019.
Anna Marbut, Katy McKinney-Bock, and Travis Wheeler. Reliable measures of spread in high dimensional latent spaces. In International Conference on Machine Learning (ICML), 2023.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Jayakrishnan Nair, Adam Wierman, and Bert Zwart. The fundamentals of heavy tails: Properties, emergence, and estimation, volume 53. Cambridge University Press, 2022.
Frank Nielsen and Richard Nock. Total jensen divergences: Definition, properties and clustering. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015.
Alexander V Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities. In The 38th Annual Conference on Neural Information Processing SystemssNeur//PS), 2024.
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Razvan Pascanu, Clare Lyle, Ionut-Vlla Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, and James Martens. Optimizers qualitatively alter solutions and we should leverage this. arXiv preprint arXiv:2507. 12224, 2025.
Guilherme Penedo, Hynek Kydlíčck, Anton Lozhkov Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity deep neural networks through transient chaos. Advances in neural information processing systems, 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research (JMLR), 2020.
13

---

Published as a conference paper at ICLR 2026
Valeria Ruscio, Umberto Nanni, and Fabrizio Silvestri. What are you sinking? geometric approach on attention sink. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025.
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization accelerate training of deep neural networks. In Advances in neural information processing systems, 2016.
Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Practical efficiency of muon for pretraining, arXiv preprint arXivv2505.00222, 2025.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning (ICML), 2018.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Forty-second International Conference on Machine Learming (ICML), 2025.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. In Neurocomputing. 2024.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MIp- mip-mixer An all-mip architecture for vision. In Advances in neural information processing systems (NeurIPS), 2021.
Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thrampoulidis. How muon's 's spectral design benefits generalization: A study on imbalanced data. In The Fourteenth International Conference on Learning Representations (ICIR), 2026.
Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang. Zhuoran Yang, Mingyi Hong, and Vincent Y. F. Tan. Muon outperforms adam in tail-end associative memory learning. The Fourteenth International Conference on Learning Representations (ICLR), 2026.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. International conference on machine learning (ICML), 2020.
Lai Wei, Zhiquan Tan, Chenghai Li, Jindong Wang, and Weiran Huang. Diff-erank: A novel rank- based metric for evaluating large language models. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of atten- tion masks and layernorm in transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhiquan Luo. Why trans- formers need adam: A hessian perspective. In Advances in neural information processing systems, 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-effictiee LLM training by gradient low-rank projection. In 5th ICLR Workshop on practical for limited/low resource settings (PMLALRS), 2024.
14

---

Published as a conference paper at ICLR 2026
Implementation Details and Methodological Considerations
16
A. Framework Overview
16
A.2
Implementation Details for Computing Covariance Matrices
16
Eigenspectrum Metrics: Design Principles and Diagnostie Guide
17
B.1
Metries Design and Justifications
17
B.2
Metric Relationships and Failure Modes
18
B.3
Decomposing Nonlinearity- Induced Eigenspectrum Restructuring
19
Eigenspectral Signature
20
Spectral Signature LayerNorm Positioning: PreLN, MixLN, and PostLN
20
Spectral Signature in Larger LLaMA Models
21
Spectral Signature of Positional Encoding: NoPE RoPE
23
D L LayerNorm Positioning and FFN Width Sweep
23
Spectral Signature Across FEN Width Sweeps Baseline Models
24
Spectral Signature Across FEN Width Sweeps in Norm-free
25
Token Sub-Sampling and Low-Rank Approximation
26
Eigen-metric Fidelity and Diagnostic Validity Under Approximations
26
G.2 Spectral Signature of Sampling and Low Rank Approximation
27
H Computational and Memory Efficiency of NerVE
28
H.1
Computational Complexity and Memory Overheads of NerVE
28
Memory-efficient Eigenspectrum Analysis
28
Eigenspectrum Dynamics in TranTorantrremr Architecture
30
J
Token-Position Effects on FFN Eigenspectrum
32
FFN Figenspectrum Dynamics: LayerNorm vs RMSNorm
34
K.1
LayerNorm vs RMSNorm Ablation GPT-2
34
LayerNorm vs RMSNomm Ablation on MLP-Mixer
35
Optimizer-Dependdent FEN Eigenspectrum Dynamics
36
AdamW vs Muon vs Dion
36
2 AdamW Adafactor
................................
37
AdamW vs SGD
39
M Jimitations NerVE Framework
Discussion: Why hep-hensiness Over Tail-heas
40
15

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS AND METHODOLOGICAL CONSIDERATIONS
A.1
FRAMEWORK OVERVIEW
Eigen-metrics
Activation Collection
Covariance Matrix Computation
Eigendecomposition
Spectral Entropy (BE): Variance unitormity
PreAct (reeaadivation)
Ee Ao
SR Au
pre
Tw
FFN Layer t
Unbiassd Sample Covariance)
Participation Ratio (PR):
Pad
Effecctive dimarsionality
W up
(E
A
(-)
x B) mateix
[B, to
(X (X (V 1)
(A) Ao Ao
Sosted eigruvalues
Eiganvalion Early Enrichmant (EEE): Yop-haaviniss
Eiso Af
W dowwn
Ev = Av
puet
Jensen SShanon Divergence [25), Distributional Shilts
Per layer (f 1..
PostAct (Post-activation
(Unblased Semple Covariance)
J5 ((A7))))) (2))))))
Figure 10: For each FFN layer l, we first collect pre-activation (after Wap, before σ) and post- activation (after before Waown). Then, tokens are flattened into X E (BBxS)xD sample matrix, where the global batch size, S the sequence (context) length, and D the FFN hidden dimension. Finally, we compute the unbiased sample co- variance of mean-centered activations to get the eigen- values { and sorted them into descending order. Three eigen-metrics metrics are computed on each (pre-/post-) eigenspectrum: spectral entropy (SE) for dispersion, participation ratio (PR) for effective dimensionality. and eigenvalue early enrichment (EEE) for top-heaviness; while Sensen-Shannor Divergence (IS) quantifies the distributional shift between the pre- and post-activation spectra, captur- ing the geometric restructuring performed by the nonlinearity.
A.2 IMPLEMENTATION DETAILS FOR COMPUTING COVARIANCE MATRICES
During training, activations are collected through registered Py PyTorch hooks. For pre-activation, we forward hooks the output of the up-prop pp-projection layer. For post-activation, we use fre-forward hooks on the down-projection layer to capture inputs before they enter the down-projection.
Token aggregation. Within each logging step, flatten all tokens across the batch and sequence dimensions into a single matrix E R x where N = x S, treating each token embedding as an independent sample. We compute the mean-centered version X = X
before forming
== X X. All eigenvalues are sorted in descending order (Λι 2 ... 2 Ap > 0) before any metric computation-rreqireee for computing EEE since it depends on the cumulative sum from largest smallest (Figure TOP. All results in this paper the full-batch covariance (no token .ub-sampling). The effect of sub-sampling and low-rank approximation is discussed in Appendix G
Paired measurement. Pre- and post-activation covariance matrices are computed on the identical set of N tokens within each layer, ensuring that nonlinear transformations, including JS divergence which measures the geometric restructuring applied by the nonlinearity, are computed on the same input population rather than comparing statistics from different samples. With full-batch computation this pairing implicit; however, when sabmsanl is employed (Appendix G), the same token subset must be used for both measurement points.
Our implementation ensures numerical stability and efficiency through:
1. Precision control: Converting all tensors to float32 to avoid precision issues.
2. Numerical stability: Adding a small epsilon values (e = le - 12) to prevent division by zero in eigenvalue normalization and entropy computation.
3. Memory efficiency: Processing layers sequentially and discarding intermediate covariance matrices after cigendecomposition, so that peak GPU memory is bounded by single layer's 's covariance pair (2D2 floats) rather than accumulating across all L layers (see Appendix H.2). For distributed training, activations are gathered from all GPUs to rank 0 before covariance computation.
16

---

Published as a conference paper at ICLR 2026
4. Specialized computation: Using torch. linalg....... eigvalsh for symmetric positive semi- definite covariance matrices, which is both faster and more numerically stable.
This methodological rigor ensures that observed patterns in eigenvalue metrics reflect intrinsic proper- ties of the network architecture rather than measurement artifacts. We provide the full implementation of our framework project /ttpps:/nnareee-iigeessseeee rum. github..hh.....
B EIGENSPECTRUM METRICS: DESIGN PRINCIPLES AND DIAGNOSTIC GUIDE
Table summarizes the four eigen-metrics, their inputs, bounds, and what each captures. The remainder of this appendix provides justifications for NERVE eigen-metrics set (B.1). examines inter-metric relationships and failure modes with diagnostic reference (B.2), and decomposes the geometric work of the nonlinearity into interpretable axes B.3).
Table 4: Summary of the four NERVE eigen-metrics. Input indicates whether the metric operates on raw (A) or normalized ()) cigenvalues, and range shows their (bounded) output values, SE, PR, and EEE characterize single spectrum; while JS quantifies the divergence between the pre- and cost-activation spectral shape. All four metrics are invariant to uniform scaling of eigenvalues.
Metric
Input
Range
Sensitive to
Captures
SE
À (normalized)
I0, D)
Full spectrum (especially mid-to-tail)
Variance uniformity (dispersion)
PR
À (raw)
[1 D]
Large cigenvalues (suppresses small cigenvalues)
Effective dimensionality
EEE
À (raw)
(0, 1)
Top eigenvalues (front-loaded concentration)
Top-heaviness (anisotropy)
S
Apm Ape
In.)]
Shape difference between (pre-/post-
Geometric restructuring
B.1
METRICS DESIGN AND JUSTIFICATIONS
The metrics should jointly capture distinct geometric aspects of high-dimensional latent space-ne single metric suffices. Thus, our selection of Sigen-metrice (SE, PR, EEE, JS) are based on the following desiderata for diagnostic framework applied to high-dimensional eigenspectra:
(i) Coverage. Different eigenvalue distributions can share the same SE and PR. yet allocate variance very differently across the eigenspectrum. For instance, a spectrum with moderate variance spread uniformly across eigenmods, and spectrum with the same total spread but concentrated in few dominant directions followed by a broad law-variance tail could yield identical SE and PR values. EEE resolves this ambiguity by accounting cumulative variance of spectra, and able to differentiate the spectra having different fractions of latent-space utilization (Marbut et al.) (023). (ii) Complementary sensitivity. The metries in Table 4 emphasize different regions of the eigen- spectrum. SE, due to the logarithmic weighting to the normalized eigenvalues (Λ,), relatively less sensitive to dominani modes and more responsive to mid-to-tail spectrum. In contrast, PR downweights small eigenvalues and driven primarily by dominant modes through the quadratic term Ex X²) As result, shifts confined to the tail can change SE while leaving PR nearly unchanged, whereas redistribution among leading cigenvalues can move PR strongly with only minor changes in SE. Since EEE summarizes top-her hea-heevinsssss of cumulative variance, a handful of large eigenvalues drives EEE close to 1, even if the rest are moderate. These complementary sensitivities ensure that changes in any region of the spectrum
reflected by at least one metric. (iii) Boundedness. All four metrics have closed-fome bounds (Table enabling meaningful com- parisons across models with different dimensionality D. training checkpoints, and architectural configurations. Unbounded metrics could make woos-confiiguaaton comparison unreliable. (iv) Scale invariance. All four metries are invariant uniform scaling of the eigenvalues (x)) This is crucial in practice, as the magnitude of the covariance matrix may fluctuate due to various factors such as batch statistics, or learning rate schedules. Scale invariance ensures the metrics remain focused on the shape of the distribution, which truly governs the directionality in the latent space. Moreover, each metric accounts for the entire eigenvalue distribution, unlike simple measures such as the eigenvalue ratio (largest-to-smalll) smalless)) which are sensitive only to extremes. (v) Pre-post restructuring. While SE, PR, and EEE characterize the intrinsic properties of a single distribution, JS quantifies the information-theon Nock 2013). Thus, JS is needed to quantify conlinearty-inducee geometric restructuring, Large JS indicate significant variance redistribution across principal components, potentially creating
distance between two distributions (Nielsen &
17

---

Published as a conference paper at ICLR 2026
new directions of specialization eliminating others. Conversely, small JS highlights FFNs which rescale existing directions without fundamentally altering the latent space geometry.
Why Jensen-Shannon divergence over KL divergence The main benefits of JS divergence (Eq. 6 compared to a simpler KL divergence, for eigenspectrum analysis are:
1. Symmetry: JS( Ppre Poes) = JS(P post || Pire) enables unbiased comparison of pre-activation and post-activation eigenspectrum (Briët & Harremoës 2009), whereas KL is asymmetrie and would prioritize one distribution as the reference.
2. Boundedness and interpretability: 0 E pree || Ppma) S In(2), facilitating standardized com- parisons across layers of different dimensions, and bounded scale makes JS values interpretable under distributional shift, unlike KL which could yields unbounded values isolation.
3. Numerical stability: IS offers superior numerical stability when analyzing eigenspectra with near-zero eigenvalues, which are common in neural network representations.
B.2 METRIC RELATIONSHIPS AND FAILURE MODES
While the four metrics are selected to capture distinct spectral properties, they are not statistically independent. In many training regimes, SE and PR move together because both respond to overall spectral flattening. Understanding when the metrics agree, when they diverge, and when individual metrics can mislead is crucial for reliable diagnostics.
When metrics agree. During standard training of well-configured models (e.g., GPT-2 PreLN with GELU), SE and PR typically co-increase while EEE decreases, highlighting nonlinearity-induce rank inflation through variance reinjection into previously inactive directions. JS decreases across depth, especially in deeper layers, indicating lesser geometrie restructuring in deeper FFNs. When all four metrics align in this pattern, the diagnostic is straightforward, the FFN latent space is being utilized effectively.
When metrics diverge, and why it matters? Divergence between metrics signals that the spectral change localized to a particular part of the eigenspectrum, which diagnostically informative:
(i) SE rises but PR is stable: Variance is redistributing in the mid-to-tail eigenvalues without affecting the top eigenvalues that dominate PR. This happens when nonlincarity reshapes the spectral tail without shifting the dominant directions.
(ii) PR rises but EEE also rises: This paradoxical pattern occurs when a few new directions gain substantial variance while the rest remain near zero. PR increases because there are more active directions, but EEE increases because the new directions are still much larger than the inactive ones. This signals the trianing stages where network is allocating capacity to new features without distributing variance uniformly.
(iii) SE and PR rise but JS is zero: Both pre- and post-activation spectra are flattening, but the nonlinearity is not reshaping the spectrum. This acts as a linear scaler which suggests the FFN may not be fully leveraging nonlinear capacity.
When individual metrics mislead. In the following scenarios, relying on single eigen-metric could lead to misinterpretation of underlying learning dynamics.
(i) PR without EEE can mislead on performance: Hyperspherical normalization when used in FFN weights of norm-free model, it achieves the highest PR_post (exceeding 600) but worse perplexity than Spectral normalization (Section 3.3). as their AEEE flattening does not persist, EEE post remains high across depth.
(ii) PR in bottleneck regimes: At FFN width D = 1d (no dimensional expansion), PR is constrained to a narrow range regardless of spectral quality. PR values in this regime should not be compared directly wider configurations without normalization by D.
(iii) SE blind to absolute scale: Since SE operates on normalized eigenvalues (Å; = >r,TT((T) is insensitive to the to absolute scale, and it should be used to locate the spectral bottlenecks (layers with very low SE. See Figure IT). Nonetheless, two FFNs with identical spectral shapes but substantially different total variance (Tr(E)) would result in identical SE values. Thus, when total variance matters, for instance comparing different layers within a network (activation magnitudes vary substantially across depth), PR should be preferred over SE.
18

---

Published as a conference paper at ICLR 2026
(iv) EEE saturation: In highly anisotropic spectra (e.g., single dominant eigenvalue with near-zero rest), EEE saturates near its upper bound and becomes insensitive to further concentration. In this regime, PR (which continues to decrease toward is the more informative metric.
Diagnostic references. Table summarizes the joint metric signatures most frequently observed across our experiments, along with their interpretation. To avoid misinterpretation, it is important to report all four metrics jointly.
Table 5: Diagnostic reference for joint eigen-metris signatures. Each row describes a commonly observed pattern, its interpretation, and where the paper it is empirically demonstrated.
Joint Signatures
Interpretation
References
SEt PR EEEJ
Healthy spectral flattening (Rank inflation)
Standard training ( 3.1
SEI PR EEET
Spectral collapse
AdamW pre-activation
3.7
JSL
depth
Decreasing nonlinear restructuring
Hcalthy deep layers 3.1
SEt PRtt ISI across depth
Nonlinearity compensating for pathology
Norm-free ReLU models Spectral norm. in FFNs
EEE 11 JSt across depth
Sustained spectral flattening
Pre-act stable, post-act shifts
Repair regime
3.7 & Appendix Muon & Appendix
AdamW 3.3.7
Pre-act shifts, post-act stable
Refinement regime
B.3 DECOMPOSING NONLINEARITY- EIGENSPECTRUN RESTRUCTURING
In addition to JS metric, we use two derived cross-activation quantities: the participation ratio gain PR(Post/Pre) and the EEE difference AFEE EEF_post EEF_pre. These quantities decompose what the nonlinearity does to the eigenspectrum:
JS measures the total magnitude of spectral change, without indicating direction.
PR(Post/Pre) measures dimensional expansion, whether nonlinearity activated new directions.
ΣEEE measures top-heaviness reduction, whether the nonlinearity suppressed leading eigen- value dominance. More negative ΛEEE indicates stronger flattening.
These metrics are complementary, not redundant: gence with little PR gain, whereas tail spreading across many small modes can yield large PR gains with typically more moderate increase in JS divergence.
dominant-mode redistributions can raise JS diver-
Effort does not imply outcome. Large PR gain does not imply high PR Under AdamW (Section 3.7 nonlinearities yield the largest PR gains and highest IS, yet the lowest PR (vs. Muon/Dion), In contrast, Muon attains the highest PRpos with the smallest PR gains and lowest JS. The difference that AdamW collapses the pre-activation spectrum, so the nonlinearity expends capacity on repuir (large corrective gains) that only recover mediocre state, whereas Muon preserves a well-conditioned pre-spectrum and requires only modest refinement.
Sustained flattening matters. High PRpsst alone is not sufficient: HNorm (Section 3.3 attains larger PRpsst yet underperforms, whereas SNorm maintains consistently lower EEEpos throughout training (sustained flattening) and achieves best perplexity among normaaizioiiiiiionnn methods.
Characteristic patterns. Table 6 summarizes four diagnostically distinct combinations observed across experiments, each reflectigg qualitatively different regime of nonlinear function.
Table 6: Four regimes of nonlinear geometric work, distinguished by the joint signature of JS, PR(Post/Pre), and AEEE. JS alone cannot distinguish beneficial restructuring from compensatory repair; PR gain alone cannot distinguish beneficial restructuring from expansion without equalization.
JS
PR gain
ABEE
Interpretation
Beneficial Restructuring
High
High
Strongly negative
Active expansion and flattening (norm-frce ReI.U,
3.2
Compensatory Repair
High
High
Weakly negative
Effort wasted undoing PreAct spectral collapse (AdamW,
3.7
Expansion without equalization
Moderate
High
Near or positive
New directions but top-heaviness persists (HNorm,
3.3
Spectral Inertia
Moderate
Ro
Nonlinearity near-identity (norm-free GELU,
3.2
19

---

Published as a conference paper at ICLR 2026
C EIGENSPECTRAL SIGNATURE
C.I SPECTRAL SIGNATURE OF LAYERNORM POSITIONING: PRELN, MIXLN, AND POSTLN
Figure 11 illustrates the post-activation spectral signatures (SE post and PR post) for three LayerNorm placements-Prree PreI.,, PostLN, and MixLN across GPT2-125M 125M, LLaMA-70M, and L.LMA-1 These signatures highlight the extent of FFN latent space utilization across layers. Within each model family, the ranking of evaluation perplexities is corroborated by their spectral signatures: models with lower perplexity exhibit higher utilization.
Layer
Layee
GPT2-125M
10
32K
61K
5K
10C
24k
32K
36C
24K
32K
41K
Taining Step
Training Step
Training Step
PreLN (Eval PPL 2.714)
PosttN (Eval PPL = 2.830)
MixLN (Eval PPL = 2.808)
1250
Layer
Lagger
1032
750
0K
24
32K
61K
6K
6K
164
24k
JBC
24k
3Gk
41K
Triniin
Stop
Stop
PreLN (Eval PPL = 34.2)
PostEN (Eval PPLL 33.6)
MIXLN (Eval PPL 33.9)
Spectrrl
LLaMA-700 70M
6K
10k
10c
74
Triinin
Training Step
Training Step
34.2)
PostLN 33.6)
33.9)
Porticppaion Pntiooation Rutto
202
4<
36
6K
3k
8K
Tairnng Step
Training
Training Step
26.4)
PostLN (Eval 26.7)
Spectal Emrrpy
LLaMA- 130M
Step
PreLN (Eval PPL 26.4)
PostLIN (Eval 26.7)
MixLN (Eval
282
152
Parfi Goul foation
EK
12x
1.C
20C
8X
12k
16C
2DK
0K
4k
8K
12C
3.0K
20k
Training Stop
Training Stop
Traninng Stop
Figure 11: Eigenspectral impact of LayerNorm placement (PreLN, PostLN, MixLN) in GPT-2 and
LLaMA variants (70M, 130M). The spectral signatures are shown through post-activation spectral entropy (f) and participation ratio (t), and model's 's perplexity shown on top of each plots. GPT-2 models trained on CodeParrot and LLaMA variants on C4.
Performance follows the order PreLN (PPL = 2.714) MixLN (2.808) > PostLN (2.830). The spectral signatures follow this ranking: PreLN exhibits superior spectral entropy and participation ratio trend across layers, indicating PostLN shows the most constrained spectral characteristics. In particular, L7 in PostLN and MixLN show vow utilization compared to the PreLN configuration.
GPT2-125M:
effective FFN latent space utilization, while
20

---

Published as a conference paper at ICLR 2026
LLaMA-70M: PostLN achieves the lowest perplexity =PPL=33.6), followed by MixLN (33.9), while PreLN performs worst (34.2). The eigenspectral analysis shows that PreLN exhibits substantially lower spectral entropy in deeper layers (L7-L8) compared to PostLN and MixLN. In contrast, PostLN demonstrates superior spectral characteristics with higher participation ratios in deeper layers compared to MixLN, consistent with its lower perplexity.
LLaMA-130M: PreLN yields the best performance with perplexity of 26.4, followed closely by PosILN (26.7) and MixLN (26.8). While PostLN exhibits stronger spectral entropy and participation ratio in the mid-depth layers (L6-L9), PreLN consistently outperforms across the remaining layers, particularly L10 where PostLN deteriorates, undermining overall benefits. In contrast, MixLN shows the worse spectral profile, leads to highest perplexity.
C.2 SPECTRAL SIGNATURE IN LARGER LLAMA MODELS
Figure 12 shows the post-activation eigen-spectral signatures (SE_post and PR-post) for LayerNorm placements, PreLN and MixLN, in 250M and LLaMA-1.33 Note that PostLN is excluded since it becomes unstable at these larger scales (Li et al. 2025). These spectral signatures provide a quantitative assessment for latent space utilization in each FFNs.
LLaMA-250M: MixLN outperforms the PreLN configuration by a margin of 0.3 in terms of PPL improvement (24.2 vs 24.5). The spectral signature of PreLN exhibits consistent lower SE in layer 7 to 16, and lower PR in mid-depth regions (Figure 12 Whereas, MixLN avoid this mid-network spectral collapse pattern by confining the lower SE bands in early layers while maintaining higher SE and PR through the mid-depth layers, effectively placing most of the usable capacity where the model does the bulk of its computation. This redistribution of spectral entropy and participation ratio across depth is consistent with the lower perplexity achieved by the MixI.N model.
LLaMA-1.3B: At 1.3B scale, MixLN's 's strategic LayerNorm placement unable to facilitate favorable capacity distribution, and training collapses after 7K steps. Whereas, PreLN maintain a stable spectral entropy across layers and achieves notably higher participation ratio the deeper layers throughout training. This divergence in spectral behavior aligns with the final performance gap, as MixLN's perplexity explodes to 1457.1 compared to 21.2 for PreLN.
These results suggest that LayerNorm placement significantly influences how effectively the FFN utilizes its latent space. Pre-LN configurations appear to better preserve and amplify feature diversity, especially in deeper layers, thereby enabling nigher-dimensional latent representations. The observed gains in the MixLN setup indicate that even partial use of PreLN can compensate for the limitations of PostLN, offering potential strategy for balancing stability and expressivity.
21

---

Published as a conference paper at ICLR 2026
PreLN (Eval PPL 24.5)
MixLN (Eval 24.2)
Spectra Entropy
Layer
Layer
LLaMA-250M
0K
4K
8K
12K
16K
20K
0K
4K
12K
16K
20K
Training Step
Training Step
PreLN (Eval PPL 24.5)
MixLN (Eval PPL 24.2)
140
120
Participation Ratio
100
Layer
Layer
20
12K
16K
Training Step
Training Step
PreLN (Eval PPL = 21.2)
MixLN (Eval PPL 1457.1)
Spectra Entropy
Layer
Layer
LLaMA-1.3B
0K
16K
20K
12K
16K
20K
Training Step
Training
PreLN (Eval PPL 21.2)
MixLN (Eval PPL 1457.1)
2000
Participation Ratio
15000
Layer
Layer
1000
500
0K
4K
8K
12K
16K
20K
0K
4K
BK
12K
16K
20K
Training Step
Training Step
Figure 12: Eigenspectral impact of LayerNorm placement (PreLN vs MixLN) in LLaMA-250 and LLaMA-1.3B models, trained from scratch on C4 dataset for 20K iterations with 256 context length. The spectral signatures shown through SE post (1) and PR post (f), and model's perplexity is shown on top of each plots. MixLN variant of LLaMA-1 -1111B destabilized after 7K iterations.
22

---

Published as a conference paper at ICLR 2026
C.3 SPECTRAL SIGNATURE OF POSITIONAL ENCODING: NOPE VS ROPE
Figure 13 shows the spectral signature of rotatory positional encoding (RoPE), in contrast with no positional encoding (NoPE). In partieular, the layerwise spectral entropy and EEE values of post-activation eigenspectrun are shown.
NoPE (SE)
RoPE (SE)
7.5
7.0
3
6.0
7-
5.5
9-
10-
11
11
20K
40K
60K
80K
100K
0K
20K
40K
60K
80K
100K
Training Step
Training Step
NoPE (EEE)
RoPE (EEE)
0-
0-
1-
1-
0.75
0.70
4-
4-
5
7.
0.60
8-
8-
11
11
0.45
20K
40K
60K
adK
100K
0K
20K
40K
60K
BOK
100K
Training Step
Training Step
Figure 13: Spectral Signature of Positional Encoding (RoPE vs NoPE) in GPT-2 models trained with 512 context size on 26B token form openwebtext dataset
D LAYERNORM POSITIONING AND FFN WIDTH SWEEP
reports the normalized PR. post (PR/D) to compare the efficiency of width utilization across LayerNorm configurations. Table ( complements this with absolute PRpos values, revealing the scale of the gap that the normalized view mask. At D 6144, PreLN sustains PR As 1822 effective dimensions, compared to 71 for PostLN, a 25x difference in absolute latent capacity that Figure can understate. MixLN remains closer to PostLN throughout the width sweep, fluctuating between PRpost R= 137-278 without scaling proportionally with D, suggesting that neither MixLN nor PostLN converts additional width into usable dimensions as effectively as PreLN.
Section 3.4
Table 7: PR_post (median + MAD across 12 layers) at final training checkpoint for different LayerNorm placements and FFN widths in GPT-2 (125M).
Method D=768
D=1536
D=2048
D=3072
D=3840
D=4608
D=5376
D=6144
PreLN
292:122
5731149
8314238
10024:35
12751333
1562-429
1822-370 1721-224
MixLN
137+-111
184
2474-233
274
278-1234
159+
251+:201
233 187
PosILN
95:165
142::104
151::147
240:1231
117::110
91:883
94:184
71:662
23

---

Published as a conference paper at ICLR 2026
E SPECTRAL SIGNATURE ACROSS FFN WIDTH SWEEPS BASELINE MODELS
Bw3dd pre.r 0.987 (.astt.0..81)
D-2d prea 6.99,, rjsstt (.838)
Dw3dd Ir. pro5-..55) y p.oote-0.857)
Baddir. pre.d.192 (.t.1.22...sst... poste
Dobluation Tom
Spon cenres
Deluton
Speci
jeee
10k
25k
36k
40K
10K
20K
40K
20k
304
40%
24k
30k
40C
0-6d (c.0re--0.992. r.90tt-4.072))
0-7d Ir. www.pre-...900 2.350 p.pptt-0..000 3.855)
pr 2rt.
ar
5C
10K
20K
31K
40K
10K
206
4GK
20k
40K
0
20K
30k
40C
400
200-
620
Tutts patienattttttt
10000
Porte patee nagets
1000-
200-
#20
200-
100
200
0-
rike Stop
Traiiig Step
Voliing Step
Traling
Dedid 2.55t--..91)
D-ddd tr.pre---997 9,972, 0,955
0-76 pw..p---777 0.530)
D-itd pree 0.973, rasst 6.932)
2000
busiitonn abon Buto
20000
pategutt Ruds
1500
1000
2000
1000
Dalatov
Flypa
500
FR,
Sralicnn
Crall
10K
20K
30k
23k
10K
20K
10K
45K
10K
20K
33K
4bk
Thankng Step
Takking top
Tulnin Stp
Thaning
Figure 14: Eigen metries in GPT-2 (GELU, D=ld to &d) with Pearson eval loss (Tpre postt
B-140 pre- 0.88S, p.c1--6.06)
B-2d Or prew 0,96,, past 6.977)
Duadd Ir 510--5.952, po02--0.11))
D.odir pre.2 pact.-..334)
Spotss Choog
Spactnr
64 poet
46
51.0 pect
10k
2/K
34K
40K
10K
205
30K
4
16
20k
30x
46K
0
1/c
2/K
30K
40C
Tasto Stop
Thiiing Strp
D- pree 0.95, postt-0.521)
D-edd crpere 933, (asst=-0.511)
0474 wtw.re---..... 0,552 1.pott-0, 0,335)
wweeir..re-.0.01 p.pott 5,511
Spo01
Spucr
5t
M jest
Sc. post
jast
St. 10.00
10K
20K
30K
40K
10K
20k
30K
40K
13k
20k
30x
40c
15k
23K
30K
40C
300-
1200
660
200-
Dutt sutten Iradisatonn
400
300-
200
200
RR.5%
(R, pm
0-
43K
20K
Traiki
0.81))
1000
Periouate ruals
1500
Partoouata Well
1100-
3000
Cookiittoon
500-
Plyye
11.5%
Flye
15k.
42K
40K
Figure 15: Eigen metrics in GPT-2 (ReLU, D=ld to 8d) with Pearson r to eval loss (Fpre: Tpost)
24

---

Published as a conference paper at ICLR 2026
F SPECTRAL SIGNATURE ACROSS FFN WIDTH SWEEPS IN NORM-FREE
Croups
10C
45K
20k
40k
25K
Tranting
Sgenn SeoWe
Sisp
0:14 Drpree 204, (.701)
Dv:2d fr.ane- 0.332, r.pest--o.77 0.7330
0-34d (r.are 0.777, rop 0.876)
0v:44 fr. (r.pren-5.400 1946-9 r.post 00.870)
Loco
102
#00
600
Purt ganteettee
200-
2.0
200
280
post
post
10K
zbx
40k
15K
20C
40K
15k
20K
30K
4bK
10c
25K
30K
40k
Trakiin Step
(r.ppee--..00.. (_postt-0.877) 0.073)
D-6d fr.3re--0.300 r.peste-00000 0.837)
D-7d (17errrrr---.400 w.4.4...0stt000000 0.00
p-0dd Ir. 1r.00=-8.377 8.postt-..2001
1000
rortbbutia Rudts
1000
600-
800
0.0-
250
FR.0%
FA,E%
posk
F1.2at
Lane
Lete
10K
20K
30k
40k
1.k
20K
30K
40k
15c
20K
30K
40K
10k
2.k
30K
40K
Tunni
Thaniin
Thaiking Stop
Thinnin Stop
Figure 16: SE and PR Normalization-free GPT-2 (GELU, D=1d to 8d) with Pearson correlation to eval loss (fpres T post)
Deldd Di pre.-0.256, C. p.45)-
Diidd E. prw.d-a00, post--0.814)
D- O.e. (0.64-
Di:68 0.56. po10t-6..200)
52, pest
St peat
St Su
gool
10K
28k
30K
42K
10K
23K
30K
40x
10K
23k
30k
40
10K
20k
50k
40K
peo.0.033,
252.1
Sqetra CunsT
SONEE CrENe
jer
joee
20K
25K
10K
Dn1e 1rraae00775 ppsste....00 9081
Dw2id traee-rrr0 L.watt 6.8741
De 000.0.0000 764, rapatt 6.691
Drikd www.re.eeee............. 740,, rppate 7.72)
300
600
%0
Nura patue tare
600
000
Bork petee
16
000
300
200
PRL post
AL. pos
AA. posr
16k
Z0K
30k
40x
1/k
20c
31k
46x
16k
20x
35k
40x
20x
38k
40x
Tranking Sis
Traning
Traking
Traning Step
Buiiddd pre-taiie, partu 6.843)
1000
Duill tr pmo- pasta 6.869)
D- pro-0.8k, pasti 6.861)
1000
Duite prewiasst, pasts 6.981)
500
400
500
7
71, Pl
200
pos
11.pok.
20k
33k
40k
16K
20K
40K
20K
40K
Figure 17: SE and PR in Normalization-free GPT-2 (ReLU, D=ld to 8d) with Pearson correlation eval loss (Tpres postt)
25

---

Published as a conference paper at ICLR 2026
G TOKEN SUB-SAMPLING AND LOW-RANK APPROXIMATION
Full-batch covariance computation is exact but becomes costly at scale (see Appendix H). We evaluate two approximation strategies: token-level sub-sampling (using 5%. 10%. 25%, and 50% of tokens) and low-rank eigendecomppoiiiin. For the latter, we use RandSVD (Halko et al. 2011). which uses randomized projections (optionally with power iterations) to approximate the top-k eigenspectrum, and Lanczos iteration (Lanczos 1950). We evaluate both at ranks &t E {256.. 512), covering two widely used paradigms for scalable eigendecomposition (Ghorbani al. 2019).
G.1 EIGEN-METRIC FIDELITY AND DIAGNOSTIC VALIDITY UNDER APPROXIMATIONS
Token sub-sampling better preserves metric fidelity than low-rank truncation Table reports the percentage error relative to full-batch eigendecomposition for GPT-2 (GELU). Token sub-sampling preserves all four metrics with minimal distortion-al 10% sampling, the worst-case error is 10.24% (PR_post), with SE and EEE errors below 2%. By contrast, low-rank methods introduce substantial bias: because SE, PR, EEE, and JS aggregate information across the full spectrum, truncating tail eigenvalues systematically distorts the metrics, with PR post errors exceeding 90% even at rank 512. Therefore, token sub-san sampling should be used over low-rank approximation when computational constraints require approximate eigendecomposition.
Table 8: Percentage of error when sampling and low-rank approximation methods are used in GPT-2.
Sampling
Ra
Lanczos
Metric
5%
10%
25%
50%
256
512
256
512
SE pre (%)
1.0
0.5
0.5
0.4
50.7
50.1
7.9
1.7
SE_ (%)
4.8
1.9
1.3
1.3
63.7
61.0
28.8
19.3
PR pre (%)
4.9
4.1
5.1
4.3
90.4
90.2
18.0
3.6
PR_post (%)
16.7
10.2
33.5
30.2
94.9
94.8
68.4
53.9
EEE pre (%)
0.4
0.1
0.1
0.1
4.2
1.6
38.8
26.3
EEE post (%)
20.7
8.6
1.4
1.2
35.7
38.6
30.5
28.8
JS (%)
21.5
8.3
3.0
3.4
67.5
63.2
81.1
70.5
Approximation preserves pre-activation but not post-activation diagnostic power. Low absolute error in metric does not necessarily preserve its diagnostic utility: approximations alter the rank- ordering of configurations and weaken their correlation with network's generalization performance. Table evaluaaes this correlations between each eigen-metree metric and validation loss under approximation. Table 9: Effect of token-sampiin sampling and low-rank approximation (etric- correlation, evaluated on GPT-2 (125M). poken-sampling preserves pre-metric trends but degrades post-metric correlations since tail eigenvalues under- snder-sampled; low-rank truncation distorts both.
Sampling
RandSVD
Lanezos
Metric
5%
10%
25%
50%
256
512
256
512
SP_rre
-0.97
-0.972
-0.971
-0.972
-0.106
-0.174
-0.822
-0.959
SE_post -0.34
-0.376
-0.332
-0.363
0.612
0.568
0.335
0.08
PR pre
-0.914
-0.915
-0.918
-0.912
0,014
0.047
-0.788
-0,901
PR. post
-0.122
-0.138
0.049
-0.235
0.594
0.575
0.314
0.136
Token sub- sub-sampling largely preserves pre-activation correlations: SEpre remains |r| > 0.97 and Peec remains |r| > 0.91 across sampling ratios (5%-50%). In contrast, post-activation correlations are less stable under sub-sampling (SF drops to |r| x 0.34-0.38 and PR to |r| < 0.24, with sign flip at 25%), suggesting that sub-sampling noise disproportionately affects the mid-to-tail spectrum.
Low-rank approximations degrade correlations more severely: RandS yields weak correlations for both pre- and post-activation metrics (SEpr |r| < 0.188, PR |r| < 0.05). Lanczos performs better for pre-activation metrics (e.g., Lanczos-512. SEpe |r| 0.96, PRpr Ir| = 0.90), but post-activation correlations remain low (SEpost |r| < 0.34, PRpsstt |r| < 0.14). Overall, token sub-sampling at > 10% preserves most pre- pre-activation diagnostic power with minimal loss, whereas correlation- based analyses of post-activation metrics benefit from full-batch covariance estimation ([rl > 0.84, Table 3). Low-rank truncation is unsuitable when preserving metric-loss correlations is the primary objective.
26

---

Published as a conference paper at ICLR 2026
G.2 SPECTRAL SIGNATURE OF SAMPLING AND Low RANK APPROXIMATION
Speciia Exbryy
Eullation Liss
Spectral trtenry
Eooluttonn to
Spectea catnayy de
Eulusston Losa
spectral prbnary oe o.
Townlen
Eowrkons
Towrbone
10K 20K 30K
40K
10K
20K 30K
48K
10K
20K
30K
40K
10K
20K
30K
40K
Thanng
Taning Step
Twining
Thining Step
5% pra:--.911,, post--0.122)
10% pre:--1.915, post-00.130)
25% or. pm 00.918 r post: 0.8091
50% pra:-4.912, post--0.2251
600
Exaluation LoS
Nertoppee on talto
800
800
Exllattinn
800
Eraluation Los
Settepe on tale
600
800
200
400
Ps.re
post
400
200
200
400
PR. post
PR
200
400
20K
Taaing
Trating Step
Figure 18: SE and PR in GPT-2 with Pearson r to eval loss under sub-sampling (5, 10, 25, 50%)
F. p.setto.612))
1774 548)
(r p..pee-...e. (.pot_00.38))
pwp.p..o..... p.
Speetra Ererap
Fosi irom loom
Sietral Ereupp
Spetta Eerrry
2 w
Scedral
F.0
3.5
20K
40K
20K
40K
20K
40K
20K
40K
Brining Skpp
Trariin Stop
Training Step
Thainig Siop
8-256 p.o...0000, p.55
R-512 Ir. pre:0.077, 0.57..
1-256 (r. pww. 788, 0,3140
L-S12 (r.preu 0,901, postuo.13))
Puctpaaton hate
Nartouatann hute
40
Esailiat on Uni
Pwtccpattnn Reto
150-
Exailutton on unia
Tut optappaee tate
300
30-
:00-
200
20-
7R. post
10-
7R, post
70, pm
78, post
78, pr
100
7R. post
78,
Eva
Tval--ae vas
Zvai
Eval uaa
40K
20K
20K
Figure 19: SE and PR in GPT-2 with Pearson to eval loss under low-rank approximation.
sampling_1o%
sampling,
sampling
Laver
LayeT
35%
34k
85
36k
24k
12k
4.8
6K
24k
32K
86
.06
246
RanddVD-25e (EEE)
Siep
RandSUD-512 (CEE)
mg Step
Lanczoe-2366 (EEE)
Step
Lanczue-513 (C06)
Thaining
Layer
LaHdF
0K
2BC
24C
326
43K
56
30k
24x
326
425
OK
66
5Bc
24k
33K
40
13K
24k
32k
414
Figure 20: Impact of sampling (10% 25%, 50%). and low-rank approximation on EEE (GPT-2)
baseline (5)
sampling 19%
sampling 25% (1)
sampling 50% (S)
Laver
LayeE
1.5%
24K
33X
11k
24k
32N
4.5
3K
15%
24k
32K
256
24C
324
Pramiigg Siep
Tra ringg
Tritingg S0ap
Thaniing
Siep
(1S)
Layer
Layer
Laver
402
025
19C
2
5:0
11k
24R
126
3
40
16k
24k
12x
414
Figure 21: Impact of sampling (10% 25% 50%), and low-rank approximation on JS metric (GPT-2)
27

---

Published as a conference paper at ICLR 2026
H COMPUTATIONAL AND MEMORY EFFICIENCY OF NERVE
H.1 COMPUTATIONAL COMPLEXITY AND MEMORY OVERHEADS OF NERVE
Table 10 reports wall-clock time and relative overhead for computing eigenspectrum metrics on GPT-2 (a 3072 3072 covariance) across approximation methods. Full-batch eigendecomposition takes 14.41s per logging step, corresponding to 6.4% overhead when logging every 200 steps, and 1.3% when logging every 1000 steps. Token sub-sampling at 5% reduces this 9.89s, with wall-clock time increasing gradually from 9.89s to 14.41s (full batch). At the recommended logging frequency of every 1000 steps, all methods, including full batch, add approximately 1% overhead, making approximation irrelevant at this scale. Sub-sampling or low-rank methods become relevant only at substantially larger FFN dimensions (D > 3072).
Table 10: Wall-clock time and relative overhead for computing eigenspectrum metrics at various logging frequencies. Overhead reported as percentage of total training time when eigendecomposi- tion is performed every 200 and 1000 steps GPT-2 with 3072x3072 FFN covariance matrix size running on AMD EPYC 7502 server with NVIDIA RTX 3090 GPU
Sampling
RandSVD
Lanczos
Full
Metric
5%
10%
25%
50%
256
512
256
512
batch
Wall-clock time
9.89s
9.92s
11.48s
12.62s
10.28s
11.19s
12.07s
12.20s
14.41s
Overhead-200 (%)
4.37
4.39
5.08
5.58
4.55
4.95
5.34
5.40
6.38
Overhead- 1K (%)
0.87
0.88
1.02
1.12
0.91
0.99
1.07
1.08
1.28
H.2 MEMORY-EFFICIENT EIGENSPECTRUN ANALYSIS
To enable scalable eigenvalue analysis during training, implemented memory optimization strategies, listed in Algorithm [ As shown in Table T1 these memory optimization significantly reduces the peak GPU memory usage. For instance, for GPT-2 model (D=4d) the peak GPU memory
usage is restricted to x 2 x 36MB per layer rather than accumulating 2 x 12 x 36MB = 864MB across all FFNS per logging step.
Table 11: GPU memory overhead for full-batch eigen-computation across various FFN widths.
Metrie
D-ld
D-2d
D=3d
D=4d
D=5d
D=6d
D=7d
D=8d
dimn.
1768.768]
11536, 1536]
12304, 2304
13072, 3072)
[3840. 38401
14608,
55376, 5376]
[6144, 6144|
Matrix size
2,25MB
9MB
20.25 20.25MB
36MB
56.25MB
8IMB
110.25MB
144MB
GPU
The complete pipeline's wall-clock time, following the steps Algorithm I is measured as follows:
f Start timing (includes all operations below)
2 torch. Corch.ccua..sppccronneee synchronize ()
3 start_time = time. ()
Complete eigenvalue pipeline for all layers:
6 + 1. CPU-GPU data transfer activations
7 # 2. Covariance matrix computation (3072x3072)
4 3. Eigenvalue decomposition using forch.linalg.eiie
4. Zigenvalue metrics computation (SE, FR, EEE, JS)
5. Memory cleanup and GPU cache clearing
11
12 torch. coch..cuddauay.. www..uddeecoooiiieeeee..
mnd_time time.timel overhead end_time start_time
Complet wall hal1-clock time
Listing 1: Wall-clock timing measurement for cigenspectrum-basee computational overhead
To enable scalable eigenvalue analysis during training, we implement three memory optimization strategies that significantly reduce GPU memory overhead:
1. Eigenvalue-only computation: Since our eigen metrics depend only on eigenvalues, we employ torch. linalg g. e eigval sh compute eigenvalues without eigenvectors.
28

---

Published as a conference paper at ICLR 2026
### What we're using (efficient):
Only eigenvalues: 3072 values
vals torch.Linalg.eigvall.. (cov)
##F Less efficient alternatives:
Eigenvalues eigenvectors: 3072 (3072x3072)
vais, vees eigh (cov)
General eigendecomposition (even more overhead)
vals, vecs torch. werch.linalg.eer (cov)
Listing 2: Memory-efficient eigenvalue computation
2. Sequential layer processing: Rather than computing eigenvalues for all layers simultancously. we process layers sequentially with memory cleanup between computations:
Process each layer individually with cleanup for layer_idx in (sef.. sesyye_pre_actt__e acts keys Compute metrics for current layer
self.layer_pan Layer_pro_acts clac clear) co.collect torch.cuda empty_cachet
Listing 3: Sequential layer processing with memory cleanup
This approach maintains peak GPU memory usage at ~ x 36MB per layer rather than accumulat-
ing 2x x3MB 864MB across all FFNs per logging step (Table [T].
3. Hybrid storage strategy: We store activation tensors on CPU memory while performing eigen-
value computations GPU, balancing memory efficiency with computational speed.
Algorithm 1 Memory-Efficiennt Eigenspectrum Analysis
1: for layer { 1,..., do
2: Hg + GetAciivationss()
Store on CPU
3: dy eigvalsh( H, H (NN
O(d) memory
4: Compute metrics: SE, PR, EEE, JS from x
5: del H;
Immediate cleanup
6: end for
29

---

Published as a conference paper at ICLR 2026
I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE
We selected MLP-Mixer (Tolstikhin al. 2021 as non-transformer architecture family model for three key reasons: 1) it has the core architectural block that we studies in this work, wider MLPs/FENs with LayerNorm and GELU activations, while completely removing the self-attention block; 2) the channel-mixine MLPs/FFNs in MLP-Mixer are functionally analogous to FFNs in Transformers, as they expand and squeeze the latent representation with an intervening nonlinearity, and stacked across the network's s depth; and 3) it isolate the contribution of FFN nonlinear transformations from attention-specifie dynamics like rank collapse in self-attention (Dong et al., 2021).
Moreover, as the MLP-Mixer is designed for vision tasks, it has fundamentally different inductive biases compared to decode-only LLMs. More importantly, it allows us to extend the eigenspectrum analysis from language modeling to vision tasks, verifying the generalization across modalities. This lets us ask: do the eigenspectrum patterns we observe fi fundamentally on the attention mechanism, or are they a more general property of deep feedforward layers?
Model, dataset, and training hyperparameters We use MLP-Mixer B/16 models adapted for CIFAR-100 x32x32 images, 4x4 patches, 64 tokens in one sequence) since it closely resembles the key architectural parameters from GPT-2 to enable direct comparison. The architecture consists of 12 Mixer blocks with embedding dimension 768. Each block contains two MLPs: token-mixing (hidden dimension 384) and channel-mixing (hidden dimension 3072), both using LayerNorm and GELU activation. This configuration results in 57.4M trainable parameters.
Settings:
We train mixer models for 120 epochs using the Adam optimizer (Kingma 2015) with a learning rate of 1e-3 and weight decay of 5e-5, We use batch size of 128 and employ cosine annealing for learning rate scheduling with 5 epochs of linear warmup. For data augmentation, we apply Auto.Augment and CutMix to improve generalization.
Methodology. We apply our NerVE framework to compute eigenspectrum metrics (pre- and post- activation) for all channel-mixing MIPs (dimension 3072) at epochs 1, 10, 20, 40, 80, and 120. We use full-batch covariance estimation with no sampling. Hence, at each logging epoch, we accumulate the statistics across the entire training dataset single forward pass which yields covariance matrices from 3.2M samples per layer. Our GPU-optimized implementation accumulates statistics for all layers simultaneously, requiring ~432MB additional GPU memory.
For an extensive eigenspectral dynamics study, we evaluate four architectural variants by systemati- cally varying the activation functions in the token-mixing (FFN1) and channel- -mixing (FFN2) layers: (1) GELU, GELU (baseline); (2) GELU, ReLU; (3) ReLU, GELU; and (4) ReLU, ReLU. We apply the full NerVE analysis to all configurations, tracking eigenspectrum metrics across all 12 layers. Figure 22 shows the eigenspectrum dynamics for each configuration, demonstrating how activation function choice impacts spectral properties during training.
Observations. First, across all four settings, post-activation eigenspectrum consistently exhibit higher spectral entropy and participation ratio than their pre-activation counterparts, with the gap increasing rapidly in the first 20 40 epochs, indicating that the Mixer nonlinearities reliably expand the effective dimensionality of the FFN representations rather than merely reshuffling.
Second. swapping GELU for ReLU in the channel-mixing FFN (FFN2) has a much stronger effect than changing the token-mixing FFN (MLPI). Configurations with ReLU in FFN2 show larger post-activation SE/PR and a more significant drop in EEE, indicating a flatter, less top-heavy spectrum where variance is redistributed away from the leading eigenmodes.
Third, the GELU (in FFN1) and ReLU (FFN2) variant, 3rd row in Figure 22 results in most aggressively flattened spectra indicated by highest post-activation PR and lowest post-activation EEE values while maintaining small JS divergence across most layers, suggesting that ReI.U in token-mixing FFN drives a more uniformly expanded, higher effective dimensionality latent space rather than inducing sharp layer-specific distortions.
Finally, JS divergence is concentrated mostly in early layers when feature-mixing (FFN2) as GELU nonlinearity, whereas ReLU in FFN2 has more uniform pattern for layerwise JS divergence, mostly concentrated in deeper layers. This suggest that activation choice mainly modulates how much the boundary layers reshape the eigenspectrum while the interior layers act as relatively stationary propagators of the learned latent representation.
30

---

Published as a conference paper at ICLR 2026
SE COLS, FTRO- 60L8)
PL.7PAL CELB. FFN2 GELV)
EEE (EEIPAA SEUL FINO GEWV
15477HG CELS, 77MC CELA)
Fl
25
0.2
0.30
0.00
(176 ReLS PPS2 GELL1
PA 14.77A11
-Betil PPN2 -0EL1
EIE (TPN) Belli FFRO GELU)
25 2PM)
FF52
GELI
Fl,a
0.2
8 R
Fl.aot
Eerogate Mate
40
0.00
L1
PS (PPR)
FRE 17P8) CEUL PINO AMil)
02-
Fl
Ginecch
F1.20t
400
00
05 L3L30112
prnt Pekii PP1 (M
PS LPP0
2 p9st Bekkl P91 -eil
Fl,r5
Brrech
4G
40 80
40 3.0
Figure 22: Eigenspectrum dynamics in MLP-Mixer under activation ablations. Rows correspond to the four activation configurations for noken-mixing (FFN1) and channel-mixing (FFN2) layers, and columns (from left right) show spectral entropy (SE), participation ratio (PR), eigenvalue early enrichment (EEE), and Jensen-Shannon divergence (JS) for the channel-mixing FFNs (FFN2). Each panel traces pre- and post-activation metries over training, showing that ReLU in the channel-mixing MLP (3rd and 4th rows) most strongly increases SE/PR and reduces EEE, i.c., reinjects variance into low-energy directions and flattens the spectrum.
Accuracy on CIFAR-100 Table 12 shows the accuracy for activation ablation study in MLP-Mixer. The ReLU FFN2 leads to a better accuracy since it flattens the spectrum (lower EEE_post) and reduce the top heaviness of the spectrum. Whereas, GELU in FFN2 leads higher EEE values throughout the training. The combination of GELU in FFN1, and ReLU in FFN2 leads to highest PR and lowest EEE, indicating best utilization of latent space in FFN2 and results in best accuracy.
Table 12: Validation accuracy on CIFAR- 100 for different activation function configurations in Mixer. FFN1 refers to token-mixing MIP, FFN2 refers to channel-mixing MI..
FEN1
FFN2
Acc. (%)
GELU
GELU
66.96
GELU
ReLU
66.99
ReLU
GELU
67.86
ReLU
ReLU
67.20
31

---

Published as a conference paper at ICLR 2026
J TOKEN-POSITION EFFECTS ON FEN EIGENSPECTRUM
To make the FFN eigenspectrum analysis explicitly sensitive to sequential structure, we stratify tokens by their position and recompute NerVE metrics within each position groups during their training. For GPT-2, we partition tokens into three groups along the sequence dimension (early, middle, late). For MLP-Mixer CIFAR- 100 instead split patches into top bottom rows of the 4x4 patch grid. For each configuration we then track effective dimensionality of the pre- and post-eigenspectrum using participation ratio metric, both aggregated across layers and averaged per layer (see Figure 23
[Late ys Early: 0.3%)
vs Early: +224.9%
GELU
Mieiie
Miliee
150
Late
300
ReL.U
Safty
Sorly
Loyar
Nomalre GELU
Late Early: (.2.8%)
Late Early: 3.6%)
800
Garly
NonnFree BeLU
1>x(Late Early: +13.99%
a03
Early 2.22%
Saily
Mediee
Lafe
Milile
NornFree LiteUU
Late Early: +19.8%
Late Early -7.4%)
203-
Sarty
Top Bottom: 9%
Top vs Bottoom -3.5%
100
220
Top
20
Bottom
MLP-MMieer
100
20
10 33
Figure 23: Sequence-aware FFN eigenspectrum across token positions. For four GPT-2 (125M) variants (rows to 4) and non-transformer MLP-Mixer (row 5), tokens are grouped by position. Left: pre-activation PRpre: Middle: post-activation PRppsti and Right: layer-wise PRpost- In GPT-2 with baseline, both GELU and ReLU variants, carly/middllellte tokens have similar PR but diverge strongly in PR.pct. with late tokens using substantially higher effective dimensionality, mainly visible in deeper layers. However, in the normalization- conditioned differences are largely suppressed both pre- and post-activations. On the other hand, in non-transformet family model, MLP- trained with SGD optimizer, where split patches into top vs bottom rows, position effects are weak throughout. This show that strong position-dependee FFN geometry is characteristic of LayerNorm- based GPT-2
GPT-2 variants, these position-
Baseline GPT-2 exhibits strong position dependence only after the FFN nonlinearity. In both GPT-2 baseline variants (GELU and ReLU), the pre-activation PR shows almost no systematic differences between early, middle, and late tokens: the three boxes largely overlap, suggesting that the FFN representations before nonlinearity have comparable effective dimensionality across positions. In contrast, the post-activation PR shows distinctive ordering: late tokens consistently exhibits substantially higher PR than early tokens, with middle tokens in between. This effect is particularly pronounced for GELU, where late-token PR post is roughly twice that of early tokens, and remains
32

---

Published as a conference paper at ICLR 2026
visible under ReLU. The layerwise trend (right column) further reveal that this distinction emerges mainly in the second half of the network: deeper FFNs layers allocate more latent degrees of freedom to later tokens, while early layers treat positions more uniformly.
Normalization-free GPT-2 suppresses position-dependent FFN geometry. When we remove LayerNorm layers (NormFree-GGe GELU and NormFree-ReL.)) the position-induced differences reduce dramatically. Both pre- and post-activation PR distributions for early, middle, and late tokens nearly collapse onto each other, and the layerwise trends show very small gaps between position groups at all depths. That is, once normalization removed, the FFN eigenspectrum become almost dnsition-agnostic. This contrast suggests that in standard GPT-2, LayerNorm coupled with the nonlinearity amplifies positional biases in the FFN latent space, which is largely disappears in the normalization- free settings.
Von-transformer MLP-Mixer shows only weak spatial position effects. For the MLP-Mix model, we perform an analogous top-bottom split over patch positions in baseline model (GELU in both MLPs) when trained from scratch CIFAR-100. Here, position-conditioned PR differences are small in both pre- and post-iigenspectrum. and the layerwise PR trend for top vs bottom patches almost coincide. This indicates that the strong position dependence observed in GPT-2 FFNs is not a generic property of FFNs: in a non- non-transformer architecture trained on images, NerVE sees only mild spatial variation in FFN eigenspectrum.
33

---

Published as a conference paper at ICLR 2026
K FEN EIGENSPECTRUM DYNAMICS: LAYERNORM VS RMSNORM
K.1 LAYERNORM VS RMSNORM ABLATION ON GPT-2
To examine how sensitive our FFN eigenspectrum analysis is to the normalization choices, we systematically vary both the placement of the normalization (PreLN, PostLN, MixLN) and the FFN activation type (GELU, ReLU, learnable leaky ReLU), and compare their effective post-activation dimensionality using the (Figure 24
ReLU
Learnable Leaky ReLU
1500
Layenkorm
RM5Norm
1250
Participation
1000
800
750
6:00
400
500
400
200
250
200
PreLN
PostUN
MixiN
PreLN
PosiLN
MixLN
PYiLN
PastLN
MixLN
GELU
ReLU
Learnable Leaky ReLU
RMS Norm Partic Iprticipation Ratio
1500
0.1415
1200
R 0.1691
800
R 0.7708
1250
1000
600
1000
800
600
400
400
y-x
200
PostLN
1000
1500
1000
0
200
400
600
800
LayerNorm - Participation Ratio
LayerNorm Participation Ratio
LayerNorm Participation Ratio
Figure 24: Effect of LayerNorm vs RMSNorM on FEN eigenspectrum. Top row: Post-activation participation ratio pot) in GPT-2 (125M) for three FFN activations (GELU, ReLU, learnable leaky ReLU), three normalization placements (PreLN, PostLN, MixLN), and two normalization variants. Within each activation-placemerd overlap, while much larger shifts arise from changing the activation type or the norm placement. Bottom row: Layerwise comparison of post under LayerNorm (æ-axis) vs RMSNorm (y-axis). Each point represents one FFN layer. GELU and ReLU show similar PR ranges but weak layerwise correspondence (R2 0.2), while learnable leaky ReLU exhibits strong alignment (R² 0.77). This indicates that FFN eigenspectrum patterns are primarily driven by activation type and are largely robust to the choice of normalization scheme.
pair, the LayerNorm and RMSNorm boxplots substantially
FFN activation type and the placement of normalization layers is more consequential than LayerNorm RMSNorm. Figure (top row) shows the distribution of post-activation participation ratio in various normalization layer positioning and FFN activation settings. Within each activation- placement pair, the LayerNorm and RMSNorm boxplots substantially overlap: their medians and interquartile ranges are of similar magnitude, and both exhibit the same depth-wise trend. For instance, PreLN exhibits highest PR spread for GELU while PostLN being more compressed.
(top
By contrast, the activation choice induces much larger shifts in PR: GELU yields the largest PR values overall, ReLU compresses the spectrum, and learnable leaky ReLU lies in between with narrower spread. Likewise, changing the PreL. to PostLN or MixLN has clear effect on the PR distribution, whereas swapping LayerNorm RMSNorm within a fixed placement produces only second-order changes, not the qualitative trends.
Layerwise eigenspectral structure is largely preserved for LayerNorm vs RMSNorm. Figure 24 (bottom row) compares LayerNorm and RMSNorm on per-layer basis by plotting. For GELU and ReLU, the points cluster a similar numeric range and follow a weak positive trend (R² zs 0.14 and R² = 0.17, respectively): layers that are high-PR under LayerNorm tend to remain high-PR under RMSNorm, and low-PR layers remain low-PR, even though RMSNorm reshapes individual values somewhat. For learnable leaky ReL.U, the alignment much stronger (R² as 0.77). This shows
34

---

Published as a conference paper at ICLR 2026
that the ordering of layers and the qualitative depth-wise behavior of the FFN eigenspectrum (quantitatively) for LayerNorm vs RMSNorm; the main axes of variation remain activation type and the placement of normalization layer.
Recall that NerVE operates on centered FFN activations: when we construct (pre-/post-)covvv )covariace matrices, subtract the empirical mean across tokens. This removes the DC component for eigenspectral analysis regardless of whether the model uses LayerNorm (which performs centering inside the model) or RMSNorm (which does not). Combined with the empirical evidence above, these ablations reassure that our conclusions about FFN eigenspectrum dynamics do not rely on particular normalization layer; they hold for both LayerNorm and RMSNorm.
K.2 LAYERNORM Vs RMSNORM ABLATION ON MLP-MIXER
We extend our LayerNorm RMSNorm comparison to MLP-Mixer trained on CIFAR-100. Figure 25 shows acst-accivation SE, PR. and EEE throughout training, and JS the final-epoch IS across layers.
(PostAct)
PR (PoetAct)
EEE (PostAct)
LoyerNorm (Ace 98365)
LaysrNorm
0.35
Layeritorm
RM dice 66.35%)
RM45Nom
51 Wereaaee Earty Comtttemnn
RMiSWerm
0.8
0.25
Partcppation
Spaccra
80-
0.15
04
0.10
Laye Roo
2..05
RMSNNOM
20
40
20 40
0.0
20 45
0.00
LS
Epooth
Epsch
Epach
Layer
Figure 25: Effect of LayerNorm vs RMSNorm in the MLP-Mixer models. Columns (from left to right) shows post-activation SE, PR. EEE, and finally JS metric trend for LayerNorm and RMSNorm configurations throughout the training when MLP-Mixer models (GELU in both MLPs) are trained from scratch CIFAR- 100 dataset using Adam optimizer. FFN eigenspectrum dynamics remain (qualitatively) stable when LayerNorm is replaced with RMSNorm. However, in the later phases of training. the LayerNorm configuration attains higher effective dimensionality (PR) than R.MNNrmm and its spectrum becomes noticeably flatter.
Throughout training, both normalization schemes produce very similar SE and EEE dynamics while layerwise trends at final convergence almost overlap, suggesting very similar reshaping of the eigenspectrum. The main difference appears in the final training phase, where the LayerNorm model attains a slightly higher PR post and lower EEE, EEE_post than RMSNorm, and correspondingly achieves modest accuracy gain (66.96% vs 66.38%). This suggests that, even in FFN-only architectures like Mixer,, the qualitative eigenspectral behavior remain robust to the choice normalization layer.
35

---

Published as a conference paper at ICLR 2026
L OPTIMIZER-DEPENDENT FFN EIGENSPECTRUM DYNAMICS
L.1 ADAMW VS MUON VS DION
We examine the optimizer-induced eigenspectral dynamics in GPT-2 160M models, when trained
from scratch on WineWeb dataset with context size 512 (Figure 27) and 1024 (Figure 26) across Adam W. Muon (Liu et al.
2025 Shah et al. 2025 Boreiko et al.) and Dion optimizers, following the architectural and training hyperparameters settings from Ahn et al. 2025
Cata (Pout/Pra)- Adamly
Ect (Peat Pret Ademly
etC Adanii
AdantV
Layor
10
100
post
0.4
1k
24
Yurnin Strp
Taring 2up
Turing Sous
Tha ring Step
Sunicaatinn RaTK C400
PR. Gaiec (Pos) Mean
8 EZ Oefooeee (hed D. thot 1e0
140 (Post Pre) Moan
1.0
Meen
-Meen
0.2
-033
20.4
16
X
1<
2<
Turnng Step
Turnng Sup
Turnep Ster
ning Sieo
PR Gains (PostPrrll Bion
644 (Peet fret- Bion
606 -iten
J6- Bioo
110
20
-03
1K
26
24
Tooaus Step
Yoaning Stop
Tonin Stor
Tra eiro Ste
Figure 26: cedimizer-depndeen FFN eigenspectrum dynamics in 6PT2-160M (context length 1024).
Rows show AdamW (top). Muon (middle), and Dion (bottom).
7att-paton Fate Cam
0.0
1.0
Layar
10
0.6
EBE
14
2K
3x
1x
24
(Postinn)
1.0
-03
0.8
0.0
1k
1.0
1204
-04-
1k
2k
Figure 27: Opiiiizer-deendeet FFN eigenspectrum dynamics in GPT2-160M (context length 512).
Rows show AdamW (top), Muon (middle), and Dion (bottom).
Across context length 512 and 1024 GPT-2 160M models, a consistent eigenspectral trend emerges. AdamW exhibits large early PR gains and high IS divergence with relatively high post-activation EEE (EEE.post), indicating optimizer-induced pre-activation collapse followed by aggressive but incomplete nonlinear repair. In contrast, Muon shows the smallest PR gains, lowest JS, and lowest EEE.post, with flatter post-activation spectra that suggest more stable eigenvalue distributions throughout training. Dion falls between these two regimes, it improves over AdamW but does not fully match Muon's 's spectral behavior. Notably, in a few early FFN layers, Dion exhibits PR gains on par with AdamW, W, suggesting layer-dependent repair dynamics.
36

---

Published as a conference paper at ICLR 2026
AdamW
PR. pre Muon
PR_pre- Dion
dur Radio
122
100M (Conext
Partipaation
40
100
-0
75
69
10
50-
8
1k
*
Traning Step
Training Step
Training Stop
AdamW
PR_pre- - Muon
pre Dion
160M (Context 1024)
Partccoation Rato
Purtic ppatinn Rabio
16
321
Partic spation
60-
20-
21-
Tarnnn Step
Thaining Stop
Training Step
Figure 28: Activation-compatilliiy of pre-activation eigenspectrum. Columns show the effective dimensionality of pre-activation spectrum (PR_pre) when GPT2-160M models are trained from scratch with Adam W (left), Muon (middle), and Dion (right) optimizers, with 512 (top) and 1024 (bottom) context length. Muon exhibits the highest PR pre across layers, demonstrating cellicondi pre-activation spectra that place less burden on the FFN nonlinearity to restore representational rank.
To evaluate the conditioning of pre-activation cigenspectrum, and asses the extent to which FFN nonlinearity must actively inflate representational rank, we plot the layerwise evolution of PR.pre during training in Figure 28 Across settings, Muon consistently exhibits the highest effective dimensionality (PR. (PR.pre) across layers, indicating well-conditioned pre-activation spectra that place far less burden on the FFN nonlinearity to restore representational collapses. On contrary, Adam shows pronounced early-layer collapse, while Dion partially mitigates these collapses.
Increasing the context length does not change this ordering; if anything, it makes the early-layer collapse under Adam and Dion more pronounced, while Muon persistently keeps pre-spectra high--dimensional. This straighten the earlier observations that Muon produces activation-compatibe representations across both model sizes and sequence lengths (Figure & and Figure 9
Table 13: Evaluation perplexity (PPL) for GPT-2 models trained from scratch on the Wen dataset (Penedo et al. (024) using AdamW, Muon, and Dion optimizer. Muon consistently achieves the lowest perplexity across all model sizes and context lengths, while Adam results in worse perplexity.
Optimizer
GPT2-350M
GPT2-160M (Context 512)
GPT2-160M (Context 1024)
(Context 512)
AdamW (Loshchilov & Hutter 2019)
33.24
39.26
34.33
Muon (Jordan al. (024)
25.68
30.95
27.09
Dion (Ahn et al. 2025)
27.68
33.60
29.34
L.2 ADAMW VS ADAFACTOR
To further investigate the optimizer-induced effects on FFN eigenspectrum, we substitute the A with Adafactor (Shazeer & Stern 2018) in GPT-2 and evaluate both the baseline and normalization- free variants with GELU and ReLU activations. We include Adafactor for optimizer-induee study because designed reduce optimizer memory via factorized second-mone moment estimates for matrix-shaped parameters, and offers a different preconditioning geometry from AdamW while remaining practical for training large language models (Raffel et al. 2020
ablation
We begin with the baseline GPT-2 (125M) with GELU and ReLU activations to substantiate the core findings about the role nonlinearity in FFNs (see Se.tion Figure 29 demonstrates the similar pre- and post-eigenspectrum characteristics that we observed earlier in Figure and Figure 3. Precisely, the pre-activation spectrum entering the FFN are hop-heavy and anisotropic, while the post-actives pust-activation spectrum exhibit higher SE and PR, and lower EEE, indicating that the FFNs systematically reinject variance and flatten the spectrum, counteracting the rank collapse induced by
self-attention (Dong al. 2021 Noci al. 2022 Wu et al. 2024 Joseph al. 2025).
37

---

Published as a conference paper at ICLR 2026
11. OELU
CEL
Optrebe Eaty Dretmot
E08 GELU
15: GEEU
022
1520
1020
5.6
6043
021
649
PR,
688
polt
2041
10K
20C
30K
40K
10K
20C
3
60K
10K
2
50K
40C
10K
20K
30K
40K
Tukking
minp Skop
Tuaning Step
Trakiigg Shop
Ogerrdas Cerndaaccce Cany Extermet
1.0
1000
cas
Sppdie
120
40K
20k 36K
Thaiiin
Figure 29: Eigen- metrics (SE, PR, EEE, and JS) illustrate how FFN nonlinearities regulate infor- mation flow and reshape the eigenspectrum when GPT-2 (125M) models with GELU (top) and ReLU (bottom) are trained from scratch on CodeParrot (2.1B tokens) datasets using Adafactor optimizer (Shazeer & Stern 2018). Pre- and post-activation dynamics are shown for SE, PR, and EEE, highlighting how nonlmeartties reinject variance and alter spectral structure. JS heatmaps (rightmost) capture the layerwise distributional shift induced by nonlinearity.
After establishing the fundamental role of FFN nonlinearity under both AdamW and Adafactor optimizer, we next compare the layerwise distribution of FFN capacity and latent-space utilization (PR.post) side in Figure across four GPT-2 configurations (PreLN and Normalization free, with GELU and ReLU activation). Notably, Adafactor consistently achieves higher anttivccti PR than AdamW, W, and this gap is more pronounced in normalization-fre ReI.U model.
GELU
ReLU
Participation Raão
Participation Ratio
1500
Adamn/
Adafactor
1000
1000
500
500
1
3
8
11
10
11
Layer
Layar
NormFree-GELU
NormFree-ReLU
Participaatio Ratio
1000-
Particppation
1000
750
250
0
0
10
11
0
3
0
10
Lay
Layer
Figure 30: Layerwise post-activation participation ratio (PR.post) comparison for AdamW and Adafactor across four GPT-2 (125M) configurations: baseline (PreLN) with GELU and ReLU (top), and normalization-fra free GELU and ReI.U (bottom). Across settings, Adafactor systematically produces higher PR_post than AdamW, indicating higher FFN latent-space utilization, with the largest gains in the normalization-fre ReI.U model.
This shows that, while FFN nonlinearity plays the same qualitative role role-exee expands and flattens the spectrum-under both AdamW and Adafactor, the degree of this expansion is optimizer-dependen Adafactor consistently drives stronger spectral expansion, activating more FFN latent capacity, especially in the normalization-free ReLU model.
Further, to contrast how effective dimensionality evolves under these two optimizers (Adam W and Adafactor), plot the layerwise post-activation participation ratio (PR.post) over the entire training run [igguree1.. Across all four configurations, deeper layers quickly rise to a substantially higher PR value, and maintain that higher PR. PR.post throughout the training. This effect is consistently stronger with Adafactor than with Adam W. indicating that Adafactor activates more latent capacity in the later FFNs throughout training.
We also track the change in EEE, ABEE = EEE EEEpres over training (Figure 32 More negative AEEE indicates stronger suppression of top eigenvalues and flatter spectrum. Across all four
38

---

Published as a conference paper at ICLR 2026
GEW (kdsmwW
PeLL (Adami))
Hormirroe GELU (AdamrW)
HormFree RecU (6da-moi
1500
3100
1036-
1000
1250
1002
752-
500
600
600
502-
422-
252
200
332
3Ec
420
:6c
22X
S0K
40c
13K
20e
36x
4Gk
20k
40k
1753
or to (Adaffectrr
3400
ReLU Lnstofactoo forr
1330
Hormirre GELU (Adatactor)
1.000
1200
Memratroe RetU (ddaracter)
3200
1252
2000
1000
1000-
14a
502-
400
200-
20C
304
166
2:20
1.3%
200
30K
20R
00c
Figure 31: Training dynamics of post-activation participation ratio (PR) for all layers in GPT-2 (125M) across four configurations (columns) and two optimizers (rows). Deeper layers achieve and maintain higher PR PR.post throughout training, with Adafactor consistently producing larger PR_p than AdamW, especially in the deeper FFNS and in the normalization-fire ReLU models.
GPT-2 configurations, both Adam and Adafactor reduces ΛEEE well below zero, confirming the FFN's spectral flattening role. However, the deeper layers. Adafactor consistently attains more negative AFFE than AdamW, especially in the normalization-free ReI.U model, indicating stronger reduction of hea-ieavs and more aggressive spectral flattening those FFNs.
-6.3
-0.3-
-0.3
0.4
-0.4
-0.5
30%
TrARiig Sup
ring Stop
Testiin ciez
Thaning Sup
(adattccar)
0.2
-0.3
-02
-0.3
-0.1
-0.4
-0.5
-05
Triking 2mo
Tha ring Stop
Trsiiin 10e
Trainng Shop
Figure 32: Training dynamics of FFN-induced spectral flattening in GPT-2 (125M), quantified as AEEE = EEEpest EEEp.. under AdamW (top row) and Adafactor (bottom row) optimizer. More negative AEEE indicates stronger suppression of top eigenvalues and flatter spectrum. Across settings, deeper layers under Adafactor tend to reach more negative AFFE than under AdamW, indicating more aggressive spectral flattening.
L.3 ADAMW VS SGD
To examine how optimizers shape latent space utilization in a non-Transformer setting, we train the MLP-Mixer baseline (GELU in both MLPs) on CIFAR- with SGD and compare its eigenspectrum characteristics side-by-side with the Adam variant in Figure 33 For SGD, we use a learning rate of 5e-2, momentum 0.9, and weight decay le-4.
SGD outperforms Adam on MLP-Mixer (68.07% vs. 66.96%), and their spectral metrics trend explains this performance gap. Notably, SGD attains significantly higher (post-activation) spectral entropy and participation ratio throughout training, indicating higher effective dimensionality and better utilization of FFN representational capacity. The EEE trend shows that Adam remains near 1.0 throughout training which suggests the persistent concentration of variance in the top eigenvalues. JS divergence further shows that Adam induces stronger spectral transformation in the very first layer, whereas SGD-induced transformation between pre- and post-eigenspectrum remain lower and more uniform across depth.
39

---

Published as a conference paper at ICLR 2026
SE (PostAct)
PR [PostAct)
EEE (PostAct)
Adam (Act 63 90%)
Adam (Pec - 90%)
Adam
SGD JACC, 68.07%)
100
SGD JAcc 68.07%).
Ugemaaee Sungy Enriccemmm
0.30
SGD
Sqaeiie Entreyy
80
0:25
0.23
0.15
0.10
Adam JACE 66.06%
R0.05
SGD (Art 774
0.00
Epoch
Eporh
Eparh
Figure 33: Optimizer effects in MLP-Mixer (Adam vs. SGD). Columns (from left to right) shows post-activation SE, PR, EEE, and finally JS divergence metric trend when MLP-Mixer baseline model (GELU in both MLPs) is trained from scratch on CIFAR- 100 dataset using Adam and SGD optimizer. Right from the epoch 10, the MLP-Mixer with the SGD optimizer starts exhibiting superior SE_post and PR-post showing significantly better utilization of FFN2 latent space.
M LIMITATIONS OF NERVE FRAMEWORK
Per-layer independence. Metrics are computed independently per layer, with no explicit measure of how spectra relate across depth. Our repair-vs-refinemen findings ($3.7) suggest that optimizers can induce different degrees of cross-layer coherence, but NerVE does not currently quantify this. Cross-layer
measures (e.g., overlap between consecutive layers' top-k subspaces) could formalize whether smooth spectral progression associated with healthier training dynamics.
Token-pond siten-position aggregation. By flattening the [B. S, DJ tensor into single [N, DJ matrix, NerVE treats all token positions as exchangeable samples. This is a deliberate choice: covariance estimation benefits from the larger sample size, and the framework targets FFN-level geometry rather than position-specifit dynamics. However, the position-stee -staatin-t analysis (Appendix (J) reveals that this aggregation masks substantial position-dependee + +125% % PR difference between early and late tokens). When pnsition-specifi FFN geometry is the target, stratified analysis should complement the aggregate metrics.
structure in LayerNonm-biee basedd models (up to
Practical constraints. Full-batch covariance estimation and storing Dx matrices can be expensive large scale. Our approximation study in Appendix G shows that token sub-sampling can preserve much of the pre-activation diagnostic signal, whereas low-rank truncation can distort correlation- based analyses, especially for post-activation metrics. For very large FFN dimensions (e.g.. D> DD>10K), practitioners should validate approximation fidelity for their specific setting and diagnostic objective.
N DISCUSSION: WHY TOP-HEAVINESS OVER TAIL-HEAVINESS
Heavy-tail (power-law) measurements are common in spectral analysis (Nair et al. 2022 Mahoney Martin 2019), but NerVE emphasizes top-heaviness because our goal is to characterize how FFN nonlinearities reshape the dominant variance subspace of activation covariances from pre- to post-activation. The leading eigenmodes capture directions that contribute most to recond-momed energy and representation anisotropy. NerVE's EEE metric directly summarizes this dominant- subspace structure through the cumulative spectrum, providing a stable and interpretable measure of front-loading without requiring tail cutoff.
In contrast, heavy-tail descriptors (c.g., fitting a power-law exponent) require choosing a tail range and performing goodness-of fit checks. These choices can be sensitive and become harder to compare across scales. As FFN width varies, a fixed corresponds to different spectral fractions, and a fixed fraction corresponds to different absolute depths in the spectrum, complicatnng confggooninn comparisons unless one performs careful per-scale calibration.
For these reasons, do consider including tail-based measures in NerVE the framework. EEE is hyperparameter-frre, numerically stable, and directly aligned with our central question: how nonlinearities transform cominant-subspace geometry and redistribute representational capacity across FFNs and training.
40


Reviews:
[{"review_id": "NiCoSyuxZa", "reviewer": "Reviewer_yZ9g", "summary": "- The paper introduces NerVE, a unified eigenspectral framework for analyzing feed-forward network (FFN) dynamics in LLMs.\n- It tracks eigenspectrum dynamics of representations using Spectral Entropy, Participation Ratio, Eigenvalue Early Enrichment, and Jensen-Shannon Divergence to quantify variance dispersion, dimensionality, and nonlinear redistribution.\n- The main finding is that FFN nonlinearities reinject and redistribute variance, activating underused dimensions and flattening eigenspectra to enhance latent space utilization, while demonstrating a correlation with validation loss.\n- Experiments were run with GPT-2 and Llama-style architectures trained from scratch, showing that NerVE provides an interpretable, data-efficient tool for understanding FFN dynamics beyond empirical tuning.", "questions": "- When computing correlations with validation loss, are the spectral metrics measured on training data or separate held-out sets?\n- Could a similar analysis be done on the other components of the transformer blocks? This may help link the inherent sequential structure of the tokens to your analysis\n- Could NerVE’s metrics generalize to other architectures (eg. non-transformers such as ResNets)? This could test whether the observations are a general FFN property.\n- Could you discuss some practical implications based on the proposed framework?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Provides a systematic, spectral lens to study FFN dynamics, an often-overlooked but important component of transformer models.\n- The four complementary eigenspectrum metrics are theoretically interpretable, and capture distinct aspects of the representations\n- The paper covers normalization variants, activation types, etc, making the analysis framework more broadly applicable.", "weaknesses": "- If I understand correctly, the paper treats activations from different sequences as interchangeable. In that case, what aspects of the analysis are specific to LLMs or transformer architectures? From this perspective, it may be valuable to also examine other types of models with FFNs and compare their behavior to that of LLMs. Alternatively, extending the analysis to explicitly account for the sequential structure of tokens could yield further insights.\n- The models analyzed in the paper are relatively small by LLM standards (up to around 130M parameters). While it may not be feasible to train larger models from scratch, it could be informative to leverage open-weight models with available training checkpoints (e.g. Pythia) to study how the observed behaviors scale with model size.\n- It would also be helpful to include some analysis on downstream tasks, such as computing the proposed metrics on datasets or domains unseen during training, to test whether the observed patterns generalize.\n- Expanding the conclusion by summarizing the main findings and contributions would help consolidate the paper’s message.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VW0zjVBMHh", "reviewer": "Reviewer_KsxW", "summary": "The paper introduces \"NerVE,\" a framework to analyze LLM feed forward networks by tracking the eigenspectrum of their pre- and post-activation covariance matrices. Using four metrics (spectral entropy, participation ratio, eigenvalue early enrichment, and JS divergence), it argues that the FFN's primary role is to \"reinject variance\" by taking the top-heavy output of attention and \"flattening\" its spectral distribution. This \"spectral reshaping\" increases the effective dimensionality for the next layer. The authors use this framework to provide geometric explanations for the effectiveness of architectural choices like Pre-LayerNorm and RoPE.", "questions": "1. Seems the four metrics (entropy, participation ratio, eigenvalues, and JS divergence) are frequently used in related interpretability works. Can the author differentiate itself from other research that apply these similar metrics? Can the author provide a more detailed related works including what metrics are used for study what kind of phenomenon?\n2. As mentioned in weakness 1, have the authors validated your key findings—particularly the efficiency of Pre-LN and the anti-collapse function of RoPE—on any models larger than 1B parameters?\n3. As mentioned in weakness 2, have the authors considered an intervention study (e.g., adding a spectral regularizer) to demonstrate that enforcing a \"healthy\" spectral signature causes better model generalization, rather than just correlating with it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's core claim—that FFNs function as spectral reshapers to re-awaken inactive dimensions is a compelling and intuitive explanation for their role. It provides a strong conceptual model that moves beyond viewing FFNs as simple key-value memories.\n2. The chosen suite of four metrics is a strength. While SE and PR are related, the addition of EEE (to distinguish between different types of flat spectra) and, crucially, JS Divergence (to quantify the nonlinearity's effect) provides a more complete picture than any single metric.\n3. The experiments effectively isolate variables. The norm-free analysis (Section 3.2), for instance, is a good way to demonstrate the FFN's compensatory role, showing that ReLU's piece-wise linear nature provides a regularization effect that the smoother GELU lacks.", "weaknesses": "1. The paper makes claims about \"LLMs\" but bases its findings on very small models (70M-130M). These spectral dynamics are not guaranteed to hold at the 1B+ parameter scales where architectural optimization is most critical. The findings need to be validated on larger models.\n2. The paper repeatedly shows that \"healthy\" spectra (high PR, low EEE) correlate with low validation loss but fails to prove causation. It's just as likely that a well-optimized model produces these spectra as a byproduct of good performance. A direct intervention study (e.g., a spectral regularizer) is needed to make a causal claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MSMFa3REzY", "reviewer": "Reviewer_jJuL", "summary": "This papers proposed to study the MLP/FFN sub-layers in LLM transformers using a spectral analysis. The authors argue that studying the activations immediately before and after the non-linearity (GeLU or ReLU or SwiGLU) allows one to capture the non-linear dynamics of the FFN. The authors proposed 4 metrics on the eigenspectrum of the covariance matrix that capture different effects (how top heavy the spectrum is, how much it has changed after the FFN). Studying these metrics in decode-only transformers shows several effects, most notably the FFN leads to more uniform spectrum/less top heavy eigenspectrum (e.g. fig 1). The authors also study several architectural design choices and how they affect these 4 metrics in some cases.\n\nI am giving a 6 but would probably have given a 5 if there was the option.", "questions": "1. What is the effect of RMSNorm not LayerNorm, as 3.2 is motivated by the centering of LN (which does not exist in RMSNorm).\n2. Likewise, what if one uses a different optimiser than Adam?\n3. \"The EE score is... average vertical distance\" is this not just the area between?\n\ntypos:\n- \"quantifies\" not quantify line 201\n- \"yields\" not yield line 222\n- \"t' in line 69", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The topic of studying the effect of FFNs in LLMs is interesting and the idea of using the eigenspectrum dynamics to do so is well motivated.\n- Several results have interesting insights such as in section 3.1, the FFN nonlinearity redistributes the variance across various eigenvalues.\n- Studying several architectural choices such as layernorms and their positioning or spectral norm is interesting (although also see \"weaknesses\" about how thorough this is).", "weaknesses": "1. Overall, I think the paper identifies an interesting area but feels somewhat underdeveloped. Some particular areas where this comes across:\n\na. There doesn't seem to be a consistent message across section 3.2-3.5. Each of the subsections looks at a different architectural choice but the conclusions don't seem to tie together, e.g. how does the finding that rope prevents mid-to-deep spectral connect to the positioning of LN? For me the most interesting part of these sections was in section 3.4 that normalised participation ratio can be used as a diagnostic for healthy scaling, which I would maybe try to develop more into a consistent narrative across the section. For example, perhaps bad hyperparameter scaling choices (say the variance of the weight init or LR not scaling with width/depth) will become apparent via the normalised participation ratio not being consistent across scales, whereas with healthy scaling choices the PR is constant across scales (as seems to be with Pre-LN in figure 6)?\nb. It feels like there could be some theoretical analysis here that could strengthen the foundations of the paper. E.g. what should one expect the dynamics of the Nerve metrics to be before/after a non-linearity under \"healthy\" behaviour and \"unhealthy\" behaviour. Should it be scale invariant?\nc. I would study mixture of experts first before choosing to study e.g. RoPE or LN positioning, when discussing the effect of FFN architectural choices. This feels like an oversight. If it is too expensive to train and MoE then it should be possible to just take an open-source pretrained one and study the final checkpoint (or multiple checkpoints if available)?\nd. Some of the claims are quite strong for being backed up by one experiment. E.g. in the discussion around hyperspherical normalisation the authors write \"EEE post values remain high across depth, indicating the persistence of dominant directions despite the extended capacity\", which doesn't seem obvious to me looking at Figure 5 (the early layers seem higher in weight and spectral normalisation).\ne. I think the choices of Pre-FFN and Post-FFN are fine (at the top of page 3), but would also be interested to know what would happen if one studied x itself or FFN(x). Maybe more interesting for the latter is to study x + FFN(x) as maybe it's most important to understand how the FFN affects the residual stream?\n\n2. It could be argued that spectral entropy, participation ratio, and early eigenvalue enrichment all look at a similar quantity (how dominant leading eigenvalues are). Indeed, in the plots (say figures 1 or 3) the first three columns look correlated. I'd ask what the benefit of these three separate metrics is, especially given that they add a lot of acronyms to the paper which the reader has to juggle around in his/her head, which can be confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)