Spectral Scaling Laws in Language Models: How Effectively Do
  Feed-Forward Networks Use Their Latent Space?
Authors: name: Nandan Kumar Jha; name: Brandon Reagen
Published: 2025-10-01 05:38:21

Abstract: As large language models (LLMs) scale, the question is not only how large
they become, but how much of their capacity is effectively utilized. Existing
scaling laws relate model size to loss, yet overlook how components exploit
their latent space. We study feed-forward networks (FFNs) and recast width
selection as a spectral utilization problem. Using a lightweight diagnostic
suite -- Hard Rank (participation ratio), Soft Rank (Shannon rank), Spectral
Concentration, and the composite Spectral Utilization Index (SUI) -- we
quantify how many latent directions are meaningfully activated across LLaMA,
GPT-2, and nGPT families. Our key finding is an asymmetric spectral scaling
law: soft rank follows an almost perfect power law with FFN width, while hard
rank grows only sublinearly and with high variance. This asymmetry suggests
that widening FFNs mostly adds low-energy tail directions, while dominant-mode
subspaces saturate early. Moreover, at larger widths, variance further
collapses into a narrow subspace, leaving much of the latent space
under-utilized. These results recast FFN width selection as a principled
trade-off between tail capacity and dominant-mode capacity, offering concrete
guidance for inference-efficient LLM design.