The Birth of Knowledge: Emergent Features across Time, Space, and Scale
  in Large Language Models
Authors: name: Shashata Sawmya; name: Micah Adler; name: Nir Shavit
Published: 2025-05-26 02:59:54

Abstract: This paper studies the emergence of interpretable categorical features within
large language models (LLMs), analyzing their behavior across training
checkpoints (time), transformer layers (space), and varying model sizes
(scale). Using sparse autoencoders for mechanistic interpretability, we
identify when and where specific semantic concepts emerge within neural
activations. Results indicate clear temporal and scale-specific thresholds for
feature emergence across multiple domains. Notably, spatial analysis reveals
unexpected semantic reactivation, with early-layer features re-emerging at
later layers, challenging standard assumptions about representational dynamics
in transformer models.