Hugging Face model: youyouYUE/golden-window
Author: youyouYUE
Created: 2026-07-05 09:37:12
Last modified: 2026-07-05 09:38:08
Likes: 0 | Downloads: 0
Tags: interpretability, training-dynamics, model-diagnostics, structural-analysis, deep-learning, dataset:WikiText-2, license:mit, region:us
Card metadata:
title: Loss Is Not Enough: The Golden Window in Neural Network Training
authors: name: Song Yue
affiliation: Independent Researcher
abstract: We measure four dimensions of internal information processing quality across  13 open-source neural network models spanning 5 architecture families, from  7.5M to 14B parameters (four orders of magnitude). We discover that training  and fine-tuning proceeds through three structurally distinct phases—Build,  Collapse, and Rebuild—and that the standard optimization metric, loss, is  blind to two of them. During the Collapse phase, structural order degrades  by up to 49% while loss continues to improve, creating a golden window where  information processing quality peaks before being sacrificed for marginal  prediction gains. We release the FPP diagnostic tool, calibration data for  13 models, and all experimental trajectories.

tags: interpretability, training-dynamics, model-diagnostics, structural-analysis, deep-learning
license: mit
datasets: WikiText-2
models: Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen2.5-1.5B-Instruct, Qwen/Qwen2.5-7B-Instruct, Qwen/Qwen2.5-14B-Instruct, EleutherAI/pythia-160m, HuggingFaceTB/SmolLM2-360M-Instruct, HuggingFaceTB/SmolLM2-1.7B-Instruct, TinyLlama/TinyLlama-1.1B-Chat-v1.0, google/gemma-3-1b-it, google/gemma-2-9b-it, facebook/opt-1.3b