ICML 2025

Scaling Collapse Reveals Universal Dynamics in
Compute-Optimally Trained Neural Networks
Shikai Qiu 1 † Lechao Xiao 2 Andrew Gordon Wilson 1 Jeffrey Pennington 2 Atish Agarwala 2
Abstract
What scaling limits govern neural network train-
ing dynamics when model size and training time
grow in tandem? We show that despite the com-
plex interactions between architecture, training
algorithms, and data, compute-optimally trained
models exhibit a remarkably precise universality.
Specifically, loss curves from models of varying
sizes collapse onto a single universal curve when
training compute and loss are normalized to unity
at the end of training. With learning rate decay,
the collapse becomes so tight that differences in
the normalized curves across models fall below
the noise floor of individual loss curves across
random seeds, a phenomenon we term supercol-
lapse. We observe supercollapse across learning
rate schedules, datasets, and architectures, includ-
ing transformers trained on next-token prediction,
and find it breaks down when hyperparameters
are scaled suboptimally, providing a precise and
practical indicator of good scaling. We explain
these phenomena by connecting collapse to the
power-law structure in typical neural scaling laws,
and analyzing a simple yet surprisingly effective
model of SGD noise dynamics that accurately
predicts loss curves across various learning rate
schedules and quantitatively explains the origin
of supercollapse.
1. Introduction
As machine learning systems grow in scale, accurate predic-
tive models of their training dynamics become increasingly
valuable, both for interpreting costly experiments and for de-
signing robust, efficient training pipelines (Wortsman et al.,
†Work done partly during an internship at Google Deep-
Mind
1New York University 2Google DeepMind. Correspon-
dence to: Shikai Qiu <sq2129@nyu.edu>, Atish Agarwala
<thetish@google.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, C...