Title: Optimal Scaling Needs Optimal Norm
Primary area: foundation or frontier models, including LLMs
Keywords: optimal scaling, hyperparameter transfer, scion

Abstract: Despite recent progress in optimal hyperparameter transfer under model and dataset scaling, no unifying explanatory principle has been established. Using the Scion optimizer, we discover that joint optimal scaling over model and dataset size is governed by a single invariant: the operator norm of the output layer. Across models with up to 1.3B parameters trained on up to 138B tokens, the optimal learning rate/batch size pair $(\eta^\ast, B^\ast)$ consistently shares the same operator norm value – a phenomenon we term norm transfer. This constant norm condition is necessary but not sufficient: while for each dataset size, multiple $(\eta, B)$ reach the optimal norm, only a unique $(\eta^\ast, B^\ast)$ achieves the best loss. We provide the first measurement of $(\eta^\ast, B^\ast)$ scaling with the dataset size for Scion and find the scaling rules consistent with those of Adam. Tuning learning rates per layer group can also improve the model performance, with the output layer driving the sensitivity and hidden layers benefiting from slightly lower learning rates. We provide practical insights on norm-guided optimal scaling and release our Distributed Scion (Disco) implementation with logs from over two thousand runs to support research on LLM training dynamics at scale.