Fantastic Pretraining Optimizers and Where to Find Them
Authors: name: Kaiyue Wen, name: David Hall, name: Tengyu Ma, name: Percy Liang

AdamW has long been the dominant optimizer in language model pretraining,
despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We
posit that two methodological shortcomings have obscured fair comparisons and
hindered practical adoption: (i) unequal hyperparameter tuning and (ii) limited
or misleading evaluation setups. To address these two issues, we conduct a
systematic study of ten deep learning optimizers across four model scales
(0.1B-1.2B parameters) and data-to-model ratios (1-8x the Chinchilla optimum).
We find that fair and informative comparisons require rigorous hyperparameter
tuning and evaluations across a range of model scales and data-to-model ratios,
performed at the end of training. First, optimal hyperparameters for one
optimizer may be suboptimal for another, making blind hyperparameter transfer
unfair. Second, the actual speedup of many proposed optimizers over well-tuned
baselines is lower than claimed and decreases with model size to only 1.1x for
1.2B parameter models. Thirdly, comparing intermediate checkpoints before
reaching the target training budgets can be misleading, as rankings between two
optimizers can flip during training due to learning rate decay. Through our
thorough investigation, we find that all the fastest optimizers such as Muon
and Soap, use matrices as preconditioners -- multiplying gradients with
matrices rather than entry-wise scalars. However, the speedup of matrix-based
optimizers is inversely proportional to model scale, decreasing from 1.4x over
AdamW for 0.1B parameter models to merely 1.1x for 1.2B parameter models.