Benchmarking Optimizers for Large Language Model Pretraining
Authors: name: Andrei Semenov, name: Matteo Pagliardini, name: Martin Jaggi

The recent development of Large Language Models (LLMs) has been accompanied
by an effervescence of novel ideas and methods to better optimize the loss of
deep learning models. Claims from those methods are myriad: from faster
convergence to removing reliance on certain hyperparameters. However, the
diverse experimental protocols used to validate these claims make direct
comparisons between methods challenging. This study presents a comprehensive
evaluation of recent optimization techniques across standardized LLM
pretraining scenarios, systematically varying model size, batch size, and
training duration. Through careful tuning of each method, we provide guidance
to practitioners on which optimizer is best suited for each scenario. For
researchers, our work highlights promising directions for future optimization
research. Finally, by releasing our code and making all experiments fully
reproducible, we hope our efforts can help the development and rigorous
benchmarking of future methods.