Practical Efficiency of Muon for Pretraining
Authors: name: Essential AI, name: Ishaan Shah, name: Anthony M. Polloreno, name: Karl Stratos, name: Philip Monk, name: Adarsh Chaluvaraju, name: Andrew Hojel, name: Andrew Ma, name: Anil Thomas, name: Ashish Tanwer, name: Darsh J Shah, name: Khoi Nguyen, name: Kurt Smith, name: Michael Callahan, name: Michael Pust, name: Mohit Parmar, name: Peter Rushton, name: Platon Mazarakis, name: Ritvik Kapila, name: Saurabh Srivastava, name: Somanshu Singla, name: Tim Romanski, name: Yash Vanjani, name: Ashish Vaswani

We demonstrate that Muon, the simplest instantiation of a second-order
optimizer, explicitly expands the Pareto frontier over AdamW on the
compute-time tradeoff. We find that Muon is more effective than AdamW in
retaining data efficiency at large batch sizes, far beyond the so-called
critical batch size, while remaining computationally efficient, thus enabling
more economical training. We study the combination of Muon and the maximal
update parameterization (muP) for efficient hyperparameter transfer and present
a simple telescoping algorithm that accounts for all sources of error in muP
while introducing only a modest overhead in resources. We validate our findings
through extensive experiments with model sizes up to four billion parameters
and ablations on the data distribution and architecture.