The Surprising Agreement Between Convex Optimization Theory and
  Learning-Rate Scheduling for Large Model Training
Authors: name: Fabian Schaipp; name: Alexander Hägele; name: Adrien Taylor; name: Umut Simsekli; name: Francis Bach
Published: 2025-01-31 08:55:56

Abstract: We show that learning-rate schedules for large model training behave
surprisingly similar to a performance bound from non-smooth convex optimization
theory. We provide a bound for the constant schedule with linear cooldown; in
particular, the practical benefit of cooldown is reflected in the bound due to
the absence of logarithmic terms. Further, we show that this surprisingly close
match between optimization theory and practice can be exploited for
learning-rate tuning: we achieve noticeable improvements for training 124M and
210M Llama-type models by (i) extending the schedule for continued training
with optimal learning-rate, and (ii) transferring the optimal learning-rate
across schedules.