The Road Less Scheduled
Authors: name: Aaron Defazio; name: Xingyu; name: Yang; name: Harsh Mehta; name: Konstantin Mishchenko; name: Ahmed Khaled; name: Ashok Cutkosky
Published: 2024-05-24 16:20:46

Abstract: Existing learning rate schedules that do not require specification of the
optimization stopping step T are greatly out-performed by learning rate
schedules that depend on T. We propose an approach that avoids the need for
this stopping time by eschewing the use of schedules entirely, while exhibiting
state-of-the-art performance compared to schedules across a wide family of
problems ranging from convex problems to large-scale deep learning problems.
Our Schedule-Free approach introduces no additional hyper-parameters over
standard optimizers with momentum. Our method is a direct consequence of a new
theory we develop that unifies scheduling and iterate averaging. An open source
implementation of our method is available
(https://github.com/facebookresearch/schedule_free).