Rethinking learning rate schedules for stochastic optimization
There is a stark disparity between the learning rate schedules used in the practice of large scale machine learning and what are considered admissible learning rate schedules prescribed in the theory of stochastic approximation.Recent results, such as in the super-convergence methods which use oscillating learning rates, serve to emphasize this point even more.", "One plausible explanation is that non-convex neural network training procedures are better suited to the use of fundamentally different learning rate schedules, such as the cut the learning rate every constant number of epochs method; note that this widely used schedule is in stark contrast to the polynomial decay schemes prescribed in the stochastic approximation literature, which are indeed shown to be optimal for classes of convex optimization problems.The main contribution of this work shows that the picture is far more nuanced, where we do not even need to move to non-convex optimization to show other learning rate schemes can be far more effective.In fact, even for the simple case of stochastic linear regression with a fixed time horizon, the rate achieved by any polynomial decay scheme is sub-optimal compared to the statistical minimax rate; in contrast the `cut the learning rate every constant number of epochs provides an exponential improvement compared to any polynomial decay scheme. ", Finally, it is important to ask if our theoretical insights are somehow fundamentally tied to quadratic loss minimization?Here, we conjecture that recent results which make the gradient norm small at a near optimal rate, for both convex and non-convex optimization, may also provide more insights into learning rate schedules used in practice.
This paper presents a rigorous study of why practically used learning rate schedules (for a given computational budget) offer significant advantages even though these schemes are not advocated by the classical theory of Stochastic Approximation.This paper presents a theoretical study of different learning rate schedules that resulted in statistical minimax lower bounds for both polynomial and constant-and-cut schemes.The paper studies the effect of learning-rate choices for stochastic optimization, focusing on least-mean-squares with decaying stepsizes