Stochastic optimization under time drift: iterate averaging, step-decay schedules, and high probability guarantees

We consider the problem of minimizing a convex function that is evolving in time according to unknown and possibly stochastic dynamics . Such problems abound in the machine learning and signal processing literature , under the names of concept drift and stochastic tracking . We provide novel non-asymptotic convergence guarantees for stochastic algorithms with iterate averaging , focusing on bounds valid both in expectation and with high probability . Notably , we show that the tracking efficiency of the proximal stochastic gradient method depends only logarithmically on the initialization quality when equipped with a step-decay schedule . 1 Introduction . Stochastic optimization underpins much of machine learning theory and practice . Significant progress has been made over the last two decades in the finite-time analysis of stochastic approximation algorithms ; see , e.g. , [ 1 , 2 , 6 , 7 , 8 , 21 , 24 , 29 , 30 ] . The predominant assumption in this line of work is that the distribution generating the data is fixed throughout the run of the process . There is no shortage of problems , however , where this assumption is grossly violated for reasons beyond the learner ’ s control . Indeed , data often shifts and evolves over time for reasons that may be independent of the learning process . Two examples are worth highlighting . The first is a classical problem in signal processing related to stochastic tracking [ 19 , 27 ] , wherein the learning algorithm aims to track over time a moving target driven by an unknown stochastic process . The second example is the concept drift phenomenon in online learning [ 14 , 28 ] , wherein the true hypothesis may be changing over time , as in topic modeling or spam classification . An important goal in online problems , and the one we adopt here , is to track as closely as possible an unknown sequence of minimizers or minimal values . The tracking error efficiency of stochastic algorithms in online settings is much less developed than sample complexity guarantees for static problems . We present finite-time efficiency estimates in expectation and with high probability for the tracking error of the proximal stochastic gradient method under time drift . Our results concisely explain the interplay between the learning rate , the noise variance in the gradient oracle , and the strength of the time drift . The high-probability results merely assume that the gradient noise and time drift have light tails . Moreover , none of the results require the objectives to have bounded domains . While conventional wisdom and previous work recommend the use of constant step sizes under time drift , we show in an important regime that a significantly better step size schedule is one that is geometrically decaying to a “ critical step size ” . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Related work . Our current work fits within the broader literature on stochastic tracking , online optimization with controlled increments , and high-probability guarantees in stochastic optimization . We now survey the most relevant literature in these areas . Stochastic tracking . Stochastic gradient-type algorithms for stochastic tracking and filtering have been the subject of extensive research in the past century . Most works have focused on the so-called least mean-squares ( LMS ) algorithm and its variants , which can be viewed as a stochastic gradient method on a least-squares loss-based objective . Other stochastic algorithms that have been studied in these settings with a larger cost per iteration include recursive least-squares and the Kalman filter [ 12 ] . Recent works have revisited these methods from a more modern viewpoint [ 5 , 23 , 32 ] . In particular , the paper [ 23 ] focuses on ( accelerated ) gradient methods for deterministic tracking problems , while [ 32 ] analyzes a stochastic gradient method for online problems that is adaptive to unknown parameters . The paper [ 5 ] analyzes the dynamic regret of stochastic algorithms for time-varying problems , focusing both on lower and upper complexity bounds . Though the proof techniques in our paper share many aspects with those available in the literature , the results we obtain are distinct . Online optimization with controlled increments . Online learning under concept drift was first considered by [ 22 ] and further developed in several papers [ 3 , 17 ] . In this literature , the data distribution is typically fixed over time and the rate of variation is stated in terms of the probability of disagreement of consecutive target functions , which is assumed to be upper bounded . Another line of work assumes a time partitioning with an expert in each time interval , and the goal is to compete with the expert in each segment . Closer to this work is [ 14 , 16 ] , where in the framework of online convex optimization the bounds are stated in terms of maximum regret over any contiguous time interval ; see also [ 5 , 9 , 26 ] . In contrast to these works , in our framework we state our bounds in the same spirit as in classical stochastic approximation , that is , in terms of distance to optimum and objective function gap , and we present results holding both in expectation and with high probability . High-probability guarantees in stochastic optimization . A large part of our work revolves around high-probability guarantees in stochastic optimization . Classical references on the subject in static settings and for minimizing regret in online optimization include [ 4 , 15 , 20 , 25 ] . There exists a variety of techniques for establishing high-probability guarantees based on Freedman ’ s inequality and doubling tricks ; see , e.g. , [ 4 , 15 ] . A more recent line of work [ 13 ] establishes a generalized Freedman inequality that is custom-tailored for analyzing stochastic gradient-type methods and results in best known high-probability guarantees . Our arguments closely follow the paradigm of [ 13 ] based on the generalized Freedman inequality . 1.2 Outline . The outline of the paper is as follows . Section 2 formalizes the problem setting of time-dependent stochastic optimization and records the relevant assumptions . Sections 3 and 4 present the main results of the paper . Specifically , Section 3 focuses on efficiency estimates for tracking the minimizer , while Section 4 focuses on efficiency estimates for tracking the minimal value . Proofs of the main results appear in Section 5 of the Supplement , and illustrative numerical results appear in Section 6 of the Supplement . 2 Framework and assumptions . 2.1 Stochastic optimization under time drift . We consider the sequence of stochastic optimization problems min x ϕt ( x ) : = ft ( x ) + rt ( x ) ( 1 ) indexed by time t ∈ N. We make the standard standing assumption that ( i ) each function ft : Rd → R is µ-strongly convex and C1-smooth with L-Lipschitz continuous gradient for some common parameters µ , L > 0 , and ( ii ) each regularizer rt : Rd → R∪ { ∞ } is proper , closed and convex . The minimizer and minimal value of ( 1 ) will be denoted by x ? t and ϕ ? t , respectively . Throughout , ‖ · ‖ denotes the ` 2-norm on Rd induced by the dot product 〈· , ·〉 . As motivation , we describe two classical examples of ( 1 ) that are worth keeping in mind and that guide our framework : stochastic tracking of a drifting target and online learning under a distributional drift . Example 2.1 ( Stochastic tracking of a drifting target ) . The problem of stochastic tracking , related to the filtering problem in signal processing , is to track a moving target x ? t from observations bt = ct ( x ? t ) + t , where ct ( · ) is a known measurement map and t is a mean-zero noise vector . A typical time-dependent problem formulation takes the form min x E t ` t ( bt − ct ( x ) ) + rt ( x ) , where ` t ( · ) derives from the distribution of t and rt ( · ) encodes available side information about the target x ? t . Common choices for rt are the 1-norm and the squared 2-norm . The motion of the target x ? t is typically driven by a random walk or a diffusion [ 12 , 27 ] . Example 2.2 ( Online learning under distributional drift ) . The problem of online learning under a distributional drift is to learn while the data distribution may change over time . More formally , one problem formulation takes the form min x E w∼D ( ut ) ` ( x , w ) + r ( x ) . where D ( ut ) is a data distribution that depends on an unknown parameter sequence { ut } , which itself may evolve stochastically . The evolution of ut is often assumed to be piecewise constant in t in online learning [ 14 , 28 ] . Algorithm 1 Online Proximal Stochastic Gradient PSG ( x0 , { ηt } , T ) Input : initial x0 and step size sequence { ηt } Tt=0 ⊂ ( 0 , ∞ ) Step t = 0 , . . . , T − 1 : Set gt = ∇̃ft ( xt ) Set xt+1 = proxηtrt ( xt − ηtgt ) Return xT Online proximal stochastic gradient method . The main goal of a learning algorithm for problem ( 1 ) is to generate a sequence of points { xt } that minimize some natural performance metric . The most prevalent performance metrics in the literature are the tracking error and the dynamic regret . We will focus on two types of tracking error , ‖xt − x ? t ‖2 and ϕt ( xt ) − ϕt ( x ? t ) . We make the standing assumption that at every time t , and at every query point x , the learner may obtain an unbiased estimator ∇̃ft ( x ) of the true gradient∇ft ( x ) in order to proceed with a stochastic gradient-like optimization algorithm . With this oracle access , the online proximal stochastic gradient method—recorded as Algorithm 1 above—in each iteration t simply takes a stochastic gradient step on ft at xt followed by a proximal operation on rt : xt+1 : = proxηtrt ( xt − ηtgt ) = arg min u∈Rd { rt ( u ) + 1 2ηt ‖u− ( xt − ηtgt ) ‖2 } . The goal of our work is to obtain efficiency estimates for this procedure that hold both in expectation and with high probability . Minimizer drift . The guarantees we obtain allow both the iterates xt and the minimizers x ? t to evolve stochastically . This is convenient for example when tracking a moving target x ? t whose motion may be governed by a stochastic process such as a random walk or a diffusion ( see Example 2.1 ) . Throughout , we define the minimizer drift at time t to be the random variable ∆t : = ‖x ? t − x ? t+1‖ . Clearly , an efficiency estimate for Algorithm 1 must take into account the variation of the functions ft in time t. Two of the most popular metrics for measuring such variations are the minimizer drift ∆t and the gradient variation supx ‖∇ft ( x ) −∇ft+1 ( x ) ‖ . Given identical regularizers , a bound on the gradient variation always implies a bound on the minimizer drift . Lemma 2.3 ( Gradient variation vs. minimizer drift ) . Suppose i , t ≥ 0 are such that the regularizers ri and rt are identical . Then we have µ‖x ? i − x ? t ‖ ≤ ‖∇fi ( x ? t ) −∇ft ( x ? t ) ‖ .

Summary: This work considers the problem of stochastic gradient descent for time varying optimization. Based on the analysis of SGD in this setting, the problem actually contains two qualitatively different regimes: one with high drift-to-noise ratio, and one with low drift-to-noise ratio. The paper further provides tight convergence analysis (both in expectation and in high probability) for the two regimes.