Stochastic optimization under time drift: iterate averaging, step-decay schedules, and high probability guarantees

We consider the problem of minimizing a convex function that is evolving in time according to unknown and possibly stochastic dynamics . Such problems abound in the machine learning and signal processing literature , under the names of concept drift and stochastic tracking . We provide novel non-asymptotic convergence guarantees for stochastic algorithms with iterate averaging , focusing on bounds valid both in expectation and with high probability . Notably , we show that the tracking efficiency of the proximal stochastic gradient method depends only logarithmically on the initialization quality when equipped with a step-decay schedule . 1 Introduction . Stochastic optimization underpins much of machine learning theory and practice . Significant progress has been made over the last two decades in the finite-time analysis of stochastic approximation algorithms ; see , e.g. , [ 1 , 2 , 6 , 7 , 8 , 21 , 24 , 29 , 30 ] . The predominant assumption in this line of work is that the distribution generating the data is fixed throughout the run of the process . There is no shortage of problems , however , where this assumption is grossly violated for reasons beyond the learner ’ s control . Indeed , data often shifts and evolves over time for reasons that may be independent of the learning process . Two examples are worth highlighting . The first is a classical problem in signal processing related to stochastic tracking [ 19 , 27 ] , wherein the learning algorithm aims to track over time a moving target driven by an unknown stochastic process . The second example is the concept drift phenomenon in online learning [ 14 , 28 ] , wherein the true hypothesis may be changing over time , as in topic modeling or spam classification . An important goal in online problems , and the one we adopt here , is to track as closely as possible an unknown sequence of minimizers or minimal values . The tracking error efficiency of stochastic algorithms in online settings is much less developed than sample complexity guarantees for static problems . We present finite-time efficiency estimates in expectation and with high probability for the tracking error of the proximal stochastic gradient method under time drift . Our results concisely explain the interplay between the learning rate , the noise variance in the gradient oracle , and the strength of the time drift . The high-probability results merely assume that the gradient noise and time drift have light tails . Moreover , none of the results require the objectives to have bounded domains . While conventional wisdom and previous work recommend the use of constant step sizes under time drift , we show in an important regime that a significantly better step size schedule is one that is geometrically decaying to a “ critical step size ” . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Related work . Our current work fits within the broader literature on stochastic tracking , online optimization with controlled increments , and high-probability guarantees in stochastic optimization . We now survey the most relevant literature in these areas . Stochastic tracking . Stochastic gradient-type algorithms for stochastic tracking and filtering have been the subject of extensive research in the past century . Most works have focused on the so-called least mean-squares ( LMS ) algorithm and its variants , which can be viewed as a stochastic gradient method on a least-squares loss-based objective . Other stochastic algorithms that have been studied in these settings with a larger cost per iteration include recursive least-squares and the Kalman filter [ 12 ] . Recent works have revisited these methods from a more modern viewpoint [ 5 , 23 , 32 ] . In particular , the paper [ 23 ] focuses on ( accelerated ) gradient methods for deterministic tracking problems , while [ 32 ] analyzes a stochastic gradient method for online problems that is adaptive to unknown parameters . The paper [ 5 ] analyzes the dynamic regret of stochastic algorithms for time-varying problems , focusing both on lower and upper complexity bounds . Though the proof techniques in our paper share many aspects with those available in the literature , the results we obtain are distinct . Online optimization with controlled increments . Online learning under concept drift was first considered by [ 22 ] and further developed in several papers [ 3 , 17 ] . In this literature , the data distribution is typically fixed over time and the rate of variation is stated in terms of the probability of disagreement of consecutive target functions , which is assumed to be upper bounded . Another line of work assumes a time partitioning with an expert in each time interval , and the goal is to compete with the expert in each segment . Closer to this work is [ 14 , 16 ] , where in the framework of online convex optimization the bounds are stated in terms of maximum regret over any contiguous time interval ; see also [ 5 , 9 , 26 ] . In contrast to these works , in our framework we state our bounds in the same spirit as in classical stochastic approximation , that is , in terms of distance to optimum and objective function gap , and we present results holding both in expectation and with high probability . High-probability guarantees in stochastic optimization . A large part of our work revolves around high-probability guarantees in stochastic optimization . Classical references on the subject in static settings and for minimizing regret in online optimization include [ 4 , 15 , 20 , 25 ] . There exists a variety of techniques for establishing high-probability guarantees based on Freedman ’ s inequality and doubling tricks ; see , e.g. , [ 4 , 15 ] . A more recent line of work [ 13 ] establishes a generalized Freedman inequality that is custom-tailored for analyzing stochastic gradient-type methods and results in best known high-probability guarantees . Our arguments closely follow the paradigm of [ 13 ] based on the generalized Freedman inequality . 1.2 Outline . The outline of the paper is as follows . Section 2 formalizes the problem setting of time-dependent stochastic optimization and records the relevant assumptions . Sections 3 and 4 present the main results of the paper . Specifically , Section 3 focuses on efficiency estimates for tracking the minimizer , while Section 4 focuses on efficiency estimates for tracking the minimal value . Proofs of the main results appear in Section 5 of the Supplement , and illustrative numerical results appear in Section 6 of the Supplement . 2 Framework and assumptions . 2.1 Stochastic optimization under time drift . We consider the sequence of stochastic optimization problems min x ϕt ( x ) : = ft ( x ) + rt ( x ) ( 1 ) indexed by time t ∈ N. We make the standard standing assumption that ( i ) each function ft : Rd → R is µ-strongly convex and C1-smooth with L-Lipschitz continuous gradient for some common parameters µ , L > 0 , and ( ii ) each regularizer rt : Rd → R∪ { ∞ } is proper , closed and convex . The minimizer and minimal value of ( 1 ) will be denoted by x ? t and ϕ ? t , respectively . Throughout , ‖ · ‖ denotes the ` 2-norm on Rd induced by the dot product 〈· , ·〉 . As motivation , we describe two classical examples of ( 1 ) that are worth keeping in mind and that guide our framework : stochastic tracking of a drifting target and online learning under a distributional drift . Example 2.1 ( Stochastic tracking of a drifting target ) . The problem of stochastic tracking , related to the filtering problem in signal processing , is to track a moving target x ? t from observations bt = ct ( x ? t ) + t , where ct ( · ) is a known measurement map and t is a mean-zero noise vector . A typical time-dependent problem formulation takes the form min x E t ` t ( bt − ct ( x ) ) + rt ( x ) , where ` t ( · ) derives from the distribution of t and rt ( · ) encodes available side information about the target x ? t . Common choices for rt are the 1-norm and the squared 2-norm . The motion of the target x ? t is typically driven by a random walk or a diffusion [ 12 , 27 ] . Example 2.2 ( Online learning under distributional drift ) . The problem of online learning under a distributional drift is to learn while the data distribution may change over time . More formally , one problem formulation takes the form min x E w∼D ( ut ) ` ( x , w ) + r ( x ) . where D ( ut ) is a data distribution that depends on an unknown parameter sequence { ut } , which itself may evolve stochastically . The evolution of ut is often assumed to be piecewise constant in t in online learning [ 14 , 28 ] . Algorithm 1 Online Proximal Stochastic Gradient PSG ( x0 , { ηt } , T ) Input : initial x0 and step size sequence { ηt } Tt=0 ⊂ ( 0 , ∞ ) Step t = 0 , . . . , T − 1 : Set gt = ∇̃ft ( xt ) Set xt+1 = proxηtrt ( xt − ηtgt ) Return xT Online proximal stochastic gradient method . The main goal of a learning algorithm for problem ( 1 ) is to generate a sequence of points { xt } that minimize some natural performance metric . The most prevalent performance metrics in the literature are the tracking error and the dynamic regret . We will focus on two types of tracking error , ‖xt − x ? t ‖2 and ϕt ( xt ) − ϕt ( x ? t ) . We make the standing assumption that at every time t , and at every query point x , the learner may obtain an unbiased estimator ∇̃ft ( x ) of the true gradient∇ft ( x ) in order to proceed with a stochastic gradient-like optimization algorithm . With this oracle access , the online proximal stochastic gradient method—recorded as Algorithm 1 above—in each iteration t simply takes a stochastic gradient step on ft at xt followed by a proximal operation on rt : xt+1 : = proxηtrt ( xt − ηtgt ) = arg min u∈Rd { rt ( u ) + 1 2ηt ‖u− ( xt − ηtgt ) ‖2 } . The goal of our work is to obtain efficiency estimates for this procedure that hold both in expectation and with high probability . Minimizer drift . The guarantees we obtain allow both the iterates xt and the minimizers x ? t to evolve stochastically . This is convenient for example when tracking a moving target x ? t whose motion may be governed by a stochastic process such as a random walk or a diffusion ( see Example 2.1 ) . Throughout , we define the minimizer drift at time t to be the random variable ∆t : = ‖x ? t − x ? t+1‖ . Clearly , an efficiency estimate for Algorithm 1 must take into account the variation of the functions ft in time t. Two of the most popular metrics for measuring such variations are the minimizer drift ∆t and the gradient variation supx ‖∇ft ( x ) −∇ft+1 ( x ) ‖ . Given identical regularizers , a bound on the gradient variation always implies a bound on the minimizer drift . Lemma 2.3 ( Gradient variation vs. minimizer drift ) . Suppose i , t ≥ 0 are such that the regularizers ri and rt are identical . Then we have µ‖x ? i − x ? t ‖ ≤ ‖∇fi ( x ? t ) −∇ft ( x ? t ) ‖ .

Summary: the paper deals with tracking the minimum of a convex function which is evolving in time according to an unknown stochastic dynamic. The racking problem is classical in signal processing and automatic control (identification of linear or non-linear systems which are changing over time). It is less familiar in ML despite the fact that the problem of concept drift is very important in current practice.  The paper presents finite-time bound both in expectation and in high-probability for the tracjking of a a proximal stochastic gradient algorithm. The proposed result displays the relations between the learning rate, the excess risk and tracking error.   The paper uses rather stringent technical assumptions. In particular, the average loss $f_t$ are $\mu$-strongly convex, and the gradient  $\nabla f_{t}$  is  $L$-Lipschitz continuous.  The noise $z_{t}:=\nabla f_{t}\left(x_{t}\right)-g_{t}$ is a martingale increment with uniformly bounded variance.  The paper first provides finite time bounds for $\mathbb{E}\left\|x_{t}-x_{t}^{\star}\right\|^{2}$ under the assumption that $\mathbb{E}\left\|x_{t}^{\star}-x_{t+1}^{\star}\right\|^{2} \leq \Delta^{2}$. The bound in Theorem 3.2. has a transient term, a fluctuation term and a tracking error. The fluctuation term is proportional to the gradient noise variance and the stepsize. The tracking error is inversely proportional to the square of the stepsize.  This result coincides to the classical tracking error for linear system identification (under the considered stochastic model for the parameter). The authors identify two regimes depending on the ratio  $\Delta / \sigma$.  The high drift-to-noise is not interesting (the stepsize should then be set like in the stationary case), but the low drift-to-noise  necessitates a  learning which is typicallt smaller that $1/2L$. A finite time bound is obtained in Theorem 3.2 (for constant learning rate) and Theorem 3.8 (for a step-decay schedule). In Section 3.3, the authors develop high-probability guarantees under the assumption that the tracking error is (conditionally to the past) sub-exponential and the gradient noise is sub-gaussian. The HP bound (for fixed stepsize) is given in Theorem 3.10.