How to Improve Sample Complexity of SGD over Highly Dependent Data?

1 INTRODUCTION . Stochastic optimization algorithms have attracted great attention in the past decade due to its successful applications to a broad research areas , including deep learning ( Goodfellow et al. , 2016 ) , reinforcement learning ( Sutton & Barto , 2018 ) , online learning ( Bottou , 2010 ; Hazan , 2017 ) , control ( Marti , 2017 ) , etc . In the conventional analysis of stochastic optimization algorithms , it is usually assumed that all data samples are independently and identically distributed ( i.i.d . ) and queried . For example , data samples in the traditional empirical risk minimization framework are assumed to be queried independently from the underlying data distribution , while data samples in reinforcement learning are assumed to be queried from the stationary distribution of the underlying Markov chain . Although the i.i.d . data assumption leads to a comprehensive understanding of the statistical limit and computation complexity of SGD , it violates the nature of many practical data-generating stochastic processes , which generate highly correlated samples that depend on the history . In fact , dependent data can be found almost everywhere , e.g. , daily stock price , weather/climate data , state transitions in Markov chains , etc . To understand the impact of data dependence on the convergence and complexity of stochastic algorithms , there is a growing number of recent works that introduce various definitions to quantify data dependence . Specifically , to analyze the finite-time convergence of various stochastic reinforcement learning algorithms , recent studies assume that the dependent samples queried from the Markov decision process satisfy a geometric mixing property ( Dalal et al. , 2018 ; Zou et al. , 2019 ; Xu & Gu , 2020 ; Qu & Wierman , 2020 ) , which requires the underlying Markov chain to be uniformly ergodic or has a finite mixing time ( Even-Dar et al. , 2003 ) . On the other hand , to analyze the convergence of stochastic optimization algorithms over dependent data , Karimi et al . ( 2019 ) assumed the existence of a solution to the Poisson equation associated with the underlying Markov chain , which is a weaker condition than the uniform ergodic condition ( Glynn & Meyn , 1996 ) . Moreover , Agarwal & Duchi ( 2012 ) introduced a φ-mixing property of the datagenerating process that quantifies how fast the distribution of future data samples ( conditioned on a fixed filtration ) converges to the underlying stationary data distribution . In particular , the φ-mixing property is more general than the previous two notions of data dependence ( Douc et al. , 2018 ) . While the aforementioned works leveraged the above notions of data dependence to characterize the sample complexity of various standard stochastic algorithms over dependent data , there still lacks theoretical understanding of how algorithm structure affects the sample complexity of stochastic algorithms under different levels of data dependence . In particular , a key algorithm structure is the stochastic update scheme , which critically affects the bias and variance of the stochastic optimization process . In fact , under i.i.d . data and convex geometry , it is well known that SGD achieves the sample complexity lower bound under various stochastic update schemes ( Bottou , 2010 ) , e.g. , single-sample update and mini-batch update . However , these stochastic update schemes may lead to substantially different convergence behaviors over highly dependent data , as they are no longer unbiased . Therefore , it is of vital importance to understand the interplay among data dependence , structure of stochastic update and convergence rate of stochastic algorithms , and we want to ask the following fundamental question . • Q : How does the structure of stochastic updates affect the convergence rate and sample complexity of stochastic algorithms over dependent data ? In this paper , we provide comprehensive answers to the above fundamental question . Specifically , we conduct a comprehensive study of the convergence rate and sample complexity of the SGD algorithm over a wide spectrum of data dependence levels under various types of stochastic updates , including periodic subsampling and mini-batch sampling . Our results show that SGD with both stochastic updates achieves a substantially improved sample complexity over the standard SGD under highly dependent data . We summarize our contributions as follows . 1.1 OUR CONTRIBUTIONS . We consider the following standard stochastic optimization problem . min w∈W f ( w ) : = Eξ∼µ [ F ( w ; ξ ) ] , ( P ) where the objective function f is convex and Lipschitz continuous , and the expectation is taken over the stationary distribution µ of the underlying data-generating process P. To perform stochastic optimization , we query a stream of dependent data samples from the underlying data-generating process . Specifically , we adopt the φ-mixing model to quantify the data dependence via a decaying mixing coefficient function φξ ( k ) ( see Definition 2.2 ) ( Agarwal & Duchi , 2012 ) . We study the convergence of the stochastic gradient descent ( SGD ) algorithm over φ-mixing dependent data samples under various stochastic update schemes , including data subsampling and mini-batch sampling . We first study the convergence of SGD over φ-mixing dependent data samples under the data subsampling update scheme . In particular , the data subsampling update scheme utilizes only one data sample per r consecutive data samples by periodically skipping r − 1 samples . With this data subsampling scheme , the subsampled data samples are less dependent for a larger subsampling period r. Consequently , we show that SGD with a proper data subsampling period achieves an improved sample complexity over that of the standard SGD in the full spectrum of the convergence rate of the φ-mixing coefficient . In particular , the improvement is substantial when the data is highly dependent with an algebraic decaying φ-mixing coefficient . Moreover , we study the sample complexity of SGD over φ-mixing dependent data samples under the mini-batch update scheme . Compare to the data subsampling update , mini-batch update can substantially reduce the mini-batch data dependence without skipping data samples . Consequently , mini-batch update leverages the sample average over a mini batch of data samples to reduce both the bias ( caused by the data dependence ) and the optimization variance . Specifically , we show that SGD with mini-batch update achieves an orderwise lower sample complexity than both the standard SGD and the SGD with data subsampling in the full spectrum of the convergence rate of the φ-mixing coefficient . We summarize and compare the sample complexities of these stochastic algorithms under different φ-mixing data dependence models in Table 1 . 1.2 RELATED WORK . Stochastic Algorithms over Dependent Data Steinwart & Christmann ( 2009 ) and Modha & Masry ( 1996 ) established the convergence analysis of online stochastic algorithms for streaming data with geometric ergodicity . Duchi et al . ( 2011 ) proved that the stochastic subgradient method has strong convergence guarantee if the mixing time is uniformly bounded . Agarwal & Duchi ( 2012 ) studied the convex/strongly convex stochastic optimization problem and proved high-probability convergence bounds for general stochastic algorithms under general stationary mixing processes . Godichon-Baggioni et al . ( 2021 ) provided the non-asymptotic analysis of stochastic algorithms with strongly convex objective function over streaming mini-batch data . In a more general setting , the stochastic approximation ( SA ) problem was studied in ( Karimi et al. , 2019 ) by assuming the existence of solution to a Poisson equation . Recently , Debavelaere et al . ( 2021 ) developed the asymptotic convergence analysis of SA problem for sub-geometric Markov dynamic noises . Finite-time convergence of reinforcement learning Recently , a series of work studied the finitetime convergence of many stochastic reinforcement learning algorithms over Markovian dependent samples , including TD learning ( Dalal et al. , 2018 ; Xu et al. , 2019 ; Kaledin et al. , 2020 ) , Q-learning ( Qu & Wierman , 2020 ; Li et al. , 2021 ; Melo et al. , 2008 ; Chen et al. , 2019 ; Xu & Gu , 2020 ) , fitted Qiteration ( Mnih et al. , 2013 ; 2015 ; Agarwal et al. , 2021 ) , actor-critic algorithms ( Wang et al. , 2019 ; Yang et al. , 2019 ; Kumar et al. , 2019 ; Qiu et al. , 2019 ; Wu et al. , 2020 ; Xu et al. , 2020 ) , etc . In these studies , the dependent Markovian samples are assumed to satisfy the geometric φ-mixing property , which is satisfied when the underlying Markov chain is uniformly ergodic or time-homogeneous with finite-states . Regret of Stochastic Convex Optimization There have been many known regret bounds for online convex optimization problem . Hazan ( 2017 ) has built the standardO ( √ T ) regret bound for online SGD algorithm with assuming the bounded gradient . Xiao ( 2009 ) introduces the regret bound of online dual averaging method . To our best knowledge , there is no high-probability guaranteed regret bound for mini-batch SGD algorithm with considering the data dependence . 2 PROBLEM FORMULATION AND ASSUMPTIONS . In this section , we introduce the problem formulation and some basic assumptions . Consider a model with parameters w. For any data sample ξ , denote F ( w ; ξ ) ∈ R as the sample loss of this data sample under the model w. In this paper , we consider the following standard stochastic optimization problem that has broad applications in machine learning . min w∈W f ( w ) : = Eξ∼µ [ F ( w ; ξ ) ] . ( P ) Here , the expectation is taken over the randomness of the data sample ξ , which is drawn from an underlying distribution µ . In particular , we make the following standard assumptions regarding the problem ( P ) ( Agarwal & Duchi , 2012 ) . Assumption 2.1 . The stochastic optimization problem ( P ) satisfies 1 . For every ξ , function F ( · ; ξ ) is G-Lipschitz continuous overW , i.e. , for all w , v ∈ W , |F ( w ; ξ ) − F ( v ; ξ ) | ≤ G‖w − v‖ . 2 . Function f ( · ) is convex and bounded below , i.e. , f ( w∗ ) : = infw∈W f ( w ) > −∞ . 3 . W is a convex and compact set with bounded diameter R , i.e. , supw , v∈W ‖w − v‖ ≤ R. To solve this stochastic optimization problem , one often needs to query a set of data samples from the distribution µ to perform optimization . Unlike traditional stochastic optimization that usually assumes that the data samples are i.i.d . we consider a more general and practical dependent datagenerating process as we elaborate below . Dependent data-generating process : We consider a stochastic process P that generates a stream of data samples { ξ1 , ξ2 , ... , } , which are not necessarily independent . In particular , the stochastic process P has an underlying stationary distribution µ . To quantify the dependence of the data generation process , we introduce the following standard φ-mixing model ( Agarwal & Duchi , 2012 ) , where we denote { Ft } t as the canonical filtration generated by { ξt } t. Definition 2.2 ( φ-mixing process ) . Consider a stochastic process { ξt } t with a stationary distribution µ . Let P ( ξt+k ∈ ·|Ft ) be the distribution of the ( t+k ) -th sample conditioned on Ft , and denote dTV as the total variation distance . Then , the process { ξt } t is called φ-mixing if the following mixing coefficient φξ ( · ) converges to 0 as k tends to infinity . φξ ( k ) : = sup t∈N , A∈Ft 2dTV ( P ( ξt+k ∈ ·|A ) , µ ) . Intuitively , the φ-mixing coefficient describes how fast the distribution of sample ξt+k converges to the stationary distribution µ when conditioned on the filtration Ft , as the time gap k → ∞ . The φmixing process can be found in many applications , which involve mixing coefficients that converge to zero at different convergence rates . Below we mention some popular examples . • Geometric φ-mixing process . Such a type of process has a geometrically diminishing mixing coefficient , i.e. , φξ ( k ) ≤ φ0 exp ( −ckθ ) for some φ0 , c , θ > 0 . Examples include finite-state ergodic Markov chains and some aperiodic Harris-recurrent Markov processes ( Modha & Masry , 1996 ; Agarwal & Duchi , 2012 ; Meyn & Tweedie , 2012 ) ; • Algebraic φ-mixing process . Such a type of process has a polynomially diminishing mixing coefficient , i.e. , φξ ( k ) ≤ φ0k−θ for some φ0 , θ > 0 . Examples include a large class of Metropolis-Hastings samplers ( Jarner & Roberts , 2002 ) and some queuing systems ( Agarwal & Duchi , 2012 ) .

Summary: In stochastic convex optimization (SCO) characterized by the following objective for convex $f$, $$\min_w f(w) := \mathbb{E}_{\xi\sim \mu}[F(w; \xi)],$$ we typically assume repeated access to the noise distribution $\mu$, for sampling i.i.d. random variables $(\xi_t)_t$. This access can be used to obtain unbiased estimates $\nabla F(w; \xi_t)$ of the true gradient $\nabla f(w)$. This oracle access can be used to implement stochastic gradient methods, which are well studied and known to have optimal sample complexity under certain regularity conditions.   On the other hand in online convex optimization, at each time step, we obtain a loss function $f_t$ from some function class $\mathcal{F}$, and we have to minimize some notion of regret. This can be modeled like SCO by defining, $f_t:= F(.; \xi_t)$, but since there was no restriction on $f_t$'s, the $\xi_t$'s here can have arbitrary dependencies over time. In particular, they could be chosen adversarially.   In between both these extremes is the setting where $\xi_t$ could have some dependency structure. One such structure called $\phi$-mixing was introduced by [Agarwal and Duchi](https://arxiv.org/abs/1110.2529). Under this dependence assumption, they provide upper bounds on the convergence of SGD. This paper generalizes these upper bounds to a sub-sampling variant of SGD as well as mini-batch SGD. It shows that mini-batch SGD essentially attains the $1/\epsilon^2$ sample complexity of for i.i.d. sampling when there is a low level of dependence in $\xi_t$'s, and more generally improves over vanilla SGD across various levels of dependence captured by the $\phi$-mixing assumption.