Learning from time-dependent streaming data with online stochastic algorithms
Authors: Antoine Godichon-Baggioni
Nicklas Werge
Olivier Wintenberger
Venue: Accepted by TMLR ()

Abstract: This paper addresses stochastic optimization in a streaming setting with time-dependent and biased gradient estimates. We analyze several first-order methods, including Stochastic Gradient Descent (SGD), mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages. Our non-asymptotic analysis establishes novel heuristics that link dependence, biases, and convexity levels, enabling accelerated convergence. Specifically, our findings demonstrate that (i) time-varying mini-batch SGD methods have the capability to break long- and short-range dependence structures, (ii) biased SGD methods can achieve comparable performance to their unbiased counterparts, and (iii) incorporating Polyak-Ruppert averaging can accelerate the convergence of the stochastic optimization algorithms. To validate our theoretical findings, we conduct a series of experiments using both simulated and real-life time-dependent data.

Published in Transactions on Machine Learning Research (07/2023)
Learning from Time-dependen, Streaming Data with Online Stochastic Algorithms
Antoine Godichon-Baggione
Laboratoire de Probabilités, Statistique et Modélisation Sorbonne Université
Nicklas Werge
Laboratoire de Probabilités, Statistique et Modélisation Sorbonne Université
Olivier Wintenberger
Laborato're de Probabilités, Statistique et Modélisation Sorbonne Unéversité
Reviewed on OpenReview: /ttps://openreee optp://veepreeeee...mmmm net/ forum? id- id=kdffBulul
Abstract
This paper addresses stochastic optimization in a streaming setting with time-dependent and biased gradient estimates. We analyze several first-order methods, including Stochastic Gradient Descent (SGD), mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppen averages. Our non-asymplotic analysis establishes novel heuristics that link dependence, biases, and convexity levels, enabling accelerated convergence. Specifically, findings demonstrate that (i) time-varying mini-batch SGD methods have the capability to break long- and short-range dependence structures, (ii) biased SGD methods can achieve comparable performance to their unbiased counterparts, and (iii) incorporating Polyak- Ruppert averaging can accelerate the convergence of the stochastic optimization algorithms. To validate our theoretical findings, we conduct a series of experiments using both simulated and real-life time-dependent data.
1 Introduction
Machine learning has experienced remarkable growth and adoption across diverse domains, revolutionizing various applications and unleashing the potential of data-driven decision-making (Bishop & Nasrabadi] 20066
Goodfellow et al.] 2016 Sutton & Barto 2018 Hastie et al.) 2009 Hazan et al. 2016 Shalev-Shwartz et al. 2012). With this proliferation of machine learning, a significant. volume of new data has emerged, including streaming data that flows continuously and poses unique challenges for learning algorithms. Unlike traditional static datasets, streaming data requires algorithms to adapt in real-time, continuously updating their models to accurately predict new samples.
At the heart of machine learning lies optimization, the process of finding optimal model parameters that minimize the objective funetion and extract meaningful insights from the data (Abu-Mostafa et al. 2012). Stochastic Optimization (SO) methods have emerged powerful tools for addressing optimization tasks in machine learning (Kushner & Yin 2003 Nemirovskij & Yudin 1983 Shalev-Shwartz & Ben- Ren--avvi 2014).
These SO methods overcome the scalability limitations of traditional batch learning techniques and allow for efficient. processing of streaming data (Bottou et al. 2018).
Among the various SO mothods, Stochastic Gradient Descent (SGD) and its variants stand out the workhorses of the field (Robbins Monro 1951). extensively utilized in machine learning tasks (Hardt al. Xiao 2009 SGD and its variants provide practical and
2016 Shalev-Shwartz al. 2011 Zhang 2004

---

Published in Transactions on Machine Learning Research (07/2023)
efficient approach to optimize complex models by updating the model parameters using stochastic estimates of the gradients. These methods have proven their effectiveness in wide range of applications, including deep learning, natural language processing, and computer vision (Goodfellow et al. 2016 Sutton & BBrtto 2018 Hastic et al. 2009).
However, traditional analyses of SO problems often assume that gradient estimates are unbiased and drawn independently and identically distributed (i.id.) from an unknown data generation process (Cesa-Biande et al., 2004). Unfortunately, real-world data rarely adheres to this idcalized assumption. Streaming data, in particular, introduces additional complexity, dependencies and biases can arise due to time-dependency or other factors inherent in the data generation process. These dependencies and biases pose significant. challenges for conventional SO methods.
Notably, SGD-based methods have demonstrated the ability to converge even under biased gradient estimates [Ajallocian & Stich 2020 Bertsckas] 2016 d' d'Aspremont] 2008: Devolder et al.) 2011 Gorbunov et al. 2020a by
Schmidt et al. 2011]. However, the theoretical understanding of the convergence properties of SGD in the presence of biased gradients is not yet well-established. While empirical studies have shown promising results
in specific applications (Agarwal & Duchi 2012
Chen & Luss 2018 Karimi et al. 2019 Ma et al. 2022
Schmidt et al. 2011), more rigorous investigations are necessary to generalize these findings and comprehend the underlying mechanisms.
Contributions. In this paper, we explore SGD-based methods in the context of streaming data. We extend the analysis of the unbiased i.i.d. case by Godichon Baggion et al. (2023) to include time-dependency and biasedness. By leveraging their insights, we investigate the effectiveness of first-order SO methods in streaming sctting, where the assumption of unbiasod i.i.d. samples no longer holds. Our non-asymptotic analysis establishes novel heuristics that bridge the gap wetween dependence, biases, and the convexity levels of the SO problem. These heuristics enable accelerated convergence in complex problems, offering promising opportunities for efficient optimization in streaming settings. Our contributions can be summarized as follows:
Non-asymptotic Convergence Rates of Time-varying Mini-batch SGD-based methods. We present non-asymptotic convergence rates specifically tailored for time-varying mini-batch SGD- based methods under time-dependency and biasedness. These convergence rates offer valuable insights into achieving and enhancing convergence in applications involving time-dependent and biased inputs, providing comprehensive understanding of the optimization process in streaming settings.
Breaking Long- and Short-term Dependence. Our study demonstrates the effectiveness of SGD-based methods in overcoming long- and short-range dependence by leveraging time-varying mini-batches. These mini-batches are carefully designed to counteract the inherent dependency structures present in streaming data, leading to improved performance and convergence. This contribution expands the applicability of SGD-based methods to scenarios where dependence poses a challenge.
Robustness to Biased Gradients. We show that biased SGDD-baed methods can converge and achieve accuracy comparable to unbiased ones, as long as the bias is not excessively large. This finding highlights the robustness of SGD-based methods in the presence of bias, broadening their applicability to scenarios where biased gradient estimates are encounteeed.
Accelerated Convergence with Dolyak-Ruppent Averaging. Our study reveals that incorporat- ing Polyak-Rupper averaging into SGD-based methods accelerates convergence (Polyak & Juditsky 1992 Ruppert 1988). Importantly, our findings emphasize the continued efficacy of this technique in challenging streaming settings characterized by dependence structures and biases. Furthermore, by combining Polyak-Ruppert averaging with the variance reduction capabilities offered by time-varying mini-batches, we obtain the best of both worlds. This powerful combination not only enhances the convergence rate but also increases robustness by reducing the variance.
Overall, our contributions extend the scope of SO by considering time-dependent and biased gradient estimates in streaming framework. We provide non-asymptotic convergence rates, demonstrate the effectiveness of SGD-based methods in handling dependence and bias, and highlight the accelerated convergence achieved
2

---

Published in Transactions on Machine Learning Research (07/2023)
through Polyak-Rupper averaging. These findings enhance understanding of SO in streaming settings and offer practical strategies for researchers and practitioners to optimize their models in real-time applications.
Organization. In section 2 we introduce the streaming framework and lay the groundwork for our non- asymptotic analysis. We provide an overview of key concepts, definitions, and assumptions, with a focus on the dependency structures.
section 2.1 we discuss the assumptions related to the objective function, while section 2.2 focuses on the assumptions concerning the gradient estimates, particularly the crucial role of a-mixing conditions in validating the dependency structures. We then present our streaming variants of the SGD methods in section
In section B we present our convergence results, considering both our streaming SGD variant with and without Polyak-Ruppert averaging. We provide a detailed discussion of each result, highlighting connections to previous work in the field. The proofs of our results are provided in appendix [
Finally, in section we present experimental results that validate our findings. We conduct experiments on synthetic and real-life Lime-dependent streaming data, showcasing the practical implications of our work.
2 Problem Formulation, Assumptions, and Methods
We consider SO problems of the form
min (F(00) (0) OE
(1)
In the formulation [1. A Rd represents the parameter vector of interest. The objective function F is defined the expected value of the random loss function f(0;E), which depends the parameter vector O and the random variable ç. To minimize F, we rely on estimates of its gradients. Specifically, estimate F by evaluating the gradient estimates of f on sequence of samples (E)).
In our streaming setting, we have access to a sequence of time-varying mini-batches (Gx), rather than knowing the true underlying distribution of &. Each &, consists of n, E N individual data points, represented by the mini-batch Ee..... We extend this notion of time-varying bini-batches by defining f() f(0: &.). Consequently, (0) constitutes a sequence of differentiable (possible non-convex) random loss functions (Nesterov et al. 2018 Nemirovskij & Yudin 1983). Hence, each fe(0) comprises n4 N individual lossos, For example, consider the scenario where & represents a matc- of input-output pairs ((....BB..)) 1E In this case, for a model class {he heee, we can express (0) as a combination of a loss function 1 and a regularizer Q:
represented by the mini-batch (f.,,,...,,,,,,
(0) = ((,,, (x()))))) + 9(()).
2.1 Assumptions on Objective Functions: Quasi-strong Convexity and Lipschitz Smoothness
In accordance with previous work by Bach & Moulines (011)); Gower et al. (2019) Nguyen et al. (2019) we adopt certain assumptions regarding the objective function F. Firstly, we assume that F possesses a unique global minimizer 8* E O, where is a closed convex set in Rd. These assumptions align with techniques utilized in (online) convex optimization (Boyd & Vandenberghe, 2004 Nesterov al.] 2018 Hazan al.)
2016 Shalev-Shwartz et al. 2012). In addition, we assume that the objective function F is u-quasi-strongly ConveX Karimi et al. 2016: Necoara et al. 2019):
Assumption 1 (u-quasi-strong convexity). The objective function F is differentiable with voF(0*) = 0 and there evists conslant μ > 0 such that v0 E O, we have
F(0") 2 F(0) + (VoF()), - 0) + 11 0* 2
(2)
The y-quasi-strong convexity assumption serves as a relaxed version of strong convexity for the SO problem, providing more conservative notion. Various objective functions F employed in machine learning applications have been extensively investigated and documented by Teo et. al. (2007). Researchers have also explored
3

---

Published in Transactions on Machine Learning Research (07/2023)
milder degrees of convexity, such the Polyak-Lojasiewvan condition (Polyak) 1963
Lojasiewicz
1963
studied
by Karimi et al. 6)166) Gower et al. (2021) for SGT) methods, and their Suppert-Polyaa average investigated
by Gadat & Panloup 2023) under a Lurddyka-Lojasievizz-ty- condition (Kurdyka
1998
Lojasiewicz 1963).
Relaxing the assumption of strict convexity is essential in practice to ensure the robustness and adaptiveness
of algorithms, particularly for non-strongly convex SO problems (Bach & Moulines
2013
Nemirovski el. al.,
2009 Necoara et al., 2019: Khaled & Richtárík, 2023
F being j-quasi-strongly convex does not guarantee the j-quasi-strong convexity of (fr). It is crucial to understand that while the objective function F satisfies the j--uasi-stronn convexity assumption (2) the individual loss functions (fr) may not exhibit the same property. This distinction plays an essential role in our convergence analysis as it interacts with the time-dependennc of the problem and the presence of biases, In section 3 we will explore this relationship in detail and investigate its implications for the convergence behavior. Specifically, we will examine how the level of dependence, biases, and the p-quasi-strong convexity conditions intertwine and affect the convergence propertics of the optimization process.
To analyze the Polyak-Ruppert Ruppert averaging estimate, need to impose additional smoothness assumptions the objective function F' following the framework established in Bach & Moulines (2011); Godichon Baggioni ct al. (2023). These smoothness assumptions ensure the necessary conditions for the convergence analysis of the Tolyak-Rupper method (Ruppert 1988 Polyak & Juditsky 1992).
Assumption 2 (Cv- and smooooeeeeess). The objective function F have Cv-Lipschitz continuous gradients around 8*, i.c., there erists a constant Cr > O such that 40 E O,
VvF(()= - V&F(O')I S C=110 10
(3)
Next, the Hessian of F is C5-Liischittt-onniivvoo around 8r, that is, there exists a constant C/ 2 such that 40 O
- S 10
(4)
As highlighted in Bottou et al. (2018), 1 the assumption stated in Assumption guarantees that the gradient V&F does not exhibit arbitrary voniations. This property makes V, a valuable guide for reducing the value of F. In deterministic optimization, smooth optimization typically achieves faster convergence rates compared to non-smooth optimization. However, in the context of SO, the benefits of smoothness are limited to improvements in the associated constants (Nesterov et al. 2018
2.2 Assumptions on Gradient Estimates: Dependence, Bias, Expected Smoothness, and Gradient Noise
Let J. aff : <() denote the natural filtration of the SO problem 11. The gradients of the time-varying mini-batches (f)( serve as estimates of VoF(O). Unlike classical assumptions that typically demand unbiased (and uniformly bounded) gradient estimates (Godichon- Baggioni Brogichon-aaggioni et al. 2023), we adopt a more flexible approach. We relax these constraints by allowing the gradients (V)f((O) to be time-dependent and biased estimates of F() in the following way:
Assumption 3-p (Dyee-depeneence and Byrr-iaa)). For each t 1, the random function Veft is square- integrable, F-mmeasurable, and for a positive integer p, there exists some positive sequence (vi) and constants Dv, B, 2 such that
E EIE VoF())" "D 6 0*1 + BE).
Discussion of Assumption 3-p. Assumption 3-p follows the form of mixing conditions for weakly dependent sequences, indicating that the dependence diminishes at rate determined by it. The verification of Assumption 3-p can be accomplished using moment inequalities for partial sums of strongly mixing sequences RRi 2017,, which is commonly referred to as short-range dependence; it is also known by other aliases such as short memory or short-range persistence.
Indeed, for any positive integer p. Assumption 3 can be bounded above as follows:
[[[[([a]](())]=)) VoF(0) E (o)((() F()))PP n, (E[[
(6)
4

---

Published in Transactions on Machine Learning Research (07/2023)
using Jensen's inequality. In 6 St Ei= (0) VoF(()) represents a d-dimensional vector, and (o) ni" Di (0). Let (Voft,)) be a strictly stationary sequence, and assume the existence of some r > such that Su Qz)(()) < co, where Q(z) denotes the quantile function of V/ Suppose (Vo.t.s) ,) is strongly a-mixing according to Rosenblatt (1956), with strong mixing coefficients (at)121 satisfying ar ((t-prr(2r-2p)) Then, by Rio (2017) Corollary 6.1), we find that [[[| - Onn This implies that (6) is at most O(n, p/2). This encompasses several linear, non-linear, and Markovian time series, c.g., see Bradley [:005]; Doukhan (2012) for more examples of other mixing coefficients of weak dependence
and their relationships.
In relation to the form of Assumption 3-pl this indicates that By + in this case. However, it is possible to have B, = 0 in unbiasod examples, we will demonstrate later in scction | Another example is the short-term dependent Markovian case, where 1 = n 1/2 and the dependency constant D2 in Assumption 3-p can be explicitly expressed using the mixing times of the Markov chain. Nagaraj et al. (2020) worked out precise calculations for least squares regression. To summarize, we encounter short-range dependence when
(6) decays at most with On, p/2). Conversely, we encounter long-range dependence when 6 decays slower
than O(n,
The classical convergence analysis for SGD-based methods relies on the assumption of (uniformly) bounded gradient estimates, which is restrictive and applicable only to certain types of losses (Bottou et al. 2018 Nguyen el al. 2 In our work, we depart from this approach and instead adopt the assumptions introduced by [Bach& & Moulines (2011); ; Gower al. (2019), who analyzed SGD for quasi-strongly convex objectives as expressed in (2). Specifically, we make assumptions regarding the expected smoothness of gradients (Veft)
and the expected finiteness of (Vof:(0")):
Assumption 4-p (C,-expected smoothness). For a positive integer p, there erists some posilive constant Cn such that v0 E θ, (ef Vef((0-)(P] s CR 0*11%)
Assumption 5-p (d--gradient noisc). For a positive integer p, there erists some positive sequence (01)121 such that ([]]V So.
As for Assumption 3-p the verification of Assumption 5-p can be achieved using a-mixing conditions and analogous arguments as those employed above, resulting in of boing of the order On,, It is worth noting that in the classical i.i.d. case with unbiased gradients, alternative relaxations of Assumptions 4-p and 5-p
exist, such as the ER or ABC assumptions (Gower et al. 2021 Khaled & Richtárikk (023). However, since our primary focus is on introducing time-dependent and biased gradient estimates, we do not delve into these alternative assumptions.
In summarr, the assumptions we have presented (Assumptions 3-p to 5-p offer a more relaxed framework
compared to the standard assumptions found in the litcrature (Benveniste et al.] 2012 Kushner & Yin 2003
Bach & Moulines 2011 Boodichon-Baggione et al.) 2023 Dieuleveut et al.) 2017 Dieuleveut & Bach 2016).
Our assumptions are designed to accommodate wide range of scenarios, including both classical examples and more intricate models that involve learning from time-dependent data with biases. By adopting these assumptions, we provide a flexible and applicable framework for analyzing SO methods in such settings.
2.3 Stochastic Streaming Optimization Methods
To address the SO problem (1) within streaming setting, we employ the Stochastic Streaming Gradient (SSG) method proposed by Modichon-Baggions et al. (2023). The SSG algorithm is defined as follows
n Volts (0.-1).
7t
(SSG)
0, -0
(7)
n4
Here, 7t denotes the learning rate, which satisfies the conditions Σ 74 do and E 7 < 0c: Robbins & Monro 1951.. It is worth noting that when n, = 1, the SSG algorithm reduces to the usual SGD method.
In many models, there are often constraints imposed on the parameter space, necessitating a projection of the parameters. To address this, we introduce the Projected Stochastic Streaming Gradient (PSSG) estimate,
5

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm 1: Stochastic streaming gradient estimates (SSG/ (SSC/PSSC/ASC//PSSC))
Inputs :00 E e C Rd, project: True False, average: Truc Falsc
Outputs: : 0, 8. (resulting estimates)
Initialization. 0n E TRd
for each t 2 1, a time-varying mini-batch of ne data arrives, do
0. + 81-1 2t E Veft, (0c-1)
/* update */
if project then
0, + (o))
/* project */
if average then
0, + (Ni-1/N.)OL-1 + (me/Ne)02-1
/* average */
defined as
06-1 PPo (0. nh
(PSSG)
7t Volci (0-1))
(8)
where Pe represents the Euclidean projection onto O given by Pe(0) arg gen 0112-
If one were to examine the trajectory of the stochastic gradients (VoJt,i), it would become apparent that they exhibit high noise levels and lack robustness, which can lead to slow convergence or even prevent convergence altogether. Therefore, it intuitively makes sense to use mini-batches of gradient estimates within each inn as this reduces variance and facilitates the adjustment. of the learning rate (7t), ultimately improving the quality of each iteration.
Within this streaming framework, are interested in incorporating acceleration techniques into the existing One important extension is the Polyak-Rupper procedure (Polyak & Juditsky 1992 Ruppert 1988). which ensures optimal statistical efficiency without compromising computa- tional costs. The Averaged Stochastic Streaming Gradient (ASSG) method, denoted as (ASSC)/(APSSG), is defined by
algorithms presented in and S
1-1 0.
(ASSG)
Σ N 1=0
14110
Here, N₂ - n; represents the accumulatod sum of observations/ To ensure fair comparison of our streaming methods, it is crucial to evaluate them in terms of the number of observations used, namely N₁- Similarly, use APSSG to denote the (Polyak-Rupperr) averaged estimate of PSSG presented in ( These averaging methods sequentially aggregate past estimates, resulting in smoother curves (i.e., variance reduction in the estimation trajectories). and accelerate convergence (Polyak & Juditsky 1992
The pseudo-code for these streaming estimates, including to 99). is presented in algorithm Additionally, can be modified to a weighted average version, giving more weight to the latest estimates. This modification improves convergence while limiting the impact of poor initializations. Examples of such algorithms can be found in Boyer & Godichon- -odiioon-Baggioni (2022): Mokkadem & Pelletier (2011).
The popularity of SGD methods has prompted efforts to improve their efficiency, robustness, and user- friendliness. As a result, numerous variants, including second-order methods like Newton's method and other extensions, have been extensively explored and discussed in works such as Boyd & Vandenberghe (2004);
Nesterov et al. (2018) Byrd ot al. (2016).
The choice of the learning rate (7x) significantly affects the convergence of SGD. learning rate that is too small leads to slow convergence, while an excessively high learning rate may prevent convergence due to oscillations around the minimum of the loss function. Therefore, there is strong motivation for the
In practice, as we handle data sequentially, we employ the rewritten formula (N--1/N)00- + (mI/N))0:1, with 0 0. This allows us update the averaged estimate efficiently.

---

Our objective is Published in Transactions on Machine Learning Research (07/2023)
development of adaptive learning rate mechanisms that require less manual fine-tuning and offer improved user-friendliness Moreover, the idea of having per-dimension learning rates that adjust individually as the convergence progresses holds potential advantages (Godichon-Baggioni & Tarrago 2023
In Bottou et al. 20188 a comprehensive overview varions SO methods is provided, covering both convex and non-convex optimization scenarios. The paper also delves into strategies for parallelization and distribution, aiming to accelerate the SGD updates and improve overall performance.
3 Convergence Analysis
In this section, we analyze the convergence of the streaming methods presented 7
9
to establish non-asymplolic bounds 5, = E 0*1 and & E[l00, 0 112] which solely depend on the SO problem parameters.
To achieve this, we derive a recursive relation for 6, that allows us to non-asymptotically bound the estimates in and 8 for any given sequences of (ye), (vt), (ot), and (nc):
Lemma 1. Let 6; E[[|0: -0-00) where (0t) either follows the recursion in or and 8-1 to 5-p hold for 2. Then, for any (7t), (v:), (or), and (nt), we have
Suppose Assumplions
B " + 20 &. $ [1 (1- 2DDDun 20 +
(10)
/
To prove this lemma, we adapt classical techniques from stochastic approximations (Benveniste et al., 2012 Kushner & Yin 2003), originally developed for the unbiased i.id. setting, to our specific streaming setting. Our adaptation incorporates the time-dependence and biases introduced by the new assumptions, Assumptions 3-p and 5-p This enables us gain novel insights into the interplay between dependence and biases, leading to deeper understanding of the convergence properties of SO methods. Notably, the recursive relation for & explicibly depends on (Tr); (u+), (at), and (nt), which, to the best of our knowledge, is novel contribution. It highlights the connection between u-quasi-strong convexity and the dependence term D, as discussed in section 2.1 We will later refer to this connection as ensuring μ y-quasi-strong converity through den-decrgginng streaming batches when the dependence sequence (vi) is linked to the sime-varying mini-batches (n:).
In section 3.3 we will delve into the convergence analysis of the Polyak-Ruppers averaging estimate On. Our goal is to establish non-asymptotic bound on 8 which quantifies the convergence properties of the averaging method. This analysis builds upon the standard decomposition of the loss terms, enabling the emergence of the Cramér-Rao term (Bach & Moulines) 2011 Gadat & Panloup 2023 Godichon-Baggioni
et al. 2023). To facilitate this analysis, we also need to consider fourth-order moments. However, we will provide a comprehensive exploration of this aspect detail in section 3.3
3.1 Learning Rate (7t). Uncertainty Terms (pr) and (ot), and Time- VareinVryyng Mini-batches (nt)
Before proceeding with the convergence analysis in sections 3.2 and 3.3 we first specify the functional forms of the learning rate (1x), the uncortain terms (vr) and (at), and the streaming batch (nz).
Learning Rate (7:). Following Godichon- Goodchon-Baggioni
2023 we adopt the learning rate given by
= C.nita
where C, > 0, 3 E 10, and is chosen based the expected streaming batches This learning rate allows us to assign more weight to larger streaming batches through the hyperparameter B.
Uncertainty Terms (vr) and (at) from Assumptions 3-p and 5-p The uncortainty terms (v: and (at) in our analysis play a crucial role in capturing the dependence and noise inherent the SO problem. As
Fg.g. see Momentum Qian [999), Nesterov accelerated gradient. NNsterov 1983), Adagrad Duchi al. 201 Adadelta
Zeiler 2012). RMSprop Tieleman al. 2012), and Adam Kingma Bal 2014.

---

Published in Transactions on Machine Learning Research (07 (07/2023)
discussed in section 2.2 these terms can be viewed functions of the streaming batches (n)). We define these terms as follows:
Vt n and 0: Co
where v E oo), o E 10, and C. > 0. By setting V4 =,",
, we introduce short-range dependence when 112, 2, 0o) and long-range dependence when v E (0, 1/2). Hence, the classical i.i.d. case (Godichon-Bare Baghion et al. 2023 corresponds to // + oo. The choice a E [0.1/2] aligns with the framework proposed by Godichon- Godichon-Baggioni al. (2023), where = 1/2 represents the i.i.d. case. When 1/2, allows for the presence of noisier outputs.
1 E
Time-varying Mini-batches (ne). Inspired by the work of Modichon-Baggione et al. (0023) we adopt formulation for timo-varying mini-batches (n+) given by
ne = [C,19),
where Cp E N and p E [0, 1),, ensuring that n, N. This formulation encompasses various scenarios, including classical (online) SGD methods for Cp = and p= 0, (online) mini-batch SGD procedures with both constant and time-varying sizes when Cp E N and p = 0 or E 1), respectively, as well as the Polyak-Ruppen average of (ouline) time-varying mini-bat batches. For ease of reference, we use the term streaming batch size to refer to Cp and the term streaming rate for p.
3.2 Stochastic Streaming Gradients
Theorem 1 (SSG/PSS /PSG)). Let 5g [ 0 11])) where (6:) either follows the recursion in Suppose Assumptions explicit constants Cs, C.. Cy U such that for p3 E (1/2, 1), we have
and 3-D to 5-p hold for p 2. If fl μ
> 0, then there exist
N,C,N, N₂ 1tp8-a
C B
C%
C.C.C,
8, 50 exp
(11)
I-8-0 C&Cp
20z
2a My
2/20-61m0
N
14e
N
1+p
An explicit version of this bound is given in appendia A
Sketch of proof. Under Assumptions and (6,) defined by
3
with p 2, we can establish recursive relation for to
5-p
8, < [1 (n (m-22Daa 2D.uu) + 20 p - BEUl + 20722
for any form of (1)) (p2), (ot), and (nz). The non-asymptotic upper bound of this recursive relation 10 can be explicitly derived using classical techniques from stochastic approximations (Benveniste el al. 2012 Kushner & Yin 2003). Bounding the projected estimate in (8) can directly be obtained by noting that (P) 1 I Biilo 11.) Alternatively, the projected estimate (8) can also be proved by assuming bounded gradient, which replaces Assumptions 4-p and 5-p) Detailed discussions and alternative proofs can be found in works such as Bach & Moulines (201]): Mudichon-Baggionn et al. (2023). The discontinuity in p emerges from the introduction of an indicator that determines whether (vr is constant or decreasing. When (v1) is constant, it becomes challenging to obtain a bound that closely approaches μ.. Conversely, when (v)) decreases, there a possibility of achieving bound that can come as close as possible to μ.. In the i.i.d. case, this choice corresponds the optimal strong convexity paramcter Alternatively, we can derive better bound by considering a continuous trade-off where we minimize (vr) in the bound while minimizing the other terms. This continuous trade-off has the potential to provide smoother transition between the cases of constant and decreasing (vt).
Boodichon-Baggion. et al. 2023).
Related work. Our work exlends and aligus with previous studies in the field. First, our results replicate the outcomes of the unbiased i.id. case investigated in Godichon- Baghionn et al. (2023), specifically when B, = and = 1/2. This demonstrates the consistency of our findings with regard the unbiased i.i.d. setting. Additionally, our results are in line with the research conducted by Bach & Moulines (2011), which focused the unbiased i.i.d. case in non-streaming setting, i.e., when C, and p 0.
8

---

Published in Transactions on Machine Learning Research (07/2023)
Bound objective function. If the objective function F has gradients that are Cy-Lipschitz continuous (Assumption 2) the inequality given by (11) provides a bound on the function values of F. Specifically, we have ()) - F[ & C by applying Cauchy-Schwarz' 'ss inequality.
Ensuring y-quasi-strong convexity through non-decreasing streaming batches. The positivity of the dependence penalized convexity constant Pv = μ - 1 1p=
crucial for all terms of (11 This constraint arises from the fact that while the objective function F' is j-quaaiistrongly convex, the loss functions (ff) may not possess the same convexity properties, e-g., see discussion in section 2.1 This is
demonstrated in sections 4.1.2 and 4.1.3 for ARCH models (Werge & WWintenberger 2022
So, how should we interpret M When the streaming rate P =0, 0, the positivity of Py depends on the convexity constant H4, the streaming batch size Cp, and the imposed dependencies specified in Assumption 3-p If the dependence quantity D2 is sufficiently large that Hy becomes non non-positive, it becomes necessary to choose a larger streaming batch size Cp to ensure Hy remains positive. In other words, strong dependency structures reduce convexily, but large mini-batches counteracts this effect.
An alternative approach to ensuring convexity is by employing increasing streaming batches, i.e, setting the strcaming rate p > O. This method provides greater robustness as we no longer need to determine a specific value for the mini-batch size Cp to maintain the positivity of μv. However, combining both approaches is ideal as it ensures convexity while a larger C, effectively reduces variance, resulting in a more favorable outcome.
In section we delve into the challenge of calibrating Cp and P appropriately to achieve both stability and convergence in the optimization process. The goal of this calibration is to strike a balance between taking frequent gradient steps for faster convergence and ensuring the positivity of pay- To accomplish this, it is crucial to choose sufficiently large Cp and p that can maintain the positivity of ny while allowing for many gradient steps as possible. This delicate balance is essential in achieving optimal performance in the optimization process.
Variance reduction with larger streaming batches Cp. Unsurprisingly, larger streaming batches Cp have a variance-reducingg elfect, as illustrated section [ This effect is explicitly demonstrated in each term of [1]. Interestingly, the variance reduction resulting from large mini-batcher scales with batch size but does not increase the docay rate of 8,
Decay of initial conditions. The initial conditions in the first term of 111 decay sub-exponentially. A
detailed expression for this term can be found in appendix A The last term of I1 represents the noise term, which is influenced by the gradiont noise as described in Assumption 5-p When a - pp8 E (1/2, the noise For instance, if we set a 2/3, B 1/3, and a 1/2, the noise term decays as ON, 2/3) for any streaming rate p E [0, 1). This decay behavior is illustrated in Baggion et al. (2023). Notably, in the unbiased cases (B, = 0), this noise term corresponds to the asymptotic term (Godichon- (Godichon-Baggioni et al., 2023). Morcover, when a a B < 20, the noise assymptoii term is positively influenced by larger streaming batches Cp- This effect is demonstrated in section
term decays at rate of O(N, (y(2a-8)+a)/(1t)))
Behavior under biasedness, B, > O. The second term in [1] represents a pure bias term determined by the bias quantity B,, the level of dependence V, and the (dependence-penalizee convexity constant Pv- Importantly, the bias term is independent of the learning rate 7 C.mptt but depends on the time-varying mini-batches n, = Cpll, i.e, through the streaming batch size Cp and the streaming rate p.
The dependence term exhibits a scaling of D(N,-200/( we would need p= and v = 1/2. It remarkable that Theorem accommodatess both long- and short-range dependence. While long-range dependence leads to slow convergence (slower than ONN, 1(2)- a positive streaming rate p can break long-range dependence. In summary, by increasing the streaming batch size, we preserve p-quasi-strong convexity and alleviate both long- and short-term dependence. This results in a bound of 6, = Omaxx [n.>0) (2ppp/((Lo)) N, (p(20 (2000)+a/(((-))
For instance, to achieve a decay rate of O(,,
9

---

Published in Transactions on Machine Learning Research (07 (203))
3.3 Averaged Stochastic Streaming Gradients
In our subsequent analysis, our main focus is on the averaging estimate 0n, which is defined in ( This averaging estimate is obtained from either the SSG estimate in (7) or the PSSG estimate in (8). Our analysis relies on the standard decomposition of the loss terms, which allows the Cramér-Rao term to emerge (Bach) & MMulines 2011 Gadat & Panloup, 2023: Godichon- Baggioon et al. 2023). To facilitate this analysis, it is necessary to consider the fourth-order momenti, which require the assumptions Assumptions 3-p to 5-1 to hold for p = 4. Moreover, based on Assumption 5-D) where o, = C, with [0, 1/2], we introduce an additional assumption regarding the covariance of the score vectors associated with the parameter vector l*. Assumption 6 (Covariance of the scores). There erists non-negative self-adjoint operator E such that Vt 2 L, E Ve Vef(((*)T] E C₁ 2 0, and of E (0, 1/2].
= Et, where i is a positive symmetric matria with Tr(Ee) C',,"" 2n'
In some cases, such as the i.i.d scenario and certain unbiased situations discussed in section 4. 4.1.1 Assumption 5
holds with o = 1/2 and C* = 0 (Godichon- Baggioni et al.) 2023). This condition applics to scenarios
characterized by short-range dependence, as shown in section 4.1.1 with o = 1/2 (and C² > 0). Conversely,
the long-range dependence case occurs when a < 1/2 (and C² 0). Under Assumption 0 we can derive the
dominant term A/N,, where A = Tr F of the Cramer-Rar lower bound for the case of unbiased i.dd. samples. Specifically, the bound can be
This assumption also enables the establishment
expressed
Bi100 10 s O(ANT)) O(N, with b>1.
In order to analyze the projected averaged estimate (a.k.a. APSSG), additional assumption ion is made to avoid the computation of the sixth-order moment. We assume that (Vefe) is uniformly bounded on the compact O. However, it is worth mentioning that the derivation of the sixth-order moment can be found in Modichon--aggiinn (2016).
Assumption 7 (Ge-bounded gradients). Let De infogae ||0 00 > with de denoting the boundary of O. Moreover, there exists Ge such that VI 2 1, supnee [o] S Cg a.s.
Theorem 2 (ASSG/PASSC) Let &, E E| or [ with On given by (9) where (0.) either follows the
recursion in 7 or B). Suppose Assumptions (0.) follows the recursion in Sh. If pv
1 to 6 hold for p = 4. In addition, Assumption 7 must hold if
{p= CD.C > 0, then for a p3 E (1/2, 1), we have
1-2(0+0')
A S N +
2 A C 20++
2 C²' Cp
20-A)
(a=1/2}
(a<1/2) +
1420(0=0'1 a(1+p)
(12)
N a(tt)
NN,
((20-1)4 10e
))
+0
maxx N₂
N 20++
+ 1 (Bu Vi,
(13)
with
plo lw)
1- itp
1+2
6/2 po 2(112)
2
1):
1 =
max
N₂
20F
.N.
N₂
.N,
N₂ I+2
where 8 = =0 (p(20 33 + a)) + 40 min(p(20 3) + ax 2pr}. An explicil version of this bound is given in appendit A
Related work. It. is important to note that the bound presented Theorem 2 is for the root mean square error, while our discussions focus on 5, without taking the root. Similar to the unbiased i.idd. case explored in Goodihonn-aggionn et al. (2023), the dominant term of 8, in {12) and (13 is A/N,, which achieves the asymptotically optimal Cramer- Rao lower bound (Murata & Amari 1999) Each term in (12) directly stems from Assumption 0 Importantly, these terms remain unaffected by the choice of learning rate (tr), but
depend on the time-varying mini-batches (n.). As discussed in Gadat & Panloup (2023). the bound of & can be interpreted as bias-variance decomposition between the leading terms in 12 and the remaining terms in 13
Ensuring s-quusi--trong convexity through non-decreasing streaming batches. The interpretation of py in Theorem 2 is similar to that in Theorem 1 In both cases, the positivity of pa plays crucial role in
10

---

Published in Transactions on Machine Learning Research (07/2023)
all terms of [1], despite not being immediately apparent. Analogous to the insights gained from Theorem 1 increasing the streaming batch size allows us to preserve y-quasi-strong convexity and alleviate both long- and short-term dependence. By choosing a larger streaming batch size, we ensure the positivity of Ne, as demonstrated in the experiments conducted for ARCH models in sections 4.1.2 and 4.1.3
Accelerated decay through Polyak-Ruppert averaging. Through Supyakt averaging, is possible to achieve the leading term A/N,, which is known to attain the asymptotically optimal Cramer-Rao bound in the i..d. case GGodichon-Baggioni et al., 2023 Consequently, can achieve the optimal and irreducible rate of 8, O '). This is always accomplished in the unbiased case (B, =0) with 1/2, even under short-range dependence.
In the specific case of a = 1/2, 12 and 13 simplify significantly. The last two lerms of 12 become negligible o' > 0. The first term of 113 decays at the rate O (44))222222444)))/(H+))))
O(N (ol2o-))+a) (L+p))
2/3, 8 1/3, and = 1/2) yields a decay of Choosing at 3 such that a + (22r /3 B) = 2/3 (e-g,,
O(N, 4/$) for any p. Hence, by setting a = 2/3 and 8 = 1/3 when o 1/2, the first term of 13 robustly
achieves ONN,71/3 1/3) for any streaming rate p. Similarly, the last term of 13 is OLV, (LLN,-(1/2+)//CCC when a = 2/3 and 3 1/3. In summary, Theorem 2 with = 2/3, B 1/3, and = 1/2 simplifies to the bound:
for any
के S
A + +0 3 + [BB ( NN. N
-(1/--1
21+00
(14)
Variance reduction from larger strcaming batches Cp.
averaging accelorates conver- gence and, in particular, the decay rate. However, by laking large mini-batches, we can also achieve variance reduction. As discussed earlier (for Theorem [], larger mini-batches scale the error, which is particularly beneficial in the initial stages. Therefore, combining averaging and mini-batches offers the best of both worlds, resulting in a better slope (decay rate) and intercept.
Behavior under biasedness, By > O. The bias term B, affects only Vr, except for the second term in 13,, which impacts the docay rate Inereasing the strcaming rate p diminishes the negative influence of the bias term. Surprisingly. 1 approaches zero as t increases indefinitely for any v. However, achieving the desired decay rate of 8, ( excludes long-range dependence. Nevertheless, in our experiments (see section 4). the bias term does not seem to have significant impact.
4 Experiments
In this section, we present the details and results of our experiments, aimed at evaluating the performance of our streaming SGD-bbaed methods on both synthetic and real-world data. Our objective is to demonstrate findings that highlight the following: (i) time-varying mini-batch SGD methods have the capability to break long- and short-range dependence structures, (ii) biased SGD methods can achieve comparable performance to their unbiased counterparts, and (iii) incorporating Dolyak-Rupperr averaging can accelerate the convergence. These findings collectively showcase the effectiveness and potential of our streaming SGD-basod methods.
4.1 Synthetic Data
A way to illustrate our findings is by use of classical methods that aim to model and predict an underlying sequence of real-valued time-series (X,): here s is short notation for indexing the sequence of observations, XN. XN,,,. X
with N. = Ei= n4. The AutoRegressive (AR), Moving- Average (MA), and AutoRcgressive Moving-Average (ARMA) models are the most well-known models for lime-series (Brockwell & Davis 2009 Box et al. 2015 Hamilton 2020). The standard time-series analysis often relies on independence and constant noise, but it can be relaxed by, e.g., the AutoRegressive Conditional Hoteroskodasticctty (ARCH) model (Engle. 1982). Online learning algorithms of (both stationary and non-stationary) dependent time-series have been studied in Agarwal & Duchi (201)); Anava et al. (013))
Wintenberger 2021
11

---

Published in Transactions on Machine Learning Research (07 (07/2023)
4.1.1 AR Model
A process (X.) is called a (zero-mean) AR(1) process, if there exists real-valued parameter such that X, = 0x + es, where (ex) weak white noise with zero mean and variance o²- To illustrate the versalility of our results, we construct some (heavy-tailed) noise processes with long-range dependence: the noisiness is integrated using Student's t-distribution with degroes of froedom above four, denoted by (2,). The long-range dependence is incorporated by multiplying (zx) with the fractional Gaussian noise G2 Bx+
B, (H),, where (B,(H) is a fractional Brownian motion with Hurst index H E (0, (G,(H) can also be seen as a (zero-mean) Gaussian process with stationary and self-similar increments (Nualart 2006).
Well-specified case. Consider the well-specified case, in which, we estimate an AR(1) model from the underlying stationary AR(1) process X, = 0 xxx1 + 6, with 10 < 1. The squared loss function fe(0) = T4 -1 ++ 1 -0XN, -+ with gradient Vofz(0) = 2n, 2i XN.
(
1 1 OKNN--i- 1)-
Thus, the objective function is F(0) (o? - / 10/11110000 (0) + o² as [[x,] O and E[X²] of/(1 (8)))),
yielding Vo FoF = 20 61) (0*))). Assumption 3-p with 2 can be written as follows:
-
4(0 0 ) 2mm))
00 1- 1-
1
(1 (8 )2)
This implies that Assumption 3-p holds when (X,) has bounded moments, which is satisfied under the As a result, we can deduce that D2 > 0, Bg = 0, and in is O(n, Similarly, Assumption 5-D can be verified in the same manner, with Ø, being Om, 1 Additionally, Assumption holds with CF = =202/11 - )) and C² = 0, while Assumption holds with Σ = / (0*) and
natural constraint 10^1
Σn = O. Furthermore, for an AR(1) process (X,) constructed using the noise process = v (H)zz, with Hurst index H 2 1/2, one can verify that vs and of are Onn using the self-similarity property Nourdin 2012).
Misspecifted case. Now, assume that the underlying data generating process follows a MA(1)-proces) X, = 6a + + f3-1, with o' IR. The misspecification error of fitting an AR(1) model to a MA(1) process can be found by minimizing F(0)= B[(X, xx,-1) = o1 (o" 0)2 (6))),, where VoF(O) =
2(0- ((0-d")a -=")=² 200 Thus, 0" arg ming F(0) arg ming 0)2 + (0))² is strictly convex function in 0, have VoF(0) Thus, for any o² E I, we have 0 E 1/2/ With this in mind,
e 2(0 Φ²) 28(x²)² 0 < 20(1 + (6)) =20- + 0 00=0"(11 (Ø))))
can conduct our study of fitting an AR(1) model to a MA(1) process with o* drawn randomly from R (figure [b] Furthermore, this reparamctrization trick can be used to verify Assumption 3-p in the following way:
BIJ BIIEE/ef((0)) VoF(O)] 4(0 - 8-)²
for (EN...),
ni
where for (EN..) is finite function depending on the moments of (EN...) and o+ Consequently, we establish D, > B, = 0, and v being On, Similarly, using the reparametrization trick, we can verify that o, is O(n, 1/2 (Assumption 5-ph.
4.1.2 ARCH Model
In time series analysis, modeling hederoscedasticity of the conditional variance is crucial component, especially in capturing phenomena like volatility clustering in financial time series. ARCH models are widely recognized models that effectively incorporate this characteristic. process (c.) is called an ARCH(1) procoss with parameters ag and o₃ if it satisfies
E. Ts
(15)
0 Co + a1
where ag > 0 and 0n 2 O ensures the non-negatyvity of the conditional variance process (o and the innovations (2,) is white noise. We employ the Quasi-Maximmnn Likelihood (QML) procedure for the
Thhe verification available longer version appendix 4The verification available longer version appendix
12

---

Published in Transactions on Machine Learning Research (07/2023)
statistical inference outlined in Werge & Wintenberger (2022; : the quasi likelihood losses is given by (x()) = (c (0) with first-order derivalive
Vofa(0)= (2:00)
(0 - - (0
where (0) )T. Verification of Assumptions 3-p to 5-p can be done using mixing conditions; Francq & Zakoian 2019 Theorem 3.5) showed that stationary ARCH processes are geometrically 3 3-mixing, which implics a-mixing as well. Observe that the loss function (f.) itself is not strongly convex but only the objective function F may be strongly convex; convexity conditions of ARCH processes was investigated in Wintenberger (2021). This makes the parameters challenging to estimate in empirical applications as the optimization algorithms can quickly fail or converge to irregular solutions. There are different ways to overcome lack of convexity: first, projecting the estimates such that the (conditional) variance process (o?) stays away from zero (and close to the unconditional variance). Second, in the specific example of ARCH example using
model, one could also recover convexity by implementing variance targeting techniques; Generalized ARCH (GARCH) models can be found in Werge & Wintenberger (2022). To simplily our analysis we consider stationary ARCH(1) processes, where we fix ag at 1 and initialize it at 1/2.
4.1.3 AR-ARCH Model
We complete our experiments by considering an AR models with ARCH noise: the process (X,) is called an AR(1)-ARCH(1) process with parameters 0, oig, and an, if it satisfies
X, 0X2-1 + Es:
6 032s:
(16)
0² = 0g + a1
where the innovations (zx) is weak white noise. The statistical inference of this model is done using the squared loss for the AR-part and the QML procedure for the ARCH part, c.g., see sections 4.1.1 and 4.1.2
Assumptions 3-p and 5-p can be verified by Doukhan (1994] Proposition 6), which showed that ARMA- ARCH
processes are 3-mixing
4.1.4 Discussion
Our experimental evaluation assesses the performance using the mean quadratic error Bi]10,, 0-]]²] across one thousand replications. The initial values 0g and 0 are randomly generated according to the specifications of the models. We intentionally omit projecting our estimates to highlight the errors stemming from dependence and lack of convexity. By averaging over multiple replications, we observe a reduction in variability, which primarily benefits the SSG method. The experiments aim to demonstrate the impact of the choice of Cp and p on the measures of dependence Dv, bias B,, and the penalized convexity constant fv associated with dependence. To facilitate comparison between different data streams, set the parameters C, 1, a = 2/3, and B 0 as fixed values.
The experiments described in sections 4.1.1 to 4.1.3 can be found in figure I here (C, = 1, 0} corresponds
to the classical SG descent and its (Polyak- -urpk--- average estimate, [C, = 64, p = 0} is a mini-batch ASG/ASSG, and {C, = 64, p = 1/2] is an increasing SSG/ASSG ASSG with initial batch size of C, 64.
Let's examine the AR cases, both well-specified and misspecified, as shown in figures Ia and 1b These figures display the results for long-range dependent white noise processes with Hurst index of H 3/4. It is evident that each pair of data streams converges, but the traditional SG method exhibits significant initial noise, particularly affecting the average estimate period without affecting its decay rate. An improvement could be achieved by modifying our average estimate to a weighted average version, which could mitigate the impact of poor initializations (Mokkadem & Pelletier 2011 Boyer & Godichon- Godichon-Baggioni 2022). Nonetheless, even with this noise, the ASSG method still demonstrates better convergence. Both methods show a noticcable reduction in variance as C, increases, which is particularly advantageous the early stages. However, excessively large streaming batch sizes Cp may hinder convergence due to fewer iterations. Additionally,
13

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Figure Simulation of various data streams n4 Cpte. See section 4.1 for details.
(a) AR(1): well-specified case. See section 4.1.1 for (b) AR(1): misspecciiidd case. See section 4.1.1
details.
details.
p=00.p=00
C,=1,
ASSG --11.p000
ASSG: c--1.p0000
SSG: p=00
10f
SSG: p=0.0
p=0.0
ASSG: C,a8,
SSG: p==8.p=05
SSG: c,=8,p=05
ASSG: p=0.5
ASSG: p=0.5
Quadratic Mean Error
6,-64, p=o.
Quadratic Mean Error
6,-64 p=0.0 ASSG: c,=64, p=0.0
ASSG: =,= p=0.0
104
SSG: 6, p=0.5
SSG: C,-64, p=0.5
C p-0.5
G2 6-64, p-0.5
102
10a
109
10-2
10
10
10
100
101
102
10
10
10a
109
101
102
103
10
105
Nr
N
(c) ARCH(1). See section
1.1.2
for details.
(d) AR(1)-ARCH(1) See section
1.1.3
for details.
10a
p=00 ASSG: p=08,
ASSG: p=0.0
103
SSG: p=8.p=000
SSG: C,=8, p-0.5
ASSG: p-0.00
S50: -=
10a
SSG: C,=64, p=0.0
ASSG: C,88, p=0.5
ASSG: C,=64, p=0.0
Quadratic Mean Error
-44 p=0.0
Quadratic Mean Error
C,-64, pwo.5
101
ASSO: G6- p=0.0
ASSG: G,=64, p=0.5
SSG: G= p=0.5
ASSG: 6644
20a
10-1
20a
1
10a
101
10a
N: 10a 10
10a
10a
10a
107
10a
10a
10a
N:
figures la and 1b indicate improved decay for the SSG methods the streaming rate p increases. On the other hand, improvements in the ASSG method are not observed, we do not leverage the potential of using more observations through the 3 parameter, which could accelerate convergence (refer to section 4.2h. It is surprising that we do not observe any effects from the long-range dependent white noise processes, but this seems to be an artifact effect in the proof, as fourth-order moments are required (i.e., Assumptions 3-p to 5-p) with p = 4). Thorefore, we conjecture that only sccond-order moment propertics are responsible for the behavior of simulations and that 1/2 holds even for long-range dependent white noise processes,
proven in section 4.1.1
Moving on to figures Ic and ld we present the experiments for stationary ARCH(1) models, both with and without an AR component, as outlined in sections 4.1.2 and 4.1.3) respectively. These figures ralustrate the lack of convexity when using small streaming batch sizes Cp: such as the traditional SG descent and average estimate (C, 1,p = 0), which lcads to divergence. convexity reflected in the absence of a positive μe, which can only be counteracted by larger streaming batch sizes Cp- Moreover, figure ld excludes the traditional SG descent {C, =1,p 0} due its lack of
is important note that the lack of
14

---

Published in Transactions on Machine Learning Research (07/2023)
convexity. The figure demonstrates that larger (Cp 64) and non-decreasing (p 2 0) streaming batches converge even under challenging settings.
4.2 Historical Hourly Weather Data
To illustrate our methodology on real-life time-dependent streaming data, we consider some historical hourly weather data F This dataset comprises approximately five years (roughly 45000 data points) of high temporal resolution hourly measurements encompassing various weather attributes, including temperature, humidity, and air pressure, The dataset encompasses thirty-six cities, resulting in a dimensionality of d = 36. In our analysis, we specifically focus on the hourly temperature measurements. To account for monthly and annual seasonality, we preprocess the data by subtracting the corresponding monthly and annual averages. This ensures that the analysis is centered around the deviations from the expected seasonal patterns.
4.2.1 Geometric Median
In the presence of noisy data, robust estimators such as the geometric median are preferred. The geometric median, introduced by Haldane (Haldane 1948), is a robust generalization of the real median. Its efficiency
makes it particularly well-suited for handling tigh-dimensional streaming data (Cardot et al.) 2013 Godichon Baggioni 2016). To estimate the goometric median of X Rd we minimize the objective function F(0)
E|| 0 - ||||| using stochastic gradient estimales Vof(0)= = (X 0)/l - O||. The existence, uniqueness, and robustness (breakdown point) of the geometric median have been studied in Kemperman (1987): Gervini
2008). It is important to note that this objective function only possesses locally strong convexity properties (Cardot et al. 2013). However, by projecting the gradients, it is possible to adapt the proof of optimality in a streaming setting, as demonstrated by Gadat & Panloup (2023). Alternatively, if X is bounded, one can adapt the approach of Cardot et al. (2012) to the streaming setting, which shows that the estimates remain bounded and projection is unnecessary in such cases. In our analysis, we choose not to project our estimates, as doing so would obscure the errors we intend explore.
4.2.2 Discussion
Similar to previous evaluations, we assess the performance using the mean quadratic error of the parameter estimates over one hundred replications, denoted as EillO,, 1*. In this comparison, we contrast our estimates with the geometric median estimate obtained through Weiszfeld's algorithm (Weiszfeld & Plastria
2009). We suppose our data are standard Gaussian random variables centered (0,) Isidd, where each 0; is randomly selected from the range [-d, d]. To align with the reasoning of Cardot et al. (013)) we set C, vd and choose a= 2/3.
Figure presents the results of the geometric median estimation, following the procedure outlined in section 4.2.1 Despite the robustness of the geometric median, noticeable fluctuations observed in figure 2 which arisc from the timo-dependency and noise present in the weather mcasurements. Figure 2a emphasizes the importance of utilizing a mini-batch size C, to stabilize the opl imization process and ensure convexity through larger streaming batches Cp. However, to achieve reasonable convergence, it is crucial to employ increasing streaming batches with positive streaming rates p > O, as illustrated in figures 2b and 2c These figures demonstrate an enhanced decay of the SSG when the streaming rate p is increased. However, the lack of convergence improvement. in figure 2 can be attributed to the use of B 0, which neglects the potential benefits of leveraging additional observations to accelerate convergence. For further insights into this matter, refer to Godichon- -Baggion et al. (2023). As discussed after Theorem 2 one way to achieve this acceleration is by setting = 2/3 and 3 = 1/3. Figure 2d demonstrates that simply selecting 8 = 1/3 enables this accelcration. Morcover, B = 1/3 ensures optimal convergence robust to any streaming rate p. Selecting an appropriate 3 > particularly crucial when C, is large, as robustness a fundamental aspect of the geometric median method. Most surprisingly, excellent convergence with a final error as low as 10 can be achieved by combining increasing streaming batches with averaging. This is exemplified in figure 2
with Cp = 64, p > 0, and 1/3. Based on these real-life experiments and the discussion in section 4.1.4
TTh historical hourly weather dataset. can be found http:://ww...aggg....ca.aatttttt http://...../ttttteea///elaaagge historical-hourh soatall-oooolly-te---aaae
15

---

Published in Transactions on Machine Learning Research (07 (02)))
Figure 2: Geometric median for various data streams n4 = Cpte. See section 4.2 for details.
(a) Varying Co: P = 0, B
(b) Varying p; C2 = 1,
10a
Quadratic Mean Error
Quadratic Mean Error
10a
10a
100
ASSG: c,=1
c,=1
100
ASSG: p=0.0
SSG: p-0.0
10
C,=8
SSG: p=0.25
ASSG: C==
SSG: 6,=64
SSG: p=0.33
c,=64
ASSG: p=0.33
10-2
SSG: Cp=128
10-1
SSG: p=0.5
C,-128
109
101
102
10a
10a
109
101
10a
103
10a
N.
N:
(c) Varying p. Cp
61,
8
(d) Varying p. C, 64, 3 1/3
10a
100
Quadratic Mean Error
Quadratic Mean Error
10
10-2
10-1
10-3
p-0.
ASSG: p00.0
SSG: p=0.25
SSG: p=0.25
ASSG: p=0.25
ASSG: p=0.25
10
SSG: p=0.5
ASSG: p=0.33
SSG: p=0.5
ASSG: p=0.33
ASSG: p=0.5
ASSG: p=0.5
10
100
101
102
10a
10.
100
101
10a
10a
104
N.
N:
we speculate that the sequence of scores (O*)) constitutes a martingale difference sequence that extends beyond our specific examples. Notably, findings indicate that 1/2 holds even in the presence of long-range dependence. Thus, the complexity associated with Theorem 2 appears to be artifact of the proof, which relies on Assumptions to with 4.
5 Conclusions and Future Work
In this paper, explored SGD-based methods in the context of streaming data. We extended the analysis of the unbiased i.i.d. case by (Codichon-Baggion) et al.] (2023) to include time-dependency and biasodncss. By leveraging their insights, we investigated the effectiveness of first-order SO methods in a streaming setting, where the assumption unbiased i.i.d. samples no longer holds. Our non-asymptotic analysis established novel heuristics that bridge the gap between dependence, biases, and the convexity levels of the SO problem. These heuristics enabled accelerated convergence in complex problems, offering promising opportunities for efficient omiizizttion in streaming settings. Specifically, findings demonstrated that. (i) time-varying
16

---

Published in Transactions on Machine Learning Research (07/2023)
mini-batch SGD methods can break long- and short-range dependence structures, (ii) biased SGD methods can achieve comparable performance to their unbiased counterparts, and (iii) incorporating Polyak-Ruppert averaging can accelerate the convergence. We validated our theoretical findings by conducting series of experiments using both simulated and real-life time-dependent data.
Future perspectives. There are several ways to expand our work about stochastic streaming algorithms: (a) we can extend our analysis to include streaming batches of any size (and not as a function of streaming batch size Cp and streaming rates p), e.g.. (Godichon-Baggione et al. 20233 discuss random streaming batches
with negative and positive drift. (b) an extension to non-strongly convex objectives could be advantageous
it will provide more insight into how we should choose our learning rates (Bach & Moulines 2013 Nemirovski
et al. 2009 Necoara et al. 2019 Gadat & Panloup 2023). (c) learning rates should be made adaptive so they are robust to poor initialization and require less tuning; an adaptive learning rate is essential for practitioners
as it builds a form of universality across applications, e.g., see Duchi et al. (2011); Kingma & Ba (2014);
Godichon- Badichon-Baggion & Tarrago (2023). (d) non- non-paranne panaamartttic analysis could improve our thcoretical results for large values of d. (e) have focused on results in quadratic mean but another way to strengthen our
non-asymptotic guarantees could be high probability bounds (Durmus et al. 2021 2022); for any 8 (0, 1),
we could obtain bounds on the sequence {1lØ, 0*ll :t E N) that holds with probability at least 1 - 6.
References
Yaser S Abu-Mostafa,, Malik Magdon-Ismail, and Hsuan-Tion Tien Lin. Learning from data, volume 4. AMLBook New York, 2012.
Alekh Agarwal and John C Duchi. The generalization ability of online algorithms for dependent data. 1EEE Transactions on Information Theory, 59(1):573-587, 2012.
Ahmad Ajallocian and Scbastian U Stich. On the convergence of sgd with biased gradients. orXiv preprint arXiv:2008.0000 000011, 2020.
Oren Anava, Elad Hazan, Shie Mannor, and Ohad Shamir. Online learning for time series prediction. In Conference learning theory, pp. 172-184, PMLR, 2013.
Francis Bach and Eric Moulines. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems, 24, 2011.
Francis Bach and Eric Moulines. Non-strongly-convee smooth stochastic approximation with convergence rate (1/n). Advances in neural information processing systems, 26, 2013.
Albert Benveniste, Michel Métivier, and Pierre Priouret. Adaplive algorithms and stochastic opprozimations, volume 22. Springer Science & Business Media, 2012.
Dimitri Bertsekas. Nonlinear Programming, volume 3. Athena Scientific, 2016.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine lcarning. Siam Review, 60(2):223-311 2018.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
Claire Boyer and Antoine Gadichon-Baggioni. On the asymptotic rate of convergence of stochastic newton algorithms and their weighted averaged versions. Compudational Oplimizalion and Applications, pp. 52, 2022.
Richard C Bradley. Basic properties of strong mixing conditions. survey and some open questions. Probability surveys, 2:107-144, 144, 2005.
17

---

Published in Transactions on Machine Learning Research (07/2023)
Peter J Brockwell and Richard A Davis. Time series: theory and methods. Springer Science & Business Media, 2009.
Richard H Byrd, Samantha I, Hansen, Jorge Nocedal, and Yoram Singer. A stochastic quasi-newton method for large-scale optimization. SIAM Journal on Optimization, 26(2):1008 1031, 2016.
Hervé Cardot, Peggy Cénac, and Jean-Maric Monnez. A fast and recursive algorithm for clustcring large datasets with k-medians. Compulalional Stalislics 6 Dala Analysis, 56(6):1434 1449, 2012.
Hervé Cardot, Peggy Cénac, and Pierre-André Zilt. Efficient and fast estimation of the geometric median in hilbert spaces with an averaged stochastic gradient algorithm. Bernoulli, 19(1):18 43, 2013.
Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050 2057, 2004.
Jie Chen and Ronny Luss. Stochastic gradient descent with biased but consistent gradient estimators. arvi preprint 1xXt::807.118800 2018.
Alexandre d' Aspremont. Smooth optimization with approximate gradient. SIAM Journal Oplimization, 19(3):1171-114 1183, 2008.
Olivier Devolder et al. Stochastic first order methods in smooth convex optimization. Technical report, CORE, 2011.
Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics, 44(4):1363-1L 11113333--99,,,,,-1111 2016.
Aymeric Dieuleveut, Nicolas Flammarion, and Francis Bach. Harder, better, faster, stronger convergence rates for least-squares regression. The Journal of Machine Learning Research, 18(1):3520 3570, 2017.
Paul Doukhan. Mixing, In Mixing, pp. 15 23. Springer, 1994.
Paul Doukhan. Miring: properties and eramples, volume 85. Springer Science & Business Media, 2012.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic oplimization. Journal of machine learning reseurch, 12(7), 2011.
Alain Durmus, Eric Moulines, Alexey Naumov, Sergey Samsonov, Kevin Scaman, and Hoi-To Wai. Tight high probability bounds for linear stochastic approximation with fixed stepsize. Advances in Neural Information Processing Systems, 1830063-300744 2021.
Alain Durmus, Eric Moulines, Alexey Naumov, and Sergey Samsonov. Finite-time high-probability bounds for ruppek- averaged iterates of linear stochastic approximation. arXiv preprint orXiv:2207.04475 2022.
Robert F Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica: Journal of the econometric society, pp. 987-1007, 1982.
Christian Francq and Jean-Michel Zakoian. GARCH models: structure, statistical inference and financial applications. John Wiley & Sons, 2019.
Sébastien Gadat and Fabien Panloup. Optimal non-asymptotic analysis of the ruppert polyakk averaging stochastic algorithm. Stochastic Processes and their Applications, 156:312-348, 2023.
Daniel Gervini. Robust functional estimation using the median and spherical principal components. Biometrika, 95(3):587 600, 2008.
Antoine Godichon- Baggioni. Estimating the geometric modian in hilbert spaces with stochastic gradient algorithms: Lp and almost rates of convergence. Journal of Mullivoriate Analysis, 146:209 1462209-222, 2016.
18

---

Published in Transactions on Machine Learning Research (07/2023)
Antoine Codichon-Baggiont and Bruno Portier. An averaged projected robbins-mouro algorithm for estimating the parameters of truncated spherical distribution. Electronic Journal of Statistics, (11):1890-192)) 2017.
Antoine Modichon-Baggionn and Pierre Tarrago. Non asymptotic analysis of adaptive stochastic gradient algorithms and applications. arxi preprint orXiv:2303.01377 2023.
Antoine Codichon-Raggione Nicklas Werge, and Olivier Wintenberger. Non-asymptotic analysis of stochastic approximation algorithms for streaming data. ESAIM: PS, 27:482 514, 2023.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems, 33:15042-15053, 2020a.
Eduard Gorbunov, Dmitry Kovalev, Dmitry Makarenko, and Peter Richtárik. Linearly converging error compensated sgd. Advances in Newral Information Processing Systems, 13:20889-200900 2020b.
Robert Gower, Othmane Sebbouh, and Nicolas Loizou. Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. In International Conference on Artificial Intelligence and Statistics, pp. 1315 1323. PMLR, 2021.
Robert Mansel Gower, Nicolas Loizon, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. Sgd: General analysis and improved rates. In International Conference on Machine Learning, pp. 5200 5209. PMLR, 2019.
JBS Haldane. Note on the median of a multivariate distribution. Biometrika, 35(3-4):414 417, 1948.
James Douglas Hamilton. Time series analysis. Princcton university press, 2020.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference machine learning, pp. 1225-1234. PMLR, 2016.
Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Elad Hazan et al. Introduction to online convex optimization. Foundations and TrendsD in Optimization, 2 (3-4):157-32) 325,, 2016.
Belhal Karimi, Blazej Miasojedow, Eric Moulines, and Hoi-To Wai. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference on Learning Theory, pp. 1944-1974. PMLR, 2019.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradien methods under the polyak-lojasiewicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 795 811. Springer, 2016.
JHB Kemperman. The median of a finite measure on a banach space. Statistical data analysis based on the Li-norm and related methods (Neuchâtel, 1987), pp. 217 230, 1987.
Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69b5 2014.
Krzysztof Kurdyka. On gradients of functions definable in o-minimal structures. In Annales de linstitut Fowrier, volume 48, pp. 769-783, 1998.
H. J. Kushner and G. G. Yin. Stochastic Approzimation and Recursive Algorithms and Applications. Springer- Verlag, 2003.
19

---

Published in Transactions on Machine Learning Research (07/2023)
Stanislaw Lojasiewiez. A topological property of real analytic subsets. Coll. du CNRS, Les équations dérivées partielles, 117(87-89):2, 1963.
Shaocong Ma, Ziyi Chen, Yi Zhou, Kaiyi Ji, and Yingbin Liang. Data sampling affects the complexity of online sgd over dependent data. In Uncertainty in Artificial Intelligence, pp. 1296-1303 PMLR, 2022.
Abdelkader Mokkadem and Mariane Pelletier. A generalization of the averaging procedure: The use of two-time-scale algorithms. SIAM Journal on Control and Optimization, 49(4):1523 1543, 2011.
Noboru Murata and Shun-ichi Amari. Statistical analysis of learning dynamics. Signal Processing, 74(1): 3-28, 1999. ISSN 0165-1684.
Dheeraj Nagaraj, Xian Wu, Guy Bresler, Prateek Jain, and Praneeth Netrapalli. Least squares regression with markovian data: Fundamental limits and algorithms. Advances in neural information processing systems, 33:16666-1666 16676, 2020.
Ton Necoara, Yu Nesterov, and Francois Glineur. Linear convergence of first order methods for non-strongly convex optimization. Mathematical Programming, 175(1):69 107, 2019.
Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4):1574-1609 2009.
Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complezity and method efficiency in optamization. Viley-Interscienne 1983.
Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of convergence o (1/k* 2). In Doklady an ussr, volune 269, pp. 543 547, 1983.
Yurii Nesterov et al. Lectures on conver optimization, volume 137. Springer, 2018
Lam Nguyen, Phuong Ha Nguyen, Marten Dijk, Peter Richtárik, Katya Scheinberg, and Martin Takác. Sgd and hogwild! convergence without the bounded gradients assumption. In International Conference Machine Learning, pp. 3750-3758. PMLR, 2018.
Lam M. Nguyen, Phuong Ha Nguyen, Peter Richtárik, Katya Scheinberg, Martin Takáč, and Marten van Dijk. New convergence aspects of stochastic gradient algorithms. Journal of Machine Learning Research, 20(176):1 49, 2019.
Ivan Nourdin. Selected aspects of fractional Brownian motion, volume Springer, 2012.
David Nualart. The Malliavin calculus and related topics, volume 1995. Springer, 2006.
Boris T Polyak and Anatoli B Juditsky, Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838 8858, 1992.
Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zharnal Vychislitel noi Matematiki Matematicheskoi Fiziki, 3(4):643 653, 1963.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12(1):145-151, 1999.
Emmanuel Rio. Asymptotic theory of weakly dependent random processes, volume 80. Springer, 2017.
Herbert Robbins and Sutton Monro. stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Murray Rosenblatt. A central limit theorem and strong mixing condition. Proceedings of the National Academy of Sciences of the United States of America, 42(1):43, 1956.
David Ruppert. Efficient estimations from slowly convergent robbins- monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
20

---

Published in Transactions on Machine Learning Research (07/2023)
Mark Schmidt, Nicolas Roux, and Francis Bach. Convergence rates of inexact proximal-gradient methods for convex optimization. Advances in neural information processing systems, 24, 2011.
Shai Shalev-Shwartz and Shai Ben-David, Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Primal estimated
sub-gradient solver for sym. Mathematical programming, 123(1):337)) 3:300, 2011.
Shai Shalov-Shwartz et al. Online learning and online convex optimization. Foundations and Trends0 in Machine Learning, 4(2):107 194, 2012.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Choon Hui Teo, Alex Smola, SVN Vishwanathan, and Quoc Viet Le. A scalable modular convex solver for regularized risk minimization. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 727 736, 2007.
Tijmen Tieleman, Geoffrey Hinton, al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Endre Weiszfeld and Frank Plastria. On the point for which the sum of the distances to given points is minimum. Annals of Operations Research, 167(1):7 41, 2009.
Nicklas Werge and Olivier Wintenberger. Adavol: An adaptive recursive volatility prediction method. Econometries and Statistics, 23:19-35, 2022. ISSN 2452-3062.
Olivier Wintenberger. Stochastic online convex optimization; application to probabilistic time series forecasting. arxiv preprint orXic:2102.00799) 2021.
Lin Xiao. Dual averaging method for regularized stochastic learning and online optimization. Advances in Neural Information. Processing Systems, 22, 2009.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.500, 2012.
Tong Zhang. Solving large scale lincar prediction problems using stochastic gradient descent algorithms. Proceedings of the twenty-fors international conference on Machine leurning, pp. 116, 2004.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pp. 928-936, 2003.
A Proofs
To begin, we establish recursive relationships for the desired quantities & = These relationships hold for any values of (nx), (ut), (at), and (nx). Subsequently, substitute the specific functional forms of these parameters, which lead to the results presented in Theorem and Theorem 2
and 5, Eil@, 9-
Before presenting the proofs, it is important to revisit a recurring argument employed to non-asymppootcaall bound &, and 5,:
Proposition 1 (Godichon-Baygae et. al. 2023 Suppose (we), (oxe, (ne), and (9x) to be some non-negalive sequences satisfying the recursive relation,
[1 27a markwr-1 B
(17)
with O and X > O. Let C.,, be such that daa S for all 2 tw with tu = inf 2 1 Cwn S X). Then, for (ovx) and (nt) decreasing, we have the upper bound on (wr) given by
1 max Bi,
wt S Tt
(18)
A 1/25ise
with Ti eepp (-AEi-1/aa)) [exp| (Ca
Et Ei= mass) (wo + maxigisi Bi) + 1/2-1 Bi0s
21

---

Published in Transactions on Machine Learning Research (07/2023)
Proposition presents straightforward method for bounding (wi) in 17 The resulting bound in 18 comprises a sub-exponential term Tt and noise term X maxt/25ist 3, Hence, our primary objective is to minimize the noise term without compromising the inherent decay of the sub-exponential term. It is worth noting that the sub-exponential term T₃ diminishes exponentially as + oc.
During our prools, specific functions will be introduced, resulting various generalized harmonic numbers that can be bounded using the integral test for convergence. Additionally, to express our findings in terms of N₂ م Rg, we will utilize the incquality (NV/2C,)
tt (2N,/C,) ((C,,),))))) as demonstrated in Godichon- Godichon-Baggioni et al. 2023 For the sake of simplicity in notation, we will employ the functions v2x(t) and (t), defined as mappings from R+ \ 0 to I, given by
t'=a /(1 x)
if x 1,
80 (0-m)/+0//11-000 <
(t)
1 1+ log(t)
= 1, and ( = 1 log((
=
(19)
00 - 1)
x/
if>
with y E RR such that v1 = S/e (12 Consequently, for any x 2 0, we have Σ En S (l). Additionally, when considering 12 (t)/t, we find that if < 1, then @M(t)/t O(t ((tt(tto)(tt))) If
it is O(log(t)= And if x > 1, it becomes O(t Therefore, for any Zu. 21, V 0, can conclude that vig (1)/t = )((-()0t)/(0+))) and why, (4), (1)l1 = Ó ((11(0)t where O(-) denotes the suppression of logarithmic factors.
In Lemma ! we establish an explicit recursive relation for 5,, which provides a non-asymptotic bound on the t-th estimate of (77. We achieve this using classical techniques from stochastic approximations (Benveniste ct al.
2012 Kushner Yin. 2003).
Proof of Lemma 7 By taking the quadratic norm on expanding it, and taking the expectation, we can derive the equation
6,-1 - (r.
0-)]
(20)
with 6, 2 0. To bound the second term on the right-hand side of 20 we use Assumptions 4-p and 5-p
for
p 2. This allows us to derive the inequality,
Vol
|B (0.-)) - vosi
2 V S 2C + 20 (21)
using that || + 1/1a < ( + ||vlP)) As mentioned in Bottou et al. (2018);
: Nesterov et al. 2018
2
implies that (V,F(0), 0²) 2 0 - 1-1 for all 0 E O. Thus, as F is p-quasi-strongly convex 2 and 0z-1
is measuralle (Assumption 3-p/ we can bound the third term on the right- hand side of 20 as follows:
E(Vef((0-1)) |-
=El = (V 0,-1 0¹) ((( -) 07)
2p -
D - B -
(22)
since
-tt.://00000V--------000-----0-- - 1-
- V
0+1121
V BB) BB)++++33300
- -D B, 6,-1.
by Jensen's inequality, Cauchy-Scwarzz Schwarz inequality, Hölder's inequality, and Assumption 3-p with p 2. Hence, by applying the incqualities (1 and (22 to (20 we obtain that
6, < [1 - -2pm + 2D +C + 2B. +0?
Bτ Vē rt + 207
$ [1 (n 2D,4)) + 202
22

---

Published in Transactions on Machine Learning Research (07 (07/2023)
using Young's inequality in the second line; 2B, S 1 + B/ Bounding the projected
estimate (8) follows from the property that (()) -811 S | 01 for all Rd and 0* 6, as mentioned Zinkevich (2003).
The following corollary directly follows by applying Proposition to the recursive relation for o, derived in
Lemma 1
Corollary Let d, O. 0" 1 where (0+) either follows the recursion in or Suppose
Assumptions 1 and 3-p to 5-p hold for p Let [(m=C] and Liv, .cj indicate whether (vr) is constant or not. If Hly 1 [v,=c) 2 0, then, for any learning rate (rx), we have
2B
4 max oi76:
8, S +
max us +
uul 1/25ist By 1/25igt
with
= exp
Mv
Σ ?i
exp
1 20 (1 Σ
V17
exp
CC
Σ
%
2
N
i=//
1=1
1=1
2B
B
1/2-1
1/2-1
80 +
max V +
max of isist
+
+
0,
iist
i=1
Proof of Corollary 1 First, we introduce the indicator function for whether (v:) is constant (= C) not
(-C). With this notation, we can rewrite 5g from Lemma as follows:
B 8, $ [1 (Ju 2Dwwe))) + 202
u 207 1
(23)
with My = 1--p1pp () 2D, > Let us consider Cg = max 20% (2u/2)2, 22 c)Dr}. By doing so, we can rewrite (23 as follows:
B 6, (1 11 1o + Cs (vr + 1e) 1060-11 + 2 + 20772
(24)
Let ls denote inf (1: Cs (y + +7) S p2/ Then, for any 1 2 48, have m(w//2)2 Co% S pp/22 i.e,
Po 2e/2 2 1. Thus, the conditions of Proposition are satisfied. Then, applying Proposition 1 to inequality
24 we can conclude the proof.
Proof of Theorem 1 By substituting the functions ?/t C,n,t V/ = 0, Conz and n. = =[Ctp1] > Cpta into the bound of Corollary
we obtain that
2p B²
22t C.C.C,
8, ST +
Ma C2r +
(25)
ST + 2(2+6p0)/(+p)) B
2(T+6p)
(26)
Cm 20/(1+p) ( (1+p)
p.cp
(20-88 800 (l (10
off a, > p.q such that 1/p+ 1/9 then ah aPrP hppppp+ 18 /g/.
23

---

Published in Transactions on Machine Learning Research (07/ /023)
with My = (p- (p=0)) > 0, and
1 1,st))CsDD,CCC uapp--ll-))))
44aa- (30003)
Ft eexp
exp
exp
22
(2a - 2p88 - 1)
2
C
V0----0-0000 HCP
4(a 10 -0--)))000 0200202000
80
pby C2
(2a - 2p(B - 0) - 1)02
N, NLL+00 (3)+00-))((11))
1 (0x0))2CDDcC,C
((B-v) (2.z
I exp
exp
0
4(a C3C
2B
4Cg0,00
exp
60
(2a 2p8- 1)
C2 C p
Cpp BB 6 a- p( 20) pC
4(a p(3 σ)) 02
(27)
(2a - 2p(3 - 0) - -))
with help of an integral test for convergence the functions (t) and (t) from 19 and by use of (N./20,) (((+p) <t (2Nt/
Next, our focus turns to the analysis of the fourth-order rate, denoted as A. E1 8 111, for the recursive estimates given by equations (7) and (8). Similar to the approach in Corollary comprehensive investigation of the general case with (76), (nx), (ot), and (n)).
[ we commence a
Lemma 2. Let D. 1=110:-0*111 where (0t) either follows the recursion in or Suppose Assumptions and 3 to 5-p hold for p 4. Let iin=c and 1s, indicate whether (vr) is constant or not. If
11 2D > 0, then for any learning rate (7t), we have
A, <IT + 4B1
1024
96
max v +
max oi?? +
max o77,
1 pl 1/25ish"
mp 1/25igt
14 1/25igt
with Il given as
1 [uy-c CADA D
256C Ct
exp
Σ
7/s
exp
uaa exp
exp
21CA
Ct
73
4
11
1=t/2
Do +
4B²
y
max 15i5
1024
max 15i5t
A_2 + 1h 1515t
96
max
o.7
+
B
f/2
4
256
1/2-1
Σ
4
+
24
1/2-1
Σ x
o 4
1 pd
11
am1
Proof of Lemma 4 The derivation of the recursive step sequence for the fourth-order moment, denoted by De, in 7 follows a similar methodology as for the second-order moment in Corollary | By applying the same approach used to derive (20), we can take the quadratic norm of 7 expand the norm, square both sides of the equation, and take the conditional expectation on both sides. This leads us to the following expression:
A. - + 1ttp:///1/1////4444..0/000100111
4% - -
8*)
<A,1 + + 1111
- -
-1y
where we have made use of Cauchy-Schward inequality. Next, we can utilize Young's inequality to simplify the terms. By applying Young's inequality, we have: 4///100t11 - 0 <
Ei 20(8-0)-22 2a < (2a- 20(8 22) 2018 00
1021, Pe J0. 1), 8 (0, 1), and a- p3 (1/2,1).
24

---

Published in Transactions on Machine Learning Research (07/2023)
/ || and 812110,-1 These inequalities allow us to obtain the simplified expression:
Ve I (4rn/2)110--
101 + 22
4 5[1 + +x
- -4, - -
0-}]
In order to bound the fourth-order term E [[] we can utilize several assumptions. Firstly, we make use of the Lipschitz continuity of Veli (as stated in Assumption 4-pl- Additionally, we consider the bounds (ff) given in Assumption 5-p and the fact that 0,-1 is F, measurable (as stated in Combining these assumptions,
Assumption 3-p
show that BIll ( < + Thus,
A 1 + 12/22 256 Ohh - 256 o -
2
240
-0")
(28)
Next, by employing similar arguments as in the proof of Corollary along with Young's inequality and Assumption 3-p (with p = 4), we have that
4% 1 - or 11 -000
2 -4, -1""011111V00000000000000000
Vol 2 D/N. -o
-
1 3 D. A -
2 -3
B.
Hence, utilizing this inequality, we can bound the last term of 28 as follows,
47.B(11,,- 1-
1 0. 0-)) -,, - 0-)) + 4-, 1 VoF(O--1),0--11 -0)
0r
0" (E
2 -- - 14 3 D/u - BB
To summarize, by inserting this into 28 and incorporating the indicator function that determines whether (v)) is constant (= C) or not (-C), we obtain the following inequality:
Hv 1 [mn-c)D
256C -24C A +
2560/12 +
A <
1
+
+ 240
(29)
2
133
M
with p4, - 11-1(cm [v. (uu > O. Note that Pv from Corollary is lower bounded by 16, and it is strictly
lower bounded when (u+) constant, i.e., Hly > 14, > Let CA 2 satisfy the conditions of Proposition 1 + +256042/ // 24C%)2) /2//2, which implies
The constant Cs chosen such that Ca (1(1 tey P/ 1. This condition is possible since the sequence (ut) is non-increasing and (7x) is decreasing. At last, by applying Proposition to (29, we obtain the desired bound for A.
Corollary 2. Let A,
E 0,- 0- (11,,, where (0.) either follows the recursion in
8
Suppose
Assumptions 1 and
3-p to
5-p hold for = 4. If = μ - L (p= 2D2/ 22 > 0, then for p3 E 1),
we have
22 B3 p Cu,CCuuupu
220 2) 22C
A < TT
(30)
11, 11,C4 (2
with II, given in 31 such that II, (N(sp((N((1tt80))
Proof of Corollary 2 By substituting the functions 71 CC, v2 =n ot = =Cnn and n, Cpt
into the bound of Lemma 2 and utilizing the inequality 23 < C,O, due to p8 (1/2, 1), we obtain
25

---

Published in Transactions on Machine Learning Research (07 2273))
30
In this inequality, we have 24 H. (p- 0f 2 > 0. Furthermore, II can be bounded as follows:
M&C,
L ((00)) Co CDDAC,
11, cxp
jo8-a
exp
10(8443--a
4
1/1/2
2 C
38p8-3a
24C
Σ
j4p8-1a
exp
exp
P
4B
C
BB 1/2-1
Au
20 400 (0((-400)-
C' CAv
C
P
256 CB3 1/2-1
24C 1/2-1
j0(33-10)
A)(
+
1C
Cp
(ox0) [OA0) CDD Po--ppp-- 18- 40) (t)
CA CA
Sexp
exp
exp
23
100
BB
2 C
27 CAC
C exp
Ao
+
P
ByC,Cly p( (t/(2
(1/2)
C
C.C.
(31)
pC
C2ea
with help of the integral test for convergence
Lemma 3. Let 8; 6, (1)] with On given by 9 where (0t) either follows the recursion in
Suppose Assumptions to C hold for p 4. In addition, Assumption 7 must hold true if (0t) follows the recursion in 18) which is indicated by [(Dosoo) Then, for any learning rate (ri), we have
11/2 81/2- S N
1/2
1/2 +
2//2
1/2
2(1-0)
Σ
2(1-0-o")
Σ
n
ngoa
N.N
HNe
81/2
74+1
T1e
1/2
11
+ 22/2 (C + +Ca)
1/2
+
NNN,
76+1
7i
N1
WN
71
1/2
21/2 (C3 + 1/)1/2
CH
1/2
WNv
W.N.
ni+ 62/22 )) 1/2
2C2 + 02)1/2 NNo
(Dro)/2 + +2 22 B.)
with - and C# = C2/2 + D
Proof of Lemma The proof is presented in two parts. In the first part, we consider the case where (0t)
follows the recursion given by In the second part, we examine the case of (8) Let's assumee that (0, is
obtained using the recursion in 7 To proceed, we apply the observation made by Polyak & Juditsky 1992 which observe that
V20(0))(011-0"
Vali(0") + Vol( (V Voli(0") VoF(O.-1)
-VooF(0.))) - ((F(0))(0 -00),
8 E. 308-33 <3< 22 and Σ (4((8-0)-4a < for any & 2 0s a- p3 (1/2,1).
26

---

Published in Transactions on Machine Learning Research (07/2023)
where (8)( is invertible with lowest cigenvalue greater than μ4 i.e., V)F((*) 2 pilg. By summing the individual parts, taking the quadratic norm and expectation, and applying Minkowski's inequality, we obtain the following inequality:
R 0.
E
Vig (0-)-1
n Vosi (00)
N
E
V F (0')- 1
Σ
n; Vof: (0 (0:-1)
N₁
E
F (0') -1 1
Σ
n; Vefi (01-1) Vofi (0*) V&F VoF(O:-1))
N
n4 V (BFF() (0') 0251 (7]
V2F (0') -1 1
E
(32)
N,
First term of 32
As (O)) is a square-integrable sequences on Rd (Assumption 3-p/, we have
1
E
(0*)
Vofi (0*)
E
N
nnyBE
( (0"F(0))
Vafi (00) (OF
-oo (00)
N?
ISisjSt
Here, the first term can be bounded using Assumption 6 as follows:
E
V WVFF(B))
(
A
n
2(1
1 C²
n
211-0-0')
N
P?N?
where A denotes TV For the second term, we use Cauchy-Schwarl inequality, Hölder's incquality, and Assumptions 3-p and 5-p to show that
2
Σ
(TA0)))))" (07)
N
ISi<iSt
2
<
Bnn (0')
Vol, (0) VoF(A'))
N2N?
ISisist
2
Σ
n:y E
Voli [|
VoF(''I
1SicjEt
Σ
Banjy F.
|ioo (0-)) 1
[ [10[[ [(F+))]11
N2
15Kjst
2Br
Rn,oWy = 2B,
j-1 Σ noo,
Σ
< NPN?
NºN?
Igisist
j=2
1=1
27

---

Published in Transactions on Machine Learning Research (07 /023)
Thus, combining these finding gives
Vofi (00) (1)
A
E
(0)
n4 Vofi
Σ n2
2(1-o)
N.
N,
==1
C/
21/2 21/2 B!
1-1
((----)
+
Σ
n1:0;
(33)
uN
m.Ne
==1
Second term of 322 First, we use that 1 Ei=1 n, ((Vo(((0 No Σ
(0,-1- 0.) N..
No (0e 0r) m + N. (00 0 which leads to an upper bound on normed quantity [V)F(0)) N. Ei=nn n4 Vof(((,,))) given by
76
(')( Pill
)
1-1 1
110 1*11
B4+1
+
1 H.N.
110 - 1*1l ne + NN,
1
100
0*1 11
N.N,
7++1
Rewriting this in terms of 6, = B|[|0, - 0- [12], gives us a bound for second term of
32
8,
n++1
T2
The
07 + pon unNN
n1
63.
(34)
NNN
764-
Vi
Third term of be derived as
32
Here, we use that (
- can
1
n
- Vofi(0)) -
B
N
2
B, - - -
Vof -Vo(()))))))
Next, Cauchy-Schwarz inequality,
innguuua Assumption 4-p and
3
show that
-Vo((())))-Voo(())))-)) v. 2(C +C
Σ n 8
Similarly, for the other term, note that
< < |4 VoF(0+)11
S 2/22 -4) 2D))0)111112200
121/2(02 + 02))1222((020)0011 (
28

---

Published in Transactions on Machine Learning Research (07 (07/2023)
using Fi-1 C F- since <. J. Cauchy-Schwarrd Schwarz inquulity, Hölder's inequality, || = 2 + ([6]l²)
with p E N, Assumptions 3-p and 4- and 3 Thus, the third term of (32 can be upper bounded by
1/2
1/2
+ C3) H.N.
23/4 (C + 02)
Σ n 01
+
Σ
(Do6 + + B.)nyu Σ n
M.N
(35)
Fourth term of 32 Here, we use that (4) implics 4B, | VoF(0)- -VF()))- 8') v C6 - 8- 13/2
NNsterovv et al. 2018).
which gives the upper bound C6
N,, Σ n, ! using the definition 4, = E - 0- ||
Combining the lerms (33) lo (35 into (32). logether with shifting the indices and collecting the 6g terms,
gives use the desired bound for when (0c) follows
Now, assume that (0t) is derived from the recursion in As above, we follow the steps of Polyak & Juditsky (1992), in which, we can rewrite (S) to "T (0,-1 0.) Vafi(O--1) + Ω4 where S4 P - T: Vafr(O,-1)) (0--1-1 - Vo.e(((. Thus, summing the parts, taking the norm and expectation, and using the Minkowski's inequality, yields the same terms as (32), but with an additional term regarding 24 namely
E
ViF
V2F((0)
1
n4 Ω
T4 %
E
12
2
(36)
N
7
N
using Godichon- Baggionni 2016
Lemma 4.3). Next, we note that
||2,1 2 = IPo (0t-1 7o Vefi (0: 6 (0
<2 Pe (0:-1 Voli - 27d Voli (01-1)11
=2 Pe (0:-1 2. Vofi (0.-1)) (P-
2 Vofi (0--1)11 11 + 2 V (0, (0,-))l - 4
_2 7/ Vofi -
as Pe is Lipschitz and [())(( < G2 for any O O. This means that the inner expectation of (36 θ). Morcover, in (Godichon-Baane Baggioni & Portier 2017 Theorem 4.2) with use of Lemma 2 we know that P(0,- 7x. VaJe θ < A./D%
E E [1 [a, 1-2 Vofrte 1) 4 G2 (of((())))
where De = infoeeell 0*1l with de denoting the frontier of O. Thus, 366 can then be bounded by
R4 %
E [12
<
2Ge
1 1/2
N.N,
DD.. N₂
since the sequence (n:) is either constant or increasing, meaning Vt, mt/n/411111 S 1. At last, let C C2/2+ + [(Do<x)) 2G/ D2 indicate whether (0.) follows or not.
Proof of Theorem This result can be obtained by simplifying and bounding each term of Lemma
3 using
the bounds provided by Theorem and Lemma 2 By inserting the functions 7t C.n L-a vs n,
29

---

Published in Transactions on Machine Learning Research (07/2023)
0 = Cont", and n, Opt into the bound of Lemma
3 obtain that
A2/2
AV200-0
t
1/2
Ch-a-o'
1
1/2
8//2 < N/1/2
Σ p202--0)
1 (x=1/2) +
1 10x1/23
+
No
N.N
:o(1-a) ))
1/2
(0(1-)) B) + C2
3)
j0ll 00l--0)4-160/22
21/2
C.
jo(1-2)
5-1
N.,.N,N
pC, uCt
Optrll-)ta 61/2
Cp 1
+22 (Cx + +c})
6//2
+ 21/2tP(C2 + 122
Σ
1/2
NCC.C,,, N₁
UNv, C.ppp
UN,
)) 1/2
Cp UNo
ip ! 1/2 22/4 + 122)
(Dvo) + 22/2 (y/2BB)) jo((t
Σ ip 1P6!/3)
nCp/2 No
==
using mit1/ng S 20 and that m S (o(1- with E [0, 1). Next, as a E 1/2 and o' (0. we have Σi=1 (0(1-0-00) & -220(-0000 /(1 + 2p(1
as -(1-3))---0000 +) S 1-pp
o o')), where & S (2N,/C,) 1/(+e)), Similarly, as v E (0, oo), we have that
jo(iv
xo(1-o)
S jo(l-v)
poll-o) S [1o001(((800001000
j=l
S ((o-)) /CoD/ (1 (2Ns/Cp),
(2))) is O(N,)(ot0)//111
using the u-function defined in 11 Hence, From 25 we know that 5, S Ds/t" with
21+2pu B²
22+0(20-8)+0) C2C,0
D3 = sup T +
ieN
and 8 1(B. - 9) + a) + (B, min (p(2o - B) + a, 2pv), yielding
j-1
1-1
+ 22/2 (D,6)//++1/333
v)
Σ 108!/2) S <D!
Σ (D.D + 21/2 By)j 1/802-001//11111 (o))
1=1
+ 222 B, D!! 1/2 1/8/2 (t)p//p(p-))(
S
s D. (2N, (2N. (C)) 1/ (2N,/Cp) + 22 B.D /2/0
8/2- 1 (CN (2
if 6/2 - p 2 0. Hence 1 (B
is Ö(N, ((N0)(4400)/2((11))) and
18/2-p (2N. 1) (2N,/ ()
is (02200)111))) Next, we define म. = E/ PT: 2 E/= T.
since ,, is decreasing. Similarly, let II= El= ilI Both R, such that गं St 1 El= n. I -t coo and IT, convergences to some finite constant depending on the model's parameters. With use of these notions,
30

---

Published in Transactions on Machine Learning Research (07 2023)
have
A
10x1/22 + C2 "2.'" -((1122(00tt)))))10
8//2 <
N//2 L (o=11/2)
N (11+220)/2(+0)
2 (7+1+22((44))))2= _ 20 ------2-------- (1/2(tb)))
TCp
2 2(2-01/1+P) 12 -,0
mp C. N; 2200
N.N,
M.G.N. (C a0)
20++pl1+2 ()+0)(1+++1)))) (1+
C/ CC
+ 1 (BuzO)
PVP.CP (0/24400-20)) 20/240/0-220//22422)0 D 2
MPd 1/2 Nº ()0
23/4+ 2//2 D D3 1/2 C
4//22p
MC Ni
as a - p E (1/2, 1), where p/ = p 2D2 D Cp+ OF C' (Dooss))GG//D
I 2 / /C, C (1/C,,0, + /22 6 + 2/2+p 22 1 (B, min ( (p(20- - B) a, 2 and U,, given
D Toc
1/2
+ I0, = [B.=0) - B) + +
2//2B// / /2/ Cpp
UP (2N (1-1))
23(1+ B,C B-
MCp WCJo-M)12N No
1212 Uv
1/2 (L+pt((0t
21+0(2- B // DD Cap 1/8222 CP) (1- (C (2N./Cp) C
ICP N.
2 BECKC,BBBCCC p(2--) 1(20-1) (2N,/CO) (1
23/2+0(1+0) B,
+
1012 /12 N₂.
13/2 2/
23/2+DV BYC 1+p(8+-- B 1)-a (2N+/Cp) 1212 1/2 C,
Furthermore, using the O-notation one can show that
A1/2
21/2 1/2 1/2 2012222(1
2//2 ( 1/22)11
6//2 = 1 (/=1/2) +
(ox1/2) /N (1(1=ppoo00))/2111
N//2
\112g0)/2(10p)
2d Cf 122-002----------0
1//2 + 22220 3/2 CC
mp 1/2 c///2////////////// (2+ 2 (22((222))-
PV/, (1 2p0 20-22--aa)(111 aa)((aa)) 10
2 C. C. (1+3 20-0)22
IC2
2c (2+ (04t)))((L+)))
p4p 1/2 No
UN, NCC. 22tp0-0))
(6+ (p) 2(1 + LBB
(37)
where v,= = ON, plo))) + ( (Lox= ÖON, (1, (1+2p)/(00)))) -,,,:///0))0)0o,0oo///////////,0 ",,,::/:0..__..000,0000000000,,0
Ö(N, (1v((1+))), implying that v > 1/2 to obtain the desired rate 8, = O(N-1) if B, = 0.
B Verifications of Assumptions 3 and 5-p for the AR model
Well-specified case. Consider the well-specified case, in which, we estimate an AR(1) model from the underlying stationary AR(1) process X, X--1 + 6, with |0'| < 1. The squared loss function fe(0)
31

---

Published in Transactions on Machine Learning Research (07/2023)
Ty "'E 1+1 -xN, ++--1) with gradient Volf(O)= -2n EE XN. 1 1t: -8Nx 1+1-1).
Thus, the objective function is
1
( NN-iti - -00 NNN = 22(00 - 0)²
F(O)
E
1- (0~)2 +
ThE
using [[X,] 0 and [[ o²/ (0))²), yielding VeF(0) 20 (1) (07)2). Next, verify
Assumption 3-p for p 2, we first note that
20
2
E [o
E
x3
F
E
X
N
T4
T4
==1
2(0
8*)
The
2
X N
Fi-1]
(38)
1
11
n1
N 0 8' + EN. For the first term of xx+1 F,] = o²²1 2/) (0") )), yielding
38
we use that E[X = (0r) and
Var
+ o²n,
E N +i- [F--1] = x
(0 20)21111)
(00)2(1-1)
(1 (0+)2)
1 (0+)z
1=1
1=1
(00) )
N
(1- (6) 2m)) )o?
+ 1 - (0+)2
(1 (8)
(1 - 22)2
Next, the second term of (38 is zero by utilising that (ex) is a Martingale difference sequence, i.e., Bl(sts|FF] = and Blea4ses+) [F] = for + f j. Thus,
- Vo - 4(0-0))3(1 (8") (m))) )o0 (1 (8+)2 nm
B BIIE -
00 + 1 (0m)2)
mcaning that Assumption 3-p is verified for p 2 if (X,) has bounded moments; this is fulfilled by the natural constraint that |0-| < 1. Thus, we can deduce that D, > 0, B, = 0, and it is O(nz²). ). The remaining assumption can be verified in the same way, in particular, Assumption 5-p is satisfied with o, is O(n 112), Assumption 2 with Op 202/(1 (8))2) and CG O, and Assumption 6 with Σ = 4o
(0*)²) and
>, = 0. Furthermore, for an AR(1) process X, constructed using the noise process E, = VG (H)zz, with Hurst index H 2 1/2, one can verify that of and of is O(n! (n in Assumptions 3-p and 5-p using the self-similarty property (Nourdin 2012).
Misspecified case. Next, assume that the underlying data generating process follows the MA X, = 6 + 6 with 6² R. The misspecification error of fitting an AR(1) model to a MA(1) process can be found by minimizing
F(0) =E((X, = Bi(c,tt(CCi.... + 0- 0(c,- (. -E((e, + (0" - -A)c,-1 - A0* c 2) = 0² -
0)" (0))))
where (() = - 11000")0² + 20( )0 Ro?.
Thus, as 0" = arg min, F(0) arg ming 0)2 + ( is strictly convex function in 0, we have (0F = 0 + 2(0- +0(=))=-0-----111000))))))))))-) + 20 + (3) = 20* + e 8= 6- /11 + (6-)2). This means for any o² E R then E (-1/2,1/2) With this in mind, we can conduct our study of fitting an AR(1) model to the MA(1) process with o² drawn randomly from l (figure Tb Furthermore, this reparamctrization trick can be used to verify Assumption 3-p first, we can reparameterize VoF(0) (o2(() (1 + (2) using 0* 101(1111 + (d))2). Next, for E (O) one have that
20
2
[[Vo{(()]==
X N:
- -
Fi-il
B N. -ti -tt-ttt--n-
T4E
14
32

---

Published in Transactions on Machine Learning Research (07 (0722023)
where
B M. - N._, + + 1 11 +.. +EX [(-1]
X to + (0 EN t... +
=X,,,, + (0") +o - (6))0 2)
X (6") - -.)) + (1+ (0)) 1),
and
B|XM 1 1+7- =6" + (m -
,N, (1 + (6))))2 1),
using the same white noise properties as for the well-specified case above. This yields,
4(0 0*)²
[[ P(O))
ni Sor (EN...),
where Ja- (eN, ,) is finite function depending the moments of (eN, 1) and Φ² Hence, have D, > 0 and B, = 0 with us being Onzzt). Similarly, it can be verified that are O(n, 1/2) by use of the reparametrizztion trick.
33


Reviews:
[{"review_id": "MKCJvo9L8G", "reviewer": "Reviewer_yQ8R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---\nStrengths\nOverall, the results are technically correct and can be of interest to members of the community, so publication in TMLR is well-suited. However, the paper should first be proofchecked and the writing can be improved. \n\n---\n\n###  Questions\n\n#### Assumption 1-p\n- what does the external expectation correspond to in assumption 1-p? If the assumption is valid for any $\\mathcal F_{t-1}$ measurable variable, isn't it valid for any $\\theta$?\n\n####  On theorem 1:\n- is there any technical novelty in the proof techniques?\n- what do the authors mean by batches of size $t^\\rho$: is it an upper bound? $n_t$ is  in  $N$. The second paragraph in Section 3.2 needs to be clarified. In the proof of Theorem 1, it seems that it is simply assumed that $n_t= C_\\rho t^\\rho$ for any t, and for some $\\rho\\in [0,1)$.  \n- I am not sure to understand why constants like $\\mu_\\nu =\\mu -1_{\\rho=0} D_\\nu C_\\rho^{-\\nu}$ have to appear. The condition $\\rho=0$ seems artificial, as I could consider $\\rho=10^{-30}$ and that would seemingly change the rate. But again this is not very clear: I suspect that $C_\\delta$ depends on $\\rho$ which is probably the case from the derivation following eq. 23. If so, that dependency should be made clear in the theorems.\n- In this derivation, I suspect that C_\\delta is chosen as a constant \"possibly smaller than 1\" such that....  I fail to see how it can be ensured that $C_\\delta$ would verify the condition and be larger than  1.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides rates for stochastic gradient descent with time-dependent streaming data. \n\nAuthors consider a stochastic gradient descent scheme in which the stochastic oracles are not independent along the sequence. They provide convergence rates under classical assumptions on the function namely expected smooth and \\mu quasi strong convexity, both in terms of expected squared distance to the optimum $E[||\\theta_n-\\theta_*||^2]$ and for a weighted Polyak-Ruppert average iterate.  \nThe correlation between consecutive gradient oracles is illustrated on classical (arch, etc.) models.\n\nThe results leverage the analysis of SGD with conditionally biased stochastic gradients (Lemma 1), that are combined with assumptions on the conditional bias to obtain the main convergence theorems.\n\nThe application to the AR case is interesting and the discussions following theorem 1, on the choice of the size batches to obtain \nconvergence.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n### Writing:\nThe writing of the paper could be improved: the technical contributions are not completely clear and the are many typos (see below).\n \nIn my opinion, the paper would be easier to read if a Lemma, corresponding to eq. 9 was stated, or possibly Lemma 1 was stated immediately after Assumption 1-p to 3-p, with precise explanations on the proof technique and the novelty regarding theory of SA, if any.\nThen in a Second subsection,  stating the specific choices of $\\nu, \\kappa, \\sigma$ as _assumptions_, with a corollary.\n\nThe discussion paragraph following assumption 3-p should be reorganized:  the link between the size of the minibatch and  Assumption 1-p is critical for the rest. Making a formal statement would help.\n\n---\n### Related work \nSeveral related references are missing or mentioned but the relationship should be clarified.\n\nOn Markovian dependent LSR:\nLeast Squares Regression with Markovian Data: Fundamental Limits and Algorithms https://proceedings.neurips.cc/paper_files/paper/2020/file/c22abfa379f38b5b0411bc11fa9bf92f-Paper.pdf\n\nOn SGD with bias.\nhttps://arxiv.org/pdf/2008.00051.pdf \n\n--> Litterature on clipping\nhttps://arxiv.org/pdf/2005.10785.pdf and related refs\n\n--> Litterature on biased compression\nhttps://proceedings.neurips.cc/paper/2020/hash/ef9280fbc5317f17d480e4d4f61b3751-Abstract.html and related refs\n\n---\n### Typos \n\nFootnote 1: Momentum and Nesterov accelerated methods are generally not considered accelerated.\n\nThe article needs to be fully proofread (including the proofs as there are still several typos). For example:\npage 6:\n-  v_t stands for \\nu_t?\n- allowS\n- oF the  form\npage 7:\n- combining both more ideal\n\npage 20:\n- since L is $\\mu$ strongly convex [such an assumption is not made]\n- it's imply \n- $(\\kappa_t)$ and $(\\gamma_t)$ is", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f36KUPcuKF", "reviewer": "Reviewer_BT8S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis theory builds on a mild assumption about the bias of gradient estimate (Assumption 2-p). To my knowledge, this assumption is considerably mild.\n\nWeaknesses:\n\n- The theory seems to be an extension of [Godichon-Baggioni et al. (2021)]. Thus, it would be nice to clarify the technical challenge from this previous work.\n- I could not understand the following sentences for $\\kappa$: from *``Obviously, $(\\nu_t), (\\kappa_t),$ and $(\\sigma_t)$ may\nbe considered as uncertain terms depending on the streaming batches $(n_t)$.''* to *``Having, $\\sigma,\\kappa \\in [0,1/2]$ follows directly from Godichon-Baggioni et al. (2021), since $\\sigma=\\kappa=1/2$ corresponds to the i.i.d. case.''* If $\\kappa_t \\sim n_t^{-1/2}$  following this description, then we can deduce $\\nabla l_t(\\theta)=\\nabla l_t(\\theta^*)$ in $L_p$ as $n_t \\to \\infty$  from Assumption 2-p. Clearly, this is false in general. Please correct me if my understanding is incorrect. (Moreover, I could not find explanations or specific examples of these constants in Godichon-Baggioni et al. (2021), although the authors cite this paper here.)\n- The notation of $l_t$ is confusing. At first glance, I thought the objective $L(\\theta)=\\mathbb{E}[l_t(\\theta)]$ itself is time-dependent because of this notation. Moreover, the mini-batch $l_t=(l_{t,1},\\ldots,l_{t,n_t})$ is also confusing because of the confliction of notations.\n- Assumptions 2-p and 3-p with p=1/2 imply ABC-condition studied in the following papers. In this sense, ABC-condition is a milder assumption. It would be better to discuss the relationship.\n\n[1] Gower, Sebbouh, and Loizou. (2021a). Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation.\n\n[2] Khaled and Richtarik. (2020). Better theory for sgd in the nonconvex world. \n\n- Lack of citations to relevant papers which show the statistical optimality of SGD (for least-squares regression). For instance, see below.\n\n[3] Dieuleveut and Bach (2016). Nonparametric stochastic approximation with large step-sizes\n\n[4] Dieuleveut, Flammarion, and Bach (2017). Harder, better, faster, stronger convergence rates for least-squares regression.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the stochastic gradient descent (SGD) for the streaming data where the time-dependent and biased estimate of the gradient is available. In particular, the authors derive the general form of convergence rates under mild assumptions for SGD and averaged SGD. Finally, the theory is verified with several experiments on synthetics dataset, AR model, ARCH model, and so on.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses for the detail.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ncPX8r8HYK", "reviewer": "Reviewer_gnir", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the upside, I think the results are mostly correct (though I did not check every details in the appendix due to the time limit) and may be useful upon further polishing. \n\nHowever, as a general comment, I find the paper very challenging to read due to the extensive use of constants/notations, which maybe even harder for general readers. The reviewer believes that the presentation of the main results in this paper should be substantially simplified before publication. Detailed comments and suggestions can be found in the \"Requested Changes\". \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents several new analysis results for the convergence of stochastic (gradient) algorithms, especially it focuses on the case with time-dependent streaming data via the new conditions proposed in Assumption 1-p to 3-p.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The proof of Lemma 1 leads to a condition that depends on whether $\\upsilon_t$ is constant or not, while it seems that the same condition is changed to $\\rho = 0$ or not in the presentation of Theorem 1. In addition to the lack of justification presented in the paper, the reviewer notices that $\\rho=0$ only implies that $n_t$ will be a constant, which  doesn't necessarily imply that $\\upsilon_t$ is constant. Again, the paper shall provide a clear justification about this. \n\n2. It is confusing that paper mention about \"conserving convexity\" in the discussion of Theorem 1. As discussed in Section 2.2, the objective function $L$ is always convex and satisfies a quasi strong convexity condition in (5). The discussion after Theorem 1 seems relevant to \"conserving *strong* convexity\" instead. Please improve the discussion about it. (P.S. On a related note, there is a small typo before (22), which wrongly stated that \"$L$ is $\\mu$-strongly convex\".)\n\n3. The distinction between long-range and short-range dependence was mentioned briefly in Sec. 3.1 and the paper proposes to distinguish them by checking if $\\upsilon > 1/2$ or not. This choice of threshold is not clear from the context, and the reviewer guesses that it is related to the discussion in \"Behavior for bias\" after Theorem 1. If it is so for the latter case, then such choice should be justified as early as in Sec 3.1.\n\n4. For the results about averaged stochastic streaming gradient, it is noted that the main result Theorem 2 is presented with $p=4$ instead of $p=2$ as in Theorem 1. Why is there a change of moment's order? Such has not been discussed in the paper. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors' engagement and incorporation of the reviewer's suggestions in their paper's presentation and writing have resulted in significant improvements. Given that the authors have addressed all the reviewer's suggestions, in my opinion, the paper is now ready for acceptance without further revisions.
Decision: Accept (poster)