Title: Near-Optimal Streaming Heavy-Tailed Statistical Estimation with Clipped SGD
Venue: NeurIPS 2024
Accepted: yes
Keywords: Streaming estimation

heavy tailed estimation

clipped SGD
Abstract: $\newcommand{\Tr}{\mathsf{Tr}}$
We consider the problem of high-dimensional heavy-tailed statistical estimation in the streaming setting, which is much harder than the traditional batch setting due to memory constraints. We cast this problem as stochastic convex optimization with heavy tailed stochastic gradients, and prove that the widely used Clipped-SGD algorithm attains near-optimal sub-Gaussian statistical rates whenever the second moment of the stochastic gradient noise is finite. More precisely, with $T$ samples, we show that Clipped-SGD, for smooth and strongly convex objectives, achieves an error of $\sqrt{\frac{\Tr(\Sigma)+\sqrt{\Tr(\Sigma)\\|\Sigma\\|_2}\ln(\tfrac{\ln(T)}{\delta})}{T}}$ with probability $1-\delta$, where $\Sigma$ is the covariance of the clipped gradient. Note that the fluctuations (depending on $\tfrac{1}{\delta}$) are of lower order than the term $\Tr(\Sigma)$.
This improves upon the current best rate of
$\sqrt{\frac{\Tr(\Sigma)\ln(\tfrac{1}{\delta})}{T}}$ for Clipped-SGD, known \emph{only} for smooth and strongly convex objectives. Our results also extend to smooth convex and lipschitz convex objectives. Key to our result is a novel iterative refinement strategy for martingale concentration, improving upon the PAC-Bayes approach of \citet{catoni2018dimension}.

Near-Optimal Streaming Heavy-Tailed Statistical Estimation with Clipped SGD

Abstract

We consider the problem of high-dimensional heavy-tailed statistical estimation in the streaming setting, which is much harder than the traditional batch setting due to memory constraints. We cast this problem as stochastic convex optimization SGD algorithm attains near-optimal sub-Gaussian statistical rates whenever the second moment of the stochastic gradient noise is finite. More precisely, with T

with heavy tailed stochastic gradients, and prove that the widely used Clippedsamples, we show that Clipped-SGD, for smooth and strongly convex objectives, achieves an error of √ Tr (Σ)+ √ Tr (Σ) ∥ Σ ∥ 2 ln( ln( T ) / δ ) T with probability 1 -δ , where Σ is the covariance of the clipped gradient. Note that the fluctuations (depending on 1 / δ ) are of lower order than the term Tr (Σ) . This improves upon the current best rate of √ Tr (Σ) ln( 1 / δ ) T for Clipped-SGD, known only for smooth and strongly convex objectives. Our results also extend to smooth convex and lipschitz convex objectives. Key to our result is a novel iterative refinement strategy for martingale concentration, improving upon the PAC-Bayes approach of Catoni and Giulini [8].

1 Introduction

A fundamental problem in machine learning and statistics is the estimation of an unknown parameter of a probability distribution, given samples from that distribution. This can be expressed as the minimization of the expected loss: min x F ( x ) := E ξ ∼ P [ f ( x ; ξ )] , where x represents the parameter to be estimated, P is the underlying probability distribution which can only be accessed through samples, and f ( x ; ξ ) is a function which quantifies the loss incurred at a point ξ by parameter x . In this paper, we focus on the setting where P is a heavy-tailed distribution for which the extreme values are more likely than in distributions like the Gaussian, f ( · ; · ) is convex and the learner only has access to O ( d ) memory.

The heavy-tailed statistical estimation problem has received increased attention of late because of the prevalence of heavy-tailed distributions in many statistical applications dealing with real world data [19, 49, 57, 23]. The presence of such heavy-tailed distributions can significantly degrade the performance of statistical estimation and testing procedures designed under Gaussian (or sub-Gaussian) tail assumptions [30, 24, 53, 24]. This has spurred recent research efforts towards developing estimators specifically tailored for heavy-tailed settings (e.g., [10, 44, 16, 36]; see Section 1.2 for a more detailed literature review). Despite substantial progress on this problem in recent years, much of the

existing work has concentrated on batch learning, where the entire dataset is available upfront, and the learner can revisit data points multiple times, without memory constraints. However, the streaming setting, where data arrives sequentially and must be processed with limited memory, is increasingly pertinent in the era of large-scale models. Consequently, in this work, we focus on understanding estimators for statistical estimation under heavy-tailed distributions, in the streaming setting.

A popular approach to study heavy-tailed streaming statistical estimation casts it as a stochastic convex optimization (SCO) problem with heavy-tailed gradients [17, 44, 52, 48] - with Clipped-SGD as the favored solution due to its simplicity [42]. Indeed, clipping has become a standard component in the training of modern deep neural networks and thus, the properties of Clipped-SGD have been studied widely in the literature [1, 56, 38, 48, 52] in various contexts. Specifically, several works have shown that Clipped-SGD has sub-Exponential or sub-Gaussian tails despite the presence of heavy tailed noise in the gradient [45, 21, 52, 49]. Despite this progress, the best known rates for Clipped-SGD with smooth and strongly convex losses, under a bounded 2 nd moment assumption on gradient distribution, are of the order √ Tr (Σ) ln( 1 / δ ) T , where δ is the failure probability [52]. Note that this is still far from the optimal sub-Gaussian rates of √ Tr (Σ)+ ∥ Σ ∥ ln( 1 / δ ) T . In this work, we bridge this gap with a sharper analysis of Clipped-SGD for SCO problems, achieving nearly sub-Gaussian rates (see Section 1.1). Our approach leverages a novel technique obtained by bootstrapping the Donsker-Varadhan Variational Principle to Freedman's inequality, yielding tighter concentration inequalities for vector martingales compared to those in [8]. This enables us to derive more refined rates for a variety of settings than a direct application of Freedman's inequality as in [52].

1.1 Sub-Gaussian Error Guarantees for Statistical Estimation

Mean Estimation We motivate our style of results with the case of mean estimation. The Central Limit Theorem (CLT) posits that that the empirical mean of T independent and identically distributed (i.i.d) random variables with a finite covariance, behaves roughly like the empirical mean of Gaussian random variables with the same covariance, as T →∞ . That is, the empirical mean ˆ µ , the true mean µ and the covariance Σ are such that lim T →∞ P ( √ T ∥ ˆ µ -µ ∥ > √ Tr (Σ) + ∥ Σ ∥ 2 log( 1 δ ) ) ≤ δ . However, these asymptotic rates need not hold with a practical number of samples. Therefore, recent works on heavy-tailed high dimensional mean estimation consider algorithms and non-asymptotic guarantees which move beyond the empirical mean (see [36, 10, 9, 27, 28, 15]). Estimators such as the clipped mean estimator [8, 55], trimmed mean estimator [45], and the geometric median-of-means estimator [39, 29] achieve an error of at-most √ Tr (Σ) log( 1 δ ) / T with probability 1 -δ with a finite covariance assumption. Recent ground breaking works [37, 28, 8, 36] further improve upon these results to construct estimators which can achieve the CLT convergence rates of C √ Tr (Σ)+ ∥ Σ ∥ 2 log( 1 δ ) / T for every T and δ . Some of these estimators work under just the assumption that the second moment is bounded [37, 28, 9] and some even provide a nearly linear time algorithm [15].

General Statistical Estimation In this work, we are interested in the general statistical estimation problem. Among the various approaches, framing this problem as SCO with heavy-tailed gradients has gained traction recently (see [52] and references there in). While one obvious candidate is to use SGD with state-of-the-art optimal mean estimators for robust gradient estimation, such methods can face significant challenges. First, most optimal mean estimators aren't designed for the streaming data setting with batch-size being 1 . Second, these estimators can be complex, frequently relying on semi-definite programming or other demanding techniques. Third, and perhaps most importantly, they don't typically provide guarantees on the bias of their estimates. This lack of bias control is problematic because SGD-style algorithms, even when equipped with accurate gradient estimates, can perform poorly if those estimates are systematically biased (See [3, Theorem 4], where bias does not cancel across iterations). Given these challenges, the clipped mean estimator of [8] has emerged as a popular choice for gradient estimation in SCO, due mainly to is simplicity. Several recent works analyze the performance of SGD with clipped mean estimator for the gradients (i.e, Clipped SGD). However, as previously mentioned, the best known analysis for clipped SGD achieves a sub-optimal rate of √ Tr (Σ) ln( 1 / δ ) /T , under bounded 2 nd moment assumption. In this work, we improve upon these rates and show that with T samples, clipped-SGD obtains a sharper rate of Tr (Σ)+ √ Tr (Σ) ∥ Σ ∥ ln( ln( T ) δ ) T with probability 1 -δ , which is closer to the truly sub-Gaussian rates.

1.2 Related Work

Clipped SGD Clipped SGD and it's variants have been studied under a variety of settings including convex, strongly-convex, non-convex losses, with various assumptions on the moments of stochastic gradients. The estimators of [21, 45, 14, 40] work under the assumption of bounded 2 nd moments, but require O (1 /ϵ ) batch size, to converge to an ϵ -approximate solution. Consequently, they are not suitable for streaming setting. The recent work of [52], which is closest to our work, addresses this issue by analysing Clipped-SGD for batch size 1 for smooth, strongly convex losses. But they achieve a sub-optimal rate of √ Tr (Σ) ln( 1 / δ ) /T . These rates are improved in our work (see Table 1 for a detailed comparison). Additionally, our work provides convergence rates for convex objectives that are not strongly convex. Recent works [48, 46, 41, 34, 13] have studied Clipped-SGD with the assumption that the stochastic gradient has a finite p -th moment for some p ∈ (1 , 2] . They derive fine-grained near optimal results in terms of dependence of T and p (but their dependence on log δ -1 is sub-optimal). In contrast, our work specifically the case considers p = 2 with a focus on improving the sub-Gaussian dependence in the high probability bounds in these works from Tr (Σ) log(1 /δ ) and approaching the truly sub-Gaussian rates for estimation 1.1.

Heavy-tailed Estimation Heavy-tailed estimation has a rich history in statistics and we only review some of the recent advances. Several recent works have studied the problem of heavy-tailed mean estimation, and have derived estimators that achieve sub-Gaussian rates under the bounded 2 nd moment assumption [36, 10, 9, 27, 28, 15, 45]. Among these, the works of [15, 32] are particularly relevant to our work. The algorithm of [15] runs in linear time while requiring O ( d log δ -1 ) memory. But it is not immediately clear how to use their estimator in the framework of SGD. [32] study the trimmed mean estimator (an estimator that is closely related to clipped mean estimator, where outliers are removed instead of being clipped) and show that when T = ω (log 3 δ -1 ) , d = ω (log 2 ( δ -1 )) , the estimator achieves the optimal rates. We not that our analysis of clipped SGD, when instantiated for mean estimation, leads to similar rates. But unlike [32] which is primarily focused on mean estimation, we focus on the more general SCO problem.

Heavy-tailed linear regression has been widely studied, with classical estimators based on Huber regression [30, 50, 33] known to provide optimal rates when the response variables are heavy-tailed, but the covariates are light-tailed. Recently, there has been a surge of interest in developing estimators when both covariates and response variables are heavy tailed [5, 44, 17, 43]. However, most of these works are in the batch setting. Another line of work has considered streaming algorithms in the Huber-contamination model, which is a much harder contamination model than heavy-tails [18]. However, these algorithms when adapted to heavy-tailed setting, do not provide optimal rates.

1.3 Contributions

Iteratively Refined Martingale Concentration via PAC Bayes Our key technical result obtains fine-grained concentration guarantees for vector-valued martingales by using the Donsker-Varadhan Variational Principle to iteratively refine baseline concentration inequalities. This allows us to sharpen the PAC Bayes bounds of Catoni and Giulini [8] (and its martingale based extensions like [11]), which were used to analyze the clipped mean estimator. We believe these iterative refinement arguments could be of independent interest for developing fine-grained concentration bounds.

Sharp Analysis of Clipped SGD Leveraging these fine-grained concentration results, We perform a fine-grained analysis of clipped SGD for heavy-tailed SCO problem obtain nearly subgaussian performance guarantees in the streaming setting with a batchsize of 1 and O ( d ) space complexity. In particular, we demonstrate that the sub-optimality gap after T steps scales as Tr (Σ)+ √ ∥ Σ ∥ 2 Tr (Σ) log( 1 / δ ) , improving upon the best known scaling of Tr (Σ) log( 1 / δ ) obtained by prior works [52] only for smooth strongly convex problems. To the best of our knowledge, we derive the first such guarantees for smooth convex and lipschitz convex problems in the streaming setting.

Streaming Heavy Tailed Statistical Estimation We use the above results to develop streaming estimators for various heavy-tailed statistical estimation problems including heavy-tailed mean estimation as well as linear, logistic and Least Absolute Deviation (LAD) regression with heavy tailed covariates, all of which exhibit nearly subgaussian performance. Our mean estimation results improve upon the previous best known guarantees for trimmed mean based estimators [8, 52, 32] (either in performance or in generality) For heavy-tailed linear regression under the assumption of bounded 4 th moments for the covariates and bounded 2 nd moments for the response, our rates significantly improve upon that of the previous best known streaming estimator [52]. To the best of our knowledge, we develop the first known streaming estimators for heavy-tailed logistic regression and LAD regression which attain nearly subgaussian rates

2 Notation and Organization

̸

We work with Euclidean spaces R d equipped with the standard inner product ⟨· , ·⟩ and the induced ℓ 2 norm ∥ · ∥ . For any matrix A ∈ R m × n , we use ∥ A ∥ 2 to denote its Euclidean operator norm ∥ A ∥ = sup x =0 ∥ Ax ∥ / ∥ x ∥ . For A ∈ R d × d , we denote its trace as Tr ( A ) . For any random vector x , we denote its covariance matrix as Cov [ x ] . We use ≲ , ≳ and ≍ to denote ≤ , ≥ and = respectively, upto universal multiplicative constants. We use ∇ f ( x ) to denote the gradient of a differentiable function For any convex function f , we use ∂f ( x ) to denote an arbitrary subgradient of f at x .

3 Background and Problem Formulation

Our work studies the Stochastic Convex Optimization (SCO) problem, described as follows: Let C denote a closed convex subset of R d and let F : C → R be a convex function. We aim to solve:

assuming access to a convex projection oracle Π C and a stochastic gradient oracle , which we define as follows: Let P denote a probability measure supported on an arbitrary domain Ξ from which we can draw samples. A stochastic gradient oracle for F is a function g : C × C , which, given a point x ∈ C and a sample ξ ∼ P returns an unbiased estimate g ( x ; ξ ) of ∇ F ( x ) i.e., E ξ ∼ P [ g ( x ; ξ )] = ∇ F ( x ) . If F is nondifferentiable, E ξ ∼ P [ g ( x ; ξ )] = ∂F ( x ) . Note that we do not assume direct access to ∇ F ( x ) , which may be expensive or intractable to compute. Our objective is to (approximately) solve SCO subject to a constraint on the number of samples we can draw from P .

This is an alternative formulation of the statistical estimation problem by recognizing P as the data distribution, C as the parameter space and defining the population risk F ( x ) := E ξ ∼ P [ f ( x ; ξ )] , where f denotes the sample-level loss function. The associated stochastic gradient oracle is g ( x ; ξ ) := ∇ f ( x ; ξ ) , ξ ∼ P , which is usually easy to compute. As we shall discuss in Section 5, several statistical estimation problems such as mean estimation, linear regression, logistic regression and least absolute deviation regression naturally fit into the SCO framework.

We use n ( x ; ξ ) = g ( x ; ξ ) -∇ F ( x ) to denote the stochastic gradient noise and assume it has finite second moment , i.e., Σ( x ) = E ξ ∼ P [ n ( x ; ξ ) n ( x ; ξ ) T ] exists for every x ∈ C . Our results make use of either of the following assumptions on Σ( x ) .

Assumption 1 (Bounded Second Moment) . The exists a positive semidefinite matrix Σ such that:

Similar assumption has been made by several prior works [21, 40, 14, 45]. We also consider the following generalized assumption, which is as a refinement of the one made in Tsai et al. [52].

Assumption 2 (Second Moment with Quadratic Growth) . There exist constants α, β ≥ 0 and 1 ≤ d eff d such that the following holds for every x ∈ C

where x ∗ denotes any arbitrary minimizer of F .

Since we consider streaming statistical estimators that are robust to heavy tailed data, we only assume the existence of the second moment of the stochastic gradient noise and allow its higher moments to be infinite . That is, our results hold even when E ξ ∼ P [ | ⟨ n ( x ; ξ ) , v ⟩ | 2+ ϵ ] = ∞ for every ϵ > 0 , v ∈ R d

Our work analyzes SCO under either of the following structural assumptions assumptions on F

Assumption 3 (Convexity) . F : R d → R is a convex function if the following holds for any t ∈ [0 , 1]

Assumption 4 ( µ -Strong Convexity) . F : R d → R is a µ -strongly convex function for µ ≥ 0 if the following holds for every t ∈ [0 , 1]

In addition, we also consider either of the two regularity assumptions on F

Assumption 5 ( L -smoothness) . F : R d → R is L -smooth for some L ≥ 0 if F is continuously differentiable and satisfies the following:

Assumption 6 ( G -Lipschitzness) . F : R d → R is G -Lipschitz for some G ≥ 0 , i.e., F is continuous and satisfies the following:

4 Results

Under the Bdd. 2 nd Moment and QG 2 nd Moment assumptions, streaming algorithms for SCO such as Stochastic Gradient Descent (SGD) typically convergence bounds guarantees that hold in expectation [56, 26, 22]. However, high probability guarantees require strong assumptions on the tail behavior of the stochastic gradients (e.g. boundedness or subgaussianity) [25, 47, 31]. Our work analyzes SCO under heavy tailed stochastic gradients, which typically exhibit large fluctuations from their expected value due to its higher order moments being potentially infinite. Clipped SGD mitigates the large fluctuations typically observed in the heavy tailed stochastic gradient g ( x ; ξ ) by thresholding its norm as follows. The full algorithm is described in Algorithm 1.

We now present our performance guarantees for clipped SGD for streaming heavy tailed SCO, wherein Algorithm 1 is subject to an O ( d ) memory constraint and can access only one stochastic gradient sample per iteration. For the remainder, of this section, we use x ∗ ∈ C to denote an arbitrary minimizer of F , which is assumed to always exist, and guaranteed to be unique if F satisfies µ -Strong Convexity. We use x 1 to denote the initialization of Algorithm 1 and let D 1 = ∥ x -x ∗ ∥ .

Algorithm 1 Clipped Stochastic Gradient Descent

Input : Initialization x 1 , Horizon T , Step Sizes ( η t ) t ∈ [ T ] , Clipping Level Γ

1: for t ∈ [ T ] do

2: g t ← g ( x t ; ξ t ) , ξ t ∼ P

3: x t +1 ← Π C ( x t - η t · clip Γ ( g t ))

4: end for

5: Last Iterate : Output x

T +1

6: Average Iterate : Output ˆ x T = 1 T ∑ T t =1 x t

4.1 Smooth Strongly Convex Objectives

Theorems 1 and 2, proved in, Appendix B and C respectively, derive high probability convergence bounds for smooth and strongly convex objectives with second moment assumption. Theorem 1 (Smooth Strongly Convex Objectives) . Let the L -smoothness, µ -Strong Convexity and Bdd. 2 nd Moment assumptions be satisfied. Then, for any δ ∈ (0 , 1 / 2 ) , the last iterate of Algorithm 1 run for T ≳ ln(ln( d )) iterations with stepsize η t = 4 µ ( t + γ ) and clipping level Γ = µ ln( ln( T ) / δ ) √ ( γ +1) 2 D 2 1 + ( T + γ ) µ 2 ( Tr (Σ) + √ Tr (Σ) ∥ Σ ∥ 2 ln( ln( T ) / δ )) satisfies the following with probability at least 1 -δ

where γ ≍ max { ∥ Σ ∥ 2 κ 2 ln( ln( T ) / δ ) 2 Tr (Σ) , κ 3 / 2 ln( ln( T ) / δ ) , κ ln( ln( T ) / δ ) 2 }

We use Theorem 1 to derive sharp rates for streaming heavy tailed mean estimation in Section 5.1 and the following result to derive sharp rates for streaming heavy tailed linear regression in section 5.2 Theorem 2 (Smooth Strongly Convex Objectives with Quadratic Growth Noise Model) . Let Assumptions µ -Strong Convexity, L -smoothness and QG 2 nd Moment be satisfied and let κ = L / µ . For any δ ∈ (0 , 1 / 2 ) , the last iterate of Algorithm 1 run for T ≳ ln(ln( d )) iterations with step-size η t = 4 µ ( t + γ ) and clipping level Γ = µ ln( ln( T ) / δ ) √ ( γ +1) 2 D 2 1 + β µ 2 · ( T + γ )( d eff + √ d eff ln( ln( T ) / δ )) satisfies the following with probability at least 1 -δ

Comparison to Prior Works To the best of our knowledge, the result closest to Theorem 2 is [52, Theorem 1] which analyzes streaming strongly convex SCO and obtains a ζD 1 T + ζ + 1 µ √ βd eff ln( 1 / δ ) T + ζ rate for ζ ≍ αd eff log( 1 / δ ) µ 2 . We note that Theorem 2 obtains a significantly better confidence bound which is closer to the optimal subgaussian rate compared [52, Theorem 1].

Extra log log T term: Our bounds for the statistical error is of the form 1 µ √ β ( d eff + √ d eff ln( ln( T ) / δ )) T + γ which has an extra log log T factor in the lower order term. This is still sharper than prior works with bounds of the form 1 µ √ βd eff ln( 1 / δ ) T + γ as long as log log T ≪ √ d eff log( 1 δ ) .

4.2 Beyond Strongly Convex Objectives

Moving beyond strong convexity, we present Theorems 3 for smooth convex functions and 4 for Lipschitz convex function, proved in Appendix D and E respectively. To the best of our knowledge,

these are the first results for streaming heavy-tailed convex SCO that exhibits near-subgaussian concentration without strong convexity.

Theorem 3 (Smooth Convex Objectives) . Let Convexity, L -smoothness and Bdd. 2 nd Moment be satisfied. Then, for any δ ∈ (0 , 1 / 2 ) and T ≥ ln(ln( d )) , there exists an η ∈ (0 , 1 / 2 L ] such that the average iterate of Algorithm 1 run for T iterations with step-size η t = η and clipping level

where o T ( L, D 1 , Σ) represents terms that are of lower order in T (explicated in Appendix D)

Theorem 4 (Lipschitz Convex Objectives) . Let Assumptions Convexity, G -Lipschitzness and Bdd. 2 nd Moment be satisfied. Then, for any δ ∈ (0 , 1 / 2 ) and T ≥ ln(ln( d )) , there exists an η ∈ (0 , G / √ T ] such that the average iterate of Algorithm 1 run for T iterations with step-size η t = η and clipping

where o T ( G,D 1 , Σ) represents terms that are lower order in T (explicated in Appendix E)

Remark: We use Theorem 3 to design the first known streaming estimator for logistic regression with heavy-tailed covariates in Section 5.3 and Theorem 4 to design the first known streaming estimator for LAD regression with heavy-tailed covariates in Section 5.4.

Remark: In Theorems 3 and 4, the leading order term in the error is of the form: √ √ ( √ ) √

we note that the term dependent on the confidence level log(1 /δ ) is lower order compared to Tr (Σ) . To the best of our knowledge, this is the first work which establishes strong confidence bounds in the setting of SCO without strong convexity. Interestingly, our results also improve the best known rates for sub-Gaussian gradient noise. To be precise, [35, Theorem 3.1] shows a weaker bound of √ D 2 1 ( G 2 + Tr (Σ) log( 1 δ )) / T in the setting of Theorem 4, but when the noise is sub-Gaussian.

5 Applications to Streaming Heavy Tailed Statistical Estimation

5.1 Streaming Heavy-Tailed Mean Estimation

Consider streaming heavy tailed mean estimation with clipped SGD with access to N i.i.d samples from the distribution P . Let Ξ = C , E ξ ∼ P [ ξ ] = m ∈ C . We further assume Cov [ ξ ] ⪯ Σ and allow the higher moments to be infinite. As described in Appendix G.1, this is an SCO problem with the sample loss f ( x ; ξ ) = 1 2 ∥ x -ξ ∥ 2 . The population loss and the stochastic gradient are given by:

The following result, proved in Appendix G.1 via an application of Theorem 1, shows that the last iterate of clipped SGD attains near-subgaussian rates for the heavy tailed mean estimation problem

Corollary 1 (Heavy Tailed Mean Estimation) . Under the stochastic gradient oracle described above, implemented using N ≳ ln(ln( d )) i.i.d samples ξ 1 , . . . , ξ N ∼ P , the last iterate of Algorithm 1 when run under the parameter settings of Theorem 1 satisfies the following with probability at least 1 -δ

where γ ≍ ln( ln( N ) / δ ) 2

Comparison to Prior Works The clipped mean estimator of [8] and the clipped-SGD based estimator in [52] come with a guarantee of the form ∥ ˆ m -m ∥ ≲ √ Tr (Σ) log( 1 δ ) / N with probability 1 -δ . Our result in Corollary 1 obtains a sharper rate of convergence. In a recent work, Lee and Valiant [32] showed that the trimmed mean estimator achieves the optimal rate of √ Tr (Σ) /N when N = ω (log 3 δ -1 ) , d = ω (log 2 ( δ -1 )) . Our result matches this optimal rate in those settings, but is considerably more general, as it holds for any N,d .

5.2 Streaming Heavy Tailed Linear Regression

In the current and subsequent sections, we use θ ∈ C to denote the parameter of F . Let Ξ = R d × R . Given a target parameter θ ∗ ∈ C , P defines the following linear model:

In addition, we make the following bounded 4 th moment asumption on the covariates x

for some numerical constant C 4 ≥ 1 . Note that we allow both the covariate x and the target y to be heavy tailed, assuming only finite moments of upto order 4 for x and order 2 for y . The assumption E [ x ] = 0 is only made for ease of presentation and our arguments easily adapt to E [ x ] = 0 . Our task is to estimate θ ∗ in a streaming fashion with access to N i.i.d samples from P . As described in Appendix G.2, we reframe this problem as SCO under the sample loss f ( θ ; x , y ) = 1 2 ( ⟨ x , θ ⟩ -y ) 2 . The associated population loss F ( θ ) and the stochastic gradient oracle g ( θ ; x , y ) are given by:

̸

Corollary 2 (Heavy Tailed Linear Regression) . Under the stochastic gradient oracle described above, implemented using N ≳ ln(ln( d )) i.i.d samples from P , the last iterate of Algorithm 1 when run under the parameter settings of Theorem 2 satisfies the following with probability at least 1 -δ :

To the best of our knowledge, [52, Corollary 4] is the only other streaming estimator for this problem with subgaussian-style concentration. Our result above significantly improves upon their rates of ∥ θ 1 -θ ∗ ∥ N + ζ + σ λ min (Σ) √ ∥ Σ ∥ 2 d ln( 1 / δ ) N + ζ with ζ = C 4 dκ 2 ln( 1 / δ ) . Furthermore, our result is much closer to the optimal subgaussian rate and gracefully adapts to the stable rank or effective dimension [32], i.e., d eff = Tr (Σ) / ∥ Σ ∥ , therefore implying significant speedups over [52] in settings where d eff ≪ d .

5.3 Streaming Heavy Tailed Logistic Regression

Let Ξ = R d ×{ 0 , 1 } and given a target parameter θ ∗ ∈ C , P denote the following linear-logistic model:

where ϕ ( t ) = (1 + e -t ) -1 . The covariates x are heavy tailed, with only bounded second moments. The negative log likelihood of y | x is given by f ( θ ; x , y ) = ln(1 + exp( ⟨ x , θ ⟩ )) -y ⟨ x , θ ⟩ . The objective of the logistic regression problem is to estimate θ ∗ by minimizing the population-level negative log likelihood:

which is minimized at θ ∗ . Here, the stochastic gradient oracle is g ( θ ; x , y ) = ϕ ( ⟨ x , θ ⟩ ) x -y x . The following result applies Theorem 3 to show that the output of clipped SGD attains near-subgaussian rates for heavy tailed logistic regression. We refer to Appendix G.3 for the proof.

Corollary 3 (Heavy Tailed Logistic Regression) . Under the stochastic subgradient oracle described above, realized using N ≳ ln(ln( d )) i.i.d samples from P , the average iterate of Algorithm 1, when run under the parameter settings of Theorem 4 satisfies the following with probability at least 1 -δ :

where o N (Σ , D 1 ) represents terms that are lower order in N (explicated in Appendix G.3

Note that the standard analysis of SGD, with the assumption that ∥ x ∥ ≤ R almost surely leads to a bound of the form [4, Proposition 5]: F ( ˆ θ N ) -F ( θ ∗ ) ≲ RD 1 √ log( 1 δ ) √ N

5.4 Streaming Heavy Tailed LAD Regression

Let Ξ = R d × R . Given a target parameter θ ∗ ∈ C , P defines the following linear model:

Weallow both the covariate x and target y to be heavy tailed, assuming only bounded second moments for x . We do not assume any moment bounds on ϵ | x . The assumption E [ x ] = 0 is made for the sake of clarity and can be straightforwardly relaxed. The Least Absolute Deviation (LAD) Regression problem involves estimating θ by solving SCO with the sample loss f ( θ ; x , y ) = | ⟨ x , θ ⟩ -y | . The stochastic subgradient oracle and population risk is given by:

̸

where sgn ( t ) = t ∥ t ∥ for t = 0 and sgn (0) = 0 . The following result, whose full statement and proof is presented in Appendix G.4, applies Theorem 4 to show that the average iterate of clipped SGD attains near-subgaussian rates for heavy tailed LAD regression. To the best of our knowledge, this is the first known streaming estimator for this problem.

Corollary 4 (Heavy Tailed LAD Regression) . Under the stochastic subgradient oracle described above, realized using N ≳ ln(ln( d )) i.i.d samples from P , the average iterate of Algorithm 1, when run under the parameter settings of Theorem 4 satisfies the following with probability at least 1 -δ :

where o N denotes terms that are lower order in N (explicated in Appendix G.4)

6 Improved Martingale Concentration via Iterative Refinement

Our results are based on the following concentration result for R d valued martingales. The proof appears in Appendix F. Suppose M t for t = 0 , . . . , T is an R d valued martingale such that M 0 = 0 almost surely, the difference sequence v t := M t -M t -1 is such that ∥ v t ∥ ≤ Γ and E [ v t v ⊺ t |F t -1 ] = Σ t almost surely for every t = 1 , . . . , T for some Γ > 0 . Assume that there exist deterministic sequences p 1 , . . . , p T and q 1 , . . . , q T such that Tr (Σ t ) ≤ q t and ∥ Σ t ∥ ≤ p t almost surely.

Where g ( T, δ ) = C M [ √ ¯ q + ¯ p √ T Γ + Γ √ T log( K δ ) ] and K = ln ln(( √ ¯ qT Γ +1)log( d +1)) + C M for some universal constant C M

To prove this result, we first use Freedman's inequality [20, 51] to obtain a coarse-grained g 0 such that P (sup t ∥ M t ∥ > g 0 √ T ) ≤ δ . We then iteratively refine this inequality via a PAC Bayesian [8, 11, 12] argument to show that P (sup t ∥ M t ∥ > g k +1 √ T | B k ) ≤ δ , where B k = { sup t ∥ M t ∥ ≤ g k √ T } and g 2 k +1 ≲ Tr (Σ) + g k √ ∥ Σ 2 ∥ log( 1 / δ ) . This iterative refinement strategy, proved in Theorem 14 is one of the main technical contributions of our work, which could be of independent interest. We arrive at Theorem 5 after K ≈ log log( T log d ) refinement steps.

Remark Theorem 5 is used to control the influence of the fluctuations introduced by clipped SGD. To this end, let v t be the centered version of clip Γ ( g t ) , ensuring ∥ v t ∥ ≤ 2Γ almost surely. Suppose Σ t = Σ for some fixed Σ and let Γ = √ ∥ Σ ∥ T / log( K δ ) . Then, with probability 1 -δ : sup t ≤ T ∥ M t ∥ ≲ √ T Tr (Σ) + T ∥ Σ ∥ log( K δ ) . This is sharper than the sup t ≤ T ∥ M t ∥ ≲ √ T Tr (Σ) log( d δ ) guarantee implied by the Matrix Freedman inequality [51, Corollary 1.6].

7 Proof Sketch

We sketch our proof technique for the case of smooth convex functions considered in 3. We consider the SGD iterations x 1 , . . . , x T with clipped stochastic gradient at time t denoted by clip Γ ( g t ) = ∇ F ( x t ) + v t + b t . Here, v t is the zero mean 'variance' such that E [ v t | x t ] = 0 and ∥ v t ∥ ≤ 2Γ almost surely. b t is the non-zero mean 'bias' which arises due to clipping. Using the usual analysis of SGD for convex functions (see for instance [31]), we consider:

Considering constant step-sizes, we sum the inequalities for each t to conclude:

The 'random' terms to bound compared to gradient descent here are ∑ t ⟨ v t + b t , x t -x ∗ ⟩ and ∑ t ∥ v t ∥ 2 + ∥ b t ∥ 2 Lemma 13 shows that ∥ x t -x ∗ ∥ ≤ 2 ∥ x 1 -x ∗ ∥ with high probability. Under this event, we bound ∑ t ⟨ v t , x t -x ∗ ⟩ using the standard Freeman's inequality and ∥∇ F ( x t ) ∥ 2 by using smoothness and the fact that ∇ F ( x ⋆ ) = 0 . The bias of the estimator ∥ b t ∥ is bound using arguments similar to [8] (see Lemma 4). The main improvement of our method is given by our method of bounding 1 T ∑ t ∥ v t ∥ 2 . We show by an application of Theorem 5 that 1 T ∑ t ∥ v t ∥ 2 ≲ Tr (Σ) + √ Tr (Σ) ∥ Σ ∥ 2 log( log T δ ) with probability at-least 1 -δ whenever the clipping factor Γ is appropriately chosen. Choosing the step size η appropriately gives us the result in Theorem 3.

8 Conclusion and Limitations

Our work obtained nearly subgaussian rates for heavy-tailed SCO using clipped SGD by developing a fine-grained iterative refinement strategy for martingale concentration. As corollaries, we obtained state-of-the-art streaming estimators for various heavy tailed statistical problems. We note Clipped-SGD is widely used to optimize neural networks with highly nonconvex landscapes, which is currently outside the scope of our work. Nevertheless, we believe our techniques could be useful for providing sharp high-probability guarantees for non-convex losses. Our bounds are currently of the form √ d + √ d ln( ln( T ) / δ ) T , which is suboptimal compared to the tight subgaussian rate of √ d +ln( 1 / δ ) T . Further research is required to understand if it is possible to obtain truly subgaussian rates with clipped mean type estimators. Another notable suboptimality of our result is the ln( ln( T ) / δ ) dependence on the confidence level (as opposed to the typical ln( 1 / δ ) scaling). However, this is not a major drawback as our results continue to significantly outperform prior works unless T ≫ e exp( √ d -1) ln(1 /δ ) (which is an impractical regime). This drawback arises due to the ln(ln( T )) iterations of our iterative refinement technique and we believe it can be removed via more sophisticated martingale concentration arguments. Our work lays the foundation for several interesting avenues for future work including the analysis of heavy tailed statistical estimation under bounded p th moment assumptions (for p < 2 ) and the development of parameter free statistical estimators that do not require knowledge of problem-dependent parameter such as ∥ Σ ∥ , δ etc. (or their respective upper bounds). Deriving anytime valid guarantees for clipped SGD using our techniques is also an interesting future direction.

References

M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

N. Agarwal, S. Chaudhuri, P. Jain, D. M. Nagaraj, and P. Netrapalli. Online target q-learning with reverse experience replay: Efficiently finding the optimal policy for linear mdps. In International Conference on Learning Representations , 2021.

A. Ajalloeian and S. U. Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051 , 2020.

F. Bach. Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression. The Journal of Machine Learning Research , 15(1):595-627, 2014.

A. Bakshi and A. Prasad. Robust linear regression: Optimal rates in polynomial time. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 102-115, 2021.

S. Bubeck. Convex optimization: Algorithms and complexity. 2014. doi: 10.48550/ARXIV. 1405.4980. URL https://arxiv.org/abs/1405.4980 .

O. Catoni. Statistical learning theory and stochastic optimization: Ecole d'Eté de Probabilités de Saint-Flour, XXXI-2001 , volume 1851. Springer Science & Business Media, 2004.

O. Catoni and I. Giulini. Dimension-free pac-bayesian bounds for the estimation of the mean of a random vector. arXiv preprint arXiv:1802.04308 , 2018.

Y. Cherapanamjeri, N. Flammarion, and P. L. Bartlett. Fast mean estimation with sub-gaussian rates. In Conference on Learning Theory , pages 786-806. PMLR, 2019.

Y. Cherapanamjeri, S. B. Hopkins, T. Kathuria, P. Raghavendra, and N. Tripuraneni. Algorithms for heavy-tailed statistics: Regression, covariance estimation, and beyond. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 601-609, 2020.

B. Chugg, H. Wang, and A. Ramdas. Time-uniform confidence spheres for means of random vectors. arXiv preprint arXiv:2311.08168 , 2023.

B. Chugg, H. Wang, and A. Ramdas. A unified recipe for deriving (time-uniform) pac-bayes bounds. Journal of Machine Learning Research , 24(372):1-61, 2023.

A. Cutkosky and H. Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems , 34:4883-4895, 2021.

D. Davis, D. Drusvyatskiy, L. Xiao, and J. Zhang. From low probability to high confidence in stochastic convex optimization. Journal of machine learning research , 22(49), 2021.

J. Depersin and G. Lecué. Robust sub-gaussian estimation of a mean vector in nearly linear time. The Annals of Statistics , 50(1):511-536, 2022.

I. Diakonikolas and D. M. Kane. Algorithmic high-dimensional robust statistics . Cambridge university press, 2023.

I. Diakonikolas, G. Kamath, D. Kane, J. Li, J. Steinhardt, and A. Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning , pages 1596-1606. PMLR, 2019.

I. Diakonikolas, D. M. Kane, A. Pensia, and T. Pittas. Streaming algorithms for high-dimensional robust statistics. arXiv preprint arXiv:2204.12399 , 2022.

J. Fan, W. Wang, and Z. Zhu. A shrinkage principle for heavy-tailed data: High-dimensional robust low-rank matrix recovery. Annals of statistics , 49(3):1239, 2021.

D. A. Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

E. Gorbunov, M. Danilova, and A. Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems , 33: 15042-15053, 2020.

R. M. Gower, N. Loizou, X. Qian, A. Sailanbayev, E. Shulgin, and P. Richtárik. Sgd: General analysis and improved rates. In Proceedings of the 36th International Conference on Machine Learning , pages 5200-5209. PMLR, 2019.

M. Gurbuzbalaban, U. Simsekli, and L. Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

F. R. Hampel, E. M. Ronchetti, P. Rousseeuw, and W. A. Stahel. Robust statistics: the approach based on influence functions . Wiley-Interscience; New York, 1986.

N. J. Harvey, C. Liaw, Y. Plan, and S. Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory , pages 1579-1613. PMLR, 2019.

E. Hazan and S. Kale. Beyond the regret minimization barrier: Optimal algorithms for stochastic strongly-convex optimization. Journal of Machine Learning Research , 15:2489-2512, 2014.

S. Hopkins, J. Li, and F. Zhang. Robust and heavy-tailed mean estimation made simple, via regret minimization. Advances in Neural Information Processing Systems , 33:11902-11912, 2020.

S. B. Hopkins. Mean estimation with sub-gaussian rates in polynomial time. The Annals of Statistics , 48(2):1193-1213, 2020.

D. Hsu and S. Sabato. Loss minimization and parameter estimation with heavy tails. The Journal of Machine Learning Research , 17(1):543-582, 2016.

P. J. Huber. Robust estimation of a location parameter. Ann. Math. Statist. , 35(4):73-101, 1964.

P. Jain, D. M. Nagaraj, and P. Netrapalli. Making the last iterate of sgd information theoretically optimal. SIAM Journal on Optimization , 31(2):1108-1130, 2021.

J. C. Lee and P. Valiant. Optimal sub-gaussian mean estimation in very high dimensions. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022) . Schloss-DagstuhlLeibniz Zentrum für Informatik, 2022.

X. Li and Q. Sun. Variance-aware decision making with linear function approximation under heavy-tailed rewards. Transactions on Machine Learning Research .

Z. Liu and Z. Zhou. Stochastic nonsmooth convex optimization with heavy-tailed noises. arXiv preprint arXiv:2303.12277 , 2023.

Z. Liu, T. D. Nguyen, T. H. Nguyen, A. Ene, and H. Nguyen. High probability convergence of stochastic gradient methods. In International Conference on Machine Learning , pages 21884-21914. PMLR, 2023.

G. Lugosi and S. Mendelson. Mean estimation and regression under heavy-tailed distributions: A survey. Foundations of Computational Mathematics , 19(5):1145-1190, 2019.

G. Lugosi and S. Mendelson. Sub-gaussian estimators of the mean of a random vector. The annals of statistics , 47(2):783-794, 2019.

V. V. Mai and M. Johansson. Stability and convergence of stochastic gradient clipping: Beyond lipschitz continuity and smoothness. In International Conference on Machine Learning , pages 7325-7335. PMLR, 2021.

S. Minsker. Geometric median and robust estimation in banach spaces. Bernoulli , 21(4): 2308-2335, 2015.

A. V. Nazin, A. S. Nemirovsky, A. B. Tsybakov, and A. B. Juditsky. Algorithms of robust stochastic optimization based on mirror descent method. Automation and Remote Control , 80 (9):1607-1627, 2019.

T. D. Nguyen, T. H. Nguyen, A. Ene, and H. Nguyen. Improved convergence in high probability of clipped gradient methods with heavy tailed noise. Advances in Neural Information Processing Systems , 36:24191-24222, 2023.

R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. Pmlr, 2013.

A. Pensia, V. Jog, and P.-L. Loh. Robust regression with covariate filtering: Heavy tails and adversarial contamination. arXiv preprint arXiv:2009.12976 , 2020.

A. Prasad, A. S. Suggala, S. Balakrishnan, and P. Ravikumar. Robust estimation via robust gradient estimation. arXiv preprint arXiv:1802.06485 , 2018.

A. Prasad, S. Balakrishnan, and P. Ravikumar. A robust univariate mean estimator is all you need. In International Conference on Artificial Intelligence and Statistics , pages 4034-4044. PMLR, 2020.

N. Puchkin, E. Gorbunov, N. Kutuzov, and A. Gasnikov. Breaking the heavy-tailed noise barrier in stochastic optimization problems. In International Conference on Artificial Intelligence and Statistics , pages 856-864. PMLR, 2024.

A. Rakhlin, O. Shamir, and K. Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1571-1578, 2012.

A. Sadiev, M. Danilova, E. Gorbunov, S. Horváth, G. Gidel, P. Dvurechensky, A. Gasnikov, and P. Richtárik. High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance. In International Conference on Machine Learning , pages 29563-29648. PMLR, 2023.

V. Srinivasan, A. Prasad, S. Balakrishnan, and P. K. Ravikumar. Efficient estimators for heavy-tailed machine learning. 2020.

Q. Sun, W.-X. Zhou, and J. Fan. Adaptive huber regression. Journal of the American Statistical Association , 115(529):254-265, 2020.

J. Tropp. Freedman's inequality for matrix martingales. Electronic Communications in Probability , 16(none):262 - 270, 2011. doi: 10.1214/ECP.v16-1624. URL https://doi.org/10. 1214/ECP.v16-1624 .

C.-P. Tsai, A. Prasad, S. Balakrishnan, and P. Ravikumar. Heavy-tailed streaming statistical estimation. In International Conference on Artificial Intelligence and Statistics , pages 12511282. PMLR, 2022.

J. W. Tukey. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathematicians, Vancouver, 1975 , volume 2, pages 523-531, 1975.

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

H. Wang and A. Ramdas. Catoni-style confidence sequences for heavy-tailed mean estimation. Stochastic Processes and Their Applications , 163:168-202, 2023.

J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems , 33: 15383-15393, 2020.

W.-X. Zhou, K. Bose, J. Fan, and H. Liu. A new perspective on robust m-estimation: Finite sample theory and applications to dependence-adjusted multiple testing. Annals of statistics , 46 (5):1904, 2018.

Contents

A Preliminaries

In this section, we collect some preliminary concentration results which will be used in the future sections. For the following lemma, we refer to Exercise 2.8.5 in [54].

Lemma 1. Suppose X is a real valued random variable such that | X | ≤ Γ almost surely, E X = 0 and E X 2 = ν . Then, for any λ ∈ R such that | λ | ≤ 1 2Γ , the following holds:

Consider a R d valued martingale ( M t ) T t =0 with respect to the filtration ( F t ) T t =0 such that M 0 = 0 almost surely. We consider the martingale difference sequence v t := M t -M t -1 for t ≥ 1 . Clearly, we must have:

Definition 1. We say that the martingale M t satisfies ( g, T, δ ) uniform concentration if:

Assume that for fixed Γ > 0 and Σ ∈ R d × d that ∥ v s ∥ ≤ Γ almost surely and E [ v t v ⊺ t |F t -1 ] =: Σ t . Suppose Tr (Σ t ) ≤ q t and ∥ Σ t ∥ 2 ≤ p t almost surely for some non-random constants p t , q t . We state a high dimensional version of Freedman's inequality [20, 51] below which follows from From Corollary 1.3 of [51], we have

Theorem 6. Suppose M t satisfies the assumptions above. Let ¯ q := 1 T ∑ T s =1 q t the following is true:

That is, for any δ > 0 , the martingale ( M t ) t ≤ T obeys ( g 0 ( δ ) , T, δ ) uniform concentration, where g 0 ( δ ) = 2Γ 3 √ T log( d +1 δ ) + √ 2¯ q log( d +1 δ )

The following inequality is a corollary of Theorem 6.

Lemma 2. Let g t ∈ R d be F t -1 measurable. Then for some constant c 1 > 0 , we have:

Where A = sup 1 ≤ t ≤ T A t

In addition, we also use the following scalar version of Freedman's inequality

Lemma 3 (Freedman's Inequality) . Let h 1 , h 2 , . . . , h T be a F t adapted martingale difference sequence such that E [ h t |F t -1 ] = 0 , E [ h 2 t |F t -1 ] = σ 2 t and ∥ h t ∥ ≤ τ . Then, for any δ ∈ (0 , 1) , the following holds with probability at least 1 -δ :

The following lemma, which bounds the moments of a clipped random vector, is crucial to our analysis of the bias and variance of the clipped stochastic gradient.

Lemma 4 (Moments of a Clipped Random Vector) . Let z ∈ R d be a random vector sampled from the distribution P with mean m and covariance matrix S . For any Γ > 0 , let ˜ z = clip Γ ( z ) , and let ˜ m and ˜ S denote the mean and covariance of ˜ z respectively, i.e., ˜ m = E [˜ z ] and ˜ S = E [ (˜ z -˜ m ) (˜ z -˜ m ) T ] . Then, the following hold:

Proof. The proof of this lemma uses arguments similar to that of Catoni and Giulini [8]. We first note that for any x ∈ R d

Following the proof of Proposition 2.1 of Catoni and Giulini [8], we observe that for any t > 0 :

Define θ ( x ) = min { 1 , Γ -1 ∥ x ∥} Γ -1 ∥ x ∥ ∀ x ∈ R d . Note that clip Γ ( x ) = θ ( x ) · x . From the above inequality, we note that:

Consider any unit vector e ∈ R d . Then,

where the second step uses the definition of θ ( z ) and the last step uses equation (5). Now, substituting p = 1 and p = 2 in the first and second terms of the RHS respectively, we obtain the following:

where the second step uses the Cauchy Schwarz inequality and the last step uses the subadditivity of the square root. It follows that:

To bound ∥ ˜ S ∥ , we first note that for any x ∈ R d , 0 ≤ θ ( x ) ≤ 1 . As before, let e ∈ R d denote an arbitrary unit vector. We note that E [ ⟨ e , ˜ z -m ⟩ 2 ] = E [ ⟨ e , ˜ z -˜ m ⟩ 2 ] + E [ ⟨ e , m -˜ m ⟩ 2 ] ≥ E [ ⟨ e , ˜ z -˜ m ⟩ 2 ] . Hence, it follows that,

where the fourth step uses Jensen's inequality by noting that 0 ≤ θ ( z ) ≤ 1 and the fifth step uses equation (5).

Finally, To upper bound Tr ( ˜ S ) , we note that clip Γ is a contractive mapping as it is the projection operator onto a convex set (namely the ball of radius Γ in R d centered at the origin). To this end,

The following result, which is a corollary of Theorem 5, is vital for controlling the error introduced due to the variance of the stochastic gradients, and is one of the major components of our analysis. The proof of this result is presented in Appendix F.3

Corollary 5 (PAC Bayesian Inequality for Quadratic Variation) . Let v 1 , . . . , v T be an R d valued martingale difference sequence adapted to the filtration F 1 , . . . , F T satisfying E [ v s |F s ] = 0 , E [ v s v T s |F s ] = Σ s and ∥ v s ∥ ≤ τ almost surely. Let UP ( t ) := min( T, 2 ⌈ log 2 t ⌉ ) . Suppose ∥ Σ s ∥ 2 ≤ p s and Tr (Σ s ) ≤ q s for some fixed sequences p 1 , . . . , p T and q 1 , . . . , q T . Then, there exists a universal constant C lower such that whenever T > C lower log((1 + √ ¯ qT Γ ) log( d +1)) such that the following inequality holds with probability at least 1 -δ , for any δ ∈ (0 , 1 2 ) :

where C M > 0 is an absolute numerical constant.

B Analysis for Smooth Strongly Convex Functions

Let d eff = Tr (Σ) ∥ Σ ∥ 2 and let K = 4max { 8 , C M , ln( T ) } . For t ≥ 1 , define the filtration F t = σ ( x 1 , g s | 1 ≤ s ≤ t ) and F 0 = σ ( x 1 ) . Furthermore, let ∇ F ( x t ) = clip Γ ( g t ) + b t + v t where b t = ∇ F ( x t ) -E [ clip Γ ( g t ) |F t -1 ] and v t = E [ clip Γ ( g t ) |F t -1 ] -clip Γ ( g t ) . We note that E [ v t |F t -1 ] = 0 and

where the first step follows from the triangle inequality, the second step uses Jensen's inequality and the last step uses the definition of clip Γ . Hence v t is an F adapted almost surely bounded martingale difference sequence. Now, let D t = ∥ x t -x ∗ ∥ where x ∗ is the unique minimizer of F (guaranteed by strong convexity).Let η t = A t + γ where A ≥ 1 is a numerical constant and γ ≥ Aκ + A -1 is a constant depending on κ, d and ln( 1 / δ ) which we shall specify later. Note that our choice of γ ensures that η t ≤ 1 L + µ for t ∈ [1 : T ] We prove the following recurrence for D t by using the smoothness and strong convexity properties of F and by exploiting the choice of the step-size.

Lemma 5 (Recurrence for D t ) . The following holds for every t ∈ [1 : T ]

Now define R T,δ as follows:

It is easy to see that Γ = µ √ R T,δ ln( K / δ ) . In our proof of Theorem 1, we shall establish that the following holds with probability at least 1 -δ :

where C > 0 is an absolute numerical constant to be chosen later. To this end, we define the event E t and the random variables d t , ˜ b t , ˜ v t as follows for t ∈ [1 : T +1] :

✶ We use the following Lemma to control the bias vector ˜ b t

✶ We note that since x t is F t -1 measurable, so are ✶ { E t } , D t , d t , b t and ˜ b t . Furthermore, E [˜ v t |F t -1 ] = E [ v t |F t -1 ] { E t } = 0 .

Lemma 6 (Bias Control) . The following holds almost surely for every t ∈ [1 : T ] :

We use the following lemma to control the variance vector ˜ v t . The proof of this lemma, which uses Freedman's inequality and the PAC Bayesian martingale concentration inequality of Corollary 6.

Lemma 7 (Variance Control) . The following holds with probability at least 1 -δ uniformly for every t ∈ [ T ] whenever A ≥ 3 and γ ≥ 4 max { κ 4 / 3 C 2 / 3 ln( ln( T ) / δ ) , κ √ C ln( ln( T ) / δ ) 3 / 2 } :

where C M is the absolute numerical constant defined in Corollary 5.

Equipped with this bound on the bias and the variance, we now present the complete proof as follows:

B.1 Proof of Theorem 1

Proof. Let A ≥ 3 , γ ≥ 4 max { κ 4 / 3 C 2 / 3 ln( ln( T ) / δ ) , κ √ C ln( ln( T ) / δ ) 3 / 2 } . Now, let E denote the following event

Note that by Lemma 7, P ( E ) ≥ 1 -δ . We now claim that P ( ⋂ T +1 t =1 E t | E ) = 1 , i.e., conditioned on the event E , the following holds almost surely for every t ∈ [1 : T +1]

We prove the above claim by induction. Note that the claim is trivially true for t = 1 as R T,δ ≥ ( γ +1) 2 D 2 1 . Now, consider any t ∈ [1 : T ] and suppose the claim holds for some 1 ≤ s ≤ t . Recall that by Lemma 5

Under the induction hypothesis, ✶ { E s } = 1 ∀ s ∈ [ t ] . Hence, Under the induction hypothesis, ✶ { D 2 s ≤ CR T,δ ( s + γ -1)( s + γ -2) } = 1 and thus, d s = x s -x ∗ , b s = ˜ b s , v s = ˜ v s ∀ 1 ≤ s ≤ t . Substituting this transformation into the above inequality, we obtain the following:

We now bound each of the terms in the RHS as follows.

Bounding 1 ⃝ Since A ≥ 1 and t ≥ 1 ,

Bounding 2 ⃝ Since γ and A satisfy the conditions of Lemma 7 and we have conditioned on the event E , it follows that:

Bounding 3 ⃝ Since γ and A satisfy the conditions of Lemma 7 and we have conditioned on the event E , it follows that:

Before controlling terms 4 ⃝ and 5 ⃝ , we note that the following holds for every s ∈ [ t ] by Lemma 6

where B 1 , . . . , B 4 are defined as:

Bounding 4 ⃝ Since ✶ { E s } = 1

Hence,

We now control the first term

where the first inequality follows from the fact that A ≥ 1 and s ≤ t . We now bound the second term

Setting A ≥ 3 / 2 and using the fact that s + γ ≥ 2 , it follows that

where the last inequality follows by setting γ ≥ Cκ 2 d · ln( K / δ ) 2

To control the third term, we set A ≥ 5 / 2 and proceed as follows:

where the last inequality follows by setting γ ≥ κ 3 / 2 C 3 / 4 ln( K / δ ) . To bound the last term,

where the second inequality uses the fact that A ≥ 3 / 2 and the last inequality follows by setting γ ≥ κC 1 / 2 ln( K / δ ) 2 . Putting it all together, it follows that

by setting γ as follows

Bounding 5 ⃝ By Lemma 6 and Jensen's inequality

It follows that

The first term is controlled as follows using the fact that A ≥ 1

The second term is controlled as

where the last inequality follows because γ ≥ κ √ C d ln( K / δ ) For controlling the third term, we set A ≥ 4 to obtain

where the last inequality uses the fact that γ ≥ κ 6 / 5 C 3 / 5 ln( K / δ ) 4 / 5 To control the fourth term, we use the fact that A ≥ 2 to obtain

where the last inequality uses the fact that γ ≥ κ 2 / 3 C 1 / 3 ln( K / δ ) 4 / 3 From the obtained bounds, we conclude that 5 ⃝≤ A 2 4 A +3 R T,δ .

Hence, setting A = 4 and γ = 4 C max { ∥ Σ ∥ 2 κ 2 ln( ln( T ) / δ ) 2 Tr (Σ) , κ 3 / 2 ln( ln( T ) / δ ) , κ ln( ln( T ) / δ ) 2 } , we obtain the following

where the last inequality is obtained by setting C = ( √ 262145 + 524288 C M +75776 ) 2 . It follows that

Thus, we have proved by induction that conditioned on E , D 2 t ≤ CR T,δ ( t + γ ) 2 for every t ∈ [ T +1] . In particular, the following holds with probability at least 1 -δ :

B.2 Proof of Lemma 5

Let ϵ t = b t + v t

By the coercivity lemma in Bubeck [6] ,

It follows that,

where the second inequality follows from the strong monotonicity property of ∇ F ( x ) and the fact that η t ≤ 1 L + µ since γ ≥ Aκ + A -1 . Now, substituting η t = A µ ( t + γ ) ,

Since 1 -t ≤ e -t ∀ t ∈ R , we note that ∀ s < t :

Using the above bound to unroll the recurence (7), we obtain:

Expanding ϵ s = b s + v s and using Young's inequality, we conclude that the following holds for every t ∈ [1 : T ]

B.3 Proof of Lemma 6

Note that by definition of E t

Recall that Γ = µ √ R T,δ ln( K / δ ) i.e. √ R T,δ = γ ln( K / δ ) µ . Substituting this into the above inequality gives us: √

✶ We recall that b t = ∇ F ( x t ) -E [ clip Γ ( g t ) |F t -1 ] = E [ g t |F t -1 ] -E [ clip Γ ( g t ) |F t -1 ] . Since Cov [ g t |F t -1 ] ⪯ Σ by Assumption Bdd. 2 nd Moment, we obtain the following bound on ∥ b t ∥ by an application of Lemma 4

Since b = b { E } , it follows that

Bounding A ⃝ By definition of Γ ,

Hence A ⃝≤ µ √ R T,δ T + γ

Bounding B ⃝ Since R T,δ ≥ ∥ Σ ∥ 2 d eff ( T + γ ) µ 2 ≥ ∥ Σ ∥ 2 T µ 2 , √ ∥ Σ ∥ 2 ≤ µ √ R T,δ √ d ( T + γ ) . Substituting this into equation (8),

Hence, B ⃝≤ µ √ R T,δ · κ ln( 1 / δ ) √ C ( s + γ ) √ d ( T + γ

Bounding C ⃝ From equation (8),

Hence, C ⃝≤ µ √ R T,δ · κ 3 C 3 / 2 ln( 1 / δ ) 2 ( t + γ -1) 3

Bounding D ⃝ Recall that,

It follows that

Hence,

B.4 Proof of Lemma 7

For any s ∈ [ T ] , we recall that v s = E [ clip Γ ( g s ) |F s -1 ] -clip Γ ( g s ) . Since E [ g s |F s -1 ] = ∇ F ( x s ) and Cov [ g s |F s -1 ] ⪯ Σ , we obtain the following from Lemma 4

For s ∈ [1 : T ] define E [˜ v s ˜ v T s |F s -1 ] = ˜ Σ s . Since ✶ { E s } is F s -1 -measurable and ˜ v s = v s ✶ { E s } , it follows that ˜ Σ s = E [ v s v T s |F s ] ✶ { E s } . Hence, we conclude the following from the above inequality

Now, for s ∈ [ t ] , we define h s as follows:

Note that E [ h s |F s -1 ] = ⟨ E [˜ v s |F s -1 ] , d s ⟩ ( s + γ ) 2 A -1 ( t + γ ) 2 A -2 = 0 . Furthermore, since ∥ ˜ v s ∥ ≤ ∥ v s ∥ ≤ 2Γ and ∥ d s ∥ ≤ √ CR T,δ s + γ -1

For s ∈ [ t ] , define σ 2 s = E [ h 2 s |F s -1 ] . It follows that,

where the last inequality follows from equation (9) and the fact that d eff = Tr (Σ) / ∥ Σ ∥ 2 . We now use the above inequality to control ∑ t s =1 σ 2 s ln( K / δ ) as follows:

We now control each of the three terms in the above inequality as follows

Before controlling the remaining two terms, we recall from (8) in the proof of Lemma ?? that where Γ = µ √ R T,δ ln( K / δ ) . It follows that

Thus, we can control the second term in equation (11) as follows

where the second inequality follows by setting A ≥ 2 and the last inequality follows by setting γ ≥ κ 4 / 3 C 2 / 3 ln( K / δ ) .

To control the third term in (11), we note that by equation (8) and the definition of R T,δ

It follows that

where the second inequality follows by setting A ≥ 3 / 2 and the last inequality follows by setting γ ≥ κ √ C ln( K / δ ) 3 / 2 . Substituting the above bounds into equation (11), we note that

Thus, by Freedman's inequality (Lemma 3), we conclude that the following holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ] :

To prove the second inequality of this lemma, we define z s = ˜ v s · ( s + γ t + γ ) A -1 for s ∈ [ t ] . Note that E [ z s |F s -1 ] = 0 and ∥ z s ∥ ≤ ∥ ˜ v s ∥ ≤ 2Γ . Define the PSD matrices G s = E [ z s z T s |F s -1 ] = ( s + γ t + γ ) 2 A -2 ˜ Σ s . Recalling that Tr ( ˜ Σ s ) ≤ Tr (Σ) and the bound obtained on ∥ ˜ Σ s | 2 in equation (9), we infer the following:

Substituting (8) into the bound for ∥ G s ∥ 2 , we obtain the following

By Cauchy Schwarz Inequality,

Since T ≳ ln(ln( d )) , K = ln(ln( T )) and q s ≤ Tr (Σ) ∀ s ∈ [ T ] , our choice of Γ ensures that the conditions of Corollary 5 are satisfied. Hence, by Corollary 5, we conclude that the following holds with probability 1 -δ / 2 uniformly for all t ∈ [ T ]

Simplifying the above using equations (13), (14) and the definition of Γ , we obtain the following inequality which holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ] :

We now simplify each term in the above inequality by using the fact that UP ( t ) ≤ min { T, 2 t } . To this end, the second term is simplified as follows by using the fact that A ≥ 1

We now control the third term as follows using the definition of R T,δ and the fact that A ≥ 1 :

To control the fourth term, we use the fact that A ≥ 3 and note that for s ≤ 2 t , ( s + γ ) ≤ 2( t + γ )

We control the last term by a similar argument

where the last inequality follows by setting γ ≥ 4 κ √ C ln( K / δ ) 3 / 2 . Substituting the obtained bounds into equation (15), we conclude that the following holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ] :

The proof is completed via a union bound.

C Analysis for Smooth Strongly Convex Functions Under Quadratic Growth Noise Model

Following a convention similar to that of Section B, let K = 4max { 8 , C M , ln( T ) } . For t ≥ 1 , define the filtration F t = σ ( x 1 , g s | 1 ≤ s ≤ t ) and F 0 = σ ( x 1 ) . Furthermore, let ∇ F ( x t ) = clip Γ ( g t )+ b t + v t where b t = ∇ F ( x t ) -E [ clip Γ ( g t ) |F t -1 ] and v t = E [ clip Γ ( g t ) |F t -1 ] -clip Γ ( g t ) . As beforem, we note that E [ v t |F t -1 ] = 0 and ∥ v t ∥ ≤ 2Γ . Hence v t is an F adapted almost surely bounded martingale difference sequence. Now, let D t = ∥ x t -x ∗ ∥ where x ∗ is the unique minimizer of F (guaranteed by strong convexity). We also define Σ t = Σ( x t ) and note that ∥ Σ t ∥ ≤ αD 2 t + β and Tr (Σ t ) ≤ d eff ( αD 2 t + β ) . Furthermore Σ t is F t -1 measurable. Let η t = A t + γ where A ≥ 1 is a numerical constant and γ ≥ Aκ + A -1 is a constant depending on κ, d and ln( 1 / δ ) which we shall specify later. Note that our choice of γ ensures that η t ≤ 1 L + µ for t ∈ [1 : T ] An application of Lemma 5 shows that D t satisfies the following for every t ∈ [1 : T ]

We now define R T,δ as follows:

√

It is easy to see that Γ = µ R T,δ ln( K / δ ) . In our proof of Theorem 1, we shall establish that the following holds with probability at least 1 -δ :

where C > 0 is an absolute numerical constant to be chosen later. To this end, we define the event E t and the F t measurable random variables d t , ˜ b t , ˜ v t as follows for t ∈ [1 : T +1] :

Lemma 8 (Bias Control) . The following holds almost surely for every t ∈ [1 : T ] :

✶ We use the following Lemma to control the bias vector ˜ b t

where B 1 , . . . , B 7 are defined as follows:

We use the following lemma to control the variance vector ˜ v t . The proof of this lemma, which uses Freedman's inequality and the PAC Bayesian martingale concentration inequality of Corollary 6.

Lemma 9 (Variance Control) . The following holds with probability at least 1 -δ uniformly for every t ∈ [ T ] for A ≥ 3 and γ ≥ 4 C max { αd eff µ 2 , α ln( K / δ ) µ 2 , κ 4 / 3 ln( K / δ ) , κ ln( K / δ ) 3 / 2 , κ 2 / 3 d 1 / 3 eff α 1 / 3 µ 2 / 3 ln( K / δ ) }

where C M is the absolute numerical constant defined in Corollary 5.

Equipped with this bound on the bias and the variance, we now present the complete proof as follows:

C.1 Proof of Theorem 2

Proof. Let A ≥ 3 , γ ≥ 4 C max { αd eff µ 2 , α ln( K / δ ) µ 2 , κ 4 / 3 ln( K / δ ) , κ ln( K / δ ) 3 / 2 , κ 2 / 3 d 1 / 3 eff α 1 / 3 µ 2 / 3 ln( K / δ ) } . Now, let E denote the following event

Note that by Lemma 9, P ( E ) ≥ 1 -δ . We now claim that P ( ⋂ T +1 t =1 E t | E ) = 1 , i.e., conditioned on the event E , the following holds almost surely for every t ∈ [1 : T +1]

We prove the above claim by induction. Note that the claim is trivially true for t = 1 as R T,δ ≥ ( γ +1) 2 D 2 1 . Now, consider any t ∈ [1 : T ] and suppose the claim holds for some 1 ≤ s ≤ t .

Under the induction hypothesis, ✶ { E s } = 1 ∀ s ∈ [ t ] . Hence, Under the induction hypothesis, ✶ { D 2 s ≤ CR T,δ ( s + γ -1)( s + γ -2) } = 1 and thus, d s = x s -x ∗ , b s = ˜ b s , v s = ˜ v s ∀ 1 ≤ s ≤ t . Substituting this transformation into the above inequality, we obtain the following:

We now bound each of the terms in the RHS as follows.

Bounding 1 ⃝ Since A ≥ 1 and t ≥ 1 ,

Bounding 2 ⃝ Since γ and A satisfy the conditions of Lemma 7 and we have conditioned on the event E , it follows that:

Bounding 3 ⃝ Since γ and A satisfy the conditions of Lemma 7 and we have conditioned on the event E , it follows that:

Bounding 4 ⃝ Since ✶ { E s } = 1

We now control the first term

where the first inequality follows from the fact that A ≥ 1 and s ≤ t .

We now control the second term

where the first inequality follows from the fact that A ≥ 2 and s ≤ t and the second inequality follows by setting γ ≥ 4 αC √ d ln( K / δ ) µ 2 .

We now bound the third term as follows:

where we use the fact that A ≥ 2 and set γ ≥ 4 κ 2 ln( K / δ ) 2 d

. We now bound the fourth term as follows:

where A ≥ 2 and γ ≥ 4 κC ln( K / δ ) √ α µ

We now bound the fifth term as follows

where A ≥ 3 and γ ≥ 4 κ 3 / 2 C 3 / 4 ln( K / δ ) .

We now bound the sixth term as follows where A ≥ 3 and γ ≥ 2 κ √ C ln( K / δ ) 2

Finally, we control the seventh term as follows

where A ≥ 3 and γ ≥ 4 √ ακd ln( K / δ ) C 3 / 4 µ . Putting it all together, it follows that

by setting γ as follows

Bounding 5 ⃝ By Lemma 8 and Jensen's inequality

It follows that

The first term is controlled as follows using the fact that A ≥ 1

The second term is controlled as

where A ≥ 3 and γ ≥ 2 4 / 3 α 2 / 3 C 2 / 3 d 1 / 3 ln( K / δ ) 2 / 3 µ 4 / 3 .

The third term is controlled as

where the last inequality follows because γ ≥ κ √ C d ln( K / δ ) The fourth term is controlled as

where A ≥ 3 and γ ≥ 2 4 / 3 κ 2 / 3 C 2 / 3 ln( K / δ ) 2 / 3 α 1 / 3 µ 2 / 3 For controlling the fifth term, we set A ≥ 4 to obtain

where the last inequality uses the fact that γ ≥ κ 6 / 5 C 3 / 5 ln( K / δ ) 4 / 5

To control the sixth term, we use the fact that A ≥ 2 to obtain

where the last inequality uses the fact that γ ≥ κ 2 / 3 C 1 / 3 ln( K / δ ) 4 / 3

To control the seventh term, we set A ≥ 4 to obtain the following:

where γ ≥ 2 6 / 5 α 2 / 5 κ 2 / 5 d 1 / 5 ln( K / δ ) 4 / 5 C 3 / 5 µ 4 / 5 From the obtained bounds, we conclude that 5 ⃝ ≤ 49 A 2 4 A +1 R T,δ .

Now, we set A = 4 and γ as follows:

Under this setting of A and γ , we obtain the following

where the second inequality holds due to our choice of A and γ and the last inequality is obtained by setting C = ( √ 802817 + 6946816 C M +98304 ) 2 . It follows that

Thus, we have proved by induction that conditioned on E , D 2 t ≤ CR T,δ ( t + γ -1) 2 for every t ∈ [ T +1] . In particular, the following holds with probability at least 1 -δ :

C.2 Proof of Lemma 8

Following the same steps as in that of the proof of Lemma 6, we use Lemma 4 and the fact that Cov [ g t |F t -1 ] = Σ t to obtain:

Bounding A ⃝ Note that by Assumption QG 2 nd Moment

It follows that

Since β √ d eff ln( K / δ ) ≤ µ 2 R T,δ T + γ , we obtain

Bounding B ⃝ Note that by equation (8),

Furthermore, by Assumption QG 2 nd Moment and the definition of E s

Recalling that β ≤ µ 2 R T,δ d eff ( T + γ ) ,

Bounding C ⃝ By equation (8),

Bounding D ⃝ Since βd ≤ µ 2 R T,δ T + γ , it follows tat

Hence, we conclude that

where B 1 , . . . , B 7 are defined as follows:

C.3 Proof of Lemma 9

As before, for s ∈ [1 : T ] define E [˜ v s ˜ v T s |F s -1 ] = ˜ Σ s . Following the same steps as in that of the proof of Lemma 7, we use Lemma 4 and the fact that Cov [ g t |F t -1 ] = Σ t to obtain:

where the second inequality follows from Assumption QG 2 nd Moment and the second inequality follows by definition of E s

Furthermore, since clip Γ is a convex projection, the following holds:

Now, for s ∈ [ t ] , we define h s as follows:

Note that E [ h s |F s -1 ] = 0 . Furthermore, since ∥ ˜ v s ∥ ≤ 2Γ and ∥ d s ∥ ≤ CR T,δ

2 2

where the last inequality follows from equation (9) and the fact that d eff = Tr (Σ) / ∥ Σ ∥ 2 . We now use the above inequality to control ∑ t s =1 σ 2 s ln( K / δ ) as follows:

We now control each of the five terms in the above inequality as follows

To control the second term,

where the second inequality follows by setting A ≥ 3 / 2 and the last inequality follows by setting γ ≥ αC ln( K / δ ) µ 2 Before controlling the remaining terms, we recall from (8) in the proof of Lemma 6 that √

where Γ = µ √ R T,δ ln( K / δ )

Thus, we can control the third term in equation (20) as follows

where the second inequality follows by setting A ≥ 2 and the last inequality follows by setting γ ≥ κ 4 / 3 C 2 / 3 ln( K / δ ) .

To control the fourth term in (20), we note that by equation (8) and the definition of R T,δ

It follows that

where the second inequality follows by setting A ≥ 3 / 2 and the last inequality follows by setting γ ≥ κ √ C ln( K / δ ) 3 / 2 .

To control the fifth term in equation (20), we proceed as follows:

where the second inequality follows by setting A ≥ 2 and the last inequality follows by setting γ ≥ α 1 / 3 d 1 / 3 eff κ 2 / 3 C 2 / 3 ln( K / δ ) µ 2 / 3 Substituting the above bounds into equation (20), we note that

Thus, by Freedman's inequality (Lemma 3), we conclude that the following holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ] :

To prove the second inequality of this lemma, we define z s = ˜ v s · ( s + γ t + γ ) A -1 for s ∈ [ t ] . Note that E [ z s |F s -1 ] = 0 and ∥ z s ∥ ≤ ∥ ˜ v s ∥ ≤ 2Γ . Define the PSD matrices G s = E [ z s z T s |F s -1 ] = ( s + γ t + γ ) 2 A -2 ˜ Σ s . Recalling the bounds obtained on ∥ ˜ Σ s ∥ 2 and Tr ( ˜ Σ s ) in equations (17) and (18), we

infer the following:

Substituting equation (8) into the bound for ∥ G s ∥ 2 , we obtain the following

Since T ≳ ln(ln( d )) , K = ln(ln( T )) , our choice of Γ and the definition of R T,δ ensures that the conditions of Corollary 5 are satisfied. Hence, by Corollary 5, we conclude that the following holds with probability 1 -δ / 2 uniformly for all t ∈ [ T ]

Simplyfing the above using equations (22), (23) and the definition of Γ , we obtain the following:

We now simplify each term in the above inequality by using the fact that UP ( t ) ≤ min { T, 2 t } . To this end, the second term is simplified as follows by using A ≥ 1

We now control the third term by noting that for s ≤ 2 t , s + γ ≤ 2 t + γ ≤ 2( t + γ ) :

where the last inquality follows by setting γ ≥ 4 αCd eff µ 2 .

We now control the fourth term as follows:

We now control the fifth term as follows:

where the last inequality uses γ ≥ 4 αC ln( K / δ ) µ 2

We now simplify the sixth term as follows:

where the last inequality follows by setting γ ≥ 4 κ 4 / 3 C 2 / 3 ln( K / δ ) .

We control the seventh term as follows:

where γ ≥ 4 κ √ C ln( K / δ ) 3 / 2 .

We use a similar argument to simplify the final term as follows:

where γ ≥ 4 κ 2 / 3 d 1 / 3 eff C 2 / 3 ln( K / δ ) µ 2 / 3 . We now set A ≥ 3 and γ as follows:

Under these parameter settings, we substitute the obtained bounds into equation (15), we conclude that the following holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ] :

The proof is completed via a union bound.

D Analysis for Smooth Convex Functions

Let d eff = Tr (Σ) ∥ Σ ∥ 2 . Following a convention similar to that of Section B, let K = 4max { 8 , C M , ln( T ) } . For t ≥ 1 , define the filtration F t = σ ( x 1 , g s | 1 ≤ s ≤ t ) and F 0 = σ ( x 1 ) . Furthermore, let ∇ F ( x t ) = clip Γ ( g t ) + b t + v t where b t = ∇ F ( x t ) -E [ clip Γ ( g t ) |F t -1 ] and v t = E [ clip Γ ( g t ) |F t -1 ] -clip Γ ( g t ) . As beforem, we note that E [ v t |F t -1 ] = 0 and ∥ v t ∥ ≤ 2Γ . Hence v t is an F adapted almost surely bounded martingale difference sequence. Now, let D t = ∥ x t -x ∗ ∥ where x ∗ is the minimizer of F considered in the statement of Theorem 3. Using the smoothness and convexity properties of F , we first prove the following intermediate average iterate guarantee:

Lemma 10 (Intermediate Average Iterate Guarantee) . The following holds for η ≤ 1 / 2 L

Define the events E t and the random vectors d t , ˜ b t and ˜ v t as follows for t ∈ [ T ] :

We use the following lemma to control the bias

Lemma 11 (Bias Control) . For every t ∈ [ T ] , ∥ ˜ b t ∥ ≤ B where B is defined as follows:

We use the following lemma to control the varince

Lemma 12 (Variance Control) . Let V ≥ 0 be defined as follows:

Then the following holds with probability at least 1 -δ uniformly for every t ∈ [ T ]

where C M is a numerical constant and g 2 is defined as follows

Let E denote the following event

We define the constant A as follows:

We now set the clipping level Γ = √ AT ln( K / δ ) . For this choice of Γ , we now obtain the following bound on B :

Similarly, we bound the value of V as follows:

Equipped with the above inequality, we then bound the value of g as:

We prove the following lemma to control the growth of the iterates D t .

Lemma 13 (Iterate Bound) . Let η ≤ c min { 1 / 2 L , D 1 / B ′ T , D 1 / g ′ √ T } where c = 1 √ 8 C M +330 . Then, conditioned on the event E , D t ≤ 2 D 1 ∀ t ∈ [ T ] .

Equipped with the above lemmas, we now present a proof of the following theorem, which is a formal restatement of Theorem 3

Theorem 7 (Smooth Convex Objectives) . Let Convexity, L -smoothness and Bdd. 2 nd Moment be satisfied. Then, for any δ ∈ (0 , 1 / 2 ) and T ≥ ln(ln( d )) , there exists an η ∈ (0 , 1 / 2 L ] such that the average iterate of Algorithm 1 run for T iterations with step-size η t = η and clipping level

D.1 Proof of Theorem 7

We condition on the event E and let η = c 2 min { 1 2 L , D 1 B ′ T , D 1 g ′ √ T } where c = 1 √ 8 C M +330 . Note that this choice of η satisfies the requirements of Lemma 10 and Lemma 13. By Lemma 10, the following holds:

Where the second inequality uses Lemma 11 and the definition of the event E and the third inequality uses ab ≤ a 2 + b 2 / 4 . For the rest of the proof, we shall use C to denote an absolute numerical constant whose value can differ at every step. By our choice of the step-size

Hence, conditioned on the event E , the following holds:

Substituting the values of g ′ , B ′ and V ′ , we obtain the following:

Substituting the value of A , we conclude that the following inequality holds almost surely conditioned on the event E

The prooof is completed by observing that P ( E ) ≥ 1 -δ by Lemma 12 which implies that the above inequality also holds with probability at least 1 -δ

D.2 Proof of Lemma 10

Proof. Since Π C is a contractive operator

By the coercivity property,

Substituting this into the recurrence for D 2 t +1 , we obtain the following:

where the last inequality uses the fact that η ≤ 1 / 2 L , Rearranging and taking averages on both sides

Using the above inequality and the convexity of F , we conclude that

D.3 Proof of Lemma 11

Note that by definition of E t

∥∇ F ( x t ) ∥ ✶ { E t } ≤ LD t ✶ { E t } ≤ 2 LD 1 We recall that b t = E [ g t |F t -1 ] -E [ clip Γ ( g t ) |F t -1 ] . Since Cov [ g t |F t -1 ] ⪯ Σ by Assumption Bdd. 2 nd Moment, we obtain the following bound on ∥ b t ∥ by an application of Lemma 4

Since ˜ b t = b t { E t } , it follows that

D.4 Proof of Lemma 12

For any s ∈ [ T ] , we recall that v s = E [ clip Γ ( g s ) |F s -1 ] -clip Γ ( g s ) . Since E [ g s |F s -1 ] = ∇ F ( x s ) and Cov [ g s |F s -1 ] ⪯ Σ , we obtain the following from Lemma 4

For s ∈ [1 : T ] define E [˜ v s ˜ v T s |F s -1 ] = ˜ Σ s . Since ✶ { E s } is F s -1 -measurable and ˜ v s = v s ✶ { E s } , it follows that ˜ Σ s = E [ v s v T s |F s -1 ] ✶ { E s } . Hence, we conclude the following from the above

inequality

For s ∈ [ T ] , define h s = ⟨ ˜ v s , d s ⟩ . We note that

Hence, by Freedman's inequality (Lemma 3), we conclude that the following holds with probability at least 1 -δ / 2 :

We now apply Corollary 6 with p s = V , q s = Tr (Σ) and τ = 2Γ to conclude that the following holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ]

where

The proof is concluded by a union bound

D.5 Proof of Lemma 13

We prove the claim via induction. Clearly, the claim is true for t = 1 . Now, suppose the claim holds for every s ≤ t for some t ∈ [ T ] . Since Π C is a contractive operator

By the coercivity property,

Substituting this into the recurrence for D 2 t +1 , we obtain the following:

where we use the fact that η ≤ 1 / 2 L . Now, by the Cauchy Schwarz inequality and the fact that ab ≤ a 2 + b 2 / 4 we obtain the following:

It follows that

Unrolling the above recursion for t steps and using the fact that (1 + 1 / 2 T ) T ≤ 2 , we obtain the following:

where the second inequality follows from the Lemma 11 and the fact that we have conditioned on E . Note that by definition of g 2 and the AM-GM inequality

It follows that

where the second inequality uses the definition of η and the fact that B ′ and g ′ upper bound B and G respectively by equations (25) and (27) and the last inequality sets c = 1 √ 8 C M +330 . Hence, D t +1 ≤ 2 D 1 which proves the claim by induction.

E Analysis for Lipschitz Convex Functions

Let d eff = Tr (Σ) ∥ Σ ∥ 2 . Since Σ is positive semidefinite, 1 ≤ d eff ≤ d . Moreover, let clip Γ ( g t ) = ∂F ( x t ) + b t + v t where b t = E [ clip Γ ( g t ) |F t ] -∂F ( x t ) represents the bias due to clipping and E [ v t |F t ] = 0 . Let D t = ∥ x t -x ∗ ∥ where x ∗ is the minimizer of F considered in the statement of Theorem 3. Using the smoothness and convexity properties of F , we first prove the following intermediate average iterate guarantee:

Lemma 14 (Intermediate Average Iterate Guarantee) . The following holds for any η > 0

Define the events E t and the random vectors d t as follows for t ∈ [ T ] :

We use the following lemma to control the bias

Lemma 15 (Bias Control) . For every t ∈ [ T ] , ∥ b t ∥ ≤ B where B is defined as follows:

We use the following lemma to control the varince

Lemma 16 (Variance Control) . Let V ≥ 0 be defined as follows:

Then the following holds with probability at least 1 -δ uniformly for every t ∈ [ T ]

where C M is a numerical constant and g 2 is defined as follows

Let E denote the following event

Note that by Lemma 16, P ( E ) ≥ 1 -δ . We define the constant A as follows:

We now set the clipping level Γ = √ AT ln( K / δ ) . For this choice of Γ , we now simplify the expression for B as follows:

Similarly, the expression for V can be simplified as follows

Using the above inequality, we derive the following upper bound for g :

We also prove the following uniform upper bound on the iterates x t

Lemma 17 (Iterate Bound) . Let η ≤ c min { D 1 / BT , D 1 / g ′ √ T , D 1 / G √ T } where c = 1 √ 8 C M +334 . Then, conditioned on the event E , D t ≤ 2 D 1 ∀ t ∈ [ T ] .

Equipped with the above lemmas, we now prove the following theorem which is a formal restatement of Theorem 4

Theorem 8 (Lipschitz Convex Objectives) . Let Assumptions Convexity, G -Lipschitzness and Bdd. 2 nd Moment be satisfied. Then, for any δ ∈ (0 , 1 / 2 ) and T ≥ ln(ln( d )) , there exists an η ∈ (0 , G / √ T ] such that the average iterate of Algorithm 1 run for T iterations with step-size η t = η and clipping

E.1 Proof of Lemma 14

Proof. Since Π C is a contractive operator

where the second inequality follows from the definition of the subgradient and the G lipschitzness of F . Rearranging and taking averages on both sides

Using the above inequality and the convexity of F , we conclude that

E.2 Proof of Lemma 15

We recall that b t = E [ g t |F t -1 ] -E [ clip Γ ( g t ) |F t -1 ] -. Since Cov [ g t |F t -1 ] ⪯ Σ by Assumption Bdd. 2 nd Moment, we obtain the following bound on ∥ b t ∥ by an application of Lemma 4

E.3 Proof of Lemma 16

For any s ∈ [ T ] , we recall that v s = E [ clip Γ ( g s ) |F s -1 ] -clip Γ ( g s ) . Since E [ g s |F s -1 ] = ∂F ( x s ) and Cov [ g s |F s -1 ] ⪯ Σ , we obtain the following from Lemma 4

For s ∈ [ T ] , define h s = ⟨ v s , d s ⟩ . We note that

Hence, by Freedman's inequality (Lemma 3), we conclude that the following holds with probability at least 1 -δ / 2 :

We now apply Corollary 6 with p s = V , q s = Tr (Σ) and τ = 2Γ to conclude that the following holds with probability at least 1 -δ / 2 uniformly for every t ∈ [ T ]

where

The proof is concluded by a union bound

E.4 Proof of Lemma 17

We prove the claim via induction. Clearly, the claim is true for t = 1 . Now, suppose the claim holds for every s ≤ t for some t ∈ [ T ] . Since Π C is a contractive operator

where the second inequality follows from the definition of the subgradient and the G lipschitzness of F . Now, by the Cauchy Schwarz inequality and the fact that ab ≤ a 2 + b 2 / 4 we obtain the following:

It follows that

Unrolling the above recursion for t steps and using the fact that (1 + 1 / 2 T ) T ≤ 2 , we obtain the following:

where the second inequality follows from the Lemma 15 and the fact that we have conditioned on E . Note that by definition of g 2 and the AM-GM inequality

It follows that

where the second inequality uses the definition of η and the fact that g ′ upper bounds g , and the last inequality sets c = 1 √ 8 C M +334 . Hence, D t +1 ≤ 2 D 1 which proves the claim by induction.

F Improved Martingale Concentration via PAC Bayes Theory

We have the following re-statement of Theorem 5 for the sake of readability.

Theorem 9. Suppose M t for t = 0 , . . . , T is an R d valued martingale such that M 0 = 0 almost surely, the martingale difference sequence v t := M t -M t -1 is such that ∥ v t ∥ ≤ Γ and E [ v t v ⊺ t |F t -1 ] = Σ t almost surely for every t = 1 , . . . , T for some Γ > 0 . Assume that there are deterministic sequences p 1 , . . . , p T and q 1 , . . . , q T such that Tr (Σ t ) ≤ q t and ∥ Σ t ∥ ≤ p t almost surely.

Define the event A t ( g ) := {∥ M t ∥ ≤ g √ T } and B t ( g ) := ∩ t s =1 A s . Consider the quantity N t := ∥ M t ∥ 2 -∑ t s =1 ∥ v s ∥ 2 .

Theorem 10. Let δ ∈ (0 , 1 2 ) and g = g ( T, δ 2 ) be as defined in Theorem 9. Under the conditions of Theorem 9, the following inequality holds for some large enough universal constant C .

The next corollary is a simple consequence of the Theorems 9 and 10.

Corollary 6. Let δ ∈ (0 , 1 2 ) and g = g ( T, δ 3 ) be as specified in Theorem 9. Under the conditions of Theorem 9, the following inequality holds with probability at-least 1 -δ :

F.1 Proof of Theorem 9

The aim of this section is to prove the sharp concentration result given in Theorem 9. We now consider the concentration of norms of the martingale ∥ M t ∥ . Define the event A t := {∥ M t ∥ ≤ g √ T } and B t = ∩ t s =1 A s . Let H be any stopping time for the martingale M t . We have the following inequality which follows from PAC-Bayes theory (see Equation 5.2.1, Page 159 in [7]).

Theorem 11. Suppose π be any measure over R d and let M 1 ( R d ) denote the space of all probability measures over R d . Let γ > 0 be arbitrary. Then conditioned on B T , with probability at-least 1 -δ , the following inequality holds:

We will now bound the exponential moment: E M E θ ∼ π exp( γ ⟨ M t , θ ⟩ ) whenever π = N (0 , I ) .

is a supermartingale with respect to the filtration F t

Proof. Let Σ t := E [ v t v ⊺ t |F t -1 ] and ν t := ∥ Σ t ∥ . First, consider E θ ∼ π exp( γ ⟨ M t , θ ⟩ ) . By the properties of the Gaussians, we must have almost surely:

✶ ✶ Using the fact that ∥ M t ∥ 2 = ∥ v t ∥ 2 +2 ⟨ v t , M t -1 ⟩ + ∥ M t -1 ∥ 2 , we have:

We will now bound the quantity: E [ exp( γ 2 ∥ v t ∥ 2 2 + γ 2 ⟨ v t , M t -1 ⟩ ) ✶ ( A t ) ∣ ∣ ∣ ∣ F t -1 ] . Using the convexity of x → exp( x ) , we conclude:

In the second step, we have used the fact that exp( γ 2 ∥ v t ∥ 2 ) ✶ ( A t ) ≤ 1+ γ 2 ∥ v t ∥ 2 exp( γ 2 Γ 2 ) almost surely using the power series expansion of the exp() function. Using the power series expansion of exp( x ) , we have:

Here, ν t = ∥ Σ t ∥ op In the second step we have used the fact that E [ v t |F t -1 ] = 0 and the fact that ⟨ v t , M t -1 ⟩ ≤ Γ ∥ M t -1 ∥ almost surely. Plugging Equation (36) into Equation (35), we conclude:

Using Equation (37) and that under the event B t -1 we must have ∥ M t -1 ∥ ≤ g √ T , we conclude:

Therefore, by induction, we conclude the statement of the theorem.

Theorem 13. For any stopping time H ,

Proof. From Theorem 12 and the optional stopping theorem, we conclude that the following quantity is a super-martingale:

Therefore, we have:

Combining Theorem 13 and Equation (32), we conclude that the following inequality holds with probability at-least 1 -δ when conditioned on B T :

In the RHS of the inequality above, we replace the supremum over M 1 with the supremum over the set of all probability distributions {N ( αξ, I ) such that ξ ∈ S d -1 , α ≥ 0 } . We note that KL ( N ( αξ, I ) || π ) = α 2 2 to conclude that the following inequality holds with probability at-least 1 -δ when conditioned on B T :

Now, note that by definition,

Therefore, whenever: γ ≤ min ( 1 Γ , 1 2 √ Γ g √ T ) , we note with probability at-least 1 -δ conditioned on the event B T :

We therefore state the following theorem:

Theorem 14. Suppose δ, δ 1 ∈ (0 , 1 2 ) . If M t satisfies ( g, T, δ ) uniform concentration for some δ < 1 2 . Then M t also satisfies ( g ′ , T, δ + δ 1 ) concentration, where

for any γ ≤ min ( 1 Γ , 1 2 √ Γ g √ T ) .

That is:

Additionally, suppose g ≥ c 0 Γ √ T for some fixed constant c 0 > 0 , then we have for some constant C iter ( c 0 ) :

Proof. Since δ ≤ 1 2 , we conclude that P ( B T ) ≥ 1 2 . Given that M t satisfies ( g, T, δ ) uniform concentration. We conclude from the discussion above that for some universal constant C and any γ ≤ min ( 1 Γ , 1 2 √ Γ g √ T ) , we have:

Picking H to be the stopping time given by H = inf { t ≥ 0 : ∥ M t ∥ 2 ≥ C [ T ¯ q + γ 2 ¯ pg 2 T 2 + 1 γ 2 log ( 1 δ 1 ) ] } where C is the same constant as in the equation above, we conclude:

Only in this proof, call the event G := { sup t ≤ T ∥ M t ∥ 2 ≥ C [ T ¯ q + γ 2 ¯ pg 2 T 2 + 1 γ 2 log ( 1 δ 1 ) ] } . We have:

Whenever g ≥ c 0 Γ √ T , we can pick λ = c 1 ( c 0 ) √ Γ g √ T and conclude the result.

We now state consider Lemma 11 from [2].

Lemma 18. Suppose α, β ≤ 0 with α + β > 0 . Consider the function f : R + → R + given by f ( u ) = α + β √ u . Then, f has the unique fixed point: u ∗ := ( β + √ β 2 +4 α 2 ) 2 . For t ∈ N , denoting f ( t ) to be the t fold composition of f with itself, we have for any u ∈ R + :

We are now ready to prove the main theorem 9

Proof of Theorem 9. It is sufficient to show that there exists K = log Θ(log(Γ Td log( 1 δ )))) such that M t obeys ( g, T, δ ) uniform concentration where g = C max( Γ √ T , ¯ q + ¯ p √ T Γ + Γ √ T log( K δ ))

Let K ∈ N be any fixed integer. By Theorem 6, we conclude that the martingale M t is ( g 0 ( δ K ) , T, δ K ) uniformly concentrated. Fix some c 0 > 0 and C iter ( c 0 ) be as in Theorem 14.

Define the sequence g i := √ C iter ( c 0 )¯ q + √ C iter ( c 0 ) g i -1 G where G = ¯ p √ T Γ + Γ √ T log( K δ ) .

If g 0 ≤ c 0 Γ √ T , then the statement of the theorem follows. Suppose there exists K 1 ≤ K -1 such that g K 1 ≤ c 0 Γ √ T and suppose that it is the first such integer. If K 1 = 0 , the statement of the theorem follows from ( g 0 ( δ K ) , T, δ K ) uniform concentration of M t . Suppose 1 ≤ K 1 ≤ K -1 . Then, min( g 0 , . . . , g K 1 -1 ) ≥ c 0 Γ √ T . Then, by Theorem 14, the fact that √ x + y ≤ √ x + √ y and induction, we conclude that M t obeys ( g i , T, ( i +1) δ K ) for every i ≤ K 1 . Thus we conclude the statement of the theorem.

Suppose such a K 1 does not exist. Then, min( g 0 , . . . , g K -1 ) ≥ c 0 Γ √ T . Then, by Theorem 14, the fact that √ x + y ≤ √ x + √ y and induction, we conclude that M t obeys ( g i , T, ( i +1) δ K ) for every i ≤ K -1 . Therefore, it obeys ( g K , T, δ ) uniform concentration.

Consider the function f in Lemma 18 with α = √ C iter ( c 0 )¯ q and β = √ C iter ( c 0 ) G and let the corresponding fixed point be denoted by g ∗ . It is easy to show that the fixed point g ∗ ≲ √ ¯ q + G . After K iterations, we must have:

We can show that picking K = log Θ(log((1 + √ ¯ qT Γ ) log d )) , and the bound on Γ , we conclude the result.

F.2 Proof of Theorem 10

Proof of Theorem 10. Recall that Σ t := E [ v t v ⊺ t |F t -1 ] , ν t := ∥ Σ t ∥ and N t := ∥ M t ∥ 2 -∑ t s =1 ∥ v t ∥ 2 . Note that ν t ≤ p t and Tr (Σ t ) ≤ p t almost surely.

Let γ ∈ R . Define h N ( t ) := ∑ t s =1 log ( 1 + 4 γ 2 p s g 2 T exp(2 | γ | Γ g √ T ) ) with empty sum denoting 0 . We first show that N exp t = exp( γN t -h N ( t )) ✶ ( B T ) is a super martingale with respect to the filtration F t for 0 ≤ t ≤ T . For T ≥ t > 1 , we have:

✶ This shows that N exp t is a super-martingale. Using the fact that N exp 1 = 1 almost surely, the optional stopping theorem and the Chernoff bound, we conclude that for any stopping time H , we have for any α, γ > 0

Taking γ = 1 2Γ g √ T allows us to conclude:

Let α = Γ Cg √ T log( 2 δ ) + CνgT 3 / 2 Γ and take H to be the stopping time min(inf t { t > 0 : N t > α } , T ) where infimum of an empty set is taken to be infinity. We note that { sup t ≤ T N t > α } = { N min( T,H ) > α } . We thus conclude:

We now note that

Hence, we note that the Bdd. 2 nd Moment assumption is satisfied. Hence, the result follows by an application of Theorem 1

Taking γ negative gives the analogous proof for N t < -α .

F.3 Proof of Corollary 5

Proof. Consider the set S = { UP ( t ) : 0 ≤ t ≤ T } . The, | S | ≤ log 2 ( T ) + 1 . By Corollary 6, we have for any t 0 ∈ S , the following is true with probability 1 -δ 1+log 2 ( T )

Therefore, by union bound of the above event over every t 0 ∈ S , we have with probability 1 -δ :

Now, note that ∑ t s =1 ∥ v s ∥ 2 ≤ ∑ UP ( t ) s =1 ∥ v s ∥ 2 almost surely for every t ∈ [ T ] since t ≤ UP ( t ) . Therefore, we conclude that with probability at-least 1 -δ , the following holds for all t ∈ [ T ] simultaneously:

Using the definition of g ( , ) from Theorem 9, we conclude the result.

G Applications to Streaming Heavy Tailed Statistical Estimation

G.1 Streaming Heavy Tailed Mean Estimation : Proof of Corollary 1

Proof. Recall that for this problem, Ξ = C , E ξ ∼ P [ ξ ] = m ∈ C and Cov [ ξ ] ⪯ Σ . Consider the following quadratic loss function f : C → R :

The associated population risk function F is given by

Note that F is L -smooth and µ -strongly convex with L = µ = 1 . Thus, κ = 1 . Furthermore, m is the unique minimizer of F . Hence, solving the streaming heavy tailed mean estimation problem is equivalent to solving the SCO problem for F . To this end, we consider the following stochastic gradient oracle:

It is easy to see that E y [ g ( x ; ξ )] = ∇ F ( x ) , i.e., the stochastic gradient estimate is unbiased. The associated stochastic gradient noise n ( x ; ξ ) is given by

G.2 Streaming Heavy Tailed Linear Regression : Proof of Corollary 2

We use θ ∈ C to denote the parameter of F . Recall from Section 5.2 that Ξ = R d × R , and given a target parameter θ ∗ ∈ C , P defines the following linear model:

In addition, we make the following bounded 4 th moment asumption on the covariates x

for some numerical constant C 4 ≥ 1 . Recall that the sample loss function is given by:

Using the fact that E [ ϵ | x ] = 0 , E [ x ] = 0 and E [ xx T ] = Σ

We note that E [ ϵ 2 ] ≤ σ 2 as per our assumption hence F is well defined. Furthermore.

Thus, the population risk F is L -smooth and µ -strongly convex with L = ∥ Σ ∥ 2 and µ = λ min (Σ) , i.e., κ = ∥ Σ ∥ 2 λ min (Σ) . Furthermore, the unique minimizer of F is θ ∗ . Hence, κ = ∥ Σ ∥ 2 λ min (Σ) the linear regression task of estimating θ ∗ is equivalent to solving SCO for the above objective.

The associated stochastic gradient oracle g ( θ ; x , y ) at any θ ∈ C is given by:

We first show that g ( θ ; x , y )) is indeed an unbiased estimate of ∇ F ( θ )

The associated stochastic gradient noise n ( θ ; x , y )( θ ) is given by

Σ( θ ) = E [ n ( θ ; x , y ) n ( θ ; x , y )] . For convenience, we use M = xx T -Σ and d θ = θ -θ ∗ and note that M is symmetric. It follows that:

where we use the fact that E [ ϵ | x ] = 0 , E [ ϵ 2 | x ] ≤ σ 2 and E [ xx T ] = Σ .

We shall now upper bound ∥ Σ( θ ) ∥ 2 . To do so, we define A ( θ ) = E [ Md θ d T θ M ] and note that A ( θ ) is a PSD matrix since for any v ∈ R d , v T A ( θ ) v = E [ ( v T Md θ ) 2 ] ≥ 0 . Without loss of generality,

̸

we assume θ = θ ∗ and observe that

where we use the fourth moment assumption on the covariates in the eighth step. Note that the above bound also holds when θ = θ ∗ since in that case A ( θ ) = 0 and d θ = 0 . It follows that

We shall now derive an upper bound for Tr (Σ( θ )) as follows:

̸

We now control E [ ∥ Md θ ∥ 2 ] . Note that E [ ∥ Md θ ∥ 2 ] = 0 if θ = θ ∗ so we shall now consider the case when θ = θ ∗ . To this end, let e 1 , . . . , e d be an orthonormal basis of R d such that e 1 = d θ ∥ d θ ∥ .

For the remainder of the proof, we use Σ ij to denote Σ ij = e T i Σ e j where i, j ∈ [ d ] , which implies that Tr (Σ) = ∑ d i =1 Σ ii . Wealso note that for any two symmetric matrices B , C , ( B -C ) 2 ⪯ 2 B 2 +2 C 2 .

Hence,

Clearly, the above bound holds even when θ = θ ∗ . Hence, we infer that

From these bounds, we can conclude the following

Thus, the stochastic gradient oracle satisfies Assumption QG 2 nd Moment with α = 2( C 4 +1) ∥ Σ ∥ 2 2 , β = σ 2 ∥ Σ ∥ and d eff = Tr (Σ) / ∥ Σ ∥ . Hence, the result follows by an application of Theorem 2

G.3 Heavy Tailed Streaming Logistic Regression : Proof of Corollary 3

Recall from Section 5.4 that Ξ = R d ×{ 0 , 1 } and P denotes the following linear-logistic model:

where ϕ ( t ) = (1 + e -t ) -1 . The covariates x are heavy tailed, with only bounded second moments.

The sample-level loss is given by the negative log likelihood of y | x as follows:

The associated population loss and stochastic gradient oracle is given by

We now compute the gradient and the Hessian of F

Since 0 ≤ ϕ ( t ) ≤ 1 for every t ∈ R , we note that 0 ⪯ ∇ 2 F ( θ ) ⪯ E [ xx T ] ⪯ Σ (as E [ x ] = 0 ). Hence, F is convex and L smooth with L = ∥ Σ ∥ 2 . Furthermore, since ∇ F ( θ ∗ ) = 0 and F is convex, we conclude that θ ∗ is a minimizer of F .

It is easy to see that E [ g ( θ ; x , y )] = E [( ϕ ( ⟨ x , θ ⟩ ) -ϕ ( ⟨ x , θ ∗ ⟩ )) x ] = ∇ F ( θ ) , i.e., the stochastic gradient is unbiased. Let n ( θ ; x , y ) denote the stochastic gradient noise, i.e.,:

We shall now control the stochastic gradient covariance Σ( θ ) = E [ n ( θ ; x , y ) n ( θ ; x , y ) T ] . To this end, we define a x ( θ ) and c x ,y ( θ ) as follows:

We note that E [ c x ,y ( θ ) | x ] = 0 and E [ a x ( θ )] = 0 . Since n x ,y ( θ ) = a x ( θ ) + b x ,y ( θ ) , it follows that:

We now control each of the terms in the RHS as follows:

where we use the fact that ϕ ( t ) ≤ 1 . Similarly,

where we use the fact that E [ y 2 | x ] = ϕ ( ⟨ x , θ ∗ ⟩ ) ≤ 1 . It follows that

Thus, the stochastic gradient oracle satisfies the Bdd. 2 nd Moment assumption. Hence, the stochastic gradient oracle satisfies the Bdd. 2 nd Moment assumption. Thus, the following result, which is a formal version of Corollary 3, is implied by Theorem 7

Corollary 7 (Heavy Tailed Logistic Regression) . Under the stochastic subgradient oracle described above, realized using N ≳ ln(ln( d )) i.i.d samples from P , the average iterate of Algorithm 1, when run under the parameter settings of Theorem 4 satisfies the following with probability at least 1 -δ :

G.4 Proof of Corollary 4

Recall from Section 5.4 that Ξ = R d × R and given a target parameter θ ∗ ∈ C , P defines the following linear model:

We allow both the covariate x and target y to be heavy tailed, assuming only bounded second moments for x . We do not assume any moment bounds on ϵ | x . The Least Absolute Deviation (LAD) Regression problem involves estimating θ by solving SCO with the following sample loss

The associated population risk and one possible realization of a stochastic subgradient oracle is given by:

where sgn ( t ) = t ∥ t ∥ for t = 0 and sgn (0) = 0 . We note that for every ( x , y ) ∈ R d × R , f ( θ ; x , y ) is a convex function in θ , and thus, the population risk F is a convex function, whose subgradient is given by:

We now show that F is a Lipschitz function by bounding ∂F ( θ ) as follows:

where the second step follows from Jensen's inequality, the third step uses the fact that | sgn ( t ) | ≤ 1 and applies the Cauchy Schwarz inequality. Hence, F is G -Lipschitz with G = √ Tr (Σ) . We now show that ∂F ( θ ∗ ) = 0 which would imply that θ ∗ is a minimizer of F (as F is convex)

where we use the fact that E [ sgn ( ϵ ) | x ] = 0 , because ϵ | x is a continuous random variable with zero median.

For the stochastic gradient oracle described above, the associated stochastic gradient noise n ( θ ; x , y ) and its covariance Σ( θ ) are given as follows:

Hence, the stochastic gradient oracle satisfies the Bdd. 2 nd Moment assumption. Thus, the following result, which is a formal version of Corollary 4, is implied by Theorem 8

Corollary 8 (Heavy Tailed LAD Regression) .

NeurIPS Paper Checklist

1. Claims

̸

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide complete mathematical proofs of the claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is purely theoretical.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Paper do note include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is purely theoretical and we do foresee any societal impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: purely theoretical work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives a new concentration method to improve the bounds for the convergence of clipped-SGD when the noise has bounded second moments in the online/streaming setting. The method has also applications in streaming heavy-tailed statistical estimation, including streaming mean estimation and regression. The main idea of the method is by bootstrapping Freedman's inequality via iterative refinement using a PAC Bayesian argument. This new concentration method allows to obtain better bounds, closer to the optimal bounds than previous methods.\n\nSTRENGTHS:\n- Overall, I think the paper has made a good contribution in sharpening known bounds for heavy-tailed clipped-SGD. The idea of bootstrapping Freedman's inequality via PAC Bayesian is very interesting and I agree with the authors that it would be interesting to further investigate the applications of this argument.\n- The proposed method has a wide range of applications and the authors have presented them quite thoroughly.\n\nWEAKNESSES:\n- The second moment bound assumption used in this paper appears to be stronger than the variance bound used in prior work for analyzing clipped-SGD. My understanding, for example from the work of Gorbunov et al, is that with the bounded variance assumption they use the simple scalar version of Freedman's so the dependence on the dimension might not be as good as stated here in the paper. It would be good if the authors can discuss this more clearly, for example what if we just use the high dimension version of Freedman's?\n- Another issue with current assumption 1 is that it seems not easy to extend the work to the bounded $p$-moment case, as we also know that we can do this very easily from the work of Gorbunov et al and follow up works\n- The clipping method requires to know the time horizon, which we know is unnecessary in some prior work, eg, Nguyen et al. The bounds are also sub-optimal in $T$ (with an extra $\\log\\log T$ factor).\n- In terms of presentation, I think proofs, especially section F, can be written in a more streamlined way  and for example, with an outline. \n- Some minor details  \nLine 82: There should be a square root in the bound  \nLine 106: We not(e) that  \n\nReference\n\nE. Gorbunov, M. Danilova, and A. Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping, NeuRIPS, 2020\n\nT. D. Nguyen, T. H. Nguyen, A. Ene, and H. Nguyen.  Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise, NeuRIPS, 2023\n\nQUESTIONS:\n- I'm a bit confused about the result in table 1. I didn't do the calculation myself but the bound shown in the last line mentioned both $T$ and $\\epsilon$, which is odd; and the dependence $\\log^2 1/\\delta$ in the lower order term is also higher than other bounds. Could the authors explain?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of high-dimensional heavy-tailed statistical estimation in a streaming setting, which is more challenging than the batch setting due to memory constraints. \n\nThe authors cast the problem as stochastic convex optimization (SCO) with heavy-tailed stochastic gradients. \n\nThey demonstrate that the Clipped Stochastic Gradient Descent (Clipped-SGD) algorithm attains near-optimal sub-Gaussian statistical rates when the second moment of the stochastic gradient noise is finite. \n\nThe rate is better than the previous result in terms of the fluctuation term that depends on $1/\\delta$ in the smooth and strongly convex case and fills the blank for smooth convex and Lipschitz convex cases.\n\nSTRENGTHS:\n1. **Improved Error Bounds**: The paper improves the known error bounds for Clipped-SGD, bringing them closer to the optimal sub-Gaussian rates. \n\n2. **Extension to Various Objectives**: The results are extended to smooth convex and Lipschitz convex objectives, broadening the applicability of Clipped-SGD to a wider range of optimization problems.\n\n3. **Novel Iterative Refinement Strategy**: The introduction of an iterative refinement strategy for martingale concentration is new to me. It might inspire similar techniques.\n\n4. **Good Clarity**: The paper is overall well-written and easy to follow.\n\nWEAKNESSES:\n1. **Missing Reference**: There is one paper from my perspective that is related but not cited.\n\n2. **How to Generalize the Established Results**: The paper contains many new theoretical results. It is unclear how others could use these proposed techniques for their interested cases. There are also some cases that the author didn't mention. See the Questions for the details.\n\nQUESTIONS:\n1. **Missing Reference**: The paper should include a citation to [1*], which modifies adaptive Huber regression for online bandit and MDP with heavy-tailed rewards. This work is related and should be cited in the \"Heavy-tailed Estimation\" part of Section 1.2.\n\n    - [1*] Li, Xiang, and Qiang Sun. \"Variance-aware decision making with linear function approximation under heavy-tailed rewards.\" Transactions on Machine Learning Research.\n\n2. **Extension to Strongly Convex and Lipschitz Cases**: Is it possible to extend the results to the strongly convex and Lipschitz case? If so, how would these results differ from those already derived in this paper?\n\n3. **Optimization Error and Statistical Error**: The error rate (e.g., $\\|x_{T+1}-x^{\\star}\\|$ in Theorem 1) typically comprises two parts: the optimization error (i.e., $\\frac{\\gamma D_1}{T + \\gamma}$ in (1)) and the statistical error (the other term in (1)). The paper's most significant contribution is on the statistical error, ensuring it behaves like a sub-Gaussian rate even when the gradient noise has only bounded variance. Two questions arise:\n   - Can the optimization error be accelerated using the techniques developed in this paper to handle heavy-tailed issues? This might be achieved by another algorithm. I just wonder whether the analysis technique, i.e., the iterative refinement strategy, could be used to tighten the analysis on the statistical error.\n   - Can the iterative refinement strategy be extended to cases with bounded $1+\\delta_0$ moments, where $\\delta_0 \\in (0, 1]$, rather than just bounded variance?\n\n4. **Comparison to Freedman's and Bernstein’s Inequalities**: The improvement in this paper is analogous to the improvement from Hoeffding's inequality to Bernstein’s inequality. However, to account for the Markov structure, Freedman’s inequality and the iterative refinement strategy are used to decouple dependencies. Is this understanding correct?\n\nLIMITATIONS:\nSee the Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces significant advancements in the field of statistical estimation and optimization for heavy-tailed data. It improves the convergence complexity of the widely-used clipped-SGD algorithm in both strongly convex and general convex settings, achieving near-optimal sub-Gaussian statistical rates when the second moment of the stochastic gradient noise is finite.\n\nThe paper also introduces refined concentration guarantees for vector-valued martingales using the Donsker-Varadhan Variational Principle, enhancing the PAC-Bayes bounds of Catoni and Giulini.\nAdditionally, it provides a fine-grained analysis of clipped SGD for heavy-tailed SCO problems, achieving nearly subgaussian performance in a streaming setting with improved complexity over previous work.\nFurthermore, the paper develops streaming estimators for various heavy-tailed statistical problems, including mean estimation and regression (linear, logistic, and LAD), all exhibiting nearly subgaussian performance and improving upon prior guarantees.\n\nSTRENGTHS:\nThe paper presents several noteworthy contributions to the field of statistical estimation and optimization for heavy-tailed data, particularly in improving complexity.\n\nThe paper introduces novel improvements to the convergence complexity of the clipped-SGD algorithm, applicable to both strongly convex and general convex settings, as well as various assumption combinations. Achieving near-optimal sub-Gaussian statistical rates is a significant advancement in the field.\nThe technical rigor of the paper is evident in its thorough analysis and detailed proofs, with over 40 pages in the appendix. The authors systematically build on existing literature, as summarized in Table 1, providing a clear and logical progression of their arguments.\nThe fine-grained analysis of clipped SGD for heavy-tailed SCO problems is meticulously executed, demonstrating nearly subgaussian performance in a streaming setting. This improvement over previous work indicates a high level of quality in the research methodology and execution.\nThe paper is well-organized and clearly written, making complex theoretical concepts accessible. The authors effectively explain the significance of their contributions and how they build upon existing work, enhancing overall readability.\n\nThe improvements to the clipped-SGD algorithm have broad implications for practical applications in machine learning and statistical estimation, particularly in scenarios involving heavy-tailed data distributions.\nBy developing streaming estimators for various heavy-tailed statistical problems, the paper addresses a critical need in the field, providing robust solutions that exhibit nearly subgaussian performance. This has the potential to significantly impact both theoretical research and real-world applications.\n\nOverall, the paper excels in originality, quality, clarity, and significance, making substantial contributions to the field of statistical estimation and optimization for heavy-tailed data.\n\nWEAKNESSES:\nI am not an expert in this area and do not have time to check the proofs in detail. Therefore, my understanding relies solely on the content provided in the writing.\nDespite its significant contributions, the paper has several areas that could benefit from improvement:\n1. Lack of Technique-Proof Sketches in the Main Paper: While the paper presents numerous theorems, there is a noticeable absence of technique-proof sketches. These sketches would provide insights into how the authors build upon existing results compared to previous works. Including such sketches would enhance the clarity and transparency of the theoretical developments, making it easier for readers to grasp the novelty and progression of the contributions.\nThe current structure dedicates only half a page to showcasing the main technique in Section 6, which can be challenging for readers. A more effective approach might involve presenting one key theory with a detailed technique description in the main paper, and relegating supplementary theories to either the main paper with abbreviated details or the appendix.\n2. Theoretical Overemphasis: The paper may overly focus on presenting multiple theories without sufficiently integrating them into a cohesive narrative or demonstrating their practical application. A clearer alignment of the theoretical developments with practical implications would strengthen the paper's impact and relevance.\nFor instance, Section 5, which features many applications, could benefit from highlighting some of them in the background to help readers understand the importance and scenarios of the problems studied.\n3. Absence of Experimental Validation: Statistical estimation typically requires extensive data and iterations to observe convergence reliably. The lack of experimental validation in the paper limits its ability to demonstrate the practical effectiveness of the proposed algorithms.\nIncorporating experiments with diverse datasets and comparisons with existing methods would provide empirical evidence of the proposed techniques' efficacy and robustness in real-world scenarios.\n\nAddressing these points would enhance the paper’s clarity, relevance, and applicability, bridging the gap between theoretical developments and practical implementations in statistical estimation and optimization for heavy-tailed data.\nI welcome further discussion on these points to ensure a comprehensive evaluation. Thanks!\n\nQUESTIONS:\nSee the Weaknesses part.\n\nLIMITATIONS:\nThe paper is theoretical, and I believe there is no need to confirm societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper studies the clipping SGD algorithm and shows a refined analysis to improve the dependence on the variance. The authors also provide different applications of their new results.

STRENGTHS:
The new concentration bounds are interesting, which is the key novel part of the work.

WEAKNESSES:
1. The key drawback is that the algorithm still requires many problem-dependent parameters like $D_1$ and $\mathrm{Tr}(\Sigma)$, which are hard to know, especially in a streaming setting.

1. The time horizon $T$ is also requested to make the algorithm work. Is it possible to extend to the any-time setting?

1. There is an extra $\log{\log{T}}$ term in the bounds.

1. Please specify the stepsize $\eta$ in Theorems 3 and 4.

1. Line 82, missing a square root?

QUESTIONS:
See **Weaknesses**.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13