Title: A Guide Through the Zoo of Biased SGD
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stochastic optimization

biased SGD

Non-convex analysis
Abstract: Stochastic Gradient Descent (SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extensively over at least half a century, SGD variants relying on biased estimators are rare. Nevertheless, there has been an increased interest in this topic in recent years. However, existing literature on SGD with biased estimators lacks coherence since each new paper relies on a different set of assumptions, without any clear understanding of how they are connected, which may lead to confusion. We address this gap by establishing connections among the existing assumptions, and presenting a comprehensive map of the underlying relationships. Additionally, we introduce a new set of assumptions that is provably weaker than all previous assumptions, and use it to present a thorough analysis of BiasedSGD in both convex and non-convex settings, offering advantages over previous results. We also provide examples where biased estimators outperform their unbiased counterparts or where unbiased versions are simply not available. Finally, we demonstrate the effectiveness of our framework through experimental results that validate our theoretical findings.

A Guide Through the Zoo of Biased SGD

Abstract

Stochastic Gradient Descent ( SGD ) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extensively over at least half a century, SGD variants relying on biased estimators are rare. Nevertheless, there has been an increased interest in this topic in recent years. However, existing literature on SGD with biased estimators ( BiasedSGD ) lacks coherence since each new paper relies on a different set of assumptions, without any clear understanding of how they are connected, which may lead to confusion. We address this gap by establishing connections among the existing assumptions, and presenting a comprehensive map of the underlying relationships. Additionally, we introduce a new set of assumptions that is provably weaker than all previous assumptions, and use it to present a thorough analysis of BiasedSGD in both convex and non-convex settings, offering advantages over previous results. We also provide examples where biased estimators outperform their unbiased counterparts or where unbiased versions are simply not available. Finally, we demonstrate the effectiveness of our framework through experimental results that validate our theoretical findings.

1 Introduction

Stochastic Gradient Descent ( SGD ) [Robbins and Monro, 1951] is a widely used and effective algorithm for training various models in machine learning. The current state-of-the-art methods for training deep learning models are all variants of SGD [Goodfellow et al., 2016; Sun, 2020]. The algorithm has been extensively studied in recent theoretical works [Bottou et al., 2018; Gower et al., 2019; Khaled and Richtárik, 2023]. In practice and theory, SGD with unbiased gradient oracles is mostly used. However, there has been a recent surge of interest in SGD with biased gradient oracles, which has been studied in several papers and applied in different domains.

In distributed parallel optimization where data is partitioned across multiple nodes, communication can be a bottleneck, and techniques such as structured sparsity [Alistarh et al., 2018; Wangni et al., 2018] or asynchronous updates [Niu et al., 2011] are involved to reduce communication costs. Nonetheless, sparsified or delayed SGD -updates are not unbiased anymore and require additional analysis [Stich and Karimireddy, 2020; Beznosikov et al., 2020].

Zeroth-order methods are often utilized when there is no access to unbiased gradients, e.g., for optimization of black-box functions [Nesterov and Spokoiny, 2017], or for finding adversarial examples in deep learning [Moosavi-Dezfooli et al., 2017; Chen et al., 2017]. Many zeroth-order training methods exploit biased gradient oracles [Nesterov and Spokoiny, 2017; Liu et al., 2018; Bergou et al., 2020; Boucherouite et al., 2022]. Various other techniques as smoothing, proximate

updates and preconditioning operate with inexact gradient estimators [d'Aspremont, 2008; Schmidt et al., 2011; Devolder et al., 2014; Tappenden et al., 2016; Karimireddy et al., 2018].

The aforementioned applications illustrate that SGD can converge even if it performs biased gradient updates, provided that certain 'regularity' conditions are satisfied by the corresponding gradient estimators [Bottou et al., 2018; Ajalloeian and Stich, 2020; Beznosikov et al., 2020; Condat et al., 2022]. Moreover, biased estimators may show better performance over their unbiased equivalents in certain settings [Beznosikov et al., 2020].

In this work we study convergence properties and worst-case complexity bounds of stochastic gradient descent ( SGD ) with a biased gradient estimator ( BiasedSGD ; see Algorithm 1) for solving general optimization problems of the form

where the function f : R d → R is possibly nonconvex, satisfies several smoothness and regularity conditions.

Assumption 0 Function f is differentiable, L -smooth (i.e., ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ for all x, y ∈ R d ), and bounded from below by f ∗ ∈ R .

We write g ( x ) for the gradient estimator, which is biased (i.e., E [ g ( x )] is not equal to ∇ f ( x ) , E [ · ] stands for the expectation with respect to the randomness of the algorithm), in general. By a gradient estimator we mean a (possibly random) mapping g : R d → R d with some constraints. We denote by γ an appropriately chosen learning rate, and x 0 ∈ R d is a starting point of the algorithm.

Algorithm 1 Biased Stochastic Gradient Descent ( BiasedSGD )

Input: initial point x ∈ R ; learning rate γ > 0

0 d

1: for t = 0 , 1 , 2 , . . . do

2: Construct a (possibly biased) estimator g t def = g ( x t ) of the gradient ∇ f ( x t )

3: Compute x t +1 = x t - γg t

4: end for

In the strongly convex case, f has a unique global minimizer which we denote by x ∗ , and f ( x ∗ ) = f ∗ . In the nonconvex case, f can have many local minima and/or saddle points. It is theoretically intractable to solve this problem to global optimality [Nemirovsky and Yudin, 1983]. Depending on the assumptions on f , and given some error tolerance ε > 0 , will seek to find a random vector x ∈ R d such that one of the following inequalities holds: i) E [ f ( x ) -f ∗ ] ≤ ε (convergence in function values); ii) E ∥ x -x ∗ ∥ 2 ≤ ε ∥ ∥ x 0 -x ∗ ∥ ∥ 2 (iterate convergence); iii) E ∥∇ f ( x ) ∥ 2 ≤ ε 2 (gradient norm convergence).

2 Sources of bias

Practical applications of SGD typically involve the training of supervised machine learning models via empirical risk minimization [Shalev-Shwartz and Ben-David, 2014], which leads to optimization problems of a finite-sum structure:

In the single-machine setup, n is the number of data points, f i ( x ) represents the loss of a model x on a data point i. In this setting, data access is expensive, g ( x ) is usually constructed with subsampling techniques such as minibatching and importance sampling. Generally, a subset S ⊆ [ n ] of examples is chosen, and subsequently g ( x ) is assembled from the information stored in the gradients of ∇ f i ( x ) for i ∈ S only. This leads to estimators of the form g ( x ) = ∑ i ∈ S v i ∇ f i ( x ) , where v i are random variables typically designed to ensure the unbiasedness [Gower et al., 2019]. In practice, points might be sampled with unknown probabilities. In this scenario, a reasonable strategy to estimate the gradient is to take an average of all sampled ∇ f i . In general, the estimator obtained is biased, and

CON

Asm 3

BVD

Asm 4

BREQ

Asm 5

SG1

Asm 1

SG2

Asm 2

BND

Asm 6

FSML

Asm 8

Biased ABC

Asm 9

ABS

Asm 7

Thm2-vi

Thm2-viii

Thm 1-ii

Thm 1-iii

such sources of bias can be characterized as arising from a lack of information about the subsampling strategy.

In the distributed setting, n represemts the number of machines, and each f i represents the loss of model x on all the training data stored on machine i. Since communication is typically very expensive, modern gradient-type methods rely on various gradient compression mechanisms that are usually randomized. Given an appropriately chosen compression map C : R d → R d , the local gradients ∇ f i ( x ) are first compressed to C i ( ∇ f i ( x )) , where C i is an independent realization of C sampled by machine i in each iteration, and subsequently communicated to the master node, which performs aggregation (typically averaging). This gives rise to SGD with the gradient estimator of the form

Many important compressors performing well in practice are of biased nature (e.g., Topk, see Def. 3), which, in general, makes g ( x ) biased as well.

Biased estimators are capable of absorbing useful information in certain settings, e.g., in the heterogeneous data regime. Unbiased estimators have to be random, otherwise they are equal to the identity mapping. However, greedy deterministic gradient estimators such as Topk often lead to better practical performance. In [Beznosikov et al., 2020, Section 4] the authors show an advantage of the Topk compressor over its randomized counterpart Randk when the coordinates of the vector that we wish to compress are distributed uniformly or exponentially. In practice, deterministic biased compressors are widely used for low precision training, and exhibit great performance [Alistarh et al., 2018; Beznosikov et al., 2020].

3 Contributions

The most commonly used assumptions for analyzing SGD with biased estimators take the form of various structured bounds on the first and the second moments of g ( x ) . We argue that assumptions proposed in the literature are often too strong, and may be unrealistic as they do not fully capture how

bias and randomness in g ( x ) arise in practice. In order to retrieve meaningful theoretical insights into the operation of BiasedSGD , it is important to model the bias and randomness both correctly, so that the assumptions we impart are provably satisfied, and accurately, so as to obtain as tight bounds as possible. Our work is motivated by the need of a more accurate and informative analysis of BiasedSGD in the strongly convex and nonconvex settings, which are problems of key importance in optimization research and deep learning. Our results are generic and cover both subsampling and compression-based estimators, among others.

The key contributions of our work are:

Inspired by recent developments in the analysis of SGD in the nonconvex setting [Khaled and Richtárik, 2023], the analysis of BiasedSGD [Bottou et al., 2018; Ajalloeian and Stich, 2020], the analysis of biased compressors [Beznosikov et al., 2020], we propose a new assumption, which we call Biased ABC, for modeling the first and the second moments of the stochastic gradient.

We show in Section 5.2 that Biased ABC is the weakest, and hence the most general, among all assumptions in the existing literature on BiasedSGD we are aware of (see Figure 1), including concepts such as Contractive (CON) [Cordonnier, 2018; Stich et al., 2018; Beznosikov et al., 2020], Absolute (ABS) [Sahu et al., 2021], Bias-Variance Decomposition (BVD) [Condat et al., 2022], Bounded Relative Error Quantization (BREQ) [Khirirat et al., 2018b], Bias-Noise Decomposition (BND) [Ajalloeian and Stich, 2020], Strong Growth 1 (SG1) and Strong Growth 2 (SG2) [Beznosikov et al., 2020], and First and Second Moment Limits (FSML) [Bottou et al., 2018] estimators.

We prove that unlike the existing assumptions, which implicitly assume that the bias comes from either perturbation or compression, Biased ABC also holds in settings such as subsampling.

We recover the optimal rates for general smooth nonconvex problems and for problems under the PŁ condition in the unbiased case and prove that these rates are also optimal in the biased case.

In the strongly convex case, we establish a similar convergence result in terms of iterate norms as in [Hu et al., 2021a], however, under milder assumptions and not only for the classical version of SGD . Our proof strategy is very different and much simpler.

4 Existing models of biased gradient estimators

Since application of a gradient compressor to the gradient constitutes a gradient estimator, below we often reformulate known assumptions and results obtained for biased compressors in the more general form of biased gradient estimators. Beznosikov et al. [2020] analyze SGD under the assumption that f is µ -strongly convex, and propose three different assumptions for compressors.

Assumption 1 (Strong Growth 1, SG1 - Beznosikov et al. [2020]) Let us say that g ( x ) belongs to a set B 1 ( α, β ) of biased gradient estimators, if, for some α, β > 0 , for every x ∈ R d , g ( x ) satisfies

Assumption 2 (Strong Growth 2, SG2 - Beznosikov et al. [2020]) Let us say that g ( x ) belongs to a set B 2 ( τ, β ) of biased gradient estimators, if, for some τ, β > 0 , for every x ∈ R d , g ( x ) satisfies

Note that each of Assumptions 1 and 2 imply

Assumption 3 (Contractive, CON - Beznosikov et al. [2020]) Let us say that g ( x ) belongs to a set B 3 ( δ ) of biased gradient estimators, if, for some δ > 0 , for every x ∈ R d , g ( x ) satisfies

The last condition is an abstraction of the contractive compression property (see Appendix L). Condat et al. [2022] introduce another assumption for biased compressors, influenced by a bias-variance decomposition equation for the second moment:

Let us write the assumption itself.

Assumption 4 (Bias-Variance Decomposition, BVD - Condat et al. [2022]) Let 0 ≤ η ≤ 1 , ξ ≥ 0 , for all x ∈ R d , the gradient estimator g ( x ) satisfies

Khirirat et al. [2018b] proposed another assumption on deterministic compressors.

Assumption 5 (Bounded Relative Error Quantization, BREQ - Khirirat et al. [2018b]) For all x ∈ R d , for any ρ, ζ ≥ 0 ,

The restriction below was imposed on the gradient estimator g ( x ) by Ajalloeian and Stich [2020]. For the purpose of clarity, we rewrote it in the notation adopted in our paper. We refer the reader to Appendix O for the proof of equivalence of these two definitioins.

Assumption 6 (Bias-Noise Decomposition, BND - Ajalloeian and Stich [2020]) Let M,σ 2 , φ 2 be nonnegative constants, and let 0 ≤ m< 1 . For all x ∈ R d , g ( x ) satisfies

The following assumption was introduced by Sahu et al. [2021] (see also the work of Danilova and Gorbunov [2022]).

Assumption 7 (Absolute Estimator, ABS - Sahu et al. [2021]) For all x ∈ R d , there exists ∆ ≥ 0 such that

This condition is tightly related to the contractive compression property (see Appendix M). Further, Bottou et al. [2018] proposed the following restriction on a stochastic gradient estimator.

Assumption 8 (First and Second Moment Limits, FSML - Bottou et al. [2018]) There exist constants 0 < q ≤ u, U ≥ 0 , Q ≥ 0 , such that, for all x ∈ R d ,

Our first theorem, described informally below and stated and proved formally in the appendix, provides required counterexamples of problems and estimators for the diagram in Figure 1.

Theorem 1 (Informal) The assumptions connected by dashed lines in Figure 1 are mutually nonimplicative.

The result says that some pairs of assumptions are in a certain sense unrelated: none implies the other, and vice versa. In the next section, we introduce a new assumption, and provide deeper connections between all assumptions.

5 New approach: biased ABC assumption

5.1 Brief history

Several existing restrictions on the first moment of the estimator were very briefly outlined in the previous section (see (3), (8), (10), (13), (15)). Khaled and Richtárik [2023] recently introduced a very general and accurate Expected Smoothness assumption (we will call it the ABC-assumption in this paper) on the second moment of the unbiased estimator. Let us note that Polyak and Tsypkin [1973] explored a related assumption during their analysis of pseudogradient algorithms. They succeeded in establishing an asymptotic convergence bound for a variant of gradient descent in the unbiased scenario. In contrast, our study focuses on non-asymptotic convergence rates in the biased setting. We generalize the restrictions (3), (10), (15) on the first moment and combine them with the ABC-assumption to develop our Biased ABC framework.

Assumption 9 (Biased ABC) There exist constants A,B,C,b,c ≥ 0 such that the gradient estimator g ( x ) for every x ∈ R d satisfies 1

The term A ( f ( x ) -f ∗ ) in (19) naturally emerges when we bound the expression of the form ∑ n i =1 q i ∥∇ f i ( x ) ∥ 2 , q i ≥ 0 , i ∈ [ n ] : while it can not be confined solely by the norm of the overall gradient B ∥∇ f ( x ) ∥ 2 , nor by a constant C, smoothness suffices to bound this by A ( f ( x ) -f ∗ ) . Further, there exist quadratic stochastic optimization problems where the second moment of a stochastic gradient is precisely equal to 2( f ( x ) -f ∗ ) (see Richtárik and Takáˇ c [2020]).

Concerning the challenges in verifying the Biased ABC assumption, it is worth mentioning that in Machine Learning, loss functions are commonly bounded from below by f ∗ = 0 . In Tables 2 and 8, we provide the constants that validate the fulfillment of our assumption by a wide range of practical estimators. Furthermore, Claims 2-4 can aid in determining these constants for various sampling schemes.

5.2 Biased ABC as the weakest assumption

As discussed in Section 4, there exists a Zoo of assumptions on the stochastic gradients in literature on BiasedSGD . Our second theorem, described informally below and stated and proved formally in the appendix, says that our new Biased ABC assumption is the least restrictive of all the assumptions reviewed in Section 4.

Theorem 2 (Informal) Assumption 9 (Biased ABC) is the weakest among Assumptions 1 - 9.

Inequality (8) of BVD or inequality (13) of BND show that one can impose the restriction on the first moment by bounding the norm of the bias. We choose inequality (18) that restrains the scalar product between the estimator and the gradient on purpose: this approach turns out to be more general on its own. In the proof of Theorem 2-ix (see (46) and (47)) we show that (13) implies (18). Below we show the existence of a counterexample that the reverse implication does not hold.

Claim 1 There exists a finite-sum minimization problem for which a gradient estimator that satisfies inequality (18) of Assumption 9 does not satisfy inequality (13) of Assumption 6.

The relationships among Assumptions 1-9 are depicted in Figure 1 based on the results of Theorem 1 and Theorem 2. It is evident from Figure 1 that Assumption 6 (BND) and Assumption 8 (FSML) are mutually non-implicative and represent the most general assumptions among those proposed in Assumptions 1-8.

The most significant difference between our Assumption 9 (Biased ABC) and Assumptions 6 and 8 is the inclusion of the term A ( f ( x ) -f ∗ ) in the bound on the second moment of the estimator.

The rationale behind this inclusion was detailed in Section 5.1. In general, estimators of the form ∑ n i =1 q i |∇ f i ( x ) | 2 , where q i ≥ 0 , for i ∈ [ n ] , often arise in sampling schemes. We present two practical settings with sampling schemes (see Definitions 1 and 2) that can be described within the Biased ABC framework. These settings, in general, fall outside of the BND and FSML frameworks.

In Section D.2 (see Proof of Theorem 2, parts viii and ix) we present an example of a setting with a minimization problem and a gradient estimator that justifies the introduction of this term: BND and FSML frameworks do not capture the proposed setting, while Biased ABC does capture it.

In Table 1 we provide a representation of each of Assumptions 1 - 8 in our Biased ABC framework (based on the results of Theorem 13). Note that the constants in Table 1 are too pessimistic: given the estimator satisfying one of these assumptions, direct computation of constants in Biased ABC scope for it might lead to much more accurate results. In Table 2 we give a description of popular gradient estimators in terms of the Biased ABC framework. Finally, in Table 3 we list several popular estimators and indicate which of Assumptions 1-9 they satisfy.

6 Convergence of biased SGD under the biased ABC assumption

Convergence rates of theorems below are summarized in Table 4 and compared to their counterparts.

6.1 General nonconvex case

Theorem 3 Let Assumptions 0 and 9 hold. Let δ 0 def = f ( x 0 ) -f ∗ , and choose the stepsize such that 0 < γ ≤ b LB . Then the iterates { x t } t ≥ 0 of BiasedSGD (Algorithm (1) ) satisfy

While one can notice the possibility of an exponential blow-up in (20), by carefully controlling the stepsize we still can guarantee the convergence of BiasedSGD . In Corollaries 5 and 6 (see the appendix) we retrieve the results of Theorem 2 and Corollary 1 from [Khaled and Richtárik, 2023] for the unbiased case. In Corollary 7 (see the appendix) we retrieve the result that is worse than that in [Ajalloeian and Stich, 2020, Theorem 4] by a multiplicative factor and an extra additive term, but under milder conditions (cf. Biased ABC and BND in Figure 1; see also Claim 1). If we set A = c = 0 , we recover the result of [Bottou et al., 2018, Theorem 4.8] (see Corollary 8 in the appendix).

6.2 Convergence under PŁ-condition

One of the popular generalizations of strong convexity in the literature is the Polyak-Łojasiewicz assumption [Polyak, 1963; Karimi et al., 2016; Lei et al., 2019]. First, we define this condition.

Assumption 10 (Polyak-Łojasiewicz) There exists µ > 0 such that ∥∇ f ( x ) ∥ 2 ≥ 2 µ ( f ( x ) -f ∗ ) , for all x ∈ R d .

We now formulate a theorem that establishes the convergence of BiasedSGD for functions satisfying this assumption and Assumption 9 .

Theorem 4 Let Assumptions 0, 9 and 10 hold. Choose a stepsize such that

Letting δ 0 def = f ( x 0 ) -f ∗ , for every T ≥ 1 , we have

When c = 0 , the last term in (22) disappears, and we recover the best known rates under the PolyakŁojasiewicz condition [Karimi et al., 2016], but under milder conditions (see Corollary 10 in the appendix). Further, if we set A = 0 , we obtain a result that is slightly weaker than the one obtained by Ajalloeian and Stich [2020, Theorem 6], but under milder assumptions (cf. Biased ABC and BND in Figure 1; see also Claim 1).

6.3 Strongly convex case

Assumption 11 Let f be µ -strongly-convex and continuously differentiable.

Since Assumption 10 is more general than Assumption 11, Theorem 4 can be applied to functions that satisfy Assumption 11. If we set A = c = 0 , we recover [Bottou et al., 2018, Theorem 4.6] (see Corollary 13 in the appendix). If A = C = c = 0 , we retrieve results comparable to those in [Beznosikov et al., 2020, Theorems 12-14], up to a multiplicative factor (see Corollary 14 in the appendix). Due to µ -strong convexity, our result (22) also implies an iterate convergence, since we have ∥ ∥ x T -x ∗ ∥ ∥ 2 ≤ 2 µ E [ f ( x T ) -f ( x ∗ ) ] . However, in this case an additional factor of 2 µ arises. Below we present a stronger result, yet, at a cost of imposing a stricter condition on the control variables from Assumption 9.

Assumption 12 Let A,B,C and b be parameters from Assumption 9. Let µ be a strong convexity constant. Let L be a smoothness constant. Suppose A + L ( B +1 -2 b ) < µ holds.

Under Assumptions 9 and 12 we establish a similar result as the one obtained by Hu et al. [2021a, Theorem 1]. The authors impose a restriction of 1 κ from above on a constant with an analogous role as B +1 -2 b in Assumptions 9 and 12 with A = 0 . However, unlike us, the authors consider only a finite sum case which makes our result more general. Moreover, only a biased version of SGD with a simple sampling strategy is analyzed by Hu et al. [2021a]. Our results are applicable to a larger variety of gradient estimators and obtained under milder assumptions. Also, our proof strategy is different, and much simpler.

Theorem 5 Let Assumptions 0, 9, 11 and 12 hold. For every positive s, satisfying A + L ( B +1 -2 b ) < s < µ, choose a stepsize γ such that

Then the iterates of BiasedSGD (Algorithm 1) for every T ≥ 1 satisfy

In the standard result for (unbiased) SGD , the convergence neighborhood term has the form of γC µ , and it can be controlled by adjusting the stepsize. However, due to the generality of our analysis in the biased case, in (24) we obtain an extra uncontrollable neighborhood term of the form C +2 c s ( µ -s ) .

When A = C = c = 0 , B = 1 , b = 1 , s → 0 , we recover exactly the classical result for GD .

7 Experiments

Datasets, Hardware, and Code Implementation. The experiments utilized publicly available LibSVM datasets Chang and Lin [2011], specifically the splice , a9a , and w8a . These algorithms were developed using Python 3.8 and executed on a machine equipped with 48 cores of Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz.

Experiment: Problem Setting. To validate our theoretical findings, we conducted a series of numerical experiments on a binary classification problem. Specifically, we employed logistic regression with a non-convex regularizer:

and ( a i , y i ) ∈ R d ×{-1 , 1 } , i = 1 , . . . , n represent the training data samples. In all experiments, we set the regularization parameter λ to a fixed value of λ = 1 . We use datasets from the open LibSVM library [Chang and Lin, 2011]. We examine the performance of the proposed BiasedSGD method with biased independent sampling without replacement (we call it BiasedSGD-ind ) in various settings (see Definition 1). The primary goal of these numerical experiments is to demonstrate the alignment of our theoretical findings with the observed experimental results. To assess the performance of the methods throughout the optimization process, we monitor the metric ∥∇ f ( x t ) ∥ 2 , recomputed after every 10 iterations. The algorithms are terminated after completing 5000 iterations. For each method, we use the largest theoretical stepsize. Specifically, for BiasedSGD-ind , the stepsize is determined according to Corollary 4 and Claim 2 with γ = min { 1 √ LAK , b LB , c LC } , where c = 0 , A = max i L i min i p i , B = 0 , C = 2 A ∆ ∗ + s 2 , b = min i p i and s = 0 .

More experimental details are provided in Appendix A.

0

2000

4000

iterations

12

13

14

15

16

17

18

19

|

f

(

x k

)

|

2

BiasedSGD-ind; dataset: splice; sampling: uniform

p=0.01

p=0.1

p=0.5

0

2000

4000

iterations

10 1

|

f

(

x k

)

|

2

BiasedSGD-ind; dataset: w8a; sampling: uniform

p=0.01

p=0.1

p=0.5

0

2000

4000

iterations

10 0

10 1

|

f

(

x k

)

|

2

BiasedSGD-ind; dataset: a9a; sampling: uniform

p=0.01

p=0.1

p=0.5

Experiment: The impact of the parameter p on the convergence behavior. In the first experiment, we investigate how the convergence of BiasedSGD-ind is affected as we increase the probabilities p i , while keeping them equal for all data samples. According to the Corollary 4, larger p i values (resulting in an increase of the expected batch size) allow for a larger stepsize, which, in turn, improves the overall convergence. This behavior is evident in Figure 2. The experiment visualized in Figure 2 involves varying the probability parameter p within the set { 0 . 01 , 0 . 1 , 0 . 5 } . This manipulation directly influences the value of A , consequently affecting the theoretical stepsize γ . In the context of BiasedSGD-ind , the stepsize γ is defined as 1 √ LAK . Acomprehensive compilation of these parameters is represented in Table 7.

8 Conclusion

In this work, we consolidate various recent assumptions regarding the convergence of biasedSGD and elucidate their implication relationships. Moreover, we introduce a weaker assumption, referred to as Biased ABC. We also demonstrate that Biased ABC encompasses stochastic gradient oracles that previous assumptions excluded. With this assumption, we provide a proof of biasedSGD convergence across multiple scenarios, including strongly convex, non-convex, and under the PŁ-condition. Convergence rates that we obtain are the same up to a constant factor due to the broader setting and in some cases they coincide with the rates obtained under stricter assumptions. Furthermore, we examine the most widely used estimators in the literature related to SGD , represent them within the context of our Biased ABC framework, and analyze their compatibility with all previous frameworks.

References

Ahmad Ajalloeian and Sebastian U Stich. Analysis of SGD with biased gradient estimators. arXiv preprint arXiv:2008.00051 , 2020.

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 440-445, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-[]1045.

Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cedric Renggli. The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, pages 5973-5983, 2018.

El Houcine Bergou, Eduard Gorbunov, and Peter Richtárik. Stochastic three points method for unconstrained smooth minimization. SIAM Journal on Optimization , 30(4):2726-2749, 2020.

Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, and Mher Safaryan. On biased compression for distributed learning. arXiv preprint arXiv:2002.12410 , 2020.

Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018.

Soumia Boucherouite, Grigory Malinovsky, Peter Richtárik, and EL Bergou. Minibatch stochastic three points method for unconstrained smooth minimization. arXiv preprint arXiv:2209.07883 , 2022.

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27, 2011.

Congliang Chen, Li Shen, Haozhi Huang, and Wei Liu. Quantized Adam with error feedback. ACM Transactions on Intelligent Systems and Technology (TIST) , 12(5):1-26, 2021a.

Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security , pages 15-26, 2017.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 25294-25307. Curran Associates, Inc., 2021b.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization. IEEE Transactions on Signal Processing , 69:4937-4948, 2021c. doi: 10.1109/TSP.2021.3092377.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. A single-timescale stochastic bilevel optimization method. arXiv preprint arXiv:2102.04671 , 2021d.

Laurent Condat, Kai Yi, and Peter Richtárik. EF-BV: A unified theory of error feedback and variance reduction mechanisms for biased and unbiased compression in distributed optimization. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Jean-Baptiste Cordonnier. Convex optimization using sparsified stochastic gradient descent with memory. Technical report, 2018.

Marina Danilova and Eduard Gorbunov. Distributed methods with absolute compression and error compensation. In Mathematical Optimization Theory and Operations Research: Recent Trends: 21st International Conference, MOTOR 2022, Petrozavodsk, Russia, July 2-6, 2022, Revised Selected Papers , pages 163-177. Springer, 2022.

Alexandre d'Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Optimization , 19(3):1171-1183, 2008.

Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146(1-2):37-75, 2014.

Aritra Dutta, El Houcine Bergou, Ahmed M Abdelmoniem, Chen-Yu Ho, Atal Narayan Sahu, Marco Canini, and Panos Kalnis. On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3817-3824, 2020.

Ilyas Fatkhullin, Igor Sokolov, Eduard Gorbunov, Zhize Li, and Peter Richtárik. EF21 with bells & whistles: practical algorithmic extensions of modern error feedback. arXiv preprint arXiv:2110.03294 , 2021.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . The MIT Press, 2016.

Eduard Gorbunov, Dmitry Kovalev, Dmitry Makarenko, and Peter Richtárik. Linearly converging error compensated SGD. Advances in Neural Information Processing Systems , 33:20889-20900, 2020.

Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General analysis and improved rates. In International Conference on Machine Learning (ICML) , pages 5200-5209. PMLR, 2019.

Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning , pages 1737-1746. PMLR, 2015.

Samuel Horváth, Chen-Yu Ho, Ludovit Horváth, Atal Narayan Sahu, Marco Canini, and Peter Richtárik. Natural compression for distributed deep learning. In Mathematical and Scientific Machine Learning , pages 129-141. PMLR, 2022.

Bin Hu, Peter Seiler, and Laurent Lessard. Analysis of biased stochastic gradient descent using sequential semidefinite programs. Mathematical Programming , 187:383-408, 2021a.

Yifan Hu, Xin Chen, and Niao He. Sample complexity of sample average approximation for conditional stochastic. SIAM J. on Optimization , 30(3):2103-2133, 2020a. ISSN 1052-6234. doi: 10.1137/19M1284865. URL https://doi.org/10.1137/19M1284865 .

Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 2759-2770. Curran Associates, Inc., 2020b. URL https://proceedings.neurips.cc/ paper_files/paper/2020/file/1cdf14d1e3699d61d237cf76ce1c2dca-[]Paper.pdf .

Yifan Hu, Xin Chen, and Niao He. On the bias-variance-cost tradeoff of stochastic optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 22119-22131. Curran Associates, Inc., 2021b. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ b986700c627db479a4d9460b75de7222-[]Paper.pdf .

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Theoretical convergence of multi-step model-agnostic meta-learning. 23(1), 2022. ISSN 1532-4435.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the Polyak-Łojasiewicz condition. Machine Learning and Knowledge Discovery in Databases , pages 795--811, 2016.

Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Adaptive balancing of gradient and update computation times using global geometry and approximate subproblems. Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS , 2018.

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich, and Martin Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. In International Conference on Machine Learning (ICML) , volume 97, pages 3252-3261, 2019.

Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum? id=AU4qHN2VkS . Survey Certification.

Sarit Khirirat, Hamid Reza Feyzmahdavian, and Mikael Johansson. Distributed learning with compressed gradients. arXiv preprint arXiv:1806.06573 , 2018a.

Sarit Khirirat, Mikael Johansson, and Dan Alistarh. Gradient compression for communication-limited convex optimization. In 2018 IEEE Conference on Decision and Control (CDC) , pages 166-171. IEEE, 2018b.

Sarit Khirirat, Sindri Magnússon, and Mikael Johansson. Compressed gradient methods with Hessianaided error compensation. IEEE Transactions on Signal Processing , 69:998-1011, 2020.

Sarit Khirirat, Sindri Magnússon, and Mikael Johansson. Eco-Fedsplit: Federated learning with errorcompensated compression. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5952-5956. IEEE, 2022.

Yunwei Lei, Ting Hu, Guiying Li, and Ke Tang. Stochastic Gradient Descent for nonconvex learning without bounded gradient assumptions. In IEEE Transactions on Neural Networks and Learning Systems , pages 1-7. ISSN 2162-2388, 2019.

Rémi Leluc and François Portier. SGD with coordinate sampling: Theory and practice. Journal of Machine Learning Research , 23(342):1-47, 2022.

Daniel Levy, Yair Carmon, John C. Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems , Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zerothorder stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems (NeurIPS) , 31:3727-3737, 2018.

Konstantin Mishchenko, Bokun Wang, Dmitry Kovalev, and Peter Richtárik. IntSGD: Adaptive floatless compression of stochastic gradients. arXiv preprint arXiv:2102.08374 , 2021.

Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Arkadi Nemirovsky and David Yudin. Problem Complexity and Method Efficiency in Optimization . Wiley, New York, 1983.

Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17(2):527-566, 2017.

Feng Niu, Benjamin Recht, Christopher Re, and Stephen Wright. Hogwild: A lock-free approach to parallelizing Stochastic Gradient Descent. In Advances in Neural Information Processing Systems (NeurIPS) , volume 24, pages 693-701, 2011.

Boris Polyak. Gradient methods for minimizing functionals. U.S.S.R. Computational Mathematics and Mathematical Physics , 3(4):864-878, 1963.

Boris Polyak. Introduction to Optimization . OptimizationSoftware, Inc., 1987.

Boris Polyak and Y.Z. Tsypkin. Pseudogradient adaptation and training algorithms. Automation and Remote Control , 34:377-397, 01 1973.

Peter Richtárik and Martin Takáˇ c. Stochastic reformulations of linear systems: Algorithms and convergence theory. SIAM J. Matrix Anal. Appl. , 41(2):487-524, 2020. ISSN 0895-4798.

Peter Richtárik, Igor Sokolov, and Ilyas Fatkhullin. EF21: A new, simpler, theoretically better, and practically faster error feedback. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Peter Richtárik, Igor Sokolov, Elnur Gasanov, Ilyas Fatkhullin, Zhize Li, and Eduard Gorbunov. 3PC: Three point compressors for communication-efficient distributed training and a better theory for lazy aggregation. In International Conference on Machine Learning (ICML) , pages 18596-18648. PMLR, 2022.

Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics , 22:400-407, 1951.

Atal Sahu, Aritra Dutta, Ahmed M Abdelmoniem, Trambak Banerjee, Marco Canini, and Panos Kalnis. Rethinking gradient sparsification as total error minimization. Advances in Neural Information Processing Systems (NeurIPS , 34:8133-8146, 2021.

Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtárik. Scaling distributed machine learning with in-network aggregation. In In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21) , pages 785-808, 2021.

Mark Schmidt, Nicolas Roux, and Francis Bach. Convergence rates of inexact proximal-gradient methods for convex optimization. Advances in Neural Information Processing Systems (NeurIPS) , 24:1458-1466, 2011.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for SGD with delayed gradients and compressed updates. The Journal of Machine Learning Research , 21(1):9613-9648, 2020.

Sebastian U. Stich, J.-B. Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Nikko Ström. Scalable distributed dnn training using commodity GPU cloud computing. In Interspeech 2015 , 2015.

Ruo-Yu Sun. Optimization for deep learning: An overview. Journal of the Operations Research Society of China , 8(2):249-294, 2020.

Hanlin Tang, Xiangru Lian, Chen Yu, Tong Zhang, and Ji Liu. DoubleSqueeze: Parallel Stochastic Gradient Descent with double-pass error-compensated compression. In Proceedings of the 36th International Conference on Machine Learning (ICML) , 2020.

Rachael Tappenden, Peter Richtárik, and Jacek Gondzio. Inexact coordinate descent: Complexity and preconditioning. Journal of Optimization Theory and Applications , 170:144-176, 2016.

Jie Wang, Rui Gao, and Yao Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926 , 2021.

Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communicationefficient distributed optimization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, pages 1306-1316, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the assumptions for biasedSGD for various applications and proposes a more general and weaker ABC-assumption that covers the mentioned assumptions. The paper provides a detailed comparison with several existing literature trying to build up a unified framework for biasedSGD. It seems to be a further generalization of [Ahmad Ajalloeian and Sebastian U Stich, 2020].\n\nSTRENGTHS:\nThe paper provides a clear link of different assumption used in the BiasedSGD literature and compare them in Figure 1. In additional, it proposes a set of assumption for biasedSGD that can guarantee the same iteration complexity as the unbiased SGD. I appreciate the clarity and structure of these relationships as they can be invaluable when addressing a new problem requiring BiasedSGD, thus saving time for the reader.\n\nWEAKNESSES:\nI mainly have two concerns: missing reference on other biasedSGD methods and how the proposed ABC assumption can serve as a useful assumption. \n\nThis paper overlooks a substantial body of literature on biasedSGD in contexts beyond classical stochastic optimization/finite-sum and distributed training settings, such as\n1. Stochastic compositional/minimax/bilevel optimization. [1,2,3]\n2. Conditional stochastic optimization. [4,5,6]\n3. Distributionally robust optimization. [7,8]\n4. Meta learning. [9]\n\nThese works are centered on constructing biased gradient estimators and analyzing biasedSGD. Unlike the manuscript, these studies not only consider iteration complexity but also the total sample size needed to reach the desired solution, given the potential high sampling/computational cost of creating low-bias or unbiased gradient estimators. This angle is neglected in the manuscript, causing the title \"A Guide Through the Zoo of Biased SGD\" to seem overstated. Consequently, Section 2, addressing the source of bias, appears narrow given the wide applicability of biasedSGD.\n\nI suggest that the authors limit their claims, incorporate and discuss the above references, and address other bias sources. Specifically, it would be insightful if they could compare two biasedSGD construction methodologies:\n1. The approach in [1] that progressively reduces the bias level of the biased gradient estimators as the algorithm evolves.\n2. The approach in [5] that constructs low-bias gradient estimators at a cheap cost at each iteration. \nNotably, it seems that [1,2,3] don't meet the proposed ABC condition, but the proposed biasedSGD algorithm can still achieve optimal sample complexity. For [5], the gradient estimator constructed in [5] should satisfy ABC condition or its variants but the submitted paper fails to address the potential high costs of obtaining gradient estimators that adhere to the proposed ABC assumption. The authors should discuss these limitations.\n\nThe proposed ABC assumption appears difficult to validate. It remains unclear when this condition should be employed.\n\n\n\n[1] Chen, Tianyi, Yuejiao Sun, and Wotao Yin. \"Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems.\" Advances in Neural Information Processing Systems 34 (2021): 25294-25307.\n\n[2] Chen, Tianyi, Yuejiao Sun, and Wotao Yin. \"Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization.\" IEEE Transactions on Signal Processing 69 (2021): 4937-4948.\n\n[3] Chen, Tianyi, Yuejiao Sun, and Wotao Yin. \"A single-timescale stochastic bilevel optimization method.\" arXiv preprint arXiv:2102.04671 (2021).\n\n[4] Hu, Yifan, et al. \"Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning.\" Advances in Neural Information Processing Systems 33 (2020): 2759-2770.\n\n[5] Hu, Yifan, Xin Chen, and Niao He. \"On the bias-variance-cost tradeoff of stochastic optimization.\" Advances in Neural Information Processing Systems 34 (2021): 22119-22131.\n\n[6] Hu, Yifan, Xin Chen, and Niao He. \"Sample complexity of sample average approximation for conditional stochastic optimization.\" SIAM Journal on Optimization 30.3 (2020): 2103-2133.\n\n[7] Levy, Daniel, et al. \"Large-scale methods for distributionally robust optimization.\" Advances in Neural Information Processing Systems 33 (2020): 8847-8860.\n\n[8] Wang, Jie, Rui Gao, and Yao Xie. \"Sinkhorn distributionally robust optimization.\" arXiv preprint arXiv:2109.11926 (2021).\n\n[9] Ji, Kaiyi, Junjie Yang, and Yingbin Liang. \"Theoretical convergence of multi-step model-agnostic meta-learning.\" Journal of machine learning research (2022).\n\nQUESTIONS:\nSee weakness. Other questions:\n\n1. Ahmad Ajalloeian and Sebastian U Stich 2020 has been published. Please update the preprint references if they are published.\n2. Table 4, it says that \"In most cases, we ensure the same rate\", but the table 4 last column does not seem to suggest so. \n3. For strongly convex case, what happens when Assumption 12 fail?\n\nLIMITATIONS:\nThe main drawback of the work is its predominant focus on iteration complexity in specific contexts, such as finite-sum, zeroth-order, and distributed stochastic optimization settings. Other perspectives on BiasedSGD, such as sample costs/computational costs for different problem settings, are not sufficiently addressed. Numerous biasedSGD methods applicable to a variety of other contexts are overlooked.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes connections among the existing assumptions found in the literature on Stochastic Gradient Descent (SGD) with biased estimators. The authors provide a comprehensive map that illustrates the relationships between these assumptions, addressing the lack of coherence found in previous works. Furthermore, they introduce a new set of assumptions, namely Biased ABC that have been proven to be weaker than the previous ones. It enables a thorough analysis of Biased SGD in both convex and non-convex settings. The paper also showcases examples where biased estimators outperform their unbiased counterparts or where unbiased versions are not available. Finally, the authors validate their theoretical findings through experimental results, which further supports the effectiveness of their framework.\n\nSTRENGTHS:\nThis paper provides a comprehensive overview of the existing literature elaborating on connections between the existing assumptions and the proposed assumption used in the analysis for biased SGD.\n\nOften, when comparing two theoretical results, one may demonstrate slightly tighter result by relying on stricter assumptions, while the other may provide a slightly worse dependence but arguing they have milder assumption. Assessing which approach is superior can be a challenging task. In this regard, the authors have tackled a crucial and significant subject. They have made a good effort towards unifying the various assumptions by proposing a new set of assumption which believed to be weaker than all previous ones. These proposed assumptions cover the existing assumptions, and the convergence results can be suitably adapted accordingly.\n\nWEAKNESSES:\nThe newly proposed assumption Biased ABC is an incremental work based on existing work on biased SGD and the work done by Khaled and Richtárik [2023] for unbiased SGD. I felt there is not much novelty exists in this newly proposed assumption.\n\nAlso, in my opinion,  the balance between the discussion on the background and introduction of existing assumptions and the presentation of the authors' own theoretical results could be further improved.\n\nQUESTIONS:\n1. Is there an example to support Claim 1?\n\nLIMITATIONS:\n1. Figure 1 summarized one of the main contribution of this paper. However, the way it presented right now is not ideal. Especially after including the blue dashed lines. With so many lines that are flying around, the main message that this figure tries to deliver is not well received, in my opinion. I would recommend the author find a more concise way to present the connections in Figure 1.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper summarizes various recent assumptions about the convergence of biased SGD and exposes their implication relations. Furthermore, it proposes a weaker assumption, Biased ABC,  in terms of implication relations and analyzes the results under this assumption.\n\nSTRENGTHS:\nThis paper summarizes various recent assumptions about the convergence of biased SGD and exposes their implication relations. \nIt is well written and in particular, Figure 1 is informative.\n\nWEAKNESSES:\nIt seems that submitting a paper, which has an Appendix of 50 pages including proofs, to an international conference where the peer review period is short and there is not enough time to adequately check proofs, is a negative impression in itself.\nIn addition, the fact that the paper ends with a section on experiments and does not include a summary gives a half-hearted impression of the paper's presentation, which I suppose cannot be helped given the insufficient number of pages, but conversely, it seems to mean that this paper is not suitable for an international conference with a tight page limit.\n\nQUESTIONS:\nAlthough the implication shows that Biased ABC is the weaker assumption, at first glance it seems that this assumption is more difficult to check because it requires more variables to be shown to exist and f* is needed than the other assumptions.\nTo explain more intuitively why this assumption is weaker, please explain a situation that can be checked with this assumption but not with the other assumptions, or a situation that applies to this assumption but not to the other assumptions.\nAlso, I know that if Assumption 8 holds, then Assumption 9 holds, but I could not catch the proof that the converse does not hold, so please tell me where in the Appendix it is written.\nThis is important for showing the weakness of Assumption 9, which leads directly to raising the score.\nSince the experiment section is halfway through, wouldn't it be better to put it into the experiment part of the Appendix and write a discussion and conclusion?\n The experiment itself does not seem to make much sense in this paper.\n\nLIMITATIONS:\nToo many pages in Appendix make it difficult to check the correctness of proofs in the limited peer review period of an international conference.\nCurrently, it is difficult to see that the assumption is weaker than Assumption 8.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the relationships among various assumptions used in the analysis of biased SGD. The authors establish connections among the large number of existing assumptions present in the literature, and propose an assumption (Biased ABC) which captures all prior assumptions. The authors also show that this assumption captures stochastic gradient oracles that prior assumptions did not allow. Using this assumption, they give a simple proof of SGD convergence in a number of settings (both convex and non-convex). While in some cases, the rates are slightly suboptimal (due in part to the more general setting they consider), in many cases, these proofs recover the same (or nearly the same) bounds of prior work under weaker assumptions.\n\nSTRENGTHS:\nThis paper provides a very nice overview of the assumptions made in various works on analysis of biased SGD. They establish relationships among these assumptions, and suggest an assumption (Biased ABC) which captures all of these assumptions. This characterization will surely be of great utility to researchers interested or working in this area. By taking a more general assumption, they are able to recover a number of existing results in the literature, sometimes slightly generalizing the results by weakening the assumptions. Sometimes, however, the results obtained are slightly cruder (due to their assumption being weaker). Although there are no significant new proof techniques (at least that I saw) in this paper, I think there is a lot of value in simplifying and unifying results of prior works.\n\nWEAKNESSES:\n[Theorem 2 viii and ix]\n\nIf I am not mistaken, it appears there is a bug in the proof of “The reverse implication does not hold” part of Theorem 2 viii and ix. You consider $f_1(x) = (x_1^2 - x_2^2)/2$ and $f_2(x) = -f_1(x)$, and claim “Due to Claim 2, it satisfies Assumption 9 (functions $f_1$ and $f_2$ are $1$-smooth)”. To apply Claim 2, however, Assumption 13 must hold, which not only requires the functions $f_i$ to be smooth, but also to be bounded from below. $f_1(x)$ is clearly not bounded from below (take $x_1=0$ and $x_2 \\rightarrow +\\infty$). Thus, it seems that Claim 2 cannot be applied to your example.\n\n[Organization of the paper]\n\nI personally found the paper a bit hard to navigate. I think this is largely due to the sheer number of claims, definitions, and parts of proofs that are in the paper. I think some reorganization could greatly improve the readability of the paper.\n\nFor instance, when reading section B of the appendix, you mention the Biased independent sampling without replacement (Definition 1). I was curious why Assumption 8 can break for this setting (i.e., why you need $A>0$ in Assumption 9), but actually understanding this required finding the proof of Claim 2 and the proof of Theorem 2 viii and ix. It didn’t feel like a natural flow to have to dig so much for what seemed to be the punchline of introducing Definition 1. Is there a way to bring this all in one place?\n\nI understand that organizing a paper with so many short results is difficult. However, I think that some results are less interesting than others, and perhaps can be moved to the end of the paper to reduce clutter. For instance, I think the main interesting parts of Theorem 2 concern Assumption 9. Perhaps you can separate these parts from the others, so that these are easier to find?\n\nAnother note is that some parts of the appendix feel like they do not have enough bridge text. For instance, section C is just a long list of Definitions and Claims. Could you add some text describing these settings, or why they might be interesting to the reader?\n\nAlso, in the main body, you mention lots of assumptions, but there is no mention of actual examples where the stochastic gradients satisfy these examples. All of this is saved for the appendix. I think it would help readability of the paper if some of the less interesting assumptions were moved to the appendix (perhaps only keep a couple of the main assumptions in the main body), and you moved some of the examples to the main body (e.g., the Biased independent sampling w/o replacement, or another of your choosing).\n\nQUESTIONS:\n[Regarding Assumption ABC]\n\nI found the term $A(f(x) - f^*)$ term in eq. (19) to be somewhat mysterious, yet there is not much discussion in the main body of your paper as to why you consider this term in the variance bound. In the appendix, it becomes somewhat clear that you introduce $A$ to handle “Biased independent sampling”, but this this isn’t discussed much in the paper. Are there other reasons why you consider this term in the variance bound?\n\nIt is worth noting that a similar assumption (in the unbiased setting) was considered by Polyak and Tsypkin (1973) “Pseudogradient Adaptation and Training Algorithms” (see the first equation in their Section 3). They also don’t really discuss why they consider this term in the variance bound. I don’t think I’ve seen this type of assumption anywhere else in the literature (except in Khaled and Richtárik (2023) “Better theory for SGD in the nonconvex world”, as you mention). Perhaps you have some insights into this? Also, it is probably worth giving the Polyak and Tsypkin paper a citation, given the similarity of your assumptions.\n\n[Regarding the discussions in weaknesses section above]\n\nCould you please address if the error in the proof of Theorem 2 viii and ix is easy to overcome? (I will set my score now to weak accept due to this error). \n\nCould you please describe how you plan to incorporate the feedback on paper organization?\n\n[Regarding Theorem 3]\n\nTheorem 3 gives a guarantee on $\\min_t E[||\\nabla f(x_t)||^2]$, while, at least for the unbiased setting, the results of Ghadimi and Lan (2013) give convergence for the average iterate. Looking at your proof, it appears that you actually get convergence for a weighted average of the gradients, but where the *smaller* iterates are weighted higher. That seemed quite counter-intuitive to me, as I would have expected a higher weighting of later iterates. Do you have any intuition on why the weighting is this way? Is there any hope in getting convergence of the average iterate in your framework?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}