Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problems

1 Introduction . We consider solving finite-sum optimization problems of the form F ( x ) = 1 n n∑ i=1 fi ( x ) using stochastic gradient descent ( SGD ) . Such problems are extremely common in modern machine learning ( e.g. , for empirical risk minimization ) , and stochastic gradient methods are the most popular approach for large-scale problems , where both n and the dimension are large . The classical approach to apply SGD to such problems is to repeatedly sample indices i ∈ { 1 , . . . , n } uniformly at random , and perform updates of the form xt+1 = xt − ηt∇fi ( xt ) , where ηt is a step-size parameter . With this sampling scheme , each ∇fi ( xt ) is a random unbiased estimate of ∇F ( xt ) ( conditioned on xt ) . Thus , the algorithm can be seen as a “ cheap ” noisy version of plain gradient descent on F ( · ) , where each iteration requires computing the gradient of just a single function fi ( · ) , rather than the gradient of F ( · ) ( which would require computing and averaging n individual gradients ) . This key observation facilitates the analysis of SGD , while simultaneously explaining why SGD is much more efficient than gradient descent on large-scale problems . However , the practice of SGD differs somewhat from this idealized description : In practice , it is much more common to perform without-replacement sampling of the indices , by randomly shuffling the indices { 1 , . . . , n } and processing the individual functions in that order – that is , choosing a random permutation σ on { 1 , . . . , n } , and performing updates of the form xt+1 = xt − ηt∇fσ ( t ) ( xt ) . After a full pass over the n functions , further passes are made either with the same permutation σ ( known as single shuffling ) , or with a new random permutation σ chosen before each pass ( known as random 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reshuffling ) . Such without-replacement schemes are not only more convenient to implement in many cases , they often also exhibit faster error decay than with-replacement SGD [ Bottou , 2009 , Recht and Ré , 2012 ] . Unfortunately , analyzing this phenomenon has proven to be notoriously difficult . This is because without-replacement sampling creates statistical dependencies between the iterations , so the stochastic gradients computed at each iteration can no longer be seen as unbiased estimates of gradients of F ( · ) . Nevertheless , in the past few years our understanding of this problem has significantly improved . For concreteness , let us focus on a classical setting where each fi ( · ) is a convex quadratic , and F ( · ) is strongly convex . Suppose that SGD is allowed to perform k epochs , each of which consists of passing over the n individual functions in a random order . Assuming the step size is appropriately chosen , it has been established that SGD with single-shuffling returns a point whose expected optimization error is on the order of 1/ ( nk2 ) . For random reshuffling , this further improves to 1/ ( nk ) 2 +1/ ( nk3 ) ( see further discussion in the related work section below ) . In contrast , the expected optimization error of with-replacement SGD after the same overall number of iterations ( nk ) is well-known to be on the order of 1/ ( nk ) ( e.g. , Nemirovski et al . [ 2009 ] , Rakhlin et al . [ 2012 ] ) . Moreover , these bounds are known to be unimprovable in general , due to the existence of nearly matching lower bounds [ Safran and Shamir , 2020 , Rajput et al. , 2020 ] . Thus , as the number of epochs k increases , without-replacement SGD seems to provably beat with-replacement SGD . Despite these encouraging results , it is important to note that the bounds as stated above quantify only the dependence on n , k , and ignore dependencies on other problem parameters . In particular , it is well-known that the convergence of SGD is highly sensitive to the problem ’ s strong convexity and smoothness parameters , their ratio ( a.k.a . the condition number ) , as well as the magnitude of the gradients . Unfortunately , existing lower bounds ignore some of these parameters ( treating them as constants ) , while in known upper bounds these can lead to vacuous results in realistic regimes . To give a concrete example , let us return to the case where each fi ( · ) is a convex quadratic function , and assume that F ( · ) is λ-strongly convex for some λ > 0 . In Table 1 , we present existing upper bounds for this case , as a function of both n , k as well as λ ( fixing other problem parameters , such as the smoothness parameter , as constants ) .1 The important thing to note about these bounds is that they are valid and improve over the bound for with-replacement SGD only when the number of epochs k is at least n or 1/λ ( or even more ) . Unfortunately , such a regime is problematic for two reasons : First , large-scale high-dimensional problems are often ill-conditioned , with λ being very small and n being very large , so requiring that many passes k over all functions can easily be prohibitively expensive . Second , if we allow k > 1λ , there exist much better and simpler methods than SGD : Indeed , we can simply run deterministic gradient descent for k iterations ( computing the gradient of F ( · ) at each iteration by computing and averaging the gradients of fi ( · ) in any order we please ) . Since the optimization error of gradient descent ( as a function of k , λ ) scales as exp ( −λk ) [ Nesterov , 2018 ] , we compute a nearly exact optimum ( up to error ϵ ) as long as k ≳ 1λ log ( 1 ϵ ) . Moreover , slightly more sophisticated methods such as accelerated gradient descent or the conjugate gradient method enjoy an error bound scaling as exp ( −k √ λ ) , so in fact , we can compute an ϵ-optimal point already when k ≳ 1√ λ log ( 1ϵ ) . Thus , the power of SGD is mostly when k is relatively small , and definitely smaller than quantities such as 1/λ . However , the upper bounds discussed earlier do not imply any advantage of withoutreplacement sampling in this regime . Of course , these are only upper bounds , which might possibly be loose . Thus , it is not clear if this issue is simply an artifact of the existing analyses , or a true issue of without-replacement sampling methods . In this paper , we rigorously study this question , with the following ( perhaps surprising ) conclusion : At least in the worst-case , without-replacement schemes do not significantly improve over withreplacement sampling , unless the number of epochs k is larger than the problem ’ s condition number ( which suitably defined , scales as 1/λ ) . As discussed above , this implies that the expected benefit of without-replacement schemes may not be manifest for realistic iteration budgets . In more detail , our contributions are as follows : • We prove that there is a simple quadratic function F ( · ) , which is λ-strongly convex and λmax-smooth , such that the expected optimization error of SGD with single shuffling ( using any fixed step size ) is at least Ω ( 1 λnk ·min { 1 , λmax/λ k } ) . ( See Thm . 1 . ) Comparing this to the Θ ( 1/ ( λnk ) ) bound for with-replacement SGD , we see that we can not possibly get a significant improvement unless k is larger than the condition number term λmaxλ . • For SGD with random reshuffling , we prove a similar lower bound of Ω ( 1 λnk ·min { 1 , λmax/λ nk + λ2max/λ 2 k2 } ) . ( See Thm . 2 . ) As before , this improves on the Θ ( 1/ ( λnk ) ) bound of with-replacement SGD only when k > λmaxλ . • We provide matching upper bounds in all relevant parameters ( up to constants and log factors and assuming the input dimension is fixed – See Sec . 4 ) , which apply to the class of quadratic functions with commuting quadratic terms ( which include in particular the lower bound constructions above ) . • To illustrate our theoretical results , we perform a few simple experiments comparing withand without-replacement sampling schemes on our lower bound constructions ( see Sec . 5 ) . Our results accord with our theoretical findings , and show that if the number of epochs is not greater than the condition number , then without-replacement does not necessarily improve upon with-replacement sampling . Moreover , it is observed that for some of the values of k exceeding the condition number by as much as 50 % , with-replacement still provides comparable results to without-replacement , even when averaging their performance over many instantiations . 1Focusing only on λ is enough for the purpose of our discussion here , and moreover , the dependence on other problem parameters is not always explicitly given in the results of previous papers . We conclude with a discussion of the results and open questions in Sec . 6 . We note that our lower and upper bounds apply to SGD using a fixed step size , ηt = η for all t , and for the iterate reached after k epochs . Thus , they do not exclude the possibility that better upper bounds can be obtained with a variable step size strategy , or some iterate averaging scheme . However , we conjecture that it is not true , as existing upper bounds either do not make such assumptions or do not beat the lower bounds presented here . Moreover , SGD with variable step sizes can generally be matched by SGD employing a fixed optimal step size ( dependent on the problem parameters and the overall number of iterations ) . Related Work Proving that without-replacement SGD converges faster than with-replacement SGD has been the focus of a line of recent works , which we briefly survey below ( focusing for concreteness on quadratic and strongly convex problems , as we do in this paper ) . Gürbüzbalaban et al . [ 2015a , b ] proved that without-replacement beats with-replacement in terms of the dependence on k ( 1/k2 vs. 1/k , without specifying a decay in terms of n ) . Shamir [ 2016 ] proved that without-replacement is not worse than with-replacement in terms of dependence on n , but only for k = 1 . HaoChen and Sra [ 2018 ] managed to prove a better bound for random reshuffling , scaling as 1/ ( nk ) 2 + 1/k3 . Safran and Shamir [ 2020 ] proved a lower bound of 1/ ( nk ) 2 + 1/nk3 for random reshuffling and 1/ ( nk2 ) for single shuffling , and also proved matching upper bounds in the ( very ) special case of one-dimensional quadratic functions . A series of recent works [ Jain et al. , 2019b , Rajput et al. , 2020 , Ahn et al. , 2020 , Mishchenko et al. , 2020 , Nguyen et al. , 2020 ] showed that these are indeed the optimal bounds in terms of n , k , by proving matching upper bounds which apply to general quadratic functions and beyond . Rajput et al . [ 2020 ] were also able to show that for strongly convex and smooth functions beyond quadratics , the error rate provably becomes order of 1/ ( nk2 ) for both single shuffling and random reshuffling . A related and ongoing line of work considers the question of whether without-replacement SGD can be shown to be superior to with-replacement SGD , individually on any given problem of a certain type [ Recht and Ré , 2012 , Lai and Lim , 2020 , De Sa , 2020 , Yun et al. , 2021 ] . However , this is different than comparing worst-case behavior over problem classes , which is our focus here . Without-replacement SGD was also studied under somewhat different settings than ours , such as Ying et al . [ 2018 ] , Tran et al . [ 2020 ] , Huang et al . [ 2021 ] . In our paper , we focus on the well-known and widely popular SGD algorithm , using various sampling schemes . However , we note that for finite-sum problems , different and sometimes better convergence guarantees can be obtained using other stochastic algorithms , such as variance-reduced methods , adaptive gradient schemes , or by incorporating momentum . Analyzing the effects of withoutreplacement sampling on such methods is an interesting topic ( studied for example in Tran et al . [ 2020 ] ) , which is however outside the scope of our paper .

Summary: This paper studies lower bounds for without-replacement fixed-step-size SGD  (both single shuffling and random reshuffling) for the finite-sum optimization problem with an attention to dependence on the condition number of the problem. Their primary result is to prove a lower bound on the optimization error (realized by a convex quadratic construction) that suggests that unless the number of epochs $k$ is significantly larger than the condition number, neither single shuffling nor random reshuffling variants of without-replacement SGD outperform with-replacement SGD (previously, upper bounds for the with-replacement variants of SGD were considered to beat without-replacement SGD since often results did not take the condition number into account in the bounds). Moreover, they show their lower bound is tight with respect to a class of quadratics which includes the lower bound instance, suggesting that the analysis is somewhat tight.   The restriction on $k$ being large in order for without-replacement SGD to be more efficient is problematic for practical justification for using without-replacement SGD, since if $k$ is large enough, at some point one will prefer to use methods like conjugate gradient instead of using SGD, thus making the regime in which without-replacement experiences gains irrelevant to practice (since there is a better algorithm).