Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problems

1 Introduction . We consider solving finite-sum optimization problems of the form F ( x ) = 1 n n∑ i=1 fi ( x ) using stochastic gradient descent ( SGD ) . Such problems are extremely common in modern machine learning ( e.g. , for empirical risk minimization ) , and stochastic gradient methods are the most popular approach for large-scale problems , where both n and the dimension are large . The classical approach to apply SGD to such problems is to repeatedly sample indices i ∈ { 1 , . . . , n } uniformly at random , and perform updates of the form xt+1 = xt − ηt∇fi ( xt ) , where ηt is a step-size parameter . With this sampling scheme , each ∇fi ( xt ) is a random unbiased estimate of ∇F ( xt ) ( conditioned on xt ) . Thus , the algorithm can be seen as a “ cheap ” noisy version of plain gradient descent on F ( · ) , where each iteration requires computing the gradient of just a single function fi ( · ) , rather than the gradient of F ( · ) ( which would require computing and averaging n individual gradients ) . This key observation facilitates the analysis of SGD , while simultaneously explaining why SGD is much more efficient than gradient descent on large-scale problems . However , the practice of SGD differs somewhat from this idealized description : In practice , it is much more common to perform without-replacement sampling of the indices , by randomly shuffling the indices { 1 , . . . , n } and processing the individual functions in that order – that is , choosing a random permutation σ on { 1 , . . . , n } , and performing updates of the form xt+1 = xt − ηt∇fσ ( t ) ( xt ) . After a full pass over the n functions , further passes are made either with the same permutation σ ( known as single shuffling ) , or with a new random permutation σ chosen before each pass ( known as random 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reshuffling ) . Such without-replacement schemes are not only more convenient to implement in many cases , they often also exhibit faster error decay than with-replacement SGD [ Bottou , 2009 , Recht and Ré , 2012 ] . Unfortunately , analyzing this phenomenon has proven to be notoriously difficult . This is because without-replacement sampling creates statistical dependencies between the iterations , so the stochastic gradients computed at each iteration can no longer be seen as unbiased estimates of gradients of F ( · ) . Nevertheless , in the past few years our understanding of this problem has significantly improved . For concreteness , let us focus on a classical setting where each fi ( · ) is a convex quadratic , and F ( · ) is strongly convex . Suppose that SGD is allowed to perform k epochs , each of which consists of passing over the n individual functions in a random order . Assuming the step size is appropriately chosen , it has been established that SGD with single-shuffling returns a point whose expected optimization error is on the order of 1/ ( nk2 ) . For random reshuffling , this further improves to 1/ ( nk ) 2 +1/ ( nk3 ) ( see further discussion in the related work section below ) . In contrast , the expected optimization error of with-replacement SGD after the same overall number of iterations ( nk ) is well-known to be on the order of 1/ ( nk ) ( e.g. , Nemirovski et al . [ 2009 ] , Rakhlin et al . [ 2012 ] ) . Moreover , these bounds are known to be unimprovable in general , due to the existence of nearly matching lower bounds [ Safran and Shamir , 2020 , Rajput et al. , 2020 ] . Thus , as the number of epochs k increases , without-replacement SGD seems to provably beat with-replacement SGD . Despite these encouraging results , it is important to note that the bounds as stated above quantify only the dependence on n , k , and ignore dependencies on other problem parameters . In particular , it is well-known that the convergence of SGD is highly sensitive to the problem ’ s strong convexity and smoothness parameters , their ratio ( a.k.a . the condition number ) , as well as the magnitude of the gradients . Unfortunately , existing lower bounds ignore some of these parameters ( treating them as constants ) , while in known upper bounds these can lead to vacuous results in realistic regimes . To give a concrete example , let us return to the case where each fi ( · ) is a convex quadratic function , and assume that F ( · ) is λ-strongly convex for some λ > 0 . In Table 1 , we present existing upper bounds for this case , as a function of both n , k as well as λ ( fixing other problem parameters , such as the smoothness parameter , as constants ) .1 The important thing to note about these bounds is that they are valid and improve over the bound for with-replacement SGD only when the number of epochs k is at least n or 1/λ ( or even more ) . Unfortunately , such a regime is problematic for two reasons : First , large-scale high-dimensional problems are often ill-conditioned , with λ being very small and n being very large , so requiring that many passes k over all functions can easily be prohibitively expensive . Second , if we allow k > 1λ , there exist much better and simpler methods than SGD : Indeed , we can simply run deterministic gradient descent for k iterations ( computing the gradient of F ( · ) at each iteration by computing and averaging the gradients of fi ( · ) in any order we please ) . Since the optimization error of gradient descent ( as a function of k , λ ) scales as exp ( −λk ) [ Nesterov , 2018 ] , we compute a nearly exact optimum ( up to error ϵ ) as long as k ≳ 1λ log ( 1 ϵ ) . Moreover , slightly more sophisticated methods such as accelerated gradient descent or the conjugate gradient method enjoy an error bound scaling as exp ( −k √ λ ) , so in fact , we can compute an ϵ-optimal point already when k ≳ 1√ λ log ( 1ϵ ) . Thus , the power of SGD is mostly when k is relatively small , and definitely smaller than quantities such as 1/λ . However , the upper bounds discussed earlier do not imply any advantage of withoutreplacement sampling in this regime . Of course , these are only upper bounds , which might possibly be loose . Thus , it is not clear if this issue is simply an artifact of the existing analyses , or a true issue of without-replacement sampling methods . In this paper , we rigorously study this question , with the following ( perhaps surprising ) conclusion : At least in the worst-case , without-replacement schemes do not significantly improve over withreplacement sampling , unless the number of epochs k is larger than the problem ’ s condition number ( which suitably defined , scales as 1/λ ) . As discussed above , this implies that the expected benefit of without-replacement schemes may not be manifest for realistic iteration budgets . In more detail , our contributions are as follows : • We prove that there is a simple quadratic function F ( · ) , which is λ-strongly convex and λmax-smooth , such that the expected optimization error of SGD with single shuffling ( using any fixed step size ) is at least Ω ( 1 λnk ·min { 1 , λmax/λ k } ) . ( See Thm . 1 . ) Comparing this to the Θ ( 1/ ( λnk ) ) bound for with-replacement SGD , we see that we can not possibly get a significant improvement unless k is larger than the condition number term λmaxλ . • For SGD with random reshuffling , we prove a similar lower bound of Ω ( 1 λnk ·min { 1 , λmax/λ nk + λ2max/λ 2 k2 } ) . ( See Thm . 2 . ) As before , this improves on the Θ ( 1/ ( λnk ) ) bound of with-replacement SGD only when k > λmaxλ . • We provide matching upper bounds in all relevant parameters ( up to constants and log factors and assuming the input dimension is fixed – See Sec . 4 ) , which apply to the class of quadratic functions with commuting quadratic terms ( which include in particular the lower bound constructions above ) . • To illustrate our theoretical results , we perform a few simple experiments comparing withand without-replacement sampling schemes on our lower bound constructions ( see Sec . 5 ) . Our results accord with our theoretical findings , and show that if the number of epochs is not greater than the condition number , then without-replacement does not necessarily improve upon with-replacement sampling . Moreover , it is observed that for some of the values of k exceeding the condition number by as much as 50 % , with-replacement still provides comparable results to without-replacement , even when averaging their performance over many instantiations . 1Focusing only on λ is enough for the purpose of our discussion here , and moreover , the dependence on other problem parameters is not always explicitly given in the results of previous papers . We conclude with a discussion of the results and open questions in Sec . 6 . We note that our lower and upper bounds apply to SGD using a fixed step size , ηt = η for all t , and for the iterate reached after k epochs . Thus , they do not exclude the possibility that better upper bounds can be obtained with a variable step size strategy , or some iterate averaging scheme . However , we conjecture that it is not true , as existing upper bounds either do not make such assumptions or do not beat the lower bounds presented here . Moreover , SGD with variable step sizes can generally be matched by SGD employing a fixed optimal step size ( dependent on the problem parameters and the overall number of iterations ) . Related Work Proving that without-replacement SGD converges faster than with-replacement SGD has been the focus of a line of recent works , which we briefly survey below ( focusing for concreteness on quadratic and strongly convex problems , as we do in this paper ) . Gürbüzbalaban et al . [ 2015a , b ] proved that without-replacement beats with-replacement in terms of the dependence on k ( 1/k2 vs. 1/k , without specifying a decay in terms of n ) . Shamir [ 2016 ] proved that without-replacement is not worse than with-replacement in terms of dependence on n , but only for k = 1 . HaoChen and Sra [ 2018 ] managed to prove a better bound for random reshuffling , scaling as 1/ ( nk ) 2 + 1/k3 . Safran and Shamir [ 2020 ] proved a lower bound of 1/ ( nk ) 2 + 1/nk3 for random reshuffling and 1/ ( nk2 ) for single shuffling , and also proved matching upper bounds in the ( very ) special case of one-dimensional quadratic functions . A series of recent works [ Jain et al. , 2019b , Rajput et al. , 2020 , Ahn et al. , 2020 , Mishchenko et al. , 2020 , Nguyen et al. , 2020 ] showed that these are indeed the optimal bounds in terms of n , k , by proving matching upper bounds which apply to general quadratic functions and beyond . Rajput et al . [ 2020 ] were also able to show that for strongly convex and smooth functions beyond quadratics , the error rate provably becomes order of 1/ ( nk2 ) for both single shuffling and random reshuffling . A related and ongoing line of work considers the question of whether without-replacement SGD can be shown to be superior to with-replacement SGD , individually on any given problem of a certain type [ Recht and Ré , 2012 , Lai and Lim , 2020 , De Sa , 2020 , Yun et al. , 2021 ] . However , this is different than comparing worst-case behavior over problem classes , which is our focus here . Without-replacement SGD was also studied under somewhat different settings than ours , such as Ying et al . [ 2018 ] , Tran et al . [ 2020 ] , Huang et al . [ 2021 ] . In our paper , we focus on the well-known and widely popular SGD algorithm , using various sampling schemes . However , we note that for finite-sum problems , different and sometimes better convergence guarantees can be obtained using other stochastic algorithms , such as variance-reduced methods , adaptive gradient schemes , or by incorporating momentum . Analyzing the effects of withoutreplacement sampling on such methods is an interesting topic ( studied for example in Tran et al . [ 2020 ] ) , which is however outside the scope of our paper .

Summary: This paper compares the efficiency of using different variants of stochastic gradient descent to find the minimum of a function that is the sum of multiple convex functions. It focuses on the case where the minimum of the function is dramatically flatter in some directions than in others. More specifically, it compares using a random sample at each time step, going through the samples in a fixed cycle, and repeatedly going through the entire list of samples in a random order. The conclusion is that the more complicated versions do not significantly outperform regular stochastic gradient descent unless the number of timesteps is unreasonably large.