An Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders 

1 Introduction . We study the finite-sum composite optimization problem min x∈Rd F ( x ) + r ( x ) and F ( x ) = 1 n n∑ i=1 fi ( x ) . ( 1 ) where each fi ( x ) is differentiable and convex , and the regularization function r ( x ) is convex but not necessarily differentiable . This formulation arises in many problems in machine learning [ 34 , 39 , 14 ] , distributed optimization [ 20 , 3 , 19 ] , and signal processing [ 4 , 9 ] . The leading methods to solve ( 1 ) are first-order algorithms such as stochastic gradient descent ( SGD ) [ 28 , 2 ] and stochastic variance-reduced methods [ 14 , 6 , 7 , 17 , 10 , 32 ] . In the implementation of ∗Equal Contribution . Correspondence to : Kun Yuan 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these methods , each fi ( x ) can be sampled either with or without replacement . Without-replacement sampling draws each fi ( x ) exactly once during an epoch , which is numerically faster than withreplacement sampling and more cache-friendly ; see the experiments in [ 1 , 38 , 11 , 7 , 37 , 5 ] . This has triggered significant interests in understanding the theory behind without-replacement sampling . Among the most popular without-replacement approaches are cyclic sampling , random reshuffling , and shuffling-once . Cyclic sampling draws the samples in a cyclic order . Random reshuffling reorders the samples at the beginning of each sample epoch . The third approach , however , shuffles data only once before the training begins . Without-replacement sampling have been extensively studied for SGD . It was established in [ 1 , 38 , 11 , 22 , 24 ] that without-replacement sampling enables SGD with faster convergence For example , it was proved that without-replacement sampling can speed up uniform-iid-sampling SGD from Õ ( 1/k ) to Õ ( 1/k2 ) ( where k is the iteration ) for strongly-convex costs in [ 11 , 12 ] , and O ( 1/k1/2 ) to O ( 1/k ) for the convex costs in [ 24 , 22 ] . [ 31 ] establishes a tight lower bound for random reshuffling SGD . Recent works [ 27 , 22 ] close the gap between upper and lower bounds . Authors of [ 22 ] also analyzes without-replacement SGD with non-convex costs . In contrast to the mature results in SGD , variance-reduction under without-replacement sampling are less understood . Variance reduction strategies construct stochastic gradient estimators with vanishing gradient variance , which allows for much larger learning rate and hence speed up training process . Variance reduction under without-replacement sampling is difficult to analyze . In the strongly convex scenario , [ 37 , 33 ] provide linear convergence guarantees for SVRG/SAGA with random reshuffling , but the rates are worse than full-batch gradient descent ( GD ) . Authors of [ 35 , 23 ] improved the rate so that it can match with GD . In convex scenario , existing rates for without-replacement sampling with variance reduction , except for the rate established in an independent and concurrent work [ 18 ] , are still far worse than GD [ 33 , 5 ] , see Table 1 . Furthermore , no existing rates for variance reduction under without-replacement sampling orders , in either convex or strongly convex scenarios , can match those under uniform-iid-sampling which are essentially sample-size independent . There is a clear gap between the known convergence rates and superior practical performance for without-replacement sampling with variance reduction . 1.1 Main results . This paper narrows such gap by providing convergence analysis and rates for proximal DFinito , a proximal damped variant of Finito/MISO [ 7 , 17 , 26 ] , which is a well-known variance reduction algorithm , under without-replacement sampling orders . Our main achieved results are : • We develop a proximal damped variant of Finito/MISO called Prox-DFinito and establish its gradient complexities with random reshuffling , cyclic sampling , and shuffling-once , under both convex and strongly convex scenarios . All these rates match with gradient descent , and are state-of-the-art ( up to logarithm factors ) compared to existing results for without-replacement sampling with variance-reduction , see Table 1 . • Our novel analysis can gauge how a cyclic order will influence the rate of Prox-DFinito with cyclic sampling . This allows us to identify the optimal cyclic sampling ordering . ProxDFinito with optimal cyclic sampling , in the highly data-heterogeneous scenario , can attain a sample-size-independent convergence rate , which is the first result , to our knowledge , that can match with uniform-iid-sampling with variance reduction in certain scenarios . We also propose a numerical method to discover the optimal cyclic ordering cheaply . 1.2 Other related works . Our analysis on cyclic sampling is novel . Most existing analyses unify random reshuffling and cyclic sampling into the same framework ; see the SGD analysis in [ 11 ] , the variance-reduction analysis in [ 10 , 36 , 23 , 37 ] , and the coordinate-update analysis in [ 5 ] . These analyses are primarily based on the “ sampled-once-per-epoch ” property and do not analyze the orders within each epoch , so they do not distinguish cyclic sampling from random reshuffling in analysis . [ 16 ] finds that random reshuffling SGD is basically the average over all cyclic sampling trials . This implies cyclic sampling can outperform random reshuffling with a well-designed sampling order . However , [ 16 ] does not discuss how much better cyclic sampling can outperform random reshuffling and how to achieve such cyclic order . Different from existing literatures , our analysis introduces an order-specific norm to gauge how cyclic sampling performs with different fixed orders . With such norm , we are able to clarify the worst-case and best-case performance of variance reduction with cyclic sampling . Simultaneously and independently , a recent work [ 18 ] also provided an improved rates for variance reduction under without-replacement sampling orders that can match with gradient descent . However , [ 18 ] does not discuss whether and when variance reduction with replacement sampling can match with uniform sampling . In addition , [ 18 ] studies SVRG while this paper studies Finito/MISO . The convergence analyses in these two works are very different . The detailed comparison between this work and [ 18 ] can be referred to Sec . 3.3 . 1.3 Notations . Throughout the paper we let col { x1 , · · · , xn } denote a column vector formed by stacking x1 , · · · , xn . We let [ n ] : = { 1 , · · · , n } and define the proximal operator as proxαr ( x ) : = arg min y∈Rd { α r ( y ) + 1 2 ‖y − x‖2 } ( 2 ) which is single-valued when r is convex , closed and proper . In general , we say A is an operator and write A : X → Y if A maps each point in space X to another space Y . So A ( x ) ∈ Y for all x ∈ X . For simplicity , we write Ax = A ( x ) and A ◦ Bx = A ( B ( x ) ) for operator composition . Cyclic sampling . We define π : = ( π ( 1 ) , π ( 2 ) , . . . , π ( n ) ) as an arbitrary determined permutation of sample indexes . The order π is fixed throughout the entire learning process under cyclic sampling . Random reshuffling . When starting each epoch , a random permutation τ : = ( τ ( 1 ) , τ ( 2 ) , ... , τ ( n ) ) is generated to specify the order to take samples . Let τk denote the permutation of the k-th epoch . Algorithm 1 Prox-DFinito Input : z̄0 = 1n n∑ i=1 z0i , step-size α , and θ ∈ ( 0 , 1 ) ; for epoch k = 0 , 1 , 2 , · · · do for iteration t = kn+ 1 , kn+ 2 , · · · , ( k + 1 ) n do xt−1 = proxαr ( z̄ t−1 ) ; Pick it with some rule ; Update ztit and z̄ t according to ( 4a ) and ( 5 ) ; end for z ( k+1 ) n i ← ( 1− θ ) zkni + θz ( k+1 ) n i for any i ∈ [ n ] ; . a damping step z̄ ( k+1 ) n ← ( 1− θ ) z̄kn + θz̄ ( k+1 ) n ; . a damping step end for 2 Proximal Finito/MISO with Damping . The proximal gradient method to solve problem ( 1 ) is zti = x t−1 − α∇fi ( xt−1 ) , ∀ i ∈ [ n ] ( 3a ) xt = proxαr ( 1 n n∑ i=1 zti ) ( 3b ) To avoid the global average that passes over all samples , we propose to update one zi per iteration : zti = { xt−1 − α∇fi ( xt−1 ) , i = it zt−1i , i 6= it ( 4a ) xt = proxαr ( 1 n n∑ i=1 zti ) . ( 4b ) When it is invoked with uniform-iid-sampling and r ( x ) = 0 , algorithm ( 4a ) – ( 4b ) reduces to Finito/MISO [ 7 , 17 ] . When it is invoked with cyclic sampling and r ( x ) = 0 , algorithm ( 4a ) – ( 4b ) reduces to DIAG [ 23 ] and WPG [ 19 ] . We let z̄t : = 1n ∑n i=1 z t i . The update ( 4a ) yields z̄t = z̄t−1 + ( ztit − z t−1 it ) /n . ( 5 ) This update can be finished with O ( d ) operations if { zti } ni=1 are stored with O ( nd ) memory . Furthermore , to increase robustness and simplify the convergence analysis , we impose a damping step to zi and z̄ when each epoch finishes . The proximal damped Finito/MISO method is listed in Algorithm 1 . Note that the damping step does not incur additional memory requirements . A more practical implementation of Algorithm 1 is referred to Algorithm 3 in Appendix A . 2.1 Fixed-point recursion reformulation . Algorithm ( 4a ) – ( 4b ) can be reformulated into a fixed-point recursion in { zi } ni=1 . Such a fixed-point recursion will be utilized throughout the paper . To proceed , we define z = col { z1 , · · · , zn } ∈ Rnd and introduce the average operator A : Rnd → Rd as Az = 1n ∑n i=1 zi . We further define the i-th block coordinate operator Ti : Rnd → Rnd as Tiz=col { z1 , · · · , ( I−α∇fi ) ◦ proxαr ( Az ) , · · · , zn } where I denotes the identity mapping . When applying Ti , it is noted that the i-th block coordinate in z is updated while the others remain unchanged . Proposition 1 . Prox-DFinito with fixed cyclic sampling order π is equivalent to the following fixed-point recursion ( see proof in Appendix B.1 . ) z ( k+1 ) n = ( 1− θ ) zkn + θTπzkn ( 6 ) where Tπ = Tπ ( n ) ◦ · · · ◦ Tπ ( 1 ) . Furthermore , variable xt can be recovered by xt = proxαr ◦ Azt , t = 0 , 1 , 2 , · · · ( 7 ) Similar result also hold for random reshuffling scenario . Proposition 2 . Prox-DFinito with random reshuffling is equivalent to z ( k+1 ) n = ( 1− θ ) zkn + θTτkzkn ( 8 ) where Tτk = Tτk ( n ) ◦ · · · ◦ Tτk ( 1 ) . Furthermore , variable xt can be recovered by following ( 7 ) .

Summary: This paper develops a proximal damped version of the Finito algorithm. The algorithm is proved to achieve the same convergence rate as proximal GD, for cyclic sampling, random reshuffling and shuffling-once versions of the algorithm. Further, the authors claim that this is the first* shuffling based variance reduction algorithm to achieve the convergence rate. The paper also gives a new norm that captures the optimality of sampling orders and provides a heuristic based on that for importance based reshuffling.  Besides the theoretical results, the empirical results seem to suggest that the proposed algorithm is indeed faster than other variance reduction algorithms.   *: The authors cite a concurrent work (Malinovsky et al.) that also achieves the same convergence rates for general convex functions, but the algorithms in the two papers are different.