An Improved Analysis and Rates for Variance Reduction under Without-replacement Sampling Orders 

1 Introduction . We study the finite-sum composite optimization problem min x∈Rd F ( x ) + r ( x ) and F ( x ) = 1 n n∑ i=1 fi ( x ) . ( 1 ) where each fi ( x ) is differentiable and convex , and the regularization function r ( x ) is convex but not necessarily differentiable . This formulation arises in many problems in machine learning [ 34 , 39 , 14 ] , distributed optimization [ 20 , 3 , 19 ] , and signal processing [ 4 , 9 ] . The leading methods to solve ( 1 ) are first-order algorithms such as stochastic gradient descent ( SGD ) [ 28 , 2 ] and stochastic variance-reduced methods [ 14 , 6 , 7 , 17 , 10 , 32 ] . In the implementation of ∗Equal Contribution . Correspondence to : Kun Yuan 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these methods , each fi ( x ) can be sampled either with or without replacement . Without-replacement sampling draws each fi ( x ) exactly once during an epoch , which is numerically faster than withreplacement sampling and more cache-friendly ; see the experiments in [ 1 , 38 , 11 , 7 , 37 , 5 ] . This has triggered significant interests in understanding the theory behind without-replacement sampling . Among the most popular without-replacement approaches are cyclic sampling , random reshuffling , and shuffling-once . Cyclic sampling draws the samples in a cyclic order . Random reshuffling reorders the samples at the beginning of each sample epoch . The third approach , however , shuffles data only once before the training begins . Without-replacement sampling have been extensively studied for SGD . It was established in [ 1 , 38 , 11 , 22 , 24 ] that without-replacement sampling enables SGD with faster convergence For example , it was proved that without-replacement sampling can speed up uniform-iid-sampling SGD from Õ ( 1/k ) to Õ ( 1/k2 ) ( where k is the iteration ) for strongly-convex costs in [ 11 , 12 ] , and O ( 1/k1/2 ) to O ( 1/k ) for the convex costs in [ 24 , 22 ] . [ 31 ] establishes a tight lower bound for random reshuffling SGD . Recent works [ 27 , 22 ] close the gap between upper and lower bounds . Authors of [ 22 ] also analyzes without-replacement SGD with non-convex costs . In contrast to the mature results in SGD , variance-reduction under without-replacement sampling are less understood . Variance reduction strategies construct stochastic gradient estimators with vanishing gradient variance , which allows for much larger learning rate and hence speed up training process . Variance reduction under without-replacement sampling is difficult to analyze . In the strongly convex scenario , [ 37 , 33 ] provide linear convergence guarantees for SVRG/SAGA with random reshuffling , but the rates are worse than full-batch gradient descent ( GD ) . Authors of [ 35 , 23 ] improved the rate so that it can match with GD . In convex scenario , existing rates for without-replacement sampling with variance reduction , except for the rate established in an independent and concurrent work [ 18 ] , are still far worse than GD [ 33 , 5 ] , see Table 1 . Furthermore , no existing rates for variance reduction under without-replacement sampling orders , in either convex or strongly convex scenarios , can match those under uniform-iid-sampling which are essentially sample-size independent . There is a clear gap between the known convergence rates and superior practical performance for without-replacement sampling with variance reduction . 1.1 Main results . This paper narrows such gap by providing convergence analysis and rates for proximal DFinito , a proximal damped variant of Finito/MISO [ 7 , 17 , 26 ] , which is a well-known variance reduction algorithm , under without-replacement sampling orders . Our main achieved results are : • We develop a proximal damped variant of Finito/MISO called Prox-DFinito and establish its gradient complexities with random reshuffling , cyclic sampling , and shuffling-once , under both convex and strongly convex scenarios . All these rates match with gradient descent , and are state-of-the-art ( up to logarithm factors ) compared to existing results for without-replacement sampling with variance-reduction , see Table 1 . • Our novel analysis can gauge how a cyclic order will influence the rate of Prox-DFinito with cyclic sampling . This allows us to identify the optimal cyclic sampling ordering . ProxDFinito with optimal cyclic sampling , in the highly data-heterogeneous scenario , can attain a sample-size-independent convergence rate , which is the first result , to our knowledge , that can match with uniform-iid-sampling with variance reduction in certain scenarios . We also propose a numerical method to discover the optimal cyclic ordering cheaply . 1.2 Other related works . Our analysis on cyclic sampling is novel . Most existing analyses unify random reshuffling and cyclic sampling into the same framework ; see the SGD analysis in [ 11 ] , the variance-reduction analysis in [ 10 , 36 , 23 , 37 ] , and the coordinate-update analysis in [ 5 ] . These analyses are primarily based on the “ sampled-once-per-epoch ” property and do not analyze the orders within each epoch , so they do not distinguish cyclic sampling from random reshuffling in analysis . [ 16 ] finds that random reshuffling SGD is basically the average over all cyclic sampling trials . This implies cyclic sampling can outperform random reshuffling with a well-designed sampling order . However , [ 16 ] does not discuss how much better cyclic sampling can outperform random reshuffling and how to achieve such cyclic order . Different from existing literatures , our analysis introduces an order-specific norm to gauge how cyclic sampling performs with different fixed orders . With such norm , we are able to clarify the worst-case and best-case performance of variance reduction with cyclic sampling . Simultaneously and independently , a recent work [ 18 ] also provided an improved rates for variance reduction under without-replacement sampling orders that can match with gradient descent . However , [ 18 ] does not discuss whether and when variance reduction with replacement sampling can match with uniform sampling . In addition , [ 18 ] studies SVRG while this paper studies Finito/MISO . The convergence analyses in these two works are very different . The detailed comparison between this work and [ 18 ] can be referred to Sec . 3.3 . 1.3 Notations . Throughout the paper we let col { x1 , · · · , xn } denote a column vector formed by stacking x1 , · · · , xn . We let [ n ] : = { 1 , · · · , n } and define the proximal operator as proxαr ( x ) : = arg min y∈Rd { α r ( y ) + 1 2 ‖y − x‖2 } ( 2 ) which is single-valued when r is convex , closed and proper . In general , we say A is an operator and write A : X → Y if A maps each point in space X to another space Y . So A ( x ) ∈ Y for all x ∈ X . For simplicity , we write Ax = A ( x ) and A ◦ Bx = A ( B ( x ) ) for operator composition . Cyclic sampling . We define π : = ( π ( 1 ) , π ( 2 ) , . . . , π ( n ) ) as an arbitrary determined permutation of sample indexes . The order π is fixed throughout the entire learning process under cyclic sampling . Random reshuffling . When starting each epoch , a random permutation τ : = ( τ ( 1 ) , τ ( 2 ) , ... , τ ( n ) ) is generated to specify the order to take samples . Let τk denote the permutation of the k-th epoch . Algorithm 1 Prox-DFinito Input : z̄0 = 1n n∑ i=1 z0i , step-size α , and θ ∈ ( 0 , 1 ) ; for epoch k = 0 , 1 , 2 , · · · do for iteration t = kn+ 1 , kn+ 2 , · · · , ( k + 1 ) n do xt−1 = proxαr ( z̄ t−1 ) ; Pick it with some rule ; Update ztit and z̄ t according to ( 4a ) and ( 5 ) ; end for z ( k+1 ) n i ← ( 1− θ ) zkni + θz ( k+1 ) n i for any i ∈ [ n ] ; . a damping step z̄ ( k+1 ) n ← ( 1− θ ) z̄kn + θz̄ ( k+1 ) n ; . a damping step end for 2 Proximal Finito/MISO with Damping . The proximal gradient method to solve problem ( 1 ) is zti = x t−1 − α∇fi ( xt−1 ) , ∀ i ∈ [ n ] ( 3a ) xt = proxαr ( 1 n n∑ i=1 zti ) ( 3b ) To avoid the global average that passes over all samples , we propose to update one zi per iteration : zti = { xt−1 − α∇fi ( xt−1 ) , i = it zt−1i , i 6= it ( 4a ) xt = proxαr ( 1 n n∑ i=1 zti ) . ( 4b ) When it is invoked with uniform-iid-sampling and r ( x ) = 0 , algorithm ( 4a ) – ( 4b ) reduces to Finito/MISO [ 7 , 17 ] . When it is invoked with cyclic sampling and r ( x ) = 0 , algorithm ( 4a ) – ( 4b ) reduces to DIAG [ 23 ] and WPG [ 19 ] . We let z̄t : = 1n ∑n i=1 z t i . The update ( 4a ) yields z̄t = z̄t−1 + ( ztit − z t−1 it ) /n . ( 5 ) This update can be finished with O ( d ) operations if { zti } ni=1 are stored with O ( nd ) memory . Furthermore , to increase robustness and simplify the convergence analysis , we impose a damping step to zi and z̄ when each epoch finishes . The proximal damped Finito/MISO method is listed in Algorithm 1 . Note that the damping step does not incur additional memory requirements . A more practical implementation of Algorithm 1 is referred to Algorithm 3 in Appendix A . 2.1 Fixed-point recursion reformulation . Algorithm ( 4a ) – ( 4b ) can be reformulated into a fixed-point recursion in { zi } ni=1 . Such a fixed-point recursion will be utilized throughout the paper . To proceed , we define z = col { z1 , · · · , zn } ∈ Rnd and introduce the average operator A : Rnd → Rd as Az = 1n ∑n i=1 zi . We further define the i-th block coordinate operator Ti : Rnd → Rnd as Tiz=col { z1 , · · · , ( I−α∇fi ) ◦ proxαr ( Az ) , · · · , zn } where I denotes the identity mapping . When applying Ti , it is noted that the i-th block coordinate in z is updated while the others remain unchanged . Proposition 1 . Prox-DFinito with fixed cyclic sampling order π is equivalent to the following fixed-point recursion ( see proof in Appendix B.1 . ) z ( k+1 ) n = ( 1− θ ) zkn + θTπzkn ( 6 ) where Tπ = Tπ ( n ) ◦ · · · ◦ Tπ ( 1 ) . Furthermore , variable xt can be recovered by xt = proxαr ◦ Azt , t = 0 , 1 , 2 , · · · ( 7 ) Similar result also hold for random reshuffling scenario . Proposition 2 . Prox-DFinito with random reshuffling is equivalent to z ( k+1 ) n = ( 1− θ ) zkn + θTτkzkn ( 8 ) where Tτk = Tτk ( n ) ◦ · · · ◦ Tτk ( 1 ) . Furthermore , variable xt can be recovered by following ( 7 ) .

Summary: The paper proposes a new method called Prox-DFinito based on the proximal Finito with without-replacement sampling. The authors derive complexity bounds for the proposed method in convex (for making the squared norm of the gradient small) and strongly convex cases (for making the squared distance to the solution small) that match under some additional assumptions the rate of Gradient Descent. Moreover, under additional assumptions on the objective function, the authors show sample-size independent bound in the convex case. The proofs are non-standard but clean and easy to follow. However, the paper has several strong weaknesses.