Title: Tighter Convergence Bounds for Shuffled SGD via Primal-Dual Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: primal-dual analysis

cyclic methods

shuffled sgd
Abstract: Stochastic gradient descent (SGD) is perhaps the most prevalent optimization method in modern machine learning. Contrary to the empirical practice of sampling from the datasets \emph{without replacement} and with (possible) reshuffling at each epoch, the theoretical counterpart of SGD usually relies on the assumption of \emph{sampling with replacement}. It is only very recently that SGD using sampling without replacement -- shuffled SGD -- has been analyzed with matching upper and lower bounds. However, we observe that those bounds are too pessimistic to explain often superior empirical performance of data permutations (sampling without replacement) over vanilla counterparts (sampling with replacement) on machine learning problems. Through fine-grained analysis in the lens of primal-dual cyclic coordinate methods and the introduction of novel smoothness parameters, we present several results for shuffled SGD on smooth and non-smooth convex losses, where our novel analysis framework provides tighter convergence bounds over all popular shuffling schemes (IG, SO, and RR). Notably, our new bounds predict faster convergence than existing bounds in the literature -- by up to a factor of $O(\sqrt{n})$, mirroring benefits from tighter convergence bounds using component smoothness parameters in randomized coordinate methods. Lastly, we numerically demonstrate on common machine learning datasets that our bounds are indeed much tighter, thus offering a bridge between theory and practice.

Tighter Convergence Bounds for Shuffled SGD via Primal-Dual Perspective

Abstract

Stochastic gradient descent (SGD) is perhaps the most prevalent optimization method in modern machine learning. Contrary to the empirical practice of sampling from the datasets without replacement and with (possible) reshuffling at each epoch, the theoretical counterpart of SGD usually relies on the assumption of sampling with replacement . It is only very recently that SGD using sampling without replacement - shuffled SGD - has been analyzed with matching upper and lower bounds. However, we observe that those bounds are too pessimistic to explain often superior empirical performance of data permutations (sampling without replacement) over vanilla counterparts (sampling with replacement) on machine learning problems. Through fine-grained analysis in the lens of primal-dual cyclic coordinate methods and the introduction of novel smoothness parameters, we present several results for shuffled SGD on smooth and non-smooth convex losses, where our novel analysis framework provides tighter convergence bounds over all popular shuffling schemes (IG, SO, and RR). Notably, our new bounds predict faster convergence than existing bounds in the literature - by up to a factor of O ( √ n ) , mirroring benefits from tighter convergence bounds using component smoothness parameters in randomized coordinate methods. Lastly, we numerically demonstrate on common machine learning datasets that our bounds are indeed much tighter, thus offering a bridge between theory and practice.

1 Introduction

Originally proposed in [38], SGD has been broadly studied in the machine learning literature due to its effectiveness in large-scale settings, where full gradient computations are often computationally prohibitive. When applied to unconstrained finite-sum problems

SGD performs the update x t = x t -1 -η ∇ f i t ( x t -1 ) for i t ∈ [ n ] ( [ n ] := { 1 , . . . , n } ), in each iteration t . Traditional theoretical analysis for SGD builds upon the assumption of sampling i t ∈ [ n ] with replacement according to a fixed distribution p = ( p 1 , . . . , p n ) ⊤ over [ n ] , which leads to

OurTheory

+ExistingBound

OurTheory

+ExistingBound

10~0.6

+OurTheory

ExistingBound

10~0.5

f(x*)

10~0.5

*

f(x

10~0.9

I

×

10~1.0

10~1.2

10~1.0

10~1.5

10~1.5

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Datapasses

Datapasses

Datapasses

E i t [ ∇ f i t ( x t -1 ) / ( np i t )] = ∇ f ( x t -1 ) , and thus much of the (deterministic) gradient descent-style analysis can be transferred to this setting. By contrast, no such connection between the component and the full gradient can be established for shuffled SGD - which employs sampling without replacement -making its analysis much more challenging. As a result, despite its fundamental nature, there were no non-asymptotic convergence results for shuffled SGD until a very recent line of work [2, 12, 20, 21, 30, 31, 34, 35, 42]. All existing results consider general finite sum problems, with the same regularity condition constant (Lipschitz constant of f i or its gradient) assumed for all the component functions. As a result, the obtained convergence bounds are typically no better than for (full) gradient descent, and are only better than the bounds for SGD with replacement sampling if the algorithm is run for many full passes over the data [30, 34].

)

Furthermore, there is a large gap between the empirical performance of shuffled SGD and the predicted convergence rates from prior work [20, 30]. One cause for this discrepancy are overly pessimistic bounds on the step size in prior work, which are of order 1 / ( nL max ) , where L max is the maximum smoothness constant over components f i in (P). In practice, the step sizes are tuned to achieve better convergence bounds than predicted by the current theory. We illustrate how restrictions on the step size affect convergence of shuffled SGD (with random permutations in each epoch) in Fig. 1, where we plot the resulting optimality gap over full data passes when shuffled SGD is applied to logistic regression problems on standard datasets. To compare the effect of the step size η from prior work and our work, we choose take η = 1 / ( √ 2 nL max ) based on [30], and η = 1 / ( n √ ˆ L ˜ L from our work, where ˆ L, ˜ L are our novel fine-grained, data-dependent smoothness parameters defined in Section 3 for smooth convex finite-sum problems with linear predictors. As can be observed from Fig. 1, larger step sizes resulting from our theory lead to faster convergence of shuffled SGD and, as a result, our convergence bounds better predict the performance of shuffled SGD.

Building on these insights, we introduce a fined-grained theoretical analysis to transparently show how the structure of the data and the possibly different Lipschitz constants of the component functions or their gradients affect the performance of shuffled SGD, thus providing a better explanation of the heuristic success of shuffled SGD in modern machine learning.

1.1 Background and related work

SGD (with replacement) has been extensively studied in many settings (see e.g., [1, 9, 10, 38] for convex optimization). Compared to SGD, shuffled SGD usually exhibits faster convergence in practice [8, 37], and is easier and more efficient to implement [5]. For each epoch k , shuffled SGDstyle algorithms perform incremental gradient updates based on the sample ordering (permutation of the data points) denoted by π ( k ) . There are three main choices of data permutations: (i) π ( k ) ≡ π for some fixed permutation of [ n ] for all epochs, where shuffled SGD reduces to the incremental gradient (IG) method; (ii) π ( k ) ≡ ˜ π where ˜ π is randomly chosen only once, at the beginning of the first epoch, referred to as the shuffle-once (SO) scheme; (iii) π ( k ) randomly generated at the beginning of each epoch, referred to as random reshuffling (RR).

For general smooth convex settings, the convergence of shuffled SGD has been established only recently. For the number of epochs K sufficiently large, [31] proved a convergence rate

√

O (1 / nK ) for RR, which leads to the complexity matching SGD. This result was later improved to O (1 / ( n 1 / 3 K 2 / 3 )) by [12, 30, 34] for K sufficiently large and with bounded variance assumed at the minimizer, while the same rate holds for SO [30]. These results were complemented by matching lower bounds in [12], under sufficiently small step sizes as utilized in prior work. The results in [30, 34] require restricted O (1 / ( nL )) step sizes and reduce to O (1 /K ) for small K , acquiring the same iteration complexity as full-gradient methods. Unlike in strongly convex settings, we are not aware of any follow-up work with improvements under small K for smooth convex settings.

The major difficulty in analyzing shuffled SGD comes from characterizing the difference between the intermediate iterate and the iterate after one full data pass, for which current analysis (see e.g., [30] in smooth convex settings) uses the global smoothness constant with a triangle inequality. Such a bound may be too pessimistic and fail capturing the nuances of intermediate progress of shuffled SGD, which leads to a small step size and large K restrictions. To provide a more fine-grained analysis that narrows the theory-practice gap for shuffled SGD, we notice that such a proof difficulty is reminiscent of the analysis of cyclic block coordinate methods relating the partial gradients to the full one. This natural connection was further emphasized in studies of cyclic methods with random permutations [24, 47]; however, these results were limited to convex quadratics. More generally, it is possible to interpret shuffled SGD as a primal-dual method performing cyclic updates on the dual side (see (PD) in Section 2.1 and (PL-PD) in Section 3). We note here that prior work on dual coordinate methods [41] provided theoretical guarantees only for the algorithms that choose the dual coordinate to optimize uniformly at random, while the cyclic variant (related to shuffled SGD) had only been studied numerically up until this work. Further discussion of related work appears in Appendix A.

1.2 Contributions

In this work, we study the convergence rates of shuffled SGD in various settings through a unified primal-dual perspective, making intriguing connections to cyclic coordinate methods. This analysis framework is novel and allows us to leverage cyclic bias accumulation techniques on the dual side to obtain fine-grained convergence bounds. The obtained bounds mirror the improvements in randomized coordinate methods, which come from different coordinate smoothness parameters. While coordinate methods are no better than full-gradient methods in the worst case, on typical

problem instances, they are much faster and the improvements come precisely from a more finegrained view of smoothness. We see a similar phenomenon in our analysis, which highlights the usefulness of the fine-grained smoothness characterizations introduced in our work.

We provide improved bounds for all three popular data permutation strategies RR, SO and IG, in smooth convex settings. When the problem objective narrows to empirical risk minimization with linear predictors, we are able to exploit the data-dependent structure and uncouple the linear and nonlinear parts of the objective function, allowing us to provide tighter data-dependent bounds, up to a factor of O ( √ n ) . Moreover, we show that our techniques extend to non-smooth convex settings, providing improved bounds over existing work.

We summarize our results and compare them to the state of the art in Table 1. As is standard, all complexity results in Table 1 are expressed in terms of individual (component) gradient evaluations. They represent the number of gradient evaluations required to construct a solution with (expected) optimality gap ϵ, given a target error ϵ > 0 .

Extensions to mini-batching and IG. When presenting our results for general finite-sum problems (in Section 2), we consider simple updates without mini-batching for ease of presentation and to avoid introducing excessive notation. However, we emphasize that all our results can be extended to shuffled SGD with mini-batching. Our results are also the first to provide convergence bounds that demonstrate benefits of mini-batching in shuffled SGD. For completeness and generality, the proofs in the appendix are carried out for mini-batch settings with arbitrary batch sizes b ∈ { 1 , . . . , n } . Thus, all the results stated in Section 2 can be recovered by setting b = 1 . Moreover, our framework can provide similar fine-grained convergence bounds for IG. However, as IG is not as commonly used in practice compared to RR and SO and due to space constraints, we only present our results for RR and SO in the main body and include the results for IG in the appendix.

1.3 Notation

We consider a real d -dimensional Euclidean space ( R d , ∥· ∥ ) where d is finite and ∥· ∥ is the ℓ 2 -norm. For a vector x , we let x j denote its j -th coordinate. For any positive integer m , we use [ m ] to denote the set { 1 , 2 , . . . , m } . Given a matrix A , ∥ A ∥ := sup x ∈ R d , ∥ x ∥≤ 1 ∥ Ax ∥ denotes its operator norm. For a positive definite matrix Λ , ∥ · ∥ Λ denotes the Mahalanobis norm, ∥ x ∥ Λ := √ ⟨ Λ x , x ⟩ . We use I to denote the identity matrix, and diag( v ) to denote the diagonal matrix with vector v on the main diagonal. For any j ∈ [ n ] , we define I j ↑ as the matrix obtained from the identity matrix I by setting the first j diagonal elements to zero, and let I j be the matrix with only the j -th diagonal element nonzero and equal to 1 . To handle the cases with random data permutations, we use the following definitions corresponding to the data permutation π = { π 1 , π 2 , . . . , π n } of [ n ] : A π := [ a π 1 , a π 2 , . . . , a π n ] ⊤ permuting the rows based on π given a matrix A = [ a 1 , a 2 , . . . , a n ] ⊤ , and v π := ( v π 1 , v π 2 , . . . , v π n ) ⊤ permuting the coordinates/subvectors based on π given a vector v = ( v 1 , v 2 , . . . , v n ) ⊤ .

2 Primal-Dual Framework for Smooth Convex Finite-Sum Problems

Throughout this section, we make the following standard assumptions.

Assumption 1. Each f i is convex and L i -smooth, and there exists a minimizer x ∗ ∈ R d for f ( x ) .

Assumption 1 implies that f and all component functions f i are L -smooth, where L max := max i ∈ [ n ] L i . It also implies that each convex conjugate f ∗ i is 1 L i -strongly convex [3]. In this section, we define Λ = diag( L 1 , . . . , L 1 ︸ ︷︷ ︸ d , . . . , L n , . . . , L n ︸ ︷︷ ︸ d ) ∈ R nd × nd , and slightly abuse the no- tation to use Λ π = diag ( L π 1 , . . . , L π 1 ︸ ︷︷ ︸ d , . . . , L π n , . . . , L π n ︸ ︷︷ ︸ d ) given a permutation π of [ n ] . For the

permutation π k at the k -th epoch, we denote Λ k = Λ π k , for brevity.

We further assume that the variance at x ∗ is bounded, same as prior work [30, 34].

Algorithm 1 Shuffled SGD (Primal-Dual View, General Convex Smooth) 1: Input: Initial point x 0 ∈ R d , step size { η k } > 0 , number of epochs K > 0 2: for k = 1 to K do 3: Generate some permutation π k of [ n ] (either deterministic or random) 4: x k -1 , 1 = x k -1 5: for i = 1 to n in the ordering of π k do 6: y i k = arg max y i ∈ R d { 〈 y i , x k -1 ,i 〉 -f ∗ i ( y i ) } 7: x k -1 ,i +1 = arg min x ∈ R d { 〈 y i k , x 〉 + 1 2 η k ∥ x -x k -1 ,i ∥ 2 } = x k -1 ,i -η k ∇ f ( x k -1 ,i ) 8: end for 9: x k = x k -1 ,n +1 , y k = ( y 1 k , y 2 k , . . . , y n k ) ⊤ 10: end for 11: Return: ˆ x K = ∑ K k =1 η k x k / ∑ K k =1 η k

2.1 Primal-dual view of shuffled SGD

Problem (P) can be reformulated into a primal-dual form using the standard Fenchel conjugacy argument (see, e.g., [13, 14]),

where we slightly abuse the notation to denote y = ( y 1 , . . . , y n ) ⊤ ∈ R nd and f ∗ i is the convex conjugate of f i defined by f ∗ i ( y ) = sup x ∈ R d { ⟨ y , x ⟩ -f i ( x ) } . We let y x = ( y 1 x , . . . , y n x ) ⊤ ∈ R nd be the conjugate pair of x ∈ R d , i.e., y i x = arg max y ∈ R d {⟨ y , x ⟩-f ∗ i ( y ) } , and we denote y ∗ = y x ∗ .

Given a primal-dual pair ( x , y ) , the primal-dual gap of (PD) is defined by Gap ( x , y ) = max ( u , v ) {L ( x , v ) -L ( u , y ) } . In particular, we consider the pair ( x , y ∗ ) for x ∈ R d , and bound Gap v ( x , y ∗ ) := L ( x , v ) - L ( x ∗ , y ∗ ) for an arbitrary but fixed v . To finally obtain the function value gap f ( x ) -f ( x ∗ ) for (P), we only need to choose v = arg max w L ( x , w ) = y x .

Using this primal-dual formulation and standard convex conjugacy arguments, we can equivalently write the standard shuffled SGD algorithm in a primal-dual form as summarized in Algorithm 1.

Improved bounds with new smoothness constants. To prove a convergence bound for shuffled SGD in this general setting, we first construct an upper estimate of Gap v ( x k , y ∗ ) for some fixed v to be set later, as summarized in the following lemma.

Lemma 1. Under Assumption 1, for any k ∈ [ K ] , the iterates { y i k } n i =1 and { x k -1 ,i } n +1 i =1 generated by Algorithm 1 satisfy

where E k := η k Gap v ( x k , y ∗ ) + 1 2 n ∥ x ∗ -x k ∥ 2 2 -1 2 n ∥ x ∗ -x k -1 ∥ 2 2 , v k = v π ( k ) , and y ∗ ,k = y ∗ ,π ( k ) are the (block-wise) permuted vectors based on the permutation π k at the k -th epoch.

We note that the first term T 1 := η k n ∑ n i =1 〈 y i k , x k -x k -1 ,i +1 〉 from Lemma 1 can be aggregated into the terms capturing the primal progress within one epoch and cancelled by the last term in Eq. (1). The precise bound on T 1 and its proof are provided in Lemma 10 in Appendix C.1. The second term T 2 := η k n ∑ n i =1 〈 v i k -y i k , x k -x k -1 ,i 〉 requires us to relate the intermediate iterate x k -1 ,i to the iterate x k after one full data pass, which corresponds to a partial sum of the component gradients, each at different iterates { x k -1 ,j } n j = i . In contrast to prior analyses (e.g., Mishchenko et al. [30]) using the global smoothness and triangle inequality to bound this partial sum, we provide a tighter bound on T 2 that tracks the progress of the cyclic update on the dual side, in the aggregate.

To simplify the notation in the following lemmas and to clearly compare our results, we introduce the following novel definitions of smoothness constants for shuffled SGD:

where I ( di ) = ∑ di j = d ( i -1)+1 I j and E = [ I d , . . . , I d ︸ ︷︷ ︸ n ] ⊤ ∈ R nd × d . Permutation-dependent quantities

ˆ L g π and ˜ L g π defined in (2) are obtained directly from our analysis. We remark that ˆ L g is bounded by the average smoothness of f and ˜ L g is bounded by the max of individual smoothness constants of f i ; see more details in Appendix B. However, as we argue in later sections, these upper bounds on ˆ L g π and ˜ L g π are loose in general, and so the convergence bounds based on ˆ L g π and ˜ L g π that we obtain align better with the empirical performance of shuffled SGD.

Assuming that a uniformly random data shuffling strategy is used (SO or RR), the resulting bound on T 2 is summarized in Lemma 2, while its proof is deferred to Appendix B.

Lemma 2. Under Assumptions 1 and 2, for any k ∈ [ K ] , the iterates { y i k } n i =1 and { x k -1 ,i } n +1 i =1 generated by Algorithm 1 with uniformly random shuffling (RR/SO) satisfy

where T 2 := η k n ∑ n i =1 〈 v i k -y i k , x k -x k -1 ,i 〉 , v k = v π ( k ) and y ∗ ,k = y ∗ ,π ( k ) .

With Lemmas 1 and 2 in tow, we are ready to present the main result of this section.

Theorem 1. Under Assumptions 1 and 2, if η k ≤ 1 n √ 2 ˆ L g π ( k ) ˜ L g π ( k ) and H K = ∑ K k =1 η k , the output ˆ x K of Algorithm 1 with uniformly random (RR/SO) shuffling satisfies

As a consequence, for any ϵ > 0 , there exists a choice of a constant step size η k = η for which E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ after O ( n √ ˆ L g ˜ L g ∥ x 0 -x ∗ ∥ 2 2 ϵ + √ n ˜ L g σ ∗ ∥ x 0 -x ∗ ∥ 2 2 ϵ 3 / 2 ) individual gradient queries.

3 Tighter Bounds for Convex Finite-Sum Problems with Linear Predictors

To study the effect of the structure of the data on the convergence of shuffled SGD, we sharpen the focus from general finite-sum problems to convex finite-sum with linear predictors:

where a i ∈ R d ( i ∈ [ n ] ) are data vectors and ℓ i : R → R are convex and either smooth or Lipschitz nonsmooth functions associated with the linear predictors ⟨ a i , x ⟩ for i ∈ [ n ] . In addition to their explicit dependence on the data, it is worth noting that problems of the form (PL) cover most of the standard convex ERM problems where shuffled SGD is commonly applied, such as support vector machines, least absolute deviation, least squares, and logistic regression.

Problem (PL) admits an explicit primal-dual formulation using the standard Fenchel conjugacy argument (see, e.g., [13, 14]),

where A = [ a 1 , a 2 , . . . , a n ] ⊤ ∈ R n × d is the data matrix and ℓ ∗ i : R → R is the convex conjugate of ℓ i . This observation allows us to again interpret without-replacement SGD updates as cyclic

coordinate updates on the dual side. Note that due to the objective structure in (PL), the primal-dual formulation (PL-PD) can decouple the linear ( a ⊤ i x ) and the non-linear ( ℓ i ) parts within individual loss functions f i . We redefine the conjugate pair of x ∈ R d to be y x = ( y 1 x , . . . , y n x ) ⊤ ∈ R n , with y i x = arg max y i ∈ R { y i a ⊤ i x -ℓ ∗ i ( y i ) } .

In this section, we consider shuffled SGD with mini-batch estimators of size b and assume without loss of generality that n = bm for some positive integer m . The detailed primal-dual view of shuffled SGD adapted to (PL-PD) and mini-batch estimators is provided in Alg. 2 in Appendix C.

3.1 Smooth and convex objectives

Throughout this subsection, we make the following (standard) assumptions, corresponding to Assumptions 1 and 2 from Section 2.

Assumption 3. Each ℓ i is convex and L i -smooth ( i ∈ [ n ]) , i.e., | ℓ ′ i ( x ) -ℓ ′ i ( y ) | ≤ L i | x -y | for any x, y ∈ R . There exists a minimizer x ∗ ∈ arg min x ∈ R d f ( x ) .

We remark that Assumption 3 implies that both f and each component function f i ( x ) = ℓ i ( a ⊤ i x ) are L max -smooth, where L max = max i ∈ [ n ] L i ∥ a i ∥ 2 2 . Assumption 3 also implies that each convex conjugate ℓ ∗ i is 1 L i -strongly convex [3]. In the following, we let Λ = diag ( L 1 , L 2 , . . . , L n ) , and Λ π = diag ( L π 1 , L π 2 , . . . , L π n ) , given a permutation π of [ n ] .

We further assume bounded variance at x ∗ , same as prior work [30, 34, 45, 46].

Improved bounds with new smoothness constants. Our convergence bounds depend on the smoothness parameters defined in Eq. (3) below. We provide a detailed discussion on how these parameters relate to traditional smoothness parameters both in the worst case and on typical datasets, in Section 4.1, with additional numerical results provided in Appendix E.

where I ( j ) := ∑ bj i = b ( j -1)+1 I i . In comparison to the smoothness constants defined in Eq. (2) for general finite-sum problems, we note that the constants in Eq. (3) applying to generalized linear models are tighter and more informative estimates, as the data matrix A and the smoothness constants from the nonlinear part Λ are separated in Eq. (3). Thus, the constants ˆ L π and ˜ L π directly depend on the data matrix, which explicitly demonstrates how the structure of the data affects the convergence of shuffled SGD. The following theorem states the convergence of Algorithm 2 with these new refined smoothness constants, while its proof is provided in Appendix C.

Theorem 2. Under Assumptions 3 and 4, if η k ≤ b n √ 2 ˆ L π ( k ) ˜ L π ( k ) and H K = ∑ K k =1 η k , then the output ˆ x K of Alg. 1 with uniformly random (RR/SO) shuffling satisfies

As a result, given ϵ > 0 , there exists a constant step size η k = η such that E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ after O ( n √ ˆ L ˜ L ∥ x 0 -x ∗ ∥ 2 2 ϵ + √ ( n -b )( n + b ) n ( n -1) √ n ˜ Lσ ∗ ∥ x 0 -x ∗ ∥ 2 2 ϵ 3 / 2 ) individual gradient queries.

A few remarks are in order here. When b = n , we recover the standard guarantee of gradient descent, which serves as a sanity check as in this case the algorithm reduces to standard gradient descent. √

When ϵ = Ω( ( n -b )( n + b ) σ 2 ∗ n 2 ( n -1) ˆ L ) , the resulting complexity is O ( n ˆ L ˜ L ∥ x 0 -x ∗ ∥ 2 2 ϵ ) . Observe that this case can happen when either ϵ is large (compared to, say, 1 /n ) or when σ ∗ is small (it is, in fact, possible for σ ∗ to be zero, which happens, for example, when the data rows are linearly independent). Unlike

in bounds from previous work, we observe from our bounds the benefit of using shuffled SGD compared to full gradient descent, where the difference is by a factor that can be as large as √ n, as we have discussed in the introduction (see also Section 4). When ϵ = O ( ( n -b )( n + b ) σ 2 ∗ n 2 ( n -1) ˆ L ) , the second term in our complexity bound dominates. In this case, when b = 1 , we recover the state of the art results from [12, 30, 34], while for b > 1 our bound provides the Ω ( √ n ( n -1) ( n -b )( n + b ) · L ˜ L ) -factor improvement, providing insights into benefits from the mini-batching strategy commonly used in practice.

3.2 Extension to non-smooth convex objectives

In non-smooth settings, we make the following standard assumption.

Assumption 5. Each ℓ i is convex and G i -Lipschitz ( i ∈ [ n ]) , i.e., | ℓ i ( x ) -ℓ i ( y ) | ≤ G i | x -y | for any x, y ∈ R ; thus | g i ( x ) | ≤ G i where g i ( x ) ∈ ∂ℓ i ( x ) . There exists a minimizer x ∗ ∈ arg min x ∈ R d f ( x ) .

If Assumption 5 holds, each ℓ i ( a ⊤ i x ) is also G max -Lipschitz with respect to x , where G max = max i ∈ [ n ] G i ∥ a i ∥ 2 . To state our results, we define Γ := diag ( G 2 1 , G 2 2 , . . . , G 2 n ) and Γ π = diag ( G 2 π 1 , G 2 π 2 , . . . , G 2 π n ) , given a data permutation π of [ n ] .

We now extend our analysis of Algorithm 1 to convex nonsmooth Lipschitz settings, where the conjugate functions ℓ ∗ i ( y i ) are only convex. Proceeding as in Lemma 1, we obtain a bound on the primal-dual gap similar to (1), but lose two retraction terms induced by smoothness. Instead of cancelling the corresponding error terms like in the smooth case, we rely on the boundedness of the subgradients to bound these terms under a sufficiently small step size, which is common in nonsmooth Lipschitz settings. Similar to Section 2, we introduce the following quantities to obtain a tighter guarantee with respect to the data matrix and Lipschitz constants

We discuss the improvements in convergence from ˆ G π and ˜ G π in Section 4, while the convergence of Algorithm 2 is described in Theorem 3, with its proof deferred to Appendix D.

Theorem 3. Under Assumption 5, if H K = ∑ K k =1 η k and ¯ G = E π [ √ ˆ G π ˜ G π ] , the output ˆ x K of Alg. 1 with possible uniformly random shuffling satisfies

As a result, for any ϵ > 0 , there exists a step size η k = η such that E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ after O ( n ¯ G ∥ x 0 -x ∗ ∥ 2 2 ϵ 2 ) individual gradient queries.

4 Discussion of Our New Smoothness Constants and Numerical Results

To succinctly explain where our improvements come from, we now consider (PL) where ℓ i is 1 -smooth and b = 1 , ignoring the gains from the mini-batch estimators (for large K ) and our softer guarantee that handles individual smoothness constants. For this specific case, ˜ L = L max = max 1 ≤ i ≤ n ∥ a i ∥ 2 , and thus our results for the smooth case and the RR and SO variants match state of the art in the second term, which dominates when there are many ( K = Ω( L 2 max D 2 n σ 2 ∗ ) ) epochs. When there are

K = O ( L 2 max D 2 n σ 2 ∗ ) epochs in the SO and RR variants or for all regimes of K in the IG variant, the difference between our and state of the art bounds comes from the constant ˆ L that replaces L max , and our improvement is by a factor √ L max / ˆ L . Note that O ( nL max ϵ ) from prior bounds, which is the dominating term in the small K regime, is even worse than the complexity of full gradient descent, as the full gradient Lipschitz constant of f in this case is 1 n ∥ AA ⊤ ∥ 2 ≤ L max .

Given a worst-case permutation ¯ π , and denoting by A ¯ π the data matrix A with its rows permuted according to ¯ π, our constant ˆ L can be bounded above by L max using the following sequence of inequalities:

where ( i ) holds by the triangle inequality, ( ii ) holds because the operator norm of the matrix I ( j -1) ↑ A π A ⊤ π I ( j -1) ↑ (equal to the operator norm of the bottom right ( n -j +1) × ( n -j +1) submatrix of A π A ⊤ π ) is always at most ∥ A π A ⊤ π ∥ = ∥ AA ⊤ ∥ , for any permutation π , and ( iii ) holds by bounding above the operator norm of a symmetric matrix by its trace. Hence ˆ L is never larger than L max , but can generally be much smaller, due to the sequence of inequality relaxations in (4). While each of these inequalities can be loose, we emphasize that ( iii ) is almost always loose, by a factor that can be as large as n .

As a specific example where ˆ L is smaller than L max by a factor of n, consider the example of Gaussian data, where we draw n i.i.d. standard Gaussian vectors from N ( 0 , I d ) and take d = n. By standard concentration results, with high probability, all columns/rows of A ¯ π in this case are near-orthogonal (see, e.g., [7, Chapter]) and ∥ a i ∥ 2 2 ≈ d = n for all i . As a result, the operator norm to trace inequality ( iii ) is loose by a factor d = n, with high probability. Note that in this example all individual smoothness parameters of components f i are essentially the same (w.h.p.) and equal ∥ a i ∥ 2 2 , thus the improvement of our bound on the smoothness parameter does not come from averaging but from the structure of the data. This observation is important for contrasting the results from Section 2 and Section 3. In particular, focusing solely on the finite sum structure and ignoring the structure of the data matrix would provide no improvements in the resulting convergence bounds.

As further evidence, we empirically evaluate L max / ˆ L on 15 large-scale machine learning datasets and demonstrate that on those datasets L max / ˆ L is of the order n α , for α ∈ [0 . 15 , 0 . 96] (see Sec. 4.1 for more details), providing strong evidence of a tighter guarantee as a function of n .

For the nonsmooth settings, by a similar sequence of inequalities, we can show that ¯ G ≤ G 2 max , which can be loose by a factor 1 /n due to the operator norm to trace inequality. Thus, our bound is never worse than what would be obtained from the full subgradient method, but can match the bound of standard SGD, or even improve 1 upon it for at least some data matrices A .

4.1 Numerical results and discussion

In this section, we provide empirical evidence to support our claim about usefulness of the new convergence bounds obtained in our work. In particular, we conduct numerical evaluations to compare ˆ L to the classical smoothness constant L on synthetic datasets and on popular machine learning benchmark datasets.

For a more streamlined comparison and to focus on the dependence on the data matrix, we assume that the loss functions ℓ i all have the same smoothness constant, which leads to L max / ˆ L = (max 1 ≤ i ≤ n {∥ a i ∥ 2 } ) / ( 1 n 2 ∥ ∑ n j =1 I ( j -1) ↑ A ¯ π A ⊤ ¯ π I ( j -1) ↑ ∥ 2 ) . Since the scale of the smoothness constant of the loss functions is irrelevant for the ratio L max / ˆ L in this case, for simplicity, we take it to equal one. Note that assuming different smoothness constants over component loss functions would only make our bound better compared to related work (see Eq. (3) and the discussion following it).

We also compare ˆ L and L max on a number of benchmarking datasets from LIBSVM [15], MNIST [17], CIFAR10 [22], and Broad Bioimage Benchmark Collection [28]. For each dataset, we generate a uniformly random permutation π for the data matrix A and compute ˆ L π . We repeat this procedure 1000 times for all datasets and display the average L max / ˆ L π in Table 2, except for e2006train , CIFAR10 , MNIST , and BBBC005 where we do 20 repetitions due to limitations of

computation resources required for each calculation. We observe that among the datasets that we consider, which contain all three data matrix 'shapes' d >> n , d << n , and d ≈ n , our novel bound dependent on ˆ L is much tighter. For instance, for rcv1 and real-sim datasets, where d and n are of the same order, we observe that L max / ˆ L are approximately 111 and 194 , respectively. For news20 dataset where d >> n , L max / ˆ L ≈ 42 . 1 . For MNIST , where d << n, L max / ˆ L ≈ 19 . 1 . Further results are provided in Appendix E.

Finally, as a justification for using the empirical mean of ˆ L π over random permutations π in the results displayed in Table 2, we observe in our evaluations that the values of L max / ˆ L π are fairly concentrated around their empirical mean values. Histogram plots showing the empirical distributions of L max / ˆ L π for each of the datasets are provided in Appendix E.

We conclude with a few additional remarks. Our results indicate that the structure of the data is important for predicting behavior of popular machine learning methods such as variants of shuffled SGD considered in our work, and thus should be incorporated in their study: as demonstrated in the Gaussian data example, considering simple finite sum structure and ignoring the dependence on the data can lead to overly pessimistic bounds. Thus it would be interesting to provide a further theoretical study of shuffled SGD that incorporates distributional assumptions for the data. Additionally, as mentioned in the previous paragraph, we empirically observed that permutation-dependent parameter ˆ L π concentrates around its mean for permutations generated uniformly at random. Thus, it would be interesting to consider whether our theoretical results can be strengthened to depend on the mean value of ˆ L π (as opposed to maximum). We leave such considerations for future work.

References

Alekh Agarwal, Martin J Wainwright, Peter Bartlett, and Pradeep Ravikumar. Informationtheoretic lower bounds on the oracle complexity of convex optimization. In Proc. NeurIPS'09 , 2009.

Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. SGD with shuffling: optimal rates without component convexity and large epoch requirements. In Proc. NeurIPS'20 , 2020.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Amir Beck and Luba Tetruashvili. On the convergence of block coordinate descent type methods. SIAM Journal on Optimization , 23(4):2037-2060, 2013.

Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks of the Trade: Second Edition , 2012.

Dimitri P Bertsekas and John N Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization , 10(3):627-642, 2000.

Avrim Blum, John Hopcroft, and Ravi Kannan. Foundations of Data Science . Cambridge University Press, Cambridge, 2020. ISBN 9781108485067. doi: DOI:. URL https://www.cambridge.org/core/books/foundations-of-data-science/ 6A43CE830DE83BED6CC5171E62B0AA9E .

Léon Bottou. Curiously fast convergence of some stochastic gradient descent algorithms. In Proc. Symposium on Learning and Data Science, Paris'09 , 2009.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 2015.

Xufeng Cai, Chaobing Song, Stephen J Wright, and Jelena Diakonikolas. Cyclic block coordinate descent with variance reduction for composite nonconvex optimization. arXiv preprint arXiv:2212.05088 , 2022.

Jaeyoung Cha, Jaewook Lee, and Chulhee Yun. Tighter lower bounds for shuffling SGD: Random permutations and beyond. arXiv preprint arXiv:2303.07160 , 2023.

Antonin Chambolle and Thomas Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision , 40(1):120-145, 2011.

Antonin Chambolle, Matthias J Ehrhardt, Peter Richtárik, and Carola-Bibiane Schonlieb. Stochastic primal-dual hybrid gradient algorithm with arbitrary sampling and imaging applications. SIAM Journal on Optimization , 28(4):2783-2808, 2018.

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27, 2011.

Christopher M De Sa. Random reshuffling is not always better. In Proc. NeurIPS'20 , 2020.

Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

MGurbuzbalaban, Asu Ozdaglar, and Pablo A Parrilo. Convergence rate of incremental gradient and incremental Newton methods. SIAM Journal on Optimization , 29(4):2542-2565, 2019.

Mert Gurbuzbalaban, Asuman Ozdaglar, Pablo A Parrilo, and Nuri Vanli. When cyclic coordinate descent outperforms randomized coordinate descent. In Proc. NeurIPS'17 , 2017.

Mert Gürbüzbalaban, Asu Ozdaglar, and Pablo A Parrilo. Why random reshuffling beats stochastic gradient descent. Mathematical Programming , 186:49-84, 2021.

Jeff Haochen and Suvrit Sra. Random shuffling beats SGD after finite epochs. In Proc. ICML'19 , 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Zehua Lai and Lek-Heng Lim. Recht-ré noncommutative arithmetic-geometric mean conjecture is false. In Proc. ICML'20 , 2020.

Ching-Pei Lee and Stephen J Wright. Random permutations fix a worst case for cyclic coordinate descent. IMA Journal of Numerical Analysis , 39(3):1246-1275, 2019.

Xiao Li, Zhihui Zhu, Anthony Man-Cho So, and Jason D Lee. Incremental methods for weakly convex optimization. arXiv preprint arXiv:1907.11687 , 2019.

Xingguo Li, Tuo Zhao, Raman Arora, Han Liu, and Mingyi Hong. On faster convergence of cyclic block coordinate descent-type methods for strongly convex minimization. The Journal of Machine Learning Research , 18(1):6741-6764, 2017.

Cheuk Yin Lin, Chaobing Song, and Jelena Diakonikolas. Accelerated cyclic coordinate dual averaging with extrapolation for composite convex optimization. arXiv preprint arXiv:2303.16279 , 2023.

Vebjorn Ljosa, Katherine L. Sokolnicki, and Anne E Carpenter. Broad bioimage benchmark collection. https://bbbc.broadinstitute.org/image_sets , 2012. Accessed: 2023-0516.

Olvi L Mangasarian and MV Solodov. Serial and parallel backpropagation convergence via nonmonotone perturbed minimization. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 1993.

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Simple analysis with vast improvements. In Proc. NeurIPS'20 , 2020.

Dheeraj Nagaraj, Prateek Jain, and Praneeth Netrapalli. SGD without replacement: Sharper rates for general smooth convex functions. In Proc. ICML'19 , 2019.

Angelia Nedic and Dimitri P Bertsekas. Incremental subgradient methods for nondifferentiable optimization. SIAM Journal on Optimization , 12(1):109-138, 2001.

Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362, 2012.

Lam M Nguyen, Quoc Tran-Dinh, Dzung T Phan, Phuong Ha Nguyen, and Marten Van Dijk. A unified convergence analysis for shuffling-type gradient methods. The Journal of Machine Learning Research , 22(1):9397-9440, 2021.

Shashank Rajput, Anant Gupta, and Dimitris Papailiopoulos. Closing the convergence gap of SGD without replacement. In Proc. ICML'20 , 2020.

Benjamin Recht and Christopher Ré. Toward a noncommutative arithmetic-geometric mean inequality: Conjectures, case-studies, and consequences. In Proc. COLT'12 , 2012.

Benjamin Recht and Christopher Ré. Parallel stochastic gradient algorithms for large-scale matrix completion. Mathematical Programming Computation , 2013.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics , pages 400-407, 1951.

Itay Safran and Ohad Shamir. How good is SGD with random shuffling? In Proc. COLT'20 , 2020.

Ankan Saha and Ambuj Tewari. On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization , 23(1):576-601, 2013.

Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research , 14(1), 2013.

Ohad Shamir. Without-replacement sampling for stochastic gradient methods. In Proc. NeurIPS'16 , 2016.

Chaobing Song and Jelena Diakonikolas. Fast cyclic coordinate dual averaging with extrapolation for generalized variational inequalities. arXiv preprint arXiv:2102.13244 , 2021.

Ruoyu Sun and Yinyu Ye. Worst-case complexity of cyclic coordinate descent: O ( n 2 ) gap with randomized version. Mathematical Programming , 185(1):487-520, 2021.

Trang H Tran, Lam M Nguyen, and Quoc Tran-Dinh. SMG: A shuffling gradient-based method with momentum. In Proc. ICML'21 , 2021.

Trang H Tran, Katya Scheinberg, and Lam M Nguyen. Nesterov accelerated shuffling gradient method for convex optimization. In Proc. ICML'22 , 2022.

Stephen Wright and Ching-pei Lee. Analyzing random permutations for cyclic coordinate descent. Mathematics of Computation , 89(325):2217-2248, 2020.

Yangyang Xu and Wotao Yin. Block stochastic gradient iteration for convex and nonconvex optimization. SIAM Journal on Optimization , 25(3):1686-1716, 2015.

Yangyang Xu and Wotao Yin. A globally convergent algorithm for nonconvex optimization based on block coordinate update. Journal of Scientific Computing , 72(2):700-734, 2017.

Bicheng Ying, Kun Yuan, Stefan Vlaski, and Ali H Sayed. Stochastic learning under random reshuffling with constant step-sizes. IEEE Transactions on Signal Processing , 67(2):474-489, 2018.

Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond. In Proc. ICLR'22 , 2022.

Supplementary Material

Outline. The supplementary material of the paper is organized as follows:

Section A provides a brief survey on shuffled SGD and its related work.

Section B presents the proofs related to the smooth convex setting from Section 2, where we only assume each component function f i to be convex and L i -smooth.

Section C presents the proofs related to the smooth convex setting with linear predictors from Section 3.

Section D presents the proofs related to the non-smooth convex setting with linear predictors from Section 3.

Section E presents the full details of the computational experiments performed in the paper.

A Further Related Work

In this section, we continue with the discussion on the background of shuffled SGD from Section 1. Wewould like to briefly recall that shuffled SGD usually performs better in practice when compared to SGD, and is also easier and more efficient to implement. However, in terms of the theoretical analysis, sampling without replacement introduces the sampling bias at each iteration, making it difficult to approximate shuffled SGD by full gradient descent. Using empirical observations, shuffled SGD was conjectured to converge much faster than SGD with replacement, based on the noncommutative arithmetic-geometric mean inequality conjecture [36], which was later proved to be false [16, 23]. As a consequence, whether or not shuffled SGD can be faster than SGD at least in some regimes remained open [8] until a breakthrough result in [20], where it was shown that for the class of smooth strongly convex optimization problems, the convergence of the RR variant of shuffled SGD is essentially of the order(1 /K 2 ) for K full passes of the data (also called epochs), which is faster than order(1 /nK ) convergence of SGD for sufficiently large K . This bound for the smooth strongly convex case was later improved under various regimes and additional assumptions [2, 21, 30, 31, 34, 42], while the tightest of those bounds were matched by lower bounds in [12, 35, 39, 51].

Since our results are for the general (non-strongly) convex regimes, in this section we focus on the results that apply to those (convex, smooth or nonsmooth Lipschitz) regimes. For convex nonsmooth Lipschitz problems, we are only aware of the results in [42]. These results are only useful when the number of data passes K is small and the number of component functions n is large, as they contain an irreducible order1 √ n error, and are not directly comparable to our results.

For the IG variant of SGD without replacement (deterministic order), asymptotic convergence was established in [6, 29], with further convergence results for both smooth and nonsmooth settings provided in [18, 25, 30, 32, 34, 50]. As IG does not benefit from randomization, it is known to have a worse convergence bound than RR under the Lipschitz Hessian assumption [18, 21], which was also shown in more general settings [30].

In this paper, we viewed shuffled SGD as a primal-dual method where the updates are performed on the dual side in a cyclic manner, thus we can leverage techniques from general cyclic methods. However, in contrast to randomized methods (corresponding to standard SGD), cyclic methods are usually more challenging to analyze [33], basic variants exhibit much worse worst-case complexity than even full gradient methods [4, 19, 26, 26, 40, 44, 48, 49], with more refined results being established only recently [11, 27, 43]. While the inspiration for our work came from these recent results [11, 27, 43], they are completely technically disjoint. First, all these results rely on nonstandard block Lipschitz assumptions, which are not present in our work. Second, all of them leverage proximal gradient-style cyclic updates to carry out the analysis, which is inapplicable in our case for the cyclic updates on the dual side, as otherwise the method would not correspond to (shuffled) SGD. Finally, [27, 43] utilize extrapolation steps, which would break the connection to shuffled SGD in our setting, while [11] relies on a gradient descent-type descent lemma, which is impossible to establish in our setting.

B Omitted Proofs From Section 2

In this section, we consider the general finite-sum setting where we assume that each component function f i is convex and smooth, and derive the refined analysis under this setting. Here we focus on the smooth convex problems as prior work did [30, 34], since smoothness is essential to showing the advantage of shuffled SGD [31] over SGD, otherwise the rate of SGD is optimal. In particular, we study the general smooth convex finite-sum problem (P)

where each f i is convex and smooth. (P) is equivalent to

where we slightly abuse the notation in this section and use y i ∈ R d to be the i -th d elements of the vector y such that y = ( y 1 , . . . , y n ) ⊤ ∈ R nd , E = [ I d , . . . , I d ︸ ︷︷ ︸ n ] ⊤ ∈ R nd × d is the vertical concatenation of n identity matrices I d ∈ R d × d and f ∗ i is the convex conjugate of f i defined by f i ( x ) = sup y i ∈ R d 〈 y i , x 〉 -f ∗ i ( y i ) . In the following, we consider the mini-batch estimator of batch size b , and let y ( i ) ∈ R bd denote the vector comprised of the i th bd elements of y . For simplicity and without loss of generality, we assume that n = bm for some positive integer m , so that y = ( y (1) , . . . , y ( m ) ) ⊤ . Note that if choosing b = 1 , our setting is the same as the ones in [30, 34]. Then we have the primal-dual view of shuffled SGD scheme for general smooth convex minimization as in Alg. 1, where E ⊤ b = [ I d , . . . , I d ︸ ︷︷ ︸ b ] ⊤ ∈ R bd × d is the vertical concatenation of b identity matrices

I d ∈ R d × d . Given the data permutation π ( k ) = { π ( k ) 1 , π ( k ) 2 , . . . , π ( k ) n } of [ n ] at the k -th epoch, we use the same notation of v k = ( v π ( k ) 1 , . . . , v π ( k ) n ) ⊤ ∈ R nd , y ∗ ,k = ( y π ( k ) 1 ∗ , . . . , y π ( k ) n ∗ ) ⊤ ∈ R nd as in previous sections except now each v π ( k ) i , y π ( k ) i ∗ are d -dimensional subvectors. Further, we denote the permuted smoothness constant matrices by Λ k = diag( L π ( k ) 1 , . . . , L π ( k ) 1 ︸ ︷︷ ︸ d , . . . , L π ( k ) n , . . . , L π ( k ) n ︸ ︷︷ ︸ d ) ∈ R nd × nd , and we use I for I nd ∈ R nd × nd throughout this section.

New smoothness constants and comparisons. We first recall the new smoothness constants for any permutation π of [ n ] , defined in Eq. (2):

where I ( di ) = ∑ bdi j = bd ( i -1)+1 I j .

To compare ˆ L g π and L := max i ∈ [ n ] L i , we make use of the Kronecker product with notation ⊗ defined by

for two matrices A ∈ R m × n and B ∈ R p × q . The following lemma states a useful fact for the Kronecker product.

Lemma 3. For square matrices A and B of sizes p and q and with eigenvalues λ i ( i ∈ [ p ] ) and µ j ( j ∈ [ q ] ) respectively, the eigenvalues of A ⊗ B are λ i µ j for i ∈ [ p ] , j ∈ [ q ] .

We now use the following chain of inequalities to compare ˆ L g π and L for any permutation π of [ n ] :

where we define l π = (√ L π 1 , √ L π 2 , . . . , √ L π n ) ⊤ . For ( i ) , we use Lemma 3 and notice that the eigenvalues of I d all equal 1 , while the largest eigenvalue of l π l ⊤ π = ∥ l ∥ 2 2 = ∑ n i =1 L i , so the operator norm of ( l k l ⊤ k ) ⊗ I d is ∑ n i =1 L i .

To compare ˜ L g π and L , we notice that

is a block diagonal matrix whose operator norm is the maximum of the operator norms over its diagonal block submatrices, so we have

where for ( i ) we use Lemma 3 for each submatrix ( l ( i ) π l ( i ) ⊤ π ) ⊗ I d and l ( i ) π = (0 , . . . , 0 , √ L π b ( i -1)+1 , . . . , √ L π bi , 0 , . . . , 0) ⊤ . Similar to the case of generalized linear models, the inequality is tight when b = 1 but can be loose for other values of b .

Before proceeding to the omitted proofs, we first state the following standard definitions and first-order characterization of strong convexity, for completeness.

Definition 1. A function f : R d → R is said to be µ -strongly convex with parameter µ > 0 , if for any x , y ∈ R d and any λ ∈ (0 , 1) :

Lemma 4. Let f : R d → R be a continuous µ -strongly convex function with µ > 0 . Then, for any x , y ∈ R d :

where g x ∈ ∂f ( x ) , and ∂f ( x ) is the subdifferential of f at x .

We also include the following lemma on the variance bound under without-replacement sampling, which is useful for our proof.

Lemma 5. Let B be the set of |B| = b samples from [ n ] , drawn without replacement and uniformly at random. Then, ∀ x ∈ R d ,

Proof. We first expand the square on the left-hand side, as follows

̸

̸

Since the batch B is sampled uniformly and without replacement from [ n ] , the probability that any pair ( i, i ′ ) from [ n ] with i = i ′ is in B is b ( b -1) n ( n -1) . By the linearity of expectation, we have

̸

̸

where ✶ is the indicator function such that ✶ i,i ′ ∈B = 1 if both i, i ′ ∈ B and is equal to zero otherwise. Hence, we obtain

̸

where ( i ) is due to f = 1 n ∑ n i =1 f i having the finite sum structure.

Now we provide the omitted proofs from Section 2.

Lemma 6. Under Assumption 1, for any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 1 satisfy

Proof. We first note that based on Line 6 of Alg. 1, we have

̸

̸

Since the max problem defining y k is separable, we have for b ( i -1) + 1 ≤ j ≤ bi and i ∈ [ m ]

which leads to x k -1 ,i ∈ ∂f ∗ π ( k ) j ( y j k ) . Further, since each component function f ∗ j is 1 L j -strongly convex thus for b ( i -1) + 1 ≤ j ≤ bi , we also have

which leads to

Using the same argument, as x ∗ ∈ ∂f ∗ i ( y i ∗ ) for i ∈ [ n ] , we have

Thus,

Using the updating scheme of x k -1 ,i +1 and noticing that ϕ i k ( x ) = 〈 E ⊤ b y ( i ) k , x 〉 + b 2 η k ∥ x -x k -1 ,i ∥ 2 is b η k -strongly convex and minimized at x k -1 ,i +1 , we have

which leads to

Hence, combining the bounds on L ( x k , v ) and L ( x ∗ , y ∗ ) and letting

we obtain

thus completing the proof.

We note that the first inner product term T 1 := η k n ∑ m i =1 〈 E ⊤ b y ( i ) k , x k -x k -1 ,i +1 〉 in Eq. (5) can be cancelled by the last negative term -b 2 n ∑ m i =1 ∥ x k -1 ,i +1 -x k -1 ,i ∥ 2 therein, as precisely proved in Lemma 10 of Appendix C. In the following subsections, we continue our analysis and handle the remaining terms in Eq. (5) according to different shuffling and derive the final complexity.

B.1 Random reshuffling/shuffle-once schemes

We introduce the following lemma to bound the second inner product term T 2 := η k n ∑ m i =1 〈 E ⊤ b ( v ( i ) k -y ( i ) k ) , x k -x k -1 ,i 〉 in Lemma 6 when there are random permutations.

Lemma 7. Under Assumptions 1 and 2, for any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 1 with uniformly random shuffling (RR/SO) satisfy

Proof. First note that x k -x k -1 ,i = ∑ m j = i ( x k -1 ,j +1 -x k -1 ,j ) = -η k b ∑ m j = i E ⊤ b y ( j ) k = -η k b E ⊤ I bd ( i -1) ↑ y k , so we have

For the term I 1 , we use Young's inequality with α > 0 to be set later and obtain

Further, notice that

where for the last inequality we use Cauchy-Schwarz inequality. Using the same argument, we can bound

Thus, combining (6)-(8) and choosing α = 2 η k ˜ L g π ( k ) , we obtain

For the term I 2 , we again apply Young's inequality with β > 0 to be set later and obtain

Choosing β = 2 η k ˜ L g and using the fact that ˜ L g π ( k ) ≤ ˜ L g , we have

Hence, combining the above two estimates with m = n/b , we have

First, consider the RR scheme. Taking conditional expectation on both sides w.r.t. the randomness up to but not including k -th epoch, we have

For the first term, since the only randomness comes from the permutation π ( k ) , we can proceed as in the proof of Lemma 11 and obtain

where we use the linearity of expectation for ( i ) , and ( ii ) is due to Lemma 5 and the definition σ 2 ∗ = 1 n ∑ n i =1 ∥∇ f i ( x ∗ ) ∥ 2 = 1 n ∑ n i =1 ∥ y i ∗ ∥ 2 . Then taking expectation w.r.t. all randomness on both sides, we obtain

Finally, we remark that the above argument for bounding the term η 3 k ˜ L g bn E k [ ∑ m i =1 ∥ E ⊤ I bd ( i -1) ↑ y ∗ ,k ∥ 2 ] also applies to the SO scheme, in which case there is only one random permutation at the very beginning that induces the randomness.

We state the final convergence rate and complexity in the following theorem and provide the proof for completeness.

Theorem 4. Under Assumptions 1 and 2, if η k ≤ b n √ 2 ˆ L g π ( k ) ˜ L g π ( k ) and H K = ∑ K k =1 η k , the output ˆ x K of Algorithm 1 with uniformly random (RR/SO) shuffling satisfies

As a consequence, for any ϵ > 0 , there exists a choice of a constant step size η k = η for which E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ after O ( n √ ˆ L g ˜ L g ∥ x 0 -x ∗ ∥ 2 2 ϵ + √ ( n -b )( n + b ) n ( n -1) √ n ˜ L g σ ∗ ∥ x 0 -x ∗ ∥ 2 2 ϵ 3 / 2 ) gradient queries.

Proof. Combining the bounds in Lemma 10 and 2 and plugging them into Eq. (5), we obtain

For the stepsize η k such that η k ≤ b n √ 2 ˆ L g π ( k ) ˜ L g π ( k ) , we have η 3 k n ˆ L g π ( k ) ˜ L g π ( k ) b 2 -η k 2 n ≤ 0 , thus

Using our definition of E k and telescoping from k = 1 to K , we have

Noticing that L ( x , v ) is convex in x for a fixed v , we have Gap v (ˆ x K , y ∗ ) ≤ ∑ K k =1 η k Gap v ( x k , y ∗ ) /H K , where ˆ x K = ∑ K k =1 η k x k /H K and H K = ∑ K k =1 η k , which leads to

Further choosing v = y ˆ x K , we obtain

To analyze the individual gradient oracle complexity, we choose constant stepsizes η ≤ b n √ 2 ˆ L g ˜ L g , then Eq. (9) will become

̸

Without loss of generality, we assume that b = n , otherwise the method and its analysis reduce to (full) gradient descent. We consider the following two cases:

Combining these two cases by setting η = min { b n √ 2 ˆ L g ˜ L g , ( 3 b 3 ( n -1) ∥ x 0 -x ∗ ∥ 2 2 n ( n -b )( n + b ) ˜ L g Kσ 2 ∗ ) 1 / 3 } , we obtain

Hence, to guarantee E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ for ϵ > 0 , the total number of individual gradient evaluations will be

as claimed.

B.2 Incremental gradient descent (IG)

In this subsection, we provide the convergence results for incremental gradient descent which does not involve random permutations. We first prove the technical lemma below to bound the term T 2 := η k n ∑ m i =1 〈 E ⊤ b ( v ( i ) k -y ( i ) k ) , x k -x k -1 ,i 〉 in Eq. (5) of Lemma 6.

Lemma 8. For any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 1 with fixed data ordering satisfy

Proof. Proceeding as in the proof of Lemma 7, we have

For both terms I 1 and I 2 , we apply Young's inequality with α = 2 η k ˜ L g 0 and obtain

and

Wenowshow that the term η 3 k ˜ L g 0 nb ∑ m i =1 ∥ E ⊤ I bd ( i -1) ↑ y ∗ ∥ 2 2 is no larger than either η 3 k n b 2 ˆ L g 0 ˜ L g 0 ∥ y ∗ ∥ 2 Λ -1 or η 3 k ( n -b ) 2 b 2 ˜ L g 0 σ 2 ∗ . This is trivial when b = n as E ⊤ I 0 ↑ y ∗ = ∑ n i =1 y i ∗ = 0 . When b < n , to show the former one, we have

To prove the latter one, we notice that

using the fact that ∑ n i =1 y i ∗ = 0 . Then using Young's inequality we obtain

Further noticing that ∑ ( m -1) b i =1 ∥ y i ∗ ∥ 2 2 ≤ ∑ n i =1 ∥ y i ∗ ∥ 2 = nσ 2 ∗ , we have

The same bound also captures the case b = n and leads to

Hence, combining Eq. (11)-(13), we obtain

which finishes the proof.

We are now ready to state our convergence results for IGD in the following theorem, with its proof provided for completeness.

Theorem 5. Under Assumptions 1 and 2, if η k ≤ b n √ 2 ˆ L g 0 ˜ L g 0 and H K = ∑ K k =1 η k , the output ˆ x K of Algorithm 1 with a fixed permutation satisfies

As a consequence, for any ϵ > 0 , there exists a choice of a constant step size η k = η such that f (ˆ x K ) -f ( x ∗ ) ≤ ϵ after O ( n √ ˆ L g 0 ˜ L g 0 ∥ x 0 -x ∗ ∥ 2 2 ϵ + min { √ n ˆ L g 0 ˜ L g 0 ∥ y ∗ ∥ Λ -1 , ( n -b ) √ ˜ L g 0 σ ∗ } ∥ x 0 -x ∗ ∥ 2 2 ϵ 3 / 2 ) gradient queries.

Proof. Combining the bounds in Lemma 10 and 8 and plugging them into Eq. (5) in Lemma 6 without random permutations, we have

If η k ≤ b n √ 2 ˆ L g 0 ˜ L g 0 , we have η 3 k n ˆ L g 0 ˜ L g 0 b 2 -η k 2 n ≤ 0 , thus

Using the definition of E k and telescoping from k = 1 to K , we obtain

Noticing that L ( x , v ) is convex w.r.t. x , we have Gap v (ˆ x K , y ∗ ) ≤ ∑ K k =1 η k Gap v ( x k , y ∗ ) /H K , where ˆ x K = ∑ K k =1 η k x k /H K and H K = ∑ K k =1 η k , so we obtain

Further choosing v = y ˆ x K , we obtain

To analyze the individual gradient oracle complexity, we choose constant stepsizes η ≤ b n √ 2 ˆ L g 0 ˜ L g 0 and assume b < n without loss of generality, then Eq. (14) becomes

When ˆ L g 0 ∥ y ∗ ∥ 2 Λ -1 ≤ ( n -b ) 2 n σ 2 ∗ , we set η = min { b n √ 2 ˆ L g 0 ˜ L g 0 , ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n 2 ˆ L g 0 ˜ L g 0 K ∥ y ∗ ∥ 2 Λ -1 ) 1 / 3 } and consider the following two possible cases:

'Small K ' case: if η = b n √ 2 ˆ L g 0 ˜ L g 0 ≤ ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n 2 ˆ L g 0 ˜ L g 0 K ∥ y ∗ ∥ 2 Λ -1 ) 1 / 3 , we have

Combining these two cases, we have

Hence, to guarantee E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ for ϵ > 0 , the total number of required individual gradient evaluations will be

When ( n -b ) 2 n σ 2 ∗ ≤ ˆ L g 0 ∥ y ∗ ∥ 2 Λ -1 , we set η = min { b n √ 2 ˆ L g 0 ˜ L g 0 , ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n ( n -b ) 2 ˜ L g 0 Kσ 2 ∗ ) 1 / 3 } and consider the two cases as below:

'Small K ' case: if η = b n √ 2 ˆ L g 0 ˜ L g 0 ≤ ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n ( n -b ) 2 ˜ L g 0 Kσ 2 ∗ ) 1 / 3 , we have

'Large K ' case: if η = ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n ( n -b ) 2 ˜ L g 0 Kσ 2 ∗ ) 1 / 3 ≤ b n √ 2 ˆ L g 0 ˜ L g 0 , we have

Combining these two cases, we obtain

To guarantee E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ for ϵ > 0 , the total number of required individual gradient evaluations will be

Combining Eq. (15) and Eq. (16), we finally have

thus finishing the proof.

C Omitted Proofs for Smooth Convex Settting From Section 3

Before proceeding to the omitted proofs for the smooth convex settings in finite-sum with linear predictors, we first recall its primal-dual reformulation, then state the specialized version of a primaldual shuffled SGD algorithm in Algorithm 2. Recall that (PL) admits an explicit reformulation using convex conjugates of ℓ i :

where y i x = arg max y i ∈ R { y i a ⊤ i x -ℓ ∗ i ( y i ) } (different from the general smooth convex finite-sum settings in Section 2 and Appendix B). Further, for notational convenience, we assume that the partition is ordered, in the sense that for 1 ≤ j < j ′ ≤ m , max i ∈S j i < min i ′ ∈S j ′ i ′ . 2 We denote by y ( j ) the subvector of y ∈ R n indexed by the elements of S j , and by A ( j ) the submatrix obtained from A ∈ R n × d by selecting the rows indexed by S j .

Based on the formulation (PL-PD), we view shuffled SGD as a primal-dual method with block coordinate updates on the dual side, as summarized in Algorithm 2, for completeness. To see the equivalence, in i -th inner iteration of k -th epoch, we first update the i -th block y ( i ) k ∈ R b of the dual vector y k -1 ∈ R n based on x k -1 ,i as in Line 6. Since the dual update has a decomposable structure, this maximization step corresponds to computing the (sub)gradients { ℓ ′ π ( k ) j ( a ⊤ π ( k ) j x k -1 ,i ) } bi j = b ( i -1)+1

at x k -1 ,i for the batch of individual losses indexed by { π ( k ) j } bi j = b ( i -1)+1 . Then in Line 7, we perform a minimization step using y ( i ) k to compute x k -1 ,i +1 on the primal side. Combining these two steps, we have x k -1 ,i +1 = x k -1 ,i -η k b ∑ bi j = b ( i -1)+1 ℓ ′ π ( k ) j ( a ⊤ π ( k ) j x k -1 ,i ) a π ( k ) j , which is exactly the original primal shuffled SGD updating scheme .

Algorithm 2 Shuffled SGD (Primal-Dual View)

1: Input: Initial point x 0 ∈ R d , batch size b > 0 , step size { η k } > 0 , number of epochs K > 0

2: for k = 1 to K do

3: Generate any permutation π ( k ) of [ n ] (either deterministic or random)

4: x k -1 , 1 = x k -1

5: for i = 1 to m do

8: end for

10: end for

C.1 Omitted Proofs for the Random Reshuffling/Shuffle-Once Schemes

Lemma 9. Given { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 2 for k ∈ [ K ] , let E k := η k Gap v ( x k , y ∗ ) + b 2 n ∥ x ∗ -x k ∥ 2 2 -b 2 n ∥ x ∗ -x k -1 ∥ 2 2 . If Assumption 3 holds, then

Proof. By Line 6 in Alg. 2, we have y ( i ) k = arg max y ∈ R b { y ⊤ A ( i ) k x k -1 ,i -∑ b j =1 ℓ ∗ π ( k ) b ( i -1)+ j ( y j ) } for i ∈ [ m ] . Notice that since

is separable, we have y j k = arg max y ∈ R { y a ⊤ π ( k ) j x k -1 ,i -ℓ ∗ π ( k ) j ( y ) } for b ( i -1) + 1 ≤ j ≤ bi , thus a ⊤ π ( k ) j x k -1 ,i ∈ ∂ℓ ∗ π ( k ) j ( y j k ) . Since ℓ ∗ i is 1 L i -strongly convex by Assumption 3, then by Lemma 4 we obtain for b ( i -1) + 1 ≤ j ≤ bi

which leads to

Using the same argument for L ( x ∗ , y ∗ ) as a ⊤ j x ∗ ∈ ∂ℓ ∗ j ( y j ∗ ) for j ∈ [ n ] , we have

Adding and substracting the term b 2 nη k ∑ m i =1 ∥ x ∗ -x k -1 ,i ∥ 2 2 on the R.H.S. of Eq. (19), we obtain

By Line 7 of Alg. 2, we have x k -1 ,i +1 = arg min x ∈ R d { y ( i ) ⊤ k A ( i ) k x + b 2 η k ∥ x -x k -1 ,i ∥ 2 2 } . Further noticing that ϕ ( i ) k ( x ) := y ( i ) ⊤ k A ( i ) k x + b 2 η k ∥ x -x k -1 ,i ∥ 2 2 is b η k -strongly convex w.r.t. x and ∇ ϕ ( i ) k ( x k -1 ,i +1 ) = 0 , we have

which leads to

where we telescope from i = 1 to m for the term ∑ m i =1 ( ∥ x ∗ -x k -1 ,i +1 ∥ 2 2 -∥ x ∗ -x k -1 ,i ∥ 2 2 ) , and use the definitions that x k = x k -1 ,m +1 and x k -1 = x k -1 , 1 for ( i ) .

Combining the bounds from Eq. (18) and Eq. (20) and denoting

we obtain

thus completing the proof.

Lemma 10. For any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 in Algorithm 2 satisfy

Proof. By Line 7 in Alg. 2, we have A ( i ) ⊤ k y ( i ) k = b η k ( x k -1 ,i -x k -1 ,i +1 ) . Further noticing that x k -x k -1 ,i +1 = -∑ m j = i +1 ( x k -1 ,j -x k -1 ,j +1 ) , we obtain

thus completing the proof.

Lemma 11. Under Assumption 4, for any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 2 with uniformly random shuffling satisfy

Proof. By Line 7 in Alg. 2, we have x k -1 ,i -x k -1 ,i +1 = η k b A ( i ) ⊤ k y ( i ) k . Using the definition of I j ↑ for 0 ≤ j ≤ n -1 as in Section 1, we obtain

Also, we have A ( i ) ⊤ k ( v ( i ) k -y ( i ) k ) = A k I ( i ) ( v k -y k ) by the definition of I ( i ) in Section 3. Combining these two observations, we have

where we make a decomposition w.r.t. y ∗ ,k in ( i ) . For the first term in Eq. (21), we use Young's inequality for α > 0 and have

Expanding the squares and rearranging the terms in Eq. (23), we have

where we use Cauchy-Schwarz inequality for ( i ) . Using a similar argument, we also have

By the definitions of ˆ L π ( k ) and ˜ L π ( k ) , and choosing α = 2 η k ˜ L π ( k ) in Eq. (23), we obtain

For the second term in Eq. (22), we apply Young's inequality with β > 0 and proceed as above:

Noticing that ˜ L π ( k ) ≤ ˜ L , we choose β = 2 η k ˜ L and obtain

Combining Eq. (25) and Eq. (26), we have

We first assume the RR scheme. Taking conditional expectation w.r.t. the randomness up to but not including k -th epoch, we have

For the first term η 3 k ˜ L nb E k [ ∑ m i =1 ∥ A ⊤ k I b ( i -1) ↑ y ∗ ,k ∥ 2 2 ] , the only randomness is from the random permutation π ( k ) . In this case, each term A ⊤ k I b ( i -1) ↑ y ∗ ,k can be considered as a sum of a batch sampled without replacement from { y j ∗ a j } j ∈ [ n ] , while ∑ n j =1 y j ∗ a j = 0 as x ∗ is the minimizer, we then can use Lemma 5 and obtain

where ( i ) is due to the linearity of expectation, and we use our definition σ 2 ∗ = 1 n ∑ n j =1 ( y j ∗ ) 2 ∥ a j ∥ 2 2 = E j [ ∥ y j ∗ a j ∥ 2 2 ] for ( ii ) . Taking expectation w.r.t. all the randomness on both sides and using the law of total expectation, we obtain

For the SO scheme, since there is only one random permutation generated at the very beginning, we can take expectation w.r.t. all the randomness on both sides of (27), and the randomness for the term η 3 k ˜ L nb E [ ∑ m i =1 ∥ A ⊤ k I b ( i -1) ↑ y ∗ ,k ∥ 2 2 ] is only from the initial random permutation. So the above argument still applies to this case, and we complete the proof.

Theorem 2. Under Assumptions 3 and 4, if η k ≤ b n √ 2 ˆ L π ( k ) ˜ L π ( k ) and H K = ∑ K k =1 η k , then the output ˆ x K of Alg. 1 with uniformly random (RR/SO) shuffling satisfies

As a result, given ϵ > 0 , there exists a constant step size η k = η such that E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ after O ( n √ ˆ L ˜ L ∥ x 0 -x ∗ ∥ 2 2 ϵ + √ ( n -b )( n + b ) n ( n -1) √ n ˜ Lσ ∗ ∥ x 0 -x ∗ ∥ 2 2 ϵ 3 / 2 ) individual gradient queries.

Proof. Combining the bounds in Lemma 10 and 11 and plugging them into Eq. (17), we obtain

For the stepsize η k such that η k ≤ b n √ 2 ˆ L π ( k ) ˜ L π ( k ) , we have η 3 k n ˆ L π ( k ) ˜ L π ( k ) b 2 -η k 2 n ≤ 0 , thus

Noticing that E k = η k Gap v ( x k , y ∗ ) + b 2 n ∥ x ∗ -x k ∥ 2 2 -b 2 n ∥ x ∗ -x k -1 ∥ 2 2 and telescoping from k = 1 to K , we have

Noticing that L ( x , v ) is convex w.r.t. x , we have Gap v (ˆ x K , y ∗ ) ≤ ∑ K k =1 η k Gap v ( x k , y ∗ ) /H K , where ˆ x K = ∑ K k =1 η k x k /H K and H K = ∑ K k =1 η k , which leads to

Further choosing v = y ˆ x K , we obtain

To analyze the individual gradient oracle complexity, we choose constant stepsizes η ≤ b n √ 2 ˆ L ˜ L , then Eq. (28) will become

̸

Without loss of generality, we assume that b = n , otherwise the method and its analysis reduce to (full) gradient descent. We consider the following two cases:

Combining these two cases by setting η = min { b n √ 2 ˆ L ˜ L , ( 3 b 3 ( n -1) ∥ x 0 -x ∗ ∥ 2 2 n ( n -b )( n + b ) ˜ LKσ 2 ∗ ) 1 / 3 } , we obtain

Hence, to guarantee E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ for ϵ > 0 , the total number of individual gradient evaluations will be

as claimed.

C.2 Omitted Proofs for Incremental Gradient Descenet

We now provide the proof for convergence of IGD in the smooth convex settings. We first prove the following technical lemma, which bounds the inner product term T 2 := η k n ∑ m i =1 ( v ( i ) -y ( i ) k ) ⊤ A ( i ) ( x k -x k -1 ,i ) without random permutations involved.

Lemma 12. For any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 2 with fixed data ordering satisfy

Proof. Proceeding as in Lemma 11, we have

For both terms in Eq. (30) and Eq. (31), we use Young's inequality for α = 2 η k ˜ L 0 > 0 and proceed as in Eq. (24) to obtain

and

where again we used α = 2 η k ˜ L 0 . We then prove the term η 3 k ˜ L 0 nb ∑ m i =1 ∥ A ⊤ I b ( i -1) ↑ y ∗ ∥ 2 2 in Eq. (33) is no larger than the minimum of η 3 k n b 2 ˆ L 0 ˜ L 0 ∥ y ∗ ∥ 2 Λ -1 and η 3 k ( n -b ) 2 b 2 ˜ L 0 σ 2 ∗ . Note that when b = n , we have A ⊤ I (0) ↑ y ∗ = 0 , so this term disappears. When b < n , the former one can be derived as in Eq.(24), which gives

For the latter one, we notice that

by using the fact that ∑ n j =1 y j ∗ a j = 0 . Using Young's inequality, we have

By the definition that σ 2 ∗ = 1 n ∑ n j =1 ∥ y j ∗ a j ∥ 2 2 and ∑ ( m -1) b i = i ∥ y j ∗ a j ∥ 2 2 ≤ ∑ n j =1 ∥ y j ∗ a j ∥ 2 2 = nσ 2 ∗ , we obtain

Note that the bound in Eq. (34) equals to zero when b = n , which recovers the case of full gradient descent, so we have

Combining Eq. (32)-(35), we obtain

thus finishing the proof.

Theorem 6. Under Assumptions 3 and 4, if η k ≤ b n √ 2 ˆ L 0 ˜ L 0 and H K = ∑ K k =1 η k , the output ˆ x K of Alg. 2 with a fixed permutation satisfies

As a consequence, given ϵ > 0 , there exists a constant step size η k = η such that f (ˆ x K ) -f ( x ∗ ) ≤ ϵ after the number of gradient queries bounded by O ( n √ ˆ L 0 ˜ L 0 ∥ x 0 -x ∗ ∥ 2 2 ϵ + min { √ n ˆ L 0 ˜ L 0 ∥ y ∗ ∥ Λ -1 , ( n -b ) √ ˜ L 0 σ ∗ } ∥ x 0 -x ∗ ∥ 2 2 ϵ 3 / 2 ) .

Proof. Proceeding as in Lemmas 9 and 10, but without random permutations, we have

Using the bound in Lemma 12 and applying Eq. (29) into Eq. (36), we obtain

If η k ≤ b n √ 2 ˆ L 0 ˜ L 0 , we have η 3 k n ˆ L 0 ˜ L 0 b 2 -η k 2 n ≤ 0 , thus

Noticing that E k = η k Gap v ( x k , y ∗ ) + b 2 n ∥ x ∗ -x k ∥ 2 2 -b 2 n ∥ x ∗ -x k -1 ∥ 2 2 and telescoping from k = 1 to K , we have

Noticing that L ( x , v ) is convex w.r.t. x , we have Gap v (ˆ x K , y ∗ ) ≤ ∑ K k =1 η k Gap v ( x k , y ∗ ) /H K , where ˆ x K = ∑ K k =1 η k x k /H K and H K = ∑ K k =1 η k , so we obtain

Further choosing v = y ˆ x K , we obtain

To analyze the individual gradient oracle complexity, we choose constant stepsizes η ≤ b n √ 2 ˆ L 0 ˜ L 0 and assume b < n without loss of generality, then Eq. (37) becomes

When ˆ L 0 ∥ y ∗ ∥ 2 Λ -1 ≤ ( n -b ) 2 n σ 2 ∗ , we set η = min { b n √ 2 ˆ L 0 ˜ L 0 , ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n 2 ˆ L 0 ˜ L 0 K ∥ y ∗ ∥ 2 Λ -1 ) 1 / 3 } and consider the following two possible cases:

Combining these two cases, we have

Hence, to guarantee E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ for ϵ > 0 , the total number of individual gradient evaluations will be

When ( n -b ) 2 n σ 2 ∗ ≤ ˆ L 0 ∥ y ∗ ∥ 2 Λ -1 , we set η = min { b n √ 2 ˆ L 0 ˜ L 0 , ( b 3 ∥ x 0 -x ∗ ∥ 2 2 2 n ( n -b ) 2 ˜ L 0 Kσ 2 ∗ ) 1 / 3 } and consider the two cases as below:

Combining these two cases, we obtain

To guarantee E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ for ϵ > 0 , the total number of individual gradient evaluations will be

Combining Eq. (38) and Eq. (39), we finally have

thus finishing the proof.

D Omitted Proofs for Non-Smooth Convex Setting From Section 3

Before we prove Theorem 3 in convex Lipschitz settings, for completeness, we first recall the following standard first-order characterization of convexity.

Lemma 13. Let f : R d → R be a continuous convex function. Then, for any x , y ∈ R d :

where g x ∈ ∂f ( x ) , and ∂f ( x ) is the subdifferential of f at x .

The following technical lemma provides a primal-dual gap bound in convex nonsmooth settings.

Lemma 14. For any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 2 satisfy

where E k := η k ( L ( x k , v ) -L ( x ∗ , y ∗ ) ) + b 2 n ∥ x ∗ -x k ∥ 2 2 -b 2 n ∥ x ∗ -x k -1 ∥ 2 2 .

Proof. By the same argument as in the proof for Lemma 9, we know that a ⊤ π ( k ) j x k -1 ,i ∈ ∂ℓ ∗ π ( k ) j ( y j k ) for b ( i -1) + 1 ≤ j ≤ bi , then by Lemma 13 we have

which leads to

Using the same argument for L ( x ∗ , y ∗ ) as a ⊤ j x ∗ ∈ ∂ℓ ∗ j ( y j ∗ ) for j ∈ [ n ] , we have

Adding and substracting the term b 2 nη k ∑ m i =1 ∥ x ∗ -x k -1 ,i ∥ 2 2 on the R.H.S. of Eq. (42), we obtain

Denote ϕ ( i ) k ( x ) := y ( i ) ⊤ k A ( i ) k x + b 2 η k ∥ x -x k -1 ,i ∥ 2 2 , which is b η k -strongly convex w.r.t. x . Noticing that x k -1 ,i +1 = arg min x ∈ R d { y ( i ) ⊤ k A ( i ) k x + b 2 η k ∥ x -x k -1 ,i ∥ 2 } by Line 7 of Alg. 2, we have ∇ ϕ ( i ) k ( x k -1 ,i +1 ) = 0 , which leads to

Thus, we obtain

where ( i ) is by telescoping ∑ m i =1 ( ∥ x ∗ -x k -1 ,i +1 ∥ 2 2 -∥ x ∗ -x k -1 ,i ∥ 2 2 ) and using x k = x k -1 ,m +1 and x k -1 = x k -1 , 1 , which both hold by definition.

Combining the bounds from Eq. (41) and Eq. (43), and denoting

we finally obtain

thus completing the proof.

Note that we can still use Lemma 10 to bound the first inner product term in Eq. (40), as we are studying the same algorithm. The following lemma provides a bound on the second inner product term T 2 := η k n ∑ m i =1 ( v ( i ) k -y ( i ) k ) ⊤ A ( i ) k ( x k -x k -1 ,i ) in Eq. (40).

Lemma 15. Under Assumption 5, for any k ∈ [ K ] , the iterates { y ( i ) k } i m =1 and { x k -1 ,i } m +1 i =1 generated by Algorithm 2 satisfy

Proof. Proceeding as in Lemma 11, we have

Using Young's inequality for some α > 0 and proceeding as in Eq. (24), we obtain

where we use our definitions that ˆ G π ( k ) := 1 mn ∥ ∥ Γ 1 / 2 k ( ∑ m j =1 I b ( j -1) ↑ A k A ⊤ k I b ( j -1) ↑ ) Γ 1 / 2 k ∥ ∥ 2 and ˜ G π ( k ) := 1 b ∥ ∥ Γ 1 / 2 k ( ∑ m j =1 I ( j ) A k A ⊤ k I ( j ) ) Γ 1 / 2 k ∥ ∥ 2 . It remains to choose α = 2 b n √ ˜ G k ˆ G k to finish the proof.

We are now ready to prove Theorem 3 for the convergence of shuffled SGD in the convex nonsmooth Lipschitz settings.

Theorem 3. Under Assumption 5, if H K = ∑ K k =1 η k and ¯ G = E π [ √ ˆ G π ˜ G π ] , the output ˆ x K of Alg. 1 with possible uniformly random shuffling satisfies

As a result, for any ϵ > 0 , there exists a step size η k = η such that E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ after O ( n ¯ G ∥ x 0 -x ∗ ∥ 2 2 ϵ 2 ) individual gradient queries.

Proof. To simplify the presentation of our analysis, we first assume ∥ v ∥ 2 Γ -1 ≤ n , which will be later verified by our choice of v = y ˆ x K and Assumption 5.

Combining the bounds in Lemma 10 and 15 and plugging them into Eq. (40), we have

where we use Young's inequality for ∥ v k -y k ∥ 2 Γ -1 k and ∥ v k ∥ Γ -1 k = ∥ v ∥ 2 Γ -1 as v is a fixed vector for ( i ) , and ( ii ) is due to ∥ y k ∥ 2 Γ -1 k ≤ n by Assumption 5 and assuming that ∥ v ∥ 2 Γ -1 ≤ n . Proceeding as the proof for Theorem 2, we first assume the RR scheme and take conditional expectation w.r.t. the randomness up to but not including k -th epoch, then we obtain

Since the randomness only comes from the random permutation π ( k ) , we have

For notational convenience, we denote ¯ G = E π [ √ ˆ G π ˜ G π ] , and further take expectation w.r.t. all the randomness on both sides and use the law of total expectation to obtain

For the SO scheme, there is one random permutation π generated at the very beginning such that π ( k ) = π for all k ∈ [ K ] . So we can directly take expectation w.r.t. all the randomness on both sides of Eq. (45), with the randomness only from π , which leads to the same bound as Eq. (46) with E [ √ ˆ G π ( k ) ˜ G π ( k ) ] = E π [ √ ˆ G π ˜ G π ] . Note that for incremental gradient (IG) descent, we can let ¯ G = √ ˆ G 0 ˜ G 0 without randomness involved, where ˆ G 0 = ˆ G π (0) and ˜ G 0 = ˜ G π (0) w.r.t. the initial, fixed permutation π (0) of the data matrix A .

Noticing that E k = η k Gap v ( x k , y ∗ ) + b 2 n ∥ x ∗ -x k ∥ 2 2 -b 2 n ∥ x ∗ -x k -1 ∥ 2 2 and telescoping from k = 1 to K , we have

Noticing that L ( x , v ) is convex wrt x , we have Gap v (ˆ x K , y ∗ ) ≤ ∑ K k =1 η k Gap v ( x k , y ∗ ) /H K , where ˆ x K = ∑ K k =1 η k x k /H K and H K = ∑ K k =1 η k , so we obtain

Further choosing v = y ˆ x K , which also verifies ∥ v ∥ 2 Γ -1 = ∥ y ˆ x K ∥ 2 Γ -1 ≤ n by Assumption 5, we obtain

To analyze the individual gradient oracle complexity, we choose constant stepsize η . Then, the above bound becomes

Choosing η = b ∥ x 0 -x ∗ ∥ 2 2 n √ K ¯ G , we have

Hence, given ϵ > 0 , to ensure E [ f (ˆ x K ) -f ( x ∗ )] ≤ ϵ , the total number of individual gradient evaluations will be thus completing the proof.

Wenow briefly discuss this result. The total number of individual gradient queries is O ( n ¯ G ∥ x 0 -x ∗ ∥ 2 2 ϵ 2 ) , which appears independent of the batch size, but this is actually not the case, as the parameter ¯ G = E π [ √ ˆ G π ˜ G π ] depends on the block partitioning, due to Eq. (4). When b = n, as a sanity check, we recover the standard guarantee of (full) subgradient descent, which is expected, as in this case shuffled SGD reduces to subgradient descent. When b = 1 , however, the bound is worse than the corresponding bound for standard SGD, by a factor O ( n ¯ G/G 2 ) . By a similar sequence of inequalities as in Eq. (4), this factor is never worse than n, but it is typically much smaller, taking values as small as 1. We note that it is not known whether a better bound is possible for shuffled SGD in this setting, as the only seemingly tighter upper bound from [42] applies only for constant K , when n = Ω( 1 ϵ 2 ) , and under an additional boundedness assumption for the algorithm iterates.

E Experiment Details

We implement the computation of ˆ L and L max in Julia, a high-performance scientific computation programming language, and compute matrix operator norms using the default settings in the Julia Arpack Package. However, limited by computational memory and time constraint, our selection of datasets is focused on moderately large-scale datasets of n in the order of O (10 5 ) . We also include comparisons of small datasets such as a1a and sonar .

E.1 Evaluations of L max / ˜ L π on Synthetic Gaussian Datasets

We first study the gap between ˜ L π and L max for different batch sizes b , as shown in Figure 2. As in Section 4.1, we focus on their dependence on the data matrix, and assume that the loss functions ℓ i all have the same smoothness constant. In this case, the ratio L max / ˜ L π that characterizes the gap between

˜ L π and L max will become L max / ˜ L π = (max 1 ≤ i ≤ n {∥ a i ∥ 2 2 } ) / ( 1 b ∥ ∑ m j =1 I ( j ) A π A ⊤ π I ( j ) ∥ 2 ) . In particular, we run experiments on standard Gaussian data of size ( n, d ) . We fix the dimension d = 500 , and vary the number of samples with n = 100 , 500 , 1000 , 2000 . In Figure 2, we plot the ratio L max / ˜ L π versus the batch size b for 100 different random permutations π , where the dotted lines represent the mean values and the filled regions indicate the standard deviation of permutations. We observe that the ratio L max / ˜ L π is concentrated around its empirical mean and exhibits b α ( α ∈ [0 . 74 , 0 . 87] ) growth as the batch size b increases. In particular, if we choose b = √ n , the ratio can be O ( n 0 . 4 ) .

(a)

n

= 100

(b)

n

= 500

(c)

n

= 1000

(d)

n

= 2000

50

40

~L

30

7

20

10

0t

0

20

40

60

80

100

batch size b

140

120

100

~L

80

L

60

40

20

0

0

100

200

300

400

500

batch size b

200

150

~L

100

L

50

0

0

200

400

600

800

1000

batch size b

250

200

~L

150

L

100

50

0

0

batch size b

E.2 Distributions of L max / ˆ L π

In this subsection, we include histograms in Figure 3 to illustrate the spread of L max / ˆ L π with respect to random permutations, for completeness. We observe that in all the examples L max / ˆ L π is concentrated around its empirical mean. The following plots are normalized, with y-axis representing the empirical probability density. The x-axis represents L max / ˆ L π .

a1a

a9a

BBBC005

BBBC010

CIFAR10

duke

e2006train

gisette

leu

MNIST

rcv1

real-sim

sonar

tmc2007

15.0

80

12.5

10.0

60

3

7.5

40

2

5.0

2.5

20

1

0.0

5.450 5.475 5.500 5.525 5.550 5.575

0

5.485

5.490

5.495

55.500

5.505

18.2

18.3

18.4

18.5

1.0

40

0.06

0.8

0.05

30

0.6

0.04

20

0.03

0.4

0.02

0.2

10

0.01

0.0

6.0

6.5

7.0

7.5

8.0

8.5

0

10.01 10.02 10.03 10.04 10.05 10.06

0.00

30

40

50

09

70

0.06

80

120

0.05

100

60

0.04

80

0.03

40

60

40

0.02

20

20

0.01

0

5.340

5.345

5.3505.3555.360

0

3.510

3.515

3.520

3.525

0.00

40

50

15.0

5

0.8

12.5

10.0

0.6

3

7.5

0.4

5.0

2

0.2

2.5

1

0.0

19.02519.05019.07519.10019.12519.15019.175

0

41.9

42.0

42.1

42.2

42.3

42.4

0.0

1.4

1.2

2.0

25

1.0

20

1.5

0.8

15

0.6

1.0

10

0.4

0.2

0.5

5

0.0

193.00193.25193.50193.75194.00194.25

0.0

5.8

6.0

6.2

6.4

6.6

6.8

0

10.84

10.86

10.88

10.90

10.92

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly state the scope of our work and contributions, see Section 1.2.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, see detailed discussion in the introdcution.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state our assumptions in sections 2 and 3, and the proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fully disclose the public datasets and tools we use for the numerical computations in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We list the details of our experiments in Section 4.1 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We use various ways, including ribbon plots and histograms, to illustrate the variance of our numerically computed values.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We list the details of all computational tools in Section 4.1 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This is a primarily theoretical work and we conform to the rules with NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use public benchmarking datasets from LIBSVM [15], MNIST [17], CIFAR10 [22], and Broad Bioimage Benchmark Collection [28], and have properly cited and credited the asset's creators.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at improving existing bounds on random reshuffling. While lower bounds are tight in the worst case, refined smoothness definitions allow to take a larger step-size in favorable cases, which in turns allows faster convergence. These smoothness definitions involve a supremum over permutation-dependent partial averaging of individual smoothness constants. \n \nThe dependence on these refined smoothness constants is obtained through a primal-dual interpretation of SGD, though the convergence analysis does not consist in applying existing primal-dual analyses to this formulation. As a side note, results include mini-batching, which is nice to have. \n\nFor generalized linear models, individual covariances can be averaged instead of individual smoothness constants, thus allowing to gain even more especially in high-dimensions. A non-smooth extension is given, performing similar averagings over individual Lipschitz constants. \n\nTypos:\nLine 27: why does SGD rely on a standard basis vector?? This looks like a mix between SGD and coordinate descent. \nL45: we choose take \nAlgo 1: shouldn't this be a $\\nabla f_i$ on line 7?\n\nSTRENGTHS:\n- Nice use of the finite-sum assuption, which allows to use a primal-dual reformulation.\n- Improves over existing bounds for shuffled SGD, mimicking cyclic coordinate descent results.\n\nWEAKNESSES:\n- $\\tilde{L}$, the $L_{max}$-like constant, is the one that appears in Theorems 1 and 2. In particular, the more \"average smoothness\" constant (from which most of the improvement seems to come) only affect higher-order terms by using larger step-sizes to be used, but does not change anything for small step-sizes. \n- While results are strong for generalized linear models (taking into account the actual spectrum of each Hessian), general results are quite looser (though they still improve on existing ones).\n\nQUESTIONS:\n1) In the end, everything seems to depend on primal quantities. What makes an actual difference in the analysis that could not have been done in the primal (or why was it more intuitive to do it in the primal)? Why can't the full analyses be done by replacing the $y^i_k with $\\nabla f_i(x_{k,i})$?\n\nIn particular, what technical challenges do you expect moving to the non-convex setting? In the end all quantities can, but the primal-dual interpretation of SGD does not hold anymore since biconjugacy is used at some point if I'm not mistaken).\n\n2) What are the connections with existing (primal-dual) CD analyses? I see that the primal-dual reformulation allows to study a setting that is close (although different since CD would not get any residual variance). It is said in Appendix A that results are technically disjoints, so then why is this inspiration helpful still? \n\n3) It seems that the discussion from Section 4 only applies to the generalized linear models case, because it does not seem possible to benefit from structure in the general case. Do you have any intuition for why such assumptions cannot be made using Hessians, and thus benefit from similar improvements? Additionally, how would Table 2 look with these bounds instead of the generalized linear models ones?\n\n4) Do you actually expect to depend on the smoothness for the average permutation (instead of worst-case)? Is this complicated to analyze directly because of the dependence between iterates within a single epoch?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

comments: SUMMARY:
This paper studied the convergence of random shuffled SGD through the lens of dual coordinate descent. By leveraging the analysis in coordinate descent, the author(s) derived a rate that is $O(\sqrt{n})$ faster than existing rate.

STRENGTHS:
Pros:
-  The paper is well written and easy to follow, contributions are clearly stated (Table 1).
-  The rate derived by the author(s) is better than previous state-of-the-art.

WEAKNESSES:
- The primal-dual relationship between SGD and coordinate descent has a quite long history, e.g. [1]. The author(s) should mention these works in the related work section or section 2.

[1] Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization, Shai Shalev-Shwartz and Tong Zhang, 2013.

QUESTIONS:
The convergence analysis of coordinate descent usually have to assume the objective is smooth, which means the primal objective is strongly convex. But the strong convexity assumption was not used in this paper, I wonder how does the author(s) circumvent this assumption?

LIMITATIONS:
Limitations are not well-addressed. I do not see any potential negative impact of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper considers the primal-dual aspect of SGD using sampling without replacement (shuffled SGD). The authors present results for shuffled SGD on smooth and non-smooth convex settings with tight convergence bounds for several shuffling schemes (IG, SO, and RR). In some specific settings, the convergence rate can be improved with a factor of $\sqrt{n}$. The authors perform experiments to demonstrate that their bound is tight and better than prior work.

STRENGTHS:
The theoretical results seem to be solid, although I did not check the proofs. The authors choose to investigate a different type of smoothness constant compared to the prior work, thus lead to improvement in the bound. In addition, they study the primal-dual aspect for linear predictors and then extend the results to non-smooth setting.

WEAKNESSES:
The improvement in the convergence is not significant as it is only changing the Lipschitz constant. 
The authors should be more transparent about the setting where there is $\sqrt{n}$ improvement as it seems like this only applies to linear predictors (the claim in abstract, line 14-17 is not clear). The authors should include that context in your statements. Another weakness is that the bounds in Section 3 depend on the data.

QUESTIONS:
Why do you call RR and SO uniformly random shuffling? 

How the results in section 3 reduces and compares to general convex finite-sum problems when $a_i$ are all-1 vectors i.e. loss function applies on $x$?

I have read the author responses.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on SGD with shuffling for finite-sum minimization problems. While there exists tight convergence upper and lower bounds for SGD assuming a global smoothness constant $L_{\\max}$, the proposed results give a more fine-grained analysis in terms of the component-wise smoothness constants $L_i$ to show possibly faster convergence rates using Fenchel dual arguments. By replacing the $L_{\\max}$’s with smaller constants $\\tilde{L}, \\hat{L}$, the new results can improve the iteration complexity by up to a factor of $L_{\\max}/\\hat{L} = O(n)$ as demonstrated in the linear predictor example. This work also extends to non-smooth, Lipschitz objectives in the case of linear predictors, and provides empirical evidence supporting the claims.\n\nSTRENGTHS:\n- The idea of the ‘primal-dual’ formulation, which essentially allows one to exploit similar-to-coordinate-descent techniques for better dependencies on the $L_i$’s, is new to me and seems to be a potentially useful technique. I agree that using component-wise structures for improved convergence rates could be one of the many meaningful future directions for the shuffling community.\n- The paper presents results that consider general types of shuffling-based algorithms (RR, SO, IG) which I appreciate.\n\nWEAKNESSES:\n- Focusing on the results, the only big difference with previous work would be the constant $L_{\\max}$ being replaced with $\\tilde{L}, \\hat{L}$, which means that the improvements are directly linked with how small the ratios $\\tilde{L}/L_{\\max}, \\hat{L}/L_{\\max}$ could actually be. While Table 2 suggests that $L_{\\max}$ could be quite large (for the linear predictor case), it is quite hard to see how ‘small’ the values $\\tilde{L}, \\hat{L}$ are in first glance due to the relatively complicated definitions. The paper explains the gap for Gaussian data, but I think this example is a bit too ‘specific’.\n- Moreover, based on my understanding, the definitions of $\\tilde{L}, \\hat{L}$ for the general convex case and the linear predictor case seem to be a bit different. It is still unclear whether these fine-grained bounds could imply a significantly *faster* convergence rate, especially for the general convex case.\n- Minor Stuff: I might have missed this, but it seems that the definition of $\\boldsymbol{A}_{\\pi}$ (which I think is $\\\\boldsymbol{A}\\_{\\\\pi}  = [a\\_{\\pi_1} \\cdots a\\_{\\pi_n}]^{\\top}$) is missing in Section 3.1. Also, the proofs are all written for batch size $b$, might be confusing to readers: maybe it would have been better if the main body also contained statements including $b$.\n- TYPO: Line 9 of Algorithm 1, $m$ → $n$\n- TYPO: Line 473 mini-bath → mini-batch\n\nQUESTIONS:\n- Is there a way to intuitively understand when or why $\\tilde{L}/L_{\\max}, \\hat{L}/L_{\\max}$ could be ‘much’ smaller than $1$ (with possible $n^{\\alpha}$-ish dependencies), both for the  Or are there at least any examples apart from the linear predictor that demonstrate significantly small $\\tilde{L}/L_{\\max}, \\hat{L}/L_{\\max}$?\n- Based on my knowledge, in coordinate descent algorithms it is common to use *different* step sizes for different coordinates, say $\\eta_i = 1/L_i$, to improve dependencies on the $L$’s. Would it be possible to do something similar for this case if we take the primal-dual approach?\n- Can you elaborate a bit more on why the inequality $(iii)$ of $(4)$ (in Section 4) is loose by a factor of $n$ in most cases? My understanding of the linear predictor case is that the norm-trace gap corresponds to the largest VS sum of singular values, and hence if the singular values are even, then the gap can be as loose as a factor of $n$, while if we have a single large $\\lambda_{\\max}$ then the inequality can be tight. I wonder if (i) this interpretation is correct, and (ii) whether something similar could hold for the $\\tilde{L}^g, \\hat{L}^g$’s in the general convex case.\n- Minor question: Is it true Algorithm 1 is completely equivalent to the standard SGD with random reshuffling (without the $\\boldsymbol{y}$’s), and the dual variables are just there for illustrative purposes?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations, and there seems to be no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}