Title: Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributed optimization

convex optimization

second-order similarity

client sampling
Abstract: We study finite-sum distributed optimization problems involving a master node and $n-1$ local nodes under the popular $\delta$-similarity and $\mu$-strong convexity conditions. We propose two new algorithms, SVRS and AccSVRS, motivated by previous works. The non-accelerated SVRS method combines the techniques of gradient sliding and variance reduction and achieves a better communication complexity of $\tilde{\mathcal{O}}(n {+} \sqrt{n}\delta/\mu)$ compared to existing non-accelerated algorithms. Applying the framework proposed in Katyusha X, we also develop a directly accelerated version named AccSVRS with the $\tilde{\mathcal{O}}(n {+} n^{3/4}\sqrt{\delta/\mu})$ communication complexity. In contrast to existing results, our complexity bounds are entirely smoothness-free and exhibit superiority in ill-conditioned cases. Furthermore, we establish a nearly matched lower bound to verify the tightness of our AccSVRS method.

Stochastic Distributed Optimization under Average Second-order Similarity: Algorithms and Analysis

Abstract

We study finite-sum distributed optimization problems involving a master node and n -1 local nodes under the popular δ -similarity and µ -strong convexity conditions. We propose two new algorithms, SVRS and AccSVRS, motivated by previous works. The non-accelerated SVRS method combines the techniques of gradient sliding and variance reduction and achieves a better communication complexity of ˜ O ( n + √ nδ/µ ) compared to existing non-accelerated algorithms. Applying the framework proposed in Katyusha X [6], we also develop a directly accelerated version named AccSVRS with the ˜ O ( n + n 3 / 4 √ δ/µ ) communication complexity. In contrast to existing results, our complexity bounds are entirely smoothness-free and exhibit superiority in ill-conditioned cases. Furthermore, we establish a nearly matched lower bound to verify the tightness of our AccSVRS method.

1 Introduction

We have witnessed the development of distributed optimization in recent years. Distributed optimization aims to cooperatively solve a learning task over a predefined social network by exchanging information exclusively with immediate neighbors. This class of problems has found extensive applications in various fields, including machine learning, healthcare, network information processing, telecommunications, manufacturing, natural language processing tasks, and multi-agent control [54, 30, 48, 45, 60, 8]. In this paper, we focus on the following classical finite-sum optimization problem in a centralized setting:

where each f i is differentiable and corresponds to a client or node, and the target objective is their average function f . Without loss of generality, we assume f 1 is the master node and the others are local nodes. In each round, every local node can communicate with the master node certain information, such as the local parameter x , local gradient ∇ f i ( x ) , and some global information gathered at the master node. Such a scheme can also be viewed as decentralized optimization over a star network [55].

Following the wisdom of statistical similarity residing in the data at different nodes, many previous works study scenarios where the individual functions exhibit relationships or, more specifically, certain homogeneity shared among the local f i 's and f . The most common one is under the δ -second-order

similarity assumption [33, 35], that is,

Such an assumption also has different names in the literature, such as δ -related assumption, bounded Hessian dissimilarity, or function similarity [7, 32, 51, 54, 62]. The rigorous definitions are deferred to Section 2. Moreover, the second-order similarity assumption can hold with a relatively small δ compared to the smoothness coefficient of f i 's in many practical settings, such as statistical learning. More insights on this can be found in the discussion presented in [54, Section 2]. The similarity assumption indicates that the data across different clients share common information on the secondorder derivative, potentially leading to a reduction in communication among clients. Meanwhile, the cost of communication is often much higher than that of local computation in distributed optimization settings [9, 44, 30]. Hence, researchers are motivated to develop efficient algorithms characterized by low communication complexity, which is the primary objective of this paper as well.

Furthermore, we need to emphasize that prior research [25, 56, 63, 19, 43, 5] has shown tightly matched lower and upper bounds on computation complexity for the finite-sum objective in Eq. (1). These works focus on gradient complexity under (average) smoothness [63] instead of communication complexity under similarity. Indeed, we will also discuss and compare the gradient complexity as shown in [35], to explore the trade-off between communication and gradient complexity.

Although the development of distributed optimization with similarity has lasted for years, the optimal complexity under full participation was only recently achieved by Kovalev et al. [35]. They employed gradient-sliding [37] and obtained the optimal communication complexity ˜ O ( n √ δ/µ ) for µ -strongly convex f and δ -related f i 's in Eq. (1). However, the full participation model requires the calculation of the whole gradient ∇ f ( · ) , which incurs a communication cost of n -1 in each round. In contrast, partial participation could reduce the communication burden and yield improved complexity. Hence, Khaled and Jin [33] introduced client sampling, a technique that selects one client for updating in each round. They developed a non-accelerated algorithm SVRP, which achieves the communication complexity of ˜ O ( n + δ 2 /µ 2 ) . Additionally, they proposed a Catalyzed version of SVRP with the complexity ˜ O ( n + n 3 / 4 √ δ/µ ) , which is better than the rates obtained in the full participation setting.

We believe there are several potential avenues for improvement inspired by [33]. 1) Khaled and Jin [33] introduced the requirement that each individual function is strongly convex (see [33, Assumption 2]). However, this constraint is absent in prior works. Notably, in the context of full participation, even non-convexity is deemed acceptable 6 . Aprominent example is the shift-and-invert approach to solving PCA [52, 23], where each component is smooth and non-convex, but the average function remains convex. Thus we doubt the necessity of requiring strong convexity for individual components. 2) In hindsight, it seems that the directly accelerated SVRP could only achieve a bound of ˜ O ( n + √ n · δ/µ ) based on the current analysis, which is far from being satisfactory compared to its Catalyzed version. Consequently, there might be room for the development of a more effective algorithm for direct acceleration. 3) It is essential to note that the Catalyst framework introduces an additional log term in the overall complexity, along with the challenge of parameter tuning. This aspect is discussed in detail in [6, Section 1.2]. Therefore, we intend to address the aforementioned concerns, particularly on designing directly accelerated methods under the second-order similarity assumption.

1.1 Main Contributions

In this paper, we address the above concerns under the average similarity condition. Our contributions are presented in detail below and we provide a comparison with previous works in Table 1:

First, we combine gradient sliding and client sampling techniques to develop an improved non-accelerated algorithm named SVRS (Algorithms 1). SVRS achieves a communication complexity of ˜ O ( n + √ n · δ/µ ) , surpassing SVRP in ill-conditioned cases. Notably, this rate does not need component strong convexity and applies to the function value gap instead of the parameter distance.

Second, building on SVRS, we employ a classical interpolation framework motivated by Katyusha X [6] to introduce the directly accelerated SVRS (AccSVRS, Algorithm 2).

AccSVRS achieves the same communication bound of ˜ O ( n + n 3 / 4 √ δ/µ ) as Catalyzed SVRP. Specifically, our bound is entirely smoothness-free and slightly outperforms Catalyzed SVRP, featuring a log improvement and not requiring component strong convexity.

Third, by considering the proximal incremental first-order oracle in the centralized distributed framework, we establish a lower bound, which nearly matches the upper bound of AccSVRS in ill-conditioned cases.

1.2 Related Work

Gradient sliding/Oracle Complexity Separation. For optimization problems with a separated structure or multiple building blocks, such as Eq. (1), there are scenarios where computing the gradients/values of some parts (or the whole) is more expensive than the others (or a partial one). In response to this challenge, techniques such as the gradient-sliding method [37] and the concept of oracle complexity separation [28] have emerged. These methods advocate for the infrequent use of more expensive oracles compared to their less resource-intensive counterparts. This strategy has found applications in zero-order [12, 21, 28, 53], first-order [37-39, 31] and high-order methods [31, 24, 3], as well as in addressing saddle point problems [4, 13]. Our algorithms can be viewed as a variance-reduced version of gradient sliding tailored to leverage the similarity assumption.

Distributed optimization under similarity. Distributed optimization has a long history with a plethora of existing works and surveys. To streamline our discussion, we only list the most relevant references, particularly under the similarity and strong convexity assumptions. In the full participation setting, which involves deterministic methods, the first algorithm credits to DANE [51], though its analysis is limited to quadratic objectives. Subsequently, AIDE [50], DANE-LS and DANE-HB [58] improved the rates for quadratic objective; Disco [62] SPAG [27], ACN [1] and DiRegINA [18] improved the rates for self-concordant objectives. As for general strongly convex objectives, Sun et al. [54] introduced the SONATA algorithm, and Tian et al. [55] proposed accelerated SONATA. However, their complexity bounds include additional log factors. These factors have recently been removed by Accelerated Extragradient [35], whose complexity bound perfectly matches the lower bound in [7]. We highly recommend the comparison of rates in [35, Table 1] for a comprehensive overview. Once the discussion of deterministic methods is concluded, Khaled and Jin [33] shifted their focus to stochastic methods using client sampling. They proposed SVRP and its Catalyzed version, both of which exhibited superior rates compared to deterministic methods.

2 Preliminaries

̸

Notation. We denote vectors by lowercase bold letters (e.g., w , x ), and matrices by capital bold letters (e.g., A , B ). We let ∥ · ∥ be the ℓ 2 -norm for vectors, or induced ℓ 2 -norm for a given matrix: ∥ A ∥ = sup u =0 ∥ Au ∥ / ∥ u ∥ . We abbreviate [ n ] = { 1 , . . . , n } and I d ∈ R d × d is the identity matrix. We use 0 for the all-zero vector/matrix, whose size will be specified by a subscript, if necessary, and otherwise is clear from the context. We denote Unif( S ) as the uniform distribution over set S . We say T ∼ Geom( p ) for p ∈ (0 , 1] if P ( T = k ) = (1 -p ) k -1 p, ∀ k ∈ { 1 , 2 , . . . } , i.e., T obeys a geometric distribution. We adopt E k as the expectation for all randomness appeared in step k , and ✶ A as the indicator function on event A , i.e., ✶ A = 1 if event A holds, and 0 otherwise. We use O ( · ) , Ω( · ) , Θ( · ) and ˜ O ( · ) notation to hide universal constants and log-factors. We define the Bregman divergence induced by a differentiable (convex) function h : R d → R as D h ( x , y ) := h ( x ) -h ( y ) -⟨∇ h ( y ) , x -y ⟩ .

Definitions. We present the following common definitions used in this paper.

Definition 2.1 A differentiable function g : R d → R is µ -strongly convex (SC) if

Particularly, if µ = 0 , we say that g is convex.

Definition 2.2 A differentiable function g : R d → R is L -smooth if

There are many basic inequalities involving strong convexity and smoothness, see [22, Appendix A.1] for an introduction. Next, we present the definition of second-order similarity in distributed optimization.

Definition 2.3 The differentiable functions f i 's satisfy δ -average second-order similarity (AveSS) if the following inequality holds for f i 's and f = 1 n ∑ n i =1 f i :

Definition 2.4 The differentiable functions f i 's satisfy δ -component second-order similarity (SS) if the following inequality holds for f i 's and f = 1 n ∑ n i =1 f i :

Definitions 2.3 and 2.4 first appear in [33], which is an analogy to (average) smoothness in prior literature [63]. Particularly, f i 's satisfy δ -AveSS implies that ( f -f i ) 's satisfy δ -average smoothness, while f i 's satisfy δ -SS implies that ( f -f i ) 's satisfy δ -smoothness. Additionally, many researchers [32, 7, 51, 62, 54, 35] use the equivalent one defined by Hessian similarity (HS) if assuming that f i 's are twice differentiable. Thus we also list them below and leave the derivation in Appendix B.

Since our algorithm is a first-order method, we adopt the gradient description of similarity (Definitions 2.3 and 2.4) without assuming twice differentiability for brevity.

As mentioned in [7, 54], if f i 's satisfy δ -AveSS (or SS), and f is µ -strongly convex and L -smooth, then generally L ≫ δ ≫ µ > 0 for large datasets in practice. Therefore, researchers aim to develop algorithms that achieve communication complexity solely related to δ, µ (or log terms of L ). This is also our objective. To finish this section, we will clarify several straightforward yet essential propositions, and the proofs are deferred to Appendix A.

Proposition 2.5 We have the following properties among SS, AveSS, and SC: 1) δ -SS implies δ -AveSS, but δ -AveSS only implies √ nδ -SS. 2) If f i 's satisfy δ -SS and f is µ -strongly convex, then for all i ∈ [ n ] , f i ( · ) + δ -µ 2 ∥·∥ 2 is convex, i.e., f i is ( δ -µ ) -almost convex [14].

3 Algorithm and Theory

In this section, we introduce our main algorithms, which are developed to solve the distributed optimization problem in Eq. (1) under Assumption 1 below:

Assumption 1 We assume that f i 's satisfy δ -AveSS, and f is µ -strongly convex with δ ≥ µ > 0 .

Assumption 1 does not need each f i to be µ -strongly convex. In fact, it is acceptable that f i 's are non-convex, since by Proposition 2.5, f i 's are ( √ nδ -µ ) -almost convex [14]. In the following, we first propose our new algorithm SVRS, which combines the techniques of gradient sliding and variance reduction, resulting in improved rates. Then we establish the directly accelerated method motivated by [6].

3.1 No Acceleration Version: SVRS

We first show the one-epoch Stochastic Variance-Reduced Sliding ( SVRS 1ep ) method in Algorithm 1. Before delving into the theoretical analysis, we present some key insights into our method. These insights aim to enhance comprehension and facilitate connections with other algorithms.

Variance Reduction. Our algorithm can be viewed as adding variance reduction from [35]. Besides the acceleration step, the main difference lies in the proximal step, where Kovalev et al. [35] solved:

To save the heavy communication burden of calculating ∇ f ( x t ) , we apply client sampling by selecting a random ∇ f i t ( x t ) in the t -th step. However, this substitution introduces significant noise. To mitigate this, we incorporate a correction term g t = ∇ f i t ( w 0 ) -∇ f ( w 0 ) from previous wisdom [29] to reduce the variance.

Gradient sliding. Our algorithm can be viewed as adding gradient sliding from SVRP [33]. The main difference also lies in the proximal point problem, where Khaled and Jin [33] solved:

Here we adopt a fixed proximal function f 1 instead of f i t , which can be viewed as approximating f i t ( x ) ≈ f 1 ( x )+[ f i t -f 1 ]( x t )+ ⟨∇ [ f i t -f 1 ]( x t ) , x -x t ⟩ + 1 2 θ ′ ∥ x -x t ∥ 2 with a properly chosen θ ′ > 0 . Such a modification is motivated by [35], where they reformulated the objective as f ( x ) = [ f ( x ) -f 1 ( x )] + f 1 ( x ) . Thus they could employ gradient sliding to skip heavy computations of ∇ [ f -f 1 ]( x ) by utilizing the easy computations of ∇ f 1 ( x ) more times. Fixing the proximal function f 1 leads to the same metric space owned by f 1 in each step, which could benefit the analysis and alleviate the requirements on f i 's compared to SVRP. Indeed, in our setting f 1 can be replaced by any other fixed client f b , b ∈ [ n ] . In this case, the master node would be f b instead of f 1 .

Bregman-SVRG. Our algorithm can be viewed as the classical Bregman-SVRG [20] with the reference function f 1 ( · ) + 1 2 θ ∥·∥ 2 after introducing the Bregman divergence:

We need to emphasize that the proof of Bregman-SVRG requires additional structural assumptions [20, Assumpotion 3], which is not directly applicable in our setting. Hence, the rigorous proof of Bregman-SVRG under our similarity assumption is still meaningful as far as we are concerned.

3.1.1 Communication Complexity under Distributed Settings

When applied to the distributed system, the communication complexity of SVRS 1ep can be described as follows: At the beginning of each epoch, the master (corresponding to f 1 ) sends w 0 to all clients. Each client computes ∇ f i ( w 0 ) from its local data and sends it back to the master. The master then builds ∇ f ( w 0 ) after collecting all ∇ f i ( w 0 ) 's. The communication complexity is 2( n -1) in this

Algorithm 1 SVRS 1ep ( f, w 0 , θ, p )

1: Input: w 0 ∈ R d , p ∈ (0 , 1) , θ > 0

2: Initialize x 0 = w 0 , compute ∇ f ( w 0 ) , and set T ∼ Geom( p )

3: for t = 0 , 1 , 2 , . . . , T -1 do

4: Sample i t ∼ Unif ([ n ]) and compute g t = ∇ f i t ( w 0 ) -∇ f ( w 0 )

5: Approximately solve the local proximal point problem:

6: end for

7: Output: x T

case. Next, the algorithm enters into the loop iterations. In each iteration, the master only sends current x t to the chosen client i t . The i t -th client computes ∇ f i t ( x t ) and sends it to the master (the first client). Then the master solves (inexactly) the local problem (Line 5 in Algorithm 1) to get an inexact solution x t +1 . The communication complexity is 2 in this case. Thus, the total communication complexity of SVRS 1ep is 2( n -1) + 2 T . Note that E T = 1 /p and generally p = 1 /n . We obtain that one epoch communication complexity is 4 n -2 in expectation.

We would like to emphasize that our setup differs from that in [41, 46], where the authors assume the nodes can perform calculations and transmit vectors in parallel. We recognize the significance of both setups. However, there are situations where communication is more expensive than computation. For instance, in a business network or communication network the communication between any two nodes can result in charges and the risk of information leakage. To mitigate these costs, we should reduce the frequency of communication. Thus, we focus on the nonparallel setting.

3.1.2 Convergence Analysis of SVRS

Based on the one-epoch method SVRS 1ep , we could introduce our non-accelerated algorithm SVRS, which starts from w 0 ∈ R d and repeatedly performs the update 7

Now we derive the convergence rate of SVRS 8 . The main technique we apply is replacing the Euclidean distance with the Bergman divergence. Denote the reference function

By Assumption 1 and 1) in Proposition 2.5, we see that f i 's are √ nδ -SS. i.e., [ f 1 -f ]( · ) is ( √ nδ ) -smooth. Thus, h ( · ) is ( 1 θ - √ nδ ) -strongly convex and ( 1 θ + √ nδ ) -smooth if θ < 1 √ nδ , that is,

Hence, if √ nθδ = Θ(1) , h ( · ) is nearly a rescaled Euclidean norm since its condition number related to ∥·∥ is 1+ √ nθδ 1 - √ nθδ = Θ(1) . Next, we employ the properties of the Bregman divergence D h ( · , · ) to build the one-epoch progress of SVRS 1ep as shown below:

Lemma 3.1 Suppose Assumption 1 holds. Let w + = SVRS 1ep ( f, w 0 , θ, p ) with θ = 1 / (4 √ nδ ) , and the approximated solution x t +1 satisfies

Algorithm 2 Accelerated SVRS (AccSVRS)

1: Input: z 0 = y 0 ∈ R d , p, τ ∈ (0 , 1) , α, θ > 0 , K ∈ { 1 , 2 , . . . }

2: for k = 0 , 1 , 2 , . . . , K -1 do

3: x k +1 = τ z k +(1 -τ ) y k

4: y k +1 =SVRS 1ep ( f, x k +1 , θ, p )

5:

1

p

[

f

∇

-

G

k

+1

6:

k

+1

z

=

= arg min

7: end for

8: Output: y K

Then for all x ∈ R d that is independent of the indices i 1 , i 2 , . . . , i T in SVRS 1ep ( f, w 0 , θ, p ) , we have

Remark 3.2 We note that some papers [10, 11] assume the smoothness and convexity of component functions, and adopt local updates for solving the proximal step. However, we replace these assumptions with a proximal approximately solvable assumption (10) , which could even cover some nonsmooth and non-convex but proximal trackable component functions. We regard our assumption as more essential since the local updates can be viewed as partially solving this proximal step.

The proof of Lemma 3.1 is left in Appendix D.1. From Lemma 3.1, we find a well-behaved proximal operator is sufficient to ensure favorable progress. Finally, we establish the convergence rate and communication complexity of the SVRS method, and the proof is deferred to Appendix D.2.

Theorem 3.3 Suppose Assumption 1 holds. If in SVRS 1ep (Algorithm 1), the hyperparameters are set as θ = 1 / (4 √ nδ ) , p = 1 /n , and the approximate solution x t +1 in each proximal step satisfies Eq. (10) . Then for any error ε > 0 , when

i.e., after ˜ O ( n + √ nδ/µ ) communications in expectation, we obtain that E f ( w k ) -f ( x ∗ ) ≤ ε .

Remark 3.4 Our results enjoy the following advantages over SVRP [33]: The convergence of SVRP ([33, Theorem 2]) only applied to E ∥ w k -x ∗ ∥ 2 , which can also be derived by our results from strong convexity: f ( w k ) -f ( x ∗ ) ≥ µ 2 ∥ w k -x ∗ ∥ 2 . However, the reverse is not applicable since we do not assume the smoothness of f , or indeed the smoothness coefficient is very large. Moreover, for ill-conditioned problems (e.g., δ/µ ≫ √ n ), our step size 1 / (4 √ nδ ) is much larger than µ/ (2 δ 2 ) used in SVRP, and the convergence rate is also faster than SVRP: ˜ O ( n + √ nδ/µ ) vs. ˜ O ( n + δ 2 /µ 2 ) . Finally, we do not need the strong convexity assumption of component functions.

3.2 Acceleration Version: AccSVRS

Now we apply the classical interpolation technique motivated by Katyusha X [6] to establish accelerated SVRS (AccSVRS, Algorithm 2). The main difference between AccSVRS and Katyusha X is due to the different choices of distance spaces. Specifically, we adopt D h ( · , · ) instead of the Euclidean distance used in Katyusha X. Thus, the gradient mapping step (corresponding to Step 2 in [6, (4.1)]) should be built on the reference function h ( · ) defined in Eq. (8), i.e., ∇ h ( x k +1 ) -∇ h ( y k +1 ) instead of ( x k +1 -y k +1 ) /θ . Moreover, noting that ∇ h ( · ) could involve the heavy gradient computing part ∇ f ( · ) , we further employ its stochastic version (Step 5 in Algorithm 2) to reduce the overall communication complexity.

Next, we delve into the convergence analysis. We first give the core lemma for AccSVRS, which is also motivated by the framework of Katyusha X [6]. The proof is deferred to Appendix D.3.

(

k

](

1

2

α

x

k

∥

+1

z

)

-

-∇

z

[

f

∥

2

k

+

1

⟨

f

R

j

d

z

∈

-

G

k

f

k

j

+1

](

,

y

z

k

⟩

+1

+

) +

3

1

θ

µ

20

∥

z

x

(

-

k

y

+1

k

+1

-

∥

2

y

k

=

+1

)

z

k

, j

)

+0

k

∼

.

3

µα

y

k

1+0

([

+1

.

-

α

3

µα

n

G

])

k

+1

Lemma 3.5 Suppose Assumption 1 holds, and θ = 1 / (4 √ nδ ) , p = 1 /n, α ≤ nθ/ (2 τ ) in Algorithm 2, where SVRS 1ep ( f, x k +1 , θ, p ) satisfies Eq. (10) in each iteration. Then for all x ∈ R d that is independent of the random indices i ( k ) 1 , i ( k ) 2 , . . . , i ( k ) T in SVRS 1ep ( f, x k +1 , θ, p ) , we have that

Finally, we present the convergence rate and communication complexity of AccSVRS based on Lemma 3.5, and the proof is left in Appendix D.4.

Theorem 3.6 Suppose Assumption 1 holds. Consider AccSVRS with the following hyperparameters

and Eq. (10) is satisfied in each iteration of SVRS 1ep ( f, x k +1 , θ, p ) . Then for any ε > 0 , when

i.e., after ˜ O ( n + n 3 / 4 √ δ/µ ) communications in expectation, we obtain that E f ( y k ) -f ( x ∗ ) ≤ ε .

Remark 3.7 Although roughly the same as the communication complexity obtained by Catalyzed SVRP in [33, Theorem 3], our results have the following advantages.

Fewer assumptions . Except for the strong convexity of f and AveSS of f i 's, we do not need to assume component strong convexity appearing in [33, Assumption 2].

Inexact proximal step. Khaled and Jin [33, Theorem 3] require exact evaluations of the proximal operator, though they mention that this is only for the convenience of analysis. Our framework allows approximated solutions in each proximal step, and the approximation criterion (10) is errorindependent, i.e., irrelevant to the final error ε . Since the local proximal function is strongly convex, we could solve the problem in a few steps if additionally assuming the smoothness of f 1 .

Smoothness-free bound. As shown in [33, Appendix G.1] or Appendix C, even if an exact proximal step is allowed, a dependence on the smoothness coefficient would be introduced in the total communication iterations of Catalyzed SVRP, though only in a log scale. Our directly accelerated method has no dependence on the smoothness coefficient.

3.3 Gradient Complexity under Smooth Assumption

Due to the importance of total computation in the machine learning and optimization community, we consider a more common setup by additionally assuming that f 1 is L -smooth with L ≥ δ ≥ µ > 0 , which together with Assumption 1 facilitates the quantification of Eq. (10). Then we can compute the total gradient complexity for AccSVRS as shown below. By Proposition 2.5 and our assumptions, A t θ ( x ) is ( 1 θ - √ nδ ) -strongly convex and ( 1 θ + L ) -smooth. Using accelerated methods starting from x t , we can guarantee that Eq. (10) holds after T app = ˜ O (√ 1+ θL 1 - √ nθδ ) = ˜ O ( 1 + n -1 / 4 √ L/δ )

iterations with the choice of θ in Theorem 3.6. Hence, the total gradient calls in expectation are

Since δ ∈ [ µ, L ] , we recover the optimal gradient complexity ˜ O ( n + n 3 / 4 √ L/µ ) for the average smooth setting [63, Table 1] if neglecting log factors. Particularly, when δ = Θ( √ µL ) , we even obtain the nearly optimal gradient complexity ˜ O ( n + √ nL/µ ) for the component smooth setting [25, 26, 56]. We leave the details in Appendix E. Although the gradient complexity is not the primary focus of our work, we have demonstrated that the gradient complexity bound of AccSVRS is nearly optimal for certain values of δ in specific cases.

4 Lower Bound

In this section, we establish the lower bound of the communication complexity, which nearly matches the upper bound of AccSVRS.

4.1 Definition of Algorithms

In this subsection, we specify the class of algorithms to which our lower bound can apply. We first introduce the Proximal Incremental First-order Oracle (PIFO) [56, 25], which is defined as h P f i ( x , γ ) = [ f i ( x ) , ∇ f i ( x ) , prox γ f i ( x )] with γ > 0 . Here the proximal operator is defined as prox γ f i ( x ) := arg min u { f i ( u ) + 1 2 γ ∥ x -u ∥ 2 } . In addition to the local zero-order and first-order information of f i at x , the PIFO h P f i ( x , γ ) also provides some global information through the proximal operator 9 . Then we assume the algorithm has access to the PIFO and the definition of algorithms is presented as follows.

Definition 4.1 Consider a randomized algorithm A to solve problem (1) . Suppose the number of communication rounds is T . At the initialization stage, the master node 1 communicates with all the others. In round t (0 ≤ t ≤ T -1) , the algorithm samples a node i t ∼ Unif([ n ]) , and node 1 communicates with node i t . Then the algorithm samples a Bernoulli random variable a t with constant expectation c 0 /n . If a t = 1 , node 1 communicates with all the others. Define the information set I t +1 as the set of all the possible points A can obtain after round t . The algorithm updates I t +1 based on the linear-span operation and PIFO, and finally outputs a certain point in I T .

At the initialization stage, the communication cost is 2( n -1) . In each communication round, the Bernoulli random variable a t determines whether the master node communicates with all the others, i.e., whether to calculate the full gradient. Since E a t = c 0 /n , the expected communication cost of each round is of the order Θ(1) . Thus the total communication cost is of the order Θ( n + T ) and we can use T to measure the communication complexity. Moreover, one can check Algorithm 2 satisfies Definition 4.1. The formal definition and detailed analysis are deferred to Appendix F.1.

4.2 The Construction and Results

In this section, we construct a hard instance of problem (1) and then use it to establish the lower bound. Due to space limitations, we only present several key properties. The complete framework of construction is deferred to Appendix F.2.

Inspired by [25], we consider the class of matrices B ( m,ζ ) =   1 -1 . . . . . . 1 -1 ζ   ∈ R m × m . This class of matrices is widely used to establish lower bounds for minimax optimization problems [61, 49, 59], and A ( m,ζ ) := B ( m,ζ ) ⊤ B ( m,ζ ) is the well-known tridiagonal matrix in the analysis of lower bounds for convex optimization [47, 40, 15]. Denote the l -th row of B ( m,ζ ) as b l ( m,ζ ) ⊤ . We partition the row vectors of B ( m,ζ ) according to the index sets L i = { l : 1 ≤ l ≤ m, l ≡ i -1 (mod ( n -1)) } for 2 ≤ i ≤ n and L 1 = ∅ 10 . These sets are mutually exclusive and their union is [ m ] . Then we consider the following problem

̸

Here e i ∈ R m denotes the unit vector with the i -th element equal to 1 and others equal to 0 . Then one can check r ( x ; m,ζ,c ) = 1 2 x ⊤ A ( m,ζ ) x + c 2 ∥ x ∥ 2 -⟨ e 1 , x ⟩ . Clearly, r is c -strongly convex. We can also determine the AveSS parameter as follows. The proof is deferred to Appendix F.3.

Proposition 4.2 Suppose that 0 < ζ ≤ √ 2 , n ≥ 3 and m ≥ 3 . Then r i 's satisfy √ 8 n +4 -AveSS.

Define the subspaces {F k } m k =0 as F 0 = { 0 } and F k = span { e 1 , e 2 , . . . , e k } for 1 ≤ k ≤ m . The next lemma is fundamental to our analysis. The proof is deferred to Appendix F.5.

Lemma 4.3 Suppose the algorithm A satisfies Definition 4.1 and apply it to solve problem (13) with n ≥ 3 and m ≥ 4 . We have (i) I 0 = F 1 . (ii) Suppose I t ⊆ F k ( 1 ≤ k ≤ m -3 ). If i t satisfies k ∈ L i t or a t = 1 , then I t +1 ⊆ F k +3 ; otherwise, I t +1 ⊆ F k .

0

5000

10000

15000

20000

Communications

10

12

10

10

10

8

10

6

10

4

10

2

10 0

10 2

f

(

x

)

f

(

x

*

)

Synthetic Data (

n

=400,

=1,

=12.6,

L

=3390)

SVRG

KatyushaX

SVRS

AccSVRS

SVRP

AccEG

0

20000

40000

60000

80000

100000

Communications

10

12

10

10

10

8

10

6

10

4

10

2

10 0

10 2

f

(

x

)

f

(

x

*

)

Synthetic Data (

n

=400,

=0.1,

=12.6,

L

=3390)

SVRG

KatyushaX

SVRS

AccSVRS

SVRP

AccEG

0

20000

40000

60000

80000

100000

Communications

10

12

10

10

10

8

10

6

10

4

10

2

10 0

10 2

f

(

x

)

f

(

x

*

)

Synthetic Data (

n

=400,

=0.01,

=12.6,

L

=3390)

SVRG

KatyushaX

SVRS

AccSVRS

SVRP

AccEG

Lemma 4.3 guarantees that in each round, only when a specific component is sampled or the full gradient is calculated, can we expand the information set by at most three dimensions. For problem (13), we could never obtain an approximate solution unless we expand the information set to the whole space (see Proposition F.6 in Appendix F.2), while Lemma 4.3 implies that the process of expanding is very slow. Then we can establish the following lower bound.

Theorem 4.4 For any n ≥ 3 , δ, µ > 0 , algorithm A satisfying Definition 4.1 and sufficiently small ϵ > 0 , there exists a rescaled version of problem (13) such that (i) Assumption 1 holds; (ii) In order to find an ϵ -suboptimal solution ˆ x such that E r (ˆ x ) -min x r ( x ) < ϵ by A , the communication complexity in expectation is ˜ Ω( n + n 3 / 4 √ δ/µ ) .

This lower bound nearly matches the upper bound in Theorem 3.6 up to log factors, implying Algorithm 2 is nearly optimal in terms of communication complexity. The detailed statement and proof are deferred to Appendices F.2 and F.9.

5 Experiments

To demonstrate the advantages of our algorithms, we conduct the same numerical experiments as those in [35, 33]. We focus on the linear ridge regression problem with ℓ 2 regularization, where the average loss f has the formulation: f ( x ) = 1 n ∑ n i =1 [ f i ( x ) := 1 m ∑ m j =1 ( z ⊤ i,j x -y i,j ) 2 + µ 2 ∥ x ∥ 2 ] . Here z i,j ∈ R d and y i,j ∈ R , ∀ i ∈ [ n ] , j ∈ [ m ] serve as the feature and label respectively, and m can be viewed as data size in each local client. We consider a synthetic dataset generated by adding a small random noise matrix to the center matrix, ensuring a small δ . To capture the differences in convergence rates between our methods and SVRP caused by different magnitudes of µ , we vary µ = 10 -i , i ∈ { 0 , 1 , 2 } . We compare our methods (SVRS and AccSVRS) against SVRG, KatyushaX, SVRP (Catalyzed SVRP is somehow hard to tune so we omit it), and Accelerated Extragradient (AccEG) using their theoretical step sizes, except that we scale the interpolation parameter τ in KatyushaX and AccSVRS for producing practical performance (see Appendix G for detail). From Figure 1, we can observe that for a large µ , SVRP outperforms existing algorithms due to its high-order dependence on µ . However, when the problem becomes ill-conditioned with a small µ , AccSVRS exhibits significant improvements compared to other algorithms.

6 Conclusion

In this paper, we have introduced two new algorithms, SVRS and its directly accelerated version AccSVRS, and established improved communication complexity bounds for distributed optimization under the similarity assumption. Our rates are entirely smoothness-free and only require strong convexity of the objective, average similarity, and proximal friendliness of components. Moreover, our methods also have nearly optimal gradient complexity (leaving out the log term) when applied to smooth components in specific cases. It would be interesting to remove additional log terms to achieve both optimal communication and local gradient calls as [35], as well as investigating the complexity under other similarity assumptions (such as SS instead of AveSS) in future research.

Acknowledgments and Disclosure of Funding

Lin, Han, and Zhang have been supported by the National Key Research and Development Project of China (No. 2022YFA1004002) and the National Natural Science Foundation of China (No. 12271011). Ye has been supported by the National Natural Science Foundation of China (No. 12101491).

References

Artem Agafonov, Pavel Dvurechensky, Gesualdo Scutari, Alexander Gasnikov, Dmitry Kamzolov, Aleksandr Lukashevich, and Amir Daneshmand. An accelerated second-order method for distributed stochastic optimization. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 2407-2413. IEEE, 2021.

Alekh Agarwal and Leon Bottou. A lower bound for the optimization of finite sums. In ICML , 2015.

Masoud Ahookhosh and Yurii Nesterov. High-order methods beyond the classical complexity bounds, ii: inexact high-order proximal-point methods with segment search. arXiv preprint arXiv:2109.12303 , 2021.

Mohammad S Alkousa, Alexander Vladimirovich Gasnikov, Darina Mikhailovna Dvinskikh, Dmitry A Kovalev, and Fedor Sergeevich Stonyakin. Accelerated methods for saddle-point problem. Computational Mathematics and Mathematical Physics , 60:1787-1809, 2020.

Zeyuan Allen-Zhu. Katyusha: the first direct acceleration of stochastic gradient methods. The Journal of Machine Learning Research , 18(1):8194-8244, 2017.

Zeyuan Allen-Zhu. Katyusha x: Practical momentum method for stochastic sum-of-nonconvex optimization. arXiv preprint arXiv:1802.03866 , 2018.

Yossi Arjevani and Ohad Shamir. Communication complexity of distributed convex learning and optimization. Advances in neural information processing systems , 28, 2015.

Syreen Banabilah, Moayad Aloqaily, Eitaa Alsayed, Nida Malik, and Yaser Jararweh. Federated learning review: Fundamentals, enabling technologies, and future applications. Information processing & management , 59(6):103061, 2022.

Ron Bekkerman, Mikhail Bilenko, and John Langford. Scaling up machine learning: Parallel and distributed approaches . Cambridge University Press, 2011.

Aleksandr Beznosikov and Alexander Gasnikov. Compression and data similarity: Combination of two techniques for communication-efficient solving of distributed variational inequalities. In International Conference on Optimization and Applications , pages 151-162. Springer, 2022.

Aleksandr Beznosikov and Alexander Gasnikov. Similarity, compression and local steps: Three pillars of efficient communications for distributed variational inequalities. arXiv preprint arXiv:2302.07615 , 2023.

Aleksandr Beznosikov, Eduard Gorbunov, and Alexander Gasnikov. Derivative-free method for composite optimization with applications to decentralized distributed optimization. IFACPapersOnLine , 53(2):4038-4043, 2020.

Aleksandr Beznosikov, Gesualdo Scutari, Alexander Rogozin, and Alexander Gasnikov. Distributed saddle-point problems under data similarity. Advances in Neural Information Processing Systems , 34:8172-8184, 2021.

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Accelerated methods for nonconvex optimization. SIAM Journal on Optimization , 28(2):1751-1772, 2018.

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points ii: first-order methods. Mathematical Programming , 185(1-2):315-355, 2021.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

Gong Chen and Marc Teboulle. Convergence analysis of a proximal-like minimization algorithm using bregman functions. SIAM Journal on Optimization , 3(3):538-543, 1993.

Amir Daneshmand, Gesualdo Scutari, Pavel Dvurechensky, and Alexander Gasnikov. Newton method over networks is fast up to the statistical precision. In International Conference on Machine Learning , pages 2398-2409. PMLR, 2021.

Aaron Defazio. A simple practical accelerated method for finite sums. Advances in neural information processing systems , 29, 2016.

Radu Alexandru Dragomir, Mathieu Even, and Hadrien Hendrikx. Fast stochastic bregman gradient methods: Sharp analysis and variance reduction. In International Conference on Machine Learning , pages 2815-2825. PMLR, 2021.

Darina Mikhailovna Dvinskikh, Sergey Sergeevich Omelchenko, Alexander Vladimirovich Gasnikov, and AI Tyurin. Accelerated gradient sliding for minimizing a sum of functions. In Doklady Mathematics , volume 101, pages 244-246. Springer, 2020.

Alexandre d'Aspremont, Damien Scieur, Adrien Taylor, et al. Acceleration methods. Foundations and Trends® in Optimization , 5(1-2):1-245, 2021.

Dan Garber, Elad Hazan, Chi Jin, Cameron Musco, Praneeth Netrapalli, Aaron Sidford, et al. Faster eigenvector computation via shift-and-invert preconditioning. In International Conference on Machine Learning , pages 2626-2634. PMLR, 2016.

Geovani Nunes Grapiglia and Yurii Nesterov. Adaptive third-order methods for composite convex optimization. arXiv preprint arXiv:2202.12730 , 2022.

Yuze Han, Guangzeng Xie, and Zhihua Zhang. Lower complexity bounds of finite-sum optimization problems: The results and construction. arXiv preprint arXiv:2103.08280 , 2021.

Robert Hannah, Yanli Liu, Daniel O'Connor, and Wotao Yin. Breaking the span assumption yields fast finite-sum minimization. Advances in Neural Information Processing Systems , 31, 2018.

Hadrien Hendrikx, Lin Xiao, Sebastien Bubeck, Francis Bach, and Laurent Massoulie. Statistically preconditioned accelerated gradient method for distributed optimization. In International conference on machine learning , pages 4203-4227. PMLR, 2020.

Anastasiya Ivanova, Pavel Dvurechensky, Evgeniya Vorontsova, Dmitry Pasechnyuk, Alexander Gasnikov, Darina Dvinskikh, and Alexander Tyurin. Oracle complexity separation in convex optimization. Journal of Optimization Theory and Applications , 193(1-3):462-490, 2022.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems , 26, 2013.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Dmitry Kamzolov, Alexander Gasnikov, and Pavel Dvurechensky. Optimal combination of tensor optimization methods. In Optimization and Applications: 11th International Conference, OPTIMA 2020, Moscow, Russia, September 28-October 2, 2020, Proceedings 11 , pages 166183. Springer, 2020.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

Ahmed Khaled and Chi Jin. Faster federated optimization under second-order similarity. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=ElC6LYO4MfD .

Dmitry Kovalev, Samuel Horváth, and Peter Richtárik. Don't jump through hoops and remove those loops: Svrg and katyusha are better without the outer loop. In Algorithmic Learning Theory , pages 451-467. PMLR, 2020.

Dmitry Kovalev, Aleksandr Beznosikov, Ekaterina Dmitrievna Borodich, Alexander Gasnikov, and Gesualdo Scutari. Optimal gradient sliding and its application to optimal distributed optimization under similarity. In Advances in Neural Information Processing Systems , 2022.

Dmitry Kovalev, Aleksandr Beznosikov, Abdurakhmon Sadiev, Michael Persiianov, Peter Richtárik, and Alexander Gasnikov. Optimal algorithms for decentralized stochastic variational inequalities. Advances in Neural Information Processing Systems , 35:31073-31088, 2022.

Guanghui Lan. Gradient sliding for composite optimization. Mathematical Programming , 159: 201-235, 2016.

Guanghui Lan and Yuyuan Ouyang. Accelerated gradient sliding for structured convex optimization. Computational Optimization and Applications , 82(2):361-394, 2022.

Guanghui Lan and Yi Zhou. Conditional gradient sliding for convex optimization. SIAM Journal on Optimization , 26(2):1379-1409, 2016.

Guanghui Lan and Yi Zhou. An optimal randomized incremental gradient method. Mathematical programming , 171:167-215, 2018.

Kfir Y Levy. Slowcal-sgd: Slow query points improve local-sgd for stochastic convex optimization. arXiv preprint arXiv:2304.04169 , 2023.

Bingcong Li, Meng Ma, and Georgios B Giannakis. On the convergence of sarah and beyond. In International Conference on Artificial Intelligence and Statistics , pages 223-233. PMLR, 2020.

Zhize Li. Anita: An optimal loopless accelerated variance-reduced gradient method. arXiv preprint arXiv:2103.11333 , 2021.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems , 30, 2017.

Ming Liu, Stella Ho, Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang. Federated learning meets natural language processing: A survey. arXiv preprint arXiv:2107.12603 , 2021.

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750-15769. PMLR, 2022.

Yurii Nesterov et al. Lectures on convex optimization , volume 137. Springer, 2018.

Dinh C Nguyen, Quoc-Viet Pham, Pubudu N Pathirana, Ming Ding, Aruna Seneviratne, Zihuai Lin, Octavia Dobre, and Won-Joo Hwang. Federated learning for smart healthcare: A survey. ACM Computing Surveys (CSUR) , 55(3):1-37, 2022.

Yuyuan Ouyang and Yangyang Xu. Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems. Mathematical Programming , 185(1-2):1-35, 2021.

Sashank J Reddi, Jakub Koneˇ cn` y, Peter Richtárik, Barnabás Póczós, and Alex Smola. Aide: Fast and communication efficient distributed optimization. arXiv preprint arXiv:1608.06879 , 2016.

Ohad Shamir, Nati Srebro, and Tong Zhang. Communication-efficient distributed optimization using an approximate newton-type method. In International conference on machine learning , pages 1000-1008. PMLR, 2014.

Danny C Sorensen. Numerical methods for large eigenvalue problems. Acta Numerica , 11: 519-584, 2002.

Ivan Stepanov, Artyom Voronov, Aleksandr Beznosikov, and Alexander Gasnikov. One-point gradient-free methods for composite optimization with applications to distributed optimization. arXiv preprint arXiv:2107.05951 , 2021.

Ying Sun, Gesualdo Scutari, and Amir Daneshmand. Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation. SIAM Journal on Optimization , 32(2):354-385, 2022.

Ye Tian, Gesualdo Scutari, Tianyu Cao, and Alexander Gasnikov. Acceleration in distributed optimization under similarity. In International Conference on Artificial Intelligence and Statistics , pages 5721-5756. PMLR, 2022.

Blake E Woodworth and Nati Srebro. Tight complexity bounds for optimizing composite objectives. Advances in neural information processing systems , 29, 2016.

Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization , 24(4):2057-2075, 2014.

Xiao-Tong Yuan and Ping Li. On convergence of distributed approximate newton methods: Globalization, sharper bounds and beyond. The Journal of Machine Learning Research , 21(1): 8502-8552, 2020.

Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming , 194(1-2):901-935, 2022.

Sai Qian Zhang, Jieyu Lin, and Qi Zhang. A multi-agent reinforcement learning approach for efficient client selection in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9091-9099, 2022.

Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of nonconvex-strongly-concave minimax optimization. In Uncertainty in Artificial Intelligence , pages 482-492. PMLR, 2021.

Yuchen Zhang and Xiao Lin. Disco: Distributed optimization for self-concordant empirical loss. In International conference on machine learning , pages 362-370. PMLR, 2015.

Dongruo Zhou and Quanquan Gu. Lower bounds for smooth nonconvex finite-sum optimization. In International Conference on Machine Learning , pages 7574-7583. PMLR, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary** \n\nThe paper studies finite-sum minimization ($\\min_{x} f(x):= \\frac{1}{n} \\sum_{i= 1}^n f_i(x)$) in the distributed setting with a central node and $n-1$ non-central (client) nodes. The setup and the paper's assumptions are as follows. \n\n1. Each function $f_i$ is held in the $i$-th client node, and the first node is designated the central node. The machines on the non-central nodes can communicate function values and gradients with each other as well as with the central node. \n\n2. The different $f_i$'s are assumed to be related to each other via the notion of \"second-order similarity\", which essentially means that the Hessians of $f_i$ and $f$ evaluated at the same point do not differ from each other by too much in the operator norm. A benefit of this assumption is that different clients do not need to all send their Hessian information. \n\n3. The $f_i$'s are all convex, $f$ is $\\mu$-strongly convex, and the $f_i$'s all satisfy $\\delta$-average-second-order-similarity as described in 2. \n\nThe paper's results are three-fold: \n\n1. A non-accelerated algorithm called SVRS, which attains a communication cost of $\\widetilde{O}(n + \\sqrt{n} \\delta/\\mu)$. This result improves upon the previous best result of Khaled and Jin when $\\delta \\geq \\sqrt{n} \\mu$. \n\n2. An accelerated algorithm called AccSVRS, which attains a communication cost of $O((n+ n^{3/4} \\sqrt{\\delta/\\mu})\\log|\\epsilon|)$, which improves upon the previous best accelerated rate by Khaled and Jin by a factor of $\\log(L/\\mu)$. Therefore, this new rate is \"smoothness-free\". \n\n3. Lower bounds for 2. \n\nThe paper's technique is based, broadly, on the use of Bregman-SVRG: instead of calculating exactly in each iteration, we calculate the approximate entities every time and periodically offset the error by exact calculations; additionally, through the use of a Bregman divergence term, the update rule ensures that the next iterate is not too far from the current one in some chosen distance metric.\n\nSTRENGTHS:\n**Strengths** \nI think the paper scores highly on the story-telling aspect: it reads quite well! I also think the paper studies an important problem (faster communication complexity for distributed optimization).\n\nWEAKNESSES:\nIn results: \n\n1. Line 64: The paper's result from SVRS beats that of SVRP by Khaled and Jin when $\\delta \\geq \\sqrt{n} \\mu$. As of now it's not clear to me what the scope of this assumption is: are there many cases where this inequality holds? It appears to me that Lines 124 and 198 allude to this but it's not entirely clear to me. Can the authors please elaborate?  \n\n2. Line 68 - 70: The paper's result with acceleration shaves a log factor from the previous best result as well as removes a component-wise strong convexity assumption from the previous best result. It would be nice to see some intuition for why the removal of log factor here is important and why the component-wise strong convexity is that much stronger than total strong convexity. \n\n\nIn the writing: \n\n1. In lines 14 - 17, there are no references for these stated applications. It would be much more convincing to have citations for each of the stated applications. \n\n2. The problem parameter $\\mu$ should be introduced separately, rather than as part of the related work in lines 40 - 48, since it's an important parameter that comes up repeatedly. \n\n3. The inclusion of Lemma $3.1$ in the main body does not serve (in my opinion) much purpose because it's mathematically quite dense to parse Equation $(10)$. Theorem $3.2$ is ok since it clearly gives the claimed rate. Instead of Lemma $3.1$, I'd have preferred to have a proof sketch for Theorem $3.2$.\n\nQUESTIONS:\n**Questions** \n\n\n1. In Algorithm 1, should line 4 be $g_t = \\nabla f_{i_t} (w_t) - \\nabla f(w_t)$ and should there be an update rule for $w_t$ somewhere? (That's what Line 176 seems to suggest.) If not, it would be great to have intuition for why the same $w_0$ is used through the entire algorithm. \n\n2. Please see the Weaknesses section above for more questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider distributed minimization problems under data similarity (hessian similarity). The authors consider stochastic methods that reduce communication complexity via device sampling. In particular, from the stochastic point of view, the variance reduction techniques: SVRG and Katyusha, are taken. The sliding (stochastic preconditioning/mirror descent with unusual Bregman divergence) technique is used for dealing with similarity. The authors obtain record results in communication complexity (the previous ones were beaten by the logarithmic factor). To complete the picture, the authors provide lower bounds that give the optimality of their upper bounds. Synthetic experiments are also given.\n\nSTRENGTHS:\n1) Direct acceleration (without envelops) is an interesting and important result. In theory it removes the extra logarithmic factor, and in practice it works better. \n\n2) The lower bounds complete the picture.\n\nWEAKNESSES:\n1) I think the literature review is not complete. In particular I find two papers also about the hessian similarity, which also use the variance reduction technique and obtain such non-accelerated results as the authors have. A detailed comparison in approaches and results is needed. \n\nBeznosikov, A., & Gasnikov, A. (2023). Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities. arXiv preprint arXiv:2302.07615.\n\nBeznosikov, A., & Gasnikov, A. (2022, September). Compression and data similarity: Combination of two techniques for communication-efficient solving of distributed variational inequalities. In International Conference on Optimization and Applications (pp. 151-162). Cham: Springer Nature Switzerland.\n\n2) The lower bounds are a good supplement to the upper bounds, but they are to be expected. The idea of getting them is also known. Unfortunately, here the authors  don not  also give a complete summary of the literature: the problem with a matrix A is classical (the authors note it), but the partition of the problem into columns is also classical - see the papers:\n\nZhang, M., Shu, Y., & He, K. (2020). Tight Lower Complexity Bounds for Strongly Convex Finite-Sum Optimization. arXiv preprint arXiv:2010.08766.\n\nHan, Y., Xie, G., & Zhang, Z. (2021). Lower complexity bounds of finite-sum optimization problems: The results and construction. arXiv preprint arXiv:2103.08280.\n\nKovalev, D., Beznosikov, A., Sadiev, A., Persiianov, M., Richtárik, P., & Gasnikov, A. (2022). Optimal algorithms for decentralized stochastic variational inequalities. Advances in Neural Information Processing Systems, 35, 31073-31088.\n\n3) Experiments for me are not the most important thing in this paper, which is primarily theoretical. But I would still like to see real datasets (3 for symmetry with synthetic data). I would also advise authors to change not \\mu, but to change \\delta in synthetic experiments, this way the effect of similarity will be noticeable. \n\nSummary: For me, this is a borderline paper. For now I put a (weak) rejection, but I hope the authors will take part in discussion and make the changes I asked for.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nThe paper is theoretical, therefore there is no need to discuss the social negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a novel algorithm for distributed optimization, named Accelerated Stochastic Variance-Reduced Sliding (ASVRS). The authors focus on the problem of minimizing the average of a large number of smooth and strongly convex functions, a common scenario in machine learning and data analysis. The proposed ASVRS algorithm combines the techniques of gradient-sliding and variance reduction, aiming to improve the convergence rate and reduce the communication cost in distributed settings.

STRENGTHS:
The ASVRS algorithm is a novel contribution that combines gradient-sliding and variance reduction techniques in a unique way. This combination appears to be original and innovative. The authors provide detailed proofs for the convergence rate and communication complexity of the ASVRS algorithm, demonstrating its theoretical advantages over existing methods. I didn't check the details but the result seems reasonable.

WEAKNESSES:
The theoretical analysis relies on several assumptions, such as the strong convexity of the functions. This makes the application of the work rather limited.  It would be helpful to discuss the implications of these assumptions and how the algorithm's performance might be affected if they are not met.

QUESTIONS:
How sensitive is the ASVRS algorithm to its parameters? Could you provide some guidance on how to set the parameters in practice.

LIMITATIONS:
The limitation has been adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers distributed strongly convex optimization problems in the setting where the communication between nodes is bottleneck. The authors propose new methods, SVRS and AccSVRS, that guarantee new communication complexities. Also, they proved the lower bound that ensures the optimality of the AccSVRS method.\n\nSTRENGTHS:\nI think that the paper is strong. The authors provide new theoretical guarantees in the considered setting. They improve the previous methods. I haven't checked the proofs in detail and I can miss some essential parts, but the theory sounds to me.\n\nWEAKNESSES:\nIt is well known that the (Loopless-)Katyusha method converges after $n + \\sqrt{n \\frac{L}{\\mu}}$ iterations. By applying Katyusha to the authors' problem, we can get the communication complexity $n + \\sqrt{n \\frac{L}{\\mu}},$ since Katyusha requires only one gradient in each iteration. In the regime when $\\delta = L$ the Katyusha method has better communication complexity than $n + n^{3/4} \\sqrt{\\frac{L}{\\mu}}.$ Why doesn't it contradict the lower bound (Theorem G.7 and Theorem 4.4)? Does it mean that your method is only better in the regimes when $\\delta \\ll L$?\n\nMinor comments:\n\nThe paper's setup can be slightly confusing. Many other papers (e.g., \\[1,2\\]) assume that the nodes can do calculations and send vectors *in parallel,* meaning that they count each round as *one* communication, *one round = one communication*. In comparison, this paper assumes that *one round = $n$ communications* in the full participation regime. Can the authors write a small text *in the paper* explaining the difference between the setups? It seems that we have at least two different setups that both have the right to life.\n\nTypos:\n\nEq. (18): $\\nabla$ is missed.\n\n\\[1\\]: https://arxiv.org/abs/2202.09357\n\\[2\\]: https://arxiv.org/abs/2304.04169\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers finite-sum (distributed) optimization problems in the strongly convex and second-order similar regime. The authors proposed SVRS and its acceleration, AccSVRS, to solve the problem and provided the corresponding communication and computation complexities, which outperform existing works in several perspectives. The authors further characterized the lower bound for solving such problems, which validates the near optimality of the proposed AccSVRS algorithm.\n\nSTRENGTHS:\n1. The study is comprehensive in general, covering both upper and lower bounds\n2. Proposed algorithm achieves near-optimal communication/computation complexity. \n3. The design of the algorithm, which incorporates PPA, VR, gradient sliding and Katyusha, is interesting.\n4. The paper is well-written, and the flow is clear.\n\nWEAKNESSES:\n1. The design of the AccSVRS, as the core algorithm achieving near optimality, can be further elaborated. For now, I do not have a clear understanding on Steps 5 and 6 in AccSVRS. Some discussions similar to Section 3.1 connecting Katyusha X paper will be appreciated.\n2. Some more questions below.\n\nQUESTIONS:\n1. Line 232, you mentioned you recovered the optimal complexity of the average smooth setting. While with the additional AveSS setting, I may expect the complexity possibly can be (strictly) better than the classical $L$-smooth $\\mu$-strongly convex case (assuming all components are $L$-smooth here for convenience). \n\n   In your proof of Appendix E, you mentioned the obtained complexity is only smaller than that of the average smooth setting (Line 590), but should it be equivalence (rather than smaller only) here to be verified? \n  \n2. With the current result in Section 3.3, can we argue that possibly second-order similarity does not help in the gradient complexity of the classical case?\n\n3. Line 591 in Appendix E (or Line 234), why do you drop the $O(n^{3/4}(L/\\mu)^{1/4})$ term? Compared to the last $O(\\sqrt{nL/\\mu})$ term, is there a possibility that the $n^{3/4}$ term will dominate? Do I miss anything here?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}