Title: Stabilized Proximal-Point Methods for Federated Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Convex Optimization

Distributed Optimization

Proximal-Point Method

Second-order Dissimilarity

Acceleration

Line search
Abstract: In developing efficient optimization algorithms, it is crucial to account for communication constraints&mdash;a significant challenge in modern Federated Learning. 
    The best-known communication complexity among non-accelerated algorithms is achieved by DANE, a distributed proximal-point algorithm that solves local subproblems at each iteration and that can exploit second-order similarity among individual functions.
    However, to achieve such communication efficiency, the algorithm
    requires solving local subproblems sufficiently accurately resulting in slightly sub-optimal local complexity.
    Inspired by the hybrid-projection proximal-point method, in this work, we propose a novel distributed algorithm S-DANE. Compared to DANE, this method uses an auxiliary sequence of prox-centers while maintaining the same deterministic communication complexity. Moreover, the accuracy condition for solving the subproblem is milder, leading to enhanced local computation efficiency. Furthermore, S-DANE supports partial client participation and arbitrary stochastic local solvers, making it attractive in practice. We further accelerate S-DANE and show that the resulting algorithm achieves the best-known communication complexity among all existing methods for distributed convex optimization while still enjoying good local computation efficiency as S-DANE.
    Finally, we propose adaptive variants of both methods using line search, obtaining the first provably efficient adaptive algorithms that could exploit local second-order similarity without the prior knowledge of any parameters.

Stabilized Proximal-Point Methods for Federated Optimization

Abstract

In developing efficient optimization algorithms, it is crucial to account for communication constraints-a significant challenge in modern Federated Learning. The best-known communication complexity among non-accelerated algorithms is achieved by DANE, a distributed proximal-point algorithm that solves local subproblems at each iteration and that can exploit second-order similarity among individual functions. However, to achieve such communication efficiency, the algorithm requires solving local subproblems sufficiently accurately resulting in slightly sub-optimal local complexity. Inspired by the hybrid-projection proximalpoint method, in this work, we propose a novel distributed algorithm S-DANE. Compared to DANE, this method uses an auxiliary sequence of prox-centers while maintaining the same deterministic communication complexity. Moreover, the accuracy condition for solving the subproblem is milder, leading to enhanced local computation efficiency. Furthermore, S-DANE supports partial client participation and arbitrary stochastic local solvers, making it attractive in practice. We further accelerate S-DANE and show that the resulting algorithm achieves the best-known communication complexity among all existing methods for distributed convex optimization while still enjoying good local computation efficiency as S-DANE. Finally, we propose adaptive variants of both methods using line search, obtaining the first provably efficient adaptive algorithms that could exploit local second-order similarity without the prior knowledge of any parameters.

1 Introduction

Federated learning is a rapidly emerging large-scale machine learning framework that allows training from decentralized data sources (e.g. mobile phones or hospitals) while preserving basic privacy and security [43, 24, 31]. Developing efficient federated optimization algorithms becomes one of the central focuses due to its direct impact on the effectiveness of global machine learning models.

One of the key challenges in modern federated optimization is to tackle communication bottlenecks [32]. The large-scale model parameters, coupled with relatively limited network capacity and unstable client participation, often make communication highly expensive. Therefore, the primary efficiency metric of a federated optimization algorithm is the total number of communication rounds required to reach a desired accuracy level. If two algorithms share equivalent communication complexity, their local computation efficiency becomes the second important metric.

The seminal algorithm DANE [57] is an outstanding distributed optimization method. It achieves the best-known deterministic communication complexity among existing non-accelerated algorithms (on the server side) [22]. This efficiency primarily hinges upon a mild precondition regarding the Secondorder Dissimilarity δ . In numerous scenarios, like statistical learning for generalized model [19] and

semi-supervised learning [7], δ tends to be relatively small. However, to ensure such fast convergence, DANE requires each iteration subproblem to be solved with sufficiently high accuracy. This leads to sub-optimal local computation effort across the communication rounds, which is inefficient in practice. FEDRED [22] improves this weakness by using double regularization. However, this technique is only effective when using gradient descent as the local solver but cannot easily be combined with other optimization methods. For instance, applying local accelerated gradient or second-order methods cannot improve its local computation efficiency. Moreover, it is also unclear how to extend this method to the partial client participation setting relevant to federated learning.

On the other hand, the communication complexities achieved by the current accelerated methods typically cannot be directly compared with those attained by DANE, as they either depend on sub-optimal constants or additional quantities such as the number of clients n . The most relevant and state-of-the-art algorithm ACC-EXTRAGRADIENT [33] achieves a better complexity in terms of the accuracy ε but with dependency on the maximum Second-order Dissimilarity δ max which can in principle be much larger than δ (see Remark 2). Unlike most federated learning algorithms, such as FEDAVG [43], this method requires communication with all the devices at each round to compute the full gradient and then assigns one device for local computation. In contrast, FEDAVG and similar algorithms perform local computations on parallel and utilize the standard averaging to compute the global model. The follow-up work AccSVRS [40] applies variance reduction to ACCEXTRAGRADIENT which results in less frequent full gradient updates. However, the communication complexity incurs a dependency on n which is prohibitive for cross-device setting [24] where the number of clients can be potentially very large. Thus, there exists no accelerated federated algorithm that is uniformly better than DANE in terms of communication complexity.

Contributions. In this work, we aim to develop federated optimization algorithms that can achieve the best communication complexity while retaining efficient local computation. To this end, we first revisit the simple proximal-point method on a single machine. The accuracy requirement for solving the subproblem defined in this algorithm is slightly sub-optimal. Drawing inspiration from hybrid projection-proximal point method for finding zeroes of a maximal monotone operator [59], we observe that using a more stabilized prox-center improves the accuracy requirement. We make the following contributions:

We develop a novel federated optimization algorithm S-DANE that achieves the best-known communication complexity (for non-accelerated methods) while also enjoying improved local computation efficiency over DANE [57].

We develop an accelerated version of S-DANE based on the Monteiro-Svaiter acceleration [46]. The resulting algorithm ACC-S-DANE achieves the best-known communication complexity among all existing methods for distributed convex optimization.

Both algorithms support partial client participation and arbitrary stochastic local solvers, making them attractive in practice for federated optimization.

We provide a simple analysis for both algorithms. We derive convergence estimates that are continuous in the strong convexity parameter µ .

We propose adaptive variants of both algorithms using line-search in the full client participation setting. The resulting methods achieve the same communication complexity (up to a logrithmic factor) as non-adaptive ones without requiring knowledge of the similarity constant.

We illustrate strong practical performance of our proposed methods in experiments.

See also Table 1 for a summary of the main complexity results in the full-participation setting.

Related Work. Moreau first proposed the notion of the proximal approximation of a function [47]. Based on this operation, Martinet developed the first proximal-point method [42]. This method was first accelerated by Güller [17], drawing the inspiration from Nesterov's Fast gradient method [49]. Later, Lin et al. [41] introduced the celebrated CATALYST framework that builds upon Güller's acceleration. Using CATALYST acceleration, a large class of optimization algorithms can directly achieve faster convergence. In a similar spirit, Doikov and Nesterov [12] propose contracting proximal methods that can accelerate higher-order tensor methods. While Güller's acceleration has been successfully applied to many settings, its local computation is sub-optimal. Specifically, when minimizing smooth convex functions, a logarithmic dependence on the final accuracy is

0

100

200

300

400

Communication rounds

10

1

10 0

10 1

10 2

10 3

10 4

f(x)-

f

0

20000

40000

60000

Total number of gradient oracle calls

10

1

10 0

10 1

10 2

10 3

10 4

f(x)-

f

0

5

10

15

20

25

Communication rounds

10

3

10

2

10

1

10 0

10 1

10 2

value

local smoothness

local dissimilarity

(Adaptive S-DANE)

Adaptive Acc-S-DANE-GD (ours)

Adaptive S-DANE-GD (ours)

Acc-S-DANE-GD (ours)

S-DANE-GD (ours)

DANE-GD

incurred in its local computation complexity [12]. Solodov and Svaiter [59] proposed a HYBRID PROJECTION-PROXIMAL POINT method that allows significant relaxation of the accuracy condition for the proximal-point subproblems. More recent works such as ADAPTIVE CATALYST [21] and RECAPP [5] successfully get rid of the additional logarithmic factor for accelerated proximal-point methods as well.

Another type of acceleration based on the proximal extra-gradient method was introduced by Monteiro and Svaiter [46]. This method is more general in the sense that it allows arbitrary local solvers and the convergence rates depend on the these solvers. For instance, under convexity and Lipschitz secondorder derivative, the rate can be accelerated to O (1 /k 3 . 5 ) by using Newton-type method. Moreover, when the gradient method is used, Monteiro-Svaiter Acceleration recovers the rate of Güller's acceleration and its accuracy requirement for the inexact solution is weaker. For minimizing smooth convex functions, one gradient step is enough for approximately solving the local subproblem [48]. This technique has been applied to centralized composite optimization, known as gradient sliding [35, 36, 33]. A comprehensive study of acceleration can be found in [14].

We defer the literature review on distributed and federated optimization algorithms to Appendix A.

2 Problem Setup and Background

We consider the following distributed minimization problem:

where each function f i : R d → R is µ -strongly convex 2 for some µ ≥ 0 . We focus on the standard federated setting where the functions { f i } are distributed among n devices. The server coordinates the global optimization procedure among the devices. In each communication round, the server broadcasts certain information to the clients. The clients, in turn, perform local computation in parallel based on their own data and transmit the resulting local models back to the server to update the global model.

Main objective: Given the high cost of establishing connections between the server and the clients, our paramount objective is to minimize the number of required communication rounds to achieve the desired accuracy level. This represents a central metric in federated contexts, as outlined in references such as [43, 27]. Secondary objective: Efficiency in local computation represents another pivotal metric for optimization algorithms. For instance, if two algorithms share equivalent communication complexity, the algorithm with less local computational complexity is the more favorable choice.

Notation: We abbreviate [ n ] := { 1 , 2 , . . . , n } . For a set A and an integer 1 ≤ s ≤ | A | , we use ( A s ) to denote the power set comprised of all s -element subsets of A . Everywhere in this paper, ∥·∥ denotes the standard Euclidean norm (or the corresponding spectral norm for matrices). We assume problem (1) has a solution which we denote by x ⋆ ; the corresponding optimal value is denoted by f ⋆ . For a set S ∈ ( [ n ] s ) , we use f S := 1 s ∑ i ∈ S f i to denote the average function over this set. We use r to denote the index of the communication round and k to denote the index of the local step. Finally, we use the superscript and subscript to denote the global and local models, respectively; for instance, x r represents the global model at round r while x i,r is the local model computed by device i at round r .

2.1 Proximal-Point Methods on Single Machine

In this section, we provide a brief background on proximal-point methods [47, 42, 54, 51], which are the foundation for many distributed optimization algorithms.

Proximal-Point Method. Given an iterate x k , the method defines x k +1 to be an (approximate) minimizer of the proximal-point subproblem:

for an appropriately chosen parameter λ ≥ 0 . This parameter allows for a trade-off between the complexity of each iteration and the rate of convergence. If λ = 0 , the subproblem in each iteration is as difficult as solving the original problem because no regularization is applied. However, as λ increases, more regularization is added, simplifying the subproblem. For example, for a convex function f , the proximal-point method guarantees f (¯ x K ) -f ⋆ ≤ O ( λ K ∥ x 0 -x ⋆ ∥ 2 ) , where ¯ x K := 1 K ∑ K k =1 x k [51, 5]. However, to achieve such a convergence rate, the subproblem (2) has to be solved to a fairly high accuracy [54, 5]. For instance, the accuracy condition should either depend on the target accuracy ε , or increase with k : ∥∇ F k ( x k +1 ) ∥ = O ( λ k ∥ x k +1 -x k ∥ ) [58]. Indeed, when f is Lipschitz-smooth and the standard gradient descent is used as a local solver, the number of gradient steps required to solve the subproblem has a logarithmic dependence on the iteration counter k . The same issue also arises when considering accelerated proximal-point methods [12, 17].

Stabilized Proximal-Point Method. One of the key insights that we use in this work is the observation that using a different prox-center makes the accuracy condition of the subproblem weaker.

Algorithm 1 S-DANE: Stabilized DANE

The stabilized proximal-point method defines

where λ ≥ 0 is a parameter of the method and µ ≥ 0 is the strong-convexity constant of f . This algorithm updates the prox-center v k by performing an additional gradient step in each iteration. For instance, when µ = 0 , the prox-center is updated as v k +1 = v k -1 λ ∇ f ( x k +1 ) , which is often referred to as an extra-gradient update . The stabilized proximal-point method has the same convergence rate as the original method (2) but requires only that ∥∇ F k ( x k +1 ) ∥ ≤ O ( λ ∥ x k +1 -v k ∥ ) . As a result, there is no extra logarithmic factor of k in the oracle complexity estimate when f is L -smooth. Specifically, by setting λ = Θ( L ) , the previous condition can be satisfied by choosing x k +1 as the result of one gradient step from v k [48]. This shows that the stabilized proximal-point method has a better overall oracle complexity than the standard proximal-point method (c.f. Theorem 1 for the special case n = 1 ). It is worth noting that the former algorithm originates from the hybrid projection-proximal point algorithm [59] designed for solving the more general problem of finding zeroes of a monotone operator. In this work, we apply this algorithm in the distributed setting ( n ≥ 2 ).

2.2 Distributed Proximal-Point Methods

The proximal-point method can be adapted to solve the distributed optimization problem (1). This is the idea behind FEDPROX [39]. It replaces the global proximal step (2) by n subproblems defined as x i,r +1 := arg min x { f i ( x ) + λ 2 ∥ x -x r ∥ 2 } , which can be solved independently on each device, followed by the averaging step x r +1 = 1 n ∑ n i =1 x i,r +1 . Here we switch the notation from k to r to highlight that one iteration of the proximal-point method corresponds to a communication round in this setting. To ensure convergence, FEDPROX has to use a large λ that depends on the target accuracy as well as the heterogeneity among { f i } , which slows down the communication efficiency [39]. DANE [57] improves this by incorporating a drift correction term into the subproblem:

Consequently, DANE allows to choose a much smaller λ in the algorithm. Moreover, it can exploit second-order similarity and achieve the best-known communication complexity among nonaccelerated methods [22]. However, as in the original proximal-point method, the subproblem needs to be solved sufficiently accurately leading to an extra logarithmic factor in the oracle complexity estimate. To overcome this problem, we propose new algorithms described in the following section.

3 Stabilized DANE

We now describe S-DANE (Alg. 1), our proposed federated proximal-point method that employs stabilized prox-centers in its subproblems. During each communication round r , the server samples a subset of clients uniformly at random and sends v r to these clients. Then the server collects ∇ f i ( v r ) from these clients, computes ∇ f S r ( v r ) and sends ∇ f S r ( v r ) back to them. Each device in the set then calls an arbitrary local solver (which can be different on each device) to approximately solve its local subproblem. Finally, each device transmits ∇ f i ( x i,r +1 ) and x i,r +1 back to the server which then aggregates these points and computes the new global model.

As DANE, S-DANE can also achieve communication speed-up if the functions among devices are similar to each other. This is formally captured by the following assumption.

Definition 1 (Second-order Dissimilarity) . Let f 1 , . . . , f n : R d → R be functions, and let s ∈ [ n ] , δ s ≥ 0 . Then, { f i } n i =1 are said to have δ s -SOD (of size s ) if for any x , y ∈ R d and any S ∈ ( [ n ] s ) , it holds that

where h S i := f S -f i and f S := 1 s ∑ i ∈ S f i .

Definition 1 quantifies the dissimilarity between any s functions and their average, i.e., the 'internal' variation between any s functions. Clearly, δ 1 = 0 , and, when s = n , we recover the standard notion of second-order dissimilarity introduced in prior works:

Definition 2 ( δ -SOD [28, 40, 22]) . δ -SOD := δ n -SOD of size n .

When each function f i is twice continuously differentiable, a simple sufficient condition for (5) is that 1 s ∑ i ∈ S ∥∇ 2 h S i ( x ) ∥ 2 ≤ δ 2 s for any x ∈ R d . However, this is not a necessary condition (see [22] for more details).

The quantity V ( x , y ) in the left-hand side of (5) can be interpreted as the variance of the gradient difference estimator ∇ f ˆ i ( x ) -∇ f ˆ i ( y ) , where ˆ i is chosen uniformly at random from S . In particular, it can be rewritten as V ( x , y ) = 1 s ∑ i ∈ S ∥∇ f i ( x ) -∇ f i ( y ) ∥ 2 -∥∇ f S ( x ) -∇ f S ( y ) ∥ 2 . If each function f i is L i -smooth, then δ s ≤ ( 1 s ∑ i ∈ S L 2 i ) 1 / 2 for any s ∈ [ n ] . However, in general, condition (5) is weaker than assuming that each f i is Lipschitz-smooth.

Full Client Participation. We first consider the cross-silo setting where all the clients are highly reliable ( s = n ). This is typically the case with organizations and institutions having strong computing resources and stable network connection [24].

Theorem 1. Consider Algorithm 1 with s = n . Let f i : R d → R be µ -convex with µ ≥ 0 for any i ∈ [ n ] . Assume that { f i } n i =1 have δ -SOD. Let λ = 2 δ and suppose that, for any r ≥ 0 , we have

Then, for any R ≥ 1 , it holds that 3

where ¯ x R := 1 ∑ R r =1 p r ∑ R r =1 p r x r for p := 1 + µ λ , and D := ∥ x 0 -x ⋆ ∥ . To obtain f (¯ x R ) -f ⋆ ≤ ε for a given ε > 0 , it thus suffices to perform R = O ( δ + µ log(1 + µD 2 ) ) communication rounds.

µ ε

Theorem 1 provides the convergence guarantee for S-DANE in terms of the number of communication rounds. Note that the rate is continuous in µ .

Remark 2. Some previous works express complexity estimates in terms of another constant, δ max , defined by the inequality ∥∇ h i ( x ) -∇ h i ( y ) ∥ ≤ δ max ∥ x -y ∥ holding for any x , y ∈ R d and any i ∈ [ n ] , where h i = f -f i . (See for instance the second line in Table 1). Note that our δ is always not larger than δ max , and can in principle be much smaller (up to √ n times).

The proven communication complexity is the same as that of DANE [22]. However, the accuracy condition is milder. Specifically, to achieve the same guarantee, DANE requires ∑ n i =1 ∥∇ ˜ F i,r ( x i,r +1 ) ∥ 2 ≤ O ( δ 2 r 2 ∑ n i =1 ∥ x i,r +1 -x r ∥ 2 ) , where ˜ F i,r is defined as in (4), which incurs an r 2 overhead in the denominator, as in the general discussion on proximal-point methods in Section 2.1. The next corollary shows that local computations in S-DANE could be computationally very efficient.

Corollary 3. Consider the same setting as in Theorem 1. Further, assume that each f i is L -smooth. To ensure (6) with a certain first-order algorithm, each device i needs to perform at most O ( √ L δ ) computations of ∇ f i at each round r .

Remark 4. Particular examples of algorithms that could be used to achieve the result from Corollary 3 are OGM-OG by Kim and Fessler [30] and the accumulative regularization method by Lan et al. [37], both designed for the fast minimization of the gradient norm. For the standard Gradient Method (GM), the required number of oracle calls is O ( L δ ) . The standard Fast Gradient Method (FGM) [49] can further decrease the complexity to O ( √ L µ + δ log L δ ) (see Remark 18 for details). Thus, each device can run a constant number of standard local (F)GM steps to approximately solve their subproblems in S-DANE .

Partial Client Participation. Next, we turn our attention to the cross-device setting where a large number of clients (typically mobile phones) have either unstable network connection or weak computational power [24]. In such scenarios, the server typically cannot expect all the clients to be able to participate in the communication at each round. Furthermore, the clients may typically be asked to communicate only once during the whole training and are stateless [27]. Therefore, we now consider S-DANE with partial client participation and without storing any states on devices.

To prove convergence, it is necessary to assume a certain level of dissimilarity among clients. Here, we use the same assumption as in [27] to measure the gradient variance.

Definition 3 (Bounded Gradient Variance [27]) . Let f 1 , . . . , f n : R d → R be functions, and let ζ ≥ 0 . We say that { f i } n i =1 have ζ -BGV if, for any x ∈ R d and f := 1 n ∑ n i =1 f i , it holds that

Definition 3 is similar to the classical notion of uniformly bounded variance used in the context of classical stochastic gradient methods [3].

We also need the following assumption which complements Definition 1.

Definition 4 (External Dissimilarity) . Let f 1 , . . . , f n : R d → R be functions, and let s ∈ [ n ] , ∆ s ≥ 0 . Then, { f i } n i =1 are said to have ∆ s -ED (of size s ) if, for any x , y ∈ R d and any S ∈ ( [ n ] s ) , we have

where m S := f -f S and f S := 1 s ∑ i ∈ S f i .

Compared to Definition 1, the new Definition 4 quantifies the 'external' variation of any s functions w.r.t. the original function f . When each f i is twice continuously differentiable, (8) is equivalent to ∥∇ 2 m S ( x ) ∥ ≤ ∆ s for any x ∈ R d . If each f i is L -smooth, then ∆ s ≤ L for any s ∈ [ n ] . Therefore, using both Assumptions 1 and 4 is still weaker than assuming that each f i is L -smooth.

In what follows, we work with a new second-order dissimilarity measure defined as the sum δ s +∆ s . Note that δ 1 +∆ 1 = δ max and δ n +∆ n = δ .

Theorem 5. Consider Algorithm 1. Let f i : R d → R be µ -convex with µ ≥ 0 for any i ∈ [ n ] and let n ≥ 2 . Assume that { f i } n i =1 have δ s -SOD, ∆ s -ED and ζ -BGV. Let λ = 4( n -s ) s ( n -1) ζ 2 ε +2( δ s +∆ s ) , and suppose that, for any r ≥ 0 , we have

Then, to ensure that E [ f (¯ x R )] -f ( x ⋆ ) ≤ ε for a given ε > 0 , it suffices to perform at most the following number of communication rounds:

Algorithm 2 ACC-S-DANE

1: Input: λ > 0 , µ ≥ 0 , x 0 = v 0 ∈ R d , s ∈ [ n ] .

2: Set A 0 = 0 , B 0 = 1 .

3: for r = 0 , 1 , 2 , . . . do

4: Find a r +1 > 0 from the equation λ = ( A r + a r +1 ) B r a 2 r +1 .

5: A r +1 = A r + a r +1 , B r +1 = B r + µa r +1 .

6: y r = A r A r +1 x r + a r +1 A r +1 v r .

7: Sample S r ∈ ( [ n ] s ) uniformly at random without replacement.

8: for each device i ∈ S r in parallel do

9:

x

i,r

+1

≈

arg min

x

∈

R

10: x r +1 = 1 s ∑ i ∈ S r x i,r +1 .

d

F

i,r

(

x

) :=

f

i

(

x

) +

⟨∇

f

S

r

(

y

)

-∇

f

i

(

y

)

,

x

⟩

+

λ

2

∥

x

-

y

∥

.

11: v r +1 = arg min x ∈ R d { a r +1 s ∑ i ∈ S r [ ⟨∇ f i ( x i,r +1 ) , x ⟩ + µ 2 ∥ x -x i,r +1 ∥ 2 ] + B r 2 ∥ x -v r ∥ 2 } .

Theorem 5 provides the communication complexity of S-DANE with client sampling and arbitrary (deterministic) local solvers. The rate is again continuous in µ . Compared with the previous case of s = n , the efficiency now depends on the gradient variance ζ . Note that this error term gets reduced when s increases. Specifically, to achieve the O (log 1 ε ) and O ( 1 ε ) rates, it suffices to ensure that s = Θ( nζ 2 ζ 2 + nε ( δ s +∆ s ) ) . Notably, the algorithm can reach any target accuracy even when n →∞ .

Observe that the accuracy requirement (9) is the same as (6). Therefore, the discussions therein are valid in the partial-participation setting as well. In particular, if each f i is L -smooth, then the number of oracle calls to ∇ f i required at each round could be as small as O ( √ L λ ) (see Corollary 3). At the same time, it is also possible to use a stochastic optimization algorithm as a local solver (for more details, see Section C.3.1).

4 Accelerated S-DANE

In this section, we present the accelerated version of S-DANE, ACC-S-DANE (Alg. 2), that achieves a better communication complexity compared to the basic method. For simplicity, we only consider the full-participation setting and defer the partial participation to Appendix D.3.

Theorem 6. Consider Algorithm 2 with s = n . Let f i : R d → R be µ -convex with µ ≥ 0 for any i ∈ [ n ] . Assume that { f i } n i =1 have δ -SOD ( δ > 0 ). Let λ = 2 δ and suppose that, for any r ≥ 0 , we have ∑ n i =1 ∥∇ F i,r ( x i,r +1 ) ∥ 2 ≤ δ 2 ∑ n i =1 ∥ x i,r +1 -y r ∥ 2 . If µ ≤ 8 δ , then, for any R ≥ 1 ,

where D := ∥ x 0 -x ⋆ ∥ . Otherwise, f ( x R ) -f ⋆ ≤ 4 δD 2 (1+ √ µ 8 δ ) 2( R -1) for any R ≥ 1 . To ensure that f ( x R ) -f ⋆ ≤ ε for a given ε > 0 , it thus suffices to perform R = O ( √ δ + µ µ log(1+ √ min { µ,δ } D 2 ε ) ) communication rounds.

Let us consider the most interesting regime when µ ≤ 8 δ . Comparing Theorems 1 and 6, we see that ACC-S-DANE essentially extracts the square root of the corresponding communication complexity of S-DANE by improving it from ˜ O ( δ µ ) to ˜ O ( √ δ µ ) when µ > 0 , and from O ( δD 2 ε ) to O ( √ δD 2 ε ) when µ = 0 , while maintaining the same accuracy condition for solving the subproblem. Compared with ACC-EXTRAGRADIENT, the complexity depends on a better constant δ instead of δ max .

Note that we can satisfy the accuracy condition in Theorem 6 in exactly the same way as in Corollary 3.

In particular, if each f i is L -smooth, each device i needs at most O ( √ L δ ) computations of ∇ f i at each round r when using a fast algorithm for the gradient norm minimization.

Finally, let us highlight that Algorithm 2 gives a distributed framework for a generic acceleration scheme , that applies to a large class of local optimization methods-in the same spirit as in the

{

r

r

r

2

}

famous CATALYST [41] framework that applies to the case where n = 1 . However, in contrast to CATALYST, this stabilized version removes the logarithmic overhead present in the original method. Specifically, when applying Theorem 6 with n = 1 and λ = L for a smooth convex function f , we recover the same rate as CATALYST. The accuracy condition ∥∇ F r ( x r +1 ) ∥ ≤ L ∥ x r +1 -y r ∥ , or equivalently ⟨∇ f ( x r +1 ) , y r -x r +1 ⟩ ≥ 1 2 L ∥∇ f ( x r +1 ) ∥ 2 can be achieved with one gradient step x r +1 := y r -1 L ∇ f ( y r ) (see Lemma 5 in [48]).

5 Dynamic Estimation of Similarity Constant by Line Search

One drawback of Algorithms 1 and 2 is that they require the knowledge of the similarity constant δ to choose an appropriate value for λ . This similarity constant is typically unknown in practice and might be difficult to estimate. One effective solution to this problem is to dynamically adjust the coefficient λ inside the algorithms by using the classical technique of line search .

The basic idea is as follows. The server first picks an arbitrary sufficiently small constant ˜ λ as an initial approximation to the unknown 'correct' value of λ = 2 δ . Then, at every round, the server sends the current estimate of λ to each client asking them to approximately solve their local subproblem. After receiving the corresponding local solutions, the server checks a certain inequality based on the obtained information. If this inequality is satisfied, the server accepts the resulting aggregated solution and goes to the next round while decreasing λ in two times (so as to be more optimistic in the future). Otherwise, it increases λ in two times, asks the clients to solve their subproblems with the new value of λ , and checks the inequality again.

The precise versions of Algorithms 1 and 2 with line search for the full-participation setting are presented in Algorithms 3 and 4. Importantly, our adaptive schemes are not just some heuristics but are probably efficient. Specifically, their complexity estimates (in terms of the total number of communication rounds) are exactly the same as those given by Theorems 1 and 6, respectively, up to an extra additive logarithmic term of log 2 δ ˜ λ (see Theorems 27 and 28).

Another significant advantage of our adaptive algorithms is their ability to exploit local similarity, resulting in much stronger practical performance compared to the methods with fixed λ . We will demonstrate this in the next section.

6 Numerical Experiments

In this section, we illustrate the performance of our methods in numerical experiments. The implementation can be found at https://github.com/mlolab/S-DANE.

Convex quadratic minimization. We first illustrate the properties of our algorithms as applied to minimizing a simple quadratic function: f ( x ) := 1 n ∑ n i =1 f i ( x ) where f i ( x ) := 1 m ∑ m j =1 1 2 ⟨ A i,j ( x -b i,j ) , x -b i,j ⟩ where b i,j ∈ R d and A i,j ∈ R d × d . The experimental details can be found in Appendix F.1. From Figure 1, we see that S-DANE converges as fast as DANE in terms of communication rounds, but with much fewer local gradient oracle calls. ACC-S-DANE achieves the best performance among the three methods. We also test S-DANE and DANE with the same fixed number of local steps. The result can be seen in Figure E.1 where S-DANE is again more efficient. Finally, we report the strong performances of two adaptive variants (Algorithms 3 and 4 with initial ˜ λ = 10 -3 ). We see from Figure 1 that the method can automatically change λ to adapt to the local second-order dissimilarity. (We use ∥∇ f ( v r +1 ) -∇ f ( v r ) ∥ ∥ v r +1 -v r ∥ and √ 1 n ∑ n i =1 ∥∇ h i ( v r +1 ) -∇ h i ( v r ) ∥ 2 ∥ v r +1 -v r ∥ 2 to approximate the local smoothness and dissimilarity.)

Strongly-convex polyhedron feasibility problem. Wenowconsider the problem of finding a feasible point x ⋆ inside a polyhedron: P = ∩ n i =1 P i , where P i = { x : ⟨ a i,j , x ⟩ ≤ b i,j , ∀ j = 1 , . . . , m i } and a i,j , b i,j ∈ R d . Each individual function is defined as f i := n m ∑ m i j =1 [ ⟨ a i,j , x ⟩ -b i,j ] 2 + where ∑ n i =1 m i = m . We use m = 10 5 and d = 10 3 . We first generate x ⋆ randomly from a sphere with radius 10 6 . We then follow [55] to generate ( a i,j , b i,j ) such that x ⋆ is a feasible point of P and the initial point of all optimizers is outside the polyhedron. We choose the best λ from { 10 i } 3 i = -3 . We first consider the full client participation setting and use n = s = 10 . We compare our proposed methods with GD, DANE with GD [57], SCAFFOLD with control variate of option I [26], SCAFFNEW [44], FEDPROX with GD [39] and ACC-EXTRAGRADIENT [33]. The

0

25

50

75

100

Communication rounds

10

7

10

3

10 1

10 5

10 9

f(x)-

f

n=10, s=10

0

50

100

150

200

Communication rounds

10

4

10

1

10 2

10 5

10 8

10 11

f(x)-

f

n=100, s=80

0

50

100

150

200

Communication rounds

10

2

10 1

10 4

10 7

10 10

f(x)-

f

n=100, s=40

0

100

200

300

400

Communication rounds

10 0

10 2

10 4

10 6

10 8

10 10

f(x)-

f

n=100, s=10

S-DANE-GD (ours)

DANE-GD

Scaffold

Scaffnew

FedProx-GD

Acc-S-DANE-GD (ours)

AccGradSliding

0

20

40

Communication rounds

10

10

10

7

10

4

10

1

||

f

(

x

)||

ijcnn with non-iid splitting

0

20

40

Communication rounds

10

4

10

3

10

2

value

local smoothness

local dissimilarity

(Adaptive S-DANE)

0

20

40

Communication rounds

10

10

10

8

10

6

10

4

10

2

||

f

(

x

)||

ijcnn with iid splitting

2.5

5.0

7.5

10.0

12.5

Communication rounds

10

5

10

4

10

3

10

2

value

local smoothness

local dissimilarity

(Adaptive S-DANE)

Adaptive S-DANE-GD (ours)

Adaptive Acc-S-DANE-GD (ours)

DANE-GD

Scaffold

Scaffnew

FedProx-GD

result is shown in the first plot of Figure 2 where our proposed methods are consistently the best among all these algorithms. We next experiment with client sampling and use n = 100 . We decrease the number of sampled clients from s = 80 to s = 10 . In addition to our methods, we also report the performances of SCAFFOLD and FEDPROX with client sampling. From the same figure, we see that the improvement of ACC-S-DANE over S-DANE gradually disappears as s decreases.

Adaptive choice of λ . Weconsider the standard regularized logistic regression: f ( x ) = 1 n ∑ n i =1 f i ( x ) with f i ( x ) := n M ∑ m i j =1 log(1 + exp( -y i,j ⟨ a i,j , x ⟩ )) + 1 2 M ∥ x ∥ 2 where ( a i,j , y i,j ) ∈ R d +1 are features and labels and M := ∑ n i =1 m i is the total number of data points in the training dataset. We use the ijcnn dataset from LIBSVM [6]. We split the dataset into 10 subsets according to the Dirichlet distribution with α = 2 (i.i.d) and α = 0 . 2 (non-i.i.d). From Figure 4, Adaptive (ACC-)S-DANE (Algorithm 3 and 4) converge much faster than the other best-tuned algorithms for both cases. (We set the initial ˜ λ = 10 -4 for non-i.i.d and λ = 10 -5 for i.i.d respectively.)

Deep learning task. Finally, we consider the multi-class classification tasks with CIFAR10 [34] using ResNet-18 [18]. The details can be found in Appendix F.2. From Figure 3, we see that S-DANE (DL) 5 reaches 90% accuracy within 50 communication rounds while all the other methods are still below 90% after 80 epochs. The effectiveness of S-DANE on the training of other deep learning models such as Transformer requires further exploration.

7 Conclusion

We have proposed new federated optimization methods (both basic and accelerated) that simultaneously achieve the best-known communication and local computation complexities. The new methods allow partial participation and arbitrary stochastic local solvers, making them attractive in practice. We further equip both algorithms with line search and the resulting schemes can adapt to the local dissimilarity without knowing the corresponding similarity constant. However, we assume that each function f i is µ -strongly convex in all the theorems. This is stronger than assuming only µ -strongly convexity of f , which is used in some prior works. Possible directions for future research include consideration of weaker assumptions as well as empirical and theoretical analyses for non-convex problems.

0

20

40

60

80

Communication rounds

10

1

10 0

Train loss

0

20

40

60

80

Communication rounds

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test Accuary

CIFAR10-Res/glyph1197et18

0

20

40

60

80

10

2

10

1

10 0

value

local smoothness

local dissimilarity

DANE-SGD

Scaffold

Scaffnew

FedProx-SGD

FedAvg

Acknowledgments

The authors are grateful to Adrien Taylor and Thomas Pethick for the reference to [59]. The authors are thankful to the anonymous reviewers for their valuable comments and suggestions.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations , 2021. URL https://openreview . net/forum?id=B7v4QMR6Z9w .

Aleksandr Beznosikov, Martin Takáˇ c, and Alexander Gasnikov. Similarity, compression and local steps: Three pillars of efficient communications for distributed variational inequalities. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview . net/forum?id= Rvk1wdwz1L .

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review , 60(2):223-311, 2018.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Recapp: Crafting a more efficient catalyst for convex optimization. In International Conference on Machine Learning , pages 2658-2685. PMLR, 2022.

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology , 2:27:1-27:27, 2011. Software available at http: //www . csie . ntu . edu . tw/~cjlin/libsvm .

El Mahdi Chayti and Sai Praneeth Karimireddy. Optimization with access to auxiliary information. arXiv preprint arXiv:2206.00395 , 2022.

Laurent Condat and Peter Richtárik. Randprox: Primal-dual optimization algorithms with randomized proximal updates. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview . net/forum?id=cB4N3G5udUS .

Laurent Condat, Grigory Malinovsky, and Peter Richtárik. Tamuna: Accelerated federated learning with local training and partial participation. arXiv preprint arXiv:2302.09832 , 2023.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Olivier Devolder. Stochastic first order methods in smooth convex optimization. CORE Discussion Papers, 2011/70 , 2011.

Nikita Doikov and Yurii Nesterov. Contracting proximal methods for smooth convex optimization. SIAM Journal on Optimization , 30(4):3146-3169, 2020.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(61):2121-2159, 2011. URL http: //jmlr . org/papers/v12/duchi11a . html .

Alexandre d'Aspremont, Damien Scieur, Adrien Taylor, et al. Acceleration methods. Foundations and Trends® in Optimization , 5(1-2):1-245, 2021.

Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10112-10121, 2022.

Michał Grudzie´ n, Grigory Malinovsky, and Peter Richtárik. Can 5th generation local training methods support client sampling? yes! In International Conference on Artificial Intelligence and Statistics , pages 1055-1092. PMLR, 2023.

Osman Güler. New proximal point algorithms for convex minimization. SIAM Journal on Optimization , 2 (4):649-664, 1992. doi: 10 . 1137/0802032. URL https://doi . org/10 . 1137/0802032 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10 . 1109/CVPR . 2016 . 90.

Hadrien Hendrikx, Lin Xiao, Sebastien Bubeck, Francis Bach, and Laurent Massoulie. Statistically preconditioned accelerated gradient method for distributed optimization. In International conference on machine learning , pages 4203-4227. PMLR, 2020.

Zhengmian Hu and Heng Huang. Tighter analysis for ProxSkip. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 13469-13496. PMLR, 23-29 Jul 2023. URL https://proceedings . mlr . press/v202/ hu23a . html .

Anastasiya Ivanova, Dmitry Pasechnyuk, Dmitry Grishchenko, Egor Shulgin, Alexander Gasnikov, and Vladislav Matyukhin. Adaptive catalyst for smooth convex optimization. In International Conference on Optimization and Applications , pages 20-37. Springer, 2021.

Xiaowen Jiang, Anton Rodomanov, and Sebastian U Stich. Federated optimization with doubly regularized drift correction. arXiv preprint arXiv:2404.08447 , 2024.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'13, page 315-323. Curran Associates Inc., 2013.

Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Koneˇ cný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14 (1-2):1-210, 2021. URL https://arxiv . org/pdf/1912 . 04977 . pdf .

Avetik Karagulyan, Egor Shulgin, Abdurakhmon Sadiev, and Peter Richtárik. Spam: Stochastic proximal point method with momentum variance reduction for non-convex cross-device federated learning. arXiv preprint arXiv:2405.20127 , 2024.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Breaking the centralized barrier for cross-device federated learning. In Advances in Neural Information Processing Systems , 2021.

Ahmed Khaled and Chi Jin. Faster federated optimization under second-order similarity. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview . net/ forum?id=ElC6LYO4MfD .

Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local gd on heterogeneous data. arXiv preprint arXiv:1909.04715 , 2019.

Donghwan Kim and Jeffrey A Fessler. Generalizing the optimized gradient method for smooth convex minimization. SIAM Journal on Optimization , 28(2):1920-1950, 2018.

Jakub Koneˇ cn` y, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 , 2016.

Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016.

Dmitry Kovalev, Aleksandr Beznosikov, Ekaterina Borodich, Alexander Gasnikov, and Gesualdo Scutari. Optimal gradient sliding and its application to optimal distributed optimization under similarity. Advances in Neural Information Processing Systems , 35:33494-33507, 2022.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www . cs . toronto . edu/~kriz/cifar . html .

Guanghui Lan. Gradient sliding for composite optimization. Mathematical Programming , 159:201-235, 2016.

Guanghui Lan and Yuyuan Ouyang. Accelerated gradient sliding for structured convex optimization. arXiv preprint arXiv:1609.04905 , 2016.

Guanghui Lan, Yuyuan Ouyang, and Zhe Zhang. Optimal and parameter-free gradient minimization methods for smooth optimization. arXiv preprint arXiv:2310.12139 , 2023.

Bo Li, Mikkel N Schmidt, Tommy S Alstrøm, and Sebastian U Stich. On the effectiveness of partial variance reduction in federated learning with heterogeneous data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3964-3973, 2023.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Dachao Lin, Yuze Han, Haishan Ye, and Zhihua Zhang. Stochastic distributed optimization under average second-order similarity: Algorithms and analysis. Advances in Neural Information Processing Systems , 36, 2024.

Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. Advances in neural information processing systems , 28, 2015.

B. Martinet. Perturbation des méthodes d'optimisation. Applications. RAIRO. Analyse numérique , 12(2): 153-171, 1978. URL http://www . numdam . org/item/M2AN_1978__12_2_153_0/ .

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750-15769. PMLR, 2022.

Konstantin Mishchenko, Rui Li, Hongxiang Fan, and Stylianos Venieris. Federated learning under second-order data heterogeneity, 2024. URL https://openreview . net/forum?id=jkhVrIllKg .

Renato D. C. Monteiro and B. F. Svaiter. An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM Journal on Optimization , 23(2): 1092-1125, 2013. doi: 10 . 1137/110833786. URL https://doi . org/10 . 1137/110833786 .

J.J. Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de la Société Mathématique de France , 93:273-299, 1965. doi: 10 . 24033/bsmf . 1625. URL http://www . numdam . org/articles/10 . 24033/ bsmf . 1625/ .

Yu. Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming , 140 (1):125-161, 2013. doi: 10 . 1007/s10107-012-0629-5. URL https://doi . org/10 . 1007/s10107-0120629-5 .

Yurii Nesterov. Lectures on Convex Optimization . Springer Publishing Company, Incorporated, 2nd edition, 2018. ISBN 3319915770.

Yurii Nesterov and Sebastian U. Stich. Efficiency of the accelerated coordinate descent method on structured optimization problems. SIAM Journal on Optimization , 27(1):110-123, 2017. doi: 10 . 1137/16M1060182. URL https://doi . org/10 . 1137/16M1060182 .

Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends® in Optimization , 1(3): 127-239, 2014. ISSN 2167-3888. doi: 10 . 1561/2400000003. URL http://dx . doi . org/10 . 1561/ 2400000003 .

Kumar Kshitij Patel, Lingxiao Wang, Blake E Woodworth, Brian Bullins, and Nati Srebro. Towards optimal communication complexity in distributed non-convex optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 13316-13328. Curran Associates, Inc., 2022. URL https://proceedings . neurips . cc/paper_files/paper/2022/file/ 56bd21259e28ebdc4d7e1503733bf421-Paper-Conference . pdf .

Sashank J Reddi, Jakub Koneˇ cn` y, Peter Richtárik, Barnabás Póczós, and Alex Smola. Aide: Fast and communication efficient distributed optimization. arXiv preprint arXiv:1608.06879 , 2016.

R Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization , 14(5):877-898, 1976.

Anton Rodomanov, Xiaowen Jiang, and Sebastian Stich. Universality of adagrad stepsizes for stochastic optimization: Inexact oracle, acceleration and variance reduction. arXiv preprint arXiv:2406.06398 , 2024.

Abdurakhmon Sadiev, Dmitry Kovalev, and Peter Richtárik. Communication acceleration of local gradient methods via an accelerated primal-dual algorithm with an inexact prox. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview . net/forum?id=W72rB0wwLVu .

Ohad Shamir, Nati Srebro, and Tong Zhang. Communication-efficient distributed optimization using an approximate newton-type method. In International conference on machine learning , pages 1000-1008. PMLR, 2014.

M.V. Solodov and B. F. Svaiter. A unified framework for some inexact proximal point algorithms*. Numerical Functional Analysis and Optimization , 22(7-8):1013-1035, 2001. doi: 10 . 1081/NFA-100108320. URL https://doi . org/10 . 1081/NFA-100108320 .

M.V. Solodov and B.F. Svaiter. A hybrid projection-proximal point algorithm. Journal of Convex Analysis , 6(1):59-70, 1999. URL http://eudml . org/doc/120958 .

Ying Sun, Gesualdo Scutari, and Amir Daneshmand. Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation. SIAM Journal on Optimization , 32(2):354-385, 2022.

Ye Tian, Gesualdo Scutari, Tianyu Cao, and Alexander Gasnikov. Acceleration in distributed optimization under similarity. In International Conference on Artificial Intelligence and Statistics , pages 5721-5756. PMLR, 2022.

Farshid Varno, Marzie Saghayi, Laya Rafiee Sevyeri, Sharut Gupta, Stan Matwin, and Mohammad Havaei. Adabest: Minimizing client drift in federated learning via adaptive bias estimation. In European Conference on Computer Vision , pages 710-726. Springer, 2022.

Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems , 33:6281-6292, 2020.

Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems , 33:5332-5344, 2020.

Appendix

Contents

A More Related Work

In the first several years of the development for federated learning algorithms, the convergence guarantees are focused on the smoothness parameter L . The de facto standard algorithm for federated learning is FEDAVG. It reduces the communication frequency by doing multiple SGD steps on available clients before communication, which works well in practice [43]. However, in theory, if the heterogeneity among clients is large, then it suffers from the so-called client drift phenomenon [26] and might be worse than centralized mini-batch SGD [63, 29]. Numerous efforts have been made to mitigate this drift impact. FEDPROX adds an additional regularizer to the subproblem of each client based on the idea of centralized proximal point method to limit the drift of each client. However, the communication complexity still depends on the heterogeneity. The celebrated algorithm SCAFFOLD applies drift correction (similar to variance-reduction) to the update of FEDAVG and it successfully removes the impact of the heterogeneity. Afterwards, the idea of drift correction is employed in many other works [15, 62, 38, 1]. SCAFFNEW uses a special choice of control variate[44] and first illustrates the usefulness of taking standard local gradient steps under strongy-convexity, followed with more advanced methods with refined analysis and features such as client sampling and compression [20, 9, 8]. 5gCS [16] uses an approximate proximal-point step at each iteration and derives the convergence rate that is as good as SCAFFNEW, and it also supports client sampling. Later, Sadiev et al. [56] proposed APDA WITH INEXACT PROX that retains the same communication complexity as SCAFFNEW, but further provably reduces the local computation complexity. FedAC [64] applies nesterov's acceleration in the local steps and shows provably better convergence than FEDAVG under certain assumptions.

More recent works try to develop algorithms with guarantees that rely on a potentially smaller constant than L . SCAFFOLD first illustrates the usefulness of taking local steps for quadratics under Bounded Hessian Dissimilarity δ max [26]. SONATA [60] and its accelerated version [61] prove explicit communication reduction in terms of δ max under strong convexity. MIME [27] and CE-LGD [52] work on non-convex settings and show the communication improvement on δ max and the latter achieves the min-max optimal rates. ACCELERATED EXTRAGRADIENT SLIDING [33] applies gradient sliding [33] technique and shows communication reduction in terms of δ max for strongly-convex and convex functions, the local computation of which is also efficient without logarithmic dependence on the target accuracy, DANE with inexact local solvers [57, 22, 53] has been shown recently to achieve the communication dependency on δ under convexity and δ max under non-convexity. For solving convex problems, the local computation efficiency depends on the target accuracy ε . Otherwise, the accuracy condition for the subproblem should increase across the communication rounds. Hendrikx et al. [19] proposed SPAG, an accelerated method, and prove a better uniform concentration bound of the conditioning number when solving strongly-convex problems. SVRP and CATALYZED SVRP [28] transfer the idea of using the centralized proximal point method to the distributed setting and they achieve communication complexity (with a different notion) w.r.t δ . Lin et al. [40] further improves these two methods either with a better rate or with weaker assumptions based on the ACCELERATED EXTRAGRADIENT SLIDING method. Beznosikov et al. [2] uses compression to reduce the bits required to communicate and the more general problem of Variational Inequalities is considered. Under the same settings but for non-convex optimization, SABER [45] achieves communication complexity reduction with better dependency on δ max and n . Karagulyan et al. [25] proposed SPAM that allows partial client particiption.

Remark 7. Khaled and Jin [28] and Lin et al. [40] consider the total amount of information transmitted between the server and clients as the main metric, which is similar to reducing the total stochastic oracle calls in centralized learning settings. This is a particularly meaningful setting if the server prefers to or has to receive/transmit vectors one by one and can set up communications very fast. The term 'client sampling' in these works refers to sampling one client to do the local computation. However, all the clients still need to participate in the communication from time to time to provide the full gradient information. This is orthogonal to the setup of this work since we assume each device can do the calculation in parallel. In the scenarios where the number of devices is too large such that receiving all the updates becomes problematic, we consider instead the standard partial participation setting.

B Technical Preliminaries

B.1 Basic Definitions

We use the following definitions throughout the paper.

Definition 5. A differentiable function f : R d → R is called µ -convex for some µ ≥ 0 if for all x , y ∈ R d ,

If f is µ -convex, then, for any x , y ∈ R d , we have (Nesterov [49], Theorem 2.1.10):

Definition 6. A differentiable function f : R d → R is called L -smooth for some L ≥ 0 if for all x , y ∈ R d ,

If f is L -smooth, then, for any x , y ∈ R d , we have (Nesterov [49], Lemma 1.2.3)

Lemma 8 (Nesterov [49], Theorem 2.1.5) . Let f : R d → R be convex and L -smooth. Then, for any x , y ∈ R d , we have

B.2 Useful Lemmas

We frequently use the following helpful lemmas for the proofs.

Lemma 9. For any x , y ∈ R d and any γ > 0 , we have

Lemma 10 (Jiang et al. [22], Lemma 14) . Let { x i } n i =1 be a set of vectors in R d and let ¯ x := 1 n ∑ n i =1 x i . Let v ∈ R d be an arbitrary vector. Then,

Lemma 11. Let ( F k ) ∞ k =1 and ( D k ) ∞ k =0 be two non-negative sequences such that, for any k ≥ 0 , it holds that

where q ∈ (0 , 1] and ε ≥ 0 are some constants. Then for all K ≥ 1 and S K := ∑ K k =1 1 q k , we have

Proof. Indeed, for any k ≥ 0 , we have

Summing up from k = 0 to k = K -1 , we get

Dividing both sides by S K and substituting S K = 1 1 -q ( 1 q K -1) , we get the claim.

Lemma 12 (c.f. Lemma 2.2.4 in [49]) . Let ( A r ) ∞ r =0 be a non-negative non-decreasing sequence such that A 0 = 0 and, for any r ≥ 0 ,

where c > 0 and µ ≥ 0 are some constants. If µ ≤ 4 c , then for any R ≥ 0 , we have

Otherwise, for any R ≥ 1 , it holds that

Proof. Denote C r = √ µA r . For any r ≥ 0 , it holds that

Therefore, for any r ≥ 0 :

̸

When µ ≤ 4 c , by induction, one can show that, for any R ≥ 0 (see the proof of Theorem 1 in [50] for details):

When µ > 4 c , we have C r +1 -C r ≥ √ µ 4 c C r . It follows that, for any R ≥ 1 ,

Plugging in the definition of C R , we get the claims.

Lemma 13. Let { x i } n i =1 be vectors in R d with n ≥ 2 . Let s ∈ [ n ] and let S ∈ ( [ n ] s ) be sampled uniformly at random without replacement. Let ¯ x := 1 n ∑ n i =1 x i , ζ 2 := 1 n ∑ n i =1 ∥ x i -¯ x ∥ 2 , and ¯ x S := 1 s ∑ j ∈ S x j . Then,

Proof. Let ( n m ) = n ! m !( n -m )! be the binomial coefficient for any n ≥ m ≥ 1 . By the definition of ¯ x S , we have

where 1 [ E ] denotes the { 0 , 1 } -indicator of the event E . Taking the expectation on both sides, we get

Further,

Note that

Thus,

̸

̸

̸

̸

̸

Lemma 14. Suppose { f i } n i =1 satisfy ∆ s -ED of size s ∈ [ n ] and ζ -BGV with n ≥ 2 . Let f := 1 n ∑ n i =1 f i and f S := 1 s ∑ i ∈ S f i , where S ∈ ( [ n ] s ) is sampled uniformly at random without replacement. Further, let y ∈ R d be a fixed point, and let x S ∈ R d be a random point defined by a deterministic function of S . Then, for any γ > 0 , it holds that

Proof. Let h S := f -f S . Since { f i } satisfy ∆ s -ED (Definition 4), we have, in view of inequality (B.4),

Rearranging and taking the expectation on both sides, we get, for any γ > 0 ,

where the last inequality is due to (B.11). Using the fact that E S [ f ( y ) -f S ( y )] = 0 , we get the claim.

Lemma 15. Suppose { f i } n i =1 satisfy δ s -SOD of size s ∈ [ n ] . Let f S := 1 s ∑ i ∈ S f i where s ∈ [ n ] and S ∈ ( [ n ] s ) . Let v ∈ R d be a fixed point, λ > δ s , and let

where h S i := f S -f i . Let { x i } i ∈ S be a set of points in R d (such that x i ≈ arg min x F i ( x ) in the sense that ∥∇ F i ( x i ) ∥ is sufficiently small), and let ¯ x S = 1 s ∑ i ∈ S x i . Then,

Proof. Using the definition of F i , we get

Hence,

⟨∇ f i ( x i ) + ∇ h S i (¯ x S ) , v -x i ⟩ = λ ∥ v -x i ∥ 2 + ⟨∇ h S i (¯ x S ) -∇ h S i ( v ) , v -x i ⟩ + ⟨∇ F i ( x i ) , v -x i ⟩

Taking the average over i on both sides of the first display, we have

Therefore,

It follows that

where in the second equality, we use the fact that 1 s ∑ i ∈ S [ ∇ h S i (¯ x S ) -∇ h S i ( v )] = 0 .

C Proofs for S-DANE (Algorithm 1)

C.1 One-Step Recurrence

Lemma 16. Consider Algorithm 1. Let f i : R d → R be µ -convex with µ ≥ 0 for any i ∈ [ n ] . Assume that { f i } n i =1 have δ s -SOD. Then, for any r ≥ 0 , we have

Proof. By µ -convexity of f i , for any r ≥ 0 , it holds that

Recall that v r +1 is the minimizer of the final expression in x ⋆ . This expression is a (1 + µ/λ ) -convex function in x ⋆ . We can then estimate it by:

Using the convexity of f i and dropping the non-negative µ 2 λ 1 s ∑ i ∈ S r ∥ x i,r +1 -v r +1 ∥ 2 , we further get

Denote h r i := f S r -f i . Note that

where we have used:

It follows that

We now apply Lemma 15 (with x i = x i,r +1 , v = v r , S = S r and x = x r +1 ) to get

Substituting this lower bound into the previous display, we get the claim.

C.2 Full Client Participation (Proof of Theorem 1)

Proof. Applying Lemma 16 and using ∑ n i =1 ∥∇ F i,r ( x i,r +1 ) ∥ 2 ≤ δ 2 ∑ n i =1 ∥ x i,r +1 -v r ∥ 2 and λ = 2 δ , for any r ≥ 0 , we have

Rearranging, we get

where q := 1 1+ µ/λ . Applying Lemma 11 with ε = 0 and using convexity of f , we obtain

Dropping the non-negative term 1 -q 1 -q R ∥ v R -x ⋆ ∥ 2 and rearranging, we get

Plugging in the choice of λ and the definition of q , we get the claim.

Corollary 17. Under the same setting as in Theorem 1, to achieve f (¯ x R ) -f ⋆ ≤ ε , we need at most the following number of communication rounds:

Proof. Using the fact that (1 + q ) k ≥ exp( q 1+ q k ) for any q ≥ 0 and k > 0 , we get

Rearranging, we get the claim.

Proof of Corollary 3.

Proof. To achieve ∑ n i =1 ∥∇ F i,r ( x i,r +1 ) ∥ 2 ≤ δ 2 ∑ n i =1 ∥ x i,r +1 -y r ∥ 2 , for each i ∈ [ n ] , it is sufficient to ensure that ∥∇ F i,r ( x i,r +1 ) ∥ ≤ δ ∥ x i,r +1 -v r ∥ . Let x ⋆ i,r := arg min x F i,r ( x ) . Since

for any i ∈ [ n ] . According to Theorem 2 from [33] (or Theorem 3.2 from [37]), there exists a certain algorithm such that when started from the point v r , after K queries to ∇ F i,r , it generates the point v i,r +1 such that

Remark 18. Recall that F i,r is ( L + λ ) -smooth and ( µ + λ ) -convex, λ = Θ( δ ) and δ ≤ L . Suppose worker i uses the standard GD to approximately solve the local subproblem at round r starting at v r for K steps and return the last point, then by Lemma 19, we have that ∥∇ F i,r ( x i,r +1 ) ∥ 2 ≤ O ( ( L + λ ) 2 ∥ v r -x ⋆ i,r ∥ 2 K 2 ) . To satisfy the accuracy condition (C.1) , it is sufficient to make K = Θ( L δ ) local steps. Suppose worker i uses the fast gradient method, then by Theorem 3.18 from [4], we have that ∥∇ F i,r ( x i,r +1 ) ∥ 2 ≤ 2( L + λ ) ( F i,r ( x i,r +1 ) -F i,r ( x ⋆ i,r ) ) ≤ O ( ( L + λ ) 2 exp( -√ µ + λ L + λ K ) ∥ v r -x ⋆ i,r ∥ 2 ) . To satisfy the accuracy condition (C.1) , it suffices to make K = Θ( √ L + δ µ + δ log( L + δ δ )) = Θ( √ L µ + δ log( L δ )) gradient oracle calls.

Lemma 19 (Theorem 2.2.5 in [49]) . Let f : R d → R be a convex and L -smooth function. Consider the gradient method with constant stepsize:

started from some x 0 ∈ R . Then, for any K ≥ 1 , it holds that

Proof. By Theorem 2.2.5 in [49], we have that

It remains to note that the algorithm generates non-increasing ∥∇ f ( x k ) ∥ since

C.3 Partial Client Participation (Proof of Theorem 5)

The following theorem is a slight extension of Theorem 5, which includes the use of stochastic local solvers. Theorem 20. Consider Algorithm 1. Let f i : R d → R be µ -convex with µ ≥ 0 for any i ∈ [ n ] and let n ≥ 2 . Assume that { f i } n i =1 have δ s -SOD, ∆ s -ED and ζ -BGV. Let λ = 4( n -s ) s ( n -1) ζ 2 ε + 2( δ s + ∆ s ) . For any r ≥ 0 , suppose we have

for some ε > 0 , where ξ i,r denotes the randomness coming from device i when solving its subproblem at round r . We assume that { ξ i,r } are independent random variables. To reach E [ f (¯ x R ) -f ⋆ ] ≤ ε , we need at most the following number of communication rounds:

Proof. According to Lemma 16, we have for any r ≥ 0 ,

According to Lemma 14 (with S = S r , x = x r +1 and y = v r ), for any γ > 0 , we have

Adding 1 λ f ( x r +1 ) to both sides of the first display, taking the expectation over S r on both sides, substituting the previous upper bound and setting γ = s ( n -1) ε 2 ζ 2 ( n -s ) , we get

Denote all the randomness { ξ i,r } i ∈ S r by ξ r . Since ξ i,r is independent of the choice of S r for any i ∈ [ n ] , taking the expectation over ξ r on both sides of the previous display and using our assumption (C.3), we obtain

By our choice of λ , we have λ 4 -δ s +∆ s 2 -1 2 γ ≥ 0 . Taking the full expectation on both sides, we get

According to Lemma 11 and the fact that ∥ v 0 -x ⋆ ∥ = D , we get

where q := 1 1+ µ/λ . Rearranging and dropping the non-negative E [ ∥ v R -x ⋆ ∥ 2 ] , we get, for any R ≥ 1 ,

To reach ε -accuracy, it suffices to let µD 2 2[exp( µ µ + λ R ) -1] ≤ ε 2 . Rearranging gives the claim.

C.3.1 Stochastic Local Solver

Note that there exist many stochastic optimization algorithms that can also achieve the accuracy condition (C.3) such as variance reduction methods [23, 10], adaptive SGD methods [13], etc. Here, we take the simplest algorithm: SGD with constant stepsize as an example.

Corollary 21. Consider Algorithm 1 under the same settings as in Theorem 20. Further assume that each f i is L -smooth and each device has access to mini-batch stochastic gradient g i ( x , ¯ ξ i ) such that

Suppose for any r ≥ 0 , each device i ∈ S r solves its subproblem approximately by using mini-batch SGD:

where z 0 = v r and H > L + λ is the stepsize coefficient. Let ξ i,r denote ( ¯ ξ r i,k ) k . To achieve accuracy condition (C.3) for an appropriately chosen H , each device i requires at most the following number of stochastic mini-batch oracle calls:

Proof. To get (C.3), it suffices to ensure that, for any i ∈ S r , we have

For this, it suffices to ensure that

where x ⋆ i,r := arg min x F i,r ( x ) . Indeed, suppose (C.4) holds, then we have

Hence,

Plugging in this inequality into (C.4) and taking expectation w.r.t ξ i,r on both sides, we get

Rearranging gives the weaker condition.

We next consider the number of mini-batch stochastic gradient oracles required for SGD to achieve (C.4). Since F i,r is ( L + λ ) -smooth and ( µ + λ ) -convex, according to Lemma 22, we have

where ¯ z K := 1 ∑ K k =1 1 q k ∑ K k =1 z k q k and q = H -µ -λ H . Choosing now H = ( L + λ ) + 5( L + λ ) σ 2 λε , and letting the coefficient of the first part in the previous display be ≤ λ 2 , we get the claim.

10

Lemma 22. Let f be a µ -convex and L -smooth function. Consider SGD with constant stepsize H > L :

where g k := g ( x k , ξ k ) with E ξ [ g ( x , ξ )] = ∇ f ( x ) and E ξ [ ∥ g ( x , ξ ) -∇ f ( x ) ∥ 2 ] ≤ σ 2 for any x ∈ R d . Then for any K ≥ 1 , we have

where ¯ x K := 1 ∑ K k =1 1 q k ∑ K k =1 x k q k and q = H -µ H .

Proof. Indeed, for any k ≥ 0 , we have

Taking the expectation on both sides and using µ -convexity of f , we get

Applying Lemma 11, we have for any K ≥ 1 :

D Proofs for Accelerated S-DANE (Algorithm 2)

D.1 One-Step Recurrence

Lemma 23. Consider Algorithm 2. Let f i : R d → R be µ -convex with µ ≥ 0 for any i ∈ [ n ] . Assume that { f i } n i =1 have δ s -SOD. For any r ≥ 0 , we have

Proof. By µ -convexity of f i , for any r ≥ 0 , it holds that

Recall that v r +1 is the minimizer of the final expression in x ⋆ . This expression is a ( µa r +1 + B r ) -convex function in x ⋆ . By convexity and using the fact that A r +1 = A r + a r +1 and B r +1 = µa r +1 + B r , we obtain

Recall that y r = A r A r +1 x r + a r +1 A r +1 v r . Therefore,

Substituting this lower bound, using convexity of f i and dropping the non-negative µa r +1 2 1 s ∑ i ∈ S r ∥ x i,r +1 -v r +1 ∥ 2 , we further get

Denote h r i := f S r -f i . Substituting

into the previous display, we get

We now apply Lemma 15 (with x i = x i,r +1 , v = y r , S = S r and x = x r +1 ) to get

Substituting this lower bound into the previous display and using A r +1 = a 2 r +1 λ B r , we get the claim.

D.2 Full Client Participation (Proof of Theorem 6)

Proof. Applying Lemma 23 and using ∑ n i =1 ∥∇ F i,r ( x i,r +1 ) ∥ 2 ≤ δ 2 ∑ n i =1 ∥ x i,r +1 -y r ∥ 2 and λ = 2 δ , for any r ≥ 0 , we have

Subtracting A r +1 f ⋆ on both sides, we get

Recursively applying the previous display from r = 0 to r = R -1 , we get

It remains to apply Lemma 12 and plug in the estimation of the growth of A R .

Corollary 24. Under the same setting as in Theorem 6, to achieve f ( x R ) -f ⋆ ≤ ε , we need at most the following number of communication rounds:

Proof. When µ ≤ 8 δ , by using

we get

Making the right-hand side ≤ ε and rearranging, we get the claim. When µ ≥ 8 δ , it suffices to ensure that δD 2 4 R -2 ≤ ε .

D.3 Partial Client Participation

It is well known that accelerated stochastic gradient methods are not able to improve the complexity in the stochastic part compared with the basic methods [11]. A similar result is also shown for our accelerated distributed method.

Theorem 25. Consider Algorithm 2 under the same setting as in Theorem 20. Let

and suppose that, for any r ≥ 0 , we have

Denote D := ∥ x 0 -x ⋆ ∥ . Then, to ensure that E [ f ( x R )] -f ⋆ ≤ ε for some ε > 0 , we need to perform at most the following number of communication rounds:

The error term that depends on ζ 2 and ε is at the same scale as S-DANE, i.e. O ( ζ 2 ε ) when µ > 0 and O ( ζ 2 ε 2 ) when µ = 0 . Nevertheless, when s is large enough such that this second error becomes no larger than the first optimization term, then ACC-S-DANE can still be faster than S-DANE.

Proof of Theorem 25.

Proof. According to Lemma 23, we have, for any r ≥ 0 ,

According to Lemma 14 (with S = S r , x = x r +1 and y = y r ), for any γ > 0 , we have

Adding A r +1 f ( x r +1 ) to both sides of the first display, taking the expectation over S r on both sides, substituting the previous upper bound, and setting γ = s ( n -1) ε ′ 2 ζ 2 ( n -s ) with ε ′ > 0 , we get

Denote all the randomness { ξ i,r } i ∈ S r by ξ r . Since ξ i,r is independent of the choice of S r for any i ∈ [ n ] , taking the expectation over ξ r on both sides of the previous display and using the assumption that E S r ,ξ r [ 1 s ∑ i ∈ S r ∥∇ F i,r ( x i,r +1 ) ∥ 2 ] ≤ E S r ,ξ r [ λ 2 4 1 s ∑ i ∈ S r ∥ x i,r +1 -y r ∥ 2 ] + λε ′ 4 , we obtain

By choosing λ = 4 ζ 2 ( n -s ) s ( n -1) ε ′ +2( δ s +∆ s ) , we have that λ 4 -( δ s +∆ s ) 2 -1 2 γ ≥ 0 . Taking the full expectation on both sides, we get

Subtracting A r +1 f ( x ⋆ ) on both sides , summing up from r = 0 to r = R -1 and using the fact that A 0 = 0 , v 0 = x 0 and B 0 = 1 , we get

Dividing both sides by A R , setting ε ′ = ε R and using the fact that the sequence { A r } is non-decreasing, we get

We now apply Lemma 12 with c = λ to get

when µ ≤ 4 λ , and A R ≥ 1 4 λ (1 + √ µ 4 λ ) 2( R -1) when µ ≥ 4 λ . Letting these lower bounds be larger than ∥ x 0 -x ⋆ ∥ 2 ε , we get

Plugging λ = Ω ( ζ 2 ( n -s ) R snε +( δ s +∆ s ) ) into the last display and rearranging, we get the condition for R .

D.3.1 Stochastic Local Solver

Corollary 26. Consider Algorithm 2 under the same settings as in Theorem 25. Consider the same stochastic local solver used in Corollary 21. To achieve

each device i requires at most the following number of stochastic mini-batch oracle calls:

Proof. The proof is the same as that for Corollary 21.

E Dynamic Estimation of Similarity Constant by Line Search

Theorem 27. Consider Algorithm 3. Suppose that each function f i is µ -convex for some µ ≥ 0 , and { f i } n i =1 have δ -SOD for some δ > 0 . Let ˜ λ ≤ 2 δ . Then, for any R ≥ 1 , it holds that

where ¯ x R := arg min x ∈{ x 1 ,..., x R } f ( x ) . To ensure that f (¯ x R ) -f ⋆ ≤ ε for any given ε > 0 , it suffices to set

where D := ∥ x 0 -x ⋆ ∥ . Furthermore, the total number of communication rounds spent inside the r - and k -loops since the start of the algorithm and up to the moment ¯ x R has been computed is

Algorithm 3 S-DANE with line search

Proof. According to Lemma 15 (with x i = x i,r +1 , v = v r , S = [ n ] and ¯ x S = x r +1 ) and our requirement on ∥∇ F i,r ( x i,r +1 ) ∥ , whenever λ r,k ≥ 2 δ , we can estimate

Hence, at any iteration of the r -loop, the corresponding k -loop eventually terminates. Further, since λ 0 , 0 ≤ 2 δ , we can easily prove by induction that the quantities λ r,k stay reasonably bounded:

Proceeding exactly in the same way as in the proof of Lemma 16 and using the termination condition of the k -loop, we conclude that, for any r ≥ 0 , it holds that

In view of (E.1), this means that, for any r ≥ 0 ,

Following the same proof as in Appendix C.2 but with λ replaced by λ max , we obtain the first two claims.

It remains to estimate the total number of communication rounds required to construct the point ¯ x R . In order to carry out the k -loop, the server needs to compute ∇ f ( v r ) and send this vector, as well as v r and λ r, 0 , to each client, which requires O (1) communication rounds. Every iteration of the k -loop also requires O (1) communication rounds, and the total number of such iterations is k r . Thus, every iteration of the r -loop requires O ( k r +1) communication rounds. Furthermore, during the corresponding rounds, the server may also additionally compute the function value f ( x r +1 ) needed for updating the output point ¯ x r +1 ; this could be done, e.g., inside the k -loop, alongside with the computation of the gradient ∇ f ( x r +1 ,k ) needed to evaluate the 'if' condition. Thus, ¯ x R can be indeed computed after O (1) ∑ R -1 r =0 ( k r +1) communication rounds. To estimate the latter sum, observe that, by construction, for any r ≥ 0 , we have λ r +1 , 0 ≡ 1 2 λ r,k r = 2 k r -1 λ r, 0 . Taking logarithms, we see that k r = 1 + log 2 λ r +1 , 0 λ r, 0 for any r ≥ 0 . Thus,

where the final inequality is due to (E.1) and our choice of λ 0 , 0 .

Algorithm 4 ACC-S-DANE with line search

Theorem 28. Consider Algorithm 4. Suppose that each function f i is µ -convex for some µ ≥ 0 , and { f i } n i =1 have δ -SOD for some δ > 0 . Let ˜ λ ≤ 2 δ . If µ ≤ 16 δ , then, for any R ≥ 1 , it holds that

where D := ∥ x 0 -x ⋆ ∥ . Otherwise, f ( x R ) -f ⋆ ≤ 8 δD 2 (1+ √ µ 16 δ ) 2( R -1) for any R ≥ 1 . To ensure that f ( x R ) -f ⋆ ≤ ε for any given ε > 0 , it suffices to set

Furthermore, the total number of communication rounds spent inside the r - and k -loops since the start of the algorithm and up to the moment x R has been computed is

Proof. Using the same reasoning as in the proof of Theorem 27, we can justify that, at any iteration of the r -loop, the corresponding k -loop eventually terminates, and λ r,k stays uniformly bounded as in (E.1). Next, we proceed in the same way as in the proof of Lemma 23 and use the termination condition of the k -loop to obtain that, for any r ≥ 0 ,

This shows that, for any R ≥ 1 ,

To estimate the rate of growth of A R , we use the equation for a r +1 ≡ a r +1 ,k r and the bound on λ r from (E.1). This gives us, for any r ≥ 0 , the following inequality:

where B r ≡ 1 + µA r . Invoking Lemma 12, we get a lower bound on A R , and the first claim follows.

The bound on R via ε can be justified by the same argument as in the proof of Corollary 24.

To estimate the total number of communication rounds, we can follow exactly the same argument as in the proof of Theorem 27.

0

1000

2000

3000

4000

5000

6000

Communication rounds

10 0

10 1

10 2

10 3

10 4

f(x)-

f

0

10000

20000

30000

40000

50000

60000

Total number of gradient oracle calls

10 0

10 1

10 2

10 3

10 4

f(x)-

f

DANE-GD

S-DANE-GD (ours)

Algorithm 5 S-DANE (DL)

F Additional Details on Experiments

F.1 Convex Quadratics

We generate random vectors { b i,j } and diagonal matrices { A i,j } in the same way as in [22] such that max i,j {∥ A i,j ∥} = 100 and δ ≈ 5 . We use n = 10 , m = 5 and d = 1000 . We compare S-DANE and ACC-S-DANE with DANE. We use the standard gradient descent (GD) with constant stepsize 1 200 ≤ 1 2 L for all three methods as the local solver, where L is the smoothness constant of each f i . We use λ = 5 for all three methods. We use the stopping criterion ∥∇ F i,r ( x i,r +1 ) ∥ ≤ λ 2 ∥ x i,r +1 -v r ∥ for our methods ( v r becomes y r for the accelerated method). We use ∥∇ ˜ F i,r ( x i,r +1 ) ∥ ≤ λ r +1 ∥ x i,r +1 -x r ∥ for DANE.

F.2 Deep Learning Tasks

We simulate the experiment on one NVIDIA DGX A100. We split the training dataset into n = 10 parts according to the Dirichlet distribution with α = 0 . 5 . We use SGD with a batch size of 512 as a local solver for each device. For all the methods considered in Figure 3, we choose the best number of local steps among { 10 , 20 , . . . 80 } (for SCAFFNEW, this becomes the inverse of the probability) and the best learning rate among { 0 . 02 , 0 . 05 , 0 . 1 } . For this particular task, it is often observed that using control variate makes the training less efficient [38]. The possible issue comes from the fact that local smoothness is often much smaller than local dissimilarity for this task. We here remove the control variate term on line 6 in S-DANE which is defined as ⟨ x , ∇ f i ( v r ) - ∇ f S r ( v r ) ⟩ . Moreover, if we write the explicit formula for v r +1 on line 8, it becomes v r +1 = γ x r +1 +(1 -γ ) v r -η 1 s ∑ i ∈ S r ∇ f i ( x i,r +1 ) with γ ∈ [0 , 1] and η > 0 . We set γ = 0 . 99 and η to be the local learning rate in our experiment. The method can be found in Algorithm 5. Note that the only difference between it and FEDPROX is the choice of the prox-center. The best number of local steps for the algorithms without using control variates is 70 while for the others is 10 (otherwise, the training loss explodes).

F.3 Implementation

To implement Algorithms 1 and 2 (Algorithm 5 with option 2 is the same as Algorithm 1) , each device has the freedom to employ any efficient optimization algorithm, depending on its computation power and the local data size. At each communication round r , these local algorithms are called to approximately solve the sub-problems defined by { F i,r } , until the gradient norm satisfies a certain accuracy condition that is stated in the corresponding

theorems. The server only needs to perform basic vector operations. Note that G r defined in those algorithms has a unique solution so that v r +1 can be explicitly derived (in the same form as line 9 in Algorithm 5).

G Impact Statement

This paper presents work that aims to advance the field of distributed Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and the introduction, we first stately clearly what are goals of this study, which is first to achieve the best-known communication complexity and then to maintain the overall computation efficiency. We then discussed the performance of the previous state-of-the-art algorithms. Finally, we compared our methods with them (for instance in Table 1) and showed what contribution and scope we made in this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 7, we discuss the limitations of our work, in terms of the slightly stronger assumption on µ -convexity and the lack of non-convex analysis. On top of that, we also discuss potential future works building on our results.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We make clear assumptions and provide complete and concise proofs in the Appendix for all the claims written in the main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all the necessary information to reproduce our experimental results in Section 6 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the github link to the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips . cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips . cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details of the experiments to reproduce our experimental results can be found in Section 6 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Two sets of our experiments are defined in a totally deterministic setting (no randomness.) We ran the other two sets of experiments three times with different randomness seeds. The results are almost indistinguishable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information on the computing resources at the start of Section F for the deep learning experiment. The other experiments are run on a MacBook Pro laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips . cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and understood Code of Ethics and we have conducted our research in accordance with it

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide an impact statement in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our resesarch does not have any risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite and credit the original owners of the open-source datasets LIBSVM and CIFAR 10 in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode . com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of the paper extend the DANE algorithm to the stabilized-DANE (S-DANE) algorithm based on intuition from stabilized proximal point method. They also enhance the proposed S-DANE method with Monterio-Svaiter acceleration. The algorithms proposed by the authors allow for partial participation and various local solvers to some extent. Convergence analysis and experiments are provided.\n\nSTRENGTHS:\n1. The presentation of the paper is clear, logical and easy to follow.\n2. The paper proposes two novel algorithms S-DANE and its accelerated version and provides convergence analysis of them, experiments are also provided to validate the claim of the paper.\n3. The proposed algorithms allow for partial participation and various local solvers in some sense.\n\nWEAKNESSES:\n1. The partial client participation case of S-DANE and accelerated S-DANE both relies on the bounded gradient dissimilarity assumption, which is violated in many cases, and is not accurate enough when describing the effect of data heterogeneity. This raises concerns of the reviewer whether or not the algorithm is practical in the case of partial client participation.\n\n2. The paper wants to address the communication bottleneck in FL setting by reducing the total number of communication rounds the algorithm needed to reach a certain accuracy level. However, it is also important to take into account the number of bits transferred from the client to the server as they are often constrained by the bandwidth. In fact, the terms communication complexity/efficiency appeared in the paper are quite misleading.\n\n3. As the authors have suggested, the analysis is carried out in the case where each local objective $f_i$ is $\\mu$-strongly convex, which is a little restrictive. \n\n4. There are some typos (e.g., Line 265).\n\nQUESTIONS:\n1. In Table 1, why do the authors compare the number of communication rounds rather than the total amount of communication for each algorithm? Does all of those algorithms have the same communication overhead in each communication round? If the reviewer is not mistaken, some algorithms (such as SVRP) in the table work in the single client setting, is it fair to compare them directly to S-DANE in the full client participation setting?\n\n2. For S-DANE (and its accelerated version), how to determine the proper stepsize? In the convergence guarantee, $\\gamma$ depends on the unknown average Hessian dissimilarity constant $\\bar{\\delta}_s$ and the bounded gradient diversity constant $\\zeta$. Could the author illustrate on how the two constants are estimated?\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAn algorithm for distributed convex optimization with partial participation is proposed, under a similarity assumption.\n\nSTRENGTHS:\nThe proposed algorithm has complexity Acc-S-DANE has claimed communication complexity O(sqrt(delta/mu)log(1/epsilon), for the first time. (I did not check the details of the proof but the main lines look correct to me).\n\nWEAKNESSES:\n* I want you to discuss your statement \"Suppose we use the standard gradient descent as a local solver, then the number of gradient steps required to solve the subproblem increases across the iterations k\". I don't think this is correct. In fact, you should discuss the 5GCS algorithm in Grudzien et al. \"Can 5th Generation Local Training Methods Support Client Sampling? Yes!\" AISTATS 2023. 5GCS is essentially a Point-SAGA algorithm with inexact computation of the proximity operators, and different strategies to solve these subproblems are discussed. The number of GD steps does not increase, you just pay a log(L/mu) factor for that in the complexity.\n\n* You need to provide 2 tables: Table 1 in the full participation case, and a second table on existing algorithms for partial participation of $s$ clients. In particular, 5GCS and TAMUNA in \"TAMUNA: Doubly Accelerated Federated Learning with Local Training, Compression, and Partial Participation\" arXiv:2302.09832, 2023 have communication complexity $O\\left(\\sqrt{\\frac{nL}{s\\mu}}+\\frac{n}{s}\\right)\\log(1/\\epsilon)$.\n\n* The complexity of Catalyzed SVRP reported in Table 1 is not correct. This value is for a different communication model, which counts \"exchanging a vector between the server and one of the clients as a single communication step\". So, if $s$ clients communicate, the complexity is multiplied by $s$, and $s=1$ gives the best complexity in this sense, since asynchronicity is encouraged. This is clearly different from measuring the number of communication rounds, in which $s<n$ *worsens* the complexity.\n\n* You should discuss the paper Beznosikov et al. \"Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities\", arXiv:2302.07615\n\n* You should compare in your experiments to some of the methods mentioned above, in the two heterogeneous (delta = L) and homogeneous (delta<<L) regimes: does your method improve on them by exploiting the similarity, whereas other methods do not? (it might be that an algorithm also benefits from similarity, but has just not been studied under the similarity assumption in the literature).\n\n* Definition 4 is very restricting, this means that the $f_i$ are all the same up to a linear difference. This is an auxiliary result but it is not very useful in practice.\n\n\n\ntypo in several places : mu-convex -> mu-strongly convex\n\nThe paper [4] has been published in ICLR.\n\nQUESTIONS:\nDo you assume that $\\delta\\geq \\mu$? Because a complexity of $\\sqrt{\\delta/\\mu}$ is a strong statement, it means that the algorithm converges in 1 round if $\\delta \\rightarrow 0$. FedRed, for instance, has $(\\delta+\\mu)/\\mu$. Please check.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a stabilized version of DANE (S-DANE). It replaces the proximal point step with an extragradient-type step. With the well-designed subproblem criterion, the number of local gradient oracle queries improves over DANE in logarithmic terms. It further combines Monteiro-Svaiter acceleration with S-DANE, which leads to the best communication complexity. The paper also considers partial client participation.\n\nSTRENGTHS:\na) S-DANE eliminates the logarithmic term present in DANE.\n\nb) Accelerated S-DANE achieves a communication complexity of \\(O(\\sqrt{\\delta_A/\\mu \\log(1/\\epsilon)})\\), which surpasses the complexities in the existing literature.\n\nc) The clarity of the manuscript's writing is good.\n\nWEAKNESSES:\nSee \"Questions\".\n\nQUESTIONS:\n1. Apart from its resemblance to the extragradient method when \\(\\mu = 0\\), what is the underlying motivation for considering equation (3)?\n\n2. Regarding line 129 on page 4, the statement \"such computation overhead cannot be avoided\" seems imprecise. In the strongly convex case presented in [35] and both the convex and strongly convex cases described by Lan et al., 2023, the number of gradient steps required to solve the subproblem does not necessarily increase with \\(k\\).\n\nLan, G., & Li, Y. (2023). A Novel Catalyst Scheme for Stochastic Minimax Optimization. arXiv preprint arXiv:2311.02814.  \n\n3. Some suggested references for extragradient:\n\n\nKorpelevich, G. M. (1976). The extragradient method for finding saddle points and other problems. Matecon, 12, 747-756.\n\nNemirovski, A. (2004). Prox-method with rate of convergence O (1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1), 229-251.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of distributed optimization under second-order similarity under (strong) convexity and smoothness. The paper proposes a new algorithm, Stabilized DANE, which (a) matches the best-known communication complexity under Hessian similarity while (b) requiring that local computation problems are solved only up to an approximately constant accuracy (i.e. not an accuracy that depends polynomially on $1/\\epsilon$ where $\\epsilon$ is the desired solution accuracy). The authors also consider partial participation, where only a subset of the clients is available at any given time, and also provide an accelerated version of Stabilized DANE.\n\nSTRENGTHS:\n- The paper is written clearly and the new algorithms are well-motivated. The proofs are easy to follow.\n- The stabilization technique is elegant and clearly obtains both theoretical improvements (mostly in local computation complexity, as far as I can tell) and in practice (as shown in Section 5). \n- The algorithms developed build on DANE, which is well-known and already a strong algorithm.\n\nWEAKNESSES:\n1. \"It is necessary to assume a certain level of dissimilarity\" (lines 208-212), I'm not actually sure that's _necessary_. There are certainly upper bounds without this assumption. Why is it necessary? Relaxed assumptions like expected smoothness are often enough even in this setting.\n2. The complexities given for Acc-SVRS/Catalyzed SVRP in Table 1 are not consistent with how the other rates are presented in this work-- the $n$ factors should not be there, since in Table 1 for every other work full participation is assumed anyway (i.e. they should all have an $n$ factor multiplied). Which brings me to another point: What is the advantage of S-DANE over SVRS/SVRP? Is it only the higher efficiency of local computation?\n3. The paper right now has a few different settings (full participation, partial participation) that contrast to other settings in prior work. Remark 6 attempts to explain the difference but I think this should be explained more clearly and placed in the main body of the paper.\n\nQUESTIONS:\n- Please address my concerns in the weaknesses section.\n- Is the hessian similarity actually smaller than the smoothness over the training trajectory in the CIFAR10 optimization task?\n- Can you derive theory for the variant of your method with no control variates that you use in the deep learning experiment? Does it just reduce to stabilized prox then?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new variant of the existing DANE algorithm for federated learning. The paper integrates the stabilized proximal point method into DANE to form S-DANE. Convergence analyses are provided showing that S-DANE has the same rate as DANE but with better dependency on the communication round. The accelerated variant of S-DANE is also proposed which achieves better rates than S-DANE. Numerical experiments are able to show the advantage of S-DANE and Accelerated S-DANE compared to existing works.\n\nSTRENGTHS:\n- The related work discussion of the paper is good. I am able to see how the paper is positioned among existing works.\n- I have not seen stabilized proximal update applied to federated learning (FL) so the idea in the paper appears to be new for FL.\n- S-DANE and accelerated S-DANE are able to achieve the best-known rate for their corresponding setting.\n- The claims are supported by theoretical analyses.\n- The convergence results are done under both full and partial client participations.\n- Extensive experiments showing the advantage of S-DANE/Accelerated S-DANE.\n\nWEAKNESSES:\n- The design of the algorithm is a little bit unclear, see questions below.\n- Somehow I only see the full 8 algorithm in the left-most plot in Figure 2, while in the remaining 3 there are only about 4 algorithms displayed in the plots, i.e. I cannot see Snaffnew, AccGradSliding, DANE-GD, GD.\n\nQUESTIONS:\n1. In both algorithm 1 and algorithm 2, aseach device $i \\in S_r$ performs the update in parallel, I do not see how $\\nabla f_{S_{r}} (v^r)$ or $\\nabla f_{S_{r}} (y^r)$ is computed since it requires the evaluation of the gradient of $f_j$ for $j \\in S_r, j\\neq i$?\n\n2. In Figure 4, Scaffnew actually performs worse than Scaffold, this is somewhat surprising given the experiments in Mishchenko et al. (2022). What is your explanation on this?\n\nLIMITATIONS:\nThe paper adequately discusses the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-23"}