Stabilized Proximal-Point Methods for Federated Optimization
Authors: Xiaowen Jiang
Anton Rodomanov
Sebastian U Stich
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Convex Optimization
Distributed Optimization
Proximal-Point Method
Second-order Dissimilarity
Acceleration
Line search
Primary area: optimization
TL;DR: We propose novel federated optimization algorithms (both basic and accelerated) that are efficient in terms of both communication and computation.

Abstract: In developing efficient optimization algorithms, it is crucial to account for communication constraints&mdash;a significant challenge in modern Federated Learning. 
    The best-known communication complexity among non-accelerated algorithms is achieved by DANE, a distributed proximal-point algorithm that solves local subproblems at each iteration and that can exploit second-order similarity among individual functions.
    However, to achieve such communication efficiency, the algorithm
    requires solving local subproblems sufficiently accurately resulting in slightly sub-optimal local complexity.
    Inspired by the hybrid-projection proximal-point method, in this work, we propose a novel distributed algorithm S-DANE. Compared to DANE, this method uses an auxiliary sequence of prox-centers while maintaining the same deterministic communication complexity. Moreover, the accuracy condition for solving the subproblem is milder, leading to enhanced local computation efficiency. Furthermore, S-DANE supports partial client participation and arbitrary stochastic local solvers, making it attractive in practice. We further accelerate S-DANE and show that the resulting algorithm achieves the best-known communication complexity among all existing methods for distributed convex optimization while still enjoying good local computation efficiency as S-DANE.
    Finally, we propose adaptive variants of both methods using line search, obtaining the first provably efficient adaptive algorithms that could exploit local second-order similarity without the prior knowledge of any parameters.

Stabilized Proximal-Point Methods for Federated Optimization
Xiaowen Jiang Saarland University and CISPA xiaowen, jiangócispa.de
Anton Rodomanov CISPA anton rrttoooroodaaooeeisspp.....
Sebastian U. Stich CISPA* stich@cispa.de
Abstract
In developing efficient optimization algorithms, it crucial to account for com- munication constraints significant challenge in modern Federated Learning. The best-known communication complexity among con-accelerated algorithms is achieved by DANE, distributed proximal-poiie algorithm that solves local subproblems at each iteration and that can exploit second-order similarity among individual functions. However, to achieve such communication efficiency, the algorithm requires solving local subproblems sufficiently accurately resulting slightly sub-optimal local complexity. Inspired by the hybrid projeectoonooonn proximal- point method, in this work, we propose a novel distributed algorithm S-DANE. Compared to DANE, this method uses auxiliary sequence of prox-centers while maintaining the same deterministic communication complexity. Moreover, the accuracy condition for solving the subproblem is milder, leading enhanced local computation efficiency. Furthermore, S-DANE supports partial client participation and arbitrary stochastic local solvers, making it attractive in practice. We further accelerate S-DANE and show that the resulting algorithm achieves the best-known communication complexity among all existing methods for distributed convex optimization while still enjoying good local computation efficiency as S-DANE. Finally, we propose adaptive variants of both methods using line search, obtaining the first provably efficient adaptive algorithms that could exploit local second-order similarity without the prior knowledge of any parameters.
Introduction
Federated learning is rapidly emerging large-scale machine learning framework that allows training from decentralized data sources (e.g. mobile phones or hospitals) while preserving basic privacy and security [43, 24, 31]. Developing efficient federated optimization algorithms becomes one of the central focuses due to its direct impact on the effectiveness of global machine learning models.
One the key challenges in modern federated optimization is to tackle communication bottle- necks [32]. The large-scale model parameters, coupled with relatively limited network capacity and unstable client participation, often make communication highly expensive. Therefore, the pri- mary efficiency metric of federated optimization algorithm is the total number of communication rounds required to reach a desired accuracy level. two algorithms share equivalent communication complexity, their local computation efficiency becomes the second important metric.
The seminal algorithm DANE [57] is an outstanding distributed optimization method. It achieves the best-known deterministic communication complexity among existing conccceeeddtedd algorithms (on the server side) [22]. This efficiency primarily hinges upon mild precondition regarding the Second- order Dissimilarity 8. In numerous scenarios, like statistical learning for generalized model [19] and
"CISPA Helmholtz Center for Information Security, Saarbrüccenn, Germany
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

semi-supervised learning [7], 8 tends to be relatively small. However, to ensure such fast convergence, DANE requires each iteration subproblem to be solved with sufficiently high accuracy. This leads to sub-optimal local computation effort across the communication rounds, which is inefficient in practice. FEDRED [22] improves this weakness by using double regularization. However, this technique is only effective when using gradient descent as the local solver but cannot easily be combined with other optimization methods. For instance, applying local accelerated gradient or second-order methods cannot improve its local computation efficiency. Moreover, it is also unclear how to extend this method to the partial client participation setting relevant to federated learning.
On the other hand, the communication complexities achieved by the current accelerated methods typically cannot be directly compared with those attained by DANE, as they either depend on sub-optimal constants or additional quantities such as the number of clients n. The most relevant and state-of-the-ar algorithm ACC-EXTRACE ADIENNT ADIENT [33] achieves a better complexity in terms of the accuracy but with dependency on the maximum Second-order Dissimilarity 6max which can in principle be much larger than 8 (see Remark 2). Unlike most federated learning algorithms, such as FEDAVG [43], this method requires communication with all the devices at each round to compute the full gradient and then assigns one device for local computation. In contrast, FEDAVG parallel and utilize the standard averaging to
and similar algorithms perform local computations compute the global model. The follow-up work AccS [40] applies variance reduction to Acc- EXTRAGRADIENT which results in less frequent full gradient updates. However, the communication complexity incurs a dependency on n which is prohibitive for cross-device setting [24] where the number of clients can be potentially very large. Thus, there exists no accelerated federated algorithm that is uniformly better than DANE in terms of communication complexity.
Contributions. In this work, we aim develop federated optimization algorithms that can achieve the best communication complexity while retaining efficient local computation. To this end, we first revisit the simple proximal-point method on a single machine. The accuracy requirement for solving the subproblem defined in this algorithm is slightly sub-optimal. Drawing inspiration from proximal point method for finding zeroes of a maximal monotone operator [59], we observe that using a more stabilized prox-center improves the accuracy requirement. We make the following contributions:
hybrid projection-proximal
We develop novel federated optimization algorithm S-DANE that achieves the best-known communication complexity (for non-accelerated methods) while also enjoying improved local computation efficiency over DANE [57].
We develop an accelerated version of S-DANE based on the Monteiro-S -vaittr acceleration [46]. The resulting algorithm ACC-S-DANE achieves the best-known communication complexity among all existing methods for distributed convex optimization.
Both algorithms support partial client participation and arbitrary stochastic local solvers, making them attractive in practice for federated optimization.
We provide simple analysis for both algorithms. We derive convergence estimates that are continuous in the strong convexity parameter
We propose adaptive variants of both algorithms using line-search in the full client participation setting. The resulting methods achieve the same communication complexity (up logrithmic factor) as non-adaptive ones without requiring knowledge of the similarity constant.
We illustrate strong practical performance of our proposed methods in experiments.
See also Table for summary of the main complexity results in the full-participation setting.
Related Work. Moreau first proposed the notion of the proximal approximation of a function [47].
Based on this operation, Martinet developed the first proximal-point method [42]. This method was first accelerated by Güller [17], drawing the inspiration from Nesterov's 's Fast gradient method [49].
Later, Lin et al. [41] introduced the celebrated CATALYST framework that builds upon Güller's acceleration. Using CATALYST acceleration, a large class of optimization algorithms can directly achieve faster convergence. In a similar spirit, Doikov and Nesterov [12] propose contracting proximal methods that can accelerate higher-order tensor methods. While Güller's acceleration has been successfully applied to many settings, its local computation is sub-optimal, Specifically, when minimizing smooth convex functions, logarithmic dependence on the final accuracy is

---

Algorithm
Victors
General
pairmagly
Guarantov
per rrand
comm munds
# local gralient querao
A ceamn munds
# local gradicat guoies
log 2 12
expertaioon
SONATA [1]]
utkzown
Enknown
log
deteriinistic
DANB [57]
1ng
log
lout lng
determinatic
FalRol 132)
log
S-DANE (dii work Alg, 1)
hog of
y f oF
determiriatic
Incrzat AD=SONATA 1S]]"
unkeown
unknewn
Eana log Aqm loy
detemmirisiic
Excagradinn
Flaue lox
determitistic
Cuntyzed SVKP [S]
wp.1 up.
unkrown
snbnewn
expectation
AccSVRS (10)
(n + nf
he
stoyy?
ineeppetaiion
AuN-S-DANE work, Aly.
deteriniiticc
aaenan and Pantten ochern ormmn menck reprosees repeowe sonther undd commennti sines regeied moch sonrwy The wiocal gridien guedee replaodd VJ(K")
rapecisd
RANDPICK We
bew, of ooke 1.
SSINATA, buocsee Acc-SIONATA, Arc- ateeet and SYRS ony reed sorm areg connecry
"ooest procena local mope usd SONATA
which ueal inioandio betwees tee conurnsiittion counds ou cotuiions We voles Ronaak 7 detals
NCCSVRS
complesity opuaaket
(ctaked and li (24) awamee east evasestzes of proceral operater fordh ormoenece of asalous
Table I: Summary of the worst-case convergence behaviors of the considered distributed optimization methods (in the BigO-notation) assuming each fi is L-smooth and s-convex with μ. S @(5). where 8, 6max. < are defined in (2), Remark and (3), and D = ||x" x' I| The ## local gradient queries' column represents the number of gradient oracle queries required between
communication rounds to achieve the corresponding complexity, assuming the most efficient local first-order algorithms are used. The column 'Guarantee' means whether the convergence guarantee holds in expectation or deterministically. The suboptimality is defined via f f(x ) f for strongly-conee and general convex functions where x is a certain output produced by the algorithm after R number of communications.
||x I] and
10
10
10a
10a
107
107
100
101
107
10-1
10-1
smoothiness local desimibrtty (Adaptiive S-DANE)
10-
16-1
100
200
300
400
20000
40000
60000
10
15
25
Communicatonnaa rounds
Total number of gradient oracie calls
Communication rounds
Adeptive Are-S-DANR-GD (ours)
Ad Adiptice S-DANE-GD (ours)
Arc-S-DANI-GD (our))
5-DANE-GB (cum)
DANE-GO
Figure 1: Comparison of S- S-DANE and ACC-S-DANE with DANE for solving a convex quadratic minimization problem. All three methods use GD as the local solver. S-DANE has improved local computation efficiency than DANE while ACC-S-DANE further improves the communication complexity. Finally, the adaptive variants can leverage local dissimilarities achieve better performance. (The definitions local smoothness and dissimilarity can be found Section 6.)
incurred its local computation complexity [12]. Solodov and Svaiter [59] proposed HYBRID PROJECTION-PROXIM POINT method that allows significant relaxation of the accuracy condition for the proximal-poine subproblems. More recent works such as ADAPTIVE CATALYST [21] and RECAPP [5] successfully get rid of the additional logarithmic factor for accelerated proximal-point methods as well.
Another type of acceleration based on the proximal extra-gradient method was introduced by Monteiro and Svaiter [46]. This method is more general in the sense that it allows arbitrary local solvers and the convergence rates depend on the these solvers. For instance, under convexity and Lipschitz second- (1/3.3.5) (1.3.5) by using Newton-type method. Moreover, Acceleration recovers the rate of Güller's
order derivative, the rate can be accelerated O
when the gradient method is used, Monteiro-Svann acceleration and accuracy requirement for the inexact solution weaker. For minimizing smooth convex functions, one gradient step is enough for approximately solving the local subproblem [48]. This technique has been applied to centralized composite optimization, known as gradient sliding [35, 36, 33). A comprehensive study of acceleration be found in [14].

---

We defer the literature review on distributed and federated optimization algorithms to Appendix A.
2 Problem Setup and Background
We consider the following distributed minimization problem:
min XER
f(x)
(1)
where each function f:: R - R is p-strongly convex² for some μl 2 O. We focus on the standard federated setting where the functions {fi] are distributed among n devices. The server coordinates the global optimization procedure among the devices. In each communication round, the server broadcasts certain information to the clients. The clients, in turn, perform local computation in parallel based on their own data and transmit the resulting local models back to the server to update the global model.
Main objective: Given the high cost of establishing connections between the server and the clients, our paramount objective is minimize the number of required communication rounds to achieve the desired accuracy level. This represents central metric in federated contexts, as outlined in references such as [43, 27]. Secondary objective: Efficiency in local computation represents another pivotal metric for optimization algorithms. For instance, if two algorithms share equivalent communication complexity, the algorithm with less local computational complexity is the more favorable choice.
,n]. For a set A and an integer S 8 S [A], we use (2) to denote the power set comprised of all s-element subsets of A. Everywhere in this paper, ||-Il denotes the standard Euclidean norm (or the corresponding spectral norm for matrices). We assume problem (1) has a solution which we denote by x"; the corresponding optimal value is denoted by f*. For set S E (lr)). we use fs = Eies fi to denote the average function over this set. We use to denote the index of the communication round and use the superscript and subscript to denote the global and local models, respectively; for instance, x" represents the global model at round r while Xi,r is the local model computed by device i at round r.
Notation: We abbreviate [n] == {1,2,...,
denote the index of the local step. Finally, we
2.1 Proximal-Pone Point Methods on Single Machine
In this section, provide brief background on proximal-poin poont methods [47, 42, 54, 51], which are the foundation for many distributed optimization algorithms.
Proximal-Poie Method. Given an iterate xk. the method defines X to be (approximate) minimizer of the proximal- -oiit subproblem:
Xk+1 A arg min FF((x) = f(x) + 11xx x.l1?]
XER
for an appropriately chosen parameter À 2 0. This parameter allows for a trade-off between the complexity of each iteration and the rate of convergence. If X O. the subproblem in each iteration is as difficult as solving the original problem because no regularization is applied. However, as increases, more regularization is added, simplifying the subproblem. For example, for convex function f. the proximal-point method guarantees f(xx) f < O ||x - xx11²). where Rx + Σ Xx [51, However, to achieve such a convergence rate, the subproblem (2) has to be solved fairly high accuracy [54, 5]. For instance, the accuracy condition should either depend on the target accuracy E, increase with k: [X++)() and the standard gradient descent is used as a local solver, the number of gradient steps required solve the subproblem has logarithmic dependence on the iteration counter k. The same issue also arises when considering accelerated proximal point methods [12, 17].
O ||x++1 xxll) [58]. Indeed,
when if is Lipschitz-smool
Stabilized Proximal- Method. One of the key insights that use in this work is the observation that using a different prox-center makes the accuracy condition of the subproblem weaker.
If μ 0. then f; is assumed be simply convex.

---

Algorithm S-DANE: Stabilized DANE
Input: >> 0. 1 0, [r]].
E R".
2: 00,1=2,,,.....
3: Sample S, E ((.) uniformly at random without replacement.
4: for each device i E S, in parallel do
5:
Xi,r+1 22 arg min (xx(()) f= (Vss) -V/(") + -
6:
x x+ E,es,, X1,++1-
7:
v minxmnd (V
2|1x v"1
The stabilized proximal-poin method defines
Xk+1 arg min Fx (x) = [(x) + X vxll vxll?3
Vk+1 = arg min (
+|x x
-
vxll v
where À 2 0 is parameter of the method and μ 2 0 is the strong-convexic constant of f. This algorithm updates the prox-center V& by performing an additional gradient step in each iteration. For instance, when μ = 0, the prox-center is updated Vk+1 Vv V fx which is often referred to as an extra-gradde gradient update. The stabilized proximal-poing method has the same convergence rate the original method (2) but requires only that F>
Vv []) As result, there is no extra logarithmic factor of & in the oracle complexity estimate when / is L-smooth. Specifically, by setting À ( the previous condition can be satisfied by choosing x as the result of one gradient step from V& [48]. This shows that the stabilized proximal-poine method has better overall oracle complexity than the standard proximal-poin method (c.f. Theorem I for the special case n = 1). It is worth noting that the former algorithm originates from the hybrid cojeectonnproximaa point algorithm [59] designed for solving the more general problem of finding zeroes of a monotone operator. In this work, we apply this algorithm in the distributed setting (n 2 2).
O(al|x++1
2.2 Distributed Proximal-Point Methods
The proximal-ponn method can be adapted to solve the distributed optimization problem (1). This is the idea behind FEDPROX [39]. It replaces the global proximal step (2) by n subproblems defined as Xi,+1 = arg minn((ftt + ||x - x" ["], which can be solved independently on each device, followed by the averaging step x +++ 1 E 1 xr,++1 Here we switch the notation from to r to communication round in
highlight that one iteration of the proximal-point method corresponds this setting. To ensure convergence, FEDPROX has to use a large À that depends on the target accuracy well the heterogeneity among (fi), which slows down the communication efficiency [39]. DANE [57] improves this by incorporating a drift correction term into the subproblem:
Xi.++ == arg min f((x (VS(x") V/i - x²]
(4)
x
Consequently, DANE allows choose a much smaller in the algorithm. Moreover, it can exploit second-order similarity and achieve the best-known communication complexity among non- accelerated methods [22]. However, in the original proximal-point method, the subproblem needs to be solved sufficiently accurately leading to an extra logarithmic factor in the oracle complexity estimate, To overcome this problem, we propose new algorithms described in the following section.
3 Stabilized DANE
We now describe S-DANE (Alg. 1), our proposed federated proximal-poing method that employs stabilized prox-centers in its subproblems. During each communication round r, the server samples subset of clients uniformly at random and sends v" to these clients. Then the server collects V (v") from these clients, computes V.s, (v") and sends V/s, (v") back to them. Each device in the set then calls an arbitrary local solver (which can be different on each device) to approximately solve its local subproblem. Finally, each device transmits V V and x back the server which then aggregates these points and computes the new global model.
5

---

As DANE, S-DANE can also achieve communication speed-up if the functions among devices are similar each other. This is formally captured by the following assumption.
Definition 1 (Second-order Dissimilarity.. Let , I + R be functions, and let E [n]. 5g O. Then, (ft)=1 are said to have 6--SOD (of size for any x,y RRd and any S E ((r) holds that
Vh - (y) S x-
(5)
where big ss Ji and Js = Eies Jt-
Definition 1 quantifies the dissimilarity between any functions and their average, i..., the "internal" variation between any s functions. Clearly, 51 = 0, and, when s = n, we recover the standard notion of second-order dissimilarity introduced in prior works:
Definition 2 (6-SOD [28, 40, 22]). 5-SOD = 6,D of size n.
When each function f; is twice continuously differentiable, simple sufficient condition for (5) is that 1 s Eiss II 2/ (x)]]² I 8², for any x E Rd. However, this is not a necessary condition (see [22] for more details).
The quantity V(x: y) the left-hand side of (5) can interpreted as the variance of the gradient difference estimator (x( - Vf()), where is chosen uniformly at random from S. In partic- ular, it can be rewritten as V(x, y) = 1/2 8 [V - fl - [V - fss each function fi is smosthh, then 8, S ( ESe L for any E [n²] However, in general,
condition (5) is weaker than assuming that each fi is hipschitz-smoote
Full Client Participation. We first consider the cross-silo setting where all the clients are highly reliable (s n). This typically the case with organizations and institutions having strong computing resources and stable network connection [24].
Theorem 1. Consider Algorithmm 1 with s = n. Let fi: Rd + be p-convex with μ 2 0 for any i E [n). Assume that {J:) have 5-SOD. Let X = 20 and suppose that, for any r 2 0, we have
2²
Σ
Fii, 1]1
x1,+1 v²1
(6)
Then, for any R 2 1, it holds that?
n1)²
811²
f(x#) - 2[((1 +
1] S
11 nR
R
28
where x² = Σm-1" pf Σmm x" for p = 1+ t. and D = ||x x"
||. To obtain f(x²) f S
for a given E >> O, it thus suffices to perform R = ( log | + aD
communication rounds.
Theorem provides the convergence guarantee for S-DANE in terms of the number of communica- tion rounds. Note that the rate is continuous in p.
Remark 2. Some previous works express complexity estimates in terms of another constant, 6max defined by the inequality [/V,,x)) Vhy S ōwaxl| miaxx||xx yll holding for any x, y BR and any i E [n], where h = fs. (See for instance the second line in Table 1). Note that our 5 is always not larger than 8max, and can in principle be much smaller (up to v times).
The proven communication complexity is the same as that of DANE [22]. However, the accuracy condition is milder. Specifically,, to achieve the same guarantee, DANE requires En F (xir++1111 <
- 1²), where For is defined as in (4), which incurs an r² overhead in the denominator, as in the general discussion on proximal- point methods in Section 2. The next corollary shows that local computations in S S-DANE could be computationally very efficient.
Herr, for μ = 0, the expression after the first inequality should be understood as the corresponding limit when + 0; > 0, which is exactly the expression after the final inequality. The same remark applies to all other similar results.

---

Corollary 3. Consider the same setting as in Theorem 1. Further, assume that each fü is L-smooth. To ensure (6) with certain first- algorithm, each device i needs to perform at most O ( computations of fi at each round
Remark 4. Particular examples of algorithms that could be used to achieve the result from Corollary 3 are OGM-OG by Kim and Fessler [30] and the accumulative regularization method by Lan et al. [37], both designed for the fast minimization of the gradient norm. For the standard Gradient Method (GM), the required number of oracle calls is O O()) The standard Fast Gradient Method w+8
(FGM) [49] can further decrease the complexity to Ogy L log () (see Remark 18 for details).
Thus, each device can run a constant number of standard local (F)GM steps to approximately solve their subproblems in S-DANE.
Partial Client Participation. Next, we turn our attention to the cross-device setting where a large number of clients (typically mobile phones) have either unstable network connection weak computational power [24]. In such scenarios, the server typically cannot expect all the clients to be able to participate in the communication at each round. Furthermore, the clients may typically be asked to communicate only once during the whole training and are stateless [27]. Therefore, we now consider S-DANE with partial client participation and without storing any states on devices.
To prove convergence, it is necessary to assume certain level of dissimilarity among clients. Here, we use the same assumption as in [27] to measure the gradient variance.
Definition 3 (Bounded Gradient Variance [27]). Let fi...., We say that (ft) have C-BGV if. for any x E Rd and = E
.......e fn' Rd + R be functions, and let Ç 2 0.
fo. it holds that
V V 52
Definition is similar to the classical notion of uniformly bounded variance used in the context of classical stochastic gradient methods [3].
We also need the following assumption which complements Definition
Rd + IR be functions, and let E [n]. A. 2 O. Definition& 4 (External Dissimilarity). Let f1,..., fn :Rd
Then, (=) are said to have A,-ED (of size for any x, y R&d and any S E (In)), have
Vm)(x)) Vms(y)ll = AxIxx- yll.
(8)
where ms == $ and Is == Eies Ji-
Compared to Definition 1, the new Definition 4 quantifies the "external" variation of any s functions w.r.t. the original function f. When each S, is twice continuously differentiable, (8) is equivalent to [|V (x) || < A, for any x E Rdd each f. is L-smooth, then A, S L for any E [n]. Therefore. using both Assumptions and 4 is still weaker than assuming that each fi is L-smooth.
In what follows, work with Note that 5N + A = Einax and 5n + An = 6. Theorem 5. Consider Algorithm Let ft Rdd R be y-convex with μ 2 0 for any E [m] and let 2 2. Assume that {J:) huve 5,-SOD, A, and C-BGV. Let X s(n-1) and suppose that, for any O, we have
second-order dissimilarity measure defined the sum 8, + A..
4[n-s) c² 2(5, + A.),
2²1 1
Σ IIV (X, VES.
Σ
|x,,t1 - 1
(9)
4 iES,
Then, to ensure that [x f(x)) I E for a given E > O, it suffices to perform at most the following number of communication rounds:
R=0((
A. +
n - 8 2
log 1 1+ 1D22)
E ) < A
(8. + D.)
+ - 22
R R-
f n 1 sEj
E
1 SE
where R
D x'x with == 1 + S, and
||x0 x+1
5m p
7

---

Algorithm 2 ACC-S-DANE
Input: 0. p 2 0,
E [n].
2: Set Ap = Bo==
3: f 0,1,,,,,....dd
4: Find ar+1 > 0 from the equation À (Artar1)88
of+1
5:
Art1 Ar 1a-taan ar11, Br By + par41.
6:
y" = A11,
Ar+1
7:
Sample S, ((a)) uniformly at random without replacement.
8:
for each device E S, in parallel do
9:
xi,++1 2 arg minx (x.(()) = (Vss)) Vfi(y"),, + + y"
10:
x = E X-
11:
v arg min sill E
(Vf(( xirti -
By |xx - v
Theorem provides the communication complexity of S-DANE with client sampling and arbitrary (deterministic) local solvers. The rate is again continuous in μ. Compared with the previous case of 8 = n, the efficiency now depends on the gradient variance Ç. Note that this error term gets reduced when increases. Specifically, to achieve the Oloog )) and O(2) rates, it suffices to ensure that
O 0( Notably, the algorithm can reach any target accuracy even when n + co.
nc²
Observe that the accuracy requirement (9) is the same as (6). Therefore, the discussions therein are valid in the partial-partichan participation setting as well. In partieular, if each f/ is L-smooth, then the number of oracle calls Vfs required each round could be as small as O(y/ () (see Corollary 3). At the same time, it is also possible to use a stochastic optimization algorithm as local solver (for more details, see Section C.3.1).
Accelerated S-DANE
In this section, we present the accelerated version of S-DANE, AcC-S-DANE (Alg. 2), that achieves a better communication complexity compared to the basic method. For simplicity, we only consider the full- call-participttion setting and defer the partial participation to Appendix D.3.
Theorem 6. Consider Algorithm 2 with s n. Let : Rd + R be μ-convex with μ. 2 for any i E [n). Assume that (f)] have 8-SOD (6 > 0). Let À = 28 and suppose that, for any 2 0, we
have Σ /V,,, X 1,11))1 S 82 E Ex=I| |1, y"1l 112. If p S 85, then, for any 1,
2μD2
45D2
f(x f [(1 (1+ (1-
R
R
where D 1|x x- Otherwise, - (1+
46D²
for any R 2 1. To ensure that
(2(RR-) 1)
- S f.ror a given e > 0, it thus suffices to perform R = O(V 6+ log(1 + min(p.))
communication rounds.
Let us consider the most interesting regime when S 85. Comparing Theorems and 6, we see that Acc-S-DANE essentially extracts the square root of the corresponding communication complexity of S-DANE by improving it from y to 0 when μ 0, and from O ( to O 6Da
when μ 0, while maintaining the same accuracy condition for solving the subproblem. Compared with ACC-EEXTRAGRADIEENT the complexity depends on a better constant 5 instead of 8max-
Note that we can satisfy the accuracy condition in Theorem 6 in exactly the same way as in Corollary 3.
In particular, if each fo is L-smooth, each device i needs at most O(y (t) computations of V at each round when using a fast algorithm for the gradient norm minimization.
Finally, let us highlight that Algorithm 2 gives a distributed framework for a generic acceleration scheme, that applies to large class of local optimization methods-ing in the same spirit as in the

---

famous CATALYST [41] framework that applies to the case where n = 1. However, in contrast to CATALYST, this stabilized version removes the logarithmic overhead present in the original method. Specifically, when applying Theorem 6 with n = 1 and À = I. for smooth convex function f. we recover the same rate as CATALYST. The accuracy condition Fx S T.||x*+1 y"Il, or
equivalently [x(x++1), - x*+1) 2 1 [Vx(+++)) can be achieved with one gradient step
x** y L Vyy") (see Lemma 5 in [48]).
5 Dynamic Estimation of Similarity Constant by Line Search
One drawback of Algorithms and 2 is that they require the knowledge of the similarity constant to choose an appropriate value for A. This similarity constant is typically unknown in practice and might be difficult to estimate. One effective solution to this problem is to dynamically adjust the coefficient À inside the algorithms by using the classical technique of line search.
The basic idea is as follows. The server first picks an arbitrary sufficiently small constant À as an initial approximation to the unknown "correct" value of X 25. Then, at every round, the server sends the current estimate of À to each client asking them to approximately solve their local subproblem. After receiving the corresponding local solutions, the server checks a certain inequality based on the obtained information. If this inequality is satisfied, the server accepts the resulting aggregated solution and goes to the next round while decreasing in two times (so as to be more optimistic in the future). Otherwise, it increases À in two times, asks the clients to solve their subproblems with the new value of X, and checks the inequality again.
The precise versions of Algorithms 1 and 2 with line search for the full- full-participation setting are presented in Algorithms 3 and 4. Importantly, our adaptive schemes are not just some heuristics but are probably efficient. Specifically, their complexity estimates (in terms of the total number of communication rounds) are exactly the same as those given by Theorems I and 6, respectively, up to an extra additive logarithmic term of log 26 X (see Theorems 27 and 28).
Another significant advantage of our adaptive algorithms is their ability to exploit local similarity, resulting in much stronger practical performance compared to the methods with fixed x. We will demonstrate this in the next section.
6 Numerical Experiments
In this section, we illustrate the performance of our methods in numerical experiments. The imple- mentation can be found at
Convex quadratic minimization. We first illustrate the properties of our algorithms as applied to
m Ey= [A,,((x- b;,),x br.s) where bij E Rdd and Ai, E Rdxdd The experimental details can be found in Appendix F.I. From Figure 1, we see that S-DANE converges as fast as DANE in terms of communication rounds, but with much fewer local gradient oracle calls. ACC-S ACC-S-DAND -DAN achieves the best performance among the three methods. We also test S-DANE and DANE with the same fixed number of local steps. The result can be seen in Figure E. where S-DANE is again more efficient. Finally, we report the strong performances of two adaptive variants (Algorithms 3 and with initial Ā = 10 10-3). We see from Figure that the method can automatically change to adapt to the to E" E.-,IVV |yr+l- vrIp
minimizing simple quadratic function: fxx == 1 f(()) where f(() == 1
n
local second-order dissimilarity. (We use (V [1v++1-v]] and
approximate the local smoothness and dissimilarity.)
Strongly-convex polyhedron feasibility problem. We now consider the problem of finding feasible point x* inside a polyhedron: P = _ Pi, where P. = {x (ai.j. x) < bi.j Vjj = 1, -,...,,me m.)
and a, b,j E Rd. Each individual function is defined as ft = n j=ر (a.,, - b
where Ei.1 m. m. We use = 10s and d = 103. We first generate x* randomly from a sphere with radius 10a. We then follow [55] to generate (ar.j. bj.5) such that x" is feasible point of P and the initial point of all optimizers is outside the polyhedron. We choose the best from {10')=_33- We first consider the full client participation setting and use n = 10. We compare our proposed methods with GD, DANE with GD [57], SCAFFOLDD with control variate of option [26], SCAFFNEW [44], FEDPROX with GD [39] and RECC-EXTRARRADEER [33]. The

---

p-110,
1-1001 (-h
1-100, e-10
p-1100. 1-10
1071
10.00
1010
10a
103
10"
101
10"
(lx1
10*
10
10-1
10-
75
100
150
203
50
150
200
180
200
100
450
Commun andi
Communiiaat founds
Conmrrutica founds
Continunccat founds
Acctracitding
Figure Comparisons of different algorithms for solving the polyhedron feasibility problem.
Geaa with non-iid sptiting
ijenn with ad splitting
DU STANTTONEE A GOAANE
croottnees Hidaptive CANGE
10-2
10
10-
value
10
10
49
10-
10
10
20
20
25
5.0
75
10.0
12.5
Communicatton rounds
Communicaton Founds
Communication rounds
Commuricatin rounds
Adaplioe S-DNEE-GD (ours)
Adeptice DAN-S-DANE-GD (ours)
GD
DAME-CD
Scaflole
Scal
Fediroa-(2)
Figure 4: Illustration of the impact of adaptive À used in (Acc-)S-DAN) DANE on the convergence of a regularized logistic regression problem on the ijenn dataset [6].
result is shown in the first plot of Figure where our proposed methods are consistently the best among all these algorithms. We next experiment with client sampling and use = 100. We decrease the number of sampled clients from s = 80 to s = 10. In addition to our methods, we also report the performances of SCAFFOLD and FEDPROX with client sampling. From the same figure, we see that the improvement of ACC-S-DANE over S-DANE gradually disappears as 8 decreases.
Adaptive choice of A. We consider the standard regularized logistic regression: f(x)= E
with f((x) 37
د log(1 (xp( ( x))) + 237 |x||² where (ai.j, Mi.j) E Rd+1 are
features and labels and M- Σn Di=1 my is the total number of data points in the training dataset. We use the ijenn dataset from LIBSVM [6]. We split the dataset into 10 subsets according to the Dirichlet distribution with a 2 (i.i.d) and a = 0.2 (non-i.i.d), From Figure 4, Adaptive (ACC-)S-DANE (Algorithm 3 and 4) converge much faster than the other best-tuned algorithms for both cases. (We set the initial À = 10 for non-i.i.d and A = 10 for i.i.d respectively.)
Deep learning task. Finally, we consider the multi-class classification tasks with CI- FAR10 [34] using R8 [18]. The details can be found in Appendix F.2. From Figure 3, we see that S-DANE (DI.) 5 reaches 90% ac- curacy within 50 communication rounds while all the other methods are still below 90% after 80 epochs. The effectiveness of S-DANE on the training of other deep learning models such Transformer requires further exploration.
Figure 3: Comparison of S- DANE without control vari- ate against other popular optimizers on multi-class clas- sification tasks with CIFAR 10 datasets using ResNet1 88
7 Conclusion
We have proposed new federated optimization methods (both basic and accelerated) that simultane- ously achieve the best-known communication and local computation complexities. The new methods allow partial participation and arbitrary stochastic local solvers, making them attractive in practice. We further equip both algorithms with line search and the resulting schemes can adapt to the local dissimilarity without knowing the corresponding similarity constant. However, we assume that each function Ja j--strongyy convex in all the theorems. This is stronger than assuming only fu-strongly convexity of J. which is used in some prior works. Possible directions for future research include consideration of weaker assumptions as well as empirical and theoretical analyses for non-convex problems.
10

---

Acknowledgments
htt.:///oooouuudddacBaasssduu The authors are grateful to Adrien Taylor and Thomas Pethick for the reference to [59]. The authors are thankful to the anonymous reviewers for their valuable comments and suggestions.
References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. International Conference on Learning Representations, 2021. URL
[2] Aleksandr Beznosikov, Martin Takáč, and Alexander Gasnikov. Similarity, compression and local steps: Three pillars of efficient communications for distributed variational inequalities. In Thirty-seventh Con- ference Neural Information Processing Systems, 2023, URL https : http///pppeeeeeeeeeeeeeeeeeeeeee www.:nee//ooummmmmm/ RvklwduzII.
[3] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-31, 2018.
[4] Sébastien Bubeck al. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 833-4)221--357 2015.
[5] Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Recapp: Crafting efficient catalyst convex optimization. In International Conference on Machine Learning, pages 2658 2685. PMLR, 2022.
[6] Chih-Chumn Chang and Chih-Jen Lin, LIBSVM: A library for support vector machines. ACM Trans- actions on Intelligent Systems and Technology, 2:27:1-27:27, 2011. Software available at http:
www.cssie.nu..du wtt.tt/tt////////t////tt/tt cjlin/libsvm.
[7] El Mahdi Chayti and Sai Praneeth Karimireddy, Optimization with access to auxiliary information. arXiv preprint arXi::2206.003:3: 2022.
[8] Laurent Condat and Peter Richtárik. Randprox: Primal-dual optimization algorithms with randomized proximal updates. In The Eleventh International Conference Learning Representations, 2023, URL
[9] Laurent Condat, Grigory Malinovsky, and Peter Richtárik. Tamuna: Accelerated federated learning with local training and partial participation. arXiv preprint arXiv:2302. 09832, 2023.
[10] Aaron Defazio, Francis Bach, and Simon Lacoste-Julien Saga: fast incremental gradient method with support for non-strongly convex composite objectives. Advances neural information processing systems, 27, 2014.
[11] Olivier Devolder. Stochastic first order methods in smooth convex optimization. CORE Discussion Papers, 1/70, 2011.
[12] Nikita Doikoy and Yurii Nesterav. Contracting proximal methods for smooth convex optimization. SIAM Journal on Optimization, 30(4):3146-110 2020.
[13] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61):2121-211 2011. URL http: hwwmi..org/ppppre//eeee duchi html.
[14] Alexandre d' A'Spremooo,, Damien Scieur, Adrien Taylor, et al. Acceleration methods. Foundations and Trends Optimization, 5(1-2):1-245, 2021.
[15] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10112- 101211 2022.
[16] Michat Grudzieñ, Grigory Malinovsky, and Peter Richtárík. Can 5th generation local training methods support client sampling? yes! In International Conference on Artificial Intelligence and Statistics, pages 1055-1092, PMLR, 2023.
[17] Osman Güler. New proximal point algorithms for convex minimization. SIAM Journal on Optimization, 2 (4):649-666 1992. doi: 10.1137/080203.. URL
0080003..

---

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: N0.1109/CVPR.2016..
[19] Hadrien Hendrikx, Lin Xiao, Sebastien Buheck, Francis Bach, and Laurent Massoulie. Statistically preconditioned accelerated gradient method for distributed optimization. In International conference on machine learning, pages 4203-4227. PMLR, 2020.
[20] Zhengmian Hu and Heng Huang. Tighter analysis for ProxSkip. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 13469-13496. PMLR, 23-29 Jul 2023. URL httttt:://prcceediiiiiie wtw.eiiinnsmmrr.ssssssseeee.eeee hu23a html.
press/ /ooas//v022. v202/
[21] Anastasiya Ivanova, Dmitry Pasechnyuk, Dmitry Grishchenko, Egor Shulgin, Alexander Gasnikov, and Vladislav Matyukhin. Adaptive catalyst for smooth convex optimization. In International Conference on Optimization and Applications, pages 20 37. Springer, 2021.
[22] Xiaowen Jiang, Anton Rodomanov, and Sehastian Stich. Federated optimization with doubly regularized drift correction. arXiv preprint Sediv:24000044447 2024.
[23] Ric Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume
1, NIPS' 13, page 315-323, Curran Associates Inc., 2013.
[24] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafacl G. L. D' D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser. Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný. Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trends@ Machine Learning, 14 (1-2):1-210, 2021. URI. https: / wwwrxi..o.///pa.. 1///1/9111999.../
[25] Avetik Karagulyan, Egor Shulgin, Abdurakhmon Sadiev, and Peter Richtárik. Spam: Stochastic proximal point method with momentum variance reduction for non-convex cross-device federated leamiing. arXiv preprint 2005:22405.201 2024.
[26] Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national conference on machine learning, pages 5132-5144 5143. PMLR, 2020.
[27] Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Breaking the centralized barrier for cross-device foderated learning. In Advances in Neural Information Processing Systems, 2021.
[28] Ahmed Khaled and Chi Jin. Faster federated optimization under second-order similarity. In The Eleventh International Conference on Learning Representations, 2023. URL /tlLhttt///opeeeeie...ee derrum?AAdEE
[29] Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. First analysis of local gd heterogeneous data, arXiv preprint arXiv:1909.000 04715, 2019.
[30] Donghwan Kim and Jeffrey A Fessler. Generalizing the optimized gradient method for smooth convex minimization. SIAM Journal Optimization, 28(2):120-1190,, 2018
[31] Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint 1rXiv:1110.02527, 2016.
[32] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint 161iv:6110.05.. 2016.
[33] Dmitry Kovalev, Aleksandr Beznosikov, Ekaterina Borodich, Alexander Gasnikov, and Gesualdo Scutari. Optimal gradient sliding and its application to optimal distributed optimization under similarity. Advances in Neural Information Processing Systems, 35:33494-3333 33507, 2022.
12

---

[34]
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http::/www.cc. wwwp:cc..troonto.edie.co feltuu/rri//iffeee htmm..tml........
[35] Guanghui Lan. Gradient sliding for composite optimization. Mathematical Programming, 139:201-2355
2016.
[36] Guanghui Lan and Yuyuan Ouyang. Accelerated gradient sliding for structured convex optimization. arXiv preprint arXi:: 04905, 2016.
[37] Guanghui Lan, Yuyuan Ouyang, and Zhe Zhang. Optimal and parameter-free gradient minimization methods for smooth optimization. arXiv preprint arXiv:2310. 2tt.:2330..1119, 2023.
[38] Bo Li, Mikkel N Schmidt, Tommy $ Alstrøm, and Sebastian U Stich. On the effectiveness of partial variance reduction in federated learning with heterogencous data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3964-3973, 2023.
[39] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 1:429-450 2020.
[40] Dachao Lin, Yuze Han, Haishan Ye, and Zhihua Zhang. Stochastic distributed optimization under average sccond-order similarity: Algorithms and analysis. Advances in Neural Information Processing Systems, 36, 2024.
[41] Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. Advonces in neural information processing systems, 28, 2015.
[42] B. Martinet. Perturbation des méthodes d' o'optimisation. Applications. RAIRO. Analyse nonérique, 12(2):
153-171, 1978. URL http: httttt://www..mmttt www.wwwwww.mmmmmmmmmmmmmmmm httemddam...//iieeee http///ee//////// 12_2__55_/
[43] Brendan McMahan. Eider Moore. Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficcenn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282, PMLR, 2017.
[44] Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead communication acceleration! finally! International Conference on Machine Learning, pages 15750-15769. PMLR, 2022.
[45] Konstantin Mishchenko, Rui Li, Hongxiang Fan, and Stylianos Venieris. Federated learning under second-order data heterogeneity, 2024. URL
[46] Renato D. C. Monteiro and B. F. Svaiter. An accelerated hybrid proximal extragradient method for convex optimization and its implications second-order methods. SIAM Journal on Optimization, 23(2): 1092-1125, 2013. doi: 10. 1137/1 10833786.
URL https: / /ttp:///ooooooo/o//// 10. 1137 1137/100833786
[47] Moreau. Proximité dualité dans un espace hilbertien. Bulletin de la Société Mathématique de France. 93:273-299, 1965. doi: 10.24033/bsmf. URL. http: http:///wwwwmmdaaooo articles/ wt..eee............. 24033/
bamf 1665/.
[48] Yu. Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming, 140 (1):125-161, 2013. doi: 10. 00010077/10007--02--099995
0629-5.
[49] Yurii Nesterov. Lectures on Convex Optimization. Springer Publishing Company, Incorporated, 2nd edition, 2018. ISBN 3319915770..
[50] Yurii Nesterov and Sebastian U. Stich. Efficiency of the accelerated coordinate descent method on structured optimization problems. SIAM Journal on Optinization, 10-13:100-12,, 2017. doi: 00.1137/1M100011
URL htttss:///oi..ooo
[51] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends@ in Optimization, 1(3): 127-239, 2014. ISSN 2167-3888. 2400000003.
doi: 10. 10.1166//2000M0000 URL http: http:::/xx.ddo.org// 10..////0111///////
[52] Kumar Kshitij Patel, Lingxiao Wang. Blake E Woodworth, Brian Bullins, and Nati Sre- bro. Towards optimal communication complexity in distributed non-convex optimization. In S. Koyejo. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35. pages 33316-13328 Curran Asso- ciates, Inc., 2022. URL https:/ pectps://prowe 56bd21259e28ebdeee 966 Paaprr- PwweP-aaaeeerreeeeeee Conference.pds.
13

---

[53] Sashank J Reddi, Jakub Konečný, Peter Richtárik, Barnabás Póczós, and Alex Smola. Aide: Fast and communication efficient distributed optimization. arXiv preprini 0667:99608.06879, 2016.
[54] R Tyrrlll Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877-8%) 1976.
[55] Anton Rodomanov, Xiaowen Jiang, and Sebastian Stich. Universality of adagrad stepsizes for stochastic optimization: Inexact oracle, acceleration and variance reduction. arXiv preprint arXiv:2406.06338, 2024.
[56] Abdurakhmon Sadiev, Dmitry Kovalev, and Peter Richtárik. Communication acceleration of local gradient methods via an accelerated primal-dual algorithm with an inexact prox. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems. 2022. URL //opps:://////eeeeeee evweevviwwwwa...
[57] Ohad Shamir, Nati Srebro, and Tong Zhang. Communication efficient distributed optimization using an approximate newton-type method. In International conference on machine learning, pages 1000-1008. PMLR, 2014.
[58] M.V. Solodov and B. F. Svaiter. A unified framework for some inexact proximal point algorithms*. Nu- merical Functional Analysis and Optimization, (2(7-8)11013 2001. doi: 10. 1081/ 100108320. URL https: / /tt.////o/o//o////////ooo/oooo 10. 1081/N04 100108320.
[59] M.V. Solodov and B.F Svaiter. A hybrid proximal point algorithm. Journal of Convex Analysis, 6(1 6(1):59-70, 1999. URL http: // weweeedm.oorgoo. wtt.oo////d/oo....///..//
[60] Ying Sun, Gesualdo Scutari, and Amir Daneshmand. Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation. SIAM Journal Optimization, 32(2):354-335 385, 2022.
[61] Ye Tian, Gesualdo Scutari, Tianyu Cao, and Alexander Gasnikov. Acceleration in distributed optimization under similarity. International Conference on Artificial Intelligence and Statistics, pages 5721 5775.. PMLR, 2022.
[62] Farshid Varno, Marzie Saghayi, Laya Rafiee Sevyeri, Sharut Gupta, Stan Matwin, and Mohammad Havaei. Adabest: Minimizing client drift in federated learning via adaptive bias estimation. In European Conference on Computer Vision, pages 710-726. Springer, 2022.
[63] Blake Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33:6281-6223 2020.
[64] Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems, 33:5332-553 5344, 2020.
14

---

Appendix
Contents
1 Introduction
Problem Setup and Background
Proximal-Point Methods on Single Machine
Distributed Proximal-Pooe Methods
Stabilized DANE
4
Accelerated S-DANE
5 Dynamic Estimation of Similarity Constant by Line Search
6 Numerical Experiments
7 Conclusion
10
More Related Work
16
Technical Preliminaries
16
B.1 Basic Definitions
16
B.2 Useful Lemmas
17
Proofs S-DANE (Algorithm 1)
20
One-Step Recurrence
20
C.2 Full Client Participation (Proof Theorem 1)
C.3 Partial Client Participation (Proof Theorem
22
C.3.1 Stochastic Local Solver
23
Proofs Accelerated S-DANE (Algorithm 2)
25
D.I One-Step Recurrence
................................
25
D.2 Full Client Participation (Proof of Thcorem
................................
26
D.3 Partial Client Participation
................................
27
D.3.1 Stochastie Local Solver
28
Dynamic Estimation of Similarity Constant Line Search
28
F
Additional Details Experiments
31
E.1
Convex Quadratics
31
E2 Deep Learning Tasks
31
E33
Implementation
31
Impact Statement
32
15

---

A More Related Work
In the first several years of the development for federated learning algorithms, the convergence guarantees are focused on the smoothness parameter L.. The de facto standard algorithm for federated learning is FEDAVG. reduces the communication frequency by doing multiple SGD steps on available clients before communication, which works well in practice [43]. However. in theory. if the heterogeneity among clients is large. then it suffers from the so-called client drift phenomenon [26] and might be worse than centralized mini-batch SGD [63, 29]. Numerous efforts have been made to mitigate this drift impact. FEDPROX adds an additional regularizer to the subproblem of each client based on the idea of centralized proximal point method to limit the drift of cach client. However, the communication complexity still depends on the heterogencity. The celebrated algorithm SCAFFOLD applies drift correction (similar to variance-reduction) to the update of FEDAVG and it successfully removes the impact of the heterogeneity. Afterwarss, the idea of drift correction is employed in many other works [1S, 62, 38, 1]. SCAFFNEW uses a special choice of control variate[44] and first illustrates the usefulness of taking standard local gradient steps under
followed with more advanced methods with refined analysis and features such as client sampling and compression [20, 9, 8J. 5gCS [16] uses an approximate proximal-pone -oinnt step at each iteration and derives the convergence rate that is as good as SCAFFNEW. W. and it also supports client sampling. Later, Sadiev et al. [56] proposed APDA WITH INEXACT PROx that retains the same communication complexity as SCAFFNEW, but further provahly reduces the local computation complexity. FedAC [64] applies nesterov's 's accelcration in the local steps and shows provably better convergence than FEDAVG under certain assumptions.
More recent works try to develop algorithms with guarantees that rely potentially smaller constant than L. SCAFFOLD first illustrates the usefulness of taking local steps for quadratics under Bounded Hessian Dissimilarity 8max [26]. SONATA [60] and accelerated version [6]] prove explicit communication reduction in terms of dmax under strong convexity. MIME [27] and CE-LGD [52] work on non-convex settings and show the communication improvement on 6max and the latter achieves the min-max max optimal rates. ACCELERATED EXTRAGRADIENT SLIDING [33] applies gradient sliding [33] technique and shows communication reduction in terms of 6max for strongly-convex and convex functions, the local computation of which is also efficient without logarithmic dependence on the target accuracy. DANE with inexact local solvers [57, 22, 53] has been shown recently to achieve the communication dependency on 5 under convexity and 6max under non-convexity. For solving convex problems. the local computation efficiency depends on the target accuracy e. Otherwise, the accuracy condition for the subproblem should increase across the communication rounds. Hendrikx al. [19] proposed SPAG, an accelerated method, and prove a better uniform concentration bound of the conditioning number when solving strongly-conve --onvex problems. SVRP and CATALYZED SVRP [28] transfer the idea of using the centralized proximal point method to the distributed setting and they achieve communication complexity (with a different notion) w.r.t 5. Lin et al. [40] further improves these methods either with a better rate with weaker assumptions based on the ACCELERATED EXTRAGRADIENT SLIDING method. Beznosikov al. [2] uses compression to reduce the bits required to communicate and the more general problem of Variational Inequalities is considered. Under the same settings but for non-convex optimization, SABER [45] achieves communication complexity reduction with better dependency on 6max and n. Karagulyan et al. [25] proposed SPAM that allows partial client particiption.
Remark 7. Khaled and Jin [28] and Lin et al. [40] consider the total amown of information transmitted between the server and clients the main metric, which is similar reducing the total stochastic oracle calls in centralized learning settings. This is a particularly meaningful setting if the server prefers to or has to receive/transmit vectors one by one and can set up communications very fast. The term 'client sampling' in these works refers to sampling one client to do the local computation. However, all the clients still need to participate in the communication from time to time provide the full gradient information. This is orthogonal to the setup of this work since we assume each device can do the calculation in parallel. In the scenarios where the number of devices too large such that receiving all the updates becomes problematic, we consider instead the standard partial participation setting.
B
Technical Preliminaries
B.1 Basic Definitions
We use the following definitions throughout the paper.
Definition 5. A differentiable function f IRd + R is called p-comvex for some 2 ofr all x, E Rd,
f(y) 2 f(x) (VI(x),) x) - y11.
(B.1)
If f is p-ccovex, then, for any x, y E Rd, we have (Nesterov [49], Theorem 2.0:
A||x yll S [Vf(x) -Vf(x)-V(xlll..
(B.2)
16

---

funcitioon Definition 6. A differentiuble function f IR -> B. is called L-smooth for some L 2 0ff all x, E R
( Vf(y)l 1 yll-
(B.3)
If f is L-smooth, then, for any E R have (Nesterov [49], Lemma 1.2.3)
fly) [(x) (VI(x), x) _ xx|?
(B.4)
2
Lemma 8 (Nesterov [49], Theorem 2.1.5). Let / R IR be conver and L-smooth. Then, for any x, E R..
we have
1 L VV - v
(B.5)
B.2 Useful Lemmas
We frequently use the following helpful lemmas for the proofs.
Lemma 9. For any x, E R and any > O, we have
1x, y>1 $ 111111² 2n l1y
(B.6)
27
|x yll (1 +|x]|||
(1+2) ltyl
(B.7)
Lemma 10 (Jiang al. [22], Lemma 14), Let {mr) be of vectors in IK and let x == 1.2 E Let
E Thd be an arbitrary vector, Then,
Σ v = -|X-- vil--vl++
+ Σ 1x -x|
(B.8)
Lemma 11. Let (Fi) and (Dx) be non-negative sequences such for any k 2 0, holds that
FE+1 Dz+1 S 4Dx +
where g (0, and some constants, Then for all K and SK Eimz t we have
K
1 q R
1
DK <
Do te.
SK
km
4b
9k
Proof. Indeed, for any 2 0, we have
Fk41
Dx+1 gh+I
D2 e
gh+l
gh+l
Summing up from it to K we get
Fk Dr dk
< Do + SKE.
Dividing both sides by Sx and substituting Sx T-q (ot 1). we get the claim.
Lemma 12 (c.f. Lemma 2.2.4 in [49]). Let (A, be non-negative non-decreasing sequence such that Au = and, for any r 2 0,
c(A+1 -
Ar+1
1 HA,
where 0 and 2 0 some constants. If
4c, then for any R > 0, have
1
[1+
8
R²
AR 4u
- 1
>
(B.9)
4c
4c
4c
Otherwise, for any R 2 1, it holds that
1
2(R-1)
Ar 2
1
(B.10)
4e
1e
17

---

Proef. Denote C. = vudr. For any 2 0, it holds that
pC (1 (1+ S c(C,++ - C² - c(2(Cr+1 CG)) =c
1c(Cr+1-CC - C+)
Therefore, for any r 2
Cr+1 -++1- 2 P
V1+CA.
Ac
When j S 4c, by induction, one can show that, for any 2 0 (see the proof Theorem in [50] for details):
CR 2 2 (1+ V 4c
/
(1 -1
R
# 4c
R
R.
4c
When j 4c, we have Cr+1 C, 2 V Cr. It follows that, for any R
2
1,
Cn (1+ H Te
R R-1
ci a2
(1+
P
8-1
4e
de
Plugging in the definition of CR. we get the claims.
Lemma 13. Let {xx Him be vectors in R$ with Let E [n] and let S E ((=)) be sampled uniformty at
random without replacement. Let x = E 1 Ni. 6² 1 Σ xx; and and xs 1/2 Ejes X3. Then.
and
x[|ss x
12 11-8
(B.11)
Proof. Let (m) ml(n-m)! be the binomial coefficient for any n 2 m 2 1. By the definition of xs. we have
Σ x,
Σ 1li E S|x,,
X5
JES
where [ denotes the {0, 1) -iiddicatoor of the event E. Taking the expectation both sides, get
E[xs]
Σ
Pr(ieeSxx
x.
xi x.
Further,
E| x E
x X,X, x,,, x]]
188 JCS
x|| +
1
Σ
(x: - -,, xx]
32
8E8
(JE8,
1/1 E SE[[ - x|| +
1 E S) - 8,x, xx)
Pr E S
x|1
Σ
Pr E (x. x, x)
X X
(xi - 8, - x)
(xi - X, x).
8
sn(n 1)
Note that
Σ
(x: - x, - x)
(x: 8, - x)
[x x = =-n
(.j
Thus,
2²
(s - --
[[| x1|2]
8(n
11
1
18

---

Lemma 14. Suppose {sr)=n sutisfy A.-ED of size E [n]] and C-BGV with 2 2. Let + Ic and fs == 1 Eies fo. where S E (?a)) is sampled uniformly random without replacement. Further, let E TR be fixed point, and let x5 E B be random point defined by deterministic dleterministicffunciinn of S. Then, for any 7 > 0, it holds that
Es|f(xs) fs(xs)) n 8-5
A. Bs[[|x -yll")
(B.12)
2s
Proof. Let hg = f- fs. Since {fi} satisfy A,-ED (Definition 4), we have, view of inequality (B.4),
hs(ss) s hs(y) (Vhs(y) x5 y) xxs yl1
(B.6)
x
ha ( + ク/ヒ (y)((>)l' I|
XxS
y1?
A
Ixxs yl1
27
Rearranging taking the expectation on both sides, we gel, for any 7 > 0.
(x(hs(xs) hs(y)] - T
Es Bs|lxs yl +
A,
Bs Bsllxx y
2y
2
A
[s|l|x y|1)]
n-12s 2s
27
where the last incquality is due to (B.11). Using the fact that [s(f(y) fs(y)) 0, we get the claim.
Lemma 15. Suppose {fi satisfy 50-SOD of size [m]. Let fs = _ Exes fi where s E [m] and
S E ((r)). Leiv E Rd be a fixeed point, > de, and let
(x(x = f((x) (VA + f11x v1I,
where he fs - fv. Let {xx hes be a set of points I2d (such that x; Rs arg ming F. the sense that [FF(())]] is sufficiently small), and let X5 1 Eies Then,
Σ + (xs),, x.) 2x 8
1 Ev((x))
A- 5, 1
11 A
v x:|
x:|
V (x:))l
2
Proof.
Using the definition of Fú, we get
VF((x)) = fx()) + Vhg(v) + 1(xi v).
Hence.
(V]((x:)+ + Vh (xs), V xx) = Al|v x + (V (xs) (V x.) + (VE(x))) -x).
Taking the average over on both sides of the first display, we have
Σ
v((xx) a(v xs) VF(xx).
(B.13)
ES
EES
Therefore,
1 2A 8
1
Σ
1
1
N(v xs) + VF.(
2A
= |l x 12 ++ (x;));), v v-xx) + +2x [V
IE5
It follows that
Σ (V((x)))
+ VA (xx), v vxx;)
x;)
(xx
2X
1 Σ
V
1
2 -
1 -
1
Vh (Rs) Vh.
SES
19

---

+
[FF(())) &g -xxx xi)
Σ
V (x,) ||
V
2x
EES
[1.) X
V xsll*
Σ
|xx - *xs1**
Vh
(h)) x.)
2
688
1 Σ
+
(VF
(x))
&g - x1)
(x,)),)
2x
ES
(86) X | xs1l* +
2
1
Σ1
|xx xs11²
1
(Xs)
(8s) (v)
8. Σ
1x: 1 xs|1
2
26. 8 iES
2 iE8
128
A1
11
[F((x:) |1
x x=|1²
VF.(x))]1
2 iE8
2X
2X FES
1ES
(9.(88) 8,
A-
11 Σ
|| x=1l²
+
X xsl1
[x,)(x))11
2
A
IES
#ES
18.1
8,
v
x:
Σ
[VE())I
As
168
ES
where in the second equality, we use the fact that Exel [n](v]] = O.
C Proofs for S-DANE (Algorithm 1)
C.1 One-Step Recurrence
Lemma 16. Consider Algorithm 1. Let ft Bd + R be p-convex with μ 2 0, 0.ror any [n]. Assume that (fyE have 6,-SOD. Then, for any v 2 0, we have
1 - fx,,(x")] 1 2//
+1
2
$ 2 |'' x" 1|
1 0,/A1
Σ
||
x111
11 Σ (x X SE
Proof. By μ-convexity of fi, for any 2 0, it holds that
{ [x, + fIv
x11 -
11 Σ fi(x)) - xx11 x21
SES
(R.)) 11 f(xx,++1) + (V)x - -x,rtt) + - -1x - x" x²] + 1 +/5l"= -x"
8 SES,
Recall that + is the minimizer of the final expression in x". This expression is a (1 + pu/ coovex function x". We can then estimate by:
1
1
A Ss, (x") + 2
l|v
x11
(B.0
1
[((x).+11) (VSi(x).+1)) v xi.+1) + 1 -
2
1+
||
x
2
1 Using the convexity of f. and dropping the non-negative a EiEs, || 11 we further get
X (x') + 1 1
1
1
11
Eis(x" + (Vf((**)) Xirt+1 - x x1 1] 1 u/a ||r - x11
2
11
(V + 12 Σ V((x.)))) ri+1 -v))
+
As
ie8.
+ 2
1
+1 - 2
20

---

18.01 1
2 fs, (xx
1// xx1 As iES,
11 Vx((x"
x -x(T)
2
11 Σ (V -xir+1)
1
A8 iES.
2x
LES
Denote h fer
ft. Note that
Σ (V X1,+1 1x-+1-x(tt
xx
Σ
(-
Vhg ) xi.+1 v),
IES
where we have used:
x
Σ
Xir+1
and
Σ Vhī(x = 0.
ES
It follows that
1 1 + 2
21lv² x' x 18. (2*)+ 1+ 1/2
X fs, (x')
fx'(x'))
| =x'1
2
11 Σ (V + (h,(x" ),v' -xi.+1)
1
Σ
A
2x
Es
SES
We now apply Lemma 15 (with x, X v = =v.S= = S, and x = x= to get
x,,+1) ).v
1
Σ
2x
&
A - 8, 2
11 Σ (L
Ii- - x
112
A
EE
IE
Substituting this lower bound into the previous display, we get the claim.
C.2 Full Client Participation (Proof of Theorem 1)
Proof. Applying Lemma and using Σ ( S 8²
EE - v'I and 26, for
any r 2 0, we have
1 11 - - 1 +
1
1//
l1 1r | - x1112
2
$ 2 gllv - x²1 1
1-8
1 1
||v X
V Fe.- (x.r+))l
2
n
X2 n
=gllv= - x x²1 - (1111/2
-(--11
1/2
1)
1)
||v - -x.rt1ll
2 2 v|v - 11
2
Rearranging, we get
24 [f(x - 1&] s q/lv x11 |v x 112
X
where g = 14p7x' Applying Lemma with E and using convexity of f. we obtain
24
X -1((x*)
()
1
| x111 < (1/))= - |v x2
1- R1/9)=-1 D2.
q
Dropping the non-negative term 1-g | 12 and rearranging, we get
(1-
(x²) 5 24 [11/) D².
Plugging in the choice X and the definition of g, we get the claim.
Under the same setting as in Theorem 1, to uchieve 1(x - S S e. we need at most the following number of communication rounds:
Corollary 17.
R= 0 (u+ log 1+ e
)
Proof. Using the fact that (1+ g)ah 2 expl 1-g k)) for any 2 0 and t: > 0, we get
f(x - 8r S nD²
μ1²²
$
1] $ 6.
2 4 11
2 - R)
Rearranging, we get the claim.
21

---

Proof of Corollary 3.
Proef. To achieve E V.. (xxx++1)]] )1 < E |xx, - -y' for each i [n], it is sufficient to ensure that Pir ( s X1,++ v- Let Letxi, arg min Fi. Since
(R.2)
Xx,+1 2 lI xi,ll ||x,441 x1,1l l| - [V
xi,ll
and À = 26, suffices to ensure that
26 3 1|
V (x.,(()
x
(C.1)
for any i E [n]. According to Theorem from [33] (or Theorem 3.2 from [37]), there exists certain algorithm such that when started from the point v", after queries to V Fi.r. . it generates the point Vir+1 such that
[x,,((x S (L A)llV* -x
o Lilv" - x
K
K
(recall that 5 < L). Setting K B(V ) concludes the proof.
Remark 18. Recall that Fir is + smooohh and (μ + Xy-comvex, (66) and 5 I L. Suppose worker i uses the standard GD to approximately solve the local subproblem at round starting at for K steps and return the last point, then by Lemma 19, have that the accuracy condition (C.1), it is sufficient make K O local steps. Suppose worker uses the fast gradient method, then by Theorem 3.18 from [4], we have that (V 1X) - 2(L + (F(,,((xatt)))
I O (L+A) -x-=x;,-1l)] To satisfy
Fi. (x))) - ( + 2) exp( + K)ll= x1 To satisfy the accuracy condition (C.1), it suffices make ( p+8 log( )) = O
L logl 1) gradient oracle calls.
pts
Lemma 19 (Theorem 2.2.5 in [49]). Let / R2d + R be convex and L-smooth function. Consider the gradient method with constant stepsize:
X = xx I (xxx),
k 0,
started from some E R Then, for any K 2 1, holds that
L|x0 x
[ o
(C.2)
K
Proef. By Theorem 2.2.5 in [49], we have that
[f(x))] 0 7.1120 - x
min kE[K]
K
It remains to note that the algorithm generates non-increasing V [xx))] since
[[ [[V - Vx(x)) V(xx)]
[f(x+++)
(xx+1)) - vs((x),, vf(x)) + [[v
V (X(++)) V ( V)(XC+1), -Xen)) (
(
[Vf(x))) [|V/((+41) vx(xx)|| [Vx(x))
C.3 Partial Client Participation (Proof of Theorem 5)
The following theorem is slight extension of Theorem which includes the of stochastic local solvers.
Theorem 20. Consider Algorithm 1. Let fe Bd - be μ-convex with μ 2 f.ror any E [r]] and let Assume that (f.y have SODD AD-DD and (-BGV. Let suppose we have
4(n-a) < 2(8 A.). For any 2 s(n-1)
Σ Be... (X
2 1
Σ 1lx.-ll LESt
Ae 1
(C.3)
1EST
for some > O, where Eir denotes the randomness coming from device when solving its subproblem at round r. We assume that {E..r) are independent random variables. To reach [x -f") S need at most the following number of communication rounds:
5, +
A, + P
- <²
log 1 + HDD2
(6,
A.)
n 2D
R T==6
s 5
P
1 sEp
n 3ea
where R == R
pxx x²x R
p.
$. and D ||x"
x"11
22

---

Proef. According to Lemma 16, we have for any 2 0.
[(ss((x+t)) - -ss, + 1 1+ 1/²
1| r+
11
S 2 I1v - -x'11 xx11² 1- bo/
lIv" xx.++1l 11
11 Σ
II FIV FL. (x..+111))
2
X2 E.
sE8
According to Lemma 14 (with S = Sr, x= =
x and v²), for any
>> 0, we have
Bs, 1s, (x x1 1|5 n1 -1 7
1
A.
+
Bs, |||
-v'1
n- 1 2a
27
2
(8.8
266
1
+
A
Es.
Xxr+1
v']]".
1 28
27
2
seS.
Adding [x)**)) to both sides of the first display, taking the expectation over S, on both sides, substituting the previous upper bound and setting 7 (n-(n=))
s(n-1)e we get
Bs, (1) - + + 1/1 A
f-
Es, [|lv
- -x-11"
2
1 $
111' -1v--x11 2 - 2
1x113
8,
+ A
1
Σ
| - 1xc+11-'111
2
2x
24
IES
47 x Es,
Σ
IV 11
1E8
Denote all the randomness {Ee. hes, by &r.. Since &..- is independent of the choice of S. for any i E [n]. taking the expectation over &, both sides of the previous display and using our assumption (C.3), we obtain
1 X Bss.....((x"
14 1+ M/A Es,
[Iv
+1
x'191
S 2 v| - xx112 - (
1
8, + A.
1
Rs.ie
1
Σ 1|x L8
21
27
21
SES
By choice of X. have -- 5et
2x 2 0. Taking the full expectation on both sides, we get
|[((x+))- -1 1 1/1
Ell r+1 x "11 B[llv' x11
E
2
2x
According to Lemma 11 and the fact that |/v
x = D. we get
2
x 11 S
1 1-9 9 D² + x++ X c,
+ X
(1/g) R
where 9 = T+p/x.
Rearranging and dropping the non-negative El|l @[||v= xx ||3], get, for any R 2 1,
DD²
DD²
-[f(x")-f*) S
+ 2
S 2/exp( p+d R) - =
+
21
+ +1)
1]
To reach s-accuracy, it suffices to let 2[axpl
D
S
Rearranging gives the claim.
R1
C.3.1 Stochastic Local Solver
Note that there exist many stochastic optimization algorithms that can also achieve the accuracy condition (C.3) such as variance reduction methods [23, 10), adaptive SGD methods [13], etc. Here, we take the simplest algorithm: SGD with constant stepsize as example.
Corollary 21. Consider Algorithum under the same settings as in Theorem 20. Further assume that each L-smooth and each device has access to mini-batch stochastic gradient gx(x, E.) such that
Ex, E:)] = f((x),
[x,119((x,E) E.) (1
Suppose for any r 2 0, each device i E S, solves its suhproblem approximatell by using mini-batch SGD::
1k+1 = 2x
H
[9) (x, E(,) - V((('' + Vss, A(2x -v)]
0 kg K.
23

---

where 40 = and H > L + X is the stepsize coefficient. Let Eir denote (Etx)s. To achieve accuracy condition (C.3) for an appropriately chosen H, euch device requires most the following number of stochastic mini-batch oracle calls:
L [1ttA ++
(L+
A)o² *
log L 1+1)
K ==
Proof. To get (C.3), it suffices to ensure that, for any i E Sr. we have
X²
de Es, Ee,,- (1x v
( 11)
4
For this, it suffices ensure that
X2
x2,+11+ de
Es VI. (x )1
lIv
(C.4)
10
where x == arg ming (x). Indeed, suppose (C.4) holds, then we have
xxr+t1 - v01 liv" xi, ||x
rll
2 l
xirl - FF.
Hence,
2
x
(x..,( )l 2/|x v'Il
X
Plugging in this incquality into (C.4) and taking expectation w.r.t für on both sides, we get
X?
E (x:.,
B..,,
[P] E 1ll - - ge.
Rearranging gives the weaker condition.
We next consider the number of mini-batch stochastic gradient oracles required for SGD achieve (C.4). Since Fir is (L A)-smooth and (n + A)-convex, according to Lemma 22, have
E [II ( [1=) < 2(L + X) BE,,, (Zx) F[,r
(μ A)Ilv* x²,=12
a²
$ 2(L + X)
21exp((1 A)K/H) - 2(H - L
where 2K Σこ EK and g ドー Choosing now (L A) + 5 and letting the
coefficient the first part the previous display S 10. we get the claim.
Lemma 22. Let f be a p-convex and L-smooth L-smooth.function Consider SGD with constant stepsize H > L:
Xk41 1 = arg min (gk x) + ||x x
XER
where Ok g(xx,, Ex) with Re [x( E)] vf(xx and Eeg (1x1((xx)) ff(x)11 S 0² for any Rdd, Then for any K 2 1, we have
A||xo - 11 2[exp(uK/HH) -
o²
[[((xx)] -
(C.5)
2(H - L)
where Nx
Σ k=
X and
=
Proof. Indeed, for any 1s 2 0, we have
fxxx) + (or, x" xx) + ||x: - x 1
2 f(xx) (gk, xx+1 - xx) + | - xx| 2+
||x+1 - x"11
2
(43)
11 T,
H
2 ((x+11) + (gi - vf(xx), Xt-1 xx) +
||x+11 xx +
2 | xx11
2
2 f(xx+1) |1g V (xxx))
H
2 X - x²1
2(H - I.)
Taking the expectation on both sides and using u-convexity of f. we get
H
H - Flllx xx - x1
a²
E[f(xe+1) - f*] Bl x
2(H - L.)
Applying Lemma 11, we have for any
1:
A A||xo x" x11
2
S x|| - x+1
d²
[l(((x) - f
+
2 (1 -
2(H L) 2[exp([KK/H] H)
2(H L)
24

---

D Proofs for Accelerated S-DANE (Algorithm 2)
D.1 One-Step Recurrence
Lemma 23, Consider Algorithm 2. Let ft 18d + R be j-convex with & 2 0 for any i E [r]] Assume that (filzz have 6,-SOD. For any 0, we have
A. (x.) fs, (x') +
||v"
x'11
2
Br
2 Ar+15s, (x
+1)
|v x²1
+ Ar+1
1- 8g 1
||xer41 - y"1
11 X
Fir (x.et)))
8 SES,
Proof. By y-convexity of f., for any
0, it holds that
B,
Ax + ar Is. (xx)
1l x21
2
+ B.
A
f(xx') + ar+1
Σ fi(x')
|| [1''''''''' xx'11
2
E8
(x),)+))) + (Vf((x x" x..+t))++ + Br l|v - x21
Ar
+ a,+1
Σ [(x()t))) (V)(())))) xx,+1)
1 2 ||x - x'11"]
Recall that v is the minimizer of final expression in x*. This expression a (Har+1 + B,)-conver function in x". By convexity and using the fact that A,41 = A, + ars1 and B,+1 = ua,+1 + B,, we obtain
B. l|v
Ar Arfs, + art1fs, (x") +
- 1x
2 Ar41-
1 Σ f((x,++1) + par+1 Σ
-...t1-v-tlp B 2
[|-- 112
2
8E8,
E.
1 (1) A,x + ta Art1X1,44)) + Br+1 | -x"1
8 sE
2
Recall that y = Ar x" +
ar+1 Ant1
v". Therefore.
3,41
Br
||v - v r
(V)( + aetiv+t1 Art1xert1)
2
B. || v1 + ar+1 EV(((xatt)
2
EEB,
Art1 (V
xi.+1)
iegr
(86)
2
+ Ar41
Σ (X)))) - Xx.41)).
2B,
SES
ves,
Substituting this lower bound, using convexity of f. and dropping the non-negative p0, Eies, l|x v v1, we further get
B. "|v liv" -2
Arfs, (x") + (x",
2
(B.1)
1 Σ 1f (V Xirti -x-t1xx-xxx x -
Br+1 | --11--x1] x"1 11
Art1
2
SES
+ Art1 Σ (V.)(x -xir1))
Σ (x(.
2B,
tES.
25

---

Denote 16 = Is, Ss. Substituting
(V X,
Σ
V- (x
8,rt11 -y)
E
15
into the previous display, we get
Br
A.fs, + fs (x') )+ +
|v" -x'11
2
+ Br+1 || xx1
Art1ss, (x )
2
+ Ar41 Σ
(Vii(x
(x((x(t)) ).y" y" x..41)
ar ll's Σ
2B,
We now apply Lemma 15 (with x, xi,+1, S = S. and =x"+1)
) gel
Σ + + VAT V
),y" x1.r+1) 1
Σ Vi((xi,+t1)
2x
S
A 6,
11 Σ FL
Ily"
xir+ill A ES
2
ES
21 get the claim.
Substituting this lower bound into the previous display and using A,+1
D.2 Full Client Participation (Proof of Theorem 6)
Proof. Applying Lemma 23 and using Σ r 2 O. we have
= - y²1l² and A = 26, for
Br 1|v - -x"1 1²2
A. fxx'x) + aatif"
2 Art1f(x
By+1 | - Art1
A- ) Σ
2
2
IES
= =x+1
Bo+1 | -x11
x1.
2
Subtracting Ar+1f" on both sides, we get
A++1 1"]
B
- x A+[((x) 5
Br
x²
2
2
Recursively applying the previous display from to = 1, we get
Ax[f(x")- 1 Ba "
1
1| < Aul (x - 2 11 x=11
111x² x²1
remains to apply Lemma 12 and plug in the estimation of the growth AR.
Corollary 24. Under the same setting as in Theorem 6, to achieve f(x") f S we need at most the
following number of communication rounds:
0
+ 1 1
min(u, 81
log 1+
C
Proof. When μ 85, by using
(1+ / 11 86) - (1- 88
2 11 v 85)
1) - - 12expp
VPR
) -
88 vil
we get
2μ
2μD2
f(x²) 8r [11
exp(
VBR V55+vD
Making the hinh side < and rearranging, get the claim. When μ 2 85, it suffices ensure that aD3 q 2 S e.
L
26

---

D.3 Partial Client Participation
It is well known that accelerated stochastic gradient methods are not able to improve the complexity in the stochastic part compared with the basic methods [11]. A similar result is also shown for accelerated distributed method.
Theorem 25. Consider Algorithm 2 under the same setting as in Theorem 20. Let
s)R C²
== (6. + +
8(n
and suppose that, for any r 2 O, we have
Σ
EE... ( 0
1=1 Σ
Ee [|x v v
As
4R
S
sES,
Denote = ||x" - 1l Then, to ensure that B(f(x [1] the following number of communication rounds:
- f" S E for some & > 0, we need to perform at most
Vo. + A.
Vi
min (M, A
+ 5²
min AA
R=
log 1+
log 1+
V/
sEj1
(6.
D,
8 22
SE2
The error term that depends on <² and is at the same scale as S-DANE, i.c. O when ja > and O S when = O. Nevertheless, when is large enough such that this second error becomes no larger than the first optimization term, then ACC-S-DAND can still be faster than S-DANE.
Proof of Theorem 25.
Proof. According to Lemma 23, we have, for any 2 O.
Br - I|v" - -x'1l 1²2
A.fs,(x)) + artife (x'+
2
2 Artifs, (x*+1 + Br41
| 1|
2
- 1
11
+ Ar+1
||x,++1 x²1 - Art1 X se
Σ
FLV
2
iEg,
According Lemma (with S,, x
and y y"), for any 7 we have
Es. [sx[(x((+) - (x
n n 7
1
A.
Es, [|] y
- 1 2x
22
2
(a.) 7
A
1 Es,
Σ
|xx y [P]]
28
+
27
Se
Adding fx*1((x*+1) to both sides of the first display, taking the expectation over S, both sides, substituting the previous upper bound, and setting 7 (n-s)
s(n-1)e' with e' O, we get
Br
Axf(x)) a.t1f"
lI x21
2 Ar+1 Es. [f (x
Br41 Es. [[]v]] rl + =1
6, A.
1
1 Σ
+ Art1
Es.
|x y" [13]
2
27
Art1
Art1 E.s,
1
V V Fa. (x..+))))
les.
Denote all the randomness (Ei. Jes, by &.. Since &rr is independent of the choice of S, for any i E [n], taking the expectation over & on both sides of the previous display and using the assumption that Es..rr Eves, (xir,+)1)) < Es..sr 22 Exes, s, Xxr+t - - Ae' we obtain
A-s(x") )=="")>> + a.t1s" + B, l|v"
x11
2
27

---

Br+1
2 Ar+1 Es
+
Esrie [||v" - x" x'11
+ Art1
A
8, + A.
&s
1
||xi,t1 - -y'13
A Ant
27
By choosing À (n-o) on both sides, get
+2 As), we have that 2/2 (6,+
Taking the full expectation
s(n-1)e'
27
By - A BIf +1]
B,+1
B[[l= x 11 Ar+1 e'.
A, B[f(x')) + #0+1ff" +
2
2
2
Subtracting Ax on both sides, summing up from r = 1 H - 1 and using the fact that Ao 0. Xg and Bg = 1, get
Ar AR - 17+ BR El]
x" (1 - ||
x"11
Σ Ar
Dividing both sides by An, setting e' nh and using the fact that the sequence { (A,) is non-decreasing, get
B -[f(x"))-1
x
x
x
2AR
We apply Lemma 12 with À get
(1
[(1
112
exp
VI++V
BB
1]
AR
4
when & 4A, and Ar
ix (1 + v 2( when 2 4A. Letting these lower bounds be larger than
xx xx-x"11 we get
R Ω ( V V
log 1+ min t} >|| - xx 112
VB
Plugging À = 12 (n==fi + (6. + A.)) into the last display and rearranging, we get the condition for R.
D.3.1 Stochastic Local Solver
Corollary 26. Consider Algorithum 2 under the same settings as in Theorem 25, Consider the same stochastic local solver used in Corollary 21. To achieve
DFF, (
Σ
Ec, Ee.- - 4R)
4
FEB,
SES,
each device requires at most the following number of stochastic mini-butch oracle calls:
L+> F
(L+ A)o²H HT
log L L+D
K ==
+
(n
AJAE
Proof. The proof is the same that for Corollary 21.
E Dynamic Estimation of Similarity Constant by Line Search
Theorem 27, Consider Algorithum 3. Suppose that each function Ss is p-convex for some μ 2 0, and (fi)
have 8-SOD for some 8 Let X S Then, for any R 2 1, holds that
n
28D
1(x
2[(1 + 13 I 1
R
where R = arg min (xc x f(x). To ensure that f(x*) - ff S e for any given >> 0, it suffices set
6 (6+ log [+ 1T7 )):
R == e
where D = 1xx - x'I|. Furthermore, the total number of communication rounds spent inside the r- and k- loops since the start of the algorithm and up to the moment R has been computed is
R-1
25
((1)
Σ (kr + + +)
R log
X
k=9
28

---

Algorithm 3 S-DANE with line search
1: Input: x> 0, 2 0, x" =v" Rd. Let = f- ft.
2: Set λo,o = X.
3: for 0,1.2....da do
4: for k 0, 1.... do
5:
for each device E [n] in parallel do
6:
1, a arg xeRd (x) == f((x) + (Vh,(v"), (Vb.v) + A v
7:
(stop running the local solver once [ S X v'(l Aggregate local models: x x*1 I
8:
9:
Σ (V ((x...))).)))) + Vha(x* v" xi.++1k) 2 2dnk
Σ V V (x(
then
10:
k = k and break the loop.
11:
Ar 2
12:
A kr, X k,, x x +l,krr A.11.0 = 1/.
v=+ arg min E. (VV + #1 - -xxr+1lll 11 1 | v-19).
Proof. According to Lemma 15 (with x; X1,r+1, v = = [n]] and Nz = x1 and our requirement on V F (x.2+1)) whenever Ar.k 2 26, we can estimate
Vi + Vh,
-xir-1)
2Ar, n
An,b ō
1 1
X1,
V [Fi,-(x,,+1)]]
n
An, n
An.k 20
x
200.
Hence, any iteration of the r-loop, the corresponding k-loop eventually terminates. Further, since λo,o S we can easily prove by induction that the quantities Ar, stay reasonably bounded:
26,
Ne. S 26, Ay Ar, I 46 Amax;
Vr 2 0.
(E.I)
Proceeding exactly in the same way as in the proof of Lemma 16 and using the termination condition of the k-loop, we conclude that, for any 2 0, it holds that
Ar [ + 1 p/Ar
xx11 S llvv xx11 1
2
view of (E.1), this means that, for any r 2
[J ((x + 11/ /maax || | S llv x11
Amax
Following the same proof as in Appendix C.2 but with À replaced by Amax. obtain the first two claims.
It remains to estimate the total number of communication rounds required to construct the point x In order carry the k-loop, the server needs to compute Vf(("" and send this vector, as well v" and Ar,o, to each client, which requires ((1) communication rounds. Every iteration the k-loop also requires ((1) communication rounds, and the total number of such iterations is k,-. Thus, every iteration of the r-loop requires Ok: + 1) communication rounds. Furthermore, during the corresponding rounds, the server may also additionally compute the function value (x**2) needed for updating the output point **+1, this could be done, c.g., inside the k-loop, alongside with the computation of the gradient V fx(x*+1++ needed to evaluate the "if" condition. Thus, R can be indeed computed after ((1) RR-1 (ky + 1) communication rounds. estimate the latter sum, observe that, by construction, for any
have Ar-1.0 = f 1 Taking
logarithms, we see that k;- = + loga
A.11.0 for any r 2 0. Thus, An.a
R-1
Σ (k+
Σ
2 + log,
Aet1.
2R + loga
AR.0
25 X < 2R log2
X
20,0
where the final inequality is due to (E. and our choice of Ao,0-
29

---

Algorithm 4 ACC-S-DANE with line search
1: Input: i 0, 2 0, x"
-v" E Rd. Let h, = fc. A X.
2: Set Ao 0,
3: for =0,1,2........
4: for k = 000,, 1.......dd do
Find 1, from the equation An,k (Arta+41, Set Art1, = A. + a.-
5:
6:
y Ar+1
Ar x"
1.1 v". A,+1.
7:
for each device [r] in parallel do
8:
1, k arg min (F)),(()) == f((x + (7h:()"))))) Arg ||x - 11
9:
(stop running the local solver once ( S
And ||x,+I, 7)
10:
Aggregate local models: x *1, = I
Xi,
11:
if Σ (V(((xi++L.))) + Vh.(x y xi 2A, Σ V (i(xir+1ee)] then
12:
k k and break the loop.
13:
Ar,++1 2
14:
Ar Ar.k.. Xt.r+1 kk, x x * ar+1 ax Art10 2/r
15:
Art1 Ar ar+1, Br+1 B, -00+1.
16:
argggmi mingen frll D (V
- x Ry || v²1
Theorem 28. Consider Algorithm Suppose each function fa is p.-covex, for some μ 2 O, and {fi] have 8-SOD for some 8 Let À $ 26. If 168, then, for any R 2 L, it holds that
2μD²
86D
f(x²) f" (1 168 n (1
R
R
165
where D) 1|x0 x' 1|. Otherwise, fxx 8r
BSD
for any R 2 1. To ensure that
(I+v the ((R-1)
f(x f for any given e > 0, it suffices to set
R R= @
16+ log 1
14,
Furthermore, the total number of communication rounds speni inside the r- and k-loops since the start of the algorithum and up to the moment x has been computed is
R-1 0(1)
Σ (kr +1) +1) k=0
20
log
Proef. Using the same reasoning as in the proof of Theorem 27, can justify that, at any iteration of the r-loop, the corresponding k-loop eventually terminates, and Ar stays uniformly bounded as in (E. Next, proceed the same way as proof of Lemma 23 and use the termination condition k-loop to obtain that, for any r 2 O.
Br+1 | -x A.f((x) - f"]+ Br l - 2
A [x 11 f
2
This shows that, for any H 2 1.
D²
s(x
2AR
estimate the rate of growth Ar. we use equation a++1 =\t Or+1. bound on A from (E.1). This gives us, for any 2 0, the following inequality:
Art1
S Amax.
aft1
where B, = + p.y,. Invoking Lemma 12, we get a lower bound on Ag. and the first claim follows.
bound on via E can be justified by the same argument as in the proof Corollary 24.
To estimate the total number of communication rounds, we can follow exactly same argument as in the proof of Theorem 27.
30

---

101
107
10a
40000
50000
Communication rounds
Tatalnumbe gradienn oracle calls
S:DANE-GD (ours)
Figure E.1: Comparison of S-DANE against DANE for solving convex quadratic minimization problem with the number of local steps.
Algorithm 5 S-DANE (DL)
1: Input: À > 0, [0, 1), x" v RRd, E [m]
2: for r = 0,1,
3: Sample S, E (() uniformly at random without replacement
4: for each device
in parallel do 5: Set X1, R arg x (F.((x)): where
6:
(x, f((x) (x. V/((v) V.s,(v')) +2 1=1x-v1]
v/1.?
(option 1)
7:
(x) = fi(x)+ + f v
(option 2)
8:
Set x ++ = 1 Xi,++
9:
Set r 7x*+1 (1->)v² - - Eies, V(i(x.+t1)
Additional Details on Experiments
F.1 Convex Quadratics
We generate random vectors }br,} and diagonal matrices (Acs) same way as [22] such that max., { |ll = 100 and d Rs 5. We use n = 10, m 5 and d = 1000. We compare S-DANE and ACC-S-DANE with DANE. We use the standard gradient descent (GD) with constant stepsize 200 < 2z for all three methods the local solver, where L is the smoothness constant of cach f.. We use A = 5 for all three methods. We use the stopping criterion Fu, S ||x v"I for our methods (v"
becomes y for the accelerated method). We use | FLr (x(,++)) < ||x++1 x"|| for DANE.
F.2 Deep Learning Tasks
We simulate the experiment on one NVIDIA DGX A100. We split the training dataset into a = 10 parts according to the Dirichlet distribution with a = 0.5. We use SGD with a batch size of 512 as local solver for each device. For all the methods considered in Figure we choose the best number of local steps among {10, 20, .. 80} (for SCAFFNEW, this becomes (0.02, 0.05, 0.1 ). For this particular task, is often observed
inverse of the probability) and best learning rate among
using control variate makes the training less efficient [38]. The possible issue comes from the fact that local smoothness is often much smaller than local dissimilarity for this task. We here remove the control variate term on line in S-DANE which is defined (v")). Moreover, if write the explicit formula for y*+1 line 8, it becomes
(x. V.((v)) rr+l 7x + (1 v)v" ! Eies, (((xx,++) with y E [0, 1] and rj > O. set y = 0.99 and n to be the local learning rate in our experiment. The method can be found in Algorithm 5. Note that the only difference between it and FEDPROX is the choice of the prox-center. The best number of local steps for the algorithms without using control variates is 70 while for the others is 10 (otherwise, the training loss explodes).
F.3 Implementation
To implement Algorithms and (Algorithm with option is same as Algorithm 1). each device has the freedom to employ any efficient optimization algorithm, depending on its computation power and the local data size. At each communication round r. these local algorithms are called to approximately solve the sub-suubee problems defined by (fir),, ). until the gradient norm satisties certain accuracy condition that is stated in the corresponding
31

---

theorems. The server only needs to perform basic vector operations. Note that C, defined in those algorithms has unique solution so that v "+1 can be explicitly derived (in the same form line 9 in Algorithm S).
G Impact Statement
This paper presents work that aims to advance the field of distributed Machine Learning. There are many potential socictal consequences of work, of which feel must he specifically highlighted here
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and the introduction, first stately clearly what goals of this study, which is first achieve the hest-known communication complexity and then maintain the overall then discussed the performance of the previous state-of-the-ar algorithms. Finally, we compared our methods with them (for instance in Table 1) and showed what contribution and scope we made in this paper.
computation efficiency.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 7, we discuss the limitations of our work, in terms of the slightly stronger assumption on j-convexity and the lack of non-convex analysis. On top of that, also discuss potential future works building on our results.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiscless settings, model well- -peeffpcciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
their best judgment and recognize
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We make clear assumptions and provide complete and concise proofs in the Appendix for all the claims written in the main text.
33

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the necessary information to reproduce our experimental results in Section 6 and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, the contribution is specific model and empirical evaluation, may be necessary to either make possible others to replicate the model with same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) contribution a new model (e.g.. a large language model), there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: provide github link the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (bttps://nips.cc/ppll
guiider/CoodeSuumniesssooooooo. for more details.
34

---

While we encourage the release of code and data, we understand that this might not be possible. "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (https: http//iippcccccccceee cc/pubii// http://dddSuniiiiinnneeee
reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all training and test details (e-g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of the experiments to reproduce experimental results can be found in Scction 6 and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Two sets of our experiments are defined in a totally deterministic setting (no randomness.) We ran the other two sets of experiments three times with different randomness seeds. The results are almost indistinguishable.
Guidelines:
answer means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
35

---

Answer: [Yes]
Justification: provide information on the computing resources at start of Section the deep learning experiment. The other experiments are run on a MacBook Pro laptop.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ hwt.eerrip..cpppiic//cii.c...eoa
Answer: [Yes]
Justification: We have read and understood Code of Ethics and we have conducted research in accordance with
Guidelines:
The answer means that the authors have not reviewed NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide an impact statement in Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deplay- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
36

---

Justification:
Our resesarch does not have any risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of model, for example by requiring users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g.. code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite and credit the original owners of the open-source datasets LIBSVM and CIFAR 10 in the paper.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
authors should state which version
asset is and, possible, include a URL.
name of license (e.g.. CC-BY 4.0) should be included each asset. scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithcode. thcode.comm dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged. both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release any new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate details of the dataset/code/moodll as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
37

---

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
38


Reviews:
[{"review_id": "r5NO5FcIFR", "reviewer": "Reviewer_peSJ", "summary": "The paper introduces a new variant of the existing DANE algorithm for federated learning. The paper integrates the stabilized proximal point method into DANE to form S-DANE. Convergence analyses are provided showing that S-DANE has the same rate as DANE but with better dependency on the communication round. The accelerated variant of S-DANE is also proposed which achieves better rates than S-DANE. Numerical experiments are able to show the advantage of S-DANE and Accelerated S-DANE compared to existing works.", "questions": "1. In both algorithm 1 and algorithm 2, aseach device $i \\in S_r$ performs the update in parallel, I do not see how $\\nabla f_{S_{r}} (v^r)$ or $\\nabla f_{S_{r}} (y^r)$ is computed since it requires the evaluation of the gradient of $f_j$ for $j \\in S_r, j\\neq i$?\n\n2. In Figure 4, Scaffnew actually performs worse than Scaffold, this is somewhat surprising given the experiments in Mishchenko et al. (2022). What is your explanation on this?", "limitations": "The paper adequately discusses the limitation.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The related work discussion of the paper is good. I am able to see how the paper is positioned among existing works.\n- I have not seen stabilized proximal update applied to federated learning (FL) so the idea in the paper appears to be new for FL.\n- S-DANE and accelerated S-DANE are able to achieve the best-known rate for their corresponding setting.\n- The claims are supported by theoretical analyses.\n- The convergence results are done under both full and partial client participations.\n- Extensive experiments showing the advantage of S-DANE/Accelerated S-DANE.", "weaknesses": "- The design of the algorithm is a little bit unclear, see questions below.\n- Somehow I only see the full 8 algorithm in the left-most plot in Figure 2, while in the remaining 3 there are only about 4 algorithms displayed in the plots, i.e. I cannot see Snaffnew, AccGradSliding, DANE-GD, GD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great evaluation of our paper. Your every comment is important to us. We did our best to\nunderstand and reply to your constructive feedback as follows:\n\n>  W1 \\& Q1\n\nThanks for the question. The description can be found from line 158 to 163. Specifically, during each communication round $r$, the server first sends $v^r$ to the sampled clients. Then each client $i \\in S_r$ computes $\\nabla f_i(v^r)$ and sends this vector back to the server. Afterwards, the server computes $\\nabla f_{S_r}(v^r) = \\frac{1}{s}\\sum_{i \\in S_r} \\nabla f_i(v^r)$ and sends $\\nabla f_{S_r} (v^r)$ back to these clients. The same procedure was written in Section 4 in the Scaffold paper [1].\n\n> W2\n\nThanks for the great observation. The remaining 3 plots are about partial-client participation. Indeed, we do not plot these four algorithms as partial-client participation is not considered in their original papers. We will make it clear in the manuscript.\n\n> Q2\n\nThanks for the great observation. Note that Scaffold has two choices of control variates (cv). The cvs between Scaffnew and Scaffold are different in our experiments. For Scaffold, we use the same cv as DANE (option I in [1]) while for Scaffnew the cv is similar to option II in [1]. \n\n1. In the Scaffnew paper, they use the same cv for both Scaffnew and Scaffold. This is why they perform similarly (with tuned parameters).\n\n2. So far, no results have shown that the cv for Scaffnew can achieve communication reduction under second-order similarity. However, the other cv (option I) can exploit similarity [2]. This is perhaps why we can observe a better performance of Scaffold in certain experiments.\n\n3. Figure 4 is about deep learning. We expect that the cv for Scaffnew {option II} can be slightly less stable than another cv (option I). \n\n[1] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. ICML 2020. \n\n[2] Xiaowen Jiang, Anton Rodomanov, and Sebastian U Stich. Federated optimization with doubly regularized drift correction. ICML 2024.\n\nWe thank you again for your great review! If you agree that we managed to address all issues, please consider raising your score–this would totally boost our chance. If you believe this is not the case, please let us know and we will try our best to answer your every question!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rzi5qK2UMW", "reviewer": "Reviewer_QbRz", "summary": "This paper considers the problem of distributed optimization under second-order similarity under (strong) convexity and smoothness. The paper proposes a new algorithm, Stabilized DANE, which (a) matches the best-known communication complexity under Hessian similarity while (b) requiring that local computation problems are solved only up to an approximately constant accuracy (i.e. not an accuracy that depends polynomially on $1/\\epsilon$ where $\\epsilon$ is the desired solution accuracy). The authors also consider partial participation, where only a subset of the clients is available at any given time, and also provide an accelerated version of Stabilized DANE.", "questions": "- Please address my concerns in the weaknesses section.\n- Is the hessian similarity actually smaller than the smoothness over the training trajectory in the CIFAR10 optimization task?\n- Can you derive theory for the variant of your method with no control variates that you use in the deep learning experiment? Does it just reduce to stabilized prox then?", "limitations": "N/A.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is written clearly and the new algorithms are well-motivated. The proofs are easy to follow.\n- The stabilization technique is elegant and clearly obtains both theoretical improvements (mostly in local computation complexity, as far as I can tell) and in practice (as shown in Section 5). \n- The algorithms developed build on DANE, which is well-known and already a strong algorithm.", "weaknesses": "1. \"It is necessary to assume a certain level of dissimilarity\" (lines 208-212), I'm not actually sure that's _necessary_. There are certainly upper bounds without this assumption. Why is it necessary? Relaxed assumptions like expected smoothness are often enough even in this setting.\n2. The complexities given for Acc-SVRS/Catalyzed SVRP in Table 1 are not consistent with how the other rates are presented in this work-- the $n$ factors should not be there, since in Table 1 for every other work full participation is assumed anyway (i.e. they should all have an $n$ factor multiplied). Which brings me to another point: What is the advantage of S-DANE over SVRS/SVRP? Is it only the higher efficiency of local computation?\n3. The paper right now has a few different settings (full participation, partial participation) that contrast to other settings in prior work. Remark 6 attempts to explain the difference but I think this should be explained more clearly and placed in the main body of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great evaluation of our paper! Your every comment is important to us. We did our best to\nunderstand and reply to your constructive feedback as follows:\n\n> W1\n\nYes, this sentence is confusing! We will rewrite it. \n- This sentence (that was first written in [22]) particularly refers to the setting where $n$ might go to infinity and each client does not store any vectors on its device (what people call 'stateless'). Then this sentence says to prove convergence, we need to assume a certain level of dissimilarity, which is correct. This sentence does not necessarily refer to using Definition 4.\n- When $n$ is sufficiently small, then we can use SAGA-type methods (considered in Scaffold) that ask each device to store certain control variates. Then yes, no dissimilarity assumption is required. Meanwhile, we need to assume the $L$-smoothness of the function and the final complexity depends on $L$.\n- In this paper, we consider the same setting as in paper [22] (with potentially very large $n$). Then as you mentioned, the question is if using Definition 4 is too strong.  Interestingly, we do not use any smoothness-related assumptions to prove the communication complexity. Therefore, the function classes also include Lipschitz and non-smooth functions. In this case, we believe using Definition 4 is necessary.\n- However, as you mentioned, if we further assume that the function $f$ is $L$-smooth, then Definition 4 can exactly be relaxed. For instance, one can consider $\\frac{1}{n}\\sum_{i=1}^n ||\\nabla f_i (x) - \\nabla f(x)||^2 \\le \\zeta^2 + \\beta ||\\nabla f(x)||^2 \\le \\zeta^2 + 2\\beta L (f(x) - f^\\star)$ with $\\beta > 0$, which captures the growth behaviour of the dissimilarity. The only change in the proof will appear in line 589 where we have an additional error term which can be canceled by choosing small $\\gamma$ and large $\\lambda$ that depend also on $\\beta$ and $L$.\n\nWe will make this sentence clear and add more discussions to the manuscript according to your suggestion!\n\n> W2 \\& W3\n\nThanks for raising these points! We refer to the attached PDF for clarification of your concerns. (In Table 1, we report the number of communication rounds for all the methods including SVRP for which the number of rounds is in the same order as the total number of vectors transmitted.) Please let us know if there is anything unclear. We will move these discussions to the main paper as you suggested!\n\n> Q2\n\nThis is a very interesting question. From our experiments, we saw that the estimated dissimilarity quantity defined as in Figure 3 ranges from $10^{-4}$ to $10^{-3}$ in the CIFAR10 task (and we choose $\\lambda = 10^{-3}$ and the local learning rate $10^{-1}$ for S-DANE). Unfortunately, we did not record the local estimated smoothness quantity for our experiments. Empirically studying the relation between Hessian similarity and smoothness in different deep-learning tasks with various NN structures is indeed a very interesting future direction.\n\n> Q3\n\nYes! The proof is similar to the one for S-DANE with $1$ client participation. Let us compare Algorithm 3 with $s= n$ and Algorithm 1 with $s = 1$. The main difference is that the former considers deterministic averaging and the latter considers sampling (in the proof, we need to take expectation for the latter which is similar to the deterministic averaging for the former). Therefore, if we study Algorithm 3, we have to similarly assume a certain first-order dissimilarity condition. We are not fully sure what you mean by stabilized prox. But yes, it can be seen as stabilized FedProx.  \n\nWe thank you again for your great review and we appreciate your support! If you agree that we managed to address all issues, please consider slightly raising your score–this would totally boost our chance. If you believe this is not the case, please let us know and we will try our best to answer your every question!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aGUSBRzLK1", "reviewer": "Reviewer_iL8h", "summary": "The paper introduces a stabilized version of DANE (S-DANE). It replaces the proximal point step with an extragradient-type step. With the well-designed subproblem criterion, the number of local gradient oracle queries improves over DANE in logarithmic terms. It further combines Monteiro-Svaiter acceleration with S-DANE, which leads to the best communication complexity. The paper also considers partial client participation.", "questions": "1. Apart from its resemblance to the extragradient method when \\(\\mu = 0\\), what is the underlying motivation for considering equation (3)?\n\n2. Regarding line 129 on page 4, the statement \"such computation overhead cannot be avoided\" seems imprecise. In the strongly convex case presented in [35] and both the convex and strongly convex cases described by Lan et al., 2023, the number of gradient steps required to solve the subproblem does not necessarily increase with \\(k\\).\n\nLan, G., & Li, Y. (2023). A Novel Catalyst Scheme for Stochastic Minimax Optimization. arXiv preprint arXiv:2311.02814.  \n\n3. Some suggested references for extragradient:\n\n\nKorpelevich, G. M. (1976). The extragradient method for finding saddle points and other problems. Matecon, 12, 747-756.\n\nNemirovski, A. (2004). Prox-method with rate of convergence O (1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization, 15(1), 229-251.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a) S-DANE eliminates the logarithmic term present in DANE.\n\nb) Accelerated S-DANE achieves a communication complexity of \\(O(\\sqrt{\\delta_A/\\mu \\log(1/\\epsilon)})\\), which surpasses the complexities in the existing literature.\n\nc) The clarity of the manuscript's writing is good.", "weaknesses": "See \"Questions\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great evaluation and the support of our paper! \n\n> Q1\n\nThanks for the interesting question. The geometric meaning of equation (3) can be found in [1]. Previously, we derived this equation directly from the proof. Since we want to have the one-step recurrence of the form: \n$$ a\\bigl( f(x^{r+1}) - f(x^\\star) \\bigr) + \\frac{1+\\mu a}{2} ||v^{r+1} - x^\\star||^2 \\le \\frac{1}{2} ||v^r - x^\\star||^2 ,  $$\nthen we can first use convexity at the beginning of the proof:\n$$ a f(x^\\star) + \\frac{1}{2} ||v^r - x^\\star||^2 \\ge a \\bigl( f(x^{r+1}) + \\langle \\nabla f(x^{r+1}), x^\\star - x^{r+1}\\rangle + \\frac{\\mu}{2} ||x^{r+1} - x^\\star||^2 \\bigr) + \\frac{1}{2} ||v^r - x^\\star||^2 . $$\nThen it is natural to set $v^{r+1}$ to be the minimizer of the right expression in $x^\\star$, which is $(a\\mu + 1)$- strongly convex. After that, we can approximately get the main recurrence.\n\n[1] Svaiter B.F. and Solodov, M.V. A hybrid projection-proximal point algorithm. Journal of Convex Analysis. 1999.\n\n> Q2\n\nYes, this statement is wrong for accelerated proximal-point methods. Indeed, apart from your references, we also found several adaptive catalyst frameworks [2,3] that successfully remove the logarithmic dependency on $k$. Many thanks for pointing this out and providing the important reference. \n\n[2] Carmon, Yair, et al. Recapp: Crafting a more efficient catalyst for convex optimization. ICML 2022.\n\n[3] Ivanova, Anastasiya, et al. Adaptive Catalyst for Smooth Convex Optimization. International Conference on Optimization and Applications. 2021.\n\n> Q3\n\nWe appreciate the reviewer for providing these nice references and we will add them to our manuscript!\n\nWe thank the reviewer again for your great review and appreciate your help in improving the paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4pC9dvMlb", "reviewer": "Reviewer_vDCp", "summary": "An algorithm for distributed convex optimization with partial participation is proposed, under a similarity assumption.", "questions": "Do you assume that $\\delta\\geq \\mu$? Because a complexity of $\\sqrt{\\delta/\\mu}$ is a strong statement, it means that the algorithm converges in 1 round if $\\delta \\rightarrow 0$. FedRed, for instance, has $(\\delta+\\mu)/\\mu$. Please check.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed algorithm has complexity Acc-S-DANE has claimed communication complexity O(sqrt(delta/mu)log(1/epsilon), for the first time. (I did not check the details of the proof but the main lines look correct to me).", "weaknesses": "* I want you to discuss your statement \"Suppose we use the standard gradient descent as a local solver, then the number of gradient steps required to solve the subproblem increases across the iterations k\". I don't think this is correct. In fact, you should discuss the 5GCS algorithm in Grudzien et al. \"Can 5th Generation Local Training Methods Support Client Sampling? Yes!\" AISTATS 2023. 5GCS is essentially a Point-SAGA algorithm with inexact computation of the proximity operators, and different strategies to solve these subproblems are discussed. The number of GD steps does not increase, you just pay a log(L/mu) factor for that in the complexity.\n\n* You need to provide 2 tables: Table 1 in the full participation case, and a second table on existing algorithms for partial participation of $s$ clients. In particular, 5GCS and TAMUNA in \"TAMUNA: Doubly Accelerated Federated Learning with Local Training, Compression, and Partial Participation\" arXiv:2302.09832, 2023 have communication complexity $O\\left(\\sqrt{\\frac{nL}{s\\mu}}+\\frac{n}{s}\\right)\\log(1/\\epsilon)$.\n\n* The complexity of Catalyzed SVRP reported in Table 1 is not correct. This value is for a different communication model, which counts \"exchanging a vector between the server and one of the clients as a single communication step\". So, if $s$ clients communicate, the complexity is multiplied by $s$, and $s=1$ gives the best complexity in this sense, since asynchronicity is encouraged. This is clearly different from measuring the number of communication rounds, in which $s<n$ *worsens* the complexity.\n\n* You should discuss the paper Beznosikov et al. \"Similarity, Compression and Local Steps: Three Pillars of Efficient Communications for Distributed Variational Inequalities\", arXiv:2302.07615\n\n* You should compare in your experiments to some of the methods mentioned above, in the two heterogeneous (delta = L) and homogeneous (delta<<L) regimes: does your method improve on them by exploiting the similarity, whereas other methods do not? (it might be that an algorithm also benefits from similarity, but has just not been studied under the similarity assumption in the literature).\n\n* Definition 4 is very restricting, this means that the $f_i$ are all the same up to a linear difference. This is an auxiliary result but it is not very useful in practice.\n\n\n\ntypo in several places : mu-convex -> mu-strongly convex\n\nThe paper [4] has been published in ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great evaluation of our paper. Your every comment is important to us. We did our best to\nunderstand and reply to your constructive feedback as follows:\n\n> W1\n\nThanks for the nice question. \n\n1. This statement only (appears and) refers to the accuracy condition written in line 124. To satisfy this condition, if using GD, then the number of steps should depend on $k$, which is correct. \n\n2. This condition is only stated for the original proximal point method (eq 2). It is well-known that the local computation has an unnecessary logarithmic dependency on either k or the final accuracy for this method, which can be improved by using more advanced variants [1,2,3]. \n\n3. 5GCS is different from the original proximal point method when $n=1$. In Table 1, many previous works (with proximal-point steps) also do not require increasing the number of GD steps, such as AccSVRS. These facts do not contradict this statement because these are different methods. \n\n4. Acc-S-DANE is strictly better than 5GCS at least in the full-client participation settings.  **1)** the communication complexity depends on $\\delta_A$ instead of $L$ for 5GCS  **2)** Even if $\\delta_A = L$, the number of local steps is 1 instead of $\\tilde{O}( \\sqrt{L/\\mu} )$ for 5GCS. 3) we prove convergence for the function value which is stronger than the squared distance for 5GCS. 4) it works well when $\\mu \\to 0$ while 5GCS seems not well-defined when $\\mu = 0$.\n\nNevertheless, 5GCS is an excellent method in some settings and we will add more discussion on it. \n\n[1] Svaiter B.F. and Solodov, M.V. A hybrid projection-proximal point algorithm. Journal of Convex Analysis. 1999.\n\n[2] Carmon, Yair, et al. Recapp: Crafting a more efficient catalyst for convex optimization. ICML 2022.\n\n[3] Ivanova, Anastasiya, et al. Adaptive Catalyst for Smooth Convex Optimization. International Conference on Optimization and Applications. 2021.\n\n> W2 \n\nThanks for the suggestion! However, it might be difficult to compare these two methods directly with our methods and others, as different settings are considered: for instance, 1) we consider cross-device setting where $n$ might be very large and each client is stateless (no control variates are stored), and hence SAGA-type methods are not applicable 2) certain first-order dissimilarity condition has to be assumed in this setting 3) as a result, our rates do not have a dependency on $\\frac{n}{s}$ but instead on $\\zeta^2$. \n\nWe will add the rates and discussions of these two excellent methods in our manuscript.\n\n> W3\n\nThanks for the great question. The complexity is correct. In Table 1, we report the number of communication rounds for all the methods including SVRP (for which the number of rounds is in the same order as the total number of vectors transmitted). We refer to the attached PDF where the settings and how SVRP works are explained in detail. The main metric of SVRP/SVRS is different and we separated these two algorithms in Table 1 and added Remark 6.\n\n> W4\n\nThanks for the reference. This is a strong work. However, 1) The main target is the same as SVRP/SVRS, i.e. minimizing the total number of bits transmitted across rounds, which is different from this work. 2) It does not consider acceleration (while AccSVRS does) but uses compression to reduce the bits (while AccSVRS does not) and 3) in the end, it requires $\\tilde{O}( n + \\delta \\sqrt{n} / \\mu )$ communication rounds scaling with $n$.\n\n> W5\n\nThanks for the interesting suggestion. The paper is based on the existing theoretical results. Studying if certain previous algorithms can exploit similarity is a bit irrelevant to the focus of this paper, which could instead be an interesting future work.\nAlso in many cases, by tuning hyperparameters, some methods can outperform their theoretical upper bounds. For instance, Scaffnew and Scaffold have a similar performance by tuning stepsize while Scaffold has not been proven to achieve acceleration. But as you suggested, we will try to see if some of your mentioned methods can benefit from the similarity in our toy example.\n\n> W6\n\n1. Interestingly, we do not use any smoothness-related assumptions to prove the communication complexity. \n    Definition 4 says the difference function is Lipschitz, and the original functions can be non-smooth. \n\n2. Lipschitz functions are not just linear functions. The log-sum-exp is not linear but Lipschitz.\n\n3. If we further assume that the function $f$ is $L$-smooth, then we agree that Definition 4 can be relaxed. For instance, one can consider $\\frac{1}{n}\\sum_{i=1}^n ||\\nabla f_i (x) - \\nabla f(x)||^2 \\le \\zeta^2 + \\beta ||\\nabla f(x)||^2 \\le \\zeta^2 + 2\\beta L (f(x) - f^\\star)$ with $\\beta > 0$, which captures the growth behaviour of the dissimilarity. The only change in the proof will appear in line 589 where we have an additional error term which can be canceled by choosing small $\\gamma$ and large $\\lambda$ that depend also on $\\beta$ and $L$.\n\n4. The interesting regime of considering partial client participation is when the number of clients $n$ is potentially very large. Then Definition 4 allows an almost unbounded distribution of the gradient dissimilarity  (similar to the notion of 'stochastic gradient noise' for centralized SGD). Therefore, we believe using this assumption is still reasonable in practice. More detailed discussions can be found in Section 2 from [22] where the same assumption is used.\n\n> Typos and reference\n\nThanks for finding the typos and we will update this reference.\n\n> Q1\n\nMany thanks. Yes, in Table 1, we reported the interesting regime $\\mu \\le \\Theta(\\delta)$ and we will add it to the text.\n\nWe thank you for your great review and appreciate your help in improving the paper! If you agree that we managed to address your concerns, please consider raising your score–this would totally boost our chance. If you believe this is not the case, please let us know and we will try our best to answer your every question!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yG99alTOXT", "reviewer": "Reviewer_TYfb", "summary": "The authors of the paper extend the DANE algorithm to the stabilized-DANE (S-DANE) algorithm based on intuition from stabilized proximal point method. They also enhance the proposed S-DANE method with Monterio-Svaiter acceleration. The algorithms proposed by the authors allow for partial participation and various local solvers to some extent. Convergence analysis and experiments are provided.", "questions": "1. In Table 1, why do the authors compare the number of communication rounds rather than the total amount of communication for each algorithm? Does all of those algorithms have the same communication overhead in each communication round? If the reviewer is not mistaken, some algorithms (such as SVRP) in the table work in the single client setting, is it fair to compare them directly to S-DANE in the full client participation setting?\n\n2. For S-DANE (and its accelerated version), how to determine the proper stepsize? In the convergence guarantee, $\\gamma$ depends on the unknown average Hessian dissimilarity constant $\\bar{\\delta}_s$ and the bounded gradient diversity constant $\\zeta$. Could the author illustrate on how the two constants are estimated?", "limitations": "No.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The presentation of the paper is clear, logical and easy to follow.\n2. The paper proposes two novel algorithms S-DANE and its accelerated version and provides convergence analysis of them, experiments are also provided to validate the claim of the paper.\n3. The proposed algorithms allow for partial participation and various local solvers in some sense.", "weaknesses": "1. The partial client participation case of S-DANE and accelerated S-DANE both relies on the bounded gradient dissimilarity assumption, which is violated in many cases, and is not accurate enough when describing the effect of data heterogeneity. This raises concerns of the reviewer whether or not the algorithm is practical in the case of partial client participation.\n\n2. The paper wants to address the communication bottleneck in FL setting by reducing the total number of communication rounds the algorithm needed to reach a certain accuracy level. However, it is also important to take into account the number of bits transferred from the client to the server as they are often constrained by the bandwidth. In fact, the terms communication complexity/efficiency appeared in the paper are quite misleading.\n\n3. As the authors have suggested, the analysis is carried out in the case where each local objective $f_i$ is $\\mu$-strongly convex, which is a little restrictive. \n\n4. There are some typos (e.g., Line 265).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great evaluation of our paper. Your every comment is important to us. We did our best to\nunderstand and reply to your constructive feedback as follows:\n\n> W1\n\nThanks for the great question. Here are some justifications. \n\n1. Interestingly, we do not use any smoothness-related assumptions to prove the communication complexity. Therefore, the function classes also include Lipschitz and non-smooth functions. In this case, we believe using Definition 4 is reasonable. \n\n2. If we further assume that the function $f$ is $L$-smooth, then we agree that Definition 4 can be relaxed. For instance, one can consider $\\frac{1}{n}\\sum_{i=1}^n ||\\nabla f_i (x) - \\nabla f(x)||^2 \\le \\zeta^2 + \\beta ||\\nabla f(x)||^2 \\le \\zeta^2 + 2\\beta L (f(x) - f^\\star)$ with $\\beta > 0$, which captures the growth behaviour of the dissimilarity. The only change in the proof will appear in line 589 where we have an additional error term which can be canceled by choosing small $\\gamma$ and large $\\lambda$ that depend also on $\\beta$ and $L$.\n\n3. The interesting regime of considering partial client participation is when the number of clients $n$ is potentially very large. Then Definition 4 allows almost unbounded distribution of the gradient dissimilarity  (similar to the notion of 'stochastic gradient noise' for centralized SGD). Therefore, we believe using this assumption is still reasonable in practice. More detailed discussions can be found in Section 2 from [22] where the same assumption is used.\n\n> W2 \\& Q1\n\nThanks for this nice question! We refer to the attached pdf for the clarification of your concerns. Please let us know if there is anything unclear. We will add the discussions to our main manuscript.\n\n> W3\n\nYes! However, note that our analysis works for standard convex functions ($\\mu = 0$) as well. In the strongly convex case, when the function is of the form $f(x) = \\frac{1}{n} \\sum_{i = 1}^n \\phi_i(x) + \\frac{\\mu}{2} || x ||^2$, we can rewrite it as $f(x) = \\frac{1}{n} \\sum_{i=1}^n f_i(x)$ with $\\mu$-strongly convex $f_i(x) = \\phi_i(x) + \\frac{\\mu}{2} || x ||^2$. \n\nIn practice, the strong convexity often comes from adding explicit regularization of the local convex loss function (e.g. regularized logistic regression). Then each function typically is strongly-convex and $\\mu$ is often known. \n\nWe can actually propose the version of our method for composite functions $F(x) = f(x) + \\psi(x)$ for which such an artificial split would be unnecessary but we do not do it for simplicity.\n\n> W4: typos\n\nThanks for finding the typos!\n\n> Q2\n\nThanks for asking this important question. We refer to the section of 'adaptive S-DANE with line-search' in the attached pdf for answering this question.\n\nWe thank you again for your great review! If you agree that we managed to address all issues, please consider raising your score–this would totally boost our chance. If you believe this is not the case, please let us know and we will try our best to answer your every question!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive evaluations of our manuscript and we appreciate all the help from the reviewers for improving the paper.

In this work, we aim to develop federated optimization algorithms that 1) minimize the number of required communication rounds to reach the desired accuracy and 2) achieve high efficiency in local computation. These two represent central metrics in federated contexts, given the high cost of establishing connections between the server and the clients.

Specifically, we:
- developed novel algorithms: S-DANE (basic version) and Acc-S-DANE (accelerated version), which achieve the best-known communication complexity (in terms of the number of rounds) and local computation efficiency among all existing basic and accelerated methods (in the full client participation setting). This is achieved by using a more stabilized prox-center in the proximal step.
- further provided auxiliary results about partial client participation and using arbitrary stochastic local solvers, making them attractive in practice.
- provided a simple analysis for both algorithms. We derive convergence estimates that are continuous in the strong convexity parameter $\mu$.

Three reviewers asked about the setting and the SVRP algorithm. We refer to the attached PDF for clarifications.

After submission, we realize the following improvements can be made:
- The proof can be much simplified. We can completely remove the sequence $\\{ v_{i,r+1} \\}$ and only keep $\\{ v^{r+1} \\}$.
- We found an important reference [1] which we will include in the related work section. Our proposed method (S-DANE) recovers [1] for the special case of $n=1$. The paper [1] also contains a geometric explanation of why the stabilized proximal-point method has better local efficiency which could be of interest to the readers of our paper.
- Both of our algorithms can be made fully adaptive by using line-search (in the full-client participation setting). The details can be found in the attached PDF. This is, as far as we know, the first result of adaptive hyper-parameter tuning in the setting of exploiting second-order dissimilarity.

We anticipate an interactive discussion with you, and we will be most happy to answer any remaining questions.

[1] Svaiter B.F. and Solodov, M.V. A hybrid projection-proximal point algorithm. Journal of Convex Analysis, 6 (1):59–70, 1999.

Meta-review: Paper presents a new Federated Optimization algorithm, S-DANE, for convex minimization. First authors identify that employing an extra-gradient step to proximal point method relaxes the tolerance required for proximal point step. They call this technique stabilized proximal point method. This is exploited to reduce the amount of local computations done on clients to constant number of steps as opposed to logarithmic number of steps in DANE algorithm. Additionally, this method can support arbitrary local client solver and partial client participation. Further they employ Monteiro-Svaiter acceleration in the servers to obtain accelerated converge and reduced overall communication complexity. This achieves new state of the art results. Finally, they provide both convex and non-convex deep learning experiments.

All reviewers agreed this is a well written paper. Contributions are clearly highlighted and explained. Paper is novel and relevant for this conference. During rebuttal period authors also found 25 year old paper which proposes the same idea for n=1 case. Reviewers noted that in partial client participation current theorem requires strict bounded gradient dissimilarity condition. Additionally, current theory also assumes all client objectives are strong convex. There we also certain imprecise statements and overclaiming in the initial manuscript. During the rebuttal phase authors managed to address some of these issues and impress upon the reviewers the value of their work. I ask the reviewers to modify the paper and experiemnts as promised to the reviewers.
Decision: Accept (spotlight)