SLowcalSGD : Slow Query Points Improve Local-SGD for Stochastic Convex Optimization
Authors: Tehila Dahan
Kfir Yehuda Levy
Venue: NeurIPS 2024 poster (2024)
Keywords: Stochastic Convex Optimization
Primary area: optimization
TL;DR: The first parallel training method that provably benefits over Minibatch-SGD in Convex heterogeneous training scenarios.

Abstract: We consider  distributed learning scenarios where $M$ machines interact with a parameter server along several communication rounds in order to minimize a joint objective function. 
Focusing on the heterogeneous case, where different machines may draw samples from different data-distributions, we design the first local update method that provably benefits over the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. 
Key to our approach is a slow querying technique  that we customize to the distributed setting, which in turn enables a better mitigation of the bias caused by local updates.

SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimization
Tehila Dahan
Kfir Levy
Department of Electrical Engineering Technion
Department of Electrical Engineering Techmion
Haifa, Israel
Haifa, Israel
t. wwd.aaaaaaapuse www.rppuuu.eecc.ooo.o..o.. www.teechhon..... ac il
kfiryleeryttechiion.acc
Abstract
We consider distributed learning scenarios where M machines interact with parameter server along several communication rounds in order to minimize joint objective function. Focusing the heterogeneous case, where different machines may draw samples from different dattadistribbtionss, we design the first local update method that provably benefits the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. Key to our approach is a slow querying technique that we customize the distributed setting, which in turn enables better mitigation of the bias caused by local updates.
Introduction
Federated Learning (FL) is a framework that enables huge scale collaborative learning among large number of heterogeneous' clients (or machines). FL may potentially promote faimess among participants, by allowing clients with small scale datasets to participate in the learning process and affect the resulting model. Additionally, participants are not required to directly share data, which may improve privacy. Due to these reasons, FL has gained popularity in the past years, and found use in applications like voice recognition [1, 4), fraud detection [2], drug discovery [3], and more [34].
The two most prominent algorithmic approaches towards federated learning are Min bintbbahhhh cattchh-GG SGD [10] and Local-SGD (a.k.. Anderatted-Aeraggnng [27, 28, 33]. In Minibatch-SCD all machines (or clients) always compute unbiased gradient estimates the same query points, while using large batch sizes; and it is well known that this approach is not degraded due to data heterogeneity [37]. On the downside, the number of model updates made by Minibat ch-ibbatthhaeee ch-SGD SGD may be considerably smaller compared to the number of gradient queries made by each machine; which is due to the use of minibatches. This suggests that there may be room to improve over this approach by employing local update methods like Local-SCD, where the number of model updates and the number of gradient queries are the same. And indeed, in the past years, local update methods have been extensively investigated, e.g. [19] and references therein.
We can roughly divide the research on FL into two scenarios: the homogeneous case, where is assumed that the data on each machine is drawn from the same distribution; and to the more realistic heterogeneous case where
assumed that data distributions may vary between machines.
For the homogeneous case it was shown in [36, 13] that the standard Local-SCE method is not supe- rior to Minibat SGD. Nevertheless, [39] have designed an accelerated variant of Local-sGD that provably benefits over the Minibatch baseline. These results are established for the fundamental Stochastic Convex Optimization (SCO) setting, which assumes that the learning objective is convex.
'Heterogemeous here refers to the data of each client, and assume that statistical properties may vary between different clients.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table I: We compare the best known guarantees for parallel learning, our SLowcal-SGD approach for the heterogeneous SCO case. The bolded term the Rate column is the one that compares least favourably against Minibatch SGD. Where G and G. relate to the dissimilarity measures that are defined in Equations (1) and (3), The Rmin column presents the minimal number of communication rounds that are required obtain linear speedup (we fixed values of K, M and take 1). Note
that we omit methods that do not enable a wall-clock linear speedup with M, e.g. [29, 30].
Method
Rate
Rmin (a = 1)
MiniBatch SGD
MK
[10]
VMKR
Accelerated MiniBatch SGD [10, 25]
(MK) 1/3
VMKR
Local SGD
G2/3 R273
+ 2//
2/3 + KR VMRR
Ca (M&)* + M
[37]
SCAFFOLD
MK
[21]
VMKR
SLowcal-SGD
(This paper)
,//24,G/// a
K
+ KR
+ R175
RE VMKR
G. M// + MK1/2
Lower Bound: Local-SGD
G2/3
2//3
G GMMMK) M
[39]
R=73
[VKR)278
KR VMKR
Similarly to the homogeneous case, it was shown in [37, 13] that Local-SGD is not superior to Minibatch-SGD in heterogeneous scenarios. Nevertheless, several local approaches that compare with the Minibatch baseline were designed in [21, 14]. Unfortunately, we have so far been missing a local method that provably benefits over the Minibatch baseline in the heterogeneous SCO setting.
Our work focuses on the latter heterogeneous SCO setting, and provide a new Local- the minibatch baseline. Our algorithm named SLow
algorithm that provably benefits builds on customizing a recent technique for incorporating a Howly-changinn sequence of query points [9, 22], which in turn enables better mitigate the bias induced by the local updates. Curiously, we also found importance weighting to be crucial in order to surpass the minibatch baseline.
In Table we compare our results to the tote--of-the-art methods for the heterogeneous SCO setting. We denote M to be the number of machines, K the number of local updates per round, and the number of communications rounds. Additionally, G (or G.) measures the dissimilarity between machines. Our table shows that Local-SCa requires much more communication rounds compared to Minibatch-SCDD and that the dissimilarity G (or G*) substantially degrades its performance. Conversely, one can see that even if the dissimilarity measure is G. = Ó(1), our approach sLowcal- SGD still requires less communication rounds compared to Minibatch-SGD.
Similarly to the homogeneous case, accelerated Sosptted-MMniibbe www.iitttniitiiiaaaaa [10, 25], obtains the best perfor- mance among all current methods, and it is still open to understand whether one outperform this accelerated minibatch baseline. In App. we elaborate on the computations of Rmin in Table 1.
Related Work. We focus here on centralized learning problems, where we aim to employ M machines in order minimize a joint learning objective. We allow the machines to synchronize during R communication rounds through a central machine called the Parameter Server (PS); and allow each machine draw samples and perform local gradient computations in every such communication round. We assume that each machine i may draw i.i.d. samples from a distribution D. which may vary between machines.
The most natural approach in this context is Mi n ch- and its accelerate variant [10], which have been widely adopted both in academy and in industry, see e.g. [16, 32, 38]. Local update methods like Local-SGD [28], have recently gained much popularity due to the rise of FL, and have been extensively explored in the past years.
Focusing on the SCO setting, it is well known that the standard Local-san is not superior (actually in most regimes it is inferior) to Winibatch-Sae [36, 37, 13]. Nevertheless, [39] devised novel accelerated local approach that provably surpasses the Minibatch baseline in the homogeneous case.
The heterogeneous SCO case has also been extensively investigated, with several original and elegant approaches [24, 23, 21, 37, 30, 14, 29, 31]. Nevertheless, so far we have been missing a local approach that provably benefits over Minibat chs.c Note that [30, 29] improve the communication

---

complexity with respect to the condition number of the objective; However their performance does not improve as we increase the number of machines M 2, which is inferior the minibatch baseline.
The heterogeneous non-convex setting was also extensively explored [21, 20, 14]; and the recent work of [31] has developed novel algorithm that provably benefits over the minibatch baseline in this case. The latter work also provides a lower bound which demonstrates that their upper bound is almost tight. Finally, for the special case of quadratic loss functions, was shown in [36] and in [21] that it is possible to surpass the minibatch baseline.
It is important to note that excluding the special case of quadratic losses, there does not exist local update algorithm that provably benefits over applies to both homogeneous and heterogeneous SCO problems.
[10]. And the latter
Our local update algorithm utilizes recent technique of employing slowly changing query points in SCO problems [9]. The latter has shown to be useful in designing universal accelerated methods [22, 12, 5], as well in improving asynchronous training methods [6].
2 Setting: Parallel Stochastic Optimization
We consider Parallel stochastic optimization problems where the objective f ; Rd + IR is convex and is of the following form,
1 M
1 M
f(x)
Σ f(x) (E[M]
Σ ((x=)) VE
Thus, the objective is an average of M functions {f+ Rd r REMM) and each such f.(+) can be written as an expectation over losses Ji(,,²") where the 2² are drawn from some distribution D; which is unknown to the learner. For ease of notation, in what follows we will not explicitly denote Eg but rather use to denote the expectation w.r.t. all randomization.
We assume that there exist M machines (computation units), and that each machine may independently draw samples from the distribution D;, and can therefore compute unbiased gradient estimates the gradients of f-(.). Most commonly, we allow the machines to synchronize during R communication rounds through central machine called the Parameter Server (PS); and allow each machine to perform K local computations in every such communication round.
We consider first order optimization methods that iteratively employ samples and generate sequence of query points and eventually output a solution Toutput- Our performance measure is the expected excess loss, ExcessLoss (fouupu) )] f(w") where the expectation is w.r.t. the randomization of the samples, and w* is global minimum of f(-) in Rdd i.e., w" E arg min f(x).
More concretely, at every computation step, each machine i E [M] may draw fresh sample 2² ~ D.. and compute a gradient estimate g at given point E RR as follows, == f'((x,z)). and note that Elg Ø| (z)), i.c. g is an ubiased estimate of (z()..
General Parallelization Scheme. A general scheme for parallel stochastic optimization is described in Alg. 1. It can be seen that the PS communicates with the machines along It communication rounds. In every round E [R] the PS distributes an anchor point O, which is starting point for the local computations in that round. Based on θ, each machine performs K local gradient computations based on K i..dd draws from D,, and yields message Φ At the end of round r the PS aggregates the messages from all machines and updates the anchor point O, Finally, after the last round, the PPS outputs Toutput, which is computed based the anchor points (O,)
Ideally, one would hope that using M machines in parallel will enable to accelerate the learning process by a factor of M. And there exists rich line of works that have shown that this is indeed possible to some extent, depending on K, Rt and on the parallelization algorithm.
Next, we describe the two most prominent approaches to first-order Parallel Optimization,
Thii implies that such methods do not obtain a wall-clock speedup as we increase the number of machines
M.
3

---

Algorithm Parallel Stochastic Optimization Template
Input: M machines, Parameter Server PS, #Communication rounds R. #Local computations K, initial point Z0 PS Computes initial anchor point O using for r = 0, R do
Distributing anchor: PS distributes anchor O all M machines
Local Computations: Each machine i [M] performs K local gradient computations based K i..dd draws from D,, and yields message & Aggregation: PS aggregates end for
from all machines, and computes new anchor 0,+1
output: PS computes Foutput based on {
(i) Minibatch SGD: In terms of Alg. 1, one can describe Minibatch-SGD as an algorithm in which the PS sends weight vector I, Rd in every round as the anchor point θ, Based on that anchor O, == each machine i computes an unbiased gradient estimate based on K independent samples from D,, i.e. g. k Σx= fi <kr++)) and communicates 9f. the message Φ the PS. The latter aggregates the messages { == ge- and compute the next anchor point Zy+11
1 M
Σ g. BE[M]
Iy+1 2, 7
where 7) > is the learning rate of the algorithm. The benefit in this approach is that all machines always compute gradient estimates at the same anchor points }r, which highly simplifies its analysis. On the downside, in this approach the number of gradient updates R is smaller compared to the number of stochastic gradient computations made by each machine which K.R. This gives the hope that there is room to improve upon Minibatch SGD, by mending this issue.
(ii) Local SGD: In terms of Alg. 1, one can deseribe Local-SGD as an algorithm in which the PS sends weight vector ZrK E TR in every round E [R] as the anchor information θ,- Based on the anchor θ, == KyK, each machine performs sequence of local gradient updates based on K independent samples from D, as follows, vk E [K],
F+K+ TyK+ Ti V. 2k
where for all machines i [M] we initialize 5 = KrK = O,, and > is the learning rate of the algorithm. At the end of round r each machine communicates ofr+ 1)K as the message & the PS and the latter computes the next anchor as follows,
1 M.
Σ 4( SE[M]
In local SGD the number of gradient steps is equal to the number of stochastic gradient computations made by each machine which is K R. The latter suggests that such an approach may potentially surpass Minibatch SGD. Nevertheless, this potential benefit is hindered by the bias that is introduced between different machines during the local updates. And indeed, we show in Table 1, this approach is inferior to Minibatch SGD in the prevalent case where = 0(1).
Assumptions. We assume that f() is convex, and that the f+(.) are smooth i.e. EL. 0uchh,
[ (v)l s L| - yll Vi e [M], Vx, E Rd
We also assume that variance of the gradient estimates is bounded, i.e. that there exists such,
f1; - ff(()) < o² Vz Rd Vi [M] Letting w* be global minimum of f(.), we assume there exist G. 2 0 such that,
1
Σ [V < G./2, (G. (G.-Dissiillarit) iE[M
(1)
M

---

The above assumption together with the smoothness and convexity imply (see App. B),
Σ
[V((2)) & C. ((ww))) Vo E IRd
(2)
M
iE|M]
A stronger dissimilarity assumption that is often used in the literature is the following.
1
Σ
V G²/2, Yo E Bd (G-Dissimilarity)
(3)
M
SE[M]
Notation: For feete we denote yt,:t = Di=e, Vr. For N g we denote [N] .O....,N. 13.
3 Our Approach
Section 3.1 describes a basic (single machine) algorithmic template called Anytime-GD GDd. Section 3.2 describes our SLowcal-So algorithm, which is Local-SGD style algorithm in the spirit of Anytime GD. We describe our method in Alg. 2, and state its guarantees Thm.
3.1 Anytime GD
The standard GD algorithm computes sequence of iterates {wo fre and queries the gradients theses iterates. was recently shown that one can design a GD-style scheme that computes sequence of iterates {wr (Weee yet queries the gradients at different sequence (x+ hejz which may be slowly-changing. in the sense that || 2, || may be considerably smaller than || - well-
Concretely, the Anytime-GD algorithm [9, 22] that we describe in Equations (4) and (5), employs a learning rate 17 > 0 and a sequence of non-negative weights {az): The algorithm maintains two sequences {we he, h that updated follows Vt,
Wt+1 = Wt 1jokSt Vt E [T], where gr- = Vf(())
and then,
Cndt +
0+++1
Tt+1 00:4+
Wr++
(5)
000:2++
0o:++ - 0₁ Wy i.e. the H, are weighted averages of the wy we's. Thus, every iterate the gradient Sr is queried at xa which weighted average of past iterates, and then #++1 is updated similarly to GD with weight on the gradient Gt- Moreover, at initialization xog = wp-
be shown that the above implies that T441
Curiously, it was shown in [9] that Anytime-GD obtains the same convergence rates as GD for convex loss functions (both smooth and non-smooth). It was further shown and that one can employ stochastic version (Anytime-SGD) where we query noisy gradients at instead of the exact ones, and that approach performs similarly to SGD.
Slowly changing query points. A recent work [6], demonstrates that if we use projected Anytime- given bounded convex domain; then one can immediately
SGD, i.e project the w, sequence show that for both a₃ = 1 and = ( + 1 we obtain || - xz / < D//1. where D) is the diameter of the convex domain. Conversely, for standard SGD we have ||0e+1 w S 171lge Il. where gi here is a (possibly noisy) unbiased estimate of V f (wx)). Thus, while the change between consecutive SGD queries is controlled by 1 which is usually 1/Vt, and by magnitude of stochastic gradients; for Anytime-SGD the change decays with time, irrespective of the learning rate 77. In [6], this used to design better and more robust asynchronous training methods.
Relation to Momentum. In the appendix we show that Anytime-SGD can explicitly written as a momentum method, and therefore quite different from standard SGD. Concretely, for a 1 we show that 1++1 R3 Iz -2-1) (T=t(t Where 9r here a (possibly noisy) unbiased estimate of fx(x)). This momentum interpretation provides complementary intuition regarding the benefit of Anytime-SGD in the context of local update methods. Momentum brings more stability to the optimization process which in turn reduces the bias between different machines.
and for a t we show that 2441 2, 11 (T=((/))
For the sake of this paper we will require specific theorem that does not necessarily regard Anytime- GD, but is rather more general. We will require the following definition,
5

---

Algorithm 2 LLoweal-SGD
Input: M machines, Parameter Server PS, #Communication rounds R. #Loccal computations K, initial point zo, learning rate 71 > weights {ar)t Initialize: set Wo 20, initialize anchor point Oo == (wo, zo), and set = 0 for = ....,,
do
Distributing anchor: PS distributes anchor 0r == (we, zz) to all machines, each machine E [M] initializes (w), =/) = θ (W,, Ft)
for = 0, ....., K
do
Set = rk + k
Every machine i E [M] draws fresh sample 2 ~ Dj. and computes gi = fi 2)
Update w = w naugh. and r2+1 = = 0ce11 )xr ottl w
cou+1
on..+1
end for
Aggregation: PS aggregates {(wt+1 xt4 HE(M from all machines, and computes new
anchor 0r+1 == (Wr+1,T(+1) 1/4 DiEMMI wi+1 w EsEIM ai+1)
end for output: PS outputs dr (recall T = RR)
Definition Let {or 2 0) be a sequence of non-negative weights, and let {we E (R)), be an arbitrary sequence. We say that a sequence (x, &R² is an {atht weighted average of (wehe if To Wb, and for any > 0 Eq. (5) is satisfied.
Next, we state the main theorem for this section, which applies for any sequence {w, E RR
Theorem (Rephrased from Theorem in 19). Let f IRd + R be a convex function with a global minimum w" Also let fas 2 O) and {wr E Rd), {aze R such that {rehe h is fas l weighted average of [we]t. Then the following holds for any 2 O.
S aoot (f(x,) f(w")) S Σ ar vf(x)) (w, w²)
7=0
3.2 SLowcal-SGD
Our approach is to employ an Anytime version of Local-SGD, which name by SLowcal-SGD. Notation: Prior to describing our algorithm we will define t to be the total of per- ner-machine local updates up to step & of round r, resulting denote the iterates and samples using t, rather than explicitly denoting t= rK + k. Additionally we use {ar]r to denote pre-defined sequence of non-negative weights. Finally, we denote 7 = RK.
rK + k. In what follows,
will often find it useful to
In the spirit of Anytime-SGDD our approach to maintain two sequences per machine E [M]] {wj Rd and {aj E {.). Our approach is depicted explicitly in Alg. 2. Next we describe our algorithm in terms of the scheme depicted in Alg. 1:
(i) Distributing anchor. At the beginning of round the PS distributes 0, (w, m:) (w,K, (yK Rdd Idd to all machines.
(ii) Local Computations. For t = rK, every machine initializes (w), xt) = θ,, and for the next K rounds, i.c. for any rK < t < (r + 1)K every machine performs sequence of local Anytime-SGD steps as follows,
w w| ya
(6)
where similarly Anytime-SDD query the gradients at the averages sf, meaning of V ff(a), 2t) And query points are updated as weighted averages of past iterates [weh...
2
a++1
)art
0++11
VrK <t (r
(7)
00:+1
At the end round r, i.e. 1 (r+ 1)K, each machine communicates (wi, 2²t) message the PS. (iii) Aggregation. The PS aggregates the messages and computes the next anchor point 0 =
(w, xx) EiE[M] $ = M E wi, M Eie(an æi). where ( = (r 1)K.

---

Remark: Note that for = rK our notation for x/) is inconsistent: at the end of round r 1 these values may vary between different machines, while the beginning of round r these values are all equal to θ == (Wy, 22). Nevertheless, for simplicity we will abuse notation, and explicitly state the right definition when needed. Importantly, in most of our analysis we will mainly need to refer to the averages (+ EiE[M] wi, M Eie(an m1) and note the latter consistent the end
and beginning of consecutive rounds due to the definition of θ, and -
3.2.1 Guarantees & Intuition
Below we state our main result for SLowcal-SGD (Alg. 2),
Theorem 2. Let f() be convex and T--smoohh function.. Then under the assumption that we make in Sec. 2, invoking Alg. 2 with weights {a== = +t+ 1htm and an appropriate learning rate 1) ensures,
aBo +
( + - B
EAT S 0 LBB
n22 +
KR
K1/3
VMKR
KV/R
where AT := ((TT) f(x),, Bo = Iww w and we choose the learning rate follows,
U1100 - w= IIM
1ww0 w 11
7 min
(8)
18T((T 1)" 10LK 4T + 1)2/3
0/3/2
LV2 R G!/²)
As Table shows, Thm. 2 implies that SLowcal-SGD improves over all existing upper bounds for Minibatch and Local SGD, by allowing less communication rounds to obtain a linear speedup of M. Intuition. The degradation in local SGD schemes (both standard and Anytime) is due to the bias that it introduces between different machines during each round, which leads to a bias in their gradients. Intuitively, this bias is small if the machines query the gradients at a sequence of slowly changing query points. This is exactly the benefit of SLowcal-SGD which queries the gradients at averaged iterates z;'s, Intuitively these averages are slowly changing compared to the iterates themselves wt; and recall that the latter are the query points used by standard Local-SGD. complementary intuition to the benefit of our approach, is the interpretation of Anytime-SGD as momentum method (see Sec. 3.1 and the appendix) which leads to decreased bias between machines.
To further simplify the more technical discussion here, we will assume the homogeneous case, i.c., that for any E [M] we have D; = DD and f((-) f()).
So a bit more formally, let us discuss the bias between query points in a given round r E [R], and let us denote to rK. The following holds for standard Local SGD,
t-1
w
Wto 77
Σ of, Vi [M], t E [to: to + K] r=
(9)
where 9 is the noisy gradients that Machine computes in wi and we can write of = f(( + E
where E is the noisy component of the gradient. Thus, for two machines i + j we can write,
+-1
El El|w: - - == 1
Σ oイ 9
Σ rm
V V
Σ -
to
And it was shown in [36), that the noisy term is dominant and therefore we can bound,
E Ellw mi
E
Σ
ed
R tt-tt to K.
(10)
Similarly, for SLowcal-SGD we would like to bound E| 2E|l² for two machines i j: and in order to simplify the discussion
will assume uniform weights i.c., ar 1. Vt E [T]. Now the
update rule for the iterates wf. is of the same form as in Eq. (9), only now gf == vf(x + c where & is the noisy component of the gradient. Consequently.
t-1 () rt) = -nk
Σ (wi w!) R 7 Σ (t r=to
> (of 9.)
t=to
r=to
7

---

where we took crude approximation of t T ~ K.
Now, by definition of 2 and ds = 1, x t Ita Di wi vi E [M] E [to, to + K] Thus, for two machines + j we have,
1 Ella
E
Σ
2
1
n²K²
17
E
Σ
t-1
97
g
112
12
T=to
T=to
k E
K2
V
-
E
ta
t²
r=to
As we show in our analysis, the noisy term is dominant, so we can therefore bound,
K2 E
K2
t2 1 K3
272 Ellat 2il
(11)
Taking t =e T RK above yields a bound of O(K/R²). Thus Equations (10), (11), illustrate that the bias of sLowcal-SGD is smaller by factor of R compared to the bias of standard Local-SGD. In the appendix we demonstrate the same benefit of Anytime-SGD over SGD when both use 04 x t.
Finally, note that the biases introduced by the local updates come into play in slightly different manner in Local-SGD compared to SLowcal-SGD 3, Consequently, the above discussion does not enable to demonstrate the exact rates that we derive. Nevertheless, it provides some intuition regarding the benefit of our approach. The full and exact derivations appear the appendix.
Importance Weights. One may wonder whether it is necessary to employ increasing weights a = + + 1, rather than employing standard uniform weights a₁ = 1, Vt. Surprisingly. in our analysis we have found that increasing weights are crucial in order to obtain benefit over Minibatch-SGD and that upon using uniform weights SLowcal-SGD performs worse compared to Minibatch SGD! We elaborate this in Appendix L.. Below we provide intuitive explanation.
Intuitive Explanation. The intuition behind the importance of using increasing weights is the following: Increasing weights are technical tool put more emphasis the last rounds. Now, in the context of Local update methods, the iterates of the last rounds are more attractive since the bias between different machines shrinks as we progress. Intuitively, this happens since we progress with the optimization process, the expected value of the gradients that we compute goes to zero (since we converge); and consequently the bias between different machines shrinks as we progress.
3.3 Proof Sketch for Theorem 2
Proof Sketch Theorem 2. As a starting point for the analysis, for every iteration t E [T] define the averages of (wi, x/i of) across all machines as follows,
we will
Σ wi & x4 *E[M]
1 M
Σ x KE[M]
1 M
Σ ge
Wr == M
9t
xE[M]
Note that Alg. 2 explicitly computes (w,, 27) only once every K local updates, and that theses identical to the local copies of every machine at the beginning of every round. Combining the above definitions with Eq. (6) yields,
W++1 = we - nangt. Vt E [T]
(12)
Further combining these definitions with Eq. (7) yields,
2+41 = (1 0++1 )a, + a++1 Wi+1, Vt E [T]
(13)
00:4+1
a0:1+1
The above implies that the (#: sequence is an {arheery weighted average of (weerr. This enables to employ Thm. which yields, A < E'_ ar V((,) (w, w²), where we denote A, := fxx) - ff(ww, This bound highlights the challenge in the analysis: our algorithm does not directly compute unbiased estimates of zt, except for the first iterate of each round. Concretely,
Eq. (12) implies that our algorithm effectively updates using It which a biased estimate of V (xx).
'A major challenge in our analysis is that for given i E [M] the {ai} sequence not necessarily an {at) weighted average of the {wi].
8

---

It is therefore natural to decompose vf(x,) = 9r + (Vf(=++ 9r) in the above bound, yielding,
ar
(w, w²) V [V),)) 9r) (we w²)
(14)
r=0
(A)
(B)
Thus, we intend to bound the weighted error Co by bounding two terms: (A) which related to
the update rule of the algorithm, and (B) which accounts for the bias between 9r and V fzz).
Notation: what follows we will find the following notation useful, 9: == 37 ΣiEMa V V(((z)) and
note that gt = E. [9el [=c][=])) We will also employ the following notations: V, Er a g
fx(x)) Il and D. = W/ wt 11 where w" a global minimum of f(-). We will also denote
Doa = Er=o ||,,, - =²112
Bounding (A): Due to the update rule of Eq. (12), one can show by standard regret analysis that: (A) = Erzo 0rgr (w, w*)=
||a0o-""
+ Er= az 1|
2n
Bounding (B): We bound (B) in expectation using V2 and Da:: follows for any p
[ I 2p EV ED
Combining (A) and (B): Combining the above boounds on (A) and (B) into Eq. (14) we obtain the following bound which holds for any > and t E [T).
T a
Ckg EA,
E
11l 11? + 2p
EVT + L LEDor
(15)
2m
-=
Now, to simplify the proof sketch we shall assume that D, < Do Vt, implying that Dor S TDo _ gives,
Plugging this into the above equation and taking p 1,
| w
og EA,
1
a flg 112 4nTEV..
(16)
"
Next we will bound (*) and EVr, and plug them back into Eq. (16).
Bounding (+): To bound (x) it is natural to decompose 9r (gr g+)+ (gr + Using this decomposition we show that, (*) z 3 Σ
1= a + 3EVT + 12LE E'= O.:
Bounding E The definition of Vi shows that is encompasses the bias that is introduced due to the local updates, which in turn relates to the distances |>: x4 ll Vi, [M]. Thus, EVr therefore directly related to the dissimilarity between the machines. Our analysis shows the following: EVT 40 K3 EF= ager (G., +4LA, + K Roo Plugging the above into Eq. (16),
and using our choice for 17, gives an almost explicit bound.
EA,
1100
w'11
+ + L3 Σ an. G. + 2(T + 1)
Σ t=
T
T
0 ao.
M
t=0
The theorem follows by plugging above the choices of ar and using a technical lemma.
Experiments
To assess the effectiveness of our proposed approach, we conducted experiments on the MNIST [26] dataset-a well-established benchmark in image classification comprising 70,000 grayscale images of handwritten digits (0-9), with 60,000 images designated for training and 10,000 for testing. The datasct was accessed via torchvision (version 0. 0.12.. We implemented logistic regression

---

model [7] using the PyTorch framework and executed all computations on an NVIDIA L40S GPU. To ensure robustness, results were averaged over three different random seeds. The complete codebase for these experiments is publicly available on our GitHub repository."
We evaluated our approach using parameters derived from our theoretical framework (a, t) in com- parison to Local-SGD and Minibatch-SGD under various configurations. Specifically, experiments were conducted with 16, 32, and 64 workers to examine the scalability and robustness of the proposed method. We also varied the number of local updates K (or minibatch sizes for Minibatch-SGD) among 4, 8. 16, 32, and 64 to investigate how different local iteration counts impact performance. Data subsets for each worker were generated using a Dirichlet distribution [17] with a = 0.1 to simulate real-world non-IID data scenarios characterized by high heterogeneity. For fairness, the learning rate was selected through grid search, with a value of 0.01 for SLowcal-SGD and
and 0.1 for Minibatch-SCC SGD. More details about the data distribution across workers and complete experimental results are provided in Appendix M.
(a) Test Accuracy (f better).
(b) Test Loss (1 is better).
Figure 1: Performance Local Iterations (K) for different numbers of workers (M).
Our results on the MNIST dataset, presented in Figure and detailed in Appendix M.2, demonstrate the effectiveness of our approach, showing consistent performance improvements compared to Local-SGD and Minibatch-SGD the number of local steps increases. Notably, this improvement becomes even more significant compared to the other methods as the number of workers increases, underscoring the scalability of our method and aligning with the theoretical guarantees outlined in framework. These results highlight the robustness of our approach in handling highly heterogeneous, distributed environments.
Upon closer inspection, when a small number of local steps are performed, the differences between the approaches are negligible, with a slight advantage for Minibatch-SGD. However, as the number of local steps increases, the minibatch size grows, and the need for significant variance reduction diminishes. In this regime, making more frequent optimization updates becomes more impactful, as demonstrated by the superior performance of the local approaches compared to Minibatch- SGD. Importantly, with SLoweal-SGD, which keeps local updates closely aligned among workers throughout the training process, we can achieve significantly better and more stable performance compared to both Minibatch-SGD and Local-SGD the number of local steps K and the number of workers M increase.
5 Conclusion
We have presented the first local approach for the heterogeneous distributed Stochastic Convex Optimization (SCO) setting that provably benefits over the two most prominent baselines, namely Sinibatch-SGDD and Local-SGD. There are several interesting avenues for future exploration: (a) developing an adaptive variant that does not require the knowledge of the problem parameters like and L: (b) Allowing a per dimension step-size that could benefit in (the prevalent) scenarios where the scale of the gradients considerably changes between different dimensions; in the spirit of the well known AdaGrad method [11]. Finally, (c) it will be interesting to understand whether we can find an algorithm that provably dominates over the Accelerated Minibach-SCI baseline, which is an open question also in the homogencous SCO setting.
'nttps: : h/t::///tttt com/dahani 98/ slowcal-sgd
10

---

Acknowledgement
This research was partially supported by Israel PBC-VATAT, the Technion Artificial Intelligent Hub (Tech.Al), and the Israel Science Foundation (grant No. 3109/24).
References
[1] Apple. designing for privacy (video and slide deck). apple wwdc, 2019.
[2] Intel and consilient. intel and consilient join forces to fight financial fraud with ai, 2020.
[3] Melloddy. melloddy project meets its year one objective: Deployment of the world's first secure platform for multi-task federated learning in drug discovery among 10 pharmaceutical companies, 2020.
[4] Google. your voice and audio data stays private while google assistant improves, 2021.
[5] Kimon Antonakopoulos, Dong Quan Vu, Volkan Cevher, Kfir Levy, and Panayotis Mertikopou- los. Undergrad: universal black-box optimization method with almost dimension-free convergence rate guarantees. In International Conference on Machine Learning, pages 772-795. PMIR, 2022.
[6] Rotem Zamir Aviv, Ido Hakimi, Assaf Schuster, and Kfir Yehuda Levy. Asynchronous dis- tributed learning: Adapting to gradient delays without prior knowledge. In International Conference on Machine Learning, pages 436 445. PMLR, 2021.
[7] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[8] Ashok Cutkosky. Lecture notes for ec525: Optimization for machine learning.
[9] Ashok Cutkosky. Anytime online-to-batch, optimism and acceleration. In International conference on machine learning, pages 1446-1454. PMLR, 2019.
[10] Ofer Dekel, Ran Bacchaahc Ohad Shamir, and Lin Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research, 13(1), 2012.
[11] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[12] Alina Ene, Huy I. Nguyen, and Adrian Vladu. Adaptive gradient methods for constrained convex optimization and variational incqualities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7314-7321, 2021.
[13] Margalit R Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local sgd) and continuous perspective. In International Conference on Artificial Intelligence and Statistics, pages 9050-9090. PMLR, 2022.
[14] Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. Local sgd: Unified theory and new efficient methods. In International Conference on Artificial Intellignnce and Statistics, pages 3556-3564. PMLR, 2021.
[15] Elad Hazan et al. Introduction online convex optimization. Foundations and Trends@ in Optimization, 2(3-4):157-333 2016.
[16] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
[17] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint (0ti::/9090063333 2019.
[18] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.

---

[19] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. et al. Advances and open problems in federated learning, Foundations and Trends in Machine Learning, 14(1-2):1-210, 2021.
[20] Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008. 2020.
[21] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 1132-514.. PMLR, 2020.
[22] Ali Kavis, Kfir Y Levy, Francis Bach, and Volkan Cevher. Unixgrad: A universal, adaptive algorithm with optimal guarantees for constrained optimization. Advances in neural information processing systems, 32, 2019.
[23] Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 4519-4529, PMLR, 2020.
[24] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pages 5381-5393. PMLR, 2020.
[25] Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133(1-2):365-337,222 2012.
[26] Yann LeCun, Corinna Cortes, Chris Burges, et al. Mnist handwritten digit database, 2010. URL http: / /yann, lecun. com/ excb/ htt///miistttttt/////mmmmmm/tttt Licensed https: http::///eeeaatieeccoooeeee
Licenesee under CC BY-SA 3.0, available at C/.
[27] Olvi L Mangasarian and Mikhail V Solodov. Backpropagation convergence via deterministic nonmonotone perturbed minimization. Advances in Neural Information Processing Systems, 1993.
[28] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282, PMLR, 2017.
[29] Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning, pages 15750-15769. PMLR, 2022.
[30] Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. Advances in Neural Information Processing Systems,
2021.
[31] Kumar Kshitij Patel, Lingxiao Wang, Blake Woodworth, Brian Bullins, and Nathan Srebro. To- wards optimal communication complexity in distributed non-convex optimization. In Advances in Neural Information Processing Systems.
[32] Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv: 1.00499, 2017.
[33] Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019.
[34] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Agüera y Areas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas N. Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horváth, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konečný,
12

---

Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtárik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake E. Woodworth, Shanshan Wu, Felix Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide to federated optimization. CoRR,
2021.
[35] Jun-Kun Wang, Jacob Abemethy, and Kfir Y Levy. No-regret dynamics in the fenchel game: unified framework for algorithmic convex optimization. arXiv e-prints, pages arXiv-2111,
2021.
[36] Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference Machine Learning, pages 10334-10343. PMLR, 2020.
[37] Blake Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heteroge- neous distributed learning. Advances in Neural Information Processing Systems, 13:6281-6292, 2020.
[38] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert 76 minutes. arXiv preprint arXiv: 009622 2019.
[39] Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems, 33:5332-553 2020.
13

---

A Explanations Regarding the Linear Speedup and Table
Here we elaborate on the computations done Table 1. First we will explain why the dominance of the term VMKR implies linear speedup by factor of M.
Recall that using SGD with a single machine M = 1, yields convergnece rate of VKR (as dominant term). Thus, in order to obtain an excess loss smaller than some 0, SGD requires RK 2 Ω (+:). Where RK is the wall-clock time required to compute the solution.
Explanation.
Now, when we use parallel optimization with R communication rounds, local computations, and M machines, the wall-clock time to compute a solution is still RK. Now, if the dominant term in the then the wall clock time to obtain an e-optimal solution
convergence rate of this algorithm VMKR
should be RK Ω Mez) ). And the latter is smaller by factor of M compared to single machine.
Computation of Amin in Table 1. The term VMKR appears in the bounds of all of the parallel optimization methods that describe. Nevertheless, it is dominant up as long the number of communication rounds is larger than some treshold value Rmin= that depends on the specific convergence rate. Clearly, smaller values of Rmin imply less communication. Thus, in the Rmin column of the table we compute Rmin for each method based on the term in the bound that compares least favourably against VMKR*
These terms bolded in the Rate column of the table.
Concretely, denoting this less favourable term by Bearallel Brarallel (M, K, R, G.) $, then Rmin the lowest R which satisfies.
gearallel
MKR
B On Heterogeneity Assumption
Let us assume that the following holds at the optimum w",
Σ
=
M
VE[M]
Then show the following relation for any w E Rdd
1
Σ |Vf(w))
Σ [V((w) Vf((w)) + fi(w")111
M ME|M]
M
*E[M]
$ M
(Vf(ww) Vf((w))1
2
Σ [V')("))||
M SE[M]
kE[M]
$ (f f(w")( + G
where we used ||a + 21l2 < 2||a|2 2116112 which holds for any a, E Rd, and the last line follows by the lemma below that we borrow from [18, 8].
Lemma 1. Let [(x) = M EvEta (x) be convex function with global minimum w', and assume that every Ja TRd R T-smooth. Then the following holds,
Σ [V(((w) vel(w")) <2L([()) {E[M]
M
Proof of Lemma 1. The lemma follows immediately from lemma 27.1 in [8], by taking w² therein.
ggarallll may also depend on a, L, ||wa - will but for simplicity of exposition we hide these dependencies in Table 1.
14

---

C Interpreting Anytime-SGD as Momentum
Here we show how interpret the Anytime-SGD algorithm that present in Equations (4).(5, as momentum method. For completeness we rewrite the update equations below,
Wt+1 = Wt 110rgh E [T] where gr. = (x+)
(17)
and then,
Ol0st
T++1
Ft. +
10++1
(18)
00:++1
000:++1
where g: is an unbiased gradient estimate Ir, and far sequence non-negative scalars, And at initialization Z0 Wt.
First note that Eq. (18) directly implies that,
I++1
ary Wr
00.41
Next, note that we can directly write,
W/ Wg
OnIn
n=0
Plugging the above into the formula for F441 yields,
++
I++1 = WD - 00::+11
ananIn
n=0
w0 =
Or an
01/0011
==0+=n+
W/o
Ontl2410mgh
0tp:++1 n=0
Thus,
1-1
1
(I1+1
x²)==
Ont1nOnhn
On On+12+10n.h
17
oud
000+11 ==0
n=0
00:++1
ant1:an9
ax+11410p9
a++1 049
010:1++1
00:-11
n=0
m=0
1
000+11
Oa+lit+1
an+ lst
On
14:9
00:4+1
00:t
000:011
n
1
a00n
1
a++1
anIn
a4410ttt
00:411
00.+11
n=0
a0:
0411 On
In
(19)
00:111111
aig.t
where we used the equality below,
000++1
0++1
On+1it
0lg.n
an+1:411
Ou+i:t = 04+1 - Ontld
a441
04+1 Ou:
Qgat
Cud
Thus we can write.
E1+1 RJ Ct - -1 00:+1
Oc+ Onn
Jn
Orga
n=0
15

---

Uniform Weights. Thus, taking uniform weights a; = 1 yields,
7
I++1 RJ Ft
1
t In
==
Linear Weights, Similarly, taking linear weights a₁ = + yields,
n
1141 24
t In
D Proof of Theorem 1
Proof of Theorem I. We rehearse the proof of Theorem from [9].
First, since wt is global minimum and 0o: are non-negative than clearly,
Costt (f(z)) - f(w")) 2 0.
Now, notice that the following holds,
ar(2; - wx) = 0i0e- 1 xze)
Using the gradient inequality for J gives.
--([(==))(ww)) Σ 0r Vf(x)) (x, (w)
Σ ar ((()) (w, -w")= (0=)+
Σ
ar ([(x)) (5, wi))
==0
Σ a, [x+)) (w,- -w")+ +
Σ
a a0:r-1
Vf((z)) (x7-1 x+)
ar ((++) (w--w") w²) + Σ 00:r- 1 ( - (1 ((+t))
where we have used the gradient inequality again which implies (f(x)) (I--1 - xt) S (xx (x+)).
Now Re-ordering obtain,
Σ (au=f((z)) - ao.r- ff(-11(()-1)) an fww)(w") S
Σ ar ff(z)) (w, - -w")
7=0
Telescoping the sum in the LHS we conclude the proof,
Obg -f(x+)
- f(w")) < a, (f(x)))((0)) - -w))
E More Intuition and Discussion Regarding the Benefit of SLowcal-SGD
More Elaborate Intuitive Explanation. The intuition is the following: We have two extreme baselines: (1) Minibatch-SGD where queries do not change at all during updates-implying that there is no bias between different machines. However, Minibatch-SGDD is "lazy" since among K R queries it only performs gradient updates. Conversely (2) Local-SGD is not "lazy" since each machine performs K R gradient updates. Nevertheless, the queries of different machines change substantially during each round, which translates to bias between machines, which in turn degrades the convergence.
16

---

Ideally, we would like to have a "non-lazy" method where each machine performs KR gradient updates (like Local-SGD), but where the queries of each machine do not change at all during rounds (like Minibatch-SGD) and therefore no bias is introduced between machines. Of course, this is too good to exist, but our method is a step in this direction: it is "non-lazy" and the query points of different machines change slowly, and therefore introduce less bias between machines. This translates to better convergence rate.
Additional Technical Intuition for 04 X t. Here we extend the technical explanation that we provide in Sec. 3.2.1 to the case where dt o I, and show again that SLowcal-SGi) yields smaller bias between different machines compared to Local-SGI..
As in the intuition for the case of uniform weights, simplify the more technical discussion, we will assume the homogeneous case, i.e., that for any i E [M]] we have D, = DD and f((-) = f(-).
Note that upon employing linear weights, the normalization factor 0g:r that plays major role in the convergence guarantees of Anytime-SGD (see Thm. 1) also grows as 0g:T x T Thus, in order to make proper comparison, appropriate weighted version of SGD; where the normalization factor Qu:T also plays a similar role in the guarantees (see e.g. [35]). This weighted SGD is follows [35], AA 2 0
should compare the bias of weighted Anytime-SGD. to the
Wt++ = 00 1149tt where is unbiased of V (wr).
(20)
and after iterations it outputs wr a0:T Σt=o O4tii And for 02 t + this version enjoys the same guarantees as standard SGD.
Next, we compare the Local-SOR version of the above weighted SGD (Eq. (20)) to our SLowcal- SGDwhen both employ O₁,- = L + 1. So a bit more formally, let us discuss the bias between query
points in a given round r E [R]. and let us denote 60 = rK. The following holds for weighted Local SGD,
1-1
mi Wh.
17
Σ T=10
Eagt. Vie [M].t E [to, to + K].
(21)
where 9: is the noisy gradients that Machine i computes in w and we can write gf == f(( + t where Ex is the noisy component of the gradient. Thus, for two machines i + j we can write,
Ellu; w = n²²E
Σ
arlst rt)
E
Σ a. (Vf(ww) T=to
v v((w)))
+ + E
Σ
( - (t)
r=to
Now, we can be generous with respect to weighted SGD and only take the second noisy term into account and neglect the first term 6, Thus we obtain.
--1
El w/
2
s
E
Σ r=to
a (c: - (
oZ
E
Σ r=ta
-
2 (rK)3. (t- S ST²
(22)
where we used ag S +K Vt S to + K, as well as oz o k = (r(K 1)² ~ (rK). We also used t t-to to 2 K.
Similarly, for SLowcal-SGD we would like to bound Ellzi x 112 for two machines i * j; while assuming linear weights i.e., a i =t+1, Yt [T]. Now the update rule for the iterates wt, is of the same form as in Eq. (21), only now gf == V f((;)) + Et. where & is the noisy component of the gradient. Consequently, we can show the following,
ar(w" - wx)) n
ar Σ
gh) R
Σ ant12 ( 9 =np2 K3 Elet
Cot -94)
where we took a crude approximation of an+14 R K S rk2 2 rK 2², In the last "s" we also change the notation of summation variable from to T.
'It was shown in [36], that the noisy term is dominant for standard SGD
17

---

Now, by definition, 24 = aodt. two machines # we have.
21o + 1
Σ
a.
vi E [M], E [to. to + K]
Thus, for
on.
0011
E Blaa
x 1
E
7 (w, w+)
1
1
K (ax:))
E
Σ
1-1
gr
o
17
00:t
Tip
1/244 K
t-1
97 8t
op
1 K
2
t-1
K
Vf(x)t) v5(x4)
+
K
Σ
c
&
r=to
where we have used On x 0nda x 1g ~ K² As show in our analysis, the noisy term is dominant,
therefore bound,
L-1
172 El 2 2 K E
Σ r=fo
k (0) K
(23)
Thus Equations (22), (23), illustrate that for a then the bias SLowcal-SGD smaller by factor of r² compared to the bias of weighted Local-SGID.
Since r2 can be as big as R² this coincides with the benefit of sLowcal-SGii over standard SGD in the case where
1, which we demonstrate in the main text.
Finally, note that upon dividing by the normalization factor 0o:7, have, that for SLowcal-SGD with either 0; = 1 or x t then,
K R
El
K
(24)
0g:77 7
R2K2
RK
VKR²
Comparably, upon dividing by the normalization factor cg.T, have, that for Local-SGD with either 0t = or O2 X f that,
1
Ellol
w/ll
R²K2 VIPK3
K
(25)
00:T
RK
VRR
Thus, with respect to the approximate and intuitive analysis that we make here SLowcal-SGD main- tains similar benefit over Local-SGD for both = 1 and + 1.
As explain in Appendix I., taking ar 1 in sLowcal-SGD does not actually enable provide benefit over Local-SGID. The reason is that for a; = 1, the condition for which the dominant term in the bias (between different machines) is the noisy term (this enables the approximate analysis that we make here and the body of the paper), leads to limitation on the learning rate which in turn degrades the performance for SLowcal-SGD with a Conversely, for o f there is no such degradation due to the limitation of the learning rate. For more details and intuition please see Appendix L.
F Proof of Thm. 2
we will define the Proof of Thm. 2. As starting point for the analysis, for every iteration t E [T] averages of (wi, x1, gt) across all machines as follows,
Wc = M
Σ w AE[M]
&
34 =
1 M
Σ ME[M
9t =
1
Σ di SE[M]
M
Note that Alg. 2 explicitly computes (We, x2) only once every K local updates, and that theses are identical to the local copies of every machine at the beginning of every round. Combining the above definitions with Eq. (6) yields,
Wr+1 We - morte Vt E [T]
(26)
18

---

Further combining these definitions with Eq. (7) yields,
2141 (1
0+++1 jzr + o++1 W++1 Vt [2]]
(27)
000:1+1
00:+1
And the above implies that the (aET sequence is an (as))TT)) weighted average of {wi hem This enables employ Thm. which yields,
ag A == =o) f(w")) S Σ a, Vf(x)) (w, w²),
T=0
where we denote A : f(x) - f(w"). The above bound highlights the challenge in the analysis: our algorithm does not directly computes unbiased estimates of 2t, except for the first iterate each round. Concretely, Eq. (26) demonstrates that our algorithm effectively updates using Jr which might be a biased estimate of V (xx).
is therefore natural to decompose vf(xx) = gr + (Vf(x,) - gr) the above bound, leading to,
a0: A. S
argr (w, w²) ar (Vf(x,) gr) (w. w")
(28)
(A)
(B)
Thus, we intend bound the weighted error a by bounding two terms: (A) which directly related to the update rule of the algorithm (Eq. (26)), and (B) which accounts for the bias between gr. and (rx).
Notation: In what follows we will find the following notation useful.
9e
1
Σ Vf((z))
(29)
M KE(M)
and the above definition implies that = E [a.l<=hheMM) We will also employ the following notations,
(2/-1) [eL-))E(M)] = E [gel(+t)
Vi Σ 22 -f(=)) & D Iw: w=
where w" is global minimum of f(-). Moreover, we will also use the notation Dos: Et=o ||w, w" 11
Bounding (A): Due to the update rule of Eq. (26), one can show by standard regret analysis (see Lemma 2 below) that,
(A) = Σ ardr (wr w²)
[Wo w
11
17
0
11 11
(30)
2m
2 T=0
Lemma 2. (OGD Regret Lemma -See e.g. [15]) Let wi E RRd and n > 0. Also assume a sequence of T non-negative weights {az 2 O'tem and T vectors {ge IRd herr.. and assume an update rule of the following form:
1jontt VI E [T].
Wh++
Then the following bound holds for any Rd, and E [T].
gr (w, u) < 1w0 u/l2
7 Σ
2
11
21
2
==0
for completeness we provide proof in Appendix G.
19

---

Bounding (B): Since our goal is to bound the expected excess loss, we will bound the expected
value of (B), thus,
E (B) = E
ar (Vf(x+)) 9-) (w, w²)
E
0r (r(V [ タ-) (w- w²))
T=1
E
2 a V VV(x) 3r11 - BEllu, w =0
2p
T=0
EV. + EDuut
(31)
2p
2
where the second line follows by the definition gr (see Eq. (29)) and due to the fact that W, is measurable with respect to {1 he [gr [o-l==ahe(an).....
he while 9r
{-+ [e(a)] implying that Elg, (w, w²)] E (w, w )}; the
third line uses Young's 's inequality a S onf [|a]|l + z l|bll which holds for any a, E Rd; and the last two lines use the definition of V and Do:r-
Combining (A) and (B): Combining Equations (30) and (31) into Eq. (28) we obtain the following bound which holds for any and / E [T].
[w w0 1
T
0g. EA,
E
Σ
a
|
112 + _
EVr + L EDDo.
(32)
2n
2p
7=0
where we have used V₁ < VT which holds for any E [T]. as well as E E of 11 which holds since S T.
S
Next, we shall bound each of the above terms. The following lemma bounds E.
Lemma 3. The following bound holds for any & E (7).
EDo.: E Σ |0, w" 112 $ |T - w01l" + 2TT 2 Σ a 11. 12 16 EVr
Combining the above lemma into Eq. (33) gives,
100
"
113
+ 25 + nTT²
T E. M
(1 2p EV + PT] - -w"l w=11 2p
(oo:EA. S
27
Since the above holds for any p > 0 let us pick specific value of p = Th/ by doing so we obtain,
Coaa EA, < | w1
T
E
a 11 2 4nTEV.
(33)
7=0
(C)
Next, we would like to bound (C); to do so it is natural to decompose 9r = (gr - gr) + (g- Vf(x+)) + (z(z)) The next lemma provides bound, and its proof goes directly through this decomposition,
Lemma 4. The following holds,
2 T S
(C)
a + 3EV + 12LE
a..
M
t=
1=
20

---

Combining the above Lemma into Eq. (33) yields,
a0 EA, 110 1
o²
3n
aj + 8y/EETT + 12 ao:r D
(34)
"
M
t=
where we have uses S 4T which holds since T 1. The next lemma provides bound for EV,.
Lemma 5. For any S T KR, Alg. with the learning choice in Eq. (8) ensures the following bound.
T EV, < 40 K
Σ du. (G² + + 902
Plugging the above bound back into Eq. (34) gives an almost explicit bound,
00:
EAc
|| [W0 w"
T
311
1t
2
1)
M
1=0
4. 10 L2') TT
00:: (G2+ 4LD,) LE
ao:
ww
w01
T
T Σ 7=0
+
+31
at +
a0:r G2
M
(=0
T
+ (12m) + 16 1
TK
E
a
t=0
w=11
31
0²
Σ
T
it
+
720L
TK +4. 10
00 G
1
E
0o. Ar,
1)
M
2(T + 1)
1=0
1=0
(35)
and we used 12nL S 1/4(T 1) which follows since 1) S TSL(T+1) (see Eq. (8)), as well as 16 10 TTK < the above bound and invoke the following lemma,
1/4(T + 1), which follows since 1) (T (see Eq. (8)). Next
Lemma 6. Let (Ather be a sequence of non-negative elements and B E R, and assume that for any t < T.
A s B
A
2(T +
t=0
Then the following bound holds,
Ar < 2B
110-w'11 + 31 DT at + TK +
Taking A, +- Eo, and B +-
1
10 10' ΣT=o Ourr G provides the following explicit bound.
2|(w0 - 10
o²
T 22
00.r Fo:TEAP
6n
Σ
Σ Go
1
M
t=0
2 2|| - w0
g²
+ 6n
(KR)3 + 103 2 18.1 G.
n
M
(36)
where we have used Σ C0. S Etr of < ER-
S K3R3 as well as
T KR.
21

---

Recalling that T = K R. and that,
Ww w" wowlvvvvvv IVM
Wo w 10 1|
7 min 48L(T+1)) 10LK (7 1)2/3*
TT3/2
1/2K7/AR(0//22 G!!2)
The above bound translates into,
Co.TEAT S
(37)
o
- ( - 1²1 o olwo w 173///
L +G!/3 11w0 - w" 13/22
+
M
(38)
Noting that co.T 2 2(T2) and using T KR gives the final bound.
EAT S
LI| LIlwo w012 Lilwo w*ll
L L w*ll2
o allwo w'll
(2/2 G!/²3
+
KR
R1/3
R2
VMKR
K1/4
which establishes the Theorem.
G Proof of Lemma 2
Proof of Lemma 2. The update rule implies for all [T]
|| u1 (w, u) na+ 1|
|| - ull - 2n (w, u) + 12 |19
Re-ordering and gives.
2ma Ir (w, u) (Blw, u/1 || - ull3) + "
Summing and telescoping obtain,
2n a.97
(w- - u) = (Ilw1 - u|2 - -110e+1 u|l2) + n
a 11-11
T=0
< [| - u1 + 17 Σ 0 11
ul
Dividing the above by 2n establishes the lemma.
H Proof of Lemma 3
Proof of Lemma 3. Recalling the notations D, := ||w, - 11 our goal is to bound EDo... To do
so, we will derive recursive formula for Do..- Indeed, the update rule of Alg. 2 implies Eq. (26), which in turn leads to the following for any E [T].
|| " 212 (w w²) 70..t || ||w, w=1l 2nogge (wx -o)=-w)) + 1700 ||
Unrolling the above equation and taking expectation gives,
E W - w01 = 1|w0 1111--21
Σ r=0
a Gr (w, - -w)++ 1 Σ a |1g 1|2
(39)
7=0
(+)
The next lemma provides bound on (x), Lemma 7. The following holds for any t E TT]
xx) < 2ny EV. E.
and recall that Do.I == ET |[, w" 3 and VT = Et af vf
22

---

Plugging the bound of Lemma 7 into Eq. (39), and using the notation of D, we conclude that for any t T].
ED; s Do + 2ny 20//V,, EVT v /EDor + r
2
1 ||
==0
S Do 2nV EVT EDoT B
19
11
t=0
where we used t T. Summing the above equation over gives,
T 112 + 2n/T. EVV VDrrrr + TTj E EDor s u w²11
_
of
19
(40)
t=0
We shall now require the following lemma,
Lemma 8. Let B, 0, and assume that A S. B CVA, then the following holds,
A $ 2B + 4C
Now, using the above Lemma with Eq. (40) implies,
T 2T E
EDorr S 2T wwl - w1112
Σ
al 167 + EVr
(41)
1=
where we have taken + < Db.r. B B - T
Σ +
of 11 and C + 2n/T VEV.
1=
Thus, Eq. (41) establishes the lemma.
H.1 Proof of Lemma
Proof of Lemma 7, Recall that (+) 22nE E1 Σr= (w, w²), we shall now focus on bounding (2)/2m,
E
a2 gr (w, w²) --E argr (w, w²)
7=0
=-E ar V (x=) (w, w²) E Σ a- (gr vf(x+)) (w, w²)
-=0
5 50+ E
2l vf(x;)]
||,, - w w1
r=0
5 Σ
Ea gr vf(2,)
E W. - w" 1
E
02
[((x))1
E
||,, w 112
7=0
r=0
E t=
lg.
vf(x))?
E t=
lw
w0
EV
E.DO.
(42)
where the first line is due to the definitions of gr and 3r appearing in Eq. (29) (this is formalized in Lemma 9 below and in its proof): the third line follows by observing that the {xa h sequence is
and {at) he weighted average of } and thus Theorem implies that E Et a, fx (w,
23

---

w²) 2 0 for any t, as well as from Sauchy-Schwarzz the fourth line follows from the Cauchy-
Schwarz inequality for random variables, which asserts that for every random variables X, Y. then E|XY] & VEX²vEY' the fifth line an application of the following inequality
Σ by
6
T=0
T=0
which holds for any two sequences {ar E Jr. fbr E R. and the above also follows from the standard Cauchy-Schwarz inequality. Thus, Eq. (42) establishes the lemma.
We are left to show that [9r (w, - -w"]] = E [ (w, - w²)] which is established in the lemma below,
Lemma 9. The following holds for any T E [T].
E 19 (w> - )] = 10- (w, - w
H.1.1 Proof of Lemma 9
Proof of Lemma 9. Let {F, brerr be the natural filtration induces by the history of samples to every time step т. Then according the definitions of St and 9t have,
E [9- (w, w""]
E E g (w, "²) -=
E [gr Fr-1l || (w, - w²)
E E [ Fr-1l (w, w²)]
[gr (w, w²)
where the first line follows by the law of total expectations; the second line follows since w, is measurable w.r.t. Fy-1i the third line follows by definition of gr and Tri and the last line uses the law of total expectations.
H.2 Proof Lemma
Proof of Lemma 8, We will divide the proof into two case.
Case 1: B 2 CVA. In this case,
BB 2B
Case 2: B S OVA. In this case,
A B CEB+CVA
dividing by V and taking the square implies,
A 40 2B + +
And therefore the lemma holds.
I Proof of Lemma 4
Proof of Lemma 4. Recalling that == a2 we will decompose gr (gr gr)
(gr vf(x,)) + (x(x)) which gives,
24

---

T
(C) = E Σ
a al( Dr) + (0r v/(x+)) (+++)
7=0
T
< 3E
2 19 19
3+112 + 3E
و - v)(+>)||
3E Σ
a
V (
0=0
7
rm0
$ 3E
af 1gr
11 + 3EV 6LE
0 T
T
3 E
3EV + 61E Σ a²Ar
M
T=0
T
$ 3
E
Σ
3EV, + 12LE Σ 00 Ar
(43)
M
T=0
where the second line uses ||a + b + cll2 < 3[1a||2 + |b6| + ||c|| which holds for any a, b, c E Rd
the third line uses the definition of VT as well as the smoothness of f()) implying that [V < (T((((=)) [(xx)) == 2TA (see Lemma below); the fourth line invokes Lemma 11; the fifth line uses a = (T 1)2 < 200..r.
Lemma 10. Let F Rd + RR be an L-smooth function with a global minimum x", then for any 12d have,
IVF(2)) S (L( - F(w*))
Lemma 11. The following bound holds for any t E [T],
E 9- F- v a²
M
I.1
Proof of Lemma 10
Proof of Lemma 10. The smoothness of means the following hold Vw, u Rd
L 2 1ul
F(x+u) s + VF(x) +
Taking - (z) get,
F(x uu) S F(x) IIVF(()) 1 |VF())| F(x) 2L [VF(z)]
2L
Thus:
[VF(2)]1 < (((F(z)=F(x+
((F(())F(x')).
where in the last inequality we used F(x*) < F(x + u) which holds since x" the global minimum.
I.2 Proof of Lemma 11
Proof of Lemma II. Recall that we can write,
9r
Jr =
(9) - gr)
M
ME
25

---

where T. = Vf( and o == fi 2f), and that " M are independent of each other. Thus, conditioning over { then (o) gr Fh are independent and zero mean i.c. Elg² g+((+]]= Consequently,
E [1l++ Đ-1 0-131
E M
Σ lo - It) |E|M]
Kztkeem
Σ
E
[llg
001191(a))111111
M²
rE[M]
Σ o²
M
(E[M]
M
Using the law of total expectation implies that Eligr
9r 1 o²
M
Proof of Lemma 5
Proof of Lemma 5, To bound EV, we will first employ the definition of xz together with the smooth- ness of f())
E (xx, - 3- E
v(((x,)
Σ f((=t)
M
M
ME[M]
SEIM]
M
iE[M
L
E 1
M
L
E
Σ x JE[M]
M
M
M
VE
L
<
Σ
E
E
- +
(44)
M
MJEM
where the first line uses the definition of gt. the second line uses Jensen's inequality, and the third line uses the smoothness of fa(.)'s. The last line follows from Jensen's 's inequality.
We use the following notation for any (T)
a.t : a xe 11? & of = o Σ
x,1 & Q
af
Σ
|,
M
JE[M]
[JE(M
(45)
and notice that ΣjE[M] g.¹ = gf. and that ELSE[M] g.¹ M2Q,. Moreover ب:"d oha VVi E [M].
Thus, according to Eq. (44) is enough bound EV, as follows,
EV. [ -. <
Qr
(46)
Next we will bound the above term.
26

---

Bounding (+): Let E [T]. if = rK for some E [R], then according to Alg. 2 2 It for any
machine i E [M.] thus x x for any two machines i, [M].
More generally, if rK + for some [R], and h E [K], then by denoting to == r we can
write = tu + k. Using this notation, the update rule for x² implies the following for any i E [M],
Σ
0+ a,
00:a
Tto
Σ r=1o+1
arw,
00:
r=lo+1
where we used r/1 Ito
Vi E [M].
Thus, for any + j write,
af
t Σ
2
af ||| x/11? = (ao.)) )
a, (ut - wff
(47)
1=
So our next goal is to derive an expression for Ei= ar wf). The update rule of Eq. (6)
implies that for any T E [to, to + K],
- = wig
angi.
Wto 17
angn
(48)
n==
n1
where the second equality is due to the initialization of each round implying that wt, = Wto Vi E [M]]
Next, we will require the following notation Ø. := V)(()) and ş == of - g.. We can therefore write, gf D. Ei and it is immediate show that E(g, a Using this notation together with Eq. (48), implies that for any E [fo. to + K] and + j we have,
ar(w,, - -wt) = n
Σ n=to+1
aran (Fn F1)
Σ
a-an(5i Eh)
n=to+1
aranl
-gh)
arannn
(49)
1=
n n=to+1
and in the last line we use the following notation : c 5 Summing Eq. (49) over
[to 1, obtain,
Σ ar (w. - w²)= = - Σ Σ
a anan - - Σ Σ
a, an
r=
7=b n=lo+1
areen (0's -gh)
1
n fo 11 rmn
nmfo + rmn
On (g - -
angannn
n=fo+1
a=to+1
a,1a, (T+ -
Σ
ar
(50)
+1
r=
where in the last equation we replace the notation of the summation index from (only done to ease notation).
A more appropriate notation would be c(i.)) = et - cd. but to ease notation we absorb the (i,j) notation
into ç.
27

---

Plugging the above equation back into Eq. (47) we obtain for any t E (to. to + K]
1 : aillo - x
wi) |
Σ
ar(wt
(2
r-
a?
Σ
ar (0) -gt) +
Σ ar fr
(ao.r)
17
r=tu+1
n=to+1
2 20
20
S (aw:)
Σ
ard - [t)
+ t1p (av:)) )2
Σ ar:
+=
1=ot+1
2
$ 1²2 2a
Σ (ar.a,,) ) 02l1 + 172 2a
(t (t- to)
Σ
ara
(ana)
(aoxt)
r=ta+1
r--tot
S 11 2a
t
20
2
K (Ka))
Σ ((((t()) {++(2)1 + +1
(000))
(oo:))
T=lo-1
r=ot+1
= 1/ 20
1
Σ ()( - + n?
2a
ar
(00.)
L
(anx)) )
r=o++1
1=
< 817 kT?
Σ
V((())) - v + (8n7/aa²)
Σ
0
2
1=6++1
r=
= S
Σ r=
a (((() - v=(+)) + +nn"aa"
Σ 1=
ar Orts
- Er
1
Oi
(51)
where the first inequality uses ||a + b12 < 2||a|| + +21bb which holds Va, E Rd, the second inequal-
ity uses |I E>>> N a,.l1 < (N2- N2) Em' 2 ||q,, 2 which holds for any {an E N
N1+1
the third inequality uses the definition of g it also uses - lo < K as well as (oxpa)²² < (Kaz)² which holds since 7 < and since both a, S ay; and the last inequality uses the fact that ag = t + 1 implying that the following holds,
S 44 & af
(ao:t)
at
Lemma 12. The following holds for any i E [M].
3L2
(g: + Φ5) + 6 (VV(((()))))
Ma?
Using the above lemma inside Eq. (51) yields,
a. == a² | x!11?
< 2412 3²
1 (gi + dlotia + 28
Σ aF + 7=
M
+ 80j0
Σ
Oir arii
- at
ST
1
2 M (gho+1 + Pootlad
1
+
Σ af (IV/(<+))1 + [[+(())))))) + B (52)
2
+=o++
28

---

where we have denoted 8
:- 1:=481fK3L2, & B : 81)2
Σ r=
ar ard
257
Summing Eq. (52) over j [M] and using the definition of Qt gives,
M²Q= = E
4::
Σ 2 M
1
22 A.MQQ. Qiotii +
8
2M
Σ a Σ
[f((zt)]
+ Σ B
L
1=
iE[M]
(JE(M]
2M0
M20 Qto +
Σ
0 Σ
[[[(<+)|11
Σ
B
(53)
r=
iE[M]
where we used,
- Σ
1 = E -&Q= N²Q
jE(M]
(E[M)
JE[M
Now dividing Eq. (53) by 2 gives Vt E [to, to + K].
Qc 5 Qu+t1 +
20
a
1
Σ
Σ
B
L
M
M
-=
iE[M]
(JE(M)
20
S 0- Qto+iit + I
Σ a (G. + (L(((()))) f(w()))
Σ
B
M
r=o++1
(JE[M]
20
S Qt+ta +
Σ o. GG, 4LA,)) (G++4LA)))
Σ B
(54)
L
M
+=
(.JE[M]
where the second line follows from the dissimilarity assumption Eq. (2), and the last line is due to the definition of A.. Thus, we can re-write the above equation as follows forall E (40, 40 + K].
Qe 5 OQootii+ + He,
(55)
where H4 = 28 Ei=to+1 o² -G + 4LD,)) + 3/= EiJelM B and recall that = 48r/2 K L2. 7,3
Now, notice that Qu. H, 2 0, and that 0 satisfies 0K 48nj K* < 1/2 since assume that 7/2 I To (see Eq. (8)). This enables to make use of Lemma 13 below to conclude,
lot
QtotliotK S 2H 21
40
Σ
o
(G? 4LA,)) +
2
Σ (.jE[M]]
B to+1:
L
M
r=6++1
lotK
2000y² K
Σ a GG 4LA,)) +
2
Σ
B
M
r=to+1
(.E[M]
(56)
Lemma 13. Let K,tK O, and assume OK < 1/2. Also assume sequence of non-negativv terms
(Q, 2 fo+K and another sequence {H, 2 0 that satisfy the following inequality for any t [to, to + K).
Q₂ S 0Q + H
Then the following holds,
+to S 22Hgg11to
*Formally we should use the i,j upper script for & in the definition of B,",, i.c. to define B 8722 of Eo Oirit By e/,o 1/ We absorb this notation into & to simplify the notation.
29

---

Recalling that we like to bound the expectation of the LHS of Eq. (56), we will next bound EE Ort Ord 6r || which is done in the following Lemma 9
Lemma 14. The following bound holds for any t [to, to + K}.
E
Or of
&
I <4K
Since the above lemma for any i.j and t E [to, to + K]
we can now bound EB
+sto++ as follows,
lo-K
EB kobkk = 8
E
argar &t ai
1=to+1
to-k
S 81
Σ
4K30²
1=ta+1
32n
32n22r1 (r + +1)
(57)
where the last lines a S (r+ 1)K for any iteration that belongs to round r.
Since the above holds for any i,j it follows that,
Big
lot 2 52-327 + 1)² (r -122
M
[JE[M
Plugging the above back into Eq. (56) gives,
to
Qtoll:ttK& S 2000)² K3
Σ
(58)
+=o++1
Final Bound on EV. EV, as follows,
Finally, using the above bound together with the Eq. (46) enables to bound
EV, <
Σ Q
$ Σ Qr
T=0
T
R-1 rK*K
< 200 K
Σ
a (G. +LA,) + Σ
Σ
a x!
r=0
k+r
R-1
< 200n)2 8
0 (G 4LA,) +
Σ
64n7 (r + 10
T
< 200ny 3
0 (G 4LA,)) + 64n K R3
< 400,)² 2 K3
Σ
0:0:7 (G² (TA,)) + +90
7=0
where we have used Σ R Consequently, we can bound
Σr=
(r + 1)2 < R3, and the last line uses af (T 1)2 < 200
T
EV, < 40
Σ
ooor (G.+ + "
which established the lemma.
recall that for simplicity of notation denote rather than go.
30

---

J.1 Proof of Lemma 12
Proof of Lemma 12. First note that by definition of I, we have for any i [M1]
|0, x
1 M
Σ
S
Σ
-
Ma? 4
(59)
M
[E[M]
[e[M]
where we have used Jensen's 's inequality, and the definition of of. Using the above inequality, we obtain,
[V)( ( ((V/(()))) - (1))))))+((00))------)------1---
s 311² (1|z,- - = + -1 (v (()(a))
S <3[V\((;)) [X
3L2
< Ma
(0) +
where the second and third lines use DN 0n 12 N >
-m= ||a.|1 which holds for any {an
R JR= we also used the smoothness of the fi(-)'s; and the last line uses Eq. (59).
J.2 Proof of Lemma 13
Proof of Lemma 13. Since the Qt's and are non-negative, we can further bound Q for all [to, + K] as follows,
Qi s 1000-11 + H, S 6Q +1 + He
Summing the above over f gives,
bo*K
Qootldot =
Σ
Q S.SK OK Qo+ldot + HiotLotK S Qo + Higt11otK
1d
where we used 0K < 1/2. Re-ordering the above equation immediately establishes the lemma.
11
J.3 Proof Lemma
Proof of Lemma 14. Letting (Fe)e be the natural filtration that is induces by the random draws up to time 6, i.e. by {{1 (e]hee],)...... {Ftheean) By the definition of & it is clear that & is measurable with respect to Fi, and that,
E = 0.
31

---

Implying that ferhe is martingale difference sequence with respect to the filtration {Fih. The following implies that,
E
Olgitltr 2 ST
Cr:ectty at
E. 11
ag
to
Kay ai
2
Σ
||E=|1
+=
k2
E B||Er11
K
E ||st - El1l
T=
S 22
E cl + (2
r=fo+1
2K
20²
7=6o+1
4KK20 (t (t)))) 1()
$ 4K302
where the first line follows by Lemma 15 below, the second line holds since a, 0z and 0rd S Ko (recall ar S ar since т S f); the fourth line follows due to & == & E/: the fifth line uses
|a++bll bll < 2||a|| 2|bb] for any a, E Rd; the sixth line follows since E||5: == Ellgi - -21l =
E V f( (z+
2 < o²; and the last line uses (t - to) I K.
Lemma 15. Let (Eite be martingale difference sequence with respect filtration {Fih then the following holds for all time indexes t1,t2 2 0
E
3r
E
ler 11
1=
7=h
J.3.1 Proof of Lemma 15
Proof of Lemma 15. We shall prove the lemma by induction over lz. The base case where t2 = t clearly holds since it boils down to the following identity,
2
4 =
E
E
IE,,
2
Σ T=h
E
11?
32

---

Now for induction step let us assume that the equality holds for t2 2 t and lets prove it holds for t2 + 1. Indeed.
t +
E
Sta+1
+
1/
2
12
E
ST
+
E
E1
11
2E
Σ Er
Ectt1
1=
E
11
2F
E
Σ
&r
E1o+1
IF
T
r
Σ E |1 2F
Σ
St
E
1
E B|s++11+0
T=h
2+1
Σ E 11
where the third line follows from the induction hypothesis, as well as from the law of total expectations; the fourth lines follows since { are measurable w.r. Ftg, and the fifth line follows since E[c.,+1 Fizl 0. Thus, have established the induction step and therefore the lemma holds. O
K
Proof of Lemma 6
Proof of Lemma 6. Summing the inequality A, S B + 2(T+1) Et= A, over & gives,
Ao. S 1
2(T + +1) Ap.r (T + 1)B
Ao.T
2
Re-ordering obtain,
Ao:r S 2(T + 1)5.
Plugging this back to the original inequality and taking = T gives,
Ar - B +
AoT - 2B. 2(T + 1)
which concludes the proof.
L The Necessity of Non-uniform Weights
One may wonder, why should we employ increasing weights x t rather than using standard uniform weights an 1 Vt. Here we explain why uniform weights are insufficient and why increasing weights e.g. 0r x I are crucial to obtain our result.
Intuitive Explanation. Prior to providing an elaborate technical explanation we will provides some intuition. The intuition behind the importance of using increasing weights is the following: Increasing weights are a technical tool to put more emphasis on the last rounds. Now, in the context of Local update methods, the iterates the last rounds are more attractive since bias between different machines shrinks process, the bias in the stochastic gradients that we compute goes zero (in expectation), and consequently the bias between different machines shrinks progress.
progress, Intuitively, this happens since as we progress with the optimization
33

---

Technical Explanation. Assume general weights farte, and let go back to the proof of Lemma (see Section J). Recall that this proof we derive a bound of the following form (see Eq. (55))
A I A + BB,
(60)
where A, :-
:0:00]lle [/]l0]] 12112. B₁ Sija
EE= ard 5r and importantly 10
2a L²
0 : r²
(om::s)
Now, crucial part of the proof the fact that 0K < 1/2, which turn enables to employ Lemma 13 in order to bound EV.
Not let inspect the constraint OK < 1/2 for polynomial weights of the form or IP where 0. This condition boils down to,
20
K K 1/2,
(0o:))
Implying the following bound should apply to for any E [7],
Cod of
1
g+1
1
t1-p
2LK²
2LK
D2p
2LK
Now since the bound should hold for any / E [T' we could divide into two cases:
Case 1: p < In this case (1-P is monotonically increasing with the above condition should be satisfied for the smallest t, namely t = 1, implying,
" S
2LK2
Of
2LK
The effect of this term on the overall error stems from the first term in the enror analysis (see e.g. Eq. (36)), namely,
1
Iwo w013
2LK2,
|w0 w*ll
Ctp:T
17
Tp-1
Now, for the extreme values p O(uniform weights) and (linear weights), the above expression results an error term of the following form,
O(KR)) Err(p 0) o ((K/R) & Err(p o(k2/r9)=(((1))))
(61)
Thus, for p 0, the error term is considerably worse compared to Minibatch-SGDD and p 1 is clearly an improvement.
Case 2: p 1. In this case ('-p is decreasing increasing with the above condition should be satisfied for the largest I, namely / = T, implying.
Coo
1
n - 2LK
of
2LK
TP-I
Now, the effect of this term on the overall error stems from the first term in the error analysis (see e.g. Eq. (36)), namely,
1
Wwo w11
2LK².
|wo - w=1l 10.11.7001 2LK². |lwo w=1
00:T
7)
Tp+1
72
Thus, for any p 2 1 we obtain,
Bm(p)= O(K3/)) 0(1RR))
(62)
Conclusion: As can seen from Equations (61) (62), using uniform weights or even polynomial weights 1P with yields strictly worse guarantees compared to taking 1.
Indeed, in the where t+ 1 we take 0 = 8r) 3 L2 2 and this is used in the proof of Lemma
34

---

M Experiments
M.1 Data Distribution Across Workers
In federated learning or distributed machine learning settings, data heterogeneity among workers often arises due non-identical data distributions. To simulate such scenarios, we split the MNIST dataset using Dirichlet distribution. The Dirichlet distribution allows control over the degree of heterogeneity in the data assigned to each worker by adjusting the dirichlet-alpha parameter. Lower values of dirichlet-alphe result in more uneven class distributions across workers, simulating highly non-IID data.
Given C' classes and M workers, the probability Pa,m of assigning data from class c to worker m is based sampling from Dirichlet distribution:
Pe,1, Pa......
Pe.M Pc.
Dirichlet(a)
where a is the concentration parameter controlling the level of heterogeneity. smaller a value results in more imbalanced distribution, meaning that each worker primarily receives data from limited subset of classes, In this experiment, we set = 0.1 to induce high heterogeneity.
The following figures present scatter plots illustrating the class frequencies assigned to individual workers for different worker configurations- 16, 32, and 64 d-uiinng specific random seed.
(a) 16 Workers.
(b) 32 Workers.
(c) 64 Workers.
Figure 2: Class distribution across workers different numbers of workers (16, 32, and 64) on the MNIST dataset. The dataset was partitioned using a Dirichlet distribution, with the dirichlet-alpha parameter set to 0.1 to induce high heterogencity. Each scatter plot illustrates class frequencies for cach worker.
M.2 Complete Experimental Results
This section presents the complete experimental results for 16, 32, and 64 workers, showing test accuracy and test loss as functions of local iterations K. The plots illustrate the method's scalability and performance, with higher accuracy (f) and lower loss (J) indicating better outcomes.
Figure 3: Test Accuracy Local Iterations (K) for 16 workers (f is better).
35

---

Madrice
Figure Test Accuracy vs, Local Iterations (K) for 32 workers (f is better).
Figure 5: Test Accuracy vs. Local Iterations (K) for 64 workers is better).
Figure 6: Test Loss vs. Local Iterations (K) for 16 workers (1 is better).
Figure Test Loss vs. Local Iterations (K) for workers (L is better).
Figure 8: Test Loss vs. Local Iterations (K) for 64 workers (1 is better).
36

---

NeurIPS paper checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: we claim to present the first local-update method that improves over the minibatch baseline in the SCO heterogeneous setting, and this is exactly what establish in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the intro and conclusion sections, we discuss that our approach does not aqpply to non-convex scenarios, and state it future direction
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
37

---

Answer: [Yes]
Justification: we indeed provide the full set of assumptions and a complete (and correct) proof
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
38

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: :ettpss:/niiee //tps//nipp///////////// cc/
publ i guides/ CodeSubmm ss oooolloloooooooo icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cc/ pttpppullcccocc httpiiic//iieeeeeeeeeeeeeeisi CodeeeeeeSSeeeeee Sumeesubmiii. cessooPolllo ooyicy for more details.
unips.ccppubicc.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
39

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::////euriie cc http://s..c.oolllcccctccccccleoo
Answer: [Yes]
Justification:
we ensured that our paper conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: we not foresee any special societal impact that arise due to our work
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
40

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' raw w thcode com/ data sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
41

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: our work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "kbUN8v0jpM", "reviewer": "Reviewer_3Xai", "summary": "This paper proposes SLowcal-SGD, which is a distributed learning algorithm that builds on customizing a recent technique for incorporating a slowly-changing sequence of query points, which in turn enables to better mitigate the bias induced by the local updates. Theoretical proof is given on the proposed algorithms.", "questions": "Please refer to the previous section.", "limitations": "The paper is theory-driven and does not have negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The idea of combining local updates and mini-batch SGD to improve the data heterogeneity case is interesting.\n* The author provides intuition on why slowcal-SGD is useful, and the comparison of proposed algorithm and other algorithms in table 1 is clear.", "weaknesses": "* While the paper is heavy on theory, it has no validation on any synthetic or real-world data. It should not be hard to verify this since this is convex problems and there are multiple ways and open sourced code to produce heterogeneous data.\n* The rates shown in table 1 is a little confusing. Comparing to accelerated mini-batch SGD, what is the advantage of proposed slowcal-SGD?\n* The assumptions of equation (1) - (3) are somewhat strong if we discuss the data heterogeneity. What is the main difference between proposed slowcal-SGD and general variance reduction SGD? (which doesn't need equation 1-3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer 3Xai**\n\nThank you for your comments, we address your concerns below and kindly ask you to raise your score accordingly.\n\n\n**Regarding weaknesses**\n\n\n\n**Q:** adding experiments\n\n**A:** We have added experiments that demonstrate the benefit of our approach and corroborate our theoretical findings. Please see the details in our response to all reviewers.\n\n\n\n\n**Q:** comparison to accelerated minibatch sgd in table 1\n\n**A:** Indeed, as we describe in our paper, currently there does not exist a baseline that improves over the accelerated-minibatch-sgd, and this also applies even for the simpler homogeneous case. \nIn the heterogeneous case, no method prior to our work was able to improve over the simpler minibatch-sgd baseline, and we are the first to establish such guarantees. We hope that the new technique that we introduced in our paper may pave towards designing an approach that will be able to improve over the  accelerated-minibatch-sgd baseline.\n\n\n\n\n**Q:** Assumption on heterogeneity\n\n**A:** Assumption in Equations (1-3) are indeed related to heterogeneity. But please note that we only consider the Assumption in Equation (1) to hold, which is the less restrictive assumption (Equation (2) is a consequence of Assumption (1), and Equation (3) is a much stronger assumption than (1) ).\n\nThe heterogeneity assumption is not required for parallel training methods like minibatch-sgd and its accelerated version, since in such methods the query points of all workers are fully synchronized. Nevertheless, in local update methods (like Local-sgd and SLowcal-sgd) there is a drift in the query points of different workers (due to the local updates) and therefore heterogeneity must comes into play (this is also evident from existing lower bound for local-sgd).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcGe700vSK", "reviewer": "Reviewer_5jSD", "summary": "This paper introduces a new federated learning algorithm called Slowcal-SGD, which basically introduces anytime-SGD into federated learning setting. The authors provide solid convergence analysis and fruitful insights on the new algorithm. They show the algorithm can provably beat both mini-batch SGD and local SGD in the heterogeneous data setting. No experiments are provided, though.", "questions": "See above comments", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- For each theorem, the authors provided insightful discussions, explaining why the algorithm works better.\n- The proposed algorithm is novel and neat.", "weaknesses": "- It'd be better to define \"query\". It could be controversial. For example, why is computing the gradients wrt a mini-batch a single query instead of B (batch size) queries?\n- No experimental results are provided. So it's hard to tell whether the proposed algorithm works in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer 5jSD**\n\nThank you for your supportive review, below we address the points that you have raised. \n\n\n**Regarding weaknesses**\n\n\n**Q:** adding experiments\n\n\n**A:** We have added experiments that demonstrate the benefit of our approach and corroborate our theoretical findings. Please see the details in our response to all reviewers.\n Since the experiments were your main concern, we will appreciate it if you can consider raising your score given the experiments we conducted.\n\n\n**Q:** definition of “query”\n\n\n **A:**  When we use the term “query point” we refer to the point (model parameter) at which we estimate the gradient (of the expected loss).\n\n\n\n**Q:** “..., why is computing the gradients wrt a mini-batch a single query instead of B (batch size) queries?”\n\n**A:**  in your question when you use the word “query” you mean the number of samples (or stochastic gradient computations) that are employed. We do not use the term “query” but rather the term samples or number of (stochastic) gradient computations. This is since we already use the tem “query point” (as we explain above) and we do not want to confuse it with the word “query”. Specifically, when we relate to a batchsize of $B$ we indeed count it as $B$ samples and $B$ (stochastic) gradient computations.\nWe will clarify this in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f5VfCyQLG1", "reviewer": "Reviewer_mxJX", "summary": "This work studies distributed model training with a parameter server; contributions are theoretical:\n* **Assumptions**: $L$-smooth convex objectives, stochastic gradient estimates, heterogeneous worker distributions, bounded difference in expected gradients of local workers' objectives at a global minima, denoted $G^\\star$.\n* **Main result**: Given a fixed set of communication rounds $R$ with the parameter server, $M$ worker machines, and a budget of $K$ local gradient queries per worker in each communication round, this work presents a simple method for improving local SGD. Given $L$-smooth convex objectives with stochastic gradients and heterogeneous worker distributions, the local-SGD variant proposed in this work achieves more favorable convergence than mini-batch SGD (where the $K$ stochastic gradient queries are used to compute a larger mini-batch gradient at each worker). The proposed method converges in $\\mathcal{O}(MK^{-1/2})$ rounds, whereas mini-batch SGD converges in $\\mathcal{O}(MK)$ rounds.\n* **Method**: The proposed method extends anytime GD (Cutkosky, ICML'19) to the distributed local-sgd parameter-server setting. In short, regular local-sgd with $K$ local steps, but where stochastic gradients are computed at an exponential moving average (ema) of the model parameters. The parameter server averages both the primal and ema parameters, and sends back the exact average to each worker at the end of each round.", "questions": "* Please consider including simple numerical experiments in the convex setting (e.g., multinomial logistic regression or least squares) to empirically evaluate the performance of the proposed method.\n* Would appreciate a discussion on the main challenges to demonstrating improved convergence relative to accelerated mini-batch SGD (not just vanilla mini-batch SGD), especially since the proposed EMA update can be given a momentum-like interpretation as stated in Appendix C.", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Clarity**\n* This work is extremely clear and well written. I have skimmed the main proofs in the appendix, but the proof sketches in the main paper provide enough detail to understand and follow the primary logic and intuition behind each result.\n\n**Quality**\n* I have quickly gone over the proofs in the appendix, and the work appears to be technically sound.\n\n**Originality**\n* To the best of my knowledge, this is the first such extension of anytime GD to the distributed stochastic local update setting, and the first result demonstrating the improvements of local sgd over mini-batch sgd in general smooth convex settings. Previous studies were either devoted to the non-convex setting or quadratic objectives, or assumed bounded second moments.", "weaknesses": "**Significance**\n* No numerical experiments conducted to explore the convergence of the proposed method or generalization capabilities. To increase impact and adoption, researchers may be interested in the generalization performance of the method; i.e., going beyond the number of communication rounds required to decrease training error. Would design considerations (e.g., choice of $\\alpha_t$ schedule) change in such a setting?\n* Not a major issue for a theoretical paper, but current hyper-parameter choices requires knowledge of problem parameters ($\\sigma$ and L-smoothness constant).\n\n**Clarity**\n* Minor complaint is that the proof sketches do not always correspond to a similar logic used in the actual proofs themselves. For instance, (16) in the proof sketch in the main paper was obtained by assuming a somewhat monotonic iterate sequence, but such a result is never explicitly proven in the appendix. Instead, Lemma 3 in Appendix H (which bounds the sum of the iterate sequence using the T times the starting error, plus additional terms depending on the gradient magnitude and variance; which should indeed converge) is used to arrive at (33) in the proof of Theorem 2 in the appendix, which corresponds to (16) in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer mxJX**\n\n\nThank you for your supportive review, below we address the points that you have raised. \n\n\n**Regarding weaknesses**\n\n\n**Q:** adding experiments\n\n\n**A:** We have added experiments that demonstrate the benefit of our approach and corroborate our theoretical findings. Please see the details in our response to all reviewers. \n\n\n**Q:** “...researchers may be interested in the generalization performance of the method…Would design considerations (e.g., choice of schedule) change in such a setting?”\n\n\n**A:** Thank you for this comment. In fact we do derive guarantees for generalization (expected excess loss).\nThis can be directly seen from our description of the problem (see Section 2 lines 81-97, and specifically lines 92-94), as well as from the guarantees that we state in Theorem 2 (lines 191-196). In our experiments we present both test loss and test error.\n\n\n**Q:** “not a major issue but…..learning rate requires knowledge of problem parameters”\n\n\n**A:** This is a good point. Our focus in the paper was on showing that one can improve over the minibatch-sgd baseline in the heterogeneous case, which was already quite challenging. We agree that developing and exploring adaptive (or parameter free) methods for local training methods is an important topic, and we hope to extend our work in this respect in the future. It is interesting to note that (as far as we know) there does not exist an adaptive (or parameter free) variant of the standard local-sgd baseline, which is in itself an attractive future direction.\n\n\n**Q:** “Minor complaint … proof sketches do not always correspond to a similar logic used in the actual proofs themselves”\n\n\n**A:** Indeed, as the reviewer noticed we made a simplification in the proof sketch, and the goal was to simplify the presentation of ideas and uncover the main ideas and intuition. Please note, that when we simplify the proof sketch we do mention this explicitly: in line 260 just before equation (16) we write “to simplify the proof sketch we shall assume that $D_t \\leq D_0$”. \nIn the final version we shall mention explicitly that this simplification does not hold in the full proof.\n\n\n\n\n\n\n\n\n\n\n**Regarding questions**\n\n\n**Q:** “a discussion on the main challenges to demonstrating improved convergence relative to accelerated mini-batch SGD”\n\n\n**A:** Thank you for raising this important point. In order to improve over accelerated minibatch-sgd, we have considered designing an accelerated variant of our approach. Nevertheless, there are several challenges in doing so: First, one can think of incorporating an “acceleration ingredient\" to both aggregated and local steps which inserts another degree of freedom to the design of the algorithm, and it is not very clear what is the right way to do so. Second, accelerated methods are more complicated to analyze. Finally, it is not clear how acceleration can better mitigate the bias between different machines, which is a main obstacle in improving over our current approach. \n\n\nIdeally, we believe that one can find a way to optimally tradeoff between acceleration and bias (by controlling the learning rate and weighting scheme), thus leading towards better guarantees. \nWe will add this discussion to the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Dear reviewers**, 

We have now added experimental results comparing our approach to several baselines. Our results showcase the practicality and benefit of our approach and complement our theoretical findings. 

We have conducted  experiments on the MNIST dataset, which is a widely used benchmark in machine learning. The dataset consists of 70,000 grayscale images of handwritten digits (0-9), with 60,000 images in the training set and 10,000 images in the test set. We have executed our experiments on the NVIDIA GeForce RTX 3090 GPU and the PyTorch framework. 

We employed a logistic regression model and compared our SLowcalSGD with LocalSGD and MinibatchSGD across various configurations. Specifically, we tested with 16, 32, and 64 workers and varied the number of local steps (or minibatch sizes for MinibatchSGD) to 1, 4, 8, 16, 32, and 64. For each local update in the SLowcalSGD and LocalSGD, we used a single sample, with the weights for SLowcalSGD set as \(\alpha_t = t\). We used a learning rate of 0.01, optimized through grid search. To ensure the reliability of our results, we conducted our experiments using three different random seeds and reported the average results across these seeds. We made one pass on the MNIST dataset for all experiments to ensure a fair comparison.

Our results indicate that as the number of local steps K (or minibatch sizes for MinibatchSGD) increases, SLowcalSGD exhibits more significant advantages over LocalSGD. Notably, both approaches, SLowcalSGD and LocalSGD achieve better performance compared to Minbatch-SGD.

Meta-review: This paper gives a convergence guarantee for a distributed learning algorithm that improves upon both baselines of mini-batch SGD and local SGD. It does so by leveraging a slowly-changing sequence of query points (see Anytime-SGD), which in turn enables to better mitigate the client drift induced by the local updates. The paper provides novel insights, is well executed and interesting, as acknowledged by the reviews.

For the camera-ready version, we urge the authors to incorporate the mentioned feedback by the reviewers.
Decision: Accept (poster)