Title: FAVAS: Federated AVeraging with ASynchronous clients
Venue: NeurIPS 2023
Accepted: no
Keywords: Asynchronous Federated Learning

Federated Averaging
Abstract: In this paper, we propose a novel centralized Asynchronous Federated Learning (FL) framework, FAVAS, for training Deep Neural Networks (DNNs) in resource-constrained environments. Despite its popularity, "classical" federated learning faces the increasingly difficult task of scaling synchronous communication over large wireless networks. Moreover, clients typically have different computing resources and therefore computing speed, which can lead to a significant bias (in favor of "fast'" clients) when the updates are asynchronous. Therefore, practical deployment of FL requires to handle users with strongly varying computing speed in communication/resource constrained setting.  We provide convergence guarantees for FAVAS in a smooth, non-convex environment and carefully compare the obtained convergence guarantees with existing bounds, when they are available. Experimental results show that the FAVAS algorithm outperforms current methods on standard benchmarks.

Abstract

In this paper, we propose a novel centralized Asynchronous Federated Learning (FL) framework, FAVAS for training Deep Neural Networks (DNNs) in resourceconstrained environments. Despite its popularity, 'classical' federated learning faces the increasingly difficult task of scaling synchronous communication over large wireless networks. Moreover, clients typically have different computing resources and therefore computing speed, which can lead to a significant bias (in favor of 'fast' clients) when the updates are asynchronous. Therefore, practical deployment of FL requires to handle users with strongly varying computing speed in communication/resource constrained setting. We provide convergence guarantees for FAVAS in a smooth, non-convex environment and carefully compare the obtained convergence guarantees with existing bounds, when they are available. Experimental results show that the FAVAS algorithm outperforms current methods on standard benchmarks.

1 Introduction

Federated learning, a promising approach for training models from networked agents, involves 15 the collaborative aggregation of locally computed updates, such as parameters, under centralized 16 orchestration (Koneˇ cn` y et al., 2015; McMahan et al., 2017; Kairouz et al., 2021). The primary 17 motivation behind this approach is to maintain privacy, as local data is never shared between agents 18 and the central server (Zhao et al., 2018; Horváth et al., 2022). However, communication of training 19 information between edge devices and the server is still necessary. The central server aggregates the 20 local models to update the global model, which is then sent back to the devices. Federated learning 21 helps alleviate privacy concerns, and it distributes the computational load among networked agents. 22 However, each agent must have more computational power than is required for inference, leading to a 23 computational power bottleneck. This bottleneck is especially important when federated learning is 24 used in heterogeneous, cross-device applications. 25

26

27

28

29

30

31

32

33

34

Most approaches to centralized federated learning (FL) rely on synchronous operations, as assumed in many studies (McMahan et al., 2017; Wang et al., 2021). At each global iteration, a copy of the current model is sent from the central server to a selected subset of agents. The agents then update their model parameters using their private data and send the model updates back to the server. The server aggregates these updates to create a new shared model, and this process is repeated until the shared model meets a desired criterion. However, device heterogeneity and communication bottlenecks (such as latency and bandwidth) can cause delays, message loss, and stragglers, and the agents selected in each round must wait for the slowest one before starting the next round of computation. This waiting time can be significant, especially since nodes may have different computation speeds.

To address this challenge, researchers have proposed several approaches that enable asynchronous 35

communication, resulting in improved scalability of distributed/federated learning (Xie et al., 2019; 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

Chen et al., 2020, 2021; Xu et al., 2021). In this case, the central server and local agents typically operate with inconsistent versions of the shared model, and synchronization in lockstep is not required, even between participants in the same round. As a result, the server can start aggregating client updates as soon as they are available, reducing training time and improving scalability in practice and theory.

Contributions. Our work takes a step toward answering this question by introducing FAVAS , a centralized federated learning algorithm designed to accommodate clients with varying computing resources and support asynchronous communication.

In this paper, we introduce a new algorithm called FAVAS that uses an unbiased aggregation scheme for centralized federated learning with asynchronous communication. Our algorithm does not assume that clients computed the same number of epochs while being contacted, and we give non-asymptotic complexity bounds for FAVAS in the smooth nonconvex setting. We emphasize that the dependence of the bounds on the total number of agents n is improved compared to Zakerinia et al. (2022) and does not depend on a maximum delay.

Experimental results show that our approach consistently outperforms other asynchronous baselines on the challenging TinyImageNet dataset (Le and Yang, 2015).

Our proposed algorithm FAVAS is designed to allow clients to perform their local steps independently of the server's round structure, using a fully local, possibly outdated version of the model. Upon entering the computation, all clients are given a copy of the global model and perform at most K ≥ 1 optimization steps based on their local data. The server randomly selects a group of s clients in each server round, which, upon receiving the server's request, submit an unbiased version of their progress. Although they may still be in the middle of the local optimization process, they send reweighted contributions so that fast and slow clients contribute equally. The central server then aggregates the models and sends selected clients a copy of the current model. The clients take this received server model as a new starting point for their next local iteration.

2 Related Works

Federated Averaging (FedAvg), also known as local SGD, is a widely used approach in federated learning. In this method, each client updates its local model using multiple steps of stochastic gradient descent (SGD) to optimize a local objective function. The local devices then submit their model updates to the central server for aggregation, and the server updates its own model parameters by averaging the client models before sending the updated server parameters to all clients. FedAvg has been shown to achieve high communication efficiency with infrequent synchronization, outperforming distributed large mini-batches SGD (Lin et al., 2019).

However, the use of multiple local epochs in FedAvg can cause each device to converge to the optima of its local objective rather than the global objective, a phenomenon known as client drift. This problem has been discussed in previous work; see (Karimireddy et al., 2020). Most of these studies have focused on synchronous federated learning methods, which have a similar update structure to FedAvg (Wang et al., 2020; Karimireddy et al., 2020; Qu et al., 2021; Makarenko et al., 2022; Mao et al., 2022; Tyurin and Richtárik, 2022). However, synchronous methods can be disadvantageous because they require all clients to wait when one or more clients suffer from high network delays or have more data, and require a longer training time. This results in idle time and wasted computing resources.

Moreover, as the number of nodes in a system increases, it becomes infeasible for the central server to perform synchronous rounds among all participants, and synchrony can degrade the performance of distributed learning. A simple approach to mitigate this problem is node sampling, e.g. Smith et al. (2017); Bonawitz et al. (2019), where the server only communicates with a subset of the nodes in a round. But if the number of stragglers is large, the overall training process still suffers from delays.

Synchronous FL methods are prone to stragglers. One important research direction is based on FedAsync (Xie et al., 2019) and subsequent works. The core idea is to update the global model immediately when the central server receives a local model. However, when staleness is important, performance is similar to FedAvg, so it is suboptimal in practice. ASO-Fed (Chen et al., 2020) proposes to overcome this problem and handles asynchronous FL with local streaming data by

introducing memory-terms on the local client side. AsyncFedED (Wang et al., 2022) also relies on 89 the FedAsync instantaneous update strategy and also proposes to dynamically adjust the learning 90 rate and the number of local epochs to staleness. Only one local updated model is involved in 91 FedAsync-like global model aggregations. As a result, a larger number of training epochs are 92 required and the frequency of communication between the server and the workers increases greatly, 93 resulting in massive bandwidth consumption. From a different perspective, QuAFL (Zakerinia et al., 94 2022) develops a concurrent algorithm that is closer to the FedAvg strategy. QuAFL incorporates 95 both asynchronous and compressed communication with convergence guarantees. Each client must 96 compute K local steps and can be interrupted by the central server at any time. The client updates 97 its model with the (compressed) central version and its current private model. The central server 98 randomly selects s clients and updates the model with the (compressed) received local progress (since 99 last contact) and the previous central model. QuAFL works with old variants of the model at each 100 step, which slows convergence. However, when time, rather than the number of server rounds, is 101 taken into account, QuAFL can provide a speedup because the asynchronous framework does not 102 suffer from delays caused by stragglers. A concurrent and asynchronous approach aggregates local 103 updates before updating the global model: FedBuff (Nguyen et al., 2022) addresses asynchrony 104 using a buffer on the server side. Clients perform local iterations, and the base station updates the 105 global model only after Z different clients have completed and sent their local updates. The gradients 106 computed on the client side may be stale. The main assumption is that the client computations 107 completed at each step come from a uniform distribution across all clients. Fedbuff is asynchronous, 108 but is also sensitive to stragglers (must wait until Z different clients have done all local updates). 109 Similarly, Koloskova et al. (2022) focus on Asynchronous SGD, and provide guarantees depending 110 on some τ max . Similar to Nguyen et al. (2022) the algorithm is also impacted by stragglers, during 111 the transitional regime at least. A recent work by Fraboni et al. (2023) extend the idea of Koloskova 112 et al. (2022) by allowing multiple clients to contribute in one round. But this scheme also favors fast 113 clients. Liu et al. (2021) does not run on buffers, but develops an Adaptive Asynchronous Federated 114 Learning (AAFL) mechanism to deal with speed differences between local devices. Similar to 115 FedBuff, in Liu et al. (2021)'s method, only a certain fraction of the locally updated models contribute 116 to the global model update. Most convergence guarantees for asynchronous distributed methods 117 depend on staleness or gradient delays (Nguyen et al., 2022; Toghani and Uribe, 2022; Koloskova 118 et al., 2022). Only Mishchenko et al. (2022) analyzes the asynchronous stochastic gradient descent 119 (SGD) independently of the delays in the gradients. However, in the heterogeneous (non-IID) setting, 120 convergence is proved up to an additive term that depends on the dissimilarity limit between the 121 gradients of the local and global objective functions. 122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

3 Algorithm

We consider optimization problems in which the components of the objective function (i.e., the data for machine learning problems) are distributed over n clients, i.e.,

where d is the number of parameters (network weights and biases), n is the total number of clients, ℓ is the training loss (e.g., cross-entropy or quadratic loss), NN( x, w ) is the DNN prediction function, p i data is the training distribution on client i . In FL, the distributions p i data are allowed to differ between clients (statistical heterogeneity).

Each client maintains three key values in its local memory: the local model w i , a counter q i , and the value of the initial model with which it started the iterations w i init . The counter q i is incremented for each SGD step the client performs locally until it reaches K , at which point the client stops updating its local model and waits for the server request. Upon the request to the client i , the local model and counter q i are reset. If a server request occurs before the K local steps are completed, the client simply pauses its current training process, reweights its gradient based on the number of local epochs (defined by E i t +1 ), and sends its current reweighted model to the server.

In Zakerinia et al. (2022), we identified the client update w i = 1 s +1 w t -1 + s s +1 w i as a major shortcoming. When the number of sampled clients s is large enough, s s +1 w i dominates the update and basically no server term are taken into consideration. This leads to a significant client drift. As a

*/

;

/*

At

Client

i

*/

15

Initialize

16

Client receives

w

0

and

K

from the Server;

17

Local variables

w

i

=

w

0

, q

i

= 0

;

18

end

19 Loop

20

Run

ClientLocalTraining()

concurrently;

21

When

Contacted by the Server

do

22

Interrupt

ClientLocalTraining()

;

23

Define

α

i

following (3) ;

24

Send

w

i

unbiased

:=

w

i

init

+

1

α

i

(

w

i

-

w

i

init

)

to the server;

25

Receive

w

t

from the server;

26

Update

w

i

init

w

t

, w

i

w

t

, q

i

0

;

27

ClientLocalTraining()

from

zero with updated variables;

28

end

29

end

30

function

ClientLocalTraining()

:

31

while

q

i

< K

do

32

Compute local stochastic gradient

˜

g

i

at

w

i

;

33

Update local model

w

i

w

i

-

η

˜

g

i

;

34

Update local counter

q

i

q

i

+1

;

35

end

36

Wait()

;

37

end function

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

consequence, QuAFL does not perform well in the heterogeneous case (see Section 5). Second, one can note that the updates in QuAFL are biased in favor of fast clients. Indeed each client computes gradients at its own pace and can reach different numbers of epochs while being contacted by the central server. It is assumed that clients compute the same number of local epochs in the analysis from Zakerinia et al. (2022), but it is not the case in practice. As a consequence, we propose FAVAS to deal with asynchronous updates without favoring fast clients. A first improvement is to update local weight directly with the received central model. Details can be found in Algorithm 1. Another idea to tackle gradient unbiasedness is to reweight the contributions from each of the s selected clients: these can be done either by dividing by the (proper) number of locally computed epochs, or by the expected value of locally computed epochs. In practice, we define the reweight α i = E [ E i t +1 ∧ K ] , or α i = P ( E i t +1 > 0)( E i t +1 ∧ K ) , where ∧ stands for min . We assume that the server performs a number of training epochs T ≥ 1 . At each time step t ∈ { 1 , . . . , T } , the server has a model w t . At initialization, the central server transmits identical parameters w 0 to all devices. At each time step t , the central server selects a subset S t of s clients uniformly at random and requests their local models. Then, the requested clients submit their reweighted local models back to the server. When all requested models arrive at the server, the server model is updated based on a simple average (see Line 10). Finally, the server multicasts the updated server model to all clients in S t . In particular, all clients i / ∈ S t continue to run their individual processes without interruption.

158

159

160

161

162

163

164

165

Remark 1. In FAVAS 's setting, we assume that each client i ∈ { 1 , ..., n } keeps a full-precision local model w i . In order to reduce the computational cost induced by the training process, FAVAS can also be implemented with a quantization function Q . First, each client computes backpropagation with respect to its quantized weights Q ( w i ) . That is, the stochastic gradients are unbiased estimates of ∇ f i ( Q ( w i )) . Moreover, the activations computed at forward propagation are quantized. Finally, the stochastic gradient obtained at backpropagation is quantized before the SGD update. In our supplementary experiments, we use the logarithmic unbiased quantization method of Chmiel et al. (2021).

166

167

168

169

170

171

172

173

174

175

182

183

184

185

186

187

4 Analysis

In this section we provide complexity bounds for FAVAS in a smooth nonconvex environment. We introduce an abstraction to model the stochastic optimization process and prove convergence guarantees for FAVAS .

Preliminaries. We abstract the optimization process to simplify the analysis. In the proposed algorithm, each client asynchronously computes its own local updates without taking into account the server time step t . Here in the analysis, we introduce a different, but statistically equivalent setting. At the beginning of each server timestep t , each client maintains a local model w i t -1 . We then assume that all n clients instantaneously compute local steps from SGD. The update in local step q for a client i is given by:

where ˜ g i represents the stochastic gradient that client i computes for the function f i . We also define 176 n independent random variables E 1 t , . . . , E n t in N . Each random variable E i t models the number of 177 local steps the client i could take before receiving the server request. We then introduce the following 178 random variable: ˜ h i t = ∑ E i t q =1 ˜ h i t,q . Compared to Zakerinia et al. (2022), we do not assume that 179 clients performed the same number of local epochs. Instead, we reweight the sum of the gradients by 180 weights α i , which can be either stochastic or deterministic : 181

And we can define the unbiased gradient estimator: ˇ h i t = 1 α i ∑ E i t ∧ K q =1 ˜ h i t,q .

Finally, a subset S t of s clients is chosen uniformly at random. This subset corresponds to the clients that send their models to the server at time step t . In the current notation, each client i ∈ S t sends the value w i t -1 -η ˇ h i t to the server. We emphasise that in our abstraction, all clients compute E i t local updates. However, only the clients in S t send their updates to the server, and each client i ∈ S t sends only the K first updates. As a result, we introduce the following update equations:

Assumptions and notations. 188

A1. Uniform Lower Bound: There exists f ∗ ∈ R such that f ( x ) ≥ f ∗ for all x ∈ R d . 189

A2. Smooth Gradients: For any client i , the gradient ∇ f i ( x ) is L -Lipschitz continuous for some 190 L > 0 , i.e. for all x, y ∈ R d : ∥∇ f i ( x ) -∇ f i ( y ) ∥ ≤ L ∥ x -y ∥ . 191

A3. Bounded Variance: For any client i , the variance of the stochastic gradients is bounded by some 192 σ 2 > 0 , i.e. for all x ∈ R d : E [ ∥ ∥ ˜ g i ( x ) -∇ f i ( x ) ∥ ∥ 2 ] ≤ σ 2 . 193

194

195

A4. Bounded Gradient Dissimilarity: There exist constants G 2 ≥ 0 and B 2 ≥ 1 , such that for all x ∈ R d : ∑ n i =1 ∥∇ f i ( x ) ∥ 2 n ≤ G 2 + B 2 ∥∇ f ( x ) ∥ 2 .

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

We define the notations required for the analysis. Consider a time step t , a client i , and a local step q . We define

the average over all node models in the system at a given time t . The first step of the proof is to compute a preliminary upper bound on the divergence between the local models and their average.

For this purpose, we introduce the Lyapunov function: Φ t = ∥ w t -µ t ∥ 2 + ∑ n i =1 ∥ ∥ w i t -µ t ∥ ∥ 2 .

Upper bounding the expected change in potential. A key result from our analysis is to upper bound the change (in expectation) of the aforementioned potential function Φ t :

Lemma 2. For any time step t > 0 we have:

The intuition behind Lemma 2 is that the potential function Φ t remains concentrated around its mean, apart from deviations induced by the local gradient steps. The full analysis involves many steps and we refer the reader to Appendix B for complete proofs. In particular, Lemmas 16 and 18 allow us to examine the scalar product between the expected node progress ∑ n i =1 ˇ h i t and the true gradient evaluated on the mean model ∇ f ( µ t ) . The next theorem allows us to compute an upper-bound on the averaged norm-squared of the gradient, a standard quantity studied in nonconvex stochastic optimization.

Convergence results. The following statement shows that FAVAS algorithm converges towards a first-order stationary point, as T the number of global epochs grows.

Theorem 3. Assume A 1 to A 4 and assume that the learning rate η satisfies η ≤ 1 20 B 2 bKLs . Then FAVAS converges at rate:

with

F

:= (

f

(

µ

0

)

-

f

∗

)

, and

Note that the previous convergence result refers to the average model µ t . In practice, this does not pose much of a problem. After training is complete, the server can ask each client to submit its final model. It should be noted that each client communicates sT n times with the server during training. Therefore, an additional round of data exchange represents only a small increase in the total amount of data transmitted.

The bound in Theorem 3 contains 3 terms. The first term is standard for a general non-convex target and expresses how initialization affects convergence. The second and third terms depend on the statistical heterogeneity of the client distributions and the fluctuation of the minibatch gradients. Table 1 compares complexity bounds along with synchronous and asynchronous methods.One can note the importance of the ratio s n . Compared to Nguyen et al. (2022) or Koloskova et al. (2022), FAVAS can potentially suffer from delayed updates when s n ≪ 1 , but FAVAS does not favor fast clients at all. In practice, it is not a major shortcoming, and FAVAS is more robust to fast/slow clients distribution than FedBuff/AsyncSGD (see Figure 2). We emphasize both FedBuff and AsyncSGD rely on strong assumptions: neither the queuing process, nor the transitional regime are taken into account

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

in their analysis. In practice, during the first iterations, only fast clients contribute. It induces a serious bias. Our experiments indicate that a huge amount of server iterations has to be accomplished to reach the stationary regime. Still, under this regime, slow clients are contributing with delayed information. Nguyen et al. (2022); Koloskova et al. (2022) propose to uniformly bound this delay by some quantity τ max . We keep this notation while reporting complexity bounds in Table 1, but argue nothing guarantee τ max is properly defined (i.e. finite). All analyses except that of Mishchenko et al. (2022) show that the number of updates required to achieve accuracy grows linearly with τ max , which can be very adverse. Specifically, suppose we have two parallel workers - a fast machine that takes only 1 unit of time to compute a stochastic gradient, and a slow machine that takes 1000 units of time. If we use these two machines to implement FedBuff/AsyncSGD, the gradient delay of the slow machine will be one thousand, because in the 1 unit of time we wait for the slow machine, the fast machine will produce one thousand updates. As a result, the analysis based on τ max deteriorates by a factor of 1000 .

In the literature, guarantees are most often expressed as a function of server steps. In the asynchronous case, this is inappropriate because a single step can take very different amounts of time depending on the method. For example, with FedAvg or Scaffold (Karimireddy et al., 2020), one must wait for the slowest client for each individual server step. Therefore, we introduce in Table 1 constants C _ that depend on the client speed and represent the unit of time to wait between two consecutive server steps. Finally, optimizing the value of the learning rate η with Lemma 12 yields the following:

Corollary 4. Assume A 1 to A 4. We can optimize the learning rate by Lemma 12 and FAVAS reaches an ϵ precision for a number of server steps T greater than (up to numerical constants):

where F = ( f ( µ 0 ) -f ∗ ) , and ( a i , b ) are defined in Theorem 3.

The second term in Corollary 4 is better than the one from the QuAFL analysis ( n 3 of Zakerinia et al., 2022). Although this ( n +1) term can be suboptimal, note that it is only present at second order from ϵ and therefore becomes negligible when ϵ goes to 0 (Lu and De Sa, 2020; Zakerinia et al., 2022).

Remark 5. Our analysis can be extended to the case of quantized neural networks. The derived complexity bounds also hold for the case when the quantization function Q is biased. We make only a weak assumption about Q (we assume that there is a constant r d such that for any x ∈ R d ∥ Q ( x ) -x ∥ 2 ≤ r d ), which holds for standard quantization methods such as stochastic rounding and deterministic rounding. The only effect of quantization would be increased variance in the stochastic gradients. We need to add to the upper bound given in Theorem 3 an "error floor" of 12 L 2 r d , which remains independent of the number of server epochs. For stochastic or deterministic rounding, r d = Θ( d 1 2 2 b ) , where b is the number of bits used. The error bound is the cost of using quantization as part of the optimization algorithm. Previous works with quantized models also include error bounds (Li et al., 2017; Li and Sa, 2019).

5 Numerical Results

We test FAVAS on three image classification tasks: MNIST (Deng, 2012), CIFAR-10 (Krizhevsky et al., 2009), and TinyImageNet (Le and Yang, 2015). For the MNIST and CIFAR-10 datasets, two training sets are considered: an IID and a non-IIID split. In the first case, the training images are randomly distributed among the n clients. In the second case, each client takes two classes (out of the ten possible) without replacement. This process leads to heterogeneity among the clients.

The standard evaluation measure for FL is the number of server rounds of communication to achieve target accuracy. However, the time spent between two consecutive server steps can be very different for asynchronous and synchronous methods. Therefore, we compare different synchronous and asynchronous methods w.r.t. total simulation time (see below). We also measured the loss and accuracy of the model in terms of server steps and total local client steps (see Appendix C.3). In all experiments, we track the performance of each algorithm by evaluating the server model against an unseen validation dataset. We present the test accuracy and variance, defined as ∑ n i =1 ∥ w i t -w t ∥ 2 .

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

We decide to focus on non-uniform timing experiments as in Nguyen et al. (2022), and we base our simulation environment on QuAFL's code 1 . After simulating n clients, we randomly group them into fast or slow nodes. We assume that at each time step t (for the central server), a set of s clients is randomly selected without replacement. We assume that the clients have different computational speeds, and refer to Appendix C.2 for more details. We assume that only one-third of the clients are slow, unless otherwise noted. We compare FAVAS with the classic synchronous approach FedAvg (McMahan et al., 2017) and two newer asynchronous metods QuAFL (Zakerinia et al., 2022) and FedBuff (Nguyen et al., 2022). Details on implementing other methods can be found in Appendix C.1.

We use the standard data augmentations and normalizations for all methods. FAVAS is implemented in Pytorch, and experiments are performed on an NVIDIA Tesla-P100 GPU. Standard multiclass cross entropy loss is used for all experiments. All models are fine-tuned with n = 100 clients, K = 20 local epochs, and a batch of size 128 . Following the guidelines of Nguyen et al. (2022), the buffer size in FedBuff is set to Z = 10 . In FedAvg, the total simulated time depends on the maximum number of local steps K and the slowest client runtime, so it is proportional to the number of local steps and the number of global steps. In QuAFL and FAVAS on the other hand, each global step has a predefined duration that depends on the central server clock. Therefore, the global steps have similar durations and the total simulated time is the sum of the durations of the global steps. In FedBuff, a global step requires filling a buffer of size Z . Consequently, both the duration of a global step and the total simulated time depend on Z and on the proportion of slow clients (see Appendix C.2 for a detailed discussion).

We first report the accuracy of a shallow neural network trained on MNIST. The learning rate is set to 0 . 5 and the total simulated time is set to 5000 . We also compare the accuracy of a Resnet20 (He et al., 2016) with the CIFAR-10 dataset (Krizhevsky et al., 2009), which consists of 50000 training images and 10000 test images (in 10 classes). For CIFAR-10, the learning rate is set to 0 . 005 and the total simulation time is set to 10000 . In Figure 1, we show the test accuracy of FAVAS and competing

0

1000

2000

3000

4000

5000

Time

0.2

0.4

0.6

0.8

FedAvg

FedBuff

QuAFL

FAVAS

methods on the MNIST dataset. We find that FAVAS and other asynchronous methods can offer a significant advantage over FedAvg when time is taken into account. However, QuAFL does not appear to be adapted to the non-IID environment. We identified client-side updating as a major shortcoming. While this is not severe when each client optimizes (almost) the same function, the QuAFL mechanism suffers from significant client drift when there is greater heterogeneity between clients. FedBuff is efficient when the number of stragglers is negligible compared to n . However, FedBuff is sensitive to the fraction of slow clients and may get stuck if the majority of clients are classified as slow and a few are classified as fast. In fact, fast clients will mainly feed the buffer, so the central updates will be heavily biased towards fast clients, and little information from slow clients will be considered. Figure 2 illustrates this phenomenon, where one-ninth of the clients are classified as fast. To provide a fair comparison, Table 2 gives the average performance of 10 random experiments with the different methods on the test set.

In Figure 3a, we report accuracy on a non-IID split of the CIFAR-10 dataset. FedBuff and FAVAS both perform better than other approaches, but FedBuff suffers from greater variance. We explain this limitation by the bias FedBuff provides in favor of fast clients. We also tested FAVAS on the TinyImageNet dataset (Le and Yang, 2015) with a ResNet18. TinyImageNet has 200 classes and each

320

321

322

323

324

325

326

327

328

329

330

331

332

0

1000

2000

3000

4000

5000

Time

0.2

0.4

0.6

0.8

FedAvg

FedBuff

QuAFL

FAVAS

0

1000

2000

3000

4000

5000

Time

0

100

200

300

400

FedAvg

FedBuff

QuAFL

FAVAS

0

2000

4000

6000

8000

10000

Time

0.05

0.10

0.15

0.20

0.25

0.30

FedAvg

FedBuff

QuAFL

FAVAS

(a) CIFAR-10 (non-IID)

0

2000

4000

6000

8000

10000

Time

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

FedAvg

QuAFL

FedBuff

FAVAS

(b) TinyImageNet (IID)

class has 500 (RGB) training images, 50 validation images and 50 test images. To train ResNet18, we follow the usual practices for training NNs: we resize the input images to 64 × 64 and then randomly flip them horizontally during training. During testing, we center-crop them to the appropriate size. The learning rate is set to 0 . 1 and the total simulated time is set to 10000 . Figure 3b illustrates the performance of FAVAS in this experimental setup. While the partitioning of the training dataset follows an IID strategy, TinyImageNet provides enough diversity to challenge federated learning algorithms. Figure 3b shows that FAVAS scales much better on large image classification tasks than any of the methods we considered.

Remark 6. We also evaluated the performance of FAVAS with and without quantization. We ran the code 2 from LUQ (Chmiel et al., 2021) and adapted it to our datasets and the FL framework. Even when the weights and activation functions are highly quantized, the results are close to their full precision counterpart (see Figure 7 in Appendix C).

6 Conclusion

We have presented FAVAS the first (centralised) Federated Learning method of federated averaging 333 that accounts for asynchrony in resource-constrained environments. We established complexity 334 bounds under verifiable assumptions with explicit dependence on all relevant constants. Empirical 335 evaluation shows that FAVAS is more efficient than synchronous and asynchronous state-of-the-art 336 mechanisms in standard CNN training benchmarks for image classification. 337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

References

Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Koneˇ cn` y, J., Mazzocchi, S., McMahan, B., et al. (2019). Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems , 1:374-388.

Chen, Y., Ning, Y., Slawski, M., and Rangwala, H. (2020). Asynchronous online federated learning for edge devices with non-iid data. In 2020 IEEE International Conference on Big Data (Big Data) , pages 15-24. IEEE.

Chen, Z., Liao, W., Hua, K., Lu, C., and Yu, W. (2021). Towards asynchronous federated learning for heterogeneous edge-powered internet of things. Digital Communications and Networks , 7(3):317-326.

Chmiel, B., Banner, R., Hoffer, E., Yaacov, H. B., and Soudry, D. (2021). Logarithmic unbiased quantization: Simple 4-bit training in deep learning. arXiv preprint arXiv:2112.10769 .

Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine , 29(6):141-142.

Fraboni, Y., Vidal, R., Kameni, L., and Lorenzi, M. (2023). A general theory for federated optimization with asynchronous and heterogeneous clients updates. Journal of Machine Learning Research , 24(110):1-43.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778.

Horváth, S., Sanjabi, M., Xiao, L., Richtárik, P., and Rabbat, M. (2022). Fedshuffle: Recipes for better use of local work in federated learning. arXiv preprint arXiv:2204.13169 .

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210.

Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. (2020). Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR.

Koloskova, A., Stich, S. U., and Jaggi, M. (2022). Sharper convergence guarantees for asynchronous sgd for distributed and federated learning. arXiv preprint arXiv:2206.08307 .

Koneˇ cn` y, J., McMahan, B., and Ramage, D. (2015). Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575 .

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.

Le, Y. and Yang, X. (2015). Tiny imagenet visual recognition challenge. CS 231N , 7(7):3.

Li, H., De, S., Xu, Z., Studer, C., Samet, H., and Goldstein, T. (2017). Training quantized nets: A deeper understanding. Advances in Neural Information Processing Systems , 30.

Li, Z. and Sa, C. D. (2019). Dimension-free bounds for low-precision training.

Lin, T., Stich, S. U., Patel, K. K., and Jaggi, M. (2019). Don't use large mini-batches, use local sgd. In International Conference on Learning Representations .

Liu, J., Xu, H., Wang, L., Xu, Y., Qian, C., Huang, J., and Huang, H. (2021). Adaptive asynchronous federated learning in resource-constrained edge computing. IEEE Transactions on Mobile Computing .

Lu, Y. and De Sa, C. (2020). Moniqua: Modulo quantized communication in decentralized sgd. In International Conference on Machine Learning , pages 6415-6425. PMLR.

Makarenko, M., Gasanov, E., Islamov, R., Sadiev, A., and Richtarik, P. (2022). Adaptive compression for communication-efficient distributed training. arXiv preprint arXiv:2211.00188 .

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

Mao, Y., Zhao, Z., Yan, G., Liu, Y., Lan, T., Song, L., and Ding, W. (2022). Communication-efficient federated learning with adaptive quantization. ACM Transactions on Intelligent Systems and Technology (TIST) , 13(4):1-26.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Mishchenko, K., Bach, F., Even, M., and Woodworth, B. (2022). Asynchronous sgd beats minibatch sgd under arbitrary delays. arXiv preprint arXiv:2206.07638 .

Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M., Malek, M., and Huba, D. (2022). Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics , pages 3581-3607. PMLR.

Qu, L., Song, S., and Tsui, C.-Y. (2021). Feddq: Communication-efficient federated learning with descending quantization. arXiv preprint arXiv:2110.02291 .

Smith, V., Chiang, C.-K., Sanjabi, M., and Talwalkar, A. S. (2017). Federated multi-task learning. Advances in neural information processing systems , 30.

Toghani, M. T. and Uribe, C. A. (2022). Unbounded gradients in federated leaning with buffered asynchronous aggregation. arXiv preprint arXiv:2210.01161 .

Tyurin, A. and Richtárik, P. (2022). Dasha: Distributed nonconvex optimization with communication compression, optimal oracle complexity, and no client synchronization. arXiv preprint arXiv:2202.01268 .

Wang, J., Charles, Z., Xu, Z., Joshi, G., McMahan, H. B., Al-Shedivat, M., Andrew, G., Avestimehr, S., Daly, K., Data, D., et al. (2021). A field guide to federated optimization. arXiv preprint arXiv:2107.06917 .

Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. (2020). Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623.

Wang, Q., Yang, Q., He, S., Shui, Z., and Chen, J. (2022). Asyncfeded: Asynchronous federated learning with euclidean distance based adaptive weight aggregation. arXiv preprint arXiv:2205.13797 .

Xie, C., Koyejo, S., and Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint arXiv:1903.03934 .

Xu, C., Qu, Y., Xiang, Y., and Gao, L. (2021). Asynchronous federated learning on heterogeneous devices: A survey. arXiv preprint arXiv:2109.04269 .

Zakerinia, H., Talaei, S., Nadiradze, G., and Alistarh, D. (2022). Quafl: Federated averaging can be both asynchronous and communication-efficient. arXiv preprint arXiv:2206.10032 .

Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V . (2018). Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new Asynchronous Federated Learning algorithm, FAVAS. They provide theoretical convergence analysis of the proposed algorithm as well as present experimental results. The main motivation behind their framework is to handle heterogeneities in the computation speed of clients in asynchronous settings. The work’s fundamental contributions are: 1) Compared to the algorithm proposed in QuAFL (Zakerinia et al., 2022), FAVAS weights updates of each contacted client according to their number of local iterates which is allowed to be different across clients. Its asymptotic bound on the total number of clients (n) is improved. 2) Authors present that the proposed algorithm is experimentally better than baselines on several datasets.\n\nSTRENGTHS:\n1. The proposed algorithm seems efficient in terms of the utilization of clients. In conventional FL analysis, clients generally start computations when they are interrupted by the server. Here, clients always do computation (until $K$ local steps) and send the update when they are interrupted.\n\n2. Although the algorithm is similar to QuAFL (Zakerinia et al., 2022), the authors redesigned local updates and local weighting so that aggregation becomes unbiased. I like how they highlighted the differences in Algorithm 1.\n\n3. The mathematical analysis of the convergence seems strong and doesn’t use any unnatural assumptions. The intuition behind the proof abstract is well explained. In particular, I like the explanations in the main paper while the full proof is presented in Appendix.\n\n4. Both in algorithmic and experimental analysis, the authors provide a good comparison with baselines in the literature. The authors successfully present their experimental setup details.\n\nWEAKNESSES:\n1. A discussion on the relation between the number of local iterates and training quality can enrich the study. In the proposed framework, the number of local iterates that slow & fast clients take are random and depend on the server interaction time and clients' computation time. On the contrary, in most baseline methods, the number of local iterates is a parameter that we set and directly affects the training time. For a fair comparison with baselines, some information (maybe the mean and standard deviation statistics) on the number of local iterates during training may be helpful. \n\n2. Lack of novelty: As the authors stated, the proof strategy and the algorithm are inspired by QuAFL (Zakerinia et al., 2022). I suspect the theoretical novelty in this work is above the threshold for the conference requirement.\n\nQUESTIONS:\n1. Line 42: Which question do authors refer to? The sentence may be changed, or the expression may be clarified. \n\n2. Line 45: The term “unbiased aggregation” can be confusing. Multiple factors can create bias in aggregation (staleness bias, client selection bias, bias in the number of local updates across clients, etc.). This can be clarified.\n\n3. Line 50: The authors claim that the algorithm's guarantee does not depend on the maximum delay. However, I think there should still be an assumption about client delays. For example, if all clients are too slow to compute even one local update before server interaction, the algorithm seems to stop. It would be good if the authors could very briefly discuss this (for reference, please see Discussion in Section 3.2 Algorithm Description of QuAFL (Zakerinia et al., 2022).)\n\n4. Line 97: In QuAFL (Zakerinia et al., 2022), I think that each client doesn’t have to compute $K$ local updates. The just send the progress taken since last contacted. For instance, “Specifically, in the analysis we treat the number of local steps as a random variable which takes values in {1, 2, ...K} and define its expected value to be H. Intuitively, a larger H gives better convergence, since this means that nodes are able to perform larger number of local steps.” part can be found in their paper. Can authors clarify what they mean?\n\nAlso, it is mentioned in Lines 142, 179 and 440. Although the authors said “QuAFL assume the same number (K) of local iterates in analysis,” I believe it is not the case in QuAFL (e.g. see appendix A.1 of their paper). They consider the number of local iterates as random variables.\n\n5. Line 119: Authors said, “Only Mishchenko et al. (2022) analyze the asynchronous SGD independently of the delays in the gradients.” I remembered QuAFL convergence doesn’t depend on delays either. Could you explain this?\n\n6. In Table 1, C_ terms seem vague to me. Although it is said to be a “constant” value for the time between two consecutive iterates, I think it is more correct to say “average” round time in the whole training process due to randomness.\n\n7. Lines 216-220: Although it may not be a problem in practice, I couldn’t get the authors’ claim here. The authors stated that an additional round after training would be enough. However, the convergence analysis does not bound $‖\\nabla f(\\mu_T)‖$, the final version of average models. The analysis proves that the average of all $\\mu_t$ s over $t=1,\\dots,T$ has a bounded gradient norm. So, to get a model whose gradient norm is bounded, we would need all $\\mu_1,\\dots,\\mu_T$. Therefore, we would need much more communication to access all $\\mu_1,\\dots,\\mu_T$. Could authors clarify this?\n\n8. Line 227: Authors stated that FAVAS doesn’t favor fast clients compared to Koloskova et al. (2022). In Koloskova et al. (2022), the client selection happens uniformly at random in their Algorithm 2. So, I am not sure why the authors think that it favors fast clients. On the other hand, I have found authors right in the same claim for comparison with Nguyen et al. (2022).\n\n9. Line 458: Is $\\mathbb{E}[Y_q]$ a typo that should be $\\mathbb{E}[Y^q]$?\n\nLIMITATIONS:\nPlease see question 3 in Questions. The authors discussed both theoretical and experimental extends of the study adequately.\n \nThe authors also discussed the “Environmental footprint” of their study. The reviewer thanks them for their responsible attitude towards our environment.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new asynchronous federated learning update scheme to deal with the biasedness of conventional asynchronous updates which favor fast clients. The proposed method involves two main contributions, one is the direct local model update and the other is to reweight the contributions for the selected clients. The authors provide theoretical convergence analysis for the proposed method under stochastic non-convex settings. The numerical experimental results of the proposed method outperform other baselines in most settings.\n\nSTRENGTHS:\n1. The paper is well structured with clear motivations. The theoretical analysis and numerical results are clear to support the intuition. \n \n2. The analysis of the proposed method can get rid of the constraint of bounded gradient and the uniformly bounded delay, which among the analysis of papers studying asynchronous FL, only very few of them do not require both conditions.\n\nWEAKNESSES:\n1. Some key information about the reweighted scheme need some further illustration. \n\n2. In my opinion, one difference between the proposed method and some previous baselines such as FedBuff and FedAsync is that the proposed FAVAS enables different clients to perform unfix steps of local training during two communication rounds, and the reweighted scheme is proposed to overcome the biasedness of unfix local steps. The independent number of local update steps to perform in each round based on the communication status shares a similar idea as [1]. Thus, the idea of FAVAS is similar to combining the QuAFL and [1] into a single framework.\n \n3. One related paper [1] about federated learning could be discussed. \n\n[1] Yang, H., Zhang, X., Khanduri, P., & Liu, J. (2022, June). Anarchic federated learning. In International Conference on Machine Learning (pp. 25331-25363). PMLR.\n\nQUESTIONS:\n1. I am still quite confused about the reweight parameter $\\alpha^i$ and the parameter $E_{t+1}^i$ in Lines 149-150. How does $E_{t+1}^i$ be defined? Is $E_{t+1}^i$ a tunable parameter or a predicted value, as in Line 177 “Each random variable $E_t^i$ models the number”, it seems a predicted one, thus how to predict it.\n \n2. What do the parameters $a^i$ and $b$ in Theorem 3 stand for? It seems they are highly related to the reweighted parameters.\n \n3. Regarding the convergence rate, it is sure that FAVAS obtains a better rate for the non-dominant term than QuAFL. However, can you show a detailed comparison between the rate of QuAFL and FAVAS in Table 1 (especially for the dominant $O(\\epsilon^{-2}$ term)? \n \n4. Could you provide some details about the choosing of reweight parameter $\\alpha^i$ and parameter $E_{t+1}^i$ for the numerical results?\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce a novel asynchronous centralized Federated Learning (FL) algorithm which corrects for clients with varying computational speeds (and delays). Theoretical convergence guarantees are provided as well as strong empirical results. Solves an interesting issue of model degradation when faster clients dominate asynchronous updates.\n\nSTRENGTHS:\n1. Well-written paper with nice tables and clear algorithms.\n2. Strong theoretical backing which align with peer algorithms (without favoring faster clients).\n3. Impressive empirical results which showcase the algorithms importance in realistic (heterogeneous) settings for FL.\n\nWEAKNESSES:\n1. Novelty of paper seems to lie solely in the reweighting of fast/slow clients. This is important but seems to be the only main contribution. Algorithm is extremely close to QuAFL otherwise.\n2. Convergence is theoretically degraded when a large subset of devices are sampled to participate (s/n ≪ 1). This is a cause for concern in realistic applications.\n\nQUESTIONS:\nOne small theoretical piece of feedback. The theory incorporates a Bounded Gradient Dissimilarity (BGD) assumption. This can likely be improved upon (to boost non-iid results) by incorporating a gradient tracking scheme. Algorithms such as BEER [R1] and D2 [R2] (not my papers I swear!) are able to remove non-iid assumptions and empirically improve their performance compared to peer algorithms by incorporating gradient tracking (utilizing consecutive gradient updates). Maybe adding this in can further boost your empirical results!\n\n\nIt seems to me that the main idea is to reweight slow and fast worker models so that they're equal when performing aggregation in the asynchroous setting. General question: is favoring faster clients that big of an issue? Slow clients would be contributing a model that has barely been trained and may drag the consensus model to a bad local optima. Based on your empirical results my intuition is wrong, but it would be nice to gain an explanation why reweighting slow clients is important. \n\n\nAs mentioned in the paper, \"FAVAS can potentially suffer from delayed updates when s/n ≪ 1,\". This could be problematic in many realistic implementations of the algorithm where only a small fraction of devices can partake in each round. Where does this s/n term come from and is it an artifcat of the theory or can it be removed?\n\nWhy does FedAvg have the largest variance? Isn't it a synchronous method? Shouldn't there should be 0 variance?\n\nIs it possible to gain intuition why FedBuff and FAVAS perform well in the non-iid setting? The non-iid setting is very difficult, and I haven't seen many instances where test accuracies are above 80% for CIFAR10.\n\nSmall note: keep colors consistent for the algorithms in the plots.\n\n[R1] Zhao, Haoyu, et al. \"BEER: Fast $O (1/T)$ Rate for Decentralized Nonconvex Optimization with Communication Compression.\" Advances in Neural Information Processing Systems 35 (2022): 31653-31667.\n\n[R2] Tang, Hanlin, et al. \"$D^ 2$: Decentralized training over decentralized data.\" International Conference on Machine Learning. PMLR, 2018.\n\nLIMITATIONS:\nLimitations are not explicitly outlined in a section. I would recommend adding into the appendix at a minimum.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose an asynchronous communication protocol based on federated averaging, called FAVAS. The proposed method builds on top of a previously released method in asynchronous settings by addressing the bias and variance limitations of the previous approach through an unbiased estimator based on local reweighting. The authors make their claims clear with both theoretical and experimental results.\n\nSTRENGTHS:\n(1) The paper is easy to follow, with good structure and motivation.\n \n(2) The paper also provides a thorough related work discussion, convergence analysis, and empirical evaluation.\n\n(3) Theoretical and empirical variance analysis reduction for asynchronous settings through unbiased local model reweighing.\n\nWEAKNESSES:\n(1) Missing background makes it hard to understand essential concepts of the proposed framework.\n\n(2) The work needs additional clarifications in the numerical evaluation and convergence bounds.\n\nQUESTIONS:\nTable 1 introduces the bounds of different methods. However, an extensive discussion of what each bound represents for each method is absent. There are a lot of notations and little semantic interpretation. This needs to be updated. What do the C_ constants represent for each method? Why is this different across methods when it only depends on client speeds, which are environment-specific?\n\nPlease clarify the reweighting factor $\\alpha$; why the expectation is considered is unclear. Do you create a distribution based on the previously completed steps? What is the intuition behind the reweighting? Similarly, in the preliminaries section, what does the $\\tilde{h}^i_{t,q}$ represent? It is hard to follow. Moreover, the motivation for introducing quantization in the proposed framework is not clearly conveyed. It is not evident why quantization is critical; some theoretical numerical results may enhance the contribution of this claim. \n\nCan you elaborate on what the non-uniform timing experiments refer to? More details are needed for this concept. You introduce the global step (for the first time) in line 293. Is this step similar to an aggregator's update? In line 290, you set K=20 and state that K represents local epochs, while in your analysis, you considered K to be the local step. Are the two Ks the same? For the 10 conducted experiments in Table 2 did you set a different seed every time? If yes, please state that. Also, Table 2 shows that while FedBuff and FAVAS lose some performance as we go from more to less fast clients (2/3 fast vs. 1/9 fast), FedAvg and QuAFL seem to improve. Some discussion on this would be interesting. It would also be interesting to conduct some experiments where all clients. \n\n\nWhen comparing the different methods in Figure 2, there is a high variance for the other baselines (FedBuff, QuAFL) at the beginning. Please provide some discussion in the main text regarding this and relate it to the units of time analysis of Table 1. It would be good also to point the reader to your corresponding Figure 6 in the appendix, presenting the variance magnitude for CIFAR-10.\n\nAn analysis of how different K values affect the computation and communication tradeoff is critical. From the current analysis, it is unclear what is a good choice for K.\n\nConsider extending your related work to account for the FedRec asynchronous approach discussed in work[1], which weights local models based on steps staleness each client has performed and for the semi-synchronous protocol where the training workload is assigned based on the time it takes the slowest client to perform an epoch.\n\nComments:\n\nline 108: Fedbuff -> FedBuff\n\nline 139: term are -> is taken\n\nline 137: we identified -> this could potentially reveal authorship\n\nline 24 (Algorithm): equation (3) is never defined\n\n\n[1] Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settings. Dimitris Stripelis, Paul M. Thompson, José Luis Ambite.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel algorithm called FAVAS for asynchronous federated learning with heterogeneous clients, that is the modification of the previous QuaFL algorithm. The paper provides theoretical convergence rate of the proposed algorithm, and experimental evaluations of the effectiveness of their method.\n\nSTRENGTHS:\n- The studied question is important and well posed. \n\n- Experimental results show a large improvement of FAVAS over the previous QuaFL, and some improvement over FedBuff.\n\nWEAKNESSES:\nOverall the paper is hard to read. The major concerns are unclear presentation of the theoretical results, and limited experimental evaluation.\n\n1. In experiments the learning rate was set to some specific parameter 0.5, which is not a fair comparison. For a fair comparison, it is good to tune the learning rate separately for each of the methods. Same as for fair comparison it is good to tune the value of Z too. FedBuff proposed to set Z=10 when the delays are distributed close to the realistic FL system. However in your simulations the delays are distributed differently and thus the other values of Z might be optimal. \n\n2. The convergence rates in Table 1 are very hard to compare between each other. First, there are no estimates of how large $C_{FedAvg}, C_{FedBuff}, C_{AsyncSGD}$. Second, it is unclear how large $\\sum_{i = 1}^n a_i / n $ might be, so it is unclear if the FAVAS leading term of convergence is better or worse than previous rates. \n\n3. The leading term  of convergence of FAVAS seems to be worse than QuAFL, as in QuaFL it is divided by E^2, while in FAVAS it is not divided by that. \n\n4. It is also hard to understand how tight the proposed analysis is. \n\n5. There are many other algorithms for asynchronous FL missing: such as R1, R2, R3. Experimental comparison to these baselines is also missing.\n\n[R1] Gu et. al., Fast Federated Learning in the Presence of Arbitrary Device Unavailability, \n[R2] Yan et al, Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availability. \n[R3] Glasgow et al, Asynchronous Distributed Optimization with Stochastic Delays.\n\nQUESTIONS:\n1. I did not understand the sentence in lines 86-87: “However, when staleness is …. “ \n2. Your algorithm is very similar to FedBuff with concurrency = n. I think your algorithm could also benefit from having concurrency smaller than n, especially when the number of nodes n is very large.\n\n\nMinor: \n1. On line 23 of Algorithm 1, the reference should be to the equation (1) instead of (3). \n2. In Lemma 2, and line 207 there is a typo in the symbol above $h$. \n3. Line 197: w_t^i is not defined.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}