Title: Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability
Venue: NeurIPS 2024
Accepted: yes
Keywords: distributed learning

non-convex optimization

federated learning

fault-tolerance
Abstract: Addressing intermittent client availability is critical for the real-world deployment of federated learning algorithms. Most prior work either overlooks the potential non-stationarity in the dynamics of client unavailability or requires substantial memory/computation overhead. We study federated learning in the presence of heterogeneous and non-stationary client availability, which may occur when the deployment environments are uncertain, or the clients are mobile. The impacts of heterogeneity and non-stationarity on client unavailability can be significant, as we illustrate using FedAvg, the most widely adopted federated learning algorithm. We propose FedAWE, which includes novel algorithmic structures that (i) compensate for missed computations due to unavailability with only $O(1)$ additional memory and computation with respect to standard FedAvg, and (ii) evenly diffuse local updates within the federated learning system through implicit gossiping, despite being agnostic to non-stationary dynamics. We show that FedAWE converges to a stationary point of even non-convex objectives while achieving the desired linear speedup property. We corroborate our analysis with numerical experiments over diversified client unavailability dynamics on real-world data sets.

Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability

Abstract

Addressing intermittent client availability is critical for the real-world deployment of federated learning algorithms. Most prior work either overlooks the potential non-stationarity in the dynamics of client unavailability or requires substantial memory/computation overhead. We study federated learning in the presence of heterogeneous and non-stationary client availability, which may occur when the deployment environments are uncertain, or the clients are mobile. The impacts of heterogeneity and non-stationarity on client unavailability can be significant, as we illustrate using FedAvg , the most widely adopted federated learning algorithm. We propose FedAWE , which includes novel algorithmic structures that (i) compensate for missed computations due to unavailability with only O (1) additional memory and computation with respect to standard FedAvg , and (ii) evenly diffuse local updates within the federated learning system through implicit gossiping, despite being agnostic to non-stationary dynamics. We show that FedAWE converges to a stationary point of even non-convex objectives while achieving the desired linear speedup property. We corroborate our analysis with numerical experiments over diversified client unavailability dynamics on real-world data sets.

1 Introduction

Federated learning is a distributed machine learning approach that enables training global models without disclosing raw local data [32, 21]. It has been adopted in commercial applications such as autonomous vehicles [6, 70, 41], the Internet of things [39], and natural language processing [63, 43].

Heterogeneous data and massive client populations are two of the defining characteristics of crossdevice federated learning systems [32, 21]. Despite intensive efforts [32, 29, 68, 45, 21], several key challenges that arise from the involvement of large-scale client populations are often overlooked in the existing literature [42]. One of the primary hurdles is the issue of client unavailability. Intuitively, more active clients drive the global model to their local optima by overfitting their local data, which biases the training. In addition, the higher the uncertainty in client unavailability, the larger the performance degradation. Concrete examples that confirm these intuitions in the context of FedAvg - the most widely adopted federated learning algorithm - can be found in Section 4. Client unavailability issues can arise from internal factors such as different working schedules and heterogeneous hardware/software constraints. External factors, such as poor network coverage and frequent handovers of base stations due to fast movements, only exacerbate these problems [50, 57, 64, 3, 21]. The intricate interplay of internal and external factors results in the non-stationarity and heterogeneity of client unavailability.

Most prior work either assumes exact knowledge of the clients' available dynamics or requires their dynamics to be benignly stationary [32, 27, 42, 54, 55, 11]. A related line of work studies asynchronous federated learning wherein clients are vulnerable to delays in message transmission and the reported model updates may be stale [59, 38, 49, 25]. The proposed methods therein assume the availability of all clients or uniformly sampled clients, making them inapplicable to our settings. A few recent works [44, 58] study non-stationary dynamics. Ribero et al. [44] consider the settings where the available probabilities follow a homogeneous Markov chain. Xiang et al. [58] require

that clients be capable of continuous local optimization regardless of communication failures. A handful of other works [14, 60] memorize the old gradients of the unavailable clients to compensate for their unavailability. However, the added memory burdens the federated learning system with substantial memory proportional to the product of the number of clients and the model dimension.

Contributions . In this work, we focus on stochastic client unavailability, where client i is available for federated learning model training with probability p t i at any time t . An illustration can be found in Fig. 1. Our contributions are four-fold:

In Section 4, via constructing concrete examples, we demonstrate that both heterogeneity and non-stationarity of p t i will result in bias and thus significant performance degradation of FedAvg .

In Section 5, we propose an algorithm named FedAWE , which features computational and memory efficiency: only O (1) additional computation and memory per client will be used when compared with FedAvg . The design of FedAWE introduces two novel algorithmic structures: adaptive innovation echoing and implicit gossiping . At a high level, these novel algorithmic structures (i) help clients catch up on the missed computation, and (ii) simultaneously enable a balanced information mixture through implicit client-client gossip, which ultimately corrects the remaining bias. Notably, no direct neighbor information exchanges are used, and the client unavailability dynamics remains unknown to all clients and the parameter server.

In Section 6, we show that FedAWE converges to a stationary point of even non-convex global objective and achieves the linear speedup property without conditions on second-order partial derivatives of the loss function in analysis.

In Section 7, we validate our analysis with numerical experiments over diversified client unavailability dynamics on real-world data sets.

2 Related Work

Dynamical client availability. There is a recent surge of efforts to study time-varying client availability [45, 44, 7, 54, 44, 42, 58, 11], which can be roughly classified into two categories depending on whether the parameter server can unilaterally determine the participating clients.

(i) Controllable participation. Earlier research [32, 29] presumes that, in each round, the parameter server could select a small set of clients either uniformly at random or in proportion to the volume of local data held by clients. More recently, Cho et al. [10] design adaptive and non-uniform client sampling to accelerate learning convergence, albeit at the cost of introducing a non-zero residual error. In another work, Cho et al. [8] study the convergence of FedAvg with cyclic client participation. Yet, the set of available clients is sampled uniformly at random per cyclic round and is decided unilaterally by the parameter server. Perazzone et al. [42] consider heterogeneous and time-varying response rates p t i under the assumptions that p t i is known a priori and that the stochastic gradients are bounded in expectation. Furthermore, the dynamics of p t i are determined by the parameter server by solving a stochastic optimization problem. Chen et al. [7] propose a client sampling scheme wherein only the clients with the most 'important' updates communicate back to the parameter server. This sampling method can achieve performance comparable to that of full client participation, provided that p t i is globally known to both the parameter server and the clients. Departing from this line of literature, our setup neither assumes any side information or prior knowledge of the response rates p t i nor assumes that the parameter server has any influence on p t i .

(ii) Uncontrollable participation. There is a handful of work on building resilience against arbitrary client availability [44, 54, 60, 14, 62, 55]. Ribero et al. [44] consider random client availability whose underlying response rates are also heterogeneous and time-varying with unknown dynamics.

However, the underlying dynamics of p t i in [44] are assumed to follow a homogeneous Markov chain. Wang and Ji [54] propose a generalized FedAvg that amplifies parameter updates every P rounds for some carefully tuned P . Despite its elegant and unified analysis and potential to accommodate non-independent unavailability dynamics, to reach a stationary point, p t i needs to satisfy some assumptions to ensure roughly equal availability of all clients over every P round. Sharing a similar spirit, Crawshaw and Liu [11] propose a SCAFFOLD variant that amplifies global parameter and local gradient updates every P round. In spite of its communication efficiency and resilience to data heterogeneity, the rolling average of p t i over every P round is assumed to be the same constant for all clients. Yang et al. [62] analyze a setting where clients participate in the training at their will. Yet, their convergence is shown to be up to a non-zero residual error. The algorithms proposed in [14, 60] share the same idea of using the memorized latest updates from unavailable clients for global aggregation. Despite superior numerical performance, both algorithms demand a substantial amount of additional memory [55]. For non-convex objectives, both [60] and [14] require an absolute bounded inactive period, and share similar technical assumptions such as almost surely bounded stochastic gradients [60] or Lipschitz Hessian [14]. Though bounded inactive periods are relevant for applications wherein the sensors wake up on a periodic schedule, this assumption is not satisfied even for the simple stochastic setting when clients are selected uniformly at random. Wang and Ji consider unknown heterogeneous p i ' in a concurrent work [55]; however, p i 's are assumed to be fixed over time.

Asynchronous federated learning. Another related line of work is asynchronous federated learning. To the best of our knowledge, Xie et al. [59] initialize the study of asynchronous federated learning, wherein the parameter server revises the global model every time it receives an update from a client. Convergence is shown under some technical assumptions such as weakly-convex global objectives, bounded delay, and bounded stochastic gradients. Zakerinia et al. [69] propose QuAFL which is shown to be resilient to computation asynchronicity and quantized communication yet under the bounded and stationary delay assumption. Nguyen et al. [38] propose FedBuff, which uses additional memory to buffer asynchronous aggregation to achieve scalability and privacy. Convergence is shown under bounded gradients and bounded staleness assumptions. In fact, most convergence guarantees in the asynchronous federated learning literature rely on bounded staleness [59, 38, 49, 25], or bounded gradients [59, 38, 25]. Recently, arbitrary delay is considered in the context of distributed SGD with bounded stochastic gradients and (0 , ζ ) -bounded inter-client heterogeneity [33] (see Assumption 4 for the definition). The convergence suffers from a non-zero residual term O ( ζ 2 ) . In contrast, our convergence guarantee is free from non-zero residual terms and does not require gradients to be bounded.

3 Problem Formulation

Afederated learning system consists of a parameter server and m clients that collaboratively minimize

where F i ( x ) ≜ E ξ i ∼D i [ ℓ i ( x ; ξ i )] is the local objective and can be non-convex, D i is the local distribution, ξ i is a stochastic sample that client i has access to, ℓ i is the local loss function, and d is the model dimension.

We use Assumption 1 to capture the uncertain non-stationary dynamics and heterogeneity. Let A t denote the set of active clients, 1 {·} an indicator function, T the number of total training rounds.

Assumption 1. There exists a δ ∈ (0 , 1] such that p t i ≜ E [ 1 { i ∈A t } ] ≥ δ , where the events { i ∈ A t } are independent across clients i and across rounds t ∈ [ T ] .

Assumption 1 subsumes uniform availability [27, 62] and stationary availability considered in [55]. Independent client unavailability is widely adopted by federated learning research [27, 29, 23, 61, 62, 55]. Analyzing non-independent unavailability, together with uncertain and non-stationary dynamics in Assumption 1, is in general challenging. Specifically, the involved entanglement of stochastic gradient and availability statistics fundamentally complicates the theoretical analysis. However, we conjecture that independence and strictly positive probabilities are only necessary for the technical convenience of our analysis. Our experiments in Section 7 suggest that our algorithm offers notable improvement even in the presence of non-independent and occasionally zero-valued probabilities.

Future work will investigate how to provably accommodate correlated or zero-valued probabilities of arbitrary probabilistic trajectories.

4 Heterogeneity and Non-stationarity May Lead to Significant Bias

In this section, we illustrate the impacts of heterogeneity and non-stationarity of client availability under the classic FedAvg . We use two examples to showcase the significant bias incurred.

Example 1 (Heterogeneity) . Suppose that m = 2 and p t i = p i for i ∈ [2] . Let F i ( x ) ≜ ∥ x -u i ∥ 2 2 / 2 , where x, u i ∈ R . The global objective (1) is

with unique minimizer x ⋆ = ( u 1 + u 2 ) / 2 . Let u 1 = 0 and u 2 = 100 . Fig. 2 illustrates how the heterogeneity in p i affects the expected output of FedAvg .

0.0

0.2

0.4

0.6

0.8

1.0

p

1

0.0

0.2

0.4

0.6

0.8

1.0

2

0

10

20

30

40

50

||

xoutput

x

||

p

Example 1 matches [55, Theorem 1], which shows that FedAvg leads to a biased global objective (3) under heterogeneous p i 's, and that (3) may be significantly away from (1) depending on p i 's.

When the probabilistic dynamics of p t i 's is non-stationary, obtaining an exact biased objective similar to (3) in a neat analytical form becomes challenging, if not impossible, due to the unstructured nonstationary dynamics. Fortunately, Example 2 helps us confirm that the complex interplay between p t i 's across rounds and clients will inevitably further degrade the performance of FedAvg algorithm.

Example 2 (Non-stationarity) . In Fig. 3, a total of m = 100 clients perform an image classification task on the SVHN dataset [37] under the FedAvg algorithm, whose local dataset distribution follows Dirichlet (0 . 1) [17]. Clients become available with probability p t i = p · [ γ · sin(0 . 1 π · t )+(1 -γ )] , ∀ i ∈ [ m ] . The hyperparameter details are deferred to Appendix J. Observations can be found in the caption.

=0.1

=0.2

=0.3

=0.5

Non-stationary Degree

68%

70%

73%

75%

78%

80%

82%

Train Acc

p

=0.1

p

=0.3

p

=0.5

p

=1.0

(a) Train accuracy.

=0.1

=0.2

=0.3

=0.5

Non-stationary Degree

68%

70%

73%

75%

78%

80%

82%

Test Acc

p

=0.1

p

=0.3

p

=0.5

p

=1.0

(b) Test accuracy

5 Federated Agile Weight Re-Equalization ( FedAWE )

To minimize (1), one natural idea is to have the entire client population performs the same number of local updates and mixes these updates carefully to ensure they are weighted equally. Unfortunately, when clients are available only intermittently, they will miss some rounds. A naive approach to equalizing the number of local updates is to have clients catch up by performing their missed local computations immediately when they become available. However, this approach requires a daunting

amount of resources and may not be possible due to hardware/software constraints. Formally, recall that A t is the set of available clients at time t . Let τ i ( t ) ≜ { t ′ : t ′ < t and i ∈ A t ′ } denote the most recent (with respect to time t ) round that client i is available. Compared with standard FedAvg , the naive 'catch-up' procedure will consume ( t -τ i ( t ) -1) · s local stochastic gradient descent updates and ( t -τ i ( t ) -1) additional stochastic samples, where s is the number of local updates per round when a client is available in standard FedAvg .

In this work, we target computation-light algorithms that, compared with FedAvg , only take O (1) additional computation without additional stochastic samples. We propose Fed erated A gile W eight ReE qualization ( FedAWE ), which is formally described in Algorithm 1. It involves two novel algorithmic structures: adaptive innovation echoing and implicit gossiping . At a high level, these novel algorithmic structures (i) help clients catch up on the missed computation, and (ii) simultaneously enable a balanced information mixture through implicit client-client gossip, which ultimately corrects the remaining bias.

In Algorithm 1, each client keeps two local variables x i and τ i , along with a few auxiliary variables used in updating x i and τ i . The algorithm inputs are rather standard: total training rounds T , local and global learning rates η l and η g , the number of local updates per round s , and the initial model x 0 . In each round t , similar to FedAvg , an available client i ∈ A t performs s steps of stochastic gradient descent on its local model x t i (lines 5-8), where ∇ ℓ i ( · ; ξ ( t,k ) i ) is the stochastic gradient of sample ξ ( t,k ) i . Next, we describe the two novel algorithmic structures used in FedAWE .

Adaptive innovation echoing. Departing from FedAvg wherein the local estimate x t i is updated as x t † i ← x ( t, 0) i -η g G t i . In FedAWE (lines 10-11), we 'echo' the local innovation G t i by multiplying it by ( t -τ i ( t )) . Intuitively, this simple echoing helps us approximately equalize the number of local improvements, as formally stated in Proposition 1. It says that the total numbers of innovations echoing are the same for all active clients for any given round and allows the unavailable clients to catch up to the missed computations when they become available.

Implicit gossiping. In FedAWE , the parameter server does not send the most recent global model to the active clients at the beginning of a round. Instead, the parameter server aggregates the locally updated models x t † i and sends the new global model x t +1 to all active clients A t

(lines 14-15). By postponing multicasting the shared global model, the active clients in A t implicitly gossip their updated local models with each other through the parameter server [58]. Though the postponed multi-cast brings in staleness, a simple coupling argument shows that the staleness is bounded (Lemma 2). In addition, our empirical results (Table 8 in Appendix J) suggest that there is

no significant slowdown when compared to vanilla FedAvg . Gossip-type algorithms were originally proposed for peer-to-peer networks and are well-known for their agility to communication failures and asynchronous information exchange in achieving average consensus [13, 4, 24, 16, 31, 36]. Intuitively, the clients' local estimates are eventually equally weighted in the final algorithm output. Note that, departing from the standard gossiping protocols therein [24, 46], information exchange in FedAWE does not involve client-client communication. The information mixing matrix under FedAWE is defined in (4), which is doubly stochastic. Let M ( t ) ≜ E [( W ( t ) ) 2 ] , ρ ( t ) ≜ λ 2 ( M ( t ) ) , J = 11 ⊤ /m , and ρ ≜ max t ρ ( t ) , where λ 2 ( · ) denotes the second largest eigenvalue. We next characterize the information mixing error, i.e., consensus error in Lemma 1.

Lemma 1 ([35, 34, 51]) . For any matrix B ∈ R d × m , it holds that E W [ ∥ B ( ∏ t r =1 W ( r ) -J ) ∥ 2 F ] ≤ ρ t ∥ B ∥ 2 F , where the expectation is taken with respect to randomness in W matrices.

6 Convergence Analysis

In this section, we analyze the convergence of FedAWE . All missing proofs and intermediate results are deferred to the Appendix. Details can be found in Table of Contents.

6.1 Assumptions

We start by stating regulatory assumptions that are common in federated learning analysis [27, 52, 23]. Assumption 2. Each local objective function ∇ F i ( x ) is L -Lipschitz, i.e.,

Assumption 3. Stochastic gradients ∇ ℓ i ( x ; ξ ) are unbiased with bounded variance, i.e.,

Assumption 4. The divergence between local and global gradients is bounded for β, ζ ≥ 0 such that

When the local data sets are homogeneous, ∇ F i ( x ) = ∇ F ( x ) holds for any client i ∈ [ m ] , resulting in β = ζ = 0 . Assumption 4 and its variants in Table 1 are often referred to as bounded gradient dissimilarity assumption to account for data heterogeneity across clients. It can be easily checked that our Assumption 4 is more relaxed or equivalent to the variants therein.

6.2 Auxiliary/Imaginary update sequence construction.

Directly analyzing the evolution of x t and x t i is challenging due to the fact that different clients update at different rounds, and that different active clients echo their local innovation G t i (line 9 in Algorithm 1) with different strength ( t -τ i ) . As such, we construct an auxiliary/imaginary update sequence z t i for client i ∈ [ m ] , whose evolution is closely coupled with x t and x t i but is easier to analyze. Note that the auxiliary/imaginary update sequence is never actually computed by clients but acts as a necessary tool in building up the analysis.

Definition 1. The auxiliary sequence { z t i } of client i ∈ [ m ] is defined as

Recall that τ i (0) = -1 . Thus, by definition, z 0 i = x 0 i according to (6). For general t , when client i ∈ A t -1 , we simply have τ i ( t ) = t -1 and thus t -1 -τ i ( t ) = t -1 -( t -1) = 0 . That is, the auxiliary model z t i and the real model x t i are identical whenever the client i becomes available in the previous round.

When i ∈ A t -1 , the iterate of z i is a bit more involved:

where (7 .a ) holds because of Definition 1 and i ∈ A t -1 , (7 .b ) because of line 10 in Algorithm 1, addition and subtraction. (7 .c ) can be expanded by (6). We defer the simplified form of (7) to (18) in Appendix C for a tidy presentation.

When i / ∈ A t -1 , z t i has a simple iterative relation:

At a high level, the sequence z t i approximately mimics the ideal descent evolution at a client as if the client performs local optimizations on its local model x i per round regardless of its availability. Mathematically, the idea is that, if the progress per iteration of the auxiliary sequence z t i is bounded, we can show the convergence of x t i when x t i and z t i are close to each other.

It is worth noting that auxiliary sequences are used in peer-to-peer distributed learning literature [47, 2, 30, 67, 48, 34]. Yet, existing constructions are not applicable to our problem due to (1) the non-convexity of the global objectives, (2) multiple local updates per round, (3) possibly unbounded gradients, and (4) the general form of bounded gradient dissimilarity. Departing from the use of staled stochastic gradients for auxiliary updates therein, we adopt the true gradient ∇ F i ( · ) to avoid the complications from the involved interplay between randomness in stochastic samples and randomness in τ i ( t ) . On the technical front, it follows from Definition 1 that ∥ x t i -z t i ∥ 2 2 ≤ η 2 l η 2 g s 2 ( t -τ i ( t ) -1) 2 ∥∇ F i ( x τ i ( t )+1 i ) ∥ 2 2 , whose bound appears to be quite challenging to derive due to the coupling of different realizations of τ i ( t ) and gradients. As such, we bound the average of ∥ x t i -z t i ∥ 2 across clients and rounds in Proposition 2.

Lemma 2 (Unavailability statistics) . Under Assumption 1 and δ defined therein. It holds for t ≥ 0 that E [ t -τ i ( t )] ≤ 1 /δ and E [ ( t -τ i ( t )) 2 ] ≤ 2 /δ 2 .

Lemma 2 yields an upper bound on the first and second moments of a client i 's unavailable duration despite the unstructured nature of clients' non-stationary and heterogeneous unavailability. In the special case where we have clients available with the same probability δ , the duration simply follows a homogeneous geometric distribution. It can be easily checked that our bounds trivially hold. However, the duration becomes a more challenging non-homogeneous geometric random variable under our non-stationary unavailability dynamics. Lemma 2 can be derived by using a simple coupling argument and by using tools from probability theory [15].

6.3 Main results.

Lemma 3 (Descent Lemma) . Let F t define the sigma algebra generated by randomness up to round t . Suppose Assumptions 2, 3 hold and η l η g ≤ 9 / (100 sL ) , it holds that

The proof of Lemma 3 follows from the standard analysis for non-convex smooth objectives but with non-trivial adaptation to account for adaptive innovation echoing and implicit gossiping . In particular, it highlights two terms unique in our derivation: the approximation error from the auxiliary sequence and the consensus error from the implicit gossiping procedure.

Proposition 2 (Approximation error) . Given Assumptions 2 and 4, it holds that

The proof of Proposition 2 starts from Definition 1. Although in general it is difficult to bound the error, Assumptions 2 and 4 allow us to break down the problem into bounding the averaged gradient norm of ¯ z t and the consensus error over all randomness instead. Next, we analyze the consensus error. Note that although implicit gossiping takes place in Algorithm 1 for x t i , its analysis is technically challenging as discussed before. So, we adopt the auxiliary z t i as an intermediary and apply Young's inequality to bound the actual consensus error. Details will be specified next. Formally, the auxiliary models can be expressed in a compact matrix form as Z ( t ) ≜ [ z t 1 , . . . , z t m ] . Their local parameter innovation matrix ˜ G t is formulated by combing (7) and (8). We refer the interested readers to (19) in Appendix C for the exact formula. Unrolling the recursion, the consensus error can be expanded as

where equality (10 .a ) holds because all clients are initiated at the same weight.

Recall that ρ bounds the expected spectral norm of the information mixing matrix W ( t ) . It is important to have ρ < 1 for an exponential decay of the consensus error (see Lemma 1). We now proceed to present the convergence rates. In the sequel, we assume it holds for η g and η l that

The proof of the consensus error borrows insights from the analysis of the gossip algorithm [35, 53] but with substantial adaptation to accommodate the novel auxiliary formulation and multi-step local updates. Under the learning rate conidtions in (11) and Assumptions 1, 2, 3 and 4, we can show that

It remains to bound the full convergence error of z t i , which is presented in Theorem 1.

Theorem 1 (Convergence error of z t i ) . Suppose that Assumptions 1, 2, 3 and 4 hold. Choose learning rates η l and η g such that the conditions in (11) are met for T ≥ 1 , it holds that

By addition, subtraction, and Young's inequality, (14) and (15) hold under Assumption 2.

Moreover, from (12), (14) and (15), it can be seen that (16) holds.

Combining (12), (13), (14) and (15), we are ready for Corollary 1.

Corollary 1 (Convergence rate of x t i ) . Suppose that Assumptions 1, 2, 3 and 4 hold. Choose learning rates as η l = 1 √ TsL , η g = √ sδm such that the conditions in (11) are met for T ≥ 1 , it holds that

Corollary 1 establishes the full convergence rate for FedAWE algorithm. It can be seen that the first and second terms dominate when T is sufficiently large, which relate to initial suboptimality gap and stochastic gradient noise σ 2 , respectively. The non-stationary client unavailability results in the third term, which relates to gradient divergence ζ 2 and also to σ 2 . The proof of Corollary 1 follows from (15) by plugging in Proposition 2 and Theorem 1. In the special case where k clients participate uniformly at random, we simply have δ max = δ = k/m . Our convergence bound attains the rate of O (1 / √ skT ) . In other words, we achieve the desired linear speedup property with respect to the number of local steps s and the number of active clients k , matching the established literature [61, 54, 65, 66]. The linear speedup property enables a large cross-device federated learning system to take advantage of a massive scale of parallelism. Notice that the consensus error (16) and the convergence rate (17) have the same asymptotic order with respect to the parameters therein. Hence, the consensus error also enjoys the desired linear speedup property when T is sufficiently large.

7 Numerical Experiments

Overview. In this section, we evaluate FedAWE on real-world data sets to corroborate our analysis and compare it with the other state-of-the-art algorithms. The missing specifications and additional results can be found in Appendix J. Specifically, we consider a federated learning system of one parameter server and m = 100 clients, wherein clients become available intermittently. The image classification tasks use CNNs and are based on SVHN [37], CIFAR-10 [26] and CINIC-10 [12] data sets. All of them include 10 classes of images of different categories. To emulate a highly heterogeneous local data distribution, the image class distribution ν i ∼ Dirichlet ( α = 0 . 1) at client i [17, 54, 55].

Non-stationary client unavailability. A total of four unavailable dynamics are evaluated in Table 2, including stationary and non -stationary with staircase, sine and interleaved sine trajectories, with their visualizations available in the same table. The classification tasks become more challenging as the list progresses due to the growing complexity in the non-stationary dynamics. Furthermore, our choices of the non-stationary dynamics are motivated by real-world federated learning participation statistics, for example, sine trajectory [3], and by generalizing the existing participation patterns such as cyclic participation [8, 55]. In particular, the interleaved sine dynamics is more challenging than the vanilla cyclic availability dynamics since clients become available during each active period with probability that is less than 1 and non-stationary simultaneously. Formally, client i 's dynamics is defined as p t i = p i · f i ( t ) , where f i ( t ) is a time-dependent function under non-stationary dynamics but f i ( t ) = 1 when stationary, and p i = ⟨ ν i , ϕ ⟩ . ϕ characterizes the unbalanced contribution of different image classes to the generated probabilities. Each element of [ ϕ ] c is drawn from Uniform (0 , Φ c ) , where a smaller Φ c leads to a less significant contribution of that image class.

Correlating the local data distribution and the probability of client availability is a common practice in the prior literature. For example, Gu et al. in [14] experiment with a formula for p i so that clients that hold images of smaller digits participate less frequently. Wang and Ji in [55] construct p i as an inner product of the clients' local data distribution ν i and an external distribution Φ ′ . It is immediately clear that the coupling of local data distribution ( ν i ∼ Dirichlet ( α = 0 . 1 )) and class contribution ϕ leads to non-independent p i 's. In addition, Assumption 1 will not hold in the case of interleaved sine non-stationary dynamics since p t i 's occasionally reach 0. Although being agnostic to the challenging client unavailability dynamics not covered by our analysis, we observe that FedAWE retains its outperformance. Comparisons will be specified next.

Benchmark algorithms and discussions. We compare FedAWE with six baseline algorithms, including FedAvg over active clients [32], FedAvg over all clients, FedAU [55], F3AST [44], FedAvg with known p t i ' [42], MIFA [14] and FedVARP [20]. The details of the algorithm and the additional results are deferred to Appendix J. It is observed that FedAWE consistently outperforms the algorithms not aided by memory or known statistics. Surprisingly, FedAWE occasionally beats MIFA and FedVARP , which are memory-heavy. We attribute it to reuse of stored gradients from the unavailable clients. Although FedAWE brings in stalenss due to implicit gossiping, our results (Table 8 in Appendix J) indicate that there is no significant slowdown for FedAWE when compared to vanilla FedAvg , where we study the first round to achieve a targeted accuracy by different algorithms. In addition, FedAWE attains competitive or even better performance than FedAvg with known probability, yet unknown to the underlying dynamics in client unavailability.

8 Conclusion

In this paper, we have shown that the impacts of heterogeneous and non-stationary client unavailability can be significant through concrete examples on FedAvg . To address this, we have proposed an algorithm FedAWE , which provably converges by adaptively echoing clients' local improvement and by evenly diffusing local updates through implicit gossiping. Theoretically, it achieves the desired linear speedup property. Experiments have validated the superiority of FedAWE over state-of-the-art algorithms under diversified non-stationary dynamics. Future work will investigate how to extend our analysis to broader unavailability dynamics such as non-independent and non-stationary unavailability and how to incorporate our findings into federated learning algorithms of different local optimization methods.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge the support from the National Science Foundation under grants 2106891, 2107062, the National Science Foundation CAREER award under grant 2340482, and the Sony Faculty Innovation Award. The research was sponsored by the Army Research Laboratory under Cooperative Agreement Number W911NF-23-2-0014. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory, the National Science Foundation, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. We also thank Connor J. McLaughlin for valuable discussions and feedback on this work.

References

Youssef Allouah, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafa¨ el Pinot, and John Stephan. Fixing by mixing: A recipe for optimal byzantine ml under heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 1232-1300. PMLR, 2023.

Dmitrii Avdiukhin and Shiva Kasiviswanathan. Federated learning under arbitrary communication patterns. In International Conference on Machine Learning , pages 425-435. PMLR, 2021.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇ cn` y, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems , 1:374-388, 2019.

Stephen Boyd, Arpita Ghosh, Balaji Prabhakar, and Devavrat Shah. Randomized gossip algorithms. IEEE Transactions on Information Theory , 52(6):2508-2530, 2006.

Dan Busbridge, Jason Ramapuram, Pierre Ablin, Tatiana Likhomanenko, Eeshan Gunesh Dhekane, Xavier Suau Cuadros, and Russell Webb. How to scale your ema. Advances in Neural Information Processing Systems , 36, 2024.

Jin-Hua Chen, Min-Rong Chen, Guo-Qiang Zeng, and Jia-Si Weng. Bdfl: a byzantine-faulttolerance decentralized federated learning method for autonomous vehicle. IEEE Transactions on Vehicular Technology , 70(9):8639-8652, 2021.

Wenlin Chen, Samuel Horv´ ath, and Peter Richt´ arik. Optimal client sampling for federated learning. Transactions on Machine Learning Research , 2022.

Yae Jee Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, and Tong Zhang. On the convergence of federated averaging with cyclic client participation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 5677-5721. PMLR, 23-29 Jul 2023.

Yae Jee Cho, Jianyu Wang, Tarun Chirvolu, and Gauri Joshi. Communication-efficient and model-heterogeneous personalized federated learning via clustered knowledge transfer. IEEE Journal of Selected Topics in Signal Processing , 2023.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 10351-10375. PMLR, 2022.

Michael Crawshaw and Mingrui Liu. Federated learning under periodic client participation and heterogeneous data: A new communication-efficient algorithm and analysis. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Morris H DeGroot. Reaching a consensus. Journal of the American Statistical association , 69(345):118-121, 1974.

Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems , 34:12052-12064, 2021.

Allan Gut. Probability: a graduate course , volume 200. Springer, 2006.

John Hajnal and MS Bartlett. Weak ergodicity in non-homogeneous markov chains. In Mathematical Proceedings of the Cambridge Philosophical Society , volume 54, pages 233-246. Cambridge Univ Press, 1958.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Xinmeng Huang, Yiming Chen, Wotao Yin, and Kun Yuan. Lower bounds and nearly optimal algorithms in distributed learning with communication compression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Mark Jerrum and Alistair Sinclair. Conductance and the rapid mixing property for markov chains: the approximation of permanent resolved. In Proceedings of the Twentieth Annual ACM Symposium on Theory of Computing , pages 235-244, 1988.

Divyansh Jhunjhunwala, Pranay Sharma, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence , pages 906-916. PMLR, 2022.

Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aur´ elien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adri` a Gasc´ on, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecn´ y, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancr` ede Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer ¨ Ozg¨ ur, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tram` er, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. In International Conference on Learning Representations . PMLR, 2022.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

David Kempe, Alin Dobra, and Johannes Gehrke. Gossip-based computation of aggregate information. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings. , pages 482-491. IEEE, 2003.

Anastasiia Koloskova, Sebastian U Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous sgd for distributed and federated learning. Advances in Neural Information Processing Systems , 35:17202-17215, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450, 2020.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smithy. Feddane: A federated newton-type method. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers , pages 1227-1231. IEEE, 2019.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2020.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Nancy A. Lynch. Distributed Algorithms . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1996.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , pages 1273-1282. PMLR, 2017.

Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake E Woodworth. Asynchronous sgd beats minibatch sgd under arbitrary delays. Advances in Neural Information Processing Systems , 35:420-433, 2022.

Angelia Nedi´ c, Alex Olshevsky, and Michael G Rabbat. Network topology and communicationcomputation tradeoffs in decentralized optimization. Proceedings of the IEEE , 106(5):953-976, 2018.

Angelia Nedic, Alex Olshevsky, and Wei Shi. Achieving geometric convergence for distributed optimization over time-varying graphs. SIAM Journal on Optimization , 27(4):2597-2633, 2017.

Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control , 54(1):48-61, 2009.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011.

John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In International Conference on Artificial Intelligence and Statistics , pages 3581-3607. PMLR, 2022.

Thien Duc Nguyen, Samuel Marchal, Markus Miettinen, Hossein Fereidooni, N Asokan, and Ahmad-Reza Sadeghi. D¨ ıot: A federated self-learning anomaly detection system for iot. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) , pages 756-767. IEEE, 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , 32, 2019.

Muzi Peng, Jiangwei Wang, Dongjin Song, Fei Miao, and Lili Su. Privacy-preserving and uncertainty-aware federated trajectory prediction for connected autonomous vehicles. In The 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023) . IEEE/RSJ, 2023.

Jake Perazzone, Shiqiang Wang, Mingyue Ji, and Kevin S Chan. Communication-efficient device scheduling for federated learning using stochastic optimization. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 1449-1458. IEEE, 2022.

Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Franc ¸oise Beaufays. Federated learning for emoji prediction in a mobile keyboard. arXiv preprint arXiv:1906.04329 , 2019.

M´ onica Ribero, Haris Vikalo, and Gustavo De Veciana. Federated learning under intermittent client availability and time-varying communication constraints. IEEE Journal of Selected Topics in Signal Processing , 17(1):98-111, 2022.

Yichen Ruan, Xiaoxi Zhang, Shu-Che Liang, and Carlee Joe-Wong. Towards flexible device participation in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 3403-3411. PMLR, 2021.

Devavrat Shah et al. Gossip algorithms. Foundations and Trends® in Networking , 3(1):1-125, 2009.

Artin Spiridonoff, Alex Olshevsky, and Ioannis Ch Paschalidis. Robust asynchronous stochastic gradient-push: Asymptotically optimal and network-independent performance for strongly convex functions. Journal of Machine Learning Research , 21(58), 2020.

Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations , 2019.

Mohammad Taha Toghani and C´ esar A Uribe. Unbounded gradients in federated learning with buffered asynchronous aggregation. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-8. IEEE, 2022.

David Tse and Pramod Viswanath. Fundamentals of wireless communication . Cambridge university press, 2005.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. The Journal of Machine Learning Research , 22(1):9709-9758, 2021.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems , 33:7611-7623, 2020.

Jianyu Wang, Anit Kumar Sahu, Gauri Joshi, and Soummya Kar. Matcha: A matching-based link scheduling strategy to speed up distributed optimization. IEEE Transactions on Signal Processing , 70:5208-5221, 2022.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Shiqiang Wang and Mingyue Ji. A lightweight method for tackling unknown participation statistics in federated averaging. In The Twelfth International Conference on Learning Representations , 2024.

Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE journal on selected areas in communications , 37(6):1205-1221, 2019.

Ming Wen, Chengchang Liu, and Yuedong Xu. Communication efficient distributed newton method over unreliable networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 15832-15840, 2024.

Ming Xiang, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong, and Lili Su. Towards bias correction of fedavg over nonuniform and time-varying communications. In 2023 62nd IEEE Conference on Decision and Control (CDC) , pages 6719-6724, 2023.

Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934 , 2019.

Yikai Yan, Chaoyue Niu, Yucheng Ding, Zhenzhe Zheng, Shaojie Tang, Qinya Li, Fan Wu, Chengfei Lyu, Yanghe Feng, and Guihai Chen. Federated optimization under intermittent client availability. INFORMS Journal on Computing , 2023.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-IID federated learning. In International Conference on Learning Representations , 2021.

Haibo Yang, Xin Zhang, Prashant Khanduri, and Jia Liu. Anarchic federated learning. In International Conference on Machine Learning , pages 25331-25363. PMLR, 2022.

Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Franc ¸oise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903 , 2018.

Hao Ye, Le Liang, and Geoffrey Ye Li. Decentralized federated learning with unreliable communications. IEEE Journal of Selected Topics in Signal Processing , 16(3):487-500, 2022.

Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning , pages 7184-7193. PMLR, 2019.

Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5693-5700, 2019.

Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization , 26(3):1835-1854, 2016.

Xiaotong Yuan and Ping Li. On convergence of fedprox: Local dissimilarity invariant bounds, non-smoothness and beyond. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Hossein Zakerinia, Shayan Talaei, Giorgi Nadiradze, and Dan Alistarh. Communication-efficient federated learning with data and client heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 3448-3456. PMLR, 2024.

Tengchan Zeng, Omid Semiari, Mingzhe Chen, Walid Saad, and Mehdi Bennis. Federated learning on the road autonomous controller design for connected and autonomous vehicles. IEEE Transactions on Wireless Communications , 21(12):10407-10423, 2022.

Appendices

Here, we provide an overview of the Appendix. In particular, the proofs of the main results are presented and backed by supporting lemmas and propositions.

A Limitations

The limitations of our work are two-fold:

The client unavailability dynamics are assumed to be independent and strictly positive across clients and rounds. While deriving guarantees is generally challenging without assuming independence and positivity (see Section 3), it is interesting to explore how to relax the client unavailability dynamics, where the probabilities can potentially have arbitrary trajectories.

Our study focuses on heterogeneous and non-stationary client unavailability in federated learning, which may vary greatly due to its inherent uncontrollable nature. Although we have shown FedAWE provably converges to a stationary point of even non-convex objectives, an interesting yet challenging future direction is to incorporate variance reduction techniques for a more robust update.

B Broader Impacts

Federated learning has become the main trend for distributed learning in recent years and has empowered commercial industries such as autonomous vehicles, the Internet of Things, and natural language processing. Our paper focuses on the practical implementation of federated learning systems in the real world and has significantly advanced the theory and algorithms for federated learning by bringing together insights from statistics, optimization, distributed computing and engineering practices. In addition, our research is important for federated learning systems to expand their outreach to more undesirable deployment environments. We are unaware of any potential negative social impacts of our work.

C Nomenclatures

In this section, we provide the notations and nomenclatures used throughout our proofs for a comprehensive presentation. However, it is worth noting that all notations have been properly introduced before their first use. We next articulate the missing definitions and equation formulas.

Missing definitions and equation formulas.

The iterate of z i when i ∈ A t -1 .

Local parameter innovation ˜ G t of the auxiliary sequence.

where the last equality holds because x t i = x τ i ( t )+1 i and re-grouping.

Decomposition in the Proof of Lemma 6. The local parameter innovation of the auxiliary sequence ˜ G t can be decomposed as ˜ G t ≜ ˜ ∆ t +∆ t + s ∇ F t x . Detailed definitions can be found below.

[ ˜ ∆ t ] i ≜ 1 { i ∈A t } ( t -τ i ( t )) ∑ s -1 r =0 ( ∇ ℓ i ( x ( t,r ) i ; ξ ( t,r ) i ) -∇ F i ( x ( t,r ) i ) ) ; · [∆ t ] i ≜ 1 { i ∈A t } ( t -τ i ( t )) ∑ s -1 r =0 ( ∇ F i ( x ( t,r ) i ) -∇ F i ( x t i ) ) ; · [ ∇ F t x ] i ≜ ∇ F i ( x t i ) .

D Useful Inequalities

For completeness and for ease of exposition, we present some common inequalities that will be frequently used in our proofs.

The followings hold for any a i ∈ R d and any i ∈ [ m ] .

Jensen's inequality.

Young's inequality (a.k.a. Peter-Paul inequality).

Equivalently, we have

Smoothness corollary. Given Assumption 2, it holds that

where ( a ) follows from Cauchy-Schwartz inequality and Assumption 2.

E Descent Lemma (Lemma 3)

In this section, we first present a bound on multi-step local computation. Then, we apply the bound to the analysis of descent lemma.

E.1 Multi-step perturbation

Lemma 5. For s ≥ 1 and under Assumption 2, 3 and η l ≤ 1 / (4 sL ) , we have

Proof of Lemma 5. The proof shares a similar road map to [61, Lemma 2], but the objective is instead to show an upper bound with respect to ∥∇ F i ( x t i ) ∥ 2 2 .

For s ≥ 1 , it holds that

where inequality ( a ) holds because of Jensen's inequality, inequality ( b ) holds because of Assumption 2. It remains to bound E [ ∥ x ( t,r ) i -x t i ∥ 2 | F t ] . In what follows, we use ∇ ℓ ( t,k ) i to denote ∇ ℓ i ( x ( t,k ) i ) and ∇ F ( t,k ) i as ∇ F i ( x ( t,k ) i ) , respectively, for ease of presentation.

where equality ( c ) holds because ∇ ℓ ( t,k ) i is an unbiased estimator of ∇ F ( t,r ) i , inequality ( d ) holds because of Young's inequality, inequality ( e ) holds because of Assumption 2.

By η l ≤ 1 4 sL , it holds that

Unroll the recursion, we have

where inequality ( f ) holds because of (1 + 1 /x ) x < exp(1) , inequality ( g ) holds because of √ 3 exp(1) -1 < 4 . Plug it back into (24), we have the desired result

E.2 Descent lemma

Proof of Lemma 3. By Assumption 2 and inequality (23), we have

The one-round innovation of ¯ z can be rewritten as

where equality ( a ) using the fact that x k i = x t i for all k such that τ i ( t ) + 1 ≤ k ≤ t , and equality ( b ) is obtained by adding and subtracting ∇ ℓ i ( x t i ; ξ ( t,r ) i ) and by the fact that ( 1 { i ∈A t } + 1 { i/ ∈A t } ) = 1 .

Bounding (A) .

Bounding (A . I)

where equality ( a ) holds because of the law of total expectation, equality ( b ) holds because 1 { i ∈A t } is by definition independent of others and Assumption 3.

Bounding (A . II)

where inequality ( c ) holds because of Young's inequality. It follows that

where inequality ( d ) holds because of Lemma 5, the last equality using the fact that x k i = x t i for all k such that τ i ( t ) + 1 ≤ k ≤ t .

Bounding (A . III) .

where inequality ( e ) follows from Young's inequality and Assumption 2. It holds that,

Bounding (A . IV)

where the equality follows from the identity in Appendix D (3). It holds that

Putting (A) together,

Bounding (B) .

Bounding (B . I) Recall that δ max ≜ sup i ∈ [ m ] ,t ∈ [ T ] p t i . It holds that,

where equality ( f ) holds by the law of total expectation and by the independence of event { i ∈ A t } , inequality ( g ) holds because of Assumption 3 and by definition p t i ≤ δ max .

Bounding (B . II) We have,

where the last equality using the fact that x k i = x t i for all k such that τ i ( t ) + 1 ≤ k ≤ t .

Bounding (B . III) . E [ (B . III) ∣ ∣ ∣ F t ] ≤ 2 η 2 l η 2 g s 2 L 3 m ∑ m i =1 ∥ x t i -z t i ∥ 2 2 . Putting (B) together, we get

Now, everything:

F Intermediate Results

In this section, we present the intermediate results that serve as handy tools in building up our proofs afterwards.

F.1 Bounding local and global dissimilarity

Proposition 3. For any t , it holds that

Proof of Proposition 3.

where inequality (a) follows from Assumptions 2 and 4.

F.2 Weight re-equalization (Proposition 1)

Proof of Proposition 1. We show Proposition 1 by induction.

When T = 1 and i ∈ A 0 , we have ∑ 0 t =0 1 { i ∈A t } ( t -τ i ( t )) = 1 { i ∈A 0 } (0 -τ i (0)) = 1 . Therefore, the base case holds.

The induction hypothesis is that ∑ K -1 t =0 1 { i ∈A t } ( t -τ i ( t )) = K holds for i ∈ A K -1 . Next, we focus on K +1 :

Now, we have two cases:

Suppose i ∈ A K -1 , then we simply have τ i ( K ) = K -1 . It follows that Eq . (25) ( a ) = K + 1 , where ( a ) follows from induction hypothesis.

Suppose i / ∈ A K -1 ,

where ( b ) follows because 1 { i ∈A t } = 0 for τ i ( K ) ≤ t ≤ K -1 and induction hypothesis that ∑ τ i ( K ) t =0 1 { i ∈A t } ( t -τ i ( t )) = τ i ( K ) + 1 for i ∈ A τ i ( K ) .

F.3 Unavailable statistics (Lemma 2)

Proof of Lemma 2.

From [15, Section 12, Theorem 12.3 (i)], we know that

where X is a non-negative random variable, and g a non-negative strictly increasing differentiable function. It follows that,

where inequality ( a ) holds because x ≤ n, ∀ x ∈ ( n -1 , n ] , inequality ( b ) holds because CCDF P { X > x } is non-increasing. In particular, for a discrete random variable, we have P { X > n -1 } = P { X ≥ n } .

Therefore,

F.4 Auxiliary sequence construction and properties (Proposition 2)

Proposition 4. For any t ≥ 0 , when i / ∈ A t , it holds that x t +1 i -z t +1 i = η l η g s ( t -τ i ( t + 1)) ∇ F i ( x τ i ( t +1)+1 i ) ; when i ∈ A t , it holds that z t † i = x t † i , z t +1 = x t +1 , and z t +1 i = x t +1 i .

Proof of Proposition 4. The proof is divided into two parts: i / ∈ A t and i ∈ A t ,

When i / ∈ A t . It holds that

where equality (a) follows from Definition 1 for inactive clients.

When i ∈ A t . Note that if z t ++ i = x t ++ i for each i ∈ A t , then by the aggregation rules, we know x t +1 = (1 / |A t | ) ∑ i ∈A t x t ++ i = (1 / |A t | ) ∑ i ∈A t z t ++ i = z t +1 . Then, we know that x t +1 i = z t +1 i , ∀ i ∈ A t . Hence, to show the Proposition, it is sufficient to show z t ++ i = x t ++ i holds for i ∈ A t , which can be shown by induction.

When t = 0 ,

Thus, the base case holds. The induction hypothesis is that z t ++ i = x t ++ i , ∀ i ∈ A t is true for all t ≥ 0 . Now, we focus on t +1 .

where equality (a) follows from the auxiliary updates z i , and equality (b) holds because of the induction hypothesis and the fact that τ i ( t +1) < t +1 and i ∈ A τ i ( t +1) .

Proof of Proposition 2. From Propositions 4, we have

Take expectation over all the randomness

where inequality ( a ) follows because by definition 1 { τ i ( t )= p } is independent of ∥ ∥ ∥ ∇ F i ( x p +1 i ) ∥ ∥ ∥ 2 2 , inequality ( b ) follows because x p +1 i = z p +1 i from Proposition 4.

where inequality ( c ) follows from re-indexing, inequality ( d )

from Lemma 2.

F.5 Consensus error of the auxiliary sequence

Lemma 6 (Consensus error of z t i ) . Assuming that η l ≤ δ/ (20 sL ) , and η l η g ≤ δ (1 -√ ρ ) / (10 sL ( √ ρ +1)) , under Assumption 2, 3 and 4, it holds that

Proof of Lemma 6. When t = 0 , Z 0 = [ z 0 , · · · , z 0 ] , which immediately leads to

For t ≥ 1 , recall that W ( t ) is a doubly stochastic matrix to characterize the information mixture, and ˜ G t , defined in (19), captures the local parameter changes in each round. It can be seen that

Expanding Z , we get

where the last follows from the fact that all clients are initiated at the same weights. Note that ∏ t -1 ℓ = q W ( ℓ ) I = ∏ t -1 ℓ = q W ( ℓ ) and ∏ t -1 ℓ = q W ( ℓ ) J = J . Thus,

where the last equality holds because that Z 0 = [ z 0 , · · · , z 0 ] , which immediately leads to

Let matrix notations ˜ ∆ t , ∆ t and ∇ F t x define as follows:

It follows that

Take expectation with respect to randomness in stochastic gradients, denote by E ξ [ · ] :

where the last inequality holds because E ξ [ ˜ ∆ q ] = 0 . Next, we take expectation over the remaining randomness.

Bounding E [(I)]

̸

where inequality ( a ) holds because of Assumption 3. It remains to bound E [ ∥ ˜ ∆ q ∥ 2 F ] .

Take expectation over the remaining randomness:

Therefore,

Bounding E [(II)]

̸

̸

̸

Next, we bound the second term, choose ϵ = ρ q -p 2 ,

̸

Plugging the upper bound in (28) into (27), we get

where inequality ( b ) follows because that √ ρ t -q ≤ √ ρ for any q ≤ t -1 , and that √ ρ t +1 ≥ 0 . It remains to bound E [ ∥ ∆ q ∥ 2 F ] . Take expectation with respect to randomness in stochastic gradients:

where the inequality holds due to Lemma 5. Next, we take expectation over the remaining randomness and plug back into (29):

where the last inequality holds because of re-index and grouping. Therefore,

Bounding E [(III)] Use a similar trick as in bounding E [(II)] , and we get

so that

Putting them together

Plug in Proposition 2.

where the last inequality holds because η l ≤ δ/ (20 sL ) and η l η g ≤ δ/ (10 sL ) . Next, plug in Proposition 3.

It follows that

which is due to the fact that η l η g ≤ 10 sL ( ρ +1) .

F.6 Spectral norm upper bound (Lemma 4)

Lemma 4 adapts from [58], we present its proof here for completeness.

Proof of Lemma 4. For ease of exposition, in this proof we drop time index t . We first get the explicit expression for E [ W 2 jj ′ | A ̸ = ∅ ] . Suppose that A ̸ = ∅ . We have

̸

̸

When k = j and k = j ′ , we have

In addition, we have W jj W j ′ j = 1 |A| 2 1 { j ∈A} 1 { j ′ ∈A} , and W j ′ j ′ W jj ′ = 1 |A| 2 1 { j ∈A} 1 { j ′ ∈A} . Thus,

̸

For j = j ′ , we have

For j = j ′ , we have

In the special case where A = ∅ , we simply have W = I by the algorithmic clauses. Therefore, E [ W jj ′ | A = ∅ ] ≥ 0 holds for any pair of j, j ′ ∈ [ m ] . It follows, by the law of total expectation and for all j, j ′ ∈ [ m ] , that

̸

For j = j ′ , it holds that

where inequality ( a ) holds because |A| ≤ m ;

For j = j ′ , it holds that

Recall that M = E [ W 2 ] . Next, we show that each element of M is lower bounded.

We note that ρ ( t ) = λ 2 ( M ) , where λ 2 is the second largest eigenvalue of matrix M . A Markov chain with M as the transition matrix is ergodic as the chain is (1) irreducible : M jj ′ ≥ δ 2 m [1 -(1 -c ) m ] > 0 for j, j ′ ∈ [ m ] and (2) aperiodic (it has self-loops). In addition, W matrix is by definition doublystochastic. Hence, M has a uniform stationary distribution π = 1 ⊤ /m. Furthermore, the irreducible Markov chain is reversible since it holds for all the states that π i M ij = π j M ji . The conductance Φ of a reversible Markov chain [19] with a transition matrix M can be bounded by

where ∣ ∣ ¯ S ∣ ∣ = m - |S| ≥ m 2 . From Cheeger's inequality, we know that 1 -λ 2 2 ≤ Φ( M ) ≤ √ 2 (1 -λ 2 ) . Finally, we have

2 8

G Convergence Error of ¯ z t (Theorem 1)

In the sequel, we recall and assume the following learning rate conditions in (11):

Recall that δ max ≜ max i ∈ [ m ] ,t ∈ [ T ] p t i and F ⋆ ≜ min x F ( x ) .

Proof of Theorem 1. Take expectation over all the randomness, plug in Lemma 6 and Proposition 2. By telescoping sum, it holds that

Next, we bound (30), (31) and (32), respectively. First, we show that

where the last inequality follows from Lemma 6.

For (30), we have

where the last inequality holds due to (33). For (31), we similarly have

For (32), we have

where inequality ( a ) holds because of (11), inequality ( b ) holds because of (33).

Putting (30), (31) and (32) together and plugging them back into the telescoping sum, it holds that

where the last inequality holds because of (11).

Combining the above and rearranging the terms, we get

where the last inequality holds because ρ < 1 . In terms of asymptotics, we have

where we use the convention that η g ≥ 1 for ease of presentation.

H Convergence Rate of ¯ x t (Corollary 1)

H.1 Convergence error of Algorithm 1

Corollary 2 (Convergence error of x t i ) . Suppose learning rates conditions in (11) are met for η l and η g , and Assumptions 1, 2, 3 and 4 hold for T ≥ 1 , it holds that

Proof of Corollary 2.

where inequality ( a ) follows from Appendix D 2, inequality ( b ) follows from Assumption 2. Further plug in Proposition 3,

Finally, plug in Lemma 6.

where the last inequality holds because ρ < 1 . In terms of asymptotics, we have

where we use the convention that η g ≥ 1 for ease of presentation.

H.2 Convergence rate of Algorithm 1

Proof of Corollary 1. Choose step-size as η l = 1 √ TsL , η g = √ sδm such that learning rate conditions in (11) are met, it holds that

I Additional Results and Interpretations

I.1 Consensus error of Algorithm 1

Corollary 3 (Consensus error of x t i ) . Suppose learning rates conditions are met in (11) for η l and η g , and Assumptions 1, 2, 3 and 4 hold for T ≥ 1 , it holds that

Proof of Corollary 3.

where inequalities ( a ) and ( b ) follow from Jensen's inequality. Plug in Proposition 2 and take expectation over all the randomness, we get

where the last inequality holds because of learning rate condition in (11). Next, plug in Lemma 6:

Finally, we plug in Theorem 1

where we use the fact that ¯ z 0 = ¯ x 0 and ρ < 1 , and the convention that η g ≥ 1 and L ≥ 1 for ease of presentation.

In terms of asymptotics, we have

I.2 Orders of the asymptotic rates

From Theorem 1, Corollary 2, Corollary 3, it is easy to see from the theorem statements that they are all of the same asymptotic order, i.e.,

In addition, by applying learning rate conditions in (11) to Lemma 6 and Proposition 2, we can also see that

Therefore, we conclude that (12), (14) and (15) hold.

J Numerical Experiments

J.1 Code

The code for reproducing our experiments is available at https://github.com/mingxiang12/ FedAWE .

J.2 Experimental setups

Hardware and Software Setups.

Hardware. The simulations are performed on a private cluster with 64 CPUs, 500 GB RAM and 8 NVIDIA A5000 GPU cards.

Software. We code the experiments based on PyTorch 1.13.1 [40] and Python 3.7.16.

Neural Network and Hyper-parameter Specifications. Ta-

ble 6 specifies details of the structures of the convolutional neural network and training. We initialize CNNs using the Kaiming initialization. The initial local learning rate η 0 and the global learning rate η g are searched, based on the best performance after 500 global rounds, over two grids { 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 , 0 . 0005 } and { 0 . 5 , 1 , 1 . 5 , 5 , 10 , 50 } , respectively. The results are presented in Table 7.

The difference between FedAvg over active clients and FedAvg over all clients is that the latter counts the contributions of unavailable clients as 0 's. We set β = 0 . 001

0

1

2

3

4

5

6

7

8

9

Data (label) class

0

2

4

6

8

10

12

14

16

18

Client Index

for F3AST [44], which is tuned over a grid of { 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 , 0 . 0005 } . In addition, as recommended by [55], we choose K = 50 in FedAU without further specification. We train CNNs on all datasets for 2000 rounds. Fig. 3 adopts the same hyperparameter setups, yet with only 1000 training rounds.

Datasets and Data Heterogeneity.

Datasets. All the datasets we evaluate contain 10 classes of images. Some data enhancement tricks that are standard in training image classifiers are applied during training. Specifically, we apply

random cropping and gradient clipping with a max norm of 0.5 to all dataset trainings. Furthermore, random horizontal flipping is applied to CIFAR-10 and CINIC-10.

One full set of experiments takes about 6 hours on SVHN and CIFAR-10 datasets, while about 10 hours on CINIC-10 dataset.

SVHN [37]. The dataset contains 32 × 32 colored images of 10 different number digits. In total, there are 73257 train images and 26032 test images.

CIFAR-10 [26]. The dataset contains 32 × 32 colored images of 10 different objects. In total, there are 50000 train images and 10000 test images.

CINIC-10[12]. The dataset contains 32 × 32 colored images of 10 different objects. In total, there are 90000 train images and 90000 test images.

Data heterogeneity. Fig. 4 visualizes an example of 20 clients, the size of each circle corresponds to the relative proportion of images from a specific class. The larger the circle, the greater the share of images associated with that particular class. Moreover, α controls the heterogeneity of the data such that a greater α entails a more non-i.i.d. local data distribution and vice versa.

J.3 Non-stationary client unavailability dynamics

Client unavailability dynamics and visualizations. As specified in Section 7, we consider a total of four client unavailable dynamics in the form of p t i = p i · f i ( t ) , where p i = ⟨ ν i , ϕ ⟩ , ν i ∼ Dirichlet ( α ) and ϕ is the distribution to characterize the uneven contributions of each image class. In detail, each element [ ϕ ] c is drawn from a uniform distribution Uniform (0 , Φ c ) . We set Φ c = 1 for the first five image classes and Φ c ′ = 0 . 5 for the remaining five image classes. Fig. 5 plots one resulting p i 's example, wherein p i 's are heterogeneous across clients.

Next, we formally introduce f i ( t ) 's under each dynamic.

Stationary: f i ( t ) ≜ 1 ;

Non-stationary with staircase trajectory:

0.2

0.4

0.6

0.8

1.0

Probability

pi

0

2

4

6

8

10

12

14

16

Count

where P defines a period, t 0 ∈ { 0 , P, 2 P, 3 P, . . . } .

Non-stationary with sine trajectory:

where γ signifies the degree of non-stationary.

Non-stationary with interleaved sine trajectory:

where g i ( t ) ≜ γ sin(2 π/P · t )+(1 -γ ) and δ 0 = 0 . 1 defines a cutting-off lower bound. Specifically, δ 0 cuts off the sine curve and brings in a period of zero-valued probabilities. As different clients have different p i 's, the cut-off points are not synchronized among clients, leading to additional availability heterogeneity.

0

10

20

30

40

50

60

70

80

Global round

t

0

1

p t

i

0.1

0.5

0.9

0

10

20

30

40

50

60

70

0.1

0.5

0.9

(a) Stationary

0

10

20

30

40

50

60

70

80

Global round

t

0

1

p t

i

0.1

0.5

0.9

0

10

20

30

40

50

60

70

0.1

0.5

0.9

(b)

Non

-stationary with staircase trajectory

0

10

20

30

40

50

60

70

80

Global round

t

0

1

p t

i

0.1

0.5

0.9

0

10

20

30

40

50

60

70

0.1

0.5

0.9

(c)

Non

-stationary with sine trajectory

0

10

20

30

40

50

60

70

80

Global round

t

0

1

p t

i

0.1

0.5

0.9

0

10

20

30

40

50

60

70

0.1

0.5

0.9

(d)

Non

-stationary with interleaved sine trajectory

We choose γ = 0 . 3 and P = 20 for all non-stationary dynamics. Next, we visualize the probability trajectories along with sampled client availability in Fig. 6. The plots confirm the intuition that interleaved dynamics is the most difficult one, e.g., no clients are available in the case of 0 . 1 therein.

J.4 Additional results

Staleness studies. Table 8 illustrates the first round to reach a targeted test accuracy under nonstationary client availability with sine trajectory. Specifications can be found in the caption. It can be easily checked that, during the initial stage (the first three quarters), FedAWE slightly lags behind FedAvg over active clients. However, when reaching the final stage (the last quarter), FedAWE attains the target accuracy in a comparable or lower number of rounds to FedAvg over active clients in the evaluations on SVHN and CINIC-10 datasets. The slowdown of FedAWE on CIFAR-10 dataset is worth further investigation. In general, we arrive numerically at the conclusion that the staleness incurred by implicit gossiping in FedAWE is mild.

Training curves. In this part, we show the training curves of FedAvg over active clients, FedAWE and MIFA . In particular, the presented results of FedAWE are after exponential moving

0

250

500

750

1000 1250 1500 1750 2000

10

2

2×10 3

3×10 3

4×10 3

6×10 3

train loss

FedAWE

FedAvg

MIFA

0

250

500

750

1000 1250 1500 1750 2000

0.2

0.4

0.6

0.8

test accuracy

FedAWE

FedAvg

MIFA

(a) Evaluation results on SVHN dataset without exponential moving average

0

250

500

750

1000 1250 1500 1750 2000

10

2

2×10 3

3×10 3

4×10 3

6×10 3

train loss

FedAWE

FedAvg

MIFA

0

250

500

750

1000 1250 1500 1750 2000

0.2

0.4

0.6

0.8

test accuracy

FedAWE

FedAvg

MIFA

(b) Evaluation results on SVHN dataset

0

250

500

750

1000 1250 1500 1750 2000

10

2

4×10 3

6×10 3

train loss

FedAWE

FedAvg

MIFA

0

250

500

750

1000 1250 1500 1750 2000

0.2

0.4

0.6

test accuracy

FedAWE

FedAvg

MIFA

(c) Evaluation results on CIFAR10 dataset

0

250

500

750

1000 1250 1500 1750 2000

10

2

6×10 3

7×10 3

8×10 3

9×10 3

train loss

FedAWE

FedAvg

MIFA

0

250

500

750

1000 1250 1500 1750 2000

0.1

0.2

0.3

0.4

0.5

test accuracy

FedAWE

FedAvg

MIFA

(d) Evaluation results on CINIC10 dataset

average [5] under a parameter 0 . 99 . Note that this is to ease down the noisy progress, and for a neat presentation only, the reported results in the main text and ablation studies are all from raw data. Fig. 7a plots the train loss and test accuracy from raw data. For example, when compared with Fig. 7b, EMA eases down the fluctuations but does not change either the trend or the order of algorithm performance results. All train losses are plotted on a logarithmic scale. The results are consistent with Table 2.

Impact of system-design parameters. In this part, we study the impact of system-design parameter including the degree of non-stationarity γ and data heterogeneity α under non-stationary with sine

trajectory. The results are in Table 9 and Table 10. Overall, FedAWE keeps outperforming the algorithms not assisted by memories or known statistics.

In Table 10, clients' local data becomes more heterogeneous when α increases. We can see a clear increase trend in accuracy. However, FedAWE remains to attain the best accuracies both train and test when compared to the algorithms not aided by memory or known statistics. Moreover, it outperforms MIFA , which consumes a lot of storage space, when α = 0 . 1 and 1 . 0 . The observations confirm the practicality of FedAWE .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have faithfully stated our contributions in both the abstract and introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Appendix A for details.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The regulatory assumptions are stated in Section 6. Due to space limitations, we are unable to present all the missing proofs and intermediate results in the main text. They are deferred to Appendix. Please refer to Table of Contents for details.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed experimental and the hyperparameter setups in Section 7 and Appendix J.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our evaluations are based on open-accessed datasets that are publically available. An official implementation code is provided through a GitHub link.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setting/details are important parts of reproducing our results. We provide the details in Section 7 and Appendix J to the best of our ability.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our results are averaged over multiple random seeds and accompanied by error bars

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please find the software/hardware specifications in Appendix J.2.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The NeurIPS code of ethics is strictly enforced throughout our research.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed broader impacts in Appendix B. We are unaware of any negative impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The existing assets used in this paper has been adequately cited or credited to.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have documented the experiment details in Section 7 and Appendix J.2. In addition, we provide our code with clear details and examples.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies heterogeneous and non-stationary client availability in FL, showing that it can have a significant impact on the performance of FedAvg. As a solution, the work presents the FedAWE algorithm, which compensate the missed computationd due to unavailability with minimal additional overhead w.r.t. FedAvg and is agnostic to non-stationary participation dynamics. The works presents a convergence proof in non-convex setting and numerical experiments to showcase the problem and assess the effectiveness of the proposed solution.\n\nSTRENGTHS:\n1. **Importance of the research problem.** The work tackles an important and often overlook problem, that is heterogeneous and non-stationary client participation patterns. The research problem is particularly important given that in FL clients are edge devices, which are heterogeneous in computing capabilities and so availability patterns.\n2. **Simple, principled and convenient approach.** The proposed approach is simple to implement following the details in algorithm 1, since it involves the calculation of a discout factor on clients' pseudo gradient and postponing the sending of the updated model.\n3. **Good results.** The provided experimental results underscore the effectiveness of the approach in addressing both non-stationary and heterogeneous client availability. A plus is that authors did provide the code.\n\nWEAKNESSES:\nThe biggest weakness of the work is in my opinion the presentation. I found the writing not clear and straightforward enough to make the reader grasp the intuitions behind the technical innovations of the proposed approach. There are too many pointer to the supplementary material, and in most cases it seems matter that needs to be in main paper (e.g. line 131). Also, the implications of the technical lemmas are often poorly discussed (e.g. proposition 1-2, lemma 1-3), leaving the reader confused about the need of that particulare piece of information.\n\nI kindly suggest authors to revise the writing of the manuscript to better highlight the most imporant information to be conveyed, and avoid as much as possible continous pointer to the appendix.\n\nQUESTIONS:\nIn section 7, heterogeneous clients' data distribution are simulated $\\nu_i \\sim \\text{Dirichlet(0.1)}$, as it is common in FL literature. Then the authors simulate the availabilty function of client $i$ as $p_i^t=p_i\\cdot f_i(t)$, where $f_i(t)$ models the non-stationary dynamics and $p_i=\\langle \\nu_i, \\phi\\rangle$, with $\\phi$ caractering the importance of each image class. Why $p_i$ is defined from the image class distribution $\\nu_i$, given that it models the probability of being sampled? I expect the data distribution and the the availability to be uncorrelated, so maybe have $\\mathbf{p} \\sim \\text{Dirichlet}(\\gamma)$, where $\\gamma$ decides how heterogeneous are the sampling probability of each client.\n\nLIMITATIONS:\nLimitations of the approach are not sufficiently discussed in section 8.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address intermittent client availability, the authors study heterogeneous and non-stationary client availability, highlighting the significant impact of such heterogeneity using FedAvg. They propose FedAWE, which (i) compensates for missed computations due to unavailability and (ii) evenly diffuses local updates through implicit gossiping, despite non-stationary dynamics. The authors demonstrate that FedAWE converges to a stationary point for non-convex objectives while achieving the desired linear speedup property, supported by numerical experiments on real-world datasets with varied client unavailability dynamics.\n\nSTRENGTHS:\n1) The authors propose a new algorithm to address the issue of intermittent client availability.\n\n2) The paper is easy to follow, and a large number of cases w.r.t. unavailable dynamics are used for validation.\n\nWEAKNESSES:\n1) The proposed algorithm is intuitive. The weight of the model update $G_i^t$ for each client is modified such that the longer the participation interval rounds, the greater the proportion of model update, which is proportional to $t-\\tau(t)$. However, when the client participation rate is very low (e.g., 1% of clients participate randomly), $t-\\tau(t)$will be expected to be 100, magnified by a factor of 100. This is somewhat unreasonable.\n\n2) The rate of convergence in corollary1, the last term on the right-hand side, apparently dose not show a linear speedup with respect to the number of local steps $s$ and the number of active clients $k$.\n\n3) In the rate of convergence in Corollary 1, it appears that $(1 + L^2)$ combines two variables with different \"physical dimensions\", which suggests that there might be an issue with the proof. For example, the \"physical dimension\" of the learning rate $\\eta$ is $1/L$, and $(1 +\\eta^2 L^2)$ is reasonable, while $(1 + L^2)$ might be incorrect.\n\n4) It is not reasonable to see the performance of \"FedAvg over all clients\" being much lower than \"FedAvg over active clients\" in Table 1.\n\nQUESTIONS:\nCan the authors provide an in-depth analysis on the algorithm's complexity?\n\nLIMITATIONS:\nThe authors have discussed the limitations of algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper primarily focuses on addressing the issue of intermittent client availability in federated learning, where the problem scenario involves heterogeneity in participation and non-stationary client availability.The authors draw on ideas from other federated learning algorithms, including asynchronous federated learning, and introduce two novel algorithmic structures, \"Adaptive Innovation Echoing\" and \"Implicit Gossiping\", to address the issues of heterogeneity and non-stationary client unavailability.The authors demonstrate through experiments that heterogeneity and non-stationarity significantly impact the convergence performance of federated learning algorithms. Their newly proposed algorithm, FedAWE, exhibits the property of linear speedup while also saving substantial memory. The experimental results are comprehensive and show that their algorithm outperforms others, highlighting its advantages.\n\nSTRENGTHS:\nThe author's article combines the ideas of federated learning algorithms including asynchronous federated learning and peer-to-peer networks. Two novel algorithm architectures, \"Adaptive Innovation Echoing\" and \"Implicit Gossiping\", are proposed, which can greatly reduce memory. At the same time, the article is more general for client availability, only needs to satisfy independent and strictly positive across clients and rounds.\n\nWEAKNESSES:\nIn the section \"Heterogeneity and Non-stationarity May Lead to Significant Bias\", the authors only provide experimental evidence to verify that these factors affect the convergence performance of the algorithm. Could authors perhaps reference Theorem 1 and Theorem 2 from the FedAU paper for a theoretical analysis of these two factors? I believe that Theorem 1 from the article \"A Lightweight Method for Tackling Unknown Participation Probabilities in Federated Averaging\" could encompass the authors' experimental results concerning heterogeneity, that is, if the client participation is heterogeneous and the weight selection is not appropriate, it will directly affect the effect of convergence.\n\nQUESTIONS:\nFirstly, do you believe that the conclusions regarding the heterogeneity part of your experiments in the section \"Heterogeneity and Non-stationarity May Lead to Significant Bias\" overlap with Theorem 1 from the paper \"A Lightweight Method for Tackling Unknown Participation Probabilities in Federated Averaging\"? Or do your conclusions offer any particular insights beyond this theorem?\n    Secondly, the paper “FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learning” introduces the FedVARP algorithm, which is similar to the MIFA algorithm but employs a different variance reduction method. Could you also conduct related experiments to compare your algorithm with FedVARP?\n\nLIMITATIONS:\nThe authors' work generalizes the application scenarios for federated learning by relaxing assumptions about client participation. It is hoped that in future research, the authors will incorporate variance reduction and other related techniques within the current algorithm framework to further enhance performance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}