Title: Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: federated learning

nonconvex optimization

cross device

heterogeneous data

periodic participation
Abstract: In federated learning, it is common to assume that clients are always available to participate in training, which may not be feasible with user devices in practice. Recent works analyze federated learning under more realistic participation patterns, such as cyclic client availability or arbitrary participation. However, all such works either require strong assumptions (e.g., all clients participate almost surely within a bounded window), do not achieve linear speedup and reduced communication rounds, or are not applicable in the general non-convex setting. In this work, we focus on nonconvex optimization and consider participation patterns in which the chance of participation over a fixed window of rounds is equal among all clients, which includes cyclic client availability as a special case. Under this setting, we propose a new algorithm, named Amplified SCAFFOLD, and prove that it achieves linear speedup, reduced communication, and resilience to data heterogeneity simultaneously. In particular, for cyclic participation, our algorithm is proved to enjoy $\mathcal{O}(\epsilon^{-2})$ communication rounds to find an $\epsilon$-stationary point in the non-convex stochastic setting. In contrast, the prior work under the same setting requires $\mathcal{O}(\kappa^2 \epsilon^{-4})$ communication rounds, where $\kappa$ denotes the data heterogeneity. Therefore, our algorithm significantly reduces communication rounds due to better dependency in terms of $\epsilon$ and $\kappa$. Our analysis relies on a fine-grained treatment of the nested dependence between client participation and errors in the control variates, which results in tighter guarantees than previous work. We also provide experimental results with (1) synthetic data and (2) real-world data with a large number of clients $(N = 250)$, demonstrating the effectiveness of our algorithm under periodic client participation.

Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis

Abstract

In federated learning, it is common to assume that clients are always available to participate in training, which may not be feasible with user devices in practice. Recent works analyze federated learning under more realistic participation patterns, such as cyclic client availability or arbitrary participation. However, all such works either require strong assumptions (e.g., all clients participate almost surely within a bounded window), do not achieve linear speedup and reduced communication rounds, or are not applicable in the general non-convex setting. In this work, we focus on nonconvex optimization and consider participation patterns in which the chance of participation over a fixed window of rounds is equal among all clients, which includes cyclic client availability as a special case. Under this setting, we propose a new algorithm, named Amplified SCAFFOLD, and prove that it achieves linear speedup, reduced communication, and resilience to data heterogeneity simultaneously. In particular, for cyclic participation, our algorithm is proved to enjoy O ( ϵ -2 ) communication rounds to find an ϵ -stationary point in the non-convex stochastic setting. In contrast, the prior work under the same setting requires O ( κ 2 ϵ -4 ) communication rounds, where κ denotes the data heterogeneity. Therefore, our algorithm significantly reduces communication rounds due to better dependency in terms of ϵ and κ . Our analysis relies on a fine-grained treatment of the nested dependence between client participation and errors in the control variates, which results in tighter guarantees than previous work. We also provide experimental results with (1) synthetic data and (2) real-world data with a large number of clients ( N = 250) , demonstrating the effectiveness of our algorithm under periodic client participation.

1 Introduction

Federated learning (FL) [27, 21, 16, 53] is a distributed learning paradigm that emphasizes client privacy [28, 41, 29], limited communication [19, 25], and data heterogeneity across clients [17, 55]. FL has attracted attention in recent years due to the ability to leverage data and compute from user devices while respecting privacy [49, 9]. For large-scale FL, it is common to limit the number of simultaneously participating devices, and many works do so by assuming that a random subset of clients can be sampled independently at each round [27, 3, 50, 22, 38]. However, this pattern of client participation is not always practical. If clients are user devices like mobile phones, they may not have 24/7 availability due to low battery or bad internet connection [15, 31]. In particular, if client availability is correlated with geographical location (e.g. mobile phones charging at night), then

client availability follows a cyclic pattern [54]. Therefore, it remains an important open question to design federated optimization algorithms with provable efficiency under non-i.i.d client participation.

Several works have investigated optimization in FL under non-i.i.d. client participation [10, 2, 8, 39, 46]. However, to the best of our knowledge, no existing algorithm in a non-i.i.d. participation setting provably exhibits reduced communication cost, linear speedup with respect to the number of clients, and resilience to client data heterogeneity for general non-convex optimization.

In this work, we consider FL under an arbitrary participation framework [39], where client participation during each round is a random variable with potentially unknown distribution. We focus on client participation patterns that are periodic, in the sense that all clients are expected to participate with equal frequency over a window of multiple training rounds.

For this setting, we propose Amplified SCAFFOLD, an optimization algorithm for FL under periodic client participation. Amplified SCAFFOLD utilizes (a) amplified updates across participation periods and (b) control variates computed across entire participation periods, to eliminate the effect of data heterogeneity even under non-i.i.d. participation. We show that Amplified SCAFFOLD exhibits significantly reduced communication cost, linear speedup, and is unaffected by client data heterogeneity. To the best of our knowledge, this is the first result demonstrating reduced communication or resilience to data heterogeneity without assuming i.i.d. participation. The complexity of Amplified SCAFFOLD is compared against baselines in Table 1. For cyclic participation, Amplified SCAFFOLD improves the previous best communication cost from O ( κ 2 ϵ -4 ) to O ( ϵ -2 ) .

The main challenges of achieving these properties are (1) simultaneously handling randomness from stochastic gradients and non-i.i.d. participation; and (2) controlling the error of control variates under non-i.i.d. participation. Previous work in this setting [39] performs an in-expectation analysis, by taking expectation only over randomness from stochastic gradients; this avoids (1) but cannot leverage properties of the participation pattern to reduce communication. We present a tighter analysis that addresses (1) by taking expectation over both client participation and the stochastic gradients throughout the analysis, and carefully treating the trajectory variables which depend on both sources of randomness. We address (2) by recursively bounding the control variate errors, which involves a non-uniform average of non-uniform averages of error terms resulting from non-i.i.d. participation. We show that this nested non-uniform average can be bounded using mild regularity conditions on the participation pattern.

Our contributions are summarized below.

Weintroduce Amplified SCAFFOLD, an optimization algorithm for federated learning under non-i.i.d. client participation. Our convergence analysis demonstrates its computational and communication efficiency: Amplified SCAFFOLD exhibits reduced communication, linear speedup, and is unaffected by data heterogeneity. These guarantees are achieved with a tighter analysis than used in previous work [39], with a fine-grained treatment of the two sources of randomness: client participation and stochastic gradients. In the case of cyclic participation, we reduce the previous best communication cost of O ( κ 2 ϵ -4 ) to O ( ϵ -2 ) .

Experimental results show that Amplified SCAFFOLD converges faster than baselines on both synthetic and real-world problems under realistic non-i.i.d. client participation patterns. We also include an ablation study which demonstrates the robustness of our algorithm to changes in data heterogeneity, the number of participating clients per round, and the number of client groups in cyclic participation.

The paper is outlined as follows. We discuss related work in Section 2, and Section 3 provides a formal specification of the optimization problem. Amplified SCAFFOLD is introduced and theoretically analyzed in Section 4, and we provide experiments in Section 5. We conclude with Section 6.

2 Related Work

Federated Optimization. FedAvg [27] characterizes partial client participation and local updates in each round. FedAvg was analyzed in the full participation setting [35, 37, 50, 51, 43, 42, 18, 12]. Other federated optimization algorithms aim to improve communication efficiency [32, 52] and tackle data heterogeneity [22, 17]. The analysis of FL optimization algorithms typically either assumes full client participation or partial client participation where clients are sampled uniformly randomly [47, 38, 17, 23, 43, 1]. [30] provides lower bounds for distributed stochastic, smooth optimization with intermittent communication and non-convex objectives, both in the full and partial participation settings. They also include algorithms employing variance reduction which match (or closely match) lower bounds in the full and partial participation settings. However, none of the works above are applicable for general participation patterns such as periodic participation.

Client Participation. Cyclic data sampling was considered for stochastic convex optimization in [10], where they propose 'pluralistic" solutions instead of learning a single model for all clients. There is a recent line of work considering various participation patterns, including client selection [11, 4, 33] biased participation [34, 6, 7], independent participation across rounds [17, 22, 23], unbiased participation [36, 13], bounded rounds of unavailability [46, 14, 48], asynchronous participation [24, 2], cyclic participation [8], and arbitrary participation [39, 40]. However, none of these works enjoy linear speedup, reduced communication rounds, and resilience to data heterogeneity under the general setting of non-convex objectives and periodic client participation. Concurrent work [44] considers non-stationary client participation under the condition that for every client and every round, the probability of participation is bounded away from zero; for this setting, they propose an algorithm with linear speedup and dependence on gradient dissimilarity for non-convex, Lipschitz objectives.

See Appendix F for a detailed discussion comparing our results with a small number of closely related baselines.

3 Problem Setup

We consider a federated learning problem with N clients, with the overall objective

where each f i : R d → R is the local objective of one client. We consider the stochastic optimization problem, so that f i ( x ) = E ξ ∼D i [ F ( x ; ξ )] , and the optimization algorithm can access F i ( x ; ξ ) and ∇ F i ( x ; ξ ) for individual values of ξ . We make the following assumptions about the objectives:

Assumption 1. (a) f ( x 0 ) -min x ∈ R d f ( x ) ≤ ∆ . (b) Each f i is L -smooth, i.e., ∥∇ f i ( x ) -∇ f i ( y ) ∥ ≤ L ∥ x -y ∥ for all x , y ∈ R d . (c) The stochastic gradient has variance σ 2 , i.e., E ξ ∼D i [ ∥∇ F i ( x ; ξ ) -∇ f i ( x ) ∥ 2 ] ≤ σ 2 for all x ∈ R d .

Since each f i may be non-convex, we consider the problem of finding an ϵ -stationary point of f , that is, a point x ∈ R d such that ∥∇ f ( x ) ∥ ≤ ϵ .

3.1 Participation Framework

Weconsider a federated learning framework consisting of R rounds. For any round r ∈ { 0 , . . . , R -1 } and client i ∈ [ N ] , the availability of client i at round r is a random variable q i r , following the arbitrary participation framework of [39]. If q i r = 0 , then client i may not participate during round i . For example, under the conventional i.i.d. sampling of clients, at each round r a subset of clients S r ⊂ [ N ] is sampled uniformly without replacement, and the weights are set as q i r = 1 { i ∈S r } S .

For some P ∈ N , let Q r 0 be the filtration generated by { q i r : r 0 ≤ r < r 0 + P, i ∈ [ N ] } , let Q be the filtration generated by Q 0 , . . . , Q R -P , and let G be the filtration generated by { ξ i r,k : 0 ≤ r < R, 0 ≤ k < I, i ∈ [ N ] } , where ξ i r,k is the random sampling of the stochastic gradient of round r , step k , client i . We make the following assumptions about the participation distribution.

Assumption 2. For all r ∈ { 0 , . . . , R -1 } : (a) ∑ N i =1 q i r = 1 and ∑ N i =1 ( q i r ) 2 ≤ ρ 2 . (b) The distribution of { q i r } is unbiased across clients over every window of P rounds, i.e., E Q r 0 [ 1 P ∑ ( m +1) P -1 r = mP q i r ] = 1 /N for every m < R/P and i ∈ [ N ] . (c) Each client has a nonzero probability of being sampled over every window of P rounds, i.e., P Q r 0 ( 1 P ∑ ( m +1) P -1 r = mP q i r > 0) > p sample for every m<R/P and i ∈ [ N ] . (d) Q and G are independent.

For each round, Assumption 2(a) enforces that the participation weights q i r are normalized to sum to 1, and characterizes the spread of participation weights across clients with the constant ρ 2 . Assumption 2(b) requires that the set of rounds can be partitioned into windows of length P within which clients are expected to participate with equal frequency. Lastly, Assumption 2(c) enforces that within each window, for each client the probability of being sampled is nonzero. Conventional i.i.d client sampling satisfies Assumption 2 with ρ = S -1 / 2 , P = 1 , and p sample = S/N .

An important difference from conventional i.i.d. participation is that here, client participation is not necessarily independent across rounds. Accordingly, we emphasize that the expectation and probability in Assumptions 2(b)-(c) are taken only over Q r 0 . Therefore, the mean participation weight in Assumption 2(b) may itself be a random variable if client participation at some rounds is dependent on the outcome of participation in previous rounds. Similarly, the sampling probability in Assumption 2(c) may be a random variable. For the participation patterns considered in the next section, Assumption 2 is satisfied even when client sampling is not independent across rounds.

3.2 Specific Participation Patterns

Regularized Participation We say that client participation is regularized [39] if ¯ q i r 0 = 1 N almost surely for all r 0 and i , where ¯ q i r 0 is defined on Line 18 of Algorithm 1 as the participation of client i averaged over rounds r 0 , . . . , r 0 + P -1 . In this case, Assumption 2 is satisfied with p sample = 1 , while P and ρ 2 are parameters of the participation pattern. Regularized participation is a relatively strong constraint, since every client must participate within each window, which may not be practical. However, it is flexible in that there is no constraint on how clients participate within each window. Regularized participation was also considered for strongly convex objectives [26].

Cyclic Participation Following the CyCP framework [8], N clients are partitioned into ¯ K equally sized subsets, and at round r only clients in group ( r mod ¯ K ) may participate. S clients are sampled without replacement from group ( r mod ¯ K ) , for whom the participation weight is q i r = 1 /S . All other clients are assigned q i r = 0 . Cyclic participation satisfies Assumption 2 with P = ¯ K , ρ = S -1 / 2 , and p sample = S ¯ K/N . Notice that i.i.d. client sampling is the special case where ¯ K = 1 .

Cyclic participation can model a situation where each client group is available at a different time of day. For example, if client devices are mobile phones, then clients are available for participation at night, when phones are charging, likely to have internet connection, and otherwise idle. If devices are spread across the globe, then client groups are naturally formed by time zones. Cyclic participation is less stringent than regularized participation since not all clients are required to participate within each window. FedAvg was analyzed under cyclic participation for PL objectives [8], although this analysis is not applicable in our setting, which uses general non-convex objectives.

Algorithm 1 Amplified SCAFFOLD

4 Algorithm and Analysis

In this section, we present Amplified SCAFFOLD, our algorithm to solve the FL problem described in Section 3. Pseudocode for Amplified SCAFFOLD is shown in Algorithm 1. The main components of the Amplified SCAFFOLD algorithm are (1) amplified updates and (2) long-range control variates.

4.1 Algorithm Overview

To deal with the non-stationarity of client availability, Amplified SCAFFOLD performs amplified updates based on information accumulated over a window of P rounds. In Algorithm 1, the variable u holds a weighted average of local updates to client models, weighted by client participation. Every P rounds, the global model is updated in the direction u scaled by the amplification factor γ . Informally, the direction u includes information from all clients with equal representation, according to Assumption 2(b). Similar amplified updates are used in Amplified FedAvg [39].

Control variates for heterogeneous federated learning were first introduced by SCAFFOLD [17]. However, SCAFFOLD-style control variates are updated every time a client participates, which may not be appropriate under periodic availability. For example, under non-i.i.d. participation control variates for different clients would be updated with different frequencies, so that some clients may have consistently less accurate control variates than others. Informally, this may lead to a bias in which some clients' objective is underweighted relative to others. To deal with this issue, Amplified SCAFFOLD updates control variates based on information accumulated over a window of P rounds, which enforces equal representation of all clients in expectation, according to Assumption 2(b).

Comparison with [17, 39] Although the two algorithmic components of Amplified SCAFFOLD individually appear in previous work [17, 39], we emphasize that our complexity results cannot be achieved by simply combining the analyses of these two works. The analysis of [39] requires ϵ -4 communication cost due to their treatment of the randomness in client participation. Here, we present

a tighter analysis with ϵ -2 cost from a more fine-grained treatment of the two sources of randomness (stochastic gradients and client sampling). See Section 4.4 for more details on our approach.

4.2 Main Results

Let ˆ x = ¯ x mP , where m is sampled uniformly from { 0 , . . . , R/P -1 } , and let r 0 ∈ { 0 , P, . . . , R/P } .

the "non-uniformity" of the client sampling distribution. We also consider a variable Λ i r 0 that depends only on the client sampling distribution and characterizes the sample size from which G i r 0 is computed. See Appendix A.1 for further discussion of these quantities. We consider convergence under the following conditions, which are satisfied by several participation patterns of interest.

Theorem 1. Suppose that Assumptions 1 and 2 hold, and that Equation 1 and Equation 2 hold. If γη ≤ p sample 60 LIP and η ≤ √ p sample 60 LIP , then Algorithm 1 satisfies

Corollary 1. For any ϵ > 0 and I ≥ 1 , there exist choices of γ and η such that E [ ∥∇ f (ˆ x ) ∥ 2 ] ≤ O ( ϵ 2 ) as long as R ≥ O ( ∆ Lρ 2 σ 2 Iϵ 4 + ∆ LP p sample ϵ 2 ) .

The complexity of Amplified SCAFFOLD has several important properties:

Reduced Communication By choosing I = Θ(∆ ρ 2 σ 2 p sample P -1 ϵ -2 ) , Amplified SCAFFOLD has communication complexity R = O ( LPp -1 sample ϵ -2 ) , which improves upon the ϵ -4 complexity of parallel SGD. We are not aware of any existing work that achieves this communication reduction for non-convex federated optimization with periodic participation.

Unaffected by Heterogeneity The iterations RI and the number of communications R are unaffected by heterogeneity, which is not achieved for periodic participation by any existing work [39, 8].

Linear Speedup The number of iterations RI = O (∆ Lρ 2 σ 2 ϵ -4 ) will exhibit linear speedup in the number of clients through the term ρ 2 , depending on the client participation pattern.

4.3 Application to Participation Patterns

The results from Section 4.2 apply under any participation pattern that satisfies Assumption 2, Equation 1, and Equation 2, and below we discuss the participation patterns discussed in Section 3.2. The complexity of Amplified SCAFFOLD for each participation pattern are shown in Table 1, and these results can be obtained by plugging ρ 2 , P , and p sample into Corollary 1, together with a choice of I as described in Section 4.2. The derivations of each result below are given in Appendix B.

Regularized Participation Recall that regularized participation satisfies Assumption 2 with p sample = 1 , and P , ρ 2 are parameters of the participation pattern. Also, under regularized participation, w i r 0 = ¯ q i r 0 = 1 /N almost surely, so that E [ w i r 0 ] = 1 /N ≤ P 2 /N and v i r 0 = 0 . Therefore Equation 1 and Equation 2 are satisfied. Plugging p sample = 1 into Corollary 1 yields

In this setting, our algorithm exhibits reduced communication and resilience to heterogeneity. To our knowledge, the only existing algorithm with theoretical guarantees for non-convex problems under regularized participation is Amplified FedAvg [39]. However, as seen in Table 1, the communication complexity of Amplified FedAvg has order ϵ -4 in terms of ϵ and suffers from a κ 2 dependence.

Cyclic Participation Recall that cyclic participation satisfies Assumption 2 with P = ¯ K , ρ = S -1 / 2 , and p sample = S/N . Also, E [ w i r 0 ] = S/N 2 ≤ P 2 /N and E [( v i r 0 ) 2 Λ i r 0 ] = ρ 2 , so that Equation 1 and Equation 2 are satisfied. Based on the above parameter values, the resulting complexities are

Again, Amplified SCAFFOLD achieves reduced communication, linear speedup, and resilience to heterogeneity. Amplified FedAvg [39] is the only existing algorithm with theoretical guarantees in this setting, but it fails to achieve resilience to heterogeneity or reduce communication cost outside of the trivial case of full participation ( ¯ K = 1 , S = N ). Also, even for the setting of PL-functions, the convergence rate of FedAvg under cyclic participation from [8] does not demonstrate an improvement with respect to the number of local steps. See Appendix F for further discussion of their results.

Recall that i.i.d. participation is a special case of cyclic participation with ¯ K = 1 . In this case, Amplified FedAvg fails to recover the reduced communication usually achieved under i.i.d. participation, such as by SCAFFOLD [17]. In fact, Amplified FedAvg fails to recover the communication cost of FedAvg under i.i.d. participation, requiring an additional factor of LN . The larger communication cost of Amplified FedAvg is a result of its convergence analysis, which does not leverage the property of unbiased participation (Assumption 2(b)) during the analysis, and requires P = O ( ϵ -2 ) in order to converge (see Appendix C for more details). In contrast, Amplified SCAFFOLD succeeds in recovering the results of SCAFFOLD under i.i.d. participation, with only a slightly worse dependence of R on N/S . This difference in the order of N S is due to a potential small issue in the analysis of SCAFFOLD, which we intentionally avoided by accepting a slightly worse dependence on N S . We provide a detailed discussion of the N/S dependence in Appendix F.

4.4 Proof Sketch

The main challenges for demonstrating convergence are (1) simultaneously handling randomness from stochastic gradients and non-i.i.d. client sampling, and (2) controlling error of control variates under non-i.i.d. client sampling. Previous work [39] subverts (1) by conditioning on Q throughout the entire analysis. However, this eliminates the possibility of utilizing the condition E [¯ q i r 0 ] = 1 /N , and ultimately incurs a dependence on the data heterogeneity (see the term ˜ δ 2 ( P ) in Theorem 3.1 of [39]). Instead, we take expectation over both sources of randomness throughout the analysis, which requires a careful treatment of each iterate's dependence, and enables communication reduction. For (2), previous analysis of federated algorithms with control variates [17] recursively bounds the error of control variates between consecutive rounds. However, this recursion crucially depends on i.i.d. client participation. We extend this analysis to our setting, establishing a recursion over the control variate error between consecutive windows of P rounds. Establishing this recursion under non-i.i.d. participation involves a non-uniform average of non-uniform averages of error terms, which we handle by invoking the regularity conditions stated in Equation 1 and Equation 2.

Using smoothness of f , the objective function decrease f (¯ x r 0 + P ) -f (¯ x r 0 ) is upper bounded by ⟨∇ f (¯ x r 0 ) , ¯ x r 0 + P -¯ x r 0 ⟩ + L 2 ∥ ¯ x r 0 + P -¯ x r 0 ∥ 2 . Letting ¯ x r,k = ∑ N i =1 q i r x i r,k be a weighted average of local models, the sum of the previous inner product and quadratic terms can be bounded by -γηIP ∥∇ f (¯ x r 0 ) ∥ 2 , plus standard noise terms, the additional 'drift" terms

and control variate errors C i r 0 = ∥∇ f i (¯ x r 0 ) -G i r 0 ∥ 2 . ˜ D r,k captures the distance between local client models, while ˜ M r,k captures the distance from local models to the previous global model ¯ x r 0 .

Taking conditional expectation D r,k = E [ ˜ D r,k |Q ] and M r,k = E [ ˜ M r,k |Q ] , Lemma 1 bounds the drift terms by establishing and unrolling a mutually recurrent relation between D r,k and M r,k . The resulting bound involves a non-uniform average over the control variate errors: ∑ N i =1 q i r E [ C i r 0 |Q ] .

Denoting the average control variate error C r 0 = 1 N ∑ N i =1 C i r 0 , we want to bound E [ C r 0 + P ] in terms of E [ C r 0 ] . C r 0 + P can be decomposed into drift terms, but the result is a non-uniform average:

Since the bound for each M s,k + D s,k from Lemma 1 involves a non-uniform average over C i r 0 , the resulting bound of C r 0 + P involves a non-uniform average of non-uniform averages of C i r 0 , instead of the uniform average C r 0 . The regularity conditions in Equation 1 and Equation 2 allow us to bound this nested non-uniform average by a uniform average, which finishes the recursion.

Putting everything together, we obtain the descent inequality

where ˜ f r 0 := f (¯ x r 0 + P ) +Φ( r 0 + P ) and Φ is a potential function that depends on the control variate errors. Theorem 1 is then obtained by averaging over r 0 and isolating the gradient.

5 Experiments

We experimentally validate our algorithm for non-i.i.d client participation under three settings: minimizing a synthetic function, logistic regression for Fashion-MNIST 1 [45], and training a CNN for CIFAR-10 [20]. We also include an ablation study on Fashion-MNIST, to investigate how each algorithm is affected by changes in data heterogeneity, the number of participating clients, and the number of client groups in cyclic participation.

5.1 Setup

All of our experiments utilize a non-i.i.d. client participation pattern similar to cyclic participation (discussed in Section 3.2). We partition the total set of N clients into ¯ K equally sized subsets, and at each training round only a single client group is available for participation. In our experiments, the available group does not change every round; instead, each group is available for g rounds at a time. Under this pattern, Assumption 2 is satisfied with P = g ¯ K . We refer to g as the availability time.

We evaluate five algorithms: FedAvg [27], FedProx [22], SCAFFOLD [17], Amplified FedAvg [39], and Amplified SCAFFOLD (ours). We tune each algorithm's parameters by grid search, including learning rate η , amplification rate γ , and FedProx's µ . The search ranges and tuned values can be found in Appendix D. All experiments were run on a single node with eight NVIDIA A6000 GPUs. Code is available at the following repository: https://github.com/ MingruiLiu-ML-Lab/FL-under-Periodic-Participation

Synthetic We evaluate each algorithm's convergence on a difficult objective based on a lower bound for FedAvg [43]. The objective maps R 4 to R , is convex, and is parameterized by a smoothness L , stochastic gradient variance σ 2 , and heterogeneity κ , so that it satisfies Assumption 1 by construction. The complete definition of the objective can be found in Appendix D. Since there are only two distinct local objectives, we set the number of clients N = 2 and the number of sampled clients S = 1 , and the number of groups ¯ K = 2 . All other settings can be found in Appendix D.

Fashion-MNIST and CIFAR-10 We evaluate each algorithm for training an image classifier, using logistic regression for Fashion-MNIST and a two-layer CNN for CIFAR-10. To simulate heterogeneous data in federated learning, we use a common protocol [17, 39], to partition each dataset into client datasets according to a data similarity parameter s . This protocol is detailed in Appendix D. Following [39], we set the number of clients N = 250 , data similarity s = 5% , and the number of sampled clients per round S = 10 . For client participation, we set the number of groups ¯ K = 5 , so that each group contains clients that have majority label from two different classes. We run all baselines with 5 different random seeds and report the mean results with error bars in Section 5.2 (the radius of each error bar is 1 standard deviation). All other settings can be found in Appendix D.

Additional experimental results are provided in Appendix E, where we compare against extra baselines (FedAdam [32], FedYogi [32], FedAvg-M [5], and Amplified FedAvg with FedProx regularization), and evaluate training under another non-i.i.d. client participation pattern.

SyntheticObjective

CIFAR-10

0.5

22222227

wnu

0.8

0.6

Value

0.4

0.007

jective

0.3

Mini

0.6

SS07

0.5

0.2

ance

0.4

Train

0.006

Test

obje

0.4

0.1

Distal

0.2

0.005

0.0

0.0

0.3

0

2000

4000

0

2000

4000

0

5000

10000

0

5000

10000

Rounds

Rounds

Rounds

Rounds

FedAvg

FedProx

SCAFFOLD

AmplifiedFedAvg

Amplified SCAFFOLD

5.2 Main Results

Results for the synthetic experiment and CIFAR-10 are shown in Figure 1, and results for FashionMNIST are shown in Figure 2. We make the following observations:

Amplified SCAFFOLD converges the fastest. In all three settings, Amplified SCAFFOLD reaches the best overall solution among all algorithms (by all metrics) and requires the fewest communication rounds. In the synthetic experiment, Amplified SCAFFOLD requires 800 communication rounds to reach an objective value of 0 . 2 , while SCAFFOLD requires 1900 rounds, and both FedAvg and Amplified FedAvg require 4800 rounds to reach the same objective value.

Amplified FedAvg is comparable to FedAvg. Amplified FedAvg shows slight improvement over FedAvg for the synthetic experiment and for Fashion-MNIST. Only for CIFAR-10 is Amplified FedAvg significantly faster than FedAvg, but there it also exhibits a reduction in stability. The underwhelming experimental performance of Amplified FedAvg corroborates our discussion from Section 4.3; Amplified FedAvg requires many communication rounds and suffers from data heterogeneity.

Contrary to our findings, the original evaluation of Amplified FedAvg [39] showed a significant improvement over FedAvg. One explanation is that the original evaluation employed pretraining using FedAvg, so that each algorithm was evaluated only for fine-tuning. Our experiments suggest that Amplified FedAvg may have limited improvement over FedAvg when training from scratch.

SCAFFOLD beats Amplified FedAvg. Despite a lack of theoretical guarantees under non-i.i.d. participation, SCAFFOLD outperforms Amplified FedAvg in all settings. This suggests that SCAFFOLD may have reasonable performance under some non-i.i.d. participation patterns. For the synthetic objective and CIFAR-10, SCAFFOLD is still significantly slower than Amplified SCAFFOLD.

5.3 Ablation Study

To understand how each algorithm's performance is affected by data heterogeneity, the number of participating clients, and the number of client groups, we perform an ablation study on FashionMNIST. First, we fix the data similarity s = 5% and number of groups ¯ K = 5 while varying the number of participating clients ( S ) over { 5 , 15 , 20 , 25 } . Next, we fix S = 10 , ¯ K = 5 while varying the similarity s over { 2 . 5% , 10% , 33% , 100% } . Lastly, we fix s = 5% , S = 10 while varying the number of client groups ¯ K ∈ { 2 , 4 , 6 , 8 } . In each of these 12 scenarios, we evaluate all five algorithms using the same settings as detailed in Section 3. We train with three random seeds for each algorithm, and report the average results in Figure 2 (right).

Amplified SCAFFOLD reaches the best solution in all settings. Similarly to Section 5.2, Amplified SCAFFOLD consistently reaches the best solution in terms of both training loss and testing accuracy. While our theoretical results provide guarantees for optimization, these experiments show that Amplified SCAFFOLD also exhibits superior generalization in a variety of settings.

Robustness to data heterogeneity. When changing from completely homogeneous data ( s = 100% ) to extremely heterogeneous data ( s = 2 . 5% ), the test accuracy of Amplified SCAFFOLD exhibits a

FashionMNIST

Fashion MNIST(Ablation)

0.0028

0.84

LoSS

0.0025

0.0026

0.82

Train

0.0020

SS07

0.0024

0.80

0.0015

0.0022

0.78

0.85

0.0020

Test

0.76

ACC.

0.0018

0.74

Test

0.80

0.0016

0.72

0.75

0.0014

0.025 0.1 0.33 1.0

5152025

24

68

0

5000

10000

0

5000

10000

Data

Participating

Groups

Rounds

Rounds

Similarity

Clients

FedAvg

FedProx

SCAFFOLD

AmplifiedFedAvg

Amplified SCAFFOLD

very small decrease from 84 . 6% to 84 . 45% , so that our algorithm behaves nearly identically with homogeneous data as with extremely heterogeneous data. All baselines suffer a larger decrease when transitioning from homogeneous data to heterogeneous data.

Robustness to number of participating clients. The number of participating clients has a smaller effect on performance than data heterogeneity, but some degradation happens in the extreme case S = 5 . In particular, SCAFFOLD has competitive performance with large S ≥ 15 , but its test accuracy drops off significantly compared to Amplified SCAFFOLD in the case S = 5 .

Robustness to number of client groups. As ¯ K increases, FedAvg and Amplified FedAvg get worse, while SCAFFOLD and Amplified SCAFFOLD maintain performance. It makes intuitive sense for an algorithm to degrade as ¯ K increases, since a larger ¯ K means that the participation is in some sense 'further" from i.i.d. participation. Still, Amplified SCAFFOLD (and SCAFFOLD) are able to maintain performance even as ¯ K increases. While the worst-case communication complexity of Amplified SCAFFOLD (listed in Table 1) actually increases with ¯ K , these experiments demonstrate that in practice, Amplified SCAFFOLD can maintain performance as ¯ K increases.

6 Conclusion

We propose Amplified SCAFFOLD, an optimization algorithm for federated learning under periodic client participation, and prove that it exhibits reduced communication cost, linear speedup, and is unaffected by data heterogeneity. We also show that Amplified SCAFFOLD experimentally outperforms baselines on standard benchmarks under non-i.i.d. client participation, and that the performance of our algorithm is robust to changes in data heterogeneity and the number of participating clients.

Limitations While our analysis covers a general class of participation patterns, it may not cover some participation patterns that appear in practice. Our framework requires that all clients have an equal chance of participation across well-defined windows of time that are known to the algorithm implementer, which may not always hold. One such practical situation is where clients may freely join or leave the federated learning process during training. Extending our algorithm and guarantees for this situation would require a reformulation of the optimization problem, and possibly additional assumptions about the participation structure. We leave such analysis for future work.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations , 2021.

Dmitrii Avdiukhin and Shiva Kasiviswanathan. Federated learning under arbitrary communication patterns. In International Conference on Machine Learning , pages 425-435. PMLR, 2021.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇ cn` y, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of machine learning and systems , 1: 374-388, 2019.

Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprint arXiv:2010.13723 , 2020.

Ziheng Cheng, Xinmeng Huang, Pengfei Wu, and Kun Yuan. Momentum benefits non-iid federated learning simply and provably. In The Twelfth International Conference on Learning Representations , 2023.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243 , 2020.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 10351-10375. PMLR, 2022.

Yae Jee Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, and Tong Zhang. On the convergence of federated averaging with cyclic client participation. arXiv preprint arXiv:2302.03109 , 2023.

Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. Advances in Neural Information Processing Systems , 30, 2017.

Hubert Eichner, Tomer Koren, Brendan McMahan, Nathan Srebro, and Kunal Talwar. Semicyclic stochastic gradient descent. In International Conference on Machine Learning , pages 1764-1773. PMLR, 2019.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In International Conference on Machine Learning , pages 3407-3416. PMLR, 2021.

Margalit R Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local sgd) and continuous perspective. In International Conference on Artificial Intelligence and Statistics , pages 9050-9090. PMLR, 2022.

Michał Grudzie´ n, Grigory Malinovsky, and Peter Richtárik. Improving accelerated federated learning with compression and importance sampling. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities , 2023.

Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems , 34:12052-12064, 2021.

Dzmitry Huba, John Nguyen, Kshitiz Malik, Ruiyu Zhu, Mike Rabbat, Ashkan Yousefpour, Carole-Jean Wu, Hongyuan Zhan, Pavel Ustinov, Harish Srinivas, et al. Papaya: Practical, private, and scalable federated learning. Proceedings of Machine Learning and Systems , 4: 814-832, 2022.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends® in machine learning , 14(1-2):1-210, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 4519-4529. PMLR, 2020.

Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019.

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning , pages 3043-3052. PMLR, 2018.

Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials , 22(3):2031-2063, 2020.

Grigory Malinovsky, Samuel Horváth, Konstantin Burlachenko, and Peter Richtárik. Federated learning with regularized client participation. arXiv preprint arXiv:2302.03662 , 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963 , 2017.

Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A survey on security and privacy of federated learning. Future Generation Computer Systems , 115:619-640, 2021.

Kumar Kshitij Patel, Lingxiao Wang, Blake E Woodworth, Brian Bullins, and Nati Srebro. Towards optimal communication complexity in distributed non-convex optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 13316-13328. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 56bd21259e28ebdc4d7e1503733bf421-Paper-Conference.pdf .

Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier van Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, et al. Federated evaluation and tuning for on-device personalization: System design & applications. arXiv preprint arXiv:2102.08503 , 2021.

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecny, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. ICLR , 2021.

Elsa Rizk, Stefan Vlaski, and Ali H Sayed. Federated learning under importance sampling. IEEE Transactions on Signal Processing , 70:5381-5396, 2022.

Yichen Ruan, Xiaoxi Zhang, Shu-Che Liang, and Carlee Joe-Wong. Towards flexible device participation in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 3403-3411. PMLR, 2021.

Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767 , 2018.

Alexander Tyurin, Lukang Sun, Konstantin Pavlovich Burlachenko, and Peter Richtárik. Sharper rates and flexible framework for nonconvex sgd with client and data sampling. Transactions on Machine Learning Research , 2022.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of communication-efficient sgd algorithms. arXiv preprint arXiv:1808.07576 , 2018.

Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan AlShedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 , 2021.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. Advances in Neural Information Processing Systems , 35:19124-19137, 2022.

Shiqiang Wang and Mingyue Ji. A lightweight method for tackling unknown participation probabilities in federated averaging. arXiv preprint arXiv:2306.03401 , 2023.

Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security , 15:3454-3469, 2020. doi: 10.1109/TIFS.2020.2988575.

Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334-10343. PMLR, 2020.

Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems , 33:6281-6292, 2020.

Ming Xiang, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong, and Lili Su. Efficient federated learning against heterogeneous and non-stationary client unavailability. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017.

Yikai Yan, Chaoyue Niu, Yucheng Ding, Zhenzhe Zheng, Fan Wu, Guihai Chen, Shaojie Tang, and Zhihua Wu. Distributed non-convex optimization with sublinear speedup under intermittent client availability. arXiv preprint arXiv:2002.07399 , 2020.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. In International Conference on Learning Representations , 2020.

Haibo Yang, Xin Zhang, Prashant Khanduri, and Jia Liu. Anarchic federated learning. In International Conference on Machine Learning , pages 25331-25363. PMLR, 2022.

Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903 , 2018.

Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , pages 7184-7193, 2019.

Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5693-5700, 2019.

Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems , 33:5332-5344, 2020.

Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Based Systems , 216:106775, 2021.

Chen Zhu, Zheng Xu, Mingqing Chen, Jakub Koneˇ cn` y, Andrew Hard, and Tom Goldstein. Diurnal or nocturnal? federated learning of multi-branch networks from periodically shifting distributions. In International Conference on Learning Representations , 2021.

Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputing , 465:371-390, 2021.

Contents

A Proof of Theorem 1

A.1 Preliminary Definitions

Let G r denote the filtration generated by { ξ i r,k : 0 ≤ k ≤ I -1 , i ∈ [ N ] } , that is, by the randomness in the stochastic gradients during round r . Also, let Q r 0 denote the filtration generated by { q i r : r 0 ≤ r < r 0 + P, i ∈ [ N ] } , that is, the randomness in client sampling between rounds r 0 and r 0 + P -1 (inclusive). Also, let G denote the filtration generated by G 0 ∪ G 1 ∪ . . . ∪ G R -1 and Q denote the filtration generated by Q 0 ∪ Q P ∪ . . . ∪ Q R -P . Similarly, let G : r denote the filtration generated by G 0 ∪ G 1 ∪ . . . ∪ G r -1 and Q : r 0 denote the filtration generated by Q 0 ∪ Q P ∪ . . . ∪ Q r 0 -P . Lastly, denote E r 0 [ · ] = E [ ·|Q : r 0 , G : r 0 ] .

In order to analyze the control variate errors (i.e., ∥∇ f i (¯ x r 0 ) -G i r 0 ∥ ), we introduce notation to refer to the iterates whose stochastic gradients were used to construct G i r 0 . These iterates are exactly the iterates during the most recent window before r 0 in which client i was sampled, i.e., where 1 P ∑ ( m +1) P -1 s = mP q i s > 0 . For each r ∈ { r 0 , r 0 +1 , . . . , r 0 + P -1 } , k ∈ { 0 , . . . , I -1 } , and i ∈ [ N ] , let

with the initialization t i ( r ) = 0 for all r ∈ {-P, . . . , -1 } . Then denote

Also, denote ¯ z i r 0 = 1 P ∑ r 0 + P -1 r = r 0 z i r . Then we can rewrite

As discussed in the main body, ¯ x r,k is a weighted average over local client models, weighted according to client participation. D r,k and M r,k are the drift terms described in the proof sketch of Section 4.4. In the proof sketch, we also informally discuss the control variate error C i r 0 = ∥∇ f i (¯ x r 0 ) -G i r 0 ∥ 2 .

Define

This error is closely related to the term S i r 0 defined above, since E [ ∥ ∥ ∇ f i (¯ x r 0 ) -E [ G i r 0 |Q ] ∥ ∥ 2 ∣ ∣ ∣ Q ] = L 2 S i r 0 .

The quantities w i r 0 , v i r 0 and Λ i r 0 arise in our proof from the use of control variates under non-i.i.d. client participation. As discussed in the proof sketch (Section 4.4), bounding the errors introduced by control variates involves a non-uniform average of non-uniform averages of error terms. This nested non-uniform average can be bounded by a uniform average as long as the term w i r 0 is not too large, which is exactly the requirement stated in Equation 1. On the other hand, the term Λ i r 0 arises while bounding the variance of the update ∥ ¯ x r 0 + P -¯ x r 0 ∥ , and it represents the sample size for which the correction G i r 0 is computed. When the client sampling distribution is closer to uniform, then each client may participate frequently during each window of P rounds, and the effective sample size used to compute each control variate is large. In this case, the variance of the update ∥ ¯ x r 0 + P -¯ x r 0 ∥ will be smaller, since larger sample size implies smaller variance. For general client sampling distributions, Λ i r 0 describes the reduction of variance of one component of the update (i.e. G i r 0 ) due to the sampling of stochastic gradients during the previous window of P rounds. In our analysis, the variance of the update ∥ ¯ x r 0 + P -¯ x r 0 ∥ can be bounded in terms of the variance of G i r 0 , which in turn depends on Λ i r 0 . The condition in Equation 2 essentially enforces that clients participate sufficiently uniformly, so that the variance of G i r 0 can be bounded. We note that both Equation 1 and Equation 2 are satisfied by i.i.d. participation, regularized participation, and cyclic participation.

Finally, for ease of exposition, one clarification must be made about the conditional expectation E [ ·|Q ] as it appears in, for example, M r,k . The filtration Q contains randomness from every round from 0 to R -1 , but the expression of which we are taking expectation may only depend on randomness for a smaller subset rounds. For example, in M r,k , the expression ∥ ¯ x r,k -¯ x r 0 ∥ 2 only depends on the randomness up to round r ≤ r 0 + P , so that

In several places throughout the proof, we will replace E [ ·|Q ] by E [ ·|Q : r 0 + P ] in similar situations.

A.2 Proofs

Lemma 1. If η ≤ 1 60 LIP and E r 0 [¯ q i r 0 ] = 1 N , then

and

Proof. Denote

Also, denote g i r,k = ∇ F i ( x i r,k ; ξ i r,k ) -G i r 0 + G r 0 and ¯ g i r,k = ∇ f i ( x i r,k ) -¯ G i r 0 + ¯ G r 0 . ¯ G i r 0 is the analogue of ¯ G r 0 that depends on deterministic gradients ∇ f i ( y i s,k ) instead of stochastic gradients.

Variance of updates We first compute the errors E [ ∥ ∥ ∥ g i r,k -¯ g i r,k ∥ ∥ ∥ 2 ∣ ∣ ∣ ∣ Q ] and E [ ∥ ∥ ∥ ∑ N i =1 q i r ( g i r,k -¯ g i r,k )∥ ∥ ∥ 2 ∣ ∣ ∣ ∣ Q ] , which will be needed in several places.

where ( i ) uses Jensen's inequality. Therefore

where the last line uses Equation 5. Also

where ( i ) uses the fact that stochastic gradient noise is independent across each client, and ( ii ) uses ρ 2 ≥ 1 N .

One-step recursive bound for D r,k For any k ≥ 0 ,

where ( i ) uses

( a ) uses Equation 6, and ( ii ) uses Young's inequality. Focusing on the second term of Equation 8:

where ( i ) uses Jensen's inequality. Using the decomposition

we have

Plugging back into Equation 8,

Now choose λ 1 = 2 I , so that 1 ≤ λ 1 2 and

where we used the condition η ≤ 1 60 LIP . This yields the following recursive bound on D r,k +1 :

One-step recursive bound for M r,k For any k ≥ 0 ,

where ( i ) uses Equation 7 and ( ii ) uses Young's inequality. Focusing on the second term in Equation 10:

Plugging back into Equation 10:

Now choose λ 2 = 2 IP , so that 1 ≤ λ 2 2 and

where we used the condition η ≤ 1 60 LIP . This yields the following recursive bound on M r,k +1 :

Note that the same argument can be used to bound M r, 0 in terms of M r -1 ,I -1 with the same recurrence relation.

Unrolling the recursion For 0 ≤ t ≤ IP , let r t = r 0 + ⌊ t/I ⌋ and k t = t -⌊ t/I ⌋ ∗ I . Then let d t = D r t ,k t and m t = M r t ,k t . Also, let

Then Equation 9 and Equation 11 imply the following mutually recursive relation over d t and m t :

where d 0 = m 0 = 0 . We will unroll this recurrence, showing a bound for m t and d t in terms of the following quantities. For any 0 ≤ t ≤ IP -1 and 0 ≤ s ≤ r t , let j ( s, t ) = min { I, t -sI } and ℓ ( s, t ) = max { 0 , t -( s +1) I } . For any s, t , if k t +1 = 0 , define

Otherwise, define

Also, define

under the initial conditions

Now we can unroll the recurrence in Equation 12, by proving the following statements by induction on t :

̸

Equation 13 and Equation 14 hold for the base case t = 0 , since d 0 = m 0 = 0 . Now suppose that Equation 13 and Equation 14 hold for some t ≤ IP , and we will show that they hold for t +1 . We consider two cases: k t +1 = 0 and k t +1 = 0 .

In the first case, r t +1 = r t , i.e., step t +1 is in the same round as step t , and k t +1 = k t +1 . Using Equation 12 together with the inductive hypothesis:

where ( i ) uses the fact that ϕ s,t +1 = ( 1 + 1 I ) ϕ s,t + Pβ s,t and the definitions of α s,t +1 , ψ s,t , and ( ii ) uses k t +1 = k t +1 and r t +1 = r t . Similarly,

where ( i ) uses the fact that for s < r t :

and for s = r t :

̸

( ii ) uses the fact that ψ s,t +1 = ( 1 + 1 IP ) ψ s,t + 1 P α s,t and the definitions of β s,t +1 , ϕ s,t , and ( iii ) uses k t +1 = k t +1 and r t +1 = r t . This completes the inductive step for the first case ( k t +1 = 0 ).

In the second case (i.e., k t +1 = 0 ), Equation 13 must hold for t + 1 , since d t +1 = 0 . So it only remains to show Equation 14 holds for t +1 . Note that r t = r t +1 -1 . As in the first case, we can

use Equation 12 together with the inductive hypothesis to obtain:

where the second line uses the fact that the r t +1 -st element of both sums is 0 , since j ( r t +1 , t +1) = 0 , ψ r t +1 ,t +1 = 0 , and ϕ r t +1 ,t +1 = 0 . This completes the inductive step for both cases, and proves Equation 13 and Equation 14.

We can now bound α s,t and β s,t separately by induction on t . First, for any s ≤ P -1 and t with sI ≤ t ≤ IP , we claim that

which we will show by induction on t . Let s ≤ P -1 be given. For the base case, Equation 15 holds when t = sI since α s,sI = 0 . Now suppose that it holds for all t ′ ≤ t . Then

and

where the last line uses the definition of q 1 and q 2 together with the condition η ≤ 1 60 LIP :

Plugging Equation 16 and Equation 17 into the definition of α s,t +1 yields

where ( i ) unrolls the recurrence on the first line until α s,r t I = 0 , ( ii ) uses P ∑ I -1 i =0 ( 1 + 1 IP ) i ≤ ∑ IP -1 i =0 ( 1 + 1 IP ) i since ( 1 + 1 IP ) i is increasing with i , and we repeatedly used ( 1 + 1 x ) x < e ≤ 3 . This completes the induction and proves Equation 15.

We will similarly prove the statement

for all s ≤ P -1 and t with sI ≤ t ≤ IP by induction on t . Let s ≤ P -1 be given. For the base case, Equation 15 holds when t = sI since β s,sI = 0 . Now suppose that it holds for all t ′ ≤ t . Then

where the last line uses the definition of q 1 and q 2 together with the condition η ≤ 1 60 LIP :

We now consider two cases: t < ( s +1) I and t ≥ ( s +1) I . In the first case, we have s = r t , and plugging Equation 20 into the definition of β s,t +1 yields

where ( i ) unrolls the recurrence on the first line until β s,sI = 0 and we repeatedly used ( 1 + 1 x ) x < e ≤ 3 . In the second case, we have s > r t , so plugging Equation 19 into the definition of β s,t +1

yields

where ( i ) unrolls the recurrence on the previous line until β s,sI = 0 and the last inequality uses the definition of q 1 and q 2 together with the condition η ≤ 1 60 LIP :

This completes the induction in both cases and proves Equation 18.

We can then use Equation 15 and Equation 18 to yield bounds for the remaining terms of Equation 13 and Equation 14 as follows:

and

Finally, we can plug these into Equation 13 to yield

and into Equation 14

where ( i ) uses and

Or, in the original notation:

where ( i ) uses the fact that the interval { r 0 , . . . , r } is contained in the interval { r 0 , . . . , r 0 + P -1 } and ( ii ) uses the condition η ≤ 1 60 LIP to simplify non-dominating terms. Also

where ( i ) and ( ii ) use the same operations as in Equation 21. Summing Equation 22 and Equation 21,

where the last inequality uses η ≤ 1 60 LIP . This proves Equation 3. Equation 4 follows by summing over r ∈ { r 0 , . . . , r 0 + P -1 } and k ∈ { 0 , . . . , I -1 } , taking total expectation, and applying the condition E r 0 [¯ q i r 0 ] = 1 N .

Proof. By the algorithm definition,

To obtain the variance of the update ¯ x r 0 + P -¯ x r 0 ,

We can bound the two terms A 1 and A 2 separately as follows. For A 1 ,

where ( i ) uses the fact that for each i , { q i r ( ∇ F i ( x i r,k ) -∇ f i ( x i r,k ) )} r,k is a martingale difference sequence with respect to G (when conditioned on Q ) and that stochastic gradient noise is independent

across clients. For A 2 ,

so

where ( i ) uses the fact that, conditioned on Q , the variables G i r 0 -¯ G i r 0 depend only on stochastic gradient noise, which is independent across clients, and the last inequality uses the condition E [ ∑ N i =1 ( v i r 0 ) 2 Λ i r 0 ] ≤ ρ 2 . Plugging Equation 25 and Equation 26 into Equation 24 yields

Therefore

where ( i ) uses Equation 27, ( ii ) uses the decomposition

The remaining term in Equation 28 can be simplified as:

Plugging back to Equation 28 yields

where ( i ) uses Equation 4 from Lemma 1 and ( ii ) uses η ≤ 1 60 LIP and γη ≤ 1 60 LIP . This proves Equation 23.

Lemma 3. Suppose that P Q r 0 (¯ q i r 0 > 0) ≥ p sample , E [ w i r 0 ∣ ∣ Q : r 0 ] ≤ P 2 N , E r 0 [¯ q i r 0 ] = 1 N , and E [ ∑ N i =1 ( v i r 0 ) 2 Λ i r 0 ] ≤ ρ 2 . If η ≤ √ p sample 60 LIP and γη ≤ p sample 60 LIP , then

Proof. Let 1 ≤ i ≤ N . We can consider the value S i r 0 + P under two cases: ¯ q i r 0 > 0 and the complement. Let A i r 0 = { ¯ q i r 0 > 0 } . Denote

Then S i r 0 + P = B i 1 + B i 2 , and we can consider the two cases separately. Notice that

Therefore

B i 1 = 1 { A i r 0 } 1 IP ¯ q i r 0 r 0 + P -1 ∑ s = r 0 I -1 ∑ k =0 q i s E [ ∥ ∥ x i s,k -¯ x r 0 + P ∥ ∥ 2 ∣ ∣ ∣ Q ] ≤ 1 { A i r 0 } 3 IP ¯ q i r 0 r 0 + P -1 ∑ s = r 0 I -1 ∑ k =0 q i s ( E [ ∥ x i s,k -¯ x s,k ∥ 2 ∣ ∣ Q ] + E [ ∥ ¯ x s,k -¯ x r 0 ∥ 2 ∣ ∣ Q ] + E [ ∥ ¯ x r 0 + P -¯ x r 0 ∥ 2 ∣ ∣ Q ] ) ≤ 1 { A i r 0 } 3 IP ¯ q i r 0 r 0 + P -1 ∑ s = r 0 I -1 ∑ k =0 q i s ( D s,k + M s,k ) + 3 1 { A i r 0 } E [ ∥ ¯ x r 0 + P -¯ x r 0 ∥ 2 ∣ ∣ Q ] . Using Equation 3 from Lemma 1, 1 { A i r 0 } 3 IP ¯ q i r 0 r 0 + P -1 ∑ s = r 0 I -1 ∑ k =0 q i s ( D s,k + M s,k ) ≤ 1 { A i r 0 } 3 IP ¯ q i r 0 r 0 + P -1 ∑ s = r 0 I -1 ∑ k =0 q i s ( 108 η 2 I 2 P 2 E [ ∥∇ f (¯ x r 0 ) ∥ 2 ∣ ∣ ∣ Q ] + ( 75 η 2 I +65 η 2 IPρ 2 ) σ 2 +109 η 2 L 2 I 2 P 2 N ∑ j =1 ( ¯ q j r 0 + 1 N ) S j r 0 +36 η 2 L 2 I 2 N ∑ j =1 q j s S j r 0 ) ≤ 324 η 2 I 2 P 2 E [ ∥∇ f (¯ x r 0 ) ∥ 2 ∣ ∣ ∣ Q ] + ( 225 η 2 I +195 η 2 IPρ 2 ) σ 2 +327 η 2 L 2 I 2 P 2 N ∑ j =1 ( ¯ q j r 0 + 1 N ) S j r 0 + 1 { A i r 0 } 3 P ¯ q i r 0 r 0 + P -1 ∑ s = r 0 q i s   36 η 2 L 2 I 2 N ∑ j =1 q j s S j r 0   = 324 η 2 I 2 P 2 E [ ∥∇ f (¯ x r 0 ) ∥ 2 ∣ ∣ ∣ Q ] + ( 225 η 2 I +195 η 2 IPρ 2 ) σ 2 +327 η 2 L 2 I 2 P 2 N ∑ j =1 ( ¯ q j r 0 + 1 N ) S j r 0 +108 η 2 L 2 I 2 N ∑ j =1 ( 1 { A i r 0 } P ¯ q i r 0 r 0 + P -1 ∑ s = r 0 q i s q j s ) S j r 0 . Denote w i,j r 0 = 1 { A i r 0 } P ¯ q i r 0 ∑ r 0 + P -1 s = r 0 q i s q j s , so that w i r 0 = 1 N ∑ N j =1 w i,j r 0 . Then B i 1 ≤ 324 η 2 I 2 P 2 E [ ∥∇ f (¯ x r 0 ) ∥ 2 ∣ ∣ ∣ Q ] + ( 225 η 2 I +195 η 2 IPρ 2 ) σ 2 +327 η 2 L 2 I 2 P 2 N ∑ j =1 ( ¯ q j r 0 + 1 N ) S j r 0 +108 η 2 L 2 I 2 N ∑ j =1 ( 1 { A i r 0 } P ¯ q i r 0 r 0 + P -1 ∑ s = r 0 q i s q j s ) S j r 0 +3 1 { A i r 0 } E [ ∥ ¯ x r 0 + P -¯ x r 0 ∥ 2 ∣ ∣ Q ] [ B i 1 ∣ ∣ Q : r 0 ] ( i ) ≤ 324 η 2 I 2 P 2 E [ ∥∇ f (¯ x r 0 ) ∥ 2 ∣ ∣ ∣ Q : r 0 ] + ( 225 η 2 I +195 η 2 IPρ 2 ) σ 2 +654 η 2 L 2 I 2 P 2 1 N N ∑ j =1 S j r 0 +108 η 2 L 2 I 2 N ∑ j =1 E [ 1 { A i r 0 } P ¯ q i r 0 r 0 + P -1 ∑ s = r 0 q i s q j s ∣ ∣ ∣ ∣ ∣ Q : r 0 ] S j r 0 +3 E [ 1 { A i r 0 } ∥ ¯ x r 0 + P -¯ x r 0 ∥ 2 ∣ ∣ Q : r 0 ]

where ( i ) uses E r 0 [¯ q i r 0 ] = 1 N and the tower property E [ E [ ·|Q ] |Q : r 0 ] = E [ ·|Q : r 0 ] , and ( ii ) uses the tower property E [ E [ ·|Q : r 0 ]] = E [ · ] .

Now consider B i 2 . Under ¯ A i r 0 , z i r = z i r -P and y i r,k = y i r -P,k for all r ∈ { r 0 , . . . , r 0 + P -1 } . Therefore

where ( i ) uses Young's inequality with an arbitrary λ > 0 . Taking conditional expectation E [ ·|Q : r 0 ] followed by total expectation yields

Adding Equation 29 and Equation 30 yields

Averaging over i ,

where the last inequality uses the condition E [ w i r 0 ∣ ∣ Q : r 0 ] ≤ P 2 N . We can then apply Lemma 2 and choose λ = 3 p sample 10(1 -p sample ) to obtain

where the last inequality uses the conditions η ≤ √ p sample 60 LIP and γη ≤ p sample 60 LIP , and we used that 3 + 1 λ ≤ 10 3 p sample .

Theorem 2 (Theorem 1 restated) . Suppose Assumptions 1 and 2 hold, and E [ w i r 0 |Q : r 0 ] ≤ P 2 N , and E [ ∑ N i =1 ( v i r 0 ) 2 Λ i r 0 ] ≤ ρ 2 . If

then Algorithm 1 satisfies

Proof. Recall that

Using the quadratic upper bound for smooth functions and taking total expectation,

where ( i ) uses the law of total expectation E [ · ] = E [ E r 0 [ · ]] , and ( ii ) uses the condition E r 0 [¯ q i r 0 ] = 1 N together with

Focusing on the inner product term of Equation 32:

where we used -⟨ a, b ⟩ = 1 2 ∥ b -a ∥ 2 -1 2 ∥ a ∥ 2 -1 2 ∥ b ∥ 2 ≤ 1 2 ∥ b -a ∥ 2 -1 2 ∥ a ∥ 2 . Also

where ( i ) uses the decomposition

( ii ) and ( iii ) use Jensen's inequality, and ( iv ) uses smoothness of each f i along with the condition E r 0 [ ¯ q i r 0 ] = 1 N . Plugging into Equation 33 yields

where ( i ) uses Equation 4 from Lemma 1.

Plugging into Equation 32 and applying Lemma 2 yields

where ( i ) uses η ≤ √ p sample 60 LIP and γη ≤ p sample 60 LIP .

Using Lemma 3,

where the last inequality uses γη ≤ p sample 60 LIP . Adding this to Equation 35 and rearranging,

where we used γη ≤ 1 60 LIP and η ≤ 1 60 LIP . Finally, we can average over r 0 ∈ { 0 , P, 2 P, . . . , R -P } and rearrange to obtain

where ( i ) uses 1 N ∑ N i =1 E [ S i R ] ≥ 0 and 1 N ∑ N i =1 E [ S i 0 ] = 0 by initialization, and ( ii ) uses P ≥ 1 and ρ ≤ 1 .

Corollary 2 (Corollary 1 restated) . Let ϵ > 0 and I ≥ 1 . Suppose Assumptions 1 and 2 hold, and that E [ w i r 0 |Q : r 0 ] ≤ P 2 N , and E [ ∑ N i =1 ( v i r 0 ) 2 Λ i r 0 ] ≤ ρ 2 . If

then Algorithm 1 satisfies

Proof. First, η ≤ ρ √ p sample 264 LIP ≤ √ p sample 60 LIP and γη ≤ p sample 60 LIP together with Assumptions 1 and 2 imply that the conditions of Theorem 2 are satisfied. Therefore

From our choice of η ,

Therefore

where we used our choice of γη . Finally, from our choice of R ,

B Proofs for Specific Participation Patterns

Here we derive the convergence rates for the specific participation patterns discussed in Section 4.3. In order to apply Corollary 1, the conditions in Assumption 2, Equation 1, and Equation 2 must be satisfied. Since we already showed that Assumption 2 is satisfied by regularized participation and cyclic participation (in Section 3.2), it only remains to show that Equation 1 and Equation 2 is satisfied, and plug the appropriate values of ρ 2 , P , and p sample into Corollary 1.

Corollary 3 (Regularized Participation) . Under regularized participation, for any ϵ > 0 , there exist choices of η, γ, I , and R such that E [ ∥∇ f (ˆ x ) ∥ 2 ] ≤ O ( ϵ 2 ) and

Proof. We first show that Equation 1 and Equation 2 are satisfied under regularized participation. Let r 0 ∈ { 0 , . . . , R -P } . Using the fact that ¯ q i r 0 = 1 N almost surely:

where we used the fact that ∑ N i =1 q i r = 1 . This shows that Equation 1 is satisfied. Also, ¯ q i r 0 = 1 N means that v i r 0 = 0 , so that E [ ( v i r 0 ) 2 Λ i r 0 ] = 0 ≤ ρ 2 , and Equation 2 is satisfied. Therefore we can apply Corollary 1. The complexity result follows by plugging in p sample = 1 and choosing I = O ( ∆ ρ 2 σ 2 Pϵ 2 ) .

Corollary 4 (Cyclic Participation) . Under cyclic participation with ¯ K groups and S participating clients in each round, for any ϵ > 0 , there exist choices of η, γ, P, I , and R such that E [ ∥∇ f (ˆ x ) ∥ 2 ] ≤ O ( ϵ 2 ) and

Proof. Again, we show that Equation 1 and Equation 2 are satisfied under cyclic participation. Let r 0 ∈ { 0 , . . . , R -P } . Denoting g ( i ) = ⌊ ¯ K N ( i -1) ⌋ and r ( i ) = r 0 + g ( i ) , it holds that, over the rounds r ∈ { r 0 , . . . , r 0 + P -1 } , client i belongs to group g ( i ) and is available only during round r ( i ) . Also, let Π( i ) denote the set of indices of clients in group g ( i ) . To simplify w i r 0 , recall that for client i in group ( r mod ¯ K ) :

and q i r = 0 for all clients i not in group ( r mod ¯ K ) . So

̸

where ( i ) uses q i r = 0 for all r = r ( i ) , ( ii ) uses q j r ( i ) = 0 for all j / ∈ Π( i ) , ( iii ) uses the fact that ¯ q j r 0 > 0 implies q j r ( i ) = 1 S and ¯ q j r 0 = 1 SP , and ( iv ) uses the fact that S clients are sampled in each round. This shows that Equation 1 is satisfied.

To see that Equation 2 is satisfied:

where ( i ) uses the tower property, ( ii ) uses the variance of ¯ q i r 0 as computed in Appendix C, and ( iii ) uses the fact that ¯ z i r 0 > 0 by construction of ¯ z i r 0 , and in this case z i r = 1 S for exactly one r ∈ { r 0 -P, . . . , r 0 -1 } and z i r = 0 for all other r . Therefore Equation 2 is satisfied.

This shows we can apply Corollary 1. The complexity result follows by plugging in p sample = S N , P = ¯ K , ρ = 1 √ S , and choosing I = O ( ∆ σ 2 ¯ KNϵ 2 ) .

C Complexity of Amplified FedAvg

In this section we derive the computation and communication complexity required for Amplified FedAvg [39] to find an ϵ -stationary point under various client participation patterns, which we listed in Table 1 of the main paper. Table 1 compares the complexity under i.i.d. participation, regularized participation, and cyclic participation. Since i.i.d. participation is a special case of cyclic participation with ¯ K = 1 groups, here we only consider regularized and cyclic participation, and the result for i.i.d. participation follows.

Many works in federated learning characterize data heterogeneity by assuming that there exists a constant κ such that

for all x . The previous analysis of Amplified FedAvg [39] instead assumes an upper bound ˜ δ ( P ) on a weighted heterogeneity that depends on client sampling. Specifically, they assume that there exists ˜ δ ( P ) such that

for all x and r 0 . We restate their Corollary 3.2 for convenience:

Corollary 5 (Corollary 3.2 [39] informally restated) . There exist parameter choices such that Amplified FedAvg satisfies

As pointed out in their Section 4.2, we can interpret ˜ δ ( P ) in terms of the conventional heterogeneity constant κ as:

Our Assumption 2(b) implies that E Q r 0 [¯ q i r 0 ] = 1 N . If we choose v ≥ 0 such that Var [¯ q i r 0 ] ≤ v 2 , then we can take expecation of the above to obtain

This 'conversion" of ˜ δ ( P ) to κ and v will allow us to compare their complexity to that of algorithms that use the conventional heterogeneity assumption by computing v for each participation pattern.

Regularized Participation In this case, ¯ q i r 0 = 1 N almost surely, so that v = 0 and accordingly ˜ δ 2 ( P ) = 0 . Therefore

Therefore the choices

imply

Cyclic Participation We can compute v in terms of the parameters of the participation pattern: number of groups ¯ K and number of participating clients S in each round. Although we chose P = ¯ K for Amplified SCAFFOLD, we can satisfy Assumption(b) 2 by choosing P = m ¯ K for any m ∈ N , and indeed to achieve E [ ˜ δ 2 ( P )] ≤ ϵ 2 we must choose P = m ¯ K with m depending on ϵ .

Let A ( i ) denote the set of rounds in { r 0 , . . . , r 0 + P -1 } during which client i is available. Note that A ( i ) has size m , since P = m ¯ K . Then

where ( i ) uses the fact that q i r = 0 for all r / ∈ A ( i ) . Therefore

where ( i ) uses the fact that { q i r } r ∈ A ( i ) are independent and ( ii ) uses the fact that q i r equals 1 S times a Bernoulli variable for r ∈ A ( i ) .

With a bound for v 2 , we can bound the remaining term in Equation 36 as follows:

Therefore, Amplified FedAvg under cyclic participation can find an ϵ -stationary point with the choices

D Experiment Details

Here we discuss experimental details deferred from the main body: client sampling parameters, heterogeneity protocol, hyperparameter tuning, definition of the synthetic objective, and specification of the CNN architecture used for the image classification experiments.

D.1 Client Sampling Parameters

For the synthetic objective, we set the number of groups ¯ K = 2 and the availability time g = 240 . We set the communication interval I = 10 and train for R = 5000 rounds.

For Fashion-MNIST, we set the communication interval I = 30 and train a logistic regression model for R = 2000 rounds, with availability time g = 4 .

For CIFAR-10, we set the communication interval I = 5 and train a two-layer CNN for R = 12000 rounds, with availability time g = 10 .

D.2 Heterogeneity Protocol

The following protocol is commonly used in the literature [17, 39] to convert a dataset into a collection of heterogeneous local datasets according to a data similarity parameter s , where s = 0% creates maximal data heterogeneity across clients, and s = 100% means that data is allocated to each client uniformly at random.

A single, non-federated dataset (e.g. CIFAR-10) is partitioned into two subsets: s % of the samples are allocated to an i.i.d. pool and randomly shuffled, and the remaining (100 -s )% of the sampled are allocated to a non-i.i.d. pool and are sorted by label. Samples are allocated to each client so that s % of each local dataset comes from the i.i.d. pool, and the remaining (100 -s )% comes from the non-i.i.d. pool, so that with a small s , the majority of each local dataset consists of a small number of labels.

D.3 Hyperparameter Tuning

For all three experiments in the main body (synthetic objective, Fashion-MNIST, and CIFAR-10), each of the four baselines are individually tuned with grid search. For algorithms that use amplified updates (Amplified FedAvg and Amplified SCAFFOLD), we tune the amplification rate γ by searching over a fixed range of values. For the other algorithms (FedAvg and SCAFFOLD), we indicate the lack of amplified updates by setting γ = 1 . We tune η by allowing γη over a fixed range of values. For FedProx's µ parameter, we also search over a fixed range of values. The search range and final values for each parameter are written in Table 2, along with the final values adopted for each algorithm.

D.4 Synthetic Objective

For the synthetic experiment, we use a difficult objective from a lower bound analysis of FedAvg [43]. As defined in the lower bound analysis, the objective is parameterized by H,κ,σ,c, µ , and L .

FashionMNIST

0.00300

0.84

0.00275

0.82

SSOT

0.00250

TestAccuracy

0.80

0.00225

0.78

0.00200

0.76

0.00175

0.74

0.00150

0.72

0

2000

4000

6000

8000

1000012000

0

2000

4000

6000

8000

10000 12000

Rounds

Rounds

FedAvg

SCAFFOLD

Amplified SCAFFOLD

FedAdam

FedAvg-M

FedProx

Amplified FedAvg

Amplified FedProx

FedYogi

The objective maps R 4 to R , and there are only two clients with corresponding local objectives

where [ x ] + := max { x, 0 } . The stochastic gradients for f 1 and f 2 are sampled from the distributions ∇ f 1 ( x )+ ξe 3 and ∇ f 2 ( x )+ ξe 3 , respectively, where ξ ∼ N (0 , σ 2 ) and e 3 denotes the third standard basis vector in R 4 . We set the parameters of the objective as follows:

D.5 CNN Architecture

We use a simple 2-layer CNN for CIFAR-10. The first layer is a convolutional layer with 64 channels, a 5 × 5 kernel, stride of 2, padding of 2, and a ReLU activation. The second layer is a fully connected layer with no activation.

E Additional Experimental Results

In this section, we provide two additional experimental results. First, in Section E.1, we add four baselines to the experimental settings of the main paper: FedAdam [32], FedYogi [32], FedAvg-M [5], and Amplified FedAvg with FedProx regularization. Second, in Section E.2, we evaluate all nine algorithms (five from the main paper and the four additional baselines) under another non-i.i.d. client participation pattern for the CIFAR-10 dataset, which we refer to as Stochastic Client Availability (SCA).

E.1 Additional Baselines

We evaluate the four additional baselines for the Fashion-MNIST and CIFAR-10 experiments from Section 5, keeping the same experimental setup. We tuned the hyperparameters of all baselines according to the hyperparameter ranges suggested in the original paper of each algorithm, and we allow the same compute budget for tuning each baseline as we did for tuning the algorithms in the original paper, in terms of the total number of hyperparameter combinations evaluated. Also, the results are averaged over five random seeds. The results are shown in Figures 3 and 4.

For FashionMNIST, FedAdam and FedYogi reach moderate training loss quickly, but are soon overtaken by Amplified SCAFFOLD and later by SCAFFOLD. FedAvg-M exhibits a minor advantage over FedAvg, but performs about the same as Amplified FedAvg. Amplified FedProx (i.e. Amplified FedAvg with FedProx regularization) performs nearly identically to Amplified FedAvg.

CIFAR-10

0.0075

0.60

0.0070

0.55

LoSS

0.0065

Accurac

0.50

0.0060

0.45

0.0055

Test

0.40

0.0050

0.35

0.0045

0.30

0

2000

4000

6000

8000

10000 12000

0

2000

4000

6000

8000

10000 12000

Rounds

Rounds

FedAvg

SCAFFOLD

Amplified SCAFFOLD

FedAdam

FedAvg-M

FedProx

Amplified FedAvg

Amplified FedProx

FedYogi

For CIFAR-10, FedAdam is more competitive, but is still outperformed by Amplified SCAFFOLD. FedYogi and FedAvg-M are further behind, though both still outperform SCAFFOLD. Amplified FedProx is again nearly identical to Amplified FedAvg.

These additional comparisons demonstrate that Amplified SCAFFOLD outperforms strong empirical baselines (FedAdam, FedYogi) under cyclic client participation, reinforcing the empirical validation of our algorithm. This performance is consistent with the fact that Amplified SCAFFOLD has convergence guarantees under periodic participation, while FedAdam and FedYogi were not designed for settings beyond i.i.d. client sampling.

E.2 CIFAR-10 with Stochastic Client Availability

Here, we include an evaluation under another non-i.i.d. participation pattern, which we refer to as Stochastic Cyclic Availability (SCA). SCA models device availability which is both periodic and unreliable. Similarly to cyclic participation, the set of clients is divided into groups, and at each round one group is deemed the "active" group, while the others are inactive. Unlike cyclic participation, in SCA not every client in the active group is always available: Instead, when a group becomes active, the clients in that group become available for sampling with probability 80% , while clients in inactive groups have probability 5% to be available for participation. The active group changes every g rounds. This stochastic availability models the real-life situation where a client device can be unavailable at a time of day when it is usually available, or vice versa. In this way, SCA is more flexible than cyclic participation and better captures the unreliability of client devices. Lastly, we reused the remaining settings ( g , ¯ K , I , etc.) and the tuned hyperparameters for each baseline from the CIFAR-10 experiment under cyclic participation. Again, we average each algorithm's performance over five random seeds.

Results for CIFAR-10 under SCA participation are shown in Figure 5. Again, Amplified SCAFFOLD outperforms all baselines under SCA participation. The relative performance of each baseline is similar as under cyclic participation, with FedAdam staying competitive with Amplified SCAFFOLD, followed by FedYogi and FedAvg-M. The remainder of the baselines have significantly worse performance, and again Amplified FedAvg has not benefitted by adding FedProx regularization.

F Extended Comparison with Baselines

Here we include an extended comparison against two relevant prior works.

FedAvg with Cyclic Participation [8] analyzes FedAvg under cyclic participation, but the resulting convergence rate does not benefit from local steps unless regularized participation is satisfied. They analyze FedAvg for L -smooth and µ -PL objectives. Using their notation, ¯ K is the number of client groups, κ is the condition number of the objective, γ is the intra-group heterogeneity, M is the total number of clients, N is the number of clients that participate in each round, and T is the number of communication rounds. Then the dominating term in their convergence rate for FedAvg under cyclic

CIFAR-10(SCA Participation)

0.0075

0.60

0.0070

0.55

LoSS

0.0065

Accurac

2

0.50

0.0060

0.0055

Test

0.45

0.40

0.0050

0.0045

0.35

0.0040

0.30

0

2000

4000

6000

8000

1000012000

0

2000

4000

6000

8000

10000 12000

Rounds

Rounds

FedAvg

SCAFFOLD

AmplifiedSCAFFOLD

FedAdam

FedAvg-M

FedProx

AmplifiedFedAvg

Amplified FedProx

FedYogi

participation (Theorem 2) is

Notice that the number of local steps (denoted τ ) does not appear in this dominating term, so there is no way to reduce the communication complexity (compared to parallel SGD) by taking local steps. The only exception is when this term is zero from N = M/ ¯ K , which is equivalent to the condition that every client participates within every cycle of availability. Therefore, this result cannot show a benefit from local steps unless the client participation is regularized.

SCAFFOLD In Section 4.3, we mentioned a discrepancy between the complexity of SCAFFOLD vs. Amplified SCAFFOLD in terms of the dependence on N/S . In Table 1, the communication complexity of SCAFFOLD and Amplified SCAFFOLD under i.i.d. participation differs by their dependence on N S . The complexity of Amplified SCAFFOLD is O ( N S ) , while that of SCAFFOLD is O ( ( N S ) 2 / 3 ) . This difference in the order of N S is due to a potential small issue in the analysis of SCAFFOLD, which we intentionally avoided by accepting a slightly worse dependence on N S .

This difference stems from an apparent mistake in the original SCAFFOLD analysis. In the proof of Lemma 16 (PMLR version of SCAFFOLD), the second-to-last equation of page 32 is obtained with an incorrect step. Namely, while the current version includes

where the last line is obtained by conditioning on the event α r i,k -1 = α r -1 i,k -1 and the complement α r -1 i,k -1 = y r i,k -1 , which have probabilities 1 -S N and S N , respectively. However, this condition is not denoted in Equation 37. A corrected version should be written

but this conditional expectation is not specified in the proof of SCAFFOLD. For the remainder of the proof of Lemma 16, these terms are treated as total expectation, leading to an inconsistency. Lemma 16 concludes by applying Lemma 15 in order to bound the term E [ ∥ E r -1 [∆ x r ] ∥ 2 ] . However, when we make the correction to include the conditional expectation, we do not have a bound of Ξ r in terms

of E [ ∥ E r -1 [∆ x r ] ∥ 2 ] , we instead have a bound in terms of E [ ∥ ∥ ∥ E r -1 [ ∆ x r | α r -1 i,k -1 = y r i,k -1 ]∥ ∥ ∥ 2 ] . But this term with a conditional expectation inside the norm can't be bounded with Lemma 15. A pessimistic solution is to use Jensen's inequality to bound

where the second line follows from the tower property. This is the step that we perform in our analysis of Amplified SCAFFOLD, and this results in the N S dependence.

Fixing their analysis to recover the same ( N S ) 2 / 3 dependence may be possible, but we have instead focused on achieving the best known complexity in terms of ϵ, κ, σ , and other problem parameters.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Every claim made in the abstract is specifically tied to a theoretical convergence property of our proposed algorithm, which are stated in Section 4.2 and proven in A and B.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussion limitations in the "limitations" paragraph of Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All of the theoretical results are stated with the full set of assumptions in Sections 3, 3.1, and 4.2. The proofs are contained in Section A and B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental details are fully specified in Section 5.1 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for our experiments is included in the supplemental material with instructions for reproduction.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The main training details are specified in Section 5.1. Details such as hyperparameters, tuning protocols, architecture choices, etc. are specified in Section D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All of our experiments show results averaged over 3-5 random seeds, and error bars are shown for learning curves. Error bar calculation is specified in Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Hardware resources are specified in Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and conformed to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper is a theoretical paper on mathematical optimization problems arising in distributed learning. We do not see any direct paths to negative applications beyond those existing for any application of distributed learning or machine learning in general.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not involve the release of any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Our paper uses existing datasets (Fashion-MNIST, CIFAR-10), and we cite original sources for both datasets in Section 5.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Amplified SCAFFOLD, which is an optimization algorithm for federated learning under periodic client participation. The authors prove that it achieves reduced communication cost, linear speedup, and resilience to data heterogeneity. Numerical experiments are provided to evaluate the performance of Amplified SCAFFOLD.\n\nSTRENGTHS:\nFor federated learning under periodic client participation, Amplified SCAFFOLD is proposed and proven to exhibit reduced communication, linear speedup, and resilience to data heterogeneity. Experimental results show that Amplified SCAFFOLD converges faster than baseline algorithms and is robust to changes in data heterogeneity and the number of participating clients.\n\nWEAKNESSES:\n1. It would be better to provide a detailed description of the algorithm rather than just an overview.\n2. Except for the synthetic data, the experiments are based only on two image datasets (Fashion-MNIST and CIFAR-10) with classification tasks. Adding more experiments would better demonstrate the performance of Amplified SCAFFOLD.\n\nMinor:\n1. Line 60-61: Delete \"non-uniform average of\".\n2. Line 99: Assumption 1 (a) is not precise. It should be for all $y$, not for all $x,y$.\n3. Line 110: $P$ first appears without explanation.\n4. Line 145: Delete \"is\".\n\nQUESTIONS:\n1. For cycle participation, why are the subsets equally sized? Can they have different sizes? In addition, can the clients sampled with replacement?\n2. In the experiments, $\\bar{K}$ is chosen to be 5. What about the other choices? What is its influence?\n\nLIMITATIONS:\nAs the authors stated, this paper only considers periodic client participation. However, this may not always be true in practice. Clients may arbitrarily join and leave the training, and they may perform different numbers of local steps in each round.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm named Amplified SCAFFOLD for federated learning in environments with periodic client participation and heterogeneous data. The authors address the realistic setting where clients (e.g., mobile devices) are not always available for participation. The proposed algorithm aims to achieve linear speedup, reduced communication rounds, and resilience to data heterogeneity under non-convex optimization. The paper includes theoretical analysis demonstrating the benefits of Amplified SCAFFOLD over existing methods and provides experimental results to validate the proposed approach.\n\nSTRENGTHS:\n1. The introduction of Amplified SCAFFOLD is well-motivated. The algorithm's design to handle non-i.i.d. client participation and its ability to provide tighter guarantees than previous work is well-supported by their theoretical analysis.\n2. The experiments with synthetic and real-world data (e.g., Fashion-MNIST and CIFAR-10) demonstrate the algorithm's effectiveness and robustness under various conditions.\n\nWEAKNESSES:\n1. Some assumption is too strong, i.e., bounded objective function gap $f(\\\\mathbf{x}) - f\\_{\\\\min}$ for all $\\\\mathbf{x} \\\\in \\\\mathbb{R}^d$ in assumption 1(a), which is not satisfied for coercive objective function, a commonly used condition in optimization. In addition, previous works [1,2] only utilized the gap between the initial point and the optimum $f(\\\\mathbf{x}_0) - f\\_{\\\\min}$.\n2. While the paper provides comparisons with several baselines, it would be helpful to see related works with more recent state-of-the-art methods that might also address similar challenges, e.g., [3].\n\n\n[1]. Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., & Suresh, A. T. (2020). Scaffold: Stochastic controlled averaging for federated learning. International conference on machine learning.\n\n[2]. Wang, S., & Ji, M. (2022). A unified analysis of federated learning with arbitrary client participation. Advances in Neural Information Processing Systems.\n\n[3]. Crawshaw, M., Bao, Y., & Liu, M. (2024). Federated learning with client subsampling, data heterogeneity, and unbounded smoothness: A new algorithm and lower bounds. Advances in Neural Information Processing Systems.\n\nQUESTIONS:\nCan the authors explain from a technical point of view how their amplified SCAFFOLD algorithm is robust against data heterogeneity? I can see that the weights of the participating customers are amplified, but no other mechanism to mitigate data heterogeneity has been identified.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the limitations of federated learning under realistic client participation patterns, specifically focusing on nonconvex optimization. The proposed algorithm, Amplified SCAFFOLD, achieves linear speedup, reduced communication, and resilience to data heterogeneity without requiring strong assumptions. Compared to previous methods, it significantly reduces the required communication rounds for finding a $\\epsilon$-stationary point in cyclic participation scenarios. The analysis provides tighter guarantees, and experimental results on both synthetic and real-world data demonstrate the algorithm's effectiveness.\n\nSTRENGTHS:\n1.) The paper is well written\n2.) There is extensive theoretical analysis\n3.) Supported by experiments on baselines.\n\nWEAKNESSES:\nSee the questions section.\n\nQUESTIONS:\n1.) While setting up the problem in section 3, I think having a notation table would have been beneficial. Because there are a ton of notations in the paper, many of which are defined later.\n\n2.) In Assumption 1.(a) what is the $\\Delta$ and what is the intuitive meaning behind this assumption? Does it have anything to do with the non-negativity of the loss function?\n\n3.) Typo: In line 133, the authors have mentioned $\\bar{q}_{r0}$.\n\n4.) Do the authors use the assumption of data heterogeneity mentioned in Table 1 in their proof? If not, then how did they avoid to use it?\n\n5.) As the authors say, the work is a culmination of algorithmic components from references [14] and [34] (references from the paper), and I agree that analysis is by no means trivial. So, combining the algorithmic component of Amplified FedAvg paper and Fedprox would also provide a better result. Can this be checked experimentally, or did the authors observe anything?\n\n6.) Since the papers signify the main contribution to achieving communication efficiency, I would also like to see some experiments with algorithms like FedAdam and FedYogi. Along similar lines of work, in [1] (check below), the authors showed that momentum-based methods can reduce client drift and perform better than methods like SCAFFOLD for full participation.\n\n7.) Did the authors also check the efficacy of the proposed algorithm with different participation strategies, as mentioned in the paper?\n\n\n[1] Cheng, Ziheng, et al. \"Momentum benefits non-iid federated learning simply and provably.\" arXiv preprint arXiv:2306.16504 (2023).\n\nLIMITATIONS:\nYes, the authors have mentioned the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors examine realistic participation scenarios, including cyclic client participation and arbitrary participation patterns. They focus on a non-convex optimization setting, which is common in practical applications but challenging to address. To tackle this, they introduce a novel method called Amplified Scaffold, designed to effectively correct client drift—a common issue where clients' updates diverge from the global model. The Amplified Scaffold method builds upon existing techniques but enhances their effectiveness in this more complex setting. The authors not only provide theoretical convergence guarantees, ensuring that their method is mathematically sound, but also back up their claims with extensive experimental results. These experiments demonstrate the practical effectiveness of Amplified Scaffold in various scenarios, highlighting its potential for real-world applications.\n\nSTRENGTHS:\nIn this work, the authors provide a detailed convergence analysis of a modified version of the well-known Scaffold method. While the theoretical framework appears to be sound, I did not thoroughly review all the proofs, so there may be some oversights. Nonetheless, the results obtained seem normal, adequate, and align with expectations.\n\nThe authors consider a wide range of settings and sampling schemes, which makes their work relevant to a broad audience, including both researchers and practitioners. This comprehensive approach addresses a significant problem in the field, highlighting its importance for real-world applications.\n\nAdditionally, the proposed method undergoes extensive experimental testing. These experiments validate the method's effectiveness and demonstrate its practical applicability in various scenarios. The combination of theoretical analysis and experimental validation strengthens the credibility and utility of the proposed approach.\n\nWEAKNESSES:\n1) The related work section in your paper is limited and could be significantly strengthened by including discussions of several closely related methods. For instance, a closely related method to Scaffold is ProxSkip (also referred to as Scaffnew), which was proposed as an improved version of Scaffold. Additionally, the Tamuna method, which incorporates partial participation, is relevant. Including a discussion about the connection to these methods will provide a broader context and highlight the advancements in the field.\n\n- Mishchenko, Konstantin, et al. \"Proxskip: Yes! Local gradient steps provably lead to communication acceleration! Finally!\" International Conference on Machine Learning. PMLR, 2022.\n- Condat, Laurent Pierre, Grigory Malinovsky, and Peter Richtárik. \"Tamuna: Accelerated federated learning with local training and partial participation.\" (2023).\n\nFurthermore, an arbitrary sampling framework has been studied in several recent papers. It would be beneficial to compare the assumptions on the sampling schemes used in these studies with those in your work. This will help clarify the differences and similarities, as well as the advantages and limitations of the various approaches.\n\n- Tyurin, Alexander, et al. \"Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling.\" Transactions on Machine Learning Research.\n- Grudzień, Michał, Grigory Malinovsky, and Peter Richtárik. \"Improving Accelerated Federated Learning with Compression and Importance Sampling.\" Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities.\n\nAdditionally, it is worth discussing the paper that addresses optimal complexity for the non-convex federated learning setting. This paper provides insights into achieving optimal communication complexity in distributed non-convex optimization, which is highly relevant to your study.\n\n- Patel, Kumar Kshitij, et al. \"Towards optimal communication complexity in distributed non-convex optimization.\" Advances in Neural Information Processing Systems 35 (2022): 13316-13328.\n\nBy incorporating these discussions, your related work section will be more comprehensive and provide a clearer understanding of how your proposed method fits within the broader landscape of federated learning research. This will not only strengthen your paper but also demonstrate the relevance and impact of your contributions.\n\n2) The major issue in your paper is that in Table 1, the comparison between methods and data heterogeneity is defined using the condition $ \\sup_x \\left\\Vert\\nabla f_i(x) - \\nabla f(x)\\right\\Vert \\leq \\kappa $, which implies uniformly bounded heterogeneity. However, in the original Scaffold paper, the authors used a different measure for bounded gradient dissimilarity:\n$$ \\frac{1}{N} \\sum_{i=1}^N \\left\\Vert \\nabla f_i(x) \\right\\Vert^2 \\leq G^2 + B^2 \\left\\Vert \\nabla f(x) \\right\\Vert^2, \\forall x. $$\n\nThis discrepancy is significant because uniformly bounded heterogeneity is a more restrictive assumption compared to bounded gradient dissimilarity. The latter is a more common and practical assumption in federated learning literature, as it better captures the variations in real-world data distributions.\n\nI strongly recommend revising Table 1 to use the less restrictive assumption of bounded gradient dissimilarity instead of uniformly bounded heterogeneity. This will make your comparisons more relevant and applicable to a broader range of scenarios.\n\nAdditionally, the notation $\\kappa$ used for bounded heterogeneity is potentially confusing. Typically, $\\kappa$ is used to denote the condition number, $\\kappa = \\frac{L}{\\mu}$, in the context of strongly convex functions. To avoid this confusion, I suggest changing the notation for bounded heterogeneity to something more standard and distinct, which will make your paper clearer and more consistent with established conventions in the field.\n\n3) Assumption 2 is quite complex and may be challenging for readers to fully grasp. I recommend providing a more detailed discussion of the class of sampling schemes considered under this assumption. Specifically, elaborating on the types of sampling schemes included, their characteristics, and why they are relevant to your study will help clarify the underlying assumptions. This additional context will make it easier for readers to understand the scope and implications of your assumptions, thereby enhancing the clarity and impact of your paper.\n\n4) In the experimental section, the rationale behind the choice of hyperparameters is not clearly explained. To improve clarity, I recommend providing a detailed explanation of the specific hyperparameters selected for your experiments. This should include the criteria used to choose these values, how they were tuned, and any relevant considerations or trade-offs. Offering this additional context will help readers understand why particular hyperparameter settings were used and how they influence the outcomes of your experiments. This level of detail will enhance the reproducibility of your work and provide valuable insights for others seeking to build upon your research.\n\n5) The main issue with the theoretical results is that they do not generalize the Scaffold method in the i.i.d. setting. For Scaffold, the convergence rate is $ \\frac{\\Delta L}{\\varepsilon^2} \\left(\\frac{N}{S}\\right)^{\\frac{2}{3}} $, while for Amplified Scaffold, the rate is $\\frac{\\Delta L}{\\varepsilon^2} \\left(\\frac{N}{S}\\right) $. This discrepancy means that Amplified Scaffold does not recover the more favorable convergence rate of the original Scaffold method under i.i.d. conditions.\n\nIt would be a significant improvement if Amplified Scaffold could match the original Scaffold's rate in the i.i.d. setting. Demonstrating this equivalence would not only strengthen the theoretical results but also enhance the practical applicability of Amplified Scaffold. If you can achieve this and show that Amplified Scaffold can indeed recover the original Scaffold's rate in the i.i.d. setting, I am willing to increase the score. This adjustment would highlight the robustness and efficiency of the Amplified Scaffold method, making it a more compelling contribution to the field.\n\nQUESTIONS:\nPlease check the Weaknesses section.\n\nLIMITATIONS:\nThe limitations section is well-written, but it should also note that the Amplified Scaffold method does not recover the rate of the original Scaffold method in the i.i.d. setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-28"}