Title: Convergence Analysis of Sequential Federated Learning on Heterogeneous Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Convergence analysis
Abstract: There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) sequential FL (SFL), where clients train models in a sequential manner. In contrast to that of PFL, the convergence theory of SFL on heterogeneous data is still lacking. In this paper, we establish the convergence guarantees of SFL for strongly/general/non-convex objectives on heterogeneous data. The convergence guarantees of SFL are better than that of PFL on heterogeneous data with both full and partial client participation. Experimental results validate the counterintuitive analysis result that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.

Convergence Analysis of Sequential Federated Learning on Heterogeneous Data

Abstract

There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) sequential FL (SFL), where clients train models in a sequential manner. In contrast to that of PFL, the convergence theory of SFL on heterogeneous data is still lacking. In this paper, we establish the convergence guarantees of SFL for strongly/general/non-convex objectives on heterogeneous data. The convergence guarantees of SFL are better than that of PFL on heterogeneous data with both full and partial client participation. Experimental results validate the counterintuitive analysis result that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.

1 Introduction

Federated Learning (FL) (McMahan et al., 2017) is a popular distributed machine learning paradigm, where multiple clients collaborate to train a global model. To preserve data privacy and security, data must be kept in clients locally cannot be shared with others, causing one severe and persistent issue, namely 'data heterogeneity'. In cross-device FL, where data is generated and kept in massively distributed resource-constrained devices (e.g., IoT devices), the negative impact of data heterogeneity would be further exacerbated (Jhunjhunwala et al., 2023).

There are two categories of methods in FL to enable distributed training across multiple clients (Qu et al., 2022): i) parallel FL (PFL), where models are trained in a parallel manner across clients with synchronization at intervals, e.g., Federated Averaging ( ❋❡❞❆✈❣ ) (McMahan et al., 2017); and ii) sequential FL (SFL), where models are trained in a sequential manner across clients, e.g., Cyclic Weight Transfer ( ❈❲❚ ) (Chang et al., 2018). However, both categories of methods suffer from the 'client drift' (Karimireddy et al., 2020), i.e., the local updates on heterogeneous clients would drift away from the right direction, resulting in performance degradation.

Motivation. Recently, SFL (more generally, the sequential training manner, see Algorithm 1) has attracted much attention in the FL community (Lee et al., 2020). Specifically, SFL demonstrates advantages on training speed (in terms of training rounds) (Zaccone et al., 2022) and small datasets (Kamp et al., 2023), and both are crucial for cross-device FL. Furthermore, the sequential manner has played a great role in Split learning (SL) (Gupta and Raskar, 2018; Thapa et al., 2022), an emerging distributed learning technology at the edge side (Zhou et al., 2019), where the full model is split into client-side and server-side portions to alleviate the excessive computation overhead for resource-constrained devices. Appendix A will show that the convergence theory in this work is also applicable to SL.

Convergence theory is critical for analyzing the learning performance of algorithms on heterogeneous data in FL. So far, there are numerous works to analyze the convergence of PFL (Li et al., 2019; Khaled et al., 2020; Koloskova et al., 2020) on heterogeneous data. However, the convergence theory of SFL on heterogeneous data, given the complexity of its sequential training manner, has not been well investigated in the literature, with only limited preliminary empirical studies Gao et al. (2020, 2021). This paper aims to establish the convergence guarantees for SFL and compare the convergence results of PFL and SFL.

Setup. In the following, we provide some preliminaries about SFL and PFL.

Problem formulation. The basic FL problem is to minimize a global objective function:

where F m , f m and D m denote the local objective function, the loss function and the local dataset of client m ( m ∈ [ M ] ), respectively. In particular, when D m has finite data samples { ξ i m : i ∈ [ |D m | ] } , the local objective function can also be written as F m ( x ) = 1 |D m | ∑ |D m | i =1 f m ( x ; ξ i m ) .

Update rule of SFL. At the beginning of each training round, the indices π 1 , π 2 , . . . , π M are sampled without replacement from { 1 , 2 , . . . , M } randomly as the clients' training order. Within a round, each client i) initializes its model with the latest parameters from its previous client; ii) performs K steps of local updates over its local dataset; and iii) passes the updated parameters to the next client. This process continues until all clients finish their local training. Let x ( r ) m,k denote the local parameters of the m -th client (i.e., client π m ) after k local steps in the r -th round, and x ( r ) denote the global parameter in the r -th round. With SGD (Stochastic Gradient Descent) as the local solver, the update rule of SFL is as follows:

where g ( r ) π m ,k := ∇ f π m ( x ( r ) m,k ; ξ ) denotes the stochastic gradient of F π m regarding parameters x ( r ) m,k and η denotes the learning rate. See Algorithm 1. Notations are summarized in Appendix C.1.

Update rule of PFL. Within a round, each client i) initializes its model with the global parameters; ii) performs K steps of local updates; and iii) sends the updated parameters to the central server. The server will aggregate the local parameters to generate the global parameters. See Algorithm 2

In this work, unless otherwise stated, we use SFL and PFL to represent the classes of algorithms that share the same update rule as Algorithm 1 and Algorithm 2, respectively.

Algorithm 1: Sequential FL

Algorithm 2: Parallel FL

2 Contributions

Brief literature review. The most relevant work is the convergence of PFL and Random Reshuffling (SGD-RR). There are a wealth of works that have analyzed the convergence of PFL on data heterogeneity (Li et al., 2019; Khaled et al., 2020; Karimireddy et al., 2020; Koloskova et al., 2020; Woodworth et al., 2020b), system heterogeneity (Wang et al., 2020), partial client participation (Li et al., 2019; Yang et al., 2021; Wang and Ji, 2022) and other variants (Karimireddy et al., 2020; Reddi et al., 2021). In this work, we compare the convergence bounds between PFL and SFL (see Subsection 3.3) on heterogeneous data.

SGD-RR (where data samples are sampled without replacement) is deemed to be more practical than SGD (where data samples are sample with replacement), and thus attracts more attention recently. Gürbüzbalaban et al. (2021); Haochen and Sra (2019); Nagaraj et al. (2019); Ahn et al. (2020); Mishchenko et al. (2020) have proved the upper bounds and Safran and Shamir (2020, 2021); Rajput et al. (2020); Cha et al. (2023) have proved the lower bounds of SGD-RR. In particular, the lower bounds in Cha et al. (2023) are shown to match the upper bounds in Mishchenko et al. (2020). In this work, we use the bounds of SGD-RR to exam the tightness of that of SFL (see Subsection 3.2).

Recently, the shuffling-based method has been applied to FL (Mishchenko et al., 2022; Yun et al., 2022; Cho et al., 2023; Malinovsky et al., 2023). The most relevant works are FL with cyclic client participation (Cho et al., 2023) and FL with shuffling client participation (Malinovsky et al., 2023). The detailed comparisons are given in Appendix B.

Challenges. The theory of SGD is applicable to SFL on homogeneous data, where SFL can be reduced to SGD. However, the theory of SGD can be no longer applicable to SFL on heterogeneous data. This is because for any pair of indices m and k (except m = 1 and k = 0 ) within a round, the stochastic gradient is not an (conditionally) unbiased estimator of the global objective:

̸

In general, the challenges of establishing convergence guarantees of SFL mainly arise from (i) the sequential training manner across clients and (ii) multiple local steps of SGD at each client.

Sequential training manner across clients (vs. PFL). In PFL, local model parameters are updated in parallel within each round and synchronized at the end of the round. In this case, the local updates across clients are mutually independent when conditional on all the randomness prior to the round. However, in SFL, client's local updates additionally depend on the randomness of all previous clients. This makes bounding the client drift of SFL more complex than that of PFL.

Multiple local steps of SGD at each client (vs. SGD-RR). SGD-RR samples data samples without replacement and then performs one step of gradient descent (GD) on each data sample. Similarly, SFL samples clients without replacement and then performs multiple steps of SGD on each local objective (i.e., at each client). In fact, SGD-RR can be regarded as a special case of SFL. Thus, the derivation of convergence guarantees of SFL is also more complex than that of SGD-RR.

Contributions. The main contributions are as follows:

We derive convergence guarantees of SFL for strongly convex, general convex and non-convex objectives on heterogeneous data with the standard assumptions in FL in Subsection 3.2.

We compare the convergence guarantees of PFL and SFL, and find a counterintuitive comparison result that the guarantee of SFL is better than that of PFL (with both full participation and partial participation) in terms of training rounds on heterogeneous data in Subsection 3.3.

We validate our comparison result with simulations on quadratic functions (Subsection 4.1) and experiments on real datasets (Subsection 4.2). The experimental results exhibit that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.

3 Convergence theory

We consider three typical cases for convergence theory, i.e., the strongly convex case, the general convex case and the non-convex case, where all local objectives F 1 , F 2 , . . . , F M are µ -strongly convex, general convex ( µ = 0 ) and non-convex.

3.1 Assumptions

We assume that (i) F is lower bounded by F ∗ for all cases and there exists a minimizer x ∗ such that F ( x ∗ ) = F ∗ for strongly and general convex cases; (ii) each local objective function is L -smooth (Assumption 1). Furthermore, we need to make assumptions on the diversities: (iii) the assumptions on the stochasticity bounding the diversity of { f m ( · ; ξ i m ) : i ∈ [ |D m | ] } with respect to i inside each client (Assumption 2); (iv) the assumptions on the heterogeneity bounding the diversity of local objectives { F m : m ∈ [ M ] } with respect to m across clients (Assumptions 3a, 3b).

Assumption 1 ( L -Smoothness) . Each local objective function F m is L -smooth, m ∈ { 1 , 2 , . . . , M } , i.e., there exists a constant L > 0 such that ∥∇ F m ( x ) -∇ F m ( y ) ∥ ≤ L ∥ x -y ∥ for all x , y ∈ R d .

Assumptions on the stochasticity. Since both Algorithms 1 and 2 use SGD (data samples are chosen with replacement) as the local solver, the stochastic gradient at each client is an (conditionally) unbiased estimate of the gradient of the local objective function: E ξ ∼D m [ f m ( x ; ξ ) | x ] = ∇ F m ( x ) . Then we use Assumption 2 to bound the stochasticity, where σ measures the level of stochasticity.

Assumption 2. The variance of the stochastic gradient at each client is bounded:

Assumptions on the heterogeneity. Now we make assumptions on the diversity of the local objective functions in Assumption 3a and Assumption 3b, also known as the heterogeneity in FL. Assumption 3a is made for non-convex cases, where the constants β and ζ measure the heterogeneity of the local objective functions, and they equal zero when all the local objective functions are identical to each other. Further, if the local objective functions are strongly and general convex, we use one weaker assumption 3b as Koloskova et al. (2020), which bounds the diversity only at the optima.

Assumption 3a. There exist constants β 2 and ζ 2 such that

Assumption 3b. There exists one constant ζ 2 ∗ such that

where x ∗ ∈ arg min x ∈ R d F ( x ) is one global minimizer.

3.2 Convergence analysis of SFL

Theorem 1. For SFL (Algorithm 1), there exist a constant effective learning rate ˜ η := MKη and weights w r , such that the weighted average of the global parameters ¯ x ( R ) := 1 W R ∑ R r =0 w r x ( r ) ( W R = ∑ R r =0 w r ) satisfies the following upper bounds:

Strongly convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate 1 µR ≤ ˜ η ≤ 1 6 L and weights w r = (1 -µ ˜ η 2 ) -( r +1) , such that it holds that

General convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate ˜ η ≤ 1 6 L and weights w r = 1 , such that it holds that

Non-convex : Under Assumptions 1, 2, 3a, there exist a constant effective learning rate ˜ η ≤ 1 6 L ( β +1) and weights w r = 1 , such that it holds that

where D := ∥ ∥ x (0) -x ∗ ∥ ∥ for the convex cases and A := F ( x (0) ) -F ∗ for the non-convex case.

The effective learning rate ˜ η := MKη is used in the upper bounds as Karimireddy et al. (2020); Wang et al. (2020) did. All these upper bounds consist of two parts: the optimization part (the first term) and the error part (the last three terms). Setting ˜ η larger can make the optimization part vanishes at a higher rate, yet cause the error part to be larger. This implies that we need to choose an appropriate ˜ η to achieve a balance between these two parts, which is actually done in Corollary 1. Here we choose the best learning rate with a prior knowledge of the total training rounds R , as done in the previous works (Karimireddy et al., 2020; Reddi et al., 2021).

Corollary 1. Applying the results of Theorem 1. By choosing a appropriate learning rate (see the proof of Theorem 1 in Appendix D), we can obtain the convergence bounds for SFL as follows:

Strongly convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate 1 µR ≤ ˜ η ≤ 1 6 L and weights w r = (1 -µ ˜ η 2 ) -( r +1) , such that it holds that

General convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate ˜ η ≤ 1 6 L and weights w r = 1 , such that it holds that

Non-convex : Under Assumptions 1, 2, 3a, there exist a constant effective learning rate ˜ η ≤ 1 6 L ( β +1) and weights w r = 1 , such that it holds that

where O omits absolute constants, ˜ O omits absolute constants and polylogarithmic factors, D := ∥ ∥ x (0) -x ∗ ∥ ∥ for the convex cases and A := F ( x (0) ) -F ∗ for the non-convex case.

Convergence rate. By Corollary 1, for sufficiently large R , the convergence rate is determined by the first term for all cases, resulting in convergence rates of ˜ O (1 /MKR ) for strongly convex cases, O (1 / √ MKR ) for general convex cases and O (1 / √ MKR ) for non-convex cases.

SGD-RR vs. SFL. Recall that SGD-RR can be seen as one special case of SFL, where one step of GD is performed on each local objective F m (i.e, K = 1 and σ = 0 ). The bound of SFL turns to ˜ O ( Lζ 2 ∗ µ 2 MR 2 + µD 2 exp ( -µR L )) when K = 1 and σ = 0 for the strongly convex case. Then let us borrow the upper bound from Mishchenko et al. (2020)'s Corollary 1,

As we can see, the bound of SGD-RR only has an advantage on the second term (marked in red), which can be omitted for sufficiently large R . The difference on the constant µ is because their bound is for E [ ∥ x ( R ) -x ∗ ∥ 2 ] (see Stich (2019b)). Furthermore, our bound also matches the lower bound Ω ( Lζ 2 ∗ µ 2 MR 2 ) of SGD-RR suggested by Cha et al. (2023)'s Theorem 3.1 for sufficiently large R . For the general convex and non-convex cases, the bounds of SFL (when K = 1 and σ = 0 ) also match that of SGD-RR (see Mishchenko et al. (2020)'s Theorems 3, 4). These all suggest our bounds are tight. Yet a specialized lower bound for SFL is still required.

Effect of local steps. Two comments are included: i) It can be seen that local updates can help the convergence with proper learning rate choices (small enough) by Corollary 1. Yet this increases the total steps (iterations), leading to a higher computation cost. ii) Excessive local updates do not benefit the dominant term of the convergence rate. Take the strongly convex case as an example. When σ 2 µMKR ≤ Lζ 2 ∗ µ 2 MR 2 , the latter turns dominant, which is unaffected by K . In other words, when the value of K exceed ˜ Ω ( σ 2 /ζ 2 ∗ · µ/L · R ) , increasing local updates will no longer benefit the dominant term of the convergence rate. Note that the maximum value of K is affected by σ 2 /ζ 2 ∗ , µ/L and R . This analysis follows Reddi et al. (2021); Khaled et al. (2020).

3.3 PFL vs. SFL on heterogeneous data

Unless otherwise stated, our comparison is in terms of training rounds, which is also adopted in Gao et al. (2020, 2021). This comparison (running for the same total training rounds R ) is fair considering the same total computation cost for both methods.

Convergence results of PFL. We summarize the existing convergence results of PFL for the strongly convex case in Table 1. Here we slightly improve the convergence result for strongly convex cases by combining the works of Karimireddy et al. (2020); Koloskova et al. (2020). Besides, we note that to derive a unified theory of Decentralized SGD, the proofs of Koloskova et al. (2020) are different from other works focusing on PFL. So we reproduce the bounds for general convex and non-convex cases based on Karimireddy et al. (2020). All our results of PFL are in Theorem 2 (see Appendix E).

The convergence guarantee of SFL is better than PFL on heterogeneous data. Take the strongly convex case as an example. According to Table 1, the upper bound of SFL is better than that of PFL, with an advantage of 1 /M on the second and third terms (marked in red). This benefits from its sequential and shuffling-based training manner. Besides, we can also note that the upper bounds of both PFL and SFL are worse than that of Minibatch SGD.

Partial client participation. In the more challenging cross-device settings, only a small fraction of clients participate in each round. Following the works (Li et al., 2019; Yang et al., 2021), we provide the upper bounds of PFL and SFL with partial client participation as follows:

where S clients are selected randomly without replacement. There are additional terms (marked in blue) for both PFL and SFL, which is due to partial client participation and random sampling (Yang et al., 2021). It can be seen that the advantage of 1 /S (marked in red) of SFL also exists, similar to the full client participation setup.

4 Experiments

We run experiments on quadratic functions (Subsection 4.1) and real datasets (Subsection 4.2) to validate our theory. The main findings are i) in extremely heterogeneous settings, SFL performs better than PFL, ii) while in moderately heterogeneous settings, this may not be the case.

4.1 Experiments on quadratic functions

According to Table 1, SFL outperforms PFL on heterogeneous settings (in the worst case). Here we show that the counterintuitive result (in contrast to Gao et al. (2020, 2021)) can appear even for simple one-dimensional quadratic functions (Karimireddy et al., 2020).

Results of simulated experiments. As shown in Table 2, we use four groups of experiments with various degrees of heterogeneity. To further catch the heterogeneity, in addition to Assumption 3b, we also use bounded Hessian heterogeneity in Karimireddy et al. (2020):

Choosing larger values of ζ ∗ and δ means higher heterogeneity. The experimental results of Table 2 are shown in Figure 1. When ζ ∗ = 0 and δ = 0 , SFL outperforms PFL (Group 1). When ζ ∗ = 1 and δ = 0 , the heterogeneity has no bad effect on the performance of PFL while hurts that of SFL significantly (Group 2). When the heterogeneity continues to increase to δ > 0 , SFL outperforms PFL with a faster rate and better result (Groups 3 and 4). This in fact tells us that the comparison between PFL and SFL can be associated with the data heterogeneity, and SFL outperforms PFL when meeting high data heterogeneity, which coincides with our theoretical conclusion.

Limitation and intuitive explanation. The bounds (see Table 1) above suggest that SFL outperforms PFL regardless of heterogeneity (the value of ζ ∗ ), while the simulated results show that it only holds in extremely heterogeneous settings. This inconsistency is because existing theoretical works (Karimireddy et al., 2020; Koloskova et al., 2020) with Assumptions 3a, 3b may underestimate the capacity of PFL, where the function of global aggregation is omitted. In particular, Wang et al. (2022) have provided rigorous analyses showing that PFL performs much better than the bounds suggest in moderately heterogeneous settings. Hence, the comparison turns vacuous under this condition. Intuitively, PFL updates the global model less frequently with more accurate gradients (with the global aggregation). In contrast, SFL updates the global model more frequently with less accurate gradients. In homogeneous (gradients of both are accurate) and extremely heterogeneous settings (gradients of both are inaccurate), the benefits of frequent updates become dominant, and thus SFL outperforms PFL. In moderately heterogeneous settings, it's the opposite.

10 1

0

2

4

6

8

10

Training rounds

10

4

10

3

10

2

10

1

10 0

Distance to the optimum

PFL,

K

=2

PFL,

K

=10

SFL,

K

=2

SFL,

K

=10

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

4.2 Experiments on real datasets

Extended Dirichlet strategy. This is to generate arbitrarily heterogeneous data across clients by extending the popular Dirichlet-based data partition strategy (Yurochkin et al., 2019; Hsu et al., 2019). The difference is to add a step of allocating classes (labels) to determine the number of classes per client (denoted by C ) before allocating samples via Dirichlet distribution (with concentrate parameter α ). Thus, the extended strategy can be denoted by ExDir ( C, α ) . The implementation is as follows (with more details deferred to Appendix G.1):

Allocating classes: We randomly allocate C different classes to each client. After assigning the classes, we can obtain the prior distribution q c for each class c .

Allocating samples: For each class c , we draw p c ∼ Dir ( α q c ) and then allocate a p c,m proportion of the samples of class c to client m . For example, q c = [1 , 1 , 0 , 0 , . . . , ] means that the samples of class c are only allocated to the first 2 clients.

Experiments in cross-device settings. We next validate the theory in cross-device settings (Kairouz et al., 2021) with partial client participation on real datasets.

Setup. We consider the common CV tasks training VGGs (Simonyan and Zisserman, 2014) and ResNets (He et al., 2016) on CIFAR-10 (Krizhevsky et al., 2009) and CINIC-10 (Darlow et al., 2018). Specifically, we use the models VGG-9 (Lin et al., 2020) and ResNet-18 (Acar et al., 2021). We partition the training sets of CIFAR-10 into 500 clients / CINIC-10 into 1000 clients by ExDir (1 , 10 . 0) and ExDir (2 , 10 . 0) ; and spare the test sets for computing test accuracy. As both partitions share the same parameter α = 10 . 0 , we use C = 1 (where each client owns samples from one class) and C = 2 (where each client owns samples from two classes) to represent them, respectively. Note that these two partitions are not rare in FL (Li et al., 2022). They are called extremely heterogeneous data and moderately heterogeneous data respectively in this paper. We fix the number of participating clients to 10 and the mini-batch size to 20. The local solver is SGD with learning rate being constant, momentem being 0 and weight decay being 1e-4. We apply gradient clipping to both algorithms (Appendix G.2) and tune the learning rate by grid search (Appendix G.3).

The best learning rate of SFL is smaller than that of PFL. We have the following observations from Figure 2: i) the best learning rates of SFL is smaller than that of PFL (by comparing PFL and SFL), and ii) the best learning rate of SFL becomes smaller as data heterogeneity increases (by comparing the top row and bottom row). These observations are critical for hyperparameter selection.

Effect of local steps. Figure 3 is aimed to study the effects of local steps. In both plots, it can be seen that the performance of SFL improves as K increases from 1 to 5. This validates the theoretical conclusion that local steps can help the convergence of SFL even on heterogeneous data. Then, the performance of SFL deteriorates as K increases from 5 to 10, whereas the upper bound of SFL always diminishes as long as K increases. This is because when K exceeds one threshold, the dominant term of the upper bound will be immune to its change as stated in Subsection 3.2. Then, considering 'catastrophic forgetting' (Kirkpatrick et al., 2017; Sheller et al., 2019) problems in SFL, it can be expected to see such phenomenon.

SFL outperforms PFL on extremely heterogeneous data. The test accuracy results for various tasks are collected in Table 3. When C = 1 (extremely heterogeneous), the performance of SFL is better than that of PFL across all

-2.5-2.0-1.5-1.0-0.5

0

20

40

60

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

10

20

30

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

10

20

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

C

=2,

K

=50

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=1

SFL,

K

=1

SFL,

K

=5

SFL,

K

=20

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=2

SFL,

K

=1

SFL,

K

=5

SFL,

K

=20

tried settings. When C = 2 (moderately heterogeneous), PFL can achieve the close or even slightly better performance than SFL in some cases (e.g., CIFAR-10/ C = 2 / K = 50 ). This is consistent with our observation and analysis in Subsection 4.1. Notably, on the more complicated dataset CINIC-10, SFL shows better for all settings, which may be due to higher heterogeneity.

5 Conclusion

In this paper, we have derived the convergence guarantees of SFL for strongly convex, general convex and non-convex objectives on heterogeneous data. Furthermore, we have compared SFL against PFL, showing that the guarantee of SFL is better than PFL on heterogeneous data. Experimental results validate that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.

Future directions include i) lower bounds for SFL (this work focuses on the upper bounds of SFL), ii) other potential factors that may affect the performance of PFL and SFL (this work focuses on data heterogeneity) and iii) new algorithms to facilitate our findings (no new algorithm in this work).

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2021YFB2900302, in part by the National Science Foundation of China under Grant 62001048, and in part by the Fundamental Research Funds for the Central Universities under Grant 2242022k60006.

We thank the reviewers in NeurIPS 2023 for the insightful suggestions. We thank Sai Praneeth Karimireddy for helping us clear some doubts when proving the bounds.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations , 2021. URL ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂ ❇✼✈✹◗▼❘✻❩✾✇ .

Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. Sgd with shuffling: optimal rates without component convexity and large epoch requirements. Advances in Neural Information Processing Systems , 33:17526-17535, 2020.

Ghadir Ayache and Salim El Rouayheb. Private weighted random walk stochastic gradient descent. IEEE Journal on Selected Areas in Information Theory , 2(1):452-463, 2021.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Jaeyoung Cha, Jaewook Lee, and Chulhee Yun. Tighter lower bounds for shuffling sgd: Random permutations and beyond. arXiv preprint arXiv:2303.07160 , 2023.

Ken Chang, Niranjan Balachandar, Carson Lam, Darvin Yi, James Brown, Andrew Beers, Bruce Rosen, Daniel L Rubin, and Jayashree Kalpathy-Cramer. Distributed deep learning networks

among institutions for medical imaging. Journal of the American Medical Informatics Association , 25(8):945-954, 2018.

Yae Jee Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, and Tong Zhang. On the convergence of federated averaging with cyclic client participation. arXiv preprint arXiv:2302.03109 , 2023.

Edwige Cyffers and Aurélien Bellet. Privacy amplification by decentralization. In International Conference on Artificial Intelligence and Statistics , pages 5334-5353. PMLR, 2022.

Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Yansong Gao, Minki Kim, Sharif Abuadbba, Yeonjae Kim, Chandra Thapa, Kyuyeon Kim, Seyit A Camtep, Hyoungshick Kim, and Surya Nepal. End-to-end evaluation of federated learning and split learning for internet of things. In 2020 International Symposium on Reliable Distributed Systems (SRDS) , pages 91-100. IEEE, 2020.

Yansong Gao, Minki Kim, Chandra Thapa, Sharif Abuadbba, Zhi Zhang, Seyit Camtepe, Hyoungshick Kim, and Surya Nepal. Evaluation and optimization of distributed machine learning techniques for internet of things. IEEE Transactions on Computers , 2021.

Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235 , 2023.

Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications , 116:1-8, 2018.

Mert Gürbüzbalaban, Asu Ozdaglar, and Pablo A Parrilo. Why random reshuffling beats stochastic gradient descent. Mathematical Programming , 186:49-84, 2021.

Jeff Haochen and Suvrit Sra. Random shuffling beats sgd after finite epochs. In International Conference on Machine Learning , pages 2624-2633. PMLR, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. In The Eleventh International Conference on Learning Representations , 2023. URL ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂■/a80/a114③◆❜❞❞❳❱ .

Björn Johansson, Maben Rabi, and Mikael Johansson. A randomized incremental subgradient method for distributed optimization in networked systems. SIAM Journal on Optimization , 20 (3):1157-1170, 2010.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Michael Kamp, Jonas Fischer, and Jilles Vreeken. Federated learning from small datasets. In The Eleventh International Conference on Learning Representations , 2023. URL ❤/a116/a116♣/a115✿ ✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂❤❉❉❱✶❧/a115❘❱✽ .

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 4519-4529. PMLR, 2020.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report , 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Jin-woo Lee, Jaehoon Oh, Sungsu Lim, Se-Young Yun, and Jae-Gil Lee. Tornadoaggregate: Accurate and scalable federated learning via the ring-based architecture. arXiv preprint arXiv:2012.03214 , 2020.

Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 965-978. IEEE, 2022.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2019.

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems , 33:2351-2363, 2020.

Grigory Malinovsky, Samuel Horváth, Konstantin Burlachenko, and Peter Richtárik. Federated learning with regularized client participation. arXiv preprint arXiv:2302.03662 , 2023.

Xianghui Mao, Kun Yuan, Yubin Hu, Yuantao Gu, Ali H Sayed, and Wotao Yin. Walkman: A communication-efficient random-walk algorithm for decentralized optimization. IEEE Transactions on Signal Processing , 68:2513-2528, 2020.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Simple analysis with vast improvements. Advances in Neural Information Processing Systems , 33:17309-17320, 2020.

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Proximal and federated random reshuffling. In International Conference on Machine Learning , pages 15718-15749. PMLR, 2022.

Dheeraj Nagaraj, Prateek Jain, and Praneeth Netrapalli. Sgd without replacement: Sharper rates for general smooth convex functions. In International Conference on Machine Learning , pages 4703-4711. PMLR, 2019.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10061-10071, 2022.

Shashank Rajput, Anant Gupta, and Dimitris Papailiopoulos. Closing the convergence gap of sgd without replacement. In International Conference on Machine Learning , pages 7964-7973. PMLR, 2020.

S Sundhar Ram, A Nedi´ c, and Venugopal V Veeravalli. Incremental stochastic subgradient algorithms for convex optimization. SIAM Journal on Optimization , 20(2):691-717, 2009.

Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations , 2021. URL ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂ ▲❦❋●✸❧❇✶✸❯✺ .

Itay Safran and Ohad Shamir. How good is sgd with random shuffling? In Conference on Learning Theory , pages 3250-3284. PMLR, 2020.

Itay Safran and Ohad Shamir. Random shuffling beats sgd only after many epochs on ill-conditioned problems. Advances in Neural Information Processing Systems , 34:15151-15161, 2021.

Micah J Sheller, G Anthony Reina, Brandon Edwards, Jason Martin, and Spyridon Bakas. Multiinstitutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part I 4 , pages 92-104. Springer, 2019.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations , 2019a. URL ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂ ❙✶❣✷❏♥❘❝❋❳ .

Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232 , 2019b.

Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed communication. arXiv preprint arXiv:1909.05350 , 2019.

Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, and Seyit A Camtepe. Advancements of federated learning towards privacy preservation: from federated learning to split learning. In Federated Learning Systems , pages 79-109. Springer, 2021.

Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8485-8493, 2022.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. The Journal of Machine Learning Research , 22(1):9709-9758, 2021.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623, 2020.

Jianyu Wang, Rudrajit Das, Gauri Joshi, Satyen Kale, Zheng Xu, and Tong Zhang. On the unreasonable effectiveness of federated averaging with heterogeneous data. arXiv preprint arXiv:2206.04723 , 2022.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. Advances in Neural Information Processing Systems , 35:19124-19137, 2022.

Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334-10343. PMLR, 2020a.

Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems , 33:6281-6292, 2020b.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-IID federated learning. In International Conference on Learning Representations , 2021. URL ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂❥❉❞③❤✺✉❧✲❞ .

Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond. In International Conference on Learning Representations , 2022. URL ❤/a116/a116♣/a115✿✴✴♦♣❡♥/a114❡✈✐❡✇✳♥❡/a116✴❢♦/a114✉♠❄✐❞❂▲❞❧✇❜❇/a80✷♠❧/a113 .

Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning , pages 7252-7261. PMLR, 2019.

Riccardo Zaccone, Andrea Rizzardi, Debora Caldarola, Marco Ciccone, and Barbara Caputo. Speeding up heterogeneous federated learning with sequentially trained superclients. In 2022 26th International Conference on Pattern Recognition (ICPR) , pages 3376-3382. IEEE, 2022.

Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning framework. arXiv preprint arXiv:2107.11621 , 2021.

Fan Zhou and Guojing Cong. On the convergence properties of a k -step averaging stochastic gradient descent algorithm for nonconvex optimization. arXiv preprint arXiv:1708.01012 , 2017.

Xingyu Zhou. On the fenchel duality between strong convexity and lipschitz continuous gradient. arXiv preprint arXiv:1803.06573 , 2018.

Zhi Zhou, Xu Chen, En Li, Liekang Zeng, Ke Luo, and Junshan Zhang. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proceedings of the IEEE , 107(8): 1738-1762, 2019.

Appendix

A Applicable to Split Learning

Split Learning is proposed to address the computation bottleneck of resource-constrained devices, where the full model is split into two parts: the client-side model (front-part) and the server-side model (back-part). There are two typical algorithms in SL, i.e., Sequential Split Learning (SSL) 2 (Gupta and Raskar, 2018) and Split Federated Learning (SplitFed) 3 (Thapa et al., 2022). The overviews of these four paradigms are illustrated in Figure 4

Communication

data

Client

smashed

data

loss

Server

...

gradient

client-side model

data

Client

...

Communication

Server

Avg

Client

full model

...

Client

data

loss

data

loss

Client

Client

SSL

PFL

data

Client

smashed

data

gradient

server-side model

data

Client

Communication

loss

loss

client-side model

Main Server

...

Avg

...

Avg

Fed Server

...

SplitFed

...

Communication

Client

full model

Client

data

loss

data

loss

SFL

Client

Client

Sequential

Parallel

Training process of SSL. Each client keeps one client-side model and the server keeps one serverside model. Thus, each client and the server can collaborate to complete the local training of a full model (over the local data kept in clients). Each local step of local training includes the following operations: 1) the client executes the forward pass on the local data, and sends the activations of the cut-layer (called smashed data ) and labels to the server; 2) the server executes the forward pass with received the smashed data and computes the loss with received labels; 3) the server executes the backward pass and send the gradients of the smashed data to the client; 4) the client executes the backward pass with the received gradients. After finishing the local training, the client sends the updated parameters of its client-side model to the next client. This process continues until all clients complete their local training. See the bottom-left subfigure and Algorithms 3, 4. For clarity, we adjust the superscripts and subscripts in this section, and provide the required notations in Table 4.

Training process of SplitFed. Each client keeps one client-side model and the server keeps (named as main server ) keeps multiple server-side models, whose quantity is the same as the number of clients. Thus, each client and its corresponding server-side model in the main server can complete the local training of a full model in parallel. The local training operations of each client in SplitFed are identical to that in SSL. After the local training with the server, clients send the updated parameters to the fed server, one server introduced in SplitFed to achieve the aggregation of client-side models. The fed server aggregates the received parameters and sends the aggregated (averaged) parameters to the clients. The main server also aggregates the parameters of server-side models it kept and updates them accordingly. See the bottom-right subfigure and Thapa et al. (2021)'s Algorithm 2.

Applicable to SL. According to the complete training process of SSL and SplitFed, we can conclude the relationships between SFL and SSL, and, PFL and SplitFed as follows:

SSL and SplitFed can be viewed as the practical implementations of SFL and PFL respectively in the context of SL.

SSL and SplitFed share the same update rules with SFL (Algorithm 1) and PFL (Algorithm 2) respectively, and hence, the same convergence results.

Algorithm 3: Sequential Split Learning (Server-side operations)

Main Server executes:

B Related work

Convergence of PFL. The convergence of PFL (also known as Local SGD, ❋❡❞❆✈❣ ) has developed rapidly recently, with weaker assumptions, tighter bounds and more complex scenarios. Zhou and Cong (2017); Stich (2019a); Khaled et al. (2020); Wang and Joshi (2021) analyzed the convergence of PFL on homogeneous data. Li et al. (2019) derived the convergence guarantees for PFL with the bounded gradients assumption on heterogeneous data. Yet this assumption has been shown too stronger (Khaled et al., 2020). To further catch the heterogeneity, Karimireddy et al. (2020); Koloskova et al. (2020) assumed the variance of the gradients of local objectives is bounded either uniformly (Assumption 3a) or on the optima (Assumption 3b). Moreover, Li et al. (2019); Karimireddy et al. (2020); Yang et al. (2021) also consider the convergence with partial client participation. The lower bounds of PFL are also studied in Woodworth et al. (2020a,b); Yun et al. (2022). There are other variants in PFL, which show a faster convergence rate than the vanilla one (Algorithm 2), e.g., ❙❈❆❋❋❖▲❉ (Karimireddy et al., 2020).

Convergence of SGD-RR. Random Reshuffling (SGD-RR) has attracted more attention recently, as it (where data samples are sampled without replacement) is more common in practice than its counterpart algorithms SGD (where data samples are sample with replacement). Early works (Gürbüzbalaban et al., 2021; Haochen and Sra, 2019) prove the upper bounds for strongly convex and twice-smooth objectives. Subsequent works (Nagaraj et al., 2019; Ahn et al., 2020; Mishchenko et al., 2020) further prove upper bounds for strongly convex, convex and non-convex cases. The lower bounds of SGD-RR are also investigated in the quadratic case Safran and Shamir (2020, 2021) and the strongly convex case (Rajput et al., 2020; Cha et al., 2023). In particular, the lower bounds in Cha et al. (2023) are shown to match the upper bounds in Mishchenko et al. (2020) for the both strongly convex and general convex cases. These works have reached a consensus that SGD-RR is better than SGD a least when the number of epochs (passes over the data) is large enough. In this paper, we use the bounds of SGD-RR to exam the tightness of our bounds of SFL.

There are also works studying the randomized incremental gradient methods are also relevant (Ram et al., 2009; Johansson et al., 2010; Ayache and El Rouayheb, 2021; Mao et al., 2020; Cyffers and Bellet, 2022), which consider a single update at each client and focus on random walks.

Shuffling-based methods in FL. Recently, shuffling-based methods have appeared in FL (Mishchenko et al., 2022; Yun et al., 2022; Cho et al., 2023). Mishchenko et al. (2022) gave the convergence result of Federated Random Reshuffling (FedRR) as a application to Federated Learning of their theory for Proximal Random Reshuffling (ProxRR). Yun et al. (2022) analyzed the convergence of Minibatch RR and Local RR, the variants of Minibatch SGD and Local SGD (Local SGD is equivalent to PFL in this work), where clients perform SGD-RR locally (in parallel) instead of SGD. Both FedRR and Local RR are different from SFL from the algorithm perspective. See Yun et al. (2022)'s Appendix A for comparison.

The most relevant works are FL with cyclic client participation (Cho et al., 2023) and FL with shuffling client participation (Malinovsky et al., 2023) (we note them when preparing this version).

Round 1

Round 2

Round 3

w

(

k,

2)

w

(

k,

1)

w

(

k,

0)

w

(

k,K

)

w

(

k

+1

,

0)

=

w

(

k,K

)

,

after the

k

th

cycle-epoch of

K

rounds

<latexi sh

1_b

64="K0

kdo

mEwP

g

Ay

35

>

c

VDLS

N

F

W

r

q

JU

I

Mu

G

T

z

9

f

O2

Z

8

v

jC

n

YH

X

p

R

Q

<latexi sh

1_b

64="j

G

OLKZE

gPoU+z5I

>

C

X

c

VBN

M

u

yFQS

v

w

k

R

pm0

7

r

W

H

9

D

3

2

8

J

/

d

T

f

n

q

<latexi sh

1_b

64="mLU2

uTf7

Dw/o

X

+

OgWR

I

>A

B

H

c

V

NS

M

E

3

r

n

q

vFY

5

9k

0

P

p

z

Z

d

y

C

K

8

J

Q

j

G

<latexi sh

1_b

64="9n5cr dTF

m

XBG2p

o

g8ZM

>A

+3

VDLS

N

W

q

w

JUk

KI

u

7

P

0

zE

/

f

O

y

v

jC

YH

R

Q

<latexi sh

1_b

64="rq3

Y

v

j

G7po

f

8

M

>A

B+H

c

V

NSw

E

W

z

CR

U9k

LX

/Q

Td

ZJ

m

u

FD

2

y

g

O

0

n

K

5

I

P

<latexi sh

1_b

64="u8+dH

q

n

D

VZBG

E

>A

3

c

LS

N

F

2pr

f

7

g

k

o

wT

R

m

MK

v

X

0

I

P

C

5

Y

O

Q

/

<latexi sh

1_b

64="wVYP

qXv

o

MjJ

/NDB

3

F

>A

+H

c

S

E

u

R

7

In

5

m

8

z

p

k

U

0

K

g

Ly

Z

d

G

C

T

2

Q

9

O

f

<latexi sh

1_b

64="

mK

dXSC

k

DrI

G0V

P

7

fg

>A

Bn

c

L

N

FJ3U

2vqE

R

o

Q

p

W

u

H

M

TZ

58

/

z

9

+

Y

j

y

w

O

Discussion about FL with cyclic client participation. Cho et al. (2023) consider the scenario where the total M clients are divided into K non-overlapping client groups such that each group contains M/K clients. In each training round, the sever selects a subset of N clients from a group without replacement for training in this round. One example is shown in Figure 5. As said in the paragraph 'Cyclic Client Participation (CyCP)' in their Section 3 (Problem Formulation), the groups' training

order of FL with cyclic client participation is pre-determined and fixed. In contrast, the clients' training order of SFL (precisely, Algorithm 1) will be shuffled at the beginning of each round.

It can be verified by Cho et al. (2023)'s Theorem 2. Letting K = 1 and N = M/K = M and K = M and N = M/K = 1 , we get the bounds for PFL and SFL, respectively:

where we have omitted the optimization term and changed their notations to ours (change α to 0, γ to ζ , ν to ζ , T to R ).

where we have omitted the optimization term and changed their notations to ours (change α to ζ , γ to 0 , ν to ζ , T to MR ). As we can see, we do not see a clear advantage of SFL like ours.

Clients

VI.

Comm. Round 1

Comm.Round 2

Comm.Round 3

2

DoubleShuffling

(C=2, M=6,N=3)

3

123

1

1

23

4

Local steps

Local steps

Local steps

5

VIO

Server

6

Discussion about FL with shuffling client participation. At the beginning of each meta epoch, FL with shuffling client participation partition all M clients into M/C cohorts, each with size C . These cohorts are obtained using the without replacement sampling of clients. Each meta epoch contains R = M/C communication rounds. At each communication round, clients in a cohort participate in the training process. One example is shown in Figure 6. It is noteworthy that SGD-RR is used as the local solver in Malinovsky et al. (2023) while SGD is used in this paper.

Letting C = 1 , R = M , T = R in their Theorem 6.1, we can get the bounds for SFL:

where we only considered the heterogeneity terms (let σ 2 ⋆ = 0 for Malinovsky et al. (2023) and σ = 0 for ours) and changed their notations to ours (change γ to η , N to K , ˜ σ ⋆ to ζ ∗ ). Their bound almost matches ours, with some differences on the constants µ , L and restrictions on η , which is caused by using different local solvers. However, their results are limited to the case where the number of local steps equals the size of the local dataset. It is still uncertain whether their results for SGD-RR can be generalized to situations with varying numbers of local steps.

C Notations and technical lemmas

C.1 Notations

Table 5 summarizes the notations appearing in this paper.

C.2 Basic identities and inequalities

These identities and inequalities are mostly from Zhou (2018); Khaled et al. (2020); Mishchenko et al. (2020); Karimireddy et al. (2020); Garrigos and Gower (2023).

For any random variable x , letting the variance can be decomposed as

In particular, its version for vectors with finite number of values gives

where vectors x 1 , . . . , x n ∈ R d are the values of x and their average is ¯ x = 1 n ∑ n i =1 x i .

Jensen's inequality. For any convex function h and any vectors x 1 , . . . , x n we have

As a special case with h ( x ) = ∥ x ∥ 2 , we obtain

Smoothness and general convexity, strong convexity. There are some useful inequalities with respect to L -smoothness (Assumption 1), convexity and µ -strong convexity. Their proofs can be found in Zhou (2018); Garrigos and Gower (2023).

Bregman Divergence associated with function h and arbitrary x , y is denoted as

When the function h is convex, the divergence is strictly non-negative. A more formal definition can be found in Orabona (2019). One corollary (Chen and Teboulle, 1993) called three-point-identity is,

where x , y , z is three points in the domain.

Let h be L -smooth. With the definition of Bregman divergence, a consequence of L -smoothness is

Further, If h is L -smooth and lower bounded by h ∗ , then

If h is L -smooth and convex (The definition of convexity can be found in Boyd et al. (2004)), then

The function h : R d → R is µ -strongly convex if and only if there exists a convex function g : R d → R such that h ( x ) = g ( x ) + µ 2 ∥ x ∥ 2 .

If h is µ -strongly convex, it holds that

C.3 Technical lemmas

Lemma 1 (Karimireddy et al. (2020)) . Let { ξ i } n i =1 be a sequence of random variables. And the random sequence { x i } n i =1 satisfy that x i ∈ R d is a function of ξ i , ξ i -1 , . . . , ξ 1 for all i . Suppose that the conditional expectation is E ξ i [ x i | ξ i -1 , . . . ξ 1 ] = e i (i.e., the vectors { x i -e i } form a martingale difference sequence with respect to { ξ i } ), and the variance is bounded by

Proof. This conclusion has appeared in Stich and Karimireddy (2019)'s Lemma 15, Karimireddy et al. (2020)'s Lemma 4 (separating mean and variance) and Wang et al. (2020)'s Lemma 2, which is useful for bounding the stochasticity.

̸

Without loss of generality, we can assume that i < j . Then the cross terms in the preceding equation can be computed by the law of total expectation:

Here E [( x j -e j ) | ξ i , . . . , ξ 1 ] = 0 can be proved by mathematical induction and the law of total expectation. Then,

which is the claim of this lemma. Note that since E ξ i [ x i | ξ i -1 , . . . ξ 1 ] = e i , the conditional expectation e i is not deterministic but a function of ξ i -1 , . . . , ξ 1 .

Lemma2 (Karimireddy et al. (2020)) . The following holds for any L -smooth and µ -strongly convex function h , and any x , y , z in the domain of h :

Proof. Using the three-point-identity , we get

Then, we get the following two inequalities using smoothness and strong convexity of h :

Further, using Jensen's inequality (i.e., ∥ y -z ∥ 2 ≤ 2( ∥ x -z ∥ 2 + ∥ y -x ∥ 2 ) ), we have

Combining all the inequalities together we have

which is the claim of this lemma.

Lemma 3 (Simple Random Sampling) . Let x 1 , x 2 , . . . , x n be fixed units (e.g., vectors). The population mean and population variance are give as

Draw s ∈ [ n ] = { 1 , 2 , . . . , n } random units x π 1 , x π 2 , . . . x π s randomly from the population. There are two possible ways of simple random sampling, well known as 'sampling with replacement (SWR)' and 'sampling without replacement (SWOR)'. For these two ways, the expectation and variance of the sample mean x π := 1 s ∑ s p =1 x π p satisfies

Proof. The proof of this lemma is mainly based on Mishchenko et al. (2020)'s Lemma 1 (A lemma for sampling without replacement) and Wang et al. (2020)'s Appendix G (Extension: Incorporating Client Sampling). Since the probability of each unit being selected equals 1 n in each draw, we can get the expectation and variance of any random unit x π p at the p -th draw:

where the preceding equations hold for both sampling ways. Thus, we can compute the expectations of the sample mean for both sampling ways as

which indicates that the sample means for both ways are unbiased estimators of the population mean. The variance of the sample mean can be decomposed as

̸

Next, we deal with these two ways separately:

SWR: It holds that Cov( x π p , x π q ) = 0 , ∀ p = q since x π p , x π q are independent for SWR. Thus, we can get E ∥ x π -x ∥ 2 = 1 s 2 ∑ s p =1 Var( x π p ) = ζ 2 s .

̸

SWOR: For p = q , we have

̸

Since there are n ( n -1) possible combinations of ( x π p , x π q ) and each has the same probability, we get Pr( x π p = x i , x π q = x j ) = 1 n ( n -1) . As a consequence, we have

̸

Thus we have E ∥ x π -x ∥ 2 = ζ 2 s -s ( s -1) s 2 · ζ 2 n -1 = ( n -s ) s ( n -1) ζ 2 .

When n is infinite (or large enough), we get ( E ∥ x π -x ∥ 2 ) SWOR ( E ∥ x π -x ∥ 2 ) SWR ≈ 1 -s n . This constant has appeared in Karimireddy et al. (2020)'s Lemma 7 (one round progress) and Woodworth et al. (2020b)'s Section 7 (Using a Subset of Machines in Each Round).

Lemma 4. Under the same conditions of Lemma 3, use the way 'sampling without replacement' and let b m,k ( i ) = { K -1 , i ≤ m -1 k -1 , i = m . Then for S ≤ M ( M ≥ 2 ), it holds that

Proof. Let us focus on the term in the following:

̸

For the first term on the right hand side in (26), using (23), we have

For the second term on the right hand side in (26), we have

For the third term on the right hand side in (26), we have

where we use (24) in the last equality, since i ∈ { 1 , 2 , . . . , m -1 } ̸ = m . With these three preceding equations, we get

Then summing the preceding terms over m and k , we can get

Then applying the fact that ∑ K -1 k =1 k = ( K -1) K 2 and ∑ K -1 k =1 k 2 = ( K -1) K (2 K -1) 6 , we can simplify the preceding equation as

which is the claim of this lemma.

D Proofs of Theorem 1

In this section, we provide the proof of Theorem 1 for the strongly convex, general convex and non-convex cases in D.1, D.2 and D.3, respectively.

In the following proof, we consider the partial client participation setting. So we assume that π = { π 1 , π 2 , . . . , π M } is a permutation of { 1 , 2 , . . . , M } in a certain training round and only the first S selected clients { π 1 , π 2 , . . . , π S } will participate in this round. Without otherwise stated, we use E [ · ] to represent the expectation with respect to both types of randomness (i.e., sampling data samples ξ and sampling clients π ).

D.1 Strongly convex case

D.1.1 Finding the recursion

Lemma 5. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are µ -strongly convex. If the learning rate satisfies η ≤ 1 6 LSK , then it holds that

Proof. According to Algorithm 1, the overall model updates of SFL after one complete training round (with S clients selected for training) is

where g ( r ) π m ,k = ∇ f π m ( x ( r ) m,k ; ξ ) is the stochastic gradient of F π m regarding the vector x ( r ) m,k . Thus,

In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

We start by substituting the overall updates:

We can apply Lemma 2 with x = x m,k , y = x ∗ , z = x and h = F π m for the second term on the right hand side in (28):

For the third term on the right hand side in (28), using Jensen's inequality, we have

Seeing the data sample ξ m,k , the stochastic gradient g π m ,k , the gradient ∇ F π m ( ξ m,k ) as ξ i , x i , e i in Lemma 1 respectively and applying the result of Lemma 1, the first term on the right hand side in (30) can be bounded by 4 SKσ 2 . For the second term on the right hand side in (30), we have

For the third term on the right hand side in (30), we have

where the last inequality is because E [ D F πm ( x , x ∗ ) ] = D F ( x , x ∗ ) . The forth term on the right hand side in (30) can be bounded by Lemma 3 as follows:

With the preceding four inequalities, we can bound the third term on the right hand side in (28):

Then substituting (29) and (32) into (28), we have

where we use the condition that η ≤ 1 6 LSK in the last inequality. The claim of this lemma follows after recovering the superscripts and taking unconditional expectation.

D.1.2 Bounding the client drift with Assumption 3b

Similar to the 'client drift' in PFL (Karimireddy et al., 2020), we define the client drift in SFL:

Lemma 6. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are µ -strongly convex. If the learning rate satisfies η ≤ 1 6 LSK , then the client drift is bounded:

Proof. According to Algorithm 1, the model updates of SFL from x ( r ) to x ( r ) m,k is

with b m,k ( i ) := { K -1 , i ≤ m -1 k -1 , i = m . In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

We use Jensen's inequality to bound the term E [ ∥ x m,k -x ∥ 2 ] = η 2 E [ ∥ ∥ ∥ ∑ m i =1 ∑ b m,k ( i ) j =0 g π i ,j ∥ ∥ ∥ 2 ] :

Applying Lemma 1 to the first term and Jensen's inequality to the second, third terms on the right hand side in the preceding inequality respectively, we can get

where B m,k := ∑ m i =1 ∑ b m,k ( i ) j =0 1 = ( m -1) K + k . The first term on the right hand side in (35) is bounded by 4 B m,k η 2 σ 2 . For the second term on the right hand side in (35), we have

For the third term on the right hand side in (35), we have

As a result, we can get

Then, returning to E r := ∑ S m =1 ∑ K -1 k =0 E [ ∥ x m,k -x ∥ 2 ] , we have

Applying Lemma 4 with x π i = ∇ F π i ( x ∗ ) and x = ∇ F ( x ∗ ) = 0 and the fact that

we can simplify the preceding inequality:

After rearranging the preceding inequality, we get

Finally, using the condition that η ≤ 1 6 LSK , which implies 1 -2 L 2 S 2 K 2 η 2 ≥ 8 9 , we have

The claim follows after recovering the superscripts and taking unconditional expectations.

D.1.3 Tuning the learning rate

Here we make a clear version of Karimireddy et al. (2020)'s Lemma 1 (linear convergence rate) based on Stich (2019b); Stich and Karimireddy (2019)'s works.

Lemma 7 (Karimireddy et al. (2020)) . Two non-negative sequences { r t } t ≥ 0 , { s t } t ≥ 0 , which satisfies the relation

for all t ≥ 0 and for parameters b > 0 , a, c ≥ 0 and non-negative learning rates { γ t } t ≥ 0 with γ t ≤ 1 d , ∀ t ≥ 0 , for a parameter d ≥ a , d > 0 .

Selection of weights for average. Then there exists a constant learning rate γ t = γ ≤ 1 d and the weights w t := (1 -aγ ) -( t +1) and W T := ∑ T t =0 w t , making it hold that:

Tuning the learning rate carefully. By tuning the learning rate in (38) , for ( T +1) ≥ 1 2 aγ , we have

Proof. We start by rearranging (37) and multiplying both sides with w t :

By summing from t = 0 to t = T , we obtain a telescoping sum:

and hence

Note that the proof of Stich (2019b)'s Lemma 2 used W T ≥ w T = (1 -aγ ) -( T +1) to estimate W T . It is reasonable given that w T is extremely larger than all the terms w t ( t < T ) when T is large. Yet Karimireddy et al. (2020) goes further, showing that W T can be estimated more precisely:

When ( T +1) ≥ 1 2 aγ , (1 -aγ ) T +1 ≤ exp( -aγ ( T +1)) ≤ e -1 2 ≤ 2 3 , so it follows that

With the estimates

we can further simplify (40):

which is the first result of this lemma.

Now the lemma follows by carefully tuning γ in (38). Consider the two cases:

aT

If otherwise 1 2 a ( T +1) ≤ 1 d ≤ ln(max { 2 ,a 2 r 0 T/c } ) aT (Note 1 2 a ( T +1) ≤ ln(2) aT ≤ ln(max { 2 ,a 2 r 0 T/c } ) aT ) then we pick γ = 1 d and get that

Combining these two cases, we get

Note that this lemma holds when ( T +1) ≥ 1 2 aγ , so it restricts the value of T , while there is no such restriction in Stich (2019b)'s Lemma 2.

D.1.4 Proof of strongly convex case of Theorem 1 and Corollary 1

Proof of the strongly convex case of Theorem 1. Substituting (34) into (27) and using η ≤ 1 6 LSK , we can simplify the recursion as,

Let ˜ η = MKη , we have

Applying Lemma 7 with t = r ( T = R ), γ = ˜ η , r t = E [ ∥ ∥ x ( r ) -x ∗ ∥ ∥ 2 ] , a = µ 2 , b = 1 3 , s t = E [ D F ( x ( r ) , x ∗ ) ] , w t = (1 -µ ˜ η 2 ) -( r +1) , c 1 = 4 σ 2 SK + 4( M -S ) ζ 2 ∗ S ( M -1) , c 2 = 6 Lσ 2 SK + 6 Lζ 2 ∗ S and 1 d = 1 6 L ( ˜ η = MKη ≤ 1 6 L ), it follows that

where ¯ x ( R ) = 1 W R ∑ R r =0 w r x ( r ) and we use Jensen's inequality ( F is convex) in the first inequality. Note that there are no terms containing γ 3 in Lemma 7. As the terms containing γ 3 is not the determinant factor for the convergence rate, Lemma 7 can also be applied to this case (Karimireddy et al., 2020; Koloskova et al., 2020). Thus, by tuning the learning rate carefully, we get

where D := ∥ ∥ x (0) -x ∗ ∥ ∥ . Eq. (42) and Eq. (43) are the upper bounds with partial client participation. In particular, when S = M , we can get the claim of the strongly convex case of Theorem 1 and Corollary 1.

D.2 General convex case

D.2.1 Tuning the learning rate

Lemma 8 (Koloskova et al. (2020)) . Two non-negative sequences { r t } t ≥ 0 , { s t } t ≥ 0 , which satisfies the relation

for all t ≥ 0 and for parameters b > 0 , c 1 , c 2 ≥ 0 and non-negative learning rates { γ t } t ≥ 0 with γ t ≤ 1 d , ∀ t ≥ 0 , for a parameter d > 0 .

Selection of weights for average. Then there exists a constant learning rate γ = γ t ≤ 1 d and the weights w t = 1 and W T = ∑ T t =0 w t , making it hod that:

Tuning the learning rate carefully. By tuning the learning rate carefully in (44) , we have

Proof. For constant learning rates γ t = γ we can derive the estimate

which is the first result (44) of this lemma. Let r 0 γ ( T +1) = c 1 γ and r 0 γ ( T +1) = c 2 γ 2 , yielding two choices of γ , γ = ( r 0 c 1 ( T +1) ) 1 2 and γ = ( r 0 c 2 ( T +1) ) 1 3 . Then choosing γ = min { ( r 0 c 1 ( T +1) ) 1 2 , ( r 0 c 2 ( T +1) ) 1 3 , 1 d } ≤ 1 d , there are three cases:

Combining these three cases, we get the second result of this lemma.

D.2.2 Proof of general convex case of Theorem 1 and Corollary 1

Proof of the general convex case of Theorem 1. Letting µ = 0 in (41), we get the recursion of the general convex case,

Applying Lemma 8 with t = r ( T = R ), γ = ˜ η , r t = E [ ∥ ∥ x ( r ) -x ∗ ∥ ∥ 2 ] , b = 1 3 , s t = E [ D F ( x ( r ) , x ∗ ) ] , w t = 1 , c 1 = 4 σ 2 SK + 4( M -S ) ζ 2 ∗ S ( M -1) , c 2 = 6 Lσ 2 SK + 6 Lζ 2 ∗ S and 1 d = 1 6 L ( ˜ η = MKη ≤ 1 6 L ), it follows that

where ¯ x ( R ) = 1 W R ∑ R r =0 w r x ( r ) and we use Jensen's inequality ( F is convex) in the first inequality. By tuning the learning rate carefully, we get

where D := ∥ ∥ x (0) -x ∗ ∥ ∥ . Eq. (46) and Eq. (47) are the upper bounds with partial client participation. In particular, when S = M , we can get the claim of the strongly convex case of Theorem 1 and Corollary 1.

D.3 Non-convex case

Lemma 9. Let Assumptions 1, 2, 3b hold. If the learning rate satisfies η ≤ 1 6 LSK , then it holds that

Proof. According to Algorithm 1, the overall model updates of SFL after one complete training round (with S clients selected for training) is

where g ( r ) π m ,k = ∇ f π m ( x ( r ) m,k ; ξ ) is the stochastic gradient of F π m regarding the vector x ( r ) m,k . Thus,

In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

Starting from the smoothness of F (applying Eq. (16), D F ( x , y ) ≤ L 2 ∥ x -y ∥ 2 with x = x +∆ x , y = x ), and substituting the overall updates, we have

For the first term on the right hand side in (49), using the fact that 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 -∥ a -b ∥ 2 with a = ∇ F ( x ) and b = ∇ F π m ( x m,k ) , we have

For the third term on the right hand side in (49), using Jensen's inequality, we have

where we apply Lemma 1 by seeing the data sample ξ m,k , the stochastic gradient g π m ,k , the gradient ∇ F π m ( ξ m,k ) as ξ i , x i , e i respectively in Lemma 1 for the first term and Jensen's inequality for the second term in the preceding inequality.

Substituting (50) and (51) into (49), we have

Since η ≤ 1 6 LSK , the last term on the right hand side in the preceding inequality is negative. Then

The claim follows after recovering the superscripts and taking unconditional expectation.

D.3.1 Bounding the client drift with Assumption 3a

Since Eq. (18), which holds only for convex functions, is used in the proof of Lemma 6 (i.e., Eq. (36)), we cannot use the result of Lemma 6. Next, we use Assumption 3a to bound the client drift (defined in (33)).

Lemma 10. Let Assumptions 1, 2, 3a hold. If the learning rate satisfies η ≤ 1 6 LSK , then the client drift is bounded:

Proof. According to Algorithm 1, the model updates of SFL from x ( r ) to x ( r ) m,k is

with b m,k ( i ) := { K -1 , i ≤ m -1 k -1 , i = m . In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

We use Jensen's inequality to bound the term E [ ∥ x m,k -x ∥ 2 ] = η 2 E [ ∥ ∥ ∥ ∑ m i =1 ∑ b m,k ( i ) j =0 g π i ,j ∥ ∥ ∥ 2 ] :

Applying Lemma 1, Jensen's inequality and Jensen's inequality to the first, third and forth terms on the right hand side in the preceding inequality respectively, we can get

where B m,k := ∑ m i =1 ∑ b m,k ( i ) j =0 1 = ( m -1) K + k . The first term on the right hand side in (53) is bounded by 4 B m,k η 2 σ 2 with Assumption 2. The second term on the right hand side in (53) can be bounded by 4 L 2 η 2 B m,k ∑ m i =1 ∑ b m,k ( i ) j =0 E [ ∥ x i,j -x ∥ 2 ] with Assumption 1. Then, we have

Then, returning to E r := ∑ S m =1 ∑ K -1 k =0 E [ ∥ x m,k -x ∥ 2 ] , we have

Applying Lemma 4 with x π i = ∇ F π i ( x ) and x = ∇ F ( x ) to the third term and ∑ S m =1 ∑ K -1 k =0 B m,k ≤ 1 2 S 2 K 2 and ∑ S m =1 ∑ K -1 k =0 B 2 m,k ≤ 1 3 S 3 K 3 to the other terms on the right hand side in the preceding inequality, we can simplify it:

After rearranging the preceding inequality, we get

Finally, using the condition that η ≤ 1 6 LSK , which implies 1 -2 L 2 S 2 K 2 η 2 ≥ 8 9 , we have

The claim follows after recovering the superscripts and taking unconditional expectations.

D.3.2 Proof of non-convex case of Theorem 1 and Corollary 1

Proof of non-convex case of Theorem 1. Substituting (52) into (48) and using η ≤ 1 6 LSK min { 1 , √ S β } , we can simplify the recursion as follows:

Letting ˜ η := SKη , subtracting F ∗ from both sides and then rearranging the terms, we have

Then applying Lemma 8 with t = r ( T = R ), γ = ˜ η , r t = E [ F ( x ( r ) ) -F ∗ ] , b = 1 3 , s t = E [ ∥ ∥ ∇ F ( x ( r ) ) ∥ ∥ 2 ] , w t = 1 , c 1 = Lσ 2 SK , c 2 = 9 L 2 σ 2 8 SK + 9 L 2 ζ 2 8 S and 1 d = 1 6 L min { 1 , √ S β } ( ˜ η = SKη ≤ 1 6 L min { 1 , √ S β } ), we have

where we use min 0 ≤ r ≤ R E [ ∥ ∥ ∇ F ( x ( r ) ) ∥ ∥ 2 ] ≤ 1 R +1 ∑ R r =0 E [ ∥ ∥ ∇ F ( x ( r ) ) ∥ ∥ 2 ] . Then, using ˜ η ≤ 1 6 L ( β +1) ≤ min { 1 , √ S β } and tuning the learning rate carefully, we get

where A := F ( x 0 ) -F ∗ . Eq. (54) and Eq. (55) are the upper bounds with partial client participation. In particular, when S = M , we get the claim of the non-convex case of Theorem 1 and Corollary 1.

E Proofs of Theorem 2

Here we slightly improve the convergence guarantee for the strongly convex case by combining the works of Karimireddy et al. (2020); Koloskova et al. (2020). Moreover, we reproduce the guarantees for the general convex and non-convex cases based on Karimireddy et al. (2020) for completeness. The results are given in Theorem 2. We provide the proof of Theorem 2 for the strongly convex, general convex and non-convex cases in Sections E.1, E.2 and E.3, respectively.

In the following proof, we consider the partial client participation setting, specifically, selecting partial clients without replacement. So we assume that π = { π 1 , π 2 , . . . , π M } is a permutation of { 1 , 2 , . . . , M } in a certain training round and only the first S selected clients { π 1 , π 2 , . . . , π S } will participate in this round. Without otherwise stated, we use E [ · ] to represent the expectation with respect to both types of randomness (i.e., sampling data samples ξ and sampling clients π ).

Theorem 2. For PFL (Algorithm 2), there exist a constant effective learning rate ˜ η := Kη and weights w r , such that the weighted average of the global parameters ¯ x ( R ) := 1 W R ∑ R r =0 w r x ( r ) ( W R = ∑ R r =0 w r ) satisfies the following upper bounds:

Strongly convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate 1 µR ≤ ˜ η ≤ 1 6 L and weights w r = (1 -µ ˜ η 2 ) -( r +1) , such that it holds that

General convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate ˜ η ≤ 1 6 L and weights w r = 1 , such that it holds that

Non-convex : Under Assumptions 1, 2, 3a, there exist a constant effective learning rate ˜ η ≤ 1 6 L ( β +1) and weights w r = 1 , such that it holds that

where D := ∥ ∥ x (0) -x ∗ ∥ ∥ for the convex cases and A := F ( x (0) ) -F ∗ for the non-convex case. Corollary 2. Applying the results of Theorem 2. By choosing a appropriate learning rate (see the proof of Theorem 2 in Appendix E), we can obtain the convergence rates for PFL as follows:

Strongly convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate 1 µR ≤ ˜ η ≤ 1 6 L and weights w r = (1 -µ ˜ η 2 ) -( r +1) , such that it holds that

General convex : Under Assumptions 1, 2, 3b, there exist a constant effective learning rate ˜ η ≤ 1 6 L and weights w r = 1 , such that it holds that

Non-convex : Under Assumptions 1, 2, 3a, there exist a constant effective learning rate ˜ η ≤ 1 6 L ( β +1) and weights w r = 1 , such that it holds that

where O omits absolute constants, ˜ O omits absolute constants and polylogarithmic factors, D := ∥ ∥ x (0) -x ∗ ∥ ∥ for the convex cases and A := F ( x (0) ) -F ∗ for the non-convex case.

E.1 Strongly convex case

E.1.1 Find the per-round recursion

Lemma 11. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are µ -strongly convex. If the learning rate satisfies η ≤ 1 6 LK , then it holds that

Proof. According to Algorithm 2, the overall model updates of PFL after one complete training round (with S clients selected for training) is

where g ( r ) π m ,k = ∇ f π m ( x ( r ) m,k ; ξ ) is the stochastic gradient of F π m regarding the vector x ( r ) m,k . Thus,

In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

We start by substituting the overall updates:

We can apply Lemma 2 with x = x m,k , y = x ∗ , z = x and h = F π m for the second term on the right hand side in (63):

For the third term on the right hand side in (63), using Jensen's inequality, we have

For the first term on the right hand side in (65), we have

where we use the fact that clients are independent to each other in the first equality and apply Lemma 1 by seeing the data sample ξ m,k , the stochastic gradient g π m ,k , the gradient ∇ F π m ( ξ m,k ) as ξ i , x i , e i in the second equality. For the second term on the right hand side in (65), we have

For the third term on the right hand side in (65), we have

where the last inequality is because E [ D F πm ( x , x ∗ ) ] = D F ( x , x ∗ ) . The forth term on the right hand side in (65) can be bounded by Lemma 3 as follows:

With the preceding four inequalities, we can bound the third term on the right hand side in (63):

Then substituting (64) and (67) into (63), we have

where we use the condition that η ≤ 1 6 LK in the last inequality. The claim of this lemma follows after recovering the superscripts and taking unconditional expectation.

E.1.2 Bounding the client drift with Assumption 3b

The 'client drift' (Karimireddy et al., 2020) in PFL is defined as follows: :

Lemma 12. Let Assumptions 1, 2, 3b hold and assume that all the local objectives are µ -strongly convex. If the learning rate satisfies η ≤ 1 6 LK , then the client drift is bounded:

Proof. According to Algorithm 2, the model updates of PFL from x ( r ) to x ( r ) m,k is

In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

Applying Lemma 1 to the first term and Jensen's inequality to the last three terms on the right hand side in the preceding inequality, respectively, we get

The first term can be bounded by 4 kη 2 σ 2 with Assumption 2. The second term can be bounded by 4 L 2 kη 2 ∑ k -1 j =0 E [ ∥ x m,j -x ∥ 2 ] with Assumption 1. The third term can be bounded by 8 Lk 2 η 2 E [ D F πm ( x , x ∗ ) ] with Eq. (18). Thus, we have

Then returning to E r := 1 S ∑ S m =1 ∑ K -1 k =0 E [ ∥ x m,k -x ∥ 2 ] , we have

Using the facts that ∑ K -1 k =1 k = ( K -1) K 2 ≤ K 2 2 and ∑ K -1 k =1 k 2 = ( K -1) K (2 K -1) 6 ≤ K 3 3 , we can simplify the preceding inequality:

After rearranging the preceding inequality, we get

Finally, using the condition that η ≤ 1 6 LK , which implies 1 -2 L 2 K 2 η 2 ≥ 8 9 , we have

The claim follows after recovering the superscripts and taking unconditional expectations.

E.1.3 Proof of strongly convex case of Theorem 2

Proof of strongly convex case of Theorem 2. Substituting (69) into (62) and using η ≤ 1 6 LK , we can simplify the recursion as,

Let ˜ η = Kη , we have

Applying Lemma 7 with t = r ( T = R ), γ = ˜ η , r t = E [ ∥ ∥ x ( r ) -x ∗ ∥ ∥ 2 ] , a = µ 2 , b = 1 3 , s t = E [ D F ( x ( r ) , x ∗ ) ] , w t = (1 -µ ˜ η 2 ) -( r +1) , c 1 = 4 σ 2 SK + 4( M -S ) ζ 2 ∗ S ( M -1) , c 2 = 6 Lσ 2 K + 4 Lζ 2 ∗ and 1 d = 1 6 L ( ˜ η = Kη ≤ 1 6 L ), it follows that

where ¯ x ( R ) = 1 W R ∑ R r =0 w r x ( r ) and we use Jensen's inequality ( F is convex) in the first inequality. Thus, by tuning the learning rate carefully, we get

where D := ∥ ∥ x (0) -x ∗ ∥ ∥ . Eq. (71) and Eq. (72) are the upper bounds with partial client participation. When M is large enough, we have ( M -S ) S ( M -1) ≈ (1 -S M ) 1 S . This is the constant appearing in Karimireddy et al. (2020); Woodworth et al. (2020b). In particular, when S = M , we can get the claim of the strongly convex case of Theorem 2 and Corollary 2.

E.2 General convex case

E.2.1 Proof of general convex case of Theorem 2 and Corollary 2

Proof of general convex case of Theorem 2. Let µ = 0 in (70), we get the simplified per-round recursion of general convex case,

Applying Lemma 8 with t = r ( T = R ), γ = ˜ η , r t = E [ ∥ ∥ x ( r ) -x ∗ ∥ ∥ 2 ] , b = 1 3 , s t = E [ D F ( x ( r ) , x ∗ ) ] , w t = 1 , c 1 = 4 σ 2 SK + 4( M -S ) ζ 2 ∗ S ( M -1) , c 2 = 6 Lσ 2 K +4 Lζ 2 ∗ and 1 d = 1 6 L ( ˜ η = Kη ≤ 1 6 L ), it follows that

where ¯ x ( R ) = 1 W R ∑ R r =0 w r x ( r ) and we use Jensen's inequality ( F is convex) in the first inequality. By tuning the learning rate carefully, we get

where D := ∥ ∥ x (0) -x ∗ ∥ ∥ . Eq. (73) and Eq. (74) are the upper bounds with partial client participation. In particular, when S = M , we can get the claim of the strongly convex case of Theorem 2 and Corollary 2.

E.3 Non-convex case

Lemma 13. Let Assumptions 1, 2, 3b hold. If the learning rate satisfies η ≤ 1 6 LK , then it holds that

Proof. According to Algorithm 2, the overall model updates of PFL after one complete training round (with S clients selected for training) is

where g ( r ) π m ,k = ∇ f π m ( x ( r ) m,k ; ξ ) is the stochastic gradient of F π m regarding the vector x ( r ) m,k . Thus,

In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

Starting from the smoothness of F (applying Eq. (16), D F ( x , y ) ≤ L 2 ∥ x -y ∥ 2 with x = x +∆ x , y = x ), and substituting the overall updates, we have

For the first term on the right hand side in (76), using the fact that 2 ⟨ a, b ⟩ = ∥ a ∥ 2 + ∥ b ∥ 2 -∥ a -b ∥ 2 with a = ∇ F ( x ) and b = ∇ F π m ( x m,k ) , we have

For the third term on the right hand side in (76), using Jensen's inequality, we have

where we use independence and Lemma 1 for the first term (see Eq. (66)) and Jensen's inequality for the second term in the preceding inequality.

Substituting (77) and (78) into (76), we have

Since η ≤ 1 6 LK , the last term on the right hand side in the preceding inequality is negative. Then

The claim follows after recovering the superscripts and taking unconditional expectation.

E.3.1 Bounding the client drift with Assumption 3a

Lemma 14. Let Assumptions 1, 2, 3b hold. If the learning rate satisfies η ≤ 1 6 LK , then the client drift is bounded:

Proof. According to Algorithm 2, the model updates of PFL from x ( r ) to x ( r ) m,k is

In the following, we focus on a single training round, and hence we drop the superscripts r for a while, e.g., writing x m,k to replace x ( r ) m,k . Specially, we would like to use x to replace x ( r ) 1 , 0 . Without otherwise stated, the expectation is conditioned on x ( r ) .

Applying Lemma 1 to the first term and Jensen's inequality to the last three terms on the right hand side in the preceding inequality, respectively, we get

The first term can be bounded by 4 kη 2 σ 2 with Assumption 2. The second term can be bounded by 4 L 2 kη 2 ∑ k -1 j =0 E [ ∥ x m,j -x ∥ 2 ] with Assumption 1. The third term can be bounded by 4 k 2 η 2 ( β 2 ∥∇ F ( x ) ∥ 2 + ζ 2 ) with Assumption 3a. Thus, we have

Then returning to E r := 1 S ∑ S m =1 ∑ K -1 k =0 E [ ∥ x m,k -x ∥ 2 ] , we have

Using the facts that ∑ K -1 k =1 k = ( K -1) K 2 ≤ K 2 2 and ∑ K -1 k =1 k 2 = ( K -1) K (2 K -1) 6 ≤ K 3 3 , we can simplify the preceding inequality:

After rearranging the preceding inequality, we get

Finally, using the condition that η ≤ 1 6 LK , which implies 1 -2 L 2 K 2 η 2 ≥ 8 9 , we have

The claim follows after recovering the superscripts and taking unconditional expectations.

E.3.2 Proof of non-convex case of Theorem 2

Proof of non-convex case of Theorem 2. Substituting (52) into (48) and using η ≤ 1 6 LK ( β +1) , we can simplify the recursion as follows:

Letting ˜ η := Kη , subtracting F ∗ from both sides and then rearranging the terms, we have

Then applying Lemma 8 with t = r ( T = R ), γ = ˜ η , r t = E [ F ( x ( r ) ) -F ∗ ] , b = 1 3 , s t = E [ ∥ ∥ ∇ F ( x ( r ) ) ∥ ∥ 2 ] , w t = 1 , c 1 = Lσ 2 SK , c 2 = 9 L 2 σ 2 8 K + 3 4 L 2 ζ 2 and 1 d = 1 6 L ( β +1) ( ˜ η = Kη ≤ 1 6 L ( β +1) ), we have

where we use min 0 ≤ r ≤ R E [ ∥ ∥ ∇ F ( x ( r ) ) ∥ ∥ 2 ] ≤ 1 R +1 ∑ R r =0 E [ ∥ ∥ ∇ F ( x ( r ) ) ∥ ∥ 2 ] . Then, tuning the learning rate carefully, we get

where A := F ( x 0 ) -F ∗ . Eq. (80) and Eq. (81) are the upper bounds with partial client participation. In particular, when S = M , we get the claim of the non-convex case of Theorem 2 and Corollary 2.

F Simulations on quadratic functions

Nine groups of simulated experiments with various degrees of heterogeneity are provided in Table 6 as a extension of the experiment in Subsection 4.1. Figure 7 plots the results of PFL and SFL with various combinations of δ and ζ ∗ .

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

(a)

δ

= 0

,

ζ

∗

= 1

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

(b)

δ

= 0

,

ζ

∗

= 10

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

PFL,

K

=2

PFL,

K

=10

SFL,

K

=2

SFL,

K

=10

(c)

δ

= 0

,

ζ

∗

= 100

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

(d)

δ

=

1

3

,

ζ

∗

= 1

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

(e)

δ

=

1

3

,

ζ

∗

= 10

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

PFL,

K

=2

PFL,

K

=10

SFL,

K

=2

SFL,

K

=10

(f)

δ

=

1

3

,

ζ

∗

= 100

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

(g)

δ

= 1

,

ζ

∗

= 1

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

(h)

δ

= 1

,

ζ

∗

= 10

0

100

200

300

400

500

Training rounds

10

4

10

3

10

2

10

1

10 0

10 1

Distance to the optimum

PFL,

K

=2

PFL,

K

=10

SFL,

K

=2

SFL,

K

=10

(i)

δ

= 1

,

ζ

∗

= 100

G More experimental details

This section serves as a supplement and enhancement to Section 4. Our code is partly from Gao et al. (2021); Zeng et al. (2021); Jhunjhunwala et al. (2023) (more references are included in the code), and it is available at ❤/a116/a116♣/a115✿✴✴❣✐/a116❤✉❜✳❝♦♠✴❧✐②✐♣❡♥❣✵✵✴❝♦♥✈❡/a114❣❡♥❝❡ .

G.1 Extended Dirichlet partition

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

0

0

5923

0

0

0

0

0

0

0

0

0

0

0

0

0

6742

0

0

0

0

0

0

0

5958

0

0

0

0

0

0

6131

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

5842

5421

0

0

0

0

0

0

0

0

0

0

0

0

0

0

5918

0

0

0

0

0

0

0

0

0

0

0

6265

0

0

0

0

0

5851

0

0

0

0

0

0

0

0

0

0

0

0

0

0

5949

0

5421

6131

5923

5851

5958

5918

6742

6265

5949

5842

0

1000

2000

3000

4000

5000

6000

(a) 10 clients, ExDir

(1

,

100

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

0

0

0

0

5923

0

0

0

0

0

0

6742

0

0

0

0

0

0

0

0

5958

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

6131

0

0

0

0

0

0

5842

0

0

0

0

0

0

0

0

0

0

0

5421

0

0

0

0

0

0

5918

0

0

0

0

0

0

0

6265

0

0

0

0

0

0

0

0

0

0

0

0

0

5851

0

0

0

0

5949

0

0

0

0

0

0

0

5958

6742

5949

6265

5923

5918

5842

5851

5421

6131

0

1000

2000

3000

4000

5000

6000

(b) 10 clients, ExDir

(1

,

10

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

0

0

5923

0

0

0

0

0

0

0

0

0

0

0

0

0

6742

0

0

0

0

0

0

5958

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

6131

0

0

0

0

5842

0

0

0

0

0

0

0

0

0

0

0

0

5421

0

0

0

5918

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

6265

0

5851

0

0

0

0

0

0

0

0

0

0

0

0

0

0

5949

0

0

0

0

5851

5918

5923

5958

5842

5949

6742

5421

6265

6131

0

1000

2000

3000

4000

5000

6000

(c) 10 clients, ExDir

(1

,

1

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

0

0

0

0

2600

0

0

0

0

3323

0

0

0

0

0

6742

0

0

0

0

0

0

0

5958

0

0

0

0

0

0

0

0

2712

0

0

0

0

0

0

3419

0

0

0

0

0

0

0

0

5842

0

0

0

0

1732

0

0

1886

1803

0

0

0

0

3012

0

0

0

2906

0

0

0

2304

0

0

0

2090

0

0

1871

0

0

2031

1610

0

0

0

0

0

0

2210

0

0

5949

0

0

0

0

0

0

0

0

4335

7559

5724

7690

4690

6742

4792

3674

8052

6742

0

1000

2000

3000

4000

5000

6000

7000

8000

(d) 10 clients, ExDir

(2

,

100

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

0

0

0

0

0

5923

0

0

0

0

0

0

6742

0

0

0

0

0

0

0

0

0

0

0

0

3225

0

0

2733

0

0

0

0

0

3547

0

0

0

0

2584

0

2783

0

0

0

0

0

3059

0

0

0

0

0

2405

3016

0

0

0

0

0

0

0

0

0

0

0

5918

0

0

0

2141

4124

0

0

0

0

0

0

0

0

1318

0

0

0

0

0

0

701

1925

1907

0

0

0

4497

0

0

1452

0

0

0

3459

6907

6742

6902

6563

9148

7370

3760

4658

4491

0

2000

4000

6000

8000

(e) 10 clients, ExDir

(2

,

10

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

1709

0

0

0

0

2258

0

0

1579

377

0

0

0

0

0

726

0

0

6016

0

0

0

4130

0

0

0

1828

0

0

0

0

226

0

131

0

0

0

5774

0

0

5842

0

0

0

0

0

0

0

0

0

0

5421

0

0

0

0

0

0

0

0

0

0

0

0

2478

0

3440

0

0

0

0

0

0

0

0

0

0

0

0

6265

0

0

443

5408

0

0

0

0

0

0

0

0

0

0

3870

0

0

2079

0

0

7551

5647

4573

5539

6348

2984

5268

7853

7595

6642

0

1000

2000

3000

4000

5000

6000

7000

(f) 10 clients, ExDir

(2

,

1

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

0

850

833

0

805

763

0

919

816

937

757

822

0

741

767

708

759

744

803

641

1378

0

0

0

1921

0

0

0

1299

1360

1037

0

899

0

1142

1112

957

984

0

0

0

1764

0

2124

0

0

0

1954

0

0

0

0

1902

0

1576

0

1943

0

0

0

0

1314

0

1025

0

1119

1189

0

0

1271

1493

0

1451

1634

0

0

0

0

1687

0

1558

1280

0

0

0

0

1480

0

0

1533

0

0

1142

1226

0

1127

0

1170

1284

0

6223

6030

6227

6750

6211

4829

6328

5771

5889

5742

0

1000

2000

3000

4000

5000

6000

(g) 10 clients, ExDir

(5

,

100

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

1330

0

1465

0

1585

942

0

0

0

601

3073

0

0

0

0

2051

0

0

0

1618

1345

0

0

1553

0

0

1370

0

1690

0

1471

0

1262

0

0

0

646

1472

1280

0

0

437

478

824

399

612

1035

432

973

652

0

993

0

980

866

0

0

998

753

831

0

941

0

1027

0

1284

0

1927

739

0

0

0

1260

0

1446

1213

1322

0

0

1024

0

1888

0

0

0

0

1935

2028

0

0

0

1383

1812

2025

729

0

0

0

0

0

7219

5642

6277

6409

5025

6102

6308

6857

5435

4726

0

1000

2000

3000

4000

5000

6000

7000

(h) 10 clients, ExDir

(5

,

10

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

1069

1293

0

1421

358

645

1074

0

0

63

720

0

1303

0

0

0

3877

842

0

0

912

572

0

572

675

180

0

196

1575

1276

0

0

0

2113

0

1668

964

0

1196

190

553

786

858

35

0

0

595

914

122

1979

419

2978

0

0

0

1067

0

957

0

0

0

1927

695

1027

2200

0

0

69

0

0

0

0

2615

0

1862

0

0

0

1788

0

0

0

0

0

1185

1910

0

0

1994

762

0

0

5949

0

0

0

0

0

0

0

3673

7556

11420

5168

6280

5470

6510

2978

6675

4270

0

2000

4000

6000

8000

10000

(i) 10 clients, ExDir

(5

,

1

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

558

585

579

624

581

737

535

603

615

506

625

695

602

705

740

629

848

583

599

716

532

663

560

686

561

595

605

636

522

598

647

636

570

862

494

679

633

570

602

438

520

669

498

609

659

615

519

547

590

616

501

485

524

479

484

470

583

648

584

663

511

590

590

554

656

549

478

616

770

604

527

703

649

743

581

620

661

723

536

522

527

597

601

581

546

611

652

599

555

582

690

619

578

605

676

548

542

592

577

522

5638

6242

5751

6448

5978

6053

6056

6117

5950

5767

1000

2000

3000

4000

5000

6000

(j) 10 clients, ExDir

(10

,

100

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

428

678

615

565

494

477

826

512

717

611

844

508

828

550

501

824

704

587

653

743

450

808

686

869

544

354

362

616

609

660

927

472

350

582

573

1069

584

459

548

567

481

433

718

859

541

607

481

673

446

603

764

298

373

756

407

296

923

555

596

453

348

851

584

452

658

708

744

535

544

494

466

1039

848

778

319

607

434

565

681

528

632

826

677

566

599

753

310

449

451

588

491

609

399

756

595

632

747

643

818

259

5831

6522

6078

6733

5231

6327

6115

5594

6063

5506

1000

2000

3000

4000

5000

6000

(k) 10 clients, ExDir

(10

,

10

.

0)

0

1

2

3

4

5

6

7

8

9

Client ID

0

1

2

3

4

5

6

7

8

9

Total

Label

744

208

1354

947

536

241

87

1255

293

258

11

193

377

340

2

1220

449

1159

1323

1668

1012

655

291

672

387

102

529

1025

840

445

633

288

505

681

218

430

1102

274

1775

225

1003

181

519

443

1290

373

995

879

2

157

417

1672

280

17

7

648

999

256

983

142

72

1098

1443

395

212

911

274

50

712

751

863

319

235

541

758

182

895

1472

641

359

797

135

24

1650

2171

149

29

0

0

896

2220

216

518

129

1241

187

585

0

0

853

7772

4965

5546

5815

6822

4443

5944

6370

6569

5754

0

1000

2000

3000

4000

5000

6000

7000

(l) 10 clients, ExDir

(10

,

1

.

0)

Baseline. There are two common partition strategies to simulate the heterogeneous settings in the FL literature. According to Li et al. (2022), they can be summarized as follows:

Quantity-based class imbalance: Under this strategy, each client is allocated data samples from a fixed number of classes. The initial implementation comes from McMahan et al. (2017), and has extended by Li et al. (2022) recently. Specifically, Li et al. first randomly assign C different classes to each client. Then, the samples of each class are divided randomly and equally into the clients which owns the class.

Distribution-based class imbalance: Under this strategy, each client is allocated a proportion of the data samples of each class according to Dirichlet distribution. The initial implementation, to the best of our knowledge, comes from Yurochkin et al. (2019). For each class c , Yurochkin et al. draw p c ∼ Dir ( α q ) and allocate a p c,m proportion of the data samples of class k to client m . Here q is the prior distribution, which is set to 1 .

Extended Dirichlet strategy. This is to generate arbitrarily heterogeneous data across clients by combining the two strategies above. The difference is to add a step of allocating classes (labels) to determine the number of classes per client (denoted by C ) before allocating samples via Dirichlet distribution (with concentrate parameter α ). Thus, the extended strategy can be denoted by ExDir ( C, α ) . The implementation is as follows (one partitioning example is shown in Figure 8):

Allocating classes: We randomly allocate C different classes to each client. After assigning the classes, we can obtain the prior distribution q c for each class c (see Figure 9).

Allocating samples: For each class c , we draw p c ∼ Dir ( α q c ) and then allocate a p c,m proportion of the samples of class c to client m . For example, q c = [1 , 1 , 0 , 0 , . . . , ] means that the samples of class c are only allocated to the first 2 clients.

This strategy have two levels, the first level to allocate classes and the second level to allocate samples. We note that Reddi et al. (2021) use a two-level partition strategy to partition the CIFAR-100 dataset. They draw a multinomial distribution from the Dirichlet prior at the root (Dir ( α ) ) and a multinomial distribution from the Dirichlet prior at each coarse label (Dir ( β ) ).

G.2 Gradient clipping.

Two partitions, the extreme setting C = 1 (i.e., ExDir (1 , 10 . 0) ) and the moderate settings C = 2 (i.e., ExDir (2 , 10 . 0) ) are used in the main body. For both settings, we use the gradient clipping to improve the stability of the algorithms as done in previous works Acar et al. (2021); Jhunjhunwala et al. (2023). Further, we note that the gradient clipping is critical for PFL and SFL to prevent divergence in the learning process on heterogeneous data, especially in the extreme setting. Empirically, we find that the fitting 'max norm' of SFL is larger than PFL. Thus we trained VGG-9 on CIFAR-10 with PFL and SFL for various values of the max norm of gradient clipping to select the fitting value, and had some empirical observations for gradient clipping in FL. The experimental results are given in Table 7, Table 8 and Figure 10. The empirical observations are summarized as follows:

The fitting max norm of SFL is larger than PFL. When the max norm is set to be 20 for both algorithms, gradient clipping helps improve the performance of PFL, while it may even hurt that of SFL (e.g., see the 12-th row in Table 8).

The fitting max norm for the case with high data heterogeneity is larger than that with low data heterogeneity. This is suitable for both algorithms. (e.g., see the 12, 24-th rows in Table 8)

The fitting max norm for the case with more local update steps is larger than that with less local update steps. This is suitable for both algorithms. (e.g., see the 4, 8, 12-th rows in Table 8)

Gradient clipping with smaller values of the max norm exhibits a preference for a larger learning rate. This means that using gradient clipping makes the best learning rate larger. This phenomenon is more obvious when choosing smaller values of the max norm (see Table 7).

The fitting max norm is additionally affected by the model architecture, model size, and so on.

Finally, taking into account the experimental results and convenience, we set the max norm of gradient clipping to 10 for PFL and 50 for SFL for all settings in this paper.

Class ID

Client ID

5

4

3

2

1

1

2

3

4

0

500

1000

1500

2000

2500

3000

3500

4000

10

20

30

40

50

60

70

Test T op1 Accuracy (%)

PFL,

C

=1,

K

=5

PFL, lr=0.03, w/o clip

PFL, lr=0.1, w/ clip=10

PFL, lr=0.3, w/ clip=5

0

500

1000

1500

2000

2500

3000

3500

4000

10

15

20

25

30

35

40

45

Test T op1 Accuracy (%)

PFL,

C

=1,

K

=50

PFL, lr=0.03, w/o clip

PFL, lr=0.03, w/ clip=20

PFL, lr=0.1, w/ clip=10

PFL, lr=0.1, w/ clip=5

0

500

1000

1500

2000

2500

3000

3500

4000

10

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

SFL,

C

=1,

K

=5

SFL, lr=0.03, w/o clip

SFL, lr=0.03, w/ clip=100

SFL, lr=0.03, w/ clip=50

SFL, lr=0.03, w/ clip=20

0

500

1000

1500

2000

2500

3000

3500

4000

10

20

30

40

50

60

70

Test T op1 Accuracy (%)

SFL,

C

=1,

K

=50

SFL, lr=0.003, w/o clip

SFL, lr=0.003, w/ clip=100

SFL, lr=0.003, w/ clip=50

SFL, lr=0.003, w/ clip=20

0

500

1000

1500

2000

2500

3000

3500

4000

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

PFL,

C

=2,

K

=5

PFL, lr=0.1, w/o clip

PFL, lr=0.1, w/ clip=10

PFL, lr=0.1, w/ clip=5

0

500

1000

1500

2000

2500

3000

3500

4000

10

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

PFL,

C

=2,

K

=50

PFL, lr=0.03, w/o clip

PFL, lr=0.03, w/ clip=20

PFL, lr=0.1, w/ clip=10

PFL, lr=0.1, w/ clip=5

0

500

1000

1500

2000

2500

3000

3500

4000

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

SFL,

C

=2,

K

=5

SFL, lr=0.03, w/o clip

SFL, lr=0.03, w/ clip=100

SFL, lr=0.03, w/ clip=50

SFL, lr=0.03, w/ clip=20

0

500

1000

1500

2000

2500

3000

3500

4000

20

30

40

50

60

70

80

Test T op1 Accuracy (%)

SFL,

C

=2,

K

=50

SFL, lr=0.01, w/o clip

SFL, lr=0.01, w/ clip=100

SFL, lr=0.01, w/ clip=50

SFL, lr=0.01, w/ clip=20

G.3 Grid search

We use the grid search to find the best learning rate on one random seed '1234'. Since we have observed that the best learning rate of PFL is smaller than SFL empirically, the grid for PFL is { 10 -2 . 0 , 10 -1 . 5 , 10 -1 . 0 , 10 -0 . 5 } ({0.01, 0.03, 0.1, 0.3} in fact) and the grid for SFL is { 10 -2 . 5 , 10 -2 . 0 , 10 -1 . 5 , 10 -1 . 0 } ({0.003, 0.01, 0.03, 0.1} in fact). We use these grids for all tasks in this paper, including MNIST and FMNIST in the next subsection.

One practical method used in Jhunjhunwala et al. (2023) to find the best learning rate is running the algorithms for a fewer training rounds and then comparing the short-run results by some metrics (e.g., training accuracy) when the computation resources are restrictive and the task is complex (e.g., CIFAR-10). However, we should pay attention to whether the chosen learning rates are appropriate in the specific scenarios, as the best learning rate in the short run may not be the best in the long run. One alternative method is using the short-run results to find some alternatives (coarse-grained search) and then using the long-run results to find the best one (fine-grained search).

In this paper, for CIFAR-10 and CINIC-10, we run the algorithms for 1000 training rounds to find the candidate learning rates (with a less then 3% difference to the best result in test accuracy), called as coarse-grained search; and then run the algorithms for 4000 training rounds with the candidate learning rates to find the best learning rate (with the highest test accuracy), called as fine-grained search. The max norm of gradient clipping is set to 10 for PFL and 50 for SFL for all settings (see subsection G.2). Other hyperparameters are identical to that in the main body.

The results of the coarse-grained search are collected in Figure 12, Table 10. Taking the setting training VGG-9 on CIFAR-10 as an example. We first find the candidate learning rates, whose short-run test accuracies are only 3% or less below the best accuracy. The candidate learning rates are summarized in Table 9. The training curves are in Figure 11. We then find the best learning rate, whose long-run test accuracy is the highest among the candidate learning rates. The final best learning rates are in Table 9. For fine-grained search of other settings, please refer to the code.

0

200

400

600

800

1000

Training rounds

10

15

20

25

30

35

Test T op1 Accuracy (%)

PFL

lr=0.01

lr=0.03

lr=0.3

0

200

400

600

800

1000

Training rounds

10

20

30

40

50

60

SFL

lr=0.003

lr=0.01

lr=0.03*

0

200

400

600

800

1000

Training rounds

10

15

20

25

30

Test T op1 Accuracy (%)

PFL

lr=0.01

lr=0.03

lr=0.3

0

200

400

600

800

1000

Training rounds

10

15

20

25

30

35

SFL

lr=0.003*

lr=0.01

lr=0.03

0

200

400

600

800

1000

Training rounds

8

10

12

14

16

18

20

22

Test T op1 Accuracy (%)

PFL

lr=0.01

lr=0.03"

lr=0.3

0

200

400

600

800

1000

Training rounds

8

10

12

14

16

18

20

22

24

SFL

lr=0.003*

lr=0.01

lr=0.03

0

200

400

600

800

1000

Training rounds

10

20

30

40

50

Test T op1 Accuracy (%)

PFL

lr=0.01

lr=0.03

lr=0.3

0

200

400

600

800

1000

Training rounds

20

30

40

50

60

70

SFL

lr=0.003

lr=0.01"

lr=0.03*

0

200

400

600

800

1000

Training rounds

20

30

40

50

60

70

Test T op1 Accuracy (%)

PFL

lr=0.01

lr=0.03

lr=0.3

0

200

400

600

800

1000

Training rounds

20

30

40

50

60

SFL

lr=0.003

lr=0.01*

lr=0.03"

0

200

400

600

800

1000

Training rounds

20

30

40

50

60

70

Test T op1 Accuracy (%)

PFL

lr=0.01

lr=0.03"

lr=0.3

0

200

400

600

800

1000

Training rounds

20

30

40

50

60

SFL

lr=0.003

lr=0.01*

lr=0.03

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

C

=2,

K

=50

-2.5-2.0-1.5-1.0-0.5

0

50

100

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

50

100

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

50

100

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

50

100

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

50

100

C

=2,

K

=50

-2.5-2.0-1.5-1.0-0.5

0

50

100

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

50

100

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

50

100

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

50

100

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

50

100

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

50

100

C

=2,

K

=50

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

C

=2,

K

=50

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

25

50

75

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

25

50

75

C

=2,

K

=50

(b) Left: LeNet-5/FMNIST. Right: CNN/FMNIST

-2.5-2.0-1.5-1.0-0.5

0

20

40

60

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

10

20

30

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

10

20

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

C

=2,

K

=50

-2.5-2.0-1.5-1.0-0.5

0

20

40

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

10

20

30

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

10

20

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

60

C

=2,

K

=50

(c) Left: VGG-9/CIFAR-10. Right: ResNet-18/CIFAR-10

-2.5-2.0-1.5-1.0-0.5

0

20

40

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

10

20

30

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

10

20

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

C

=2,

K

=50

-2.5-2.0-1.5-1.0-0.5

0

10

20

30

Test T op1 Accuracy (%)

C

=1,

K

=5

-2.5-2.0-1.5-1.0-0.5

0

10

20

30

C

=1,

K

=20

-2.5-2.0-1.5-1.0-0.5

0

10

20

C

=1,

K

=50

PFL

SFL

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

Test T op1 Accuracy (%)

C

=2,

K

=5

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

C

=2,

K

=20

-2.5-2.0-1.5-1.0-0.5

Learning rate (log10 )

0

20

40

C

=2,

K

=50

(d) Left: VGG-9/CINIC-10. Right: ResNet-18/CINIC-10

G.4 More experimental results

More experimental results are provided in this subsection. These include results on MNIST (LeCun et al., 1998), FMNIST (Xiao et al., 2017) and additional results on CIFAR-10 and CINIC-10.

Setup on MNIST and FMNIST. We consider five additional tasks: 1) training Logistic Regression on MNIST , 2) training Multi-Layer Perceptron (MLP) on MNIST, 3) training LeNet-5 (LeCun et al., 1998) on MNIST, 4) training LeNet-5 on FMNIST, 5) training CNN on FMNIST. We partition the training sets of both MNIST and FMNIST into 500 clients by extended Dirichlet strategy C = 1 and C = 2 (with α = 10 . 0 ) and spare the test sets for computing test accuracy. We apply gradient clipping with the max norm of 10 for PFL and 50 for SFL. We find the best learning rate by grid search. This is done by running algorithms for 1000 training rounds and choosing the learning rate that achieves the highest test accuracy averaged over the last 40 training rounds. Note that since tasks on MNIST/FMNIST are quite simpler than that on CIFAR-10/CINIC-10, we do not use the coarse, fine-grained search. The results of grid search are given in Table 10. Other hyperparameters without being stated explicitly are identical to that of CIFAR-10/CINIC-10 in the main body.

Results of MNIST and FMNIST. The results of these five tasks are in Figures 13, 14 and Table 11. In the tasks MNIST/MLP, MNIST/LeNet-5, FMNIST/CNN, the performance of SFL is better when C = 1 , which is consistent with the analysis in Subsection 4.2. However, we note that SFL shows worse even when C = 1 in simpler tasks MNIST/Logistic and FMNIST/LeNet-5, especially when K is large. This may be because the (objective function) heterogeneity on simple models and datasets is limited even with extreme data distributions (i.e., C = 1 ). Thus, more extensive experiments are still required before drawing conclusions, which is beyond the scope of this paper.

0

200

400

600

800

1000

Training rounds

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Training Loss

MNIST, Logisitc,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Training-R Loss

MNIST, Logisitc,

C

=1

0

200

400

600

800

1000

Training rounds

75

78

80

82

85

88

90

92

95

Training T op1 Accuracy (%)

MNIST, Logisitc,

C

=1

0

200

400

600

800

1000

Training rounds

75

78

80

82

85

88

90

92

95

Training-R T op1 Accuracy (%)

MNIST, Logisitc,

C

=1

0

200

400

600

800

1000

Training rounds

75

78

80

82

85

88

90

92

95

Test T op1 Accuracy (%)

MNIST, Logisitc,

C

=1

0

200

400

600

800

1000

Training rounds

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Training Loss

MNIST, Logisitc,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Training-R Loss

MNIST, Logisitc,

C

=2

0

200

400

600

800

1000

Training rounds

82

84

86

88

90

92

94

Training T op1 Accuracy (%)

MNIST, Logisitc,

C

=2

0

200

400

600

800

1000

Training rounds

82

84

86

88

90

92

94

Training-R T op1 Accuracy (%)

MNIST, Logisitc,

C

=2

0

200

400

600

800

1000

Training rounds

82

84

86

88

90

92

94

Test T op1 Accuracy (%)

MNIST, Logisitc,

C

=2

0

200

400

600

800

1000

Training rounds

0.10

0.20

0.30

0.40

0.50

Training Loss

MNIST, MLP,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.10

0.20

0.30

0.40

0.50

Training-R Loss

MNIST, MLP,

C

=1

0

200

400

600

800

1000

Training rounds

82

84

86

88

90

92

94

96

98

100

Training T op1 Accuracy (%)

MNIST, MLP,

C

=1

0

200

400

600

800

1000

Training rounds

82

84

86

88

90

92

94

96

98

100

Training-R T op1 Accuracy (%)

MNIST, MLP,

C

=1

0

200

400

600

800

1000

Training rounds

82

84

86

88

90

92

94

96

98

100

Test T op1 Accuracy (%)

MNIST, MLP,

C

=1

0

200

400

600

800

1000

Training rounds

0.05

0.10

0.15

0.20

0.25

0.30

Training Loss

MNIST, MLP,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.05

0.10

0.15

0.20

0.25

0.30

Training-R Loss

MNIST, MLP,

C

=2

0

200

400

600

800

1000

Training rounds

92

93

94

95

96

97

98

99

Training T op1 Accuracy (%)

MNIST, MLP,

C

=2

0

200

400

600

800

1000

Training rounds

92

93

94

95

96

97

98

99

Training-R T op1 Accuracy (%)

MNIST, MLP,

C

=2

0

200

400

600

800

1000

Training rounds

92

93

94

95

96

97

98

99

Test T op1 Accuracy (%)

MNIST, MLP,

C

=2

0

200

400

600

800

1000

Training rounds

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Training Loss

MNIST, LeNet-5,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Training-R Loss

MNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

90

92

94

96

98

100

Training T op1 Accuracy (%)

MNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

90

92

94

96

98

100

Training-R T op1 Accuracy (%)

MNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

90

92

94

96

98

100

Test T op1 Accuracy (%)

MNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

0.00

0.02

0.04

0.06

0.08

0.10

Training Loss

MNIST, LeNet-5,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.00

0.02

0.04

0.06

0.08

0.10

Training-R Loss

MNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

95

96

97

98

99

100

Training T op1 Accuracy (%)

MNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

95

96

97

98

99

100

Training-R T op1 Accuracy (%)

MNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

95

96

97

98

99

100

Test T op1 Accuracy (%)

MNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Training Loss

FMNIST, LeNet-5,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Training-R Loss

FMNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

60

65

70

75

80

85

90

Training T op1 Accuracy (%)

FMNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

60

65

70

75

80

85

90

Training-R T op1 Accuracy (%)

FMNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

60

65

70

75

80

85

90

Test T op1 Accuracy (%)

FMNIST, LeNet-5,

C

=1

0

200

400

600

800

1000

Training rounds

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Training Loss

FMNIST, LeNet-5,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Training-R Loss

FMNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

70

75

80

85

90

Training T op1 Accuracy (%)

FMNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

70

75

80

85

90

Training-R T op1 Accuracy (%)

FMNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

70

75

80

85

90

Test T op1 Accuracy (%)

FMNIST, LeNet-5,

C

=2

0

200

400

600

800

1000

Training rounds

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Training Loss

FMNIST, CNN,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Training-R Loss

FMNIST, CNN,

C

=1

0

200

400

600

800

1000

Training rounds

70

75

80

85

90

95

Training T op1 Accuracy (%)

FMNIST, CNN,

C

=1

0

200

400

600

800

1000

Training rounds

70

75

80

85

90

95

Training-R T op1 Accuracy (%)

FMNIST, CNN,

C

=1

0

200

400

600

800

1000

Training rounds

70

75

80

85

90

95

Test T op1 Accuracy (%)

FMNIST, CNN,

C

=1

0

200

400

600

800

1000

Training rounds

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Training Loss

FMNIST, CNN,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

200

400

600

800

1000

Training rounds

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Training-R Loss

FMNIST, CNN,

C

=2

0

200

400

600

800

1000

Training rounds

75

80

85

90

95

Training T op1 Accuracy (%)

FMNIST, CNN,

C

=2

0

200

400

600

800

1000

Training rounds

75

80

85

90

95

Training-R T op1 Accuracy (%)

FMNIST, CNN,

C

=2

0

200

400

600

800

1000

Training rounds

75

80

85

90

95

Test T op1 Accuracy (%)

FMNIST, CNN,

C

=2

0

1000

2000

3000

4000

Training rounds

0.50

1.00

1.50

2.00

2.50

3.00

Training Loss

CIFAR-10, VGG-9,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.50

1.00

1.50

2.00

2.50

3.00

Training-R Loss

CIFAR-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

Training T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

Training-R T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

Test T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Training Loss

CIFAR-10, VGG-9,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Training-R Loss

CIFAR-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

50

55

60

65

70

75

80

85

90

95

Training T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

50

55

60

65

70

75

80

85

90

95

Training-R T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

50

55

60

65

70

75

80

85

90

95

Test T op1 Accuracy (%)

CIFAR-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

0.00

1.00

2.00

3.00

4.00

5.00

Training Loss

CIFAR-10, ResNet-18,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.00

1.00

2.00

3.00

4.00

5.00

Training-R Loss

CIFAR-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

Training T op1 Accuracy (%)

CIFAR-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

Training-R T op1 Accuracy (%)

CIFAR-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

Test T op1 Accuracy (%)

CIFAR-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Training Loss

CIFAR-10, ResNet-18,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Training-R Loss

CIFAR-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

30

40

50

60

70

80

90

Training T op1 Accuracy (%)

CIFAR-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

30

40

50

60

70

80

90

Training-R T op1 Accuracy (%)

CIFAR-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

30

40

50

60

70

80

90

Test T op1 Accuracy (%)

CIFAR-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

0.00

1.00

2.00

3.00

4.00

5.00

Training Loss

CINIC-10, VGG-9,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.00

1.00

2.00

3.00

4.00

5.00

Training-R Loss

CINIC-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

10

20

30

40

50

60

70

80

90

100

Training T op1 Accuracy (%)

CINIC-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

10

20

30

40

50

60

70

80

90

100

Training-R T op1 Accuracy (%)

CINIC-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

10

20

30

40

50

60

70

80

90

100

Test T op1 Accuracy (%)

CINIC-10, VGG-9,

C

=1

0

1000

2000

3000

4000

Training rounds

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Training Loss

CINIC-10, VGG-9,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Training-R Loss

CINIC-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

30

40

50

60

70

80

90

100

Training T op1 Accuracy (%)

CINIC-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

30

40

50

60

70

80

90

100

Training-R T op1 Accuracy (%)

CINIC-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

30

40

50

60

70

80

90

100

Test T op1 Accuracy (%)

CINIC-10, VGG-9,

C

=2

0

1000

2000

3000

4000

Training rounds

0.00

1.00

2.00

3.00

4.00

5.00

Training Loss

CINIC-10, ResNet-18,

C

=1

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.00

1.00

2.00

3.00

4.00

5.00

Training-R Loss

CINIC-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

10

20

30

40

50

60

70

80

90

100

Training T op1 Accuracy (%)

CINIC-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

10

20

30

40

50

60

70

80

90

100

Training-R T op1 Accuracy (%)

CINIC-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

10

20

30

40

50

60

70

80

90

100

Test T op1 Accuracy (%)

CINIC-10, ResNet-18,

C

=1

0

1000

2000

3000

4000

Training rounds

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Training Loss

CINIC-10, ResNet-18,

C

=2

PFL,

K

=5

PFL,

K

=20

PFL,

K

=50

SFL,

K

=5

SFL,

K

=20

SFL,

K

=50

0

1000

2000

3000

4000

Training rounds

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Training-R Loss

CINIC-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

100

Training T op1 Accuracy (%)

CINIC-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

100

Training-R T op1 Accuracy (%)

CINIC-10, ResNet-18,

C

=2

0

1000

2000

3000

4000

Training rounds

20

30

40

50

60

70

80

90

100

Test T op1 Accuracy (%)

CINIC-10, ResNet-18,

C

=2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an intriguing exploration of the counterintuitive comparison results between Sequential SL (SSL) and Federated Averaging (FedAvg) in the context of heterogeneous data. The authors attribute the surprising performance of SSL over FedAvg in worst-case scenarios to the assumption of data heterogeneity (Assumption 3), which is derived from the convergence theory of SGD. The paper provides a compelling argument and empirical evidence for the counterintuitive performance of SSL over FedAvg in certain scenarios. It offers valuable insights into the role of data heterogeneity in the performance of these algorithms and highlights the need for careful consideration of data heterogeneity in the design and analysis of federated learning algorithms. However, the paper could benefit from a more detailed discussion of the implications of these findings for real-world applications. Further exploration of the conditions under which SSL outperforms FedAvg, as well as potential strategies for mitigating the limitations of FedAvg in extremely heterogeneous settings, would be valuable.\n\nSTRENGTHS:\n1. The paper provides a comprehensive theoretical analysis of the counterintuitive performance of Sequantial SL (SSL) over Federated Averaging (FedAvg) in certain scenarios. The author's exploration of the role of data heterogeneity and its impact on these algorithms is particularly insightful.\n\n2. The authors validate their theoretical findings with empirical experiments, which strengthens the credibility of their claims. The use of one-dimensional quadratic functions for simulation and the consideration of practical scenarios in resource-constrained edge devices add depth to their research.\n\nWEAKNESSES:\n1. The paper primarily focuses on the performance of Sequential SL(SSL) and Federated Averaging (FedAvg) in the context of data heterogeneity. While this focus is important, it might limit the generalizability of the findings. The performance of these algorithms could be influenced by other factors that are not considered in this paper.\n\n2. While the paper does consider practical scenarios in resource-constrained edge devices, it could benefit from a more detailed discussion of the implications of its findings for real-world applications. Understanding how these findings could be applied in practice would add significant value to the paper.\n\n3. The paper establishes that SSL outperforms FedAvg in extremely heterogeneous settings, but it doesn't dive deeply into why this is the case or how it could be mitigated. Further exploration of these areas would enhance the paper's contribution to the field.\n\nQUESTIONS:\nWhile the paper provides a detailed convergence analysis for the newly proposed SSL algorithm, my primary concern lies in the experimental evaluation. The authors have chosen to use relatively simple and dated datasets and models, such as MNIST and CIFAR10 with LeNet and VGG-11. These models, being almost a decade old, may not accurately reflect the complexities and challenges of current practical workloads.\n\nTo strengthen the paper, I recommend that the authors extend their experimental evaluation to include larger datasets and models. This would not only provide a more rigorous test of the SSL algorithm but also enhance the relevance and applicability of the research. If such larger experimental results can be provided, I would be inclined to increase my score by one point, as it would significantly improve the paper's contribution to the field.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Sequential SL algorithm in which the at each round, sequentially, each machine performs some local SGD iterations, and then the final iterate is passed on to the next machine.  Such an algorithm may be useful in cross-device, resource constrained settings, where local iterations are expensive.The papers compares the performance of this algorithm to the performance of Federated Averaging, where the local SGD iterations are performed in parallel rather than sequentially. They study a heterogenous data setting, where the distribution of data is different at each machine, under strongly convex, convex, and non-convex settings. While the SL algorithm has larger parallel depth than federated averaging, the authors show that the SSL algorithm outperforms FedAvg in the strongly convex setting. They also show some minor improvements in the convergence of Federated Averaging by reanalyzing this popular algorithm. They provide experiments in a heterogenous environment to show that for high levels of hetergeneity, SSL outperforms FedAvg without contradicting prior experimental evidence.\n\nSTRENGTHS:\n- The authors provide a clean analysis of SSL under standard assumptions in strongly convex, convex, and non-convex settings. They also show how to slightly improve the analysis of FedAvg in a heterogeneous setting, which cleans up some small gaps between bounds in the literature. \n- The SSL algorithm they analyze is simple and natural, yet is not immediate to understand with existing analyses.\n\nWEAKNESSES:\n*Clarity*:\n- The two last paragraphs on page 1 are somewhat confusing. The 2nd to last P seems to span about FedAvg. But then the last paragraph seems to speak about but SSL and about FedAvg (which is called SFLV1, which is confusing). Could the authors streamline these paragraphs to make it clear the two algorithms considered, and also clarify if SFLV1 = FedAvg or how they are different.\n- What does it mean that the model is trained partially at the server? I don't see this in Algorithm 1?\n- I find the emphasis on \"counterintuitive\" to be off-putting, because it seems natural that SSL, which is a sequentially deeper algorithm will perform better. \n- Could the authors make Assumption 3 clearer by breaking it into assumption 3a and 3b or break its into two assumptions environments. \n\n*Comparison to other theoretical results is confusing or sometimes misleading*\n- The discussion in the paragraph at line 154 is confusing. Perhaps a table would be helpful. Further, it is misleading that the authors say that their bounds are tight, because there seams to be a difference in depenence on M (which is quite significant), and anyways, the tightness would only be for the K = 1 case.\n- In the comparison to FedAvg (line 189 paragraph), the authors only discuss the strongly convex setting. Could they please explain the other cases too. The following paragraph (line 193) is quite confusing - I have no idea what the authors are saying. \n- In the discussion of heterogeneity and in the empirical experiments, the authors claim that SSL is better relative to FedAvg when there is more heterogeneity. But this does not completely make sense, because in the homogenous setting, where there is no heterogenity, SSL would be strictly better than FedAvg. Could the authors please discuss and clarify this.\n\nQUESTIONS:\nSee weakness above.\nMinor comment: Line 134 vanishes --> vanish\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provides the convergence analysis of a FL-like approach where clients updates the model sequentially based on local datasets. Results are provided for the strongly convex, convex, and non-convex case.  Interestingly, the analysis shows that in some setting the sequential approach can outperform the usual FL approach where multiple clients update the local model in parallel and a server aggregates the updates.\nI do not understand why split learning is mentioned in the title and in the rest of the paper, as there is no model split across clients or across the clients and server as far as I can say.\n\nSTRENGTHS:\n* The convergence analysis is interesting; it shows the authors master a number of existing techniques used to prove convergence results of distributed training procedures\n* the result about sequential training  outperforming FedAvg is very interesting and indeed counterintuitive as observed by the authors\n* This was the best paper in my batch (6 papers)\n\nWEAKNESSES:\n* The reference to split computing, while this paper does not deal with split computing.\n* The result about SSL outperforming FedAvg is interesting, but the paper does not provide any intuitive explanation about why this may be the case. Also the bounds suggest that it should always be the case in the strongly convex setting for any value of $\\zeta^*$, not only in the highly heterogeneous setting (large $\\zeta^*$).\n* I do not find the comparison with FedAvg under partial clients' participation fair. The authors should consider that at each round the clients available will sequentially perform  SSL. Instead, they assume that all clients participate, but the reduce the total number of rounds. \n* there are some papers about the convergence of sequential optimization which should be cited. I think most of them consider a single gradient update at each node and focus on random walks. Examples:\n\t* Ram, S., Nedić, A., and Veeravalli, V. (2009). Incre- mental stochastic subgradient algorithms for con- vex optimization. SIAM Journal on Optimization, 20(2):691–717.\n\t* Johansson, B., Rabi, M., and Johansson, M. (2009). A randomized incremental subgradient method for dis- tributed optimization in networked systems. SIAM Journal on Optimization, 20(3):1157–1170\n\t* Mao, X., Yuan, K., Hu, Y., Gu, Y., Sayed, A. H., and Yin, W. (2020). Walkman: A Communication- Efficient Random-Walk Algorithm for Decentralized Optimization. IEEE Transactions on Signal Process- ing, 68:2513–2528.\n\t* Ayache, G. and Rouayheb, S. E. (2020). Private Weighted Random Walk Stochastic Gradient De- scent. Technical report, arXiv:2009.01790.\n\t* Cyffers, E., Bellet, A., Privacy amplification by decentralization, AISTATS 2022\n* There are some sentence that are poorly worded, some examples\n\t* line 72: \"it cannot follow the theory of SGD\", rather  \"SGD theory does not apply\"\n\t* line 120: \"Or further, it only holds at the optimum.\", it seems the authors mean. \"In alternative, the condition is required only at a global minimizer\"\n\t* line 126 and elsewhere: \"making it hold that\", simply \"and it holds\"?\n\t* line 188, what do the authors mean by \"our comparison is persuasive\"?\n\t* line 210, the section is not really about \"feasability\", it is just providing some numerical \"evidence\" tha the counterintuitive result can appear even for simple quadratic functions. As no intuitive explanation is provided, I think this section should be merged with the experiments one.\n\t* line 218, \"which is tight to capture the gradient diversities\", what the authors mean is that the bound may be pessimistic for fedavg where averaging can limitate the effect of statistical heterogeneity, but less for SSL because of its sequential nature.\n* Others:\n\t* line 67, (Wang et al, 2020) is indicated as an example dealing with unbalanced data, but I would say it rather deals with unbalanced computation.\n\t* line 67 trival -> trivial\n\t* line 114 and elsewhere: using assumption (4) to indicate the equation (4) in assumption 3 may be confusing\n\t* There are some papers which are cited as preprint, but have been published. For example Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participa- tion in non-iid federated learning appeared at ICLR 2021\n\nQUESTIONS:\n* why do the authors refer to split computing? which model is split? Algorithm 1 shows well that all clients train the same model architecture.\n* what is denoted by SFLv1? if it denotes simply the algorithm proposed in (Thapa et al., 2020) just say it. Also in the rest of the paper, including in the experiment section, SFL is never evaluated, but he proposed solution is compared with FedAvg. \n* the paper suggests in different places that increasing the number of local iterations may have a beneficial effect at first and then be harmful, but in reality the bounds in the paper suggest that it is always beneficial to increase K but with a diminishing return, right? If so, some statements while commenting figure 2 should be modified.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides the first theoretical convergence rates for sequential split learning (SSL) in the presence of heterogeneous data. In each round of SSL, a random permutation of the machines is computed. According to this permutation, each machine performs $K$ local gradient steps on the model sequentially. When the data on the clients is non-iid, due to the sequential nature of the steps, the updates are biased gradients of the true objective, which is the average of losses on all machines. This is the main challenge in analyzing SSL.\n\nThe authors extend the proof of FedAvg to provide tight convergence rates for SSL on non-convex, convex and strongly-convex loss functions under standard assumptions of smoothness, bounded gradient noise and heterogeneity between machines.\n\n\nThey show that SSL has strictly better convergence rates than FedAvg. In particular, the heterogeneity term in the rates decreases by a\nfactor of $\\frac{1}{M}$, where $M$ is the number of machines.\n\n\nThey demonstrate the improvement of SSL over FedAvg empirically for quadratics under high heterogeneity. Additional experiments on CIFAR10 under high heterogeneity support this observation.\n\nSTRENGTHS:\n- **Convergence rates** : Clean convergence results are provided under the same assumptions as FedAvg. For local steps, $K=1$, SSL is equivalent to Random Reshuffling, and the obtained convergence rates match its lower bound.\n- **Thorough comparison to FedAvg** : The authors provide thorough comparisons to the best possible rates for FedAvg under full participation and partial participation and demonstrate the improvement due to SSL.\n\n- **Experiments validate theory**: Synthetic experiments show a clear improvement of SSL over FedAvg in high heterogeneity conditions.\n- **Detailed Appendix** : There are several additional experiments in the appendix on more quadratic problems, MNIST and FMNIST, under varying hyperparameters. It might be beneficial to move one of these experiments to the main paper.\n- **Marginally better rates for FedAvg on strongly convex functions**: The authors obtain marginally better rates for FedAvg in terms of constants by combining two existing proofs.\n\nWEAKNESSES:\n- **Analysis of Split learning without any explicit splits** :  In SSL, or any form of split learning (Gupta and Raskar, 2018), a crucial component of the algorithm is the model being split into two parts, one for the server and one for the client. The analysis considers sequential local updates only without any form of splits in the model. This could be clarified in the introduction.\n- **Minor improvement in CIFAR10 figures**: This is a minor weakness, but Fig 2 is difficult to read. Using different color schemes for FedAvg and SSL or showing only 2 values of local steps might improve clarity.\n- **SSL as a case of FedAvg with partial participation** : Note that if we perform FedAvg with $S=1$ worker per round sampled without replacement for $R$ rounds, then this algorithm is same as running SSL for $R/M$ rounds. Could the analysis of FedAvg with partial participation then recover the same rates as this paper?\n\n**References**\n- (Gupta and Raskar, 2018) : Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications.\n\nQUESTIONS:\n- A recent paper (Yun et al, 2022) considers Minibatch and Local SGD with Random Reshuffling. It would be interesting to compare its theoretical rates with SSL, which has close connections to Random Reshuffling.\n- Does the improvement in the rates for SSL over FedAvg come from a better bound on the client drift?\n\n**References**\n- (Yun et al, 2022) : Minibatch vs Local SGD with Shuffling : Tight Convergence Bounds and Beyond. ICLR 2022.\n\nLIMITATIONS:\nDescribed in Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}