Title: Convergence Analysis of Split Federated Learning on Heterogeneous Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: split federated learning

distributed learning

convergence analysis

machine learning
Abstract: Split federated learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL), due to the potential dual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data. The convergence rates are $O(1/T)$ and $O(1/\sqrt[3]{T})$, respectively, where $T$ denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and where some clients may be unavailable during training. Numerical experiments validate our theoretical results and show that SFL outperforms FL and split learning (SL) when data is highly heterogeneous across a large number of clients.

Convergence Analysis of Split Federated Learning on Heterogeneous Data

Abstract

Split federated learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL), due to the potential dual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data. The convergence rates are O (1 /T ) and O (1 / 3 √ T ) , respectively, where T denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and the scenario where some clients may be unavailable during training. Experimental experiments validate our theoretical results and show that SFL outperforms FL and split learning (SL) when data is highly heterogeneous across a large number of clients.

1 Introduction

1.1 Motivation

Federated learning (FL) [18, 9] allows distributed clients to train a global machine learning model collaboratively without sharing raw data. FL leverages the parallel computing capabilities of clients to enhance model training efficiency. However, FL is usually computationally intensive. Clients need to train the entire global model multiple times, which can be infeasible for resource-constrained edge devices. This challenge is further exacerbated as the trend towards increasingly larger model architectures demands more substantial resources [1]. Moreover, FL suffers from the client drift

...

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

Fed Server

Main Server

Clients

Cut Layer

x

c

,1

x

s

,1

x

c

,2

x

c

,

N

x

c

...

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

. . .

Fed Server

Main Server

Clients

Cut Layer

x

c

,1

x

s

x

c

,2

x

c

,

N

x

c

. . .

. . .

. . .

. . .

x

s

,2

. . .

. . .

. . .

. . .

x

s

,

N

SFL-V1

SFL-V2

problem when clients' data distributions are heterogeneous, aka non-identically and independently distributed (non-IID). A large number of studies have proposed algorithms to address the client drift issue, e.g., [15, 10, 14, 25].

Split learning (SL) [28] is another distributed approach. By splitting the model across clients and a main server, SL can substantially reduce the computational workload on edge devices. Moreover, recent studies in [34, 17] show that SL can outperform FL when data is highly heterogeneous. However, SL's sequential training among clients can lead to high latency in each training round and potential performance loss (e.g., caused by catastrophic forgetting), which impedes its practical applicability in real-world distributed systems.

In light of above challenges, Thapa et. al in [27] proposed split federated learning (SFL) as a hybrid approach that synergizes the strengths of both FL and SL. SFL combines parallel training of FL with partial model training of SL. They proposed two major SFL algorithms: SFL-V1 and SFL-V2. An illustration of these SFL algorithms are shown in Fig. 1. Specifically, the global model (to be trained) is first split at a cut layer into two parts: a client-side model and a server-side model. Then, the clients are responsible for training only the client-side model under the coordination of a fed server (similar to FL). Another server, known as the main server , is tasked with training the server-side model by collaborating with the clients (similar to SL). SFL aims to leverage parallel processing to reduce latency, while benefiting from the reduced computational workloads and enhanced data heterogeneity handling of SL.

Following [27], there has been an emerging volume of empirical studies on SFL. e.g., [22, 21, 3, 23, 8, 31, 5]. However, a convergence analysis of SFL is missing in the literature , and this paper aims to provide a comprehensive convergence analysis under different conditions. Convergence theory is crucial for understanding the learning performance of SFL, particularly in the context of heterogeneous data and partial participation scenarios. In practical distributed systems, clients are prone to have different data distributions. Moreover, not all clients may be active or available at all times. These two issues can significantly affect the learning performance of SFL. We aim to provide convergence guarantees for SFL on heterogeneous data (under both full and partial participation). We further compare the results to FL and SL, which provides insights into the practical deployment of various distributed approaches.

1.2 Related Work

Convergence theories of FL and SL . There are many convergence results on FL. Most studies focus on data heterogeneity, e.g., [30, 16, 11, 10, 12]. Some studies look at partial participation, e.g., [35, 29, 26]. There are also convergence results on Mini-Batch SGD, e.g., [24, 33, 32], where [33] argued that the key difference between FL and Mini-Batch SGD is the communication frequency.

To our best knowledge, there is only one recent study [17] discussing the convergence of SL. The major difference to SL analysis lies in the sequential training manner across clients, while SFL clients perform parallel training.

1.3 Challenges and Contributions

Challenges of SFL convergence analysis . When data is homogeneous (IID) across clients, the convergence theory in [12] (mainly developed for FL) can be applied to SFL. When data is heterogeneous, however, the theory cannot be directly applied due to the client drift problem. The challenge is intensified with clients' partial participation, which induces bias in the training process. Despite that prior FL theories have handled data heterogeneity [16] and partial participation [29], SFL convergence analysis imposes unique challenges due to the dual-paced model aggregation and model updates at the client-side and server-side. More specifically,

Dual-paced model aggregation in SFL-V1 : In SFL-V1, the main server maintains one server-side model for each client, and it periodically aggregates the server-side models. When the main server aggregates its models at the same frequency as the clients, the analysis is the same to that of FL. However, FL analysis cannot be applied when aggregations occur at different frequencies, and it is challenging to analyze the impact of such discrepancy on SFL convergence.

Dual-paced model updates in SFL-V2 : In SFL-V2, the main server only maintains one version of server-side model. The clients update the client-side models in a parallel manner while the main server updates the server-side model in a sequential fashion. Hence, each client's local update depends on the randomness of the previous clients who have interacted with the main server. While [17] handled sequential client training, their theory cannot be applied to SFL-V2 as they did not consider the aggregation of client-side models. This makes our analysis more challenging than FL and SL.

Contributions . We summarize our contributions as follows:

We provide the first comprehensive convergence analysis of SFL. The analysis is more challenging than prior FL analysis due to the dual-paced model aggregation and model updates. To this end, we derive a key decomposition result (Proposition 3.5) that enables us to analyze the convergence from the server-side and client-side separately.

Based on the decomposition result, we prove that the convergence guarantees of both SFLV1 and SFL-V2 are O (1 /T ) for strongly convex objective and O (1 / 3 √ T ) for general convex objective, where T denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and more practical scenarios where some clients may be unavailable during training.

We conduct simulations on various datasets. We show that the results are consistent with our theories. We further show two surprising results: (i) SFL achieves a better performance when clients maintain a larger portion of the global model; (ii) SFL-V2 outperforms FL and SL when clients have highly heterogeneous data and the number of client is large.

The rest of the paper is organized as follows. Sec. 2 formulates the SFL model. Sec. 3 presents the convergence results for SFL. We conduct experiments in Sec. 4 and conclude in Sec. 5.

2 Problem Formulation

2.1 Model

We consider a set of clients N = { 1 , 2 , · · · , N } , where each client n ∈ N has a local private dataset D n of size D n = |D n | . Suppose the global model parameterized by x has L layers. In SFL, the global model is split at the L c -th layer (i.e., the cut layer) into two segments: a client-side model x c (from the first layer to layer L c ) and a server-side model x s (from layer L c + 1 to layer L ), where x = [ x c ; x s ] . Let x c,n denote the local client-side model of client n . The clients train models with the help of two servers: (i) fed server, which periodically aggregates clients' local models x c,n (similar to FL), and (ii) main server, who trains the server-side model x s . In this work, we consider two major SFL algorithms: SFL-V1 and SFL-V2 [27]. In SFL-V1, the main server maintains a separate server-side model x s,n corresponding to each client n . In comparison, in SFL-V2, the main server only maintains one model x s .

Let F n ( x ; ζ n ) denote the loss of model x over client n 's mini-batch instance ζ n , which is randomly sampled from client n 's dataset D n . Let F n ( x ) ≜ E ζ n ∼D n [ F n ( x ; ζ n )] denote the expected loss of model x over client n 's dataset. The goal of SFL is to minimize the expected loss of the model x

over the datasets of all clients:

where a n ∈ [0 , 1] is the weight of client n satisfying ∑ n ∈N a n = 1 . Typically, a n = D n / ∑ n ′ ∈N D n ′ , where a client with a larger data size is assigned a larger weight [34].

2.2 Algorithm Description

We provide a brief description of SFL. Refer to Appendix B for a more detailed discussion. SFL takes a total number of T rounds to solve (1). At the beginning of each round t , clients download the recent global client-side model from the fed server, where the model is an aggregated version of the client-side models of the clients from the previous round t -1 . Each round t contains two stages:

Stage 1: model training . Clients and the main server train the full global model for τ iterations in each round. In each iteration i < τ , there are three steps:

Step 1: client forward propagation . Each client n samples a mini-batch of data ζ t,i n from D n , computes the intermediate features (e.g., activation values at the cut layer) over its current model x t,i c,n , and sends the activation to the main server. The clients perform forward propagation in parallel.

Step 2: main server training . Upon receiving the activation of each client n ,

SFL-V1: the main server computes the loss using the current server-side model x t,i s,n . It then computes the gradients over x t,i s,n to update the model. It also computes the gradient over the activation at the cut layer, and sends it to client n .

SFL-V2: the main server computes the loss F n ( { x t,i c,n , x t,i s } ) , based on which it then updates the server-side model x t,i s . It also computes and sends the gradient over activation at the cut layer to client n . Note that the main server sequentially interacts with the clients in a randomized order.

Step 3: client backward propagation . Receiving gradient at the cut layer, each client n computes the client-side gradient using the chain rule, and then updates its model x t,i c,n .

Stage 2: model aggregation . Model aggregation can occur for both client-side and server-side models. For the client side, after τ iterations of model training (i.e., at the end of round t ), each client sends its current client-side model to the fed server. The fed server aggregates the clients' models (e.g., weighted averaging), which will be downloaded in the next round t +1 :

For the server side, (i) in SFL-V1, after ˜ τ iterations of training, the main server aggregates all serverside models. Note that ˜ τ does not necessarily need to equal τ , but when equality holds, SFL-V1 can be regarded as FL (despite the model splitting). (ii) In SFL-V2, no aggregation occurs since the main server only maintains one model.

2.3 Client Participation

We consider two cases: (i) full participation where all clients are available during training. This can model the scenarios where clients are organizations or companies who likely have sufficient computation and communication resources [7]; (ii) partial participation where some clients may be unavailable during training. This can model the cases where clients are edge devices (e.g., mobile phones) that are usually resource-constrained and may be disconnected from the SFL process.

To model partial participation, we consider independent participation probabilities for each client, allowing for arbitrary and heterogeneous participation probabilities. Specifically, we use q n ∈ [0 , 1] to denote client n 's participation level (or probability), and q = ( q n , n ∈ N ) . If q n = 1 , client n participates in every round of SFL with probability one. If q n < 1 , client n is unavailable in some rounds. Denote P t ( q ) as the set of participating clients in round t . In the presence of partial

participation, we need to modify (2) (and the potential server-side aggregation) to offset the incurred bias:

3 Convergence Analysis

We first make technical assumptions in Sec. 3.1. Then, we present a key technical result in Sec. 3.2 to support the SFL convergence analysis. Finally, we provide the convergence results under full participation and partial participation in Sec. 3.3 and Sec. 3.4, respectively.

3.1 Assumptions

We start with some conventional assumptions for convergence analysis in the FL literature.

Assumption 3.1. ( S -Smoothness ) Each client n 's loss function F n is S -smooth. That is, for all x , y ∈ R d ,

The smoothness assumption holds for many loss functions in, for example, logistic regression, softmax classifier, and l 2 -norm regularized linear regression [16].

Assumption 3.2. ( Unbiased and bounded stochastic gradients with bounded variance ) The stochastic gradients g n ( · ) of F n ( · ) is unbiased with the variance bounded by σ 2 n .

Assumption 3.3. ( Bounded gradients ) The expected squared norm of stochastic gradients is bounded by G 2 .

The value of σ n measures the level of stochasticity.

Assumption 3.4. ( Heterogeneity ) There exists an ϵ 2 such that the divergence between local and global gradients is bounded by ϵ 2 .

A larger ϵ 2 indicates a larger degree of data heterogeneity.

3.2 Decomposition

As discussed in Sec. 1.3, analyzing the performance bound of SFL can be more challenging than that of conventional FL counterparts due to the dual-paced model aggregation and model updates. To address this challenge, we decompose the convergence analysis into the server-side and client-side updates, respectively. We give the decomposition below.

Proposition 3.5. (Convergence decomposition) Let x ∗ ≜ [ x ∗ c ; x ∗ s ] denote the optimal global model that minimizes f ( · ) , and x T ≜ [ x T c ; x T s ] is the global model obtained after T rounds of SFL training. Under Assumption 3.1, we have

The proof is given in Appendix C.4. Proposition 3.5 is particularly useful. It shows that despite the challenging dual-paced updates, to bound the SFL performance gap, it suffices to separately bound the gap at the server-side and client-side models. Note that our decomposition can be easily applied to other distributed approaches such as SL. In addition, such a decomposition is not necessarily loose, as our derived bounds for SFL achieve the same order as in FL (see Appendix H.2 for details).

3.3 Results under Full Participation

Built upon Proposition 3.5, we first present the convergence results under full participation. For convenience, define

and let η t represent the learning rate at round t . Let f ∗ denotes the optimal global loss, i.e., f ∗ ≜ f ( x ∗ ) . All results are obtained based on Assumptions 3.1-3.4. The convergence results for SFL-V1 and SFL-V2 are summarized in Theorems 3.6 and 3.7, respectively 1 .

Theorem 3.6. ( SFL-V1: full participation)

µ -strongly convex : Let Assumptions 3.1 - 3.3 hold, and η t = 4 µ ˜ τ ( γ + t ) for client-side model and η t = 4 µτ ( γ + t ) for server-side model,

General convex : Let Assumptions 3.1 - 3.3 hold, and η t ≤ 1 2 Sτ max ,

Non-convex : Let Assumptions 3.1, 3.2, and 3.4 hold, and η t ≤ min { 1 16 Sτ max , τ min 8 SNτ 2 max ∑ N n =1 a 2 n } ,

Theorem 3.7. ( SFL-V2: full participation)

µ -strongly convex : Let Assumptions 3.1 - 3.3 hold, and η t = 4 µ ˜ τ ( γ + t ) for client-side model and η t = 4 µτ ( γ + t ) for server-side model,

General convex : Let Assumptions 3.1 - 3.3 hold, and η t ≤ 1 2 Sτ ,

Non-convex : Let Assumptions 3.1, 3.2, and 3.4 hold, and η t ≤ min { 1 16 Sτ , 1 8 SN 2 τ } ,

Proofs of Theorems 3.6-3.7 are given in Appendices D-E, respectively. We summarize the key findings below.

Convergence rate . The convergence bounds of both SFL-V1 and SFL-V2 achieve an order of O (1 /T ) on strongly convex (and non-convex) objectives. For general convex objectives, the convergence rate becomes O (1 / 3 √ T ) . 2 Note that our bounds match the existing bounds for FL and SL (in terms of the order of T ) on heterogeneous data for strongly convex objectives. For a more detailed comparison, please refer to Appendix H.2. 3

Impact of data heterogeneity . The convergence bounds increase as the level of data heterogeneity increases. For example, in (13), the bound increases in ϵ 2 (see Assumption 3.4). This means that SFL tends to perform worse when clients' data are more heterogeneous, which is a commonly observed phenomenon in distributed learning, e.g., FL.

Choice of learning rate. One should use a smaller learning rate when the number of local iteration τ increases. This bears a similar spirit to [16]. In addition, our results indicate that a proper choice of constant learning rate suffices for SFL convergence. It would be an interesting direction to investigate whether diminishing learning rates are able to achieve faster convergence.

Comparison between SFL-V1 and SFL-V2 . The convergence results between the two SFL versions are very similar, except that a 2 n (and a n ) in SFL-V1 are replaced by a 2 n +1 (and a n +1 ) in SFL-V2. See (11) and (14) for an inspection. We will show in Sec. 4 that SFL-V1 and SFL-V2 achieve similar accuracy (except under highly heterogeneous data).

3.4 Results under Partial Participation

Now, we present the results under partial participation.

Theorem 3.8. ( SFL-V1: partial participation)

µ -strongly convex : Let Assumptions 3.1 - 3.3 hold, and η t = 4 µ ˜ τ ( γ + t ) for client-side model and η t = 4 µτ ( γ + t ) for server-side model,

General convex : Let Assumptions 3.1 - 3.3 hold, and η t ≤ 1 2 Sτ max ,

Non-convex : Let Assumptions 3.1, 3.2, and 3.4 hold, and η t ≤ min { 1 16 Sτ max , τ min 8 SNτ 2 max ∑ N n =1 a 2 n qn } ,

Theorem 3.9. ( SFL-V2: partial participation)

µ -strongly convex : Let Assumptions 3.1 - 3.3 hold, and η t = 4 µ ˜ τ ( γ + t ) for client-side model and η t = 4 µτ ( γ + t ) for server-side model,

0

50

100

150

200

Training Round

0

20

40

60

80

100

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

20

40

60

80

100

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

20

40

60

80

100

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

20

40

60

80

100

Lc

=1

Lc

=2

Lc

=3

Lc

=4

Figure 2: Impact of the choice of cut layer on SFL performance.

General convex : Let Assumptions 3.1 - 3.3 hold, and η t ≤ 1 2 Sτ ,

Non-convex : Let Assumptions 3.1, 3.2, and 3.4 hold, and η t ≤ min { 1 16 Sτ , 1 8 SN 2 τ ∑ N n =1 a 2 n qn } ,

The proofs are given in Appendices F-G.

Impact of partial participation . In practical cross-device settings, some clients may not participate in all rounds of training, i.e., q n < 1 for some n . This brings an additional term G 2 /q n to the convergence bound (e.g., see (12) and (18)), meaning that partial participation worsens SFL performance. This is also observed in FL literature (e.g., [29]) and is consistent with our experimental results.

4 Experimental Results

4.1 Setup

We conduct experiments on CIFAR-10 and CIFAR-100 [13]. 4 To simulate data heterogeneity, we adopt the widely used Dirichlet distribution [6] with a controlling parameter β . Here, a smaller β corresponds to a higher level of data heterogeneity across clients. We use ResNet-18, which contains four blocks, as the model structure and consider four types of model splitting represented by L c = { 1 , 2 , 3 , 4 } , where L c = n means the model is split after the n -th residual block. We consider two major distributed approaches as the benchmark, i.e., FL (in particular FedAvg [18]) and SL [28]. The learning rates for SFL-V1, SFL-V2, FL, and SL are set as 0 . 01 . The batchsize b s is 128, and we run experiments for T = 200 rounds. Unless stated otherwise, we use N = 10 , β = 0 . 1 , E = 5 , where E is the number of local epochs for client-side model aggregation (i.e., every E times of training performed over each client's dataset, their client-side models are aggregated at the fed server), and hence τ = ⌈ D n b s ⌉ × E . We set τ = ˜ τ for the fair comparison to vanilla FL. The experiments are run on a CPU (Intel(R) Xeon(R) Gold 5320 at 2.20GHz) and a GPU (A100-PCIE-80GB). Our codes are provided in https://github.com/TIANGeng708/ Convergence-Analysis-of-Split-Federated-Learning-on-Heterogeneous-Data .

4.2 Impact of system parameters on SFL performance

Impact of cut layer . We first investigate how the choice of the cut layer L c affects the SFL performance. The results are reported in Fig. 2. We observe that for both SFL-V1 and SFL-V2,

0

50

100

150

200

Training Round

0

20

40

60

80

100

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0

20

40

60

80

100

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0

20

40

60

80

100

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0

20

40

60

80

100

=0.1

=0.5

=1

=

(a) SFL-V1 on CIFAR-10. (b) SFL-V2 on CIFAR-10. (c) SFL-V1 on CIFAR-100. (d) SFL-V2 on CIFAR-100.

0

50

100

150

200

Training Round

0

20

40

60

80

100

q

=0.2

q

=0.5

q

=1

0

50

100

150

200

Training Round

0

20

40

60

80

100

q

=0.2

q

=0.5

q

=1

0

50

100

150

200

Training Round

0

20

40

60

80

100

q

=0.2

q

=0.5

q

=1

0

50

100

150

200

Training Round

0

20

40

60

80

100

q

=0.2

q

=0.5

q

=1

(a) SFL-V1 on CIFAR-10. (b) SFL-V2 on CIFAR-10. (c) SFL-V1 on CIFAR-100. (d) SFL-V2 on CIFAR-100.

the performance increases in L c (i.e., clients have a larger proportion of the global model). This is associated with our empirical observation that the average client gradient variance gets smaller with L c . Intuitively, a smaller gradient variance implies a lower degree of the client drift issue, which leads to a better algorithm performance. 5 Based on this observation, we use L c = 4 for SFL (and SL) for the following experiments.

Impact of data heterogeneity . We study the impact of data heterogeneity on SFL performance, where we use β ∈ { 0 . 1 , 0 . 5 , 1 , ∞} , and β = ∞ means clients have IID data. The results are reported in Fig. 3. We observe that a higher level of data heterogeneity (i.e., a smaller β ) leads to slower algorithm convergence and a lower accuracy for both SFL-V1 and SFL-V2. The observation is consistent with our convergence bound, e.g., in (16), the performance bound increases in ϵ 2 . Note that the negative impact of heterogeneity is commonly observed in distributed learning literature including FL [7] and SL [21].

Impact of partial participation . We study the impact of client participation and let q n = q ∈ { 0 . 2 , 0 . 5 , 1 } , ∀ n. The results are reported in Fig. 4. We observe that a lower level of participation leads to less stable convergence and also a smaller accuracy. This is consistent with our convergence results, e.g., in (20), the bound decreases in clients' participation level q n . Partial participation is expected in practical cross-device scenarios where clients are resourced-constrained edge devices. It is important to develop efficient algorithms as well as effective incentive mechanisms to encourage clients' participation in SFL.

4.3 Comparison among SFL, FL, and SL.

We now compare SFL to FL and SL. We consider different combinations of data heterogeneity β ∈ { 0 . 1 , 0 . 5 } and cohort sizes N ∈ { 10 , 50 , 100 } . The results are reported in Fig. 5. When data is mildly heterogeneous (i.e., β = 0 . 5 ), SFL and FL have similar convergence rates and accuracy performance. Note that SL seems to under-perform SFL and FL. We think this is mainly due to the catastrophic forgetting issue, which has been observed in [21, 2].

SFL outperforms FL and SL under highly heterogeneous data and a large client number . When data becomes more non-IID (i.e., β = 0 . 1 ), SFL-V2 tends to outperform FL and SL. The improvement becomes more significant as the cohort size gets larger. The bottleneck of FL is the client drift issue caused by data heterogeneity. The bottleneck of SL is associated with the catastrophic forgetting. SFL-V2 is a hybrid combination of FL and SL, which can lead to a better tradeoff between client drift and forgetting. By appropriately choosing the cut layer, SFL-V2 outperforms

0

50

100

150

200

Training Round

0

20

40

60

80

100

SFL-V1

SFL-V2

FL

SL

(a)

β

= 0

.

5

, N

= 10

.

0

50

100

150

200

Training Round

0

20

40

60

80

100

SFL-V1

SFL-V2

FL

SL

(b)

β

= 0

.

1

, N

= 10

.

0

50

100

150

200

Training Round

0

20

40

60

80

100

SFL-V1

SFL-V2

FL

SL

(c)

β

= 0

.

1

, N

= 50

.

0

50

100

150

200

Training Round

0

20

40

60

80

100

SFL-V1

SFL-V2

FL

SL

(d)

β

= 0

.

1

, N

= 100

.

FL and SL. This observation also indicates that SFL-V2 can be a more appealing solution than FL for practical cross-device systems, as it achieves a better performance while requiring smaller computation overheads from edge devices.

5 Conclusion

In this work, we provided the first comprehensive convergence analysis of SFL for strongly convex, general-convex, and non-convex objectives on heterogeneous data. One key challenge is the dualpaced model updates. We get around this issue by decomposing the performance gap of the global model into the client-side and server-side gaps. We further extend our analysis to the more practical scenario with partial client participation. Experimental experiments validate our theories and further show that SFL can outperform FL and SL under highly heterogeneous data and a large client number. One limitation of our work is that our bounds for SFL achieve the same order (in terms of training rounds) as in FL, yet the experiments showed that SFL outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL are to be derived, which is an important future work. For future work, one can apply our derived bounds to optimize SFL system performance, considering model accuracy, communication overhead, and computational workload of clients. It is also interesting to theoretically analyze how the choice of the cut layer affects the SFL performance.

References

Ahmed M Abdelmoniem, Atal Narayan Sahu, Marco Canini, and Suhaib A Fahmy. Refl: Resource-efficient federated learning. In Proceedings of the Eighteenth European Conference on Computer Systems , pages 215-232, 2023.

Yansong Gao, Minki Kim, Sharif Abuadbba, Yeonjae Kim, Chandra Thapa, Kyuyeon Kim, Seyit A Camtepe, Hyoungshick Kim, and Surya Nepal. End-to-end evaluation of federated learning and split learning for internet of things. arXiv preprint arXiv:2003.13376 , 2020.

Dong-Jun Han, Hasnain Irshad Bhatti, Jungmoon Lee, and Jaekyun Moon. Accelerating federated learning with split learning on locally generated losses. In ICML workshop on federated learning for user privacy and data confidentiality , 2021.

Dong-Jun Han, Do-Yeon Kim, Minseok Choi, Christopher G Brinton, and Jaekyun Moon. Splitgp: Achieving both generalization and personalization in federated learning. Proc. of IEEE INFOCOM , 2023.

Pengchao Han, Chao Huang, Xingyan Shi, Jianwei Huang, and Xin Liu. Incentivizing participation in splitfed learning: Convergence analysis and model versioning. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS) , pages 846-856, 2024.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Chao Huang, Shuqi Ke, and Xin Liu. Duopoly business competition in cross-silo federated learning. IEEE Transactions on Network Science and Engineering , 2023.

Chao Huang, Geng Tian, and Ming Tang. When minibatch sgd meets splitfed learning: Convergence analysis and performance evaluation. arXiv preprint arXiv:2308.11953 , 2023.

Yang Jiao, Kai Yang, Tiancheng Wu, Chengtao Jian, and Jianwei Huang. Provably convergent federated trilevel learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 12928-12937, 2024.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 4519-4529. PMLR, 2020.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10713-10722, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In Proc. of ICLR , 2020.

Yipeng Li and Xinchen Lyu. Convergence analysis of sequential federated learning on heterogeneous data. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn` y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295 , 2020.

Sashank Reddi, Zachary Burr Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cný, Sanjiv Kumar, and Brendan McMahan, editors. Adaptive Federated Optimization , 2021.

Jinglong Shen, Nan Cheng, Xiucheng Wang, Feng Lyu, Wenchao Xu, Zhi Liu, Khalid Aldubaikhy, and Xuemin Shen. Ringsfl: An adaptive split federated learning towards taming client heterogeneity. IEEE Transactions on Mobile Computing , 2023.

Chamani Shiranthika, Zahra Hafezi Kafshgari, Parvaneh Saeedi, and Ivan V Baji´ c. Splitfed resilience to packet loss: Where to split, that is the question. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 367-377. Springer, 2023.

Veronika Stephanie, Ibrahim Khalil, and Mohammed Atiquzzaman. Digital twin enabled asynchronous splitfed learning in e-healthcare systems. IEEE Journal on Selected Areas in Communications , 41(11):3650-3661, 2023.

Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232 , 2019.

Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, and Chengqi Zhang. Federated learning on non-iid graphs via structural knowledge sharing. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 9953-9961, 2023.

Ming Tang and Vincent WS Wong. Tackling system induced bias in federated learning: Stratification and convergence analysis. In Proc. of IEEE INFOCOM , pages 1-10, 2023.

Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proc. of AAAI , volume 36, pages 8485-8493, 2022.

Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. ICLR Workshop on AI for Social Good , 2019.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. Advances in Neural Information Processing Systems , 35:19124-19137, 2022.

Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE journal on selected areas in communications , 37(6):1205-1221, 2019.

Dinah Waref and Mohammed Salem. Split federated learning for emotion detection. In 2022 4th Novel Intelligent and Leading Emerging Sciences Conference (NILES) , pages 112-115. IEEE, 2022.

Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334-10343. PMLR, 2020.

Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems , 33:6281-6292, 2020.

Wen Wu, Mushu Li, Kaige Qu, Conghao Zhou, Xuemin Shen, Weihua Zhuang, Xu Li, and Weisen Shi. Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications , 41(4):1051-1066, 2023.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. Proc. of ICLR , 2021.

A Appendix / supplemental material

We organize the entire appendix file as follows:

In Sec. B, we provide detailed algorithmic descriptions.

In Sec. C, we provide notations and some technical lemmas.

In Sec. C.1, we provide some notations

In Sec. C.2, we recall SFL-V1 and SFL-V2

In Sec. C.3, we recall the assumptions

In Sec. C.4, we provide some useful technical lemmas together with their proofs

In Sec. D, we prove Theorem 3.6, i.e., convergence of SFL-V1 under full participation.

In Sec. D.1, we prove the strongly convex case

In Sec. D.2, we prove the general convex case

In Sec. D.3, we prove the non-convex case

In Sec. E, we prove Theorem 3.7, i.e., convergence of SFL-V2 under full participation.

In Sec. E.1, we prove the strongly convex case

In Sec. E.2, we prove the general convex case

In Sec. E.3, we prove the non-convex case

In Sec. F, we prove Theorem 3.8, i.e., convergence of SFL-V1 under partial participation.

In Sec. F.1, we prove the strongly convex case

In Sec. F.2, we prove the general convex case

In Sec. F.3, we prove the non-convex case

In Sec. G, we prove Theorem 3.9, i.e., convergence of SFL-V2 under partial participation.

In Sec. G.1, we prove the strongly convex case

In Sec. G.2, we prove the general convex case

In Sec. G.3, we prove the non-convex case

In Sec. H, we SFL to other distributed approaches, i.e., FL, SL, and Mini-Batch SGD.

In Sec. H.2, we compare their convergence bounds

In Sec. H.3, we compare their overheads in terms of communication and computation

In Sec. I, we provide more experimental results.

B Algorithm description

For version 1, the client-side model parameter and M-server-side model parameter are aggregated every τ and ˜ τ iterations, respectively. In iteration i of round t , each client n samples a mini-batch of data ζ t,i n from D n , computes the intermediate features h ( x t,i c,n ; ζ t,i n ) (e.g., activation values at the cut layer) over its current model x t,i c,n , and sends h ( x t,i c,n ; ζ t,i n ) to the M-server. For each client n , the M-server computes the loss F n ( h ( x t,i c,n ; ζ t,i n ) , x t,i s,n ) based on x t,i s,n . Let ∇ denote a gradient operator and ∇ w F represents the gradient of F w.r.t. w . The M-server computes the M-server-side gradient g t,i s,n ( x t,i s,n ; ζ t,i n ) = ∇ x s F n ( h ( x t,i c,n ; ζ t,i n ) , x t,i s,n ) , the gradient over the intermediate features (activations) at the cut layer r t,i c,n ( x t,i s,n ; ζ t,i n ) = ∇ h F n ( h ( x t,i c,n ; ζ t,i n ) , x t,i s,n ) , and sends r t,i c,n ( x t,i s,n ; ζ t,i n ) to client n . Each client n computes the client-side gradient g t,i c,n ( x t,i c,n ; ζ t,i n ) based on r t,i c,n ( x t,i s,n ; ζ t,i n ) using the chain rule.

For version 2, the client-side model is aggregated every τ iterations, while the M-server trains only one version of the M-server-side model.

C Notations and technical lemmas

C.1 Notations

Recall that the objective of SFL is given by

We define

x c and x s : global model parameter on the clients and server sides, respectively.

x c,n and x s,n : local forms of parameter on client n and on the main server corresponding to client n (in SFL-V1).

∇ F c,n ( · ) and ∇ F s,n ( · ) : the gradients of F n ( · ) over x c and x s , respectively.

g c,n ( · ) and g s,n ( · ) : the stochastic gradients of F n ( · ) over x c and x s , respectively.

For convenience, we omit the notation for mini-batch training data when referring to stochastic gradients.

Further, we recall how SFL-V1 and SFL-V2 update models below.

C.2 SFL-V1 and SFL-V2 model updates

Let q n denote the participating probability of client n and define q := { q 1 , . . . , q N } . We denote I t n as a binary variable, taking 1 if client n participates in model training in round t , and 0 otherwise. I t n follows a Bernoulli distribution with an expectation of q n . Denote P t ( q ) as the set of participating clients in round t .

Parameter update for SFL-V1:

Local training of client n : x t, 0 c,n ← x t c , x t,i +1 c,n ← x t,i c,n -η t g t,i c,n ( x t,i c,n ) , x t +1 c,n ← x t,τ c,n ;

Client-side global aggregation:

-Full participation: x t +1 c ← x t c -η t ∑ n ∈N a n ∑ τ i =0 g t,i c,n ( x t,i c,n ) ;

-Partial participation: x t +1 c ← x t c -η t ∑ n ∈P t ( q ) a n q n ∑ τ i =0 g t,i c,n ( x t,i c,n ) ;

M-server-side model update:

-Full participation: x t +1 s ← x t s -η t ∑ n ∈N a n ∑ ˜ τ -1 i =0 g t,i s,n ( x t,i s,n ) ;

-Partial participation: x t +1 s ← x t s -η t ∑ n ∈P t ( q ) a n q n ∑ ˜ τ -1 i =0 g t,i s,n ( x t,i s,n ) .

Parameter update for SFL-V2:

Local training of client n : x t, 0 c,n ← x t c , x t,i +1 c,n ← x t,i c,n -η t g t,i c,n ( x t,i c,n ) , x t +1 c,n ← x t,τ c,n ;

Client-side global aggregation:

-Full participation: x t +1 c ← x t c -η t ∑ n ∈N a n ∑ τ i =0 g t,i c,n ( x t,i c,n ) ;

-Partial participation: x t +1 c ← x t c -η t ∑ n ∈P t ( q ) a n q n ∑ τ i =0 g t,i c,n ( x t,i c,n ) ;

M-server-side model update:

-Full participation: x t +1 s ← x t s -η t ∑ n ∈N ∑ τ -1 i =0 g t,i s,n ( x t,i s,n ) ;

-Partial participation: x t +1 s ← x t s -η t ∑ n ∈P t ( q ) 1 q n ∑ τ -1 i =0 g t,i s,n ( x t,i s,n ) .

C.3 Assumptions

We further recall the following assumptions for clients' loss functions in the proof.

Assumption C.1. For each client n ∈ N :

The loss F n ( · ) is S -smooth:

The stochastic gradients of F n ( · ) are unbiased with the variance bounded by σ 2 n :

The expected squared norm of stochastic gradients is bounded by G 2 :

(Bounded gradient divergence) There exists a constant ϵ > 0 , such that the divergence between local and global gradients is bounded by ϵ 2 :

Assumption C.2. For each client n ∈ N :

The loss F n ( · ) is µ -strongly convex for some µ ≥ 0 :

Here, we allow that µ = 0 , referring to this case of the general convex.

C.4 Technical Lemmas

Lemma C.3. [Lemma 5 in [10]] The following holds for any S -smooth and µ -strongly convex function h , and any x, y, z in the domain of h :

Proof of Proposition 3.5

Proposition 3.5 (Convergence decomposition) Let x ∗ ≜ [ x ∗ c ; x ∗ s ] denote the optimal global model that minimizes f ( · ) , and x T ≜ [ x T c ; x T s ] is the global model obtained after T rounds of SFL training. Under Assumption 3.1, we have

Proof. Since F n 's are S -smooth, it is easy to show that the global loss function f ( · ) is also S -smooth. Thus, we have

Since x T ≜ [ x T c ; x T s ] , and x ∗ ≜ [ x ∗ c ; x ∗ s ] , we have

Substituting (34) into (33), we complete the proof.

Proposition C.4 (Decomposition in each round) . Under Assumption C.1, we have

Proof. The proposition can be easily proved by the S -smoothness of f ( · ) .

Lemma C.5. [Multiple iterations of local training in each round] Under Assumption C.1, if we let η t ≤ 1 √ 6 Sτ and run client n 's local model for τ iteration continuously in any round t , we have

Proof. Similar to Lemma 3 in [20], we have

where we use Assumption C.1, ( X + Y ) 2 ≤ (1 + a ) X 2 + ( 1 + 1 a ) Y 2 for some positive a , and η t ≤ 1 √ 6 Sτ .

Let

We have

We can show that

Note that A t := A t, 0 = E [ ∥ x t -x t ∥ 2 ] = 0 . Accumulate the above for τ iterations, we have

The first inequality is due to ∑ N -1 i =0 x i = x N -1 X -1 and the third line results from (1 + n x ) x ≤ e n . Thus, we finish the proof.

Lemma C.6. [Multiple iterations of local training in each round] Under Assumption C.1, if we let η t ≤ 1 √ 8 Sτ and run client n 's local model for τ iteration continuously in any round t , we have

Proof.

where we have applied Assumption C.1, ( X + Y ) 2 ≤ (1 + a ) X 2 + ( 1 + 1 a ) Y 2 for some positive a , and η t ≤ 1 √ 8 Sτ .

Let

We have

We can show that

where we use ∑ N -1 i =0 x i = x N -1 X -1 and (1 + n x ) x ≤ e n . Therefore, we complete the proof.

Lemma C.7. [Multiple iterations of local gradient accumulation in each round] Under Assumption C.1, if we let η t ≤ 1 2 Sτ and run client n 's local model for τ iteration continuously in any round t , we have

Proof.

We define the following notation for simplicity:

We have

We can show that

D Proof for Theorem 3.6

We organize the proof of Theorem 3.6 as follows:

In Sec. D.1, we prove the strongly convex case.

In Sec. D.2, we prove the general convex case.

In Sec. D.3, we prove the non-convex case.

D.1 Strongly convex case for SFL-V1

D.1.1 One-round Parallel Update for M-Server-Side Model

Lemma D.1. Under Assumptions C.1 and C.2, if η t ≤ 1 2 S ˜ τ , in round t , the M-server-side model evolves as

We prove Lemma D.1 as follows.

Proof. We use x t,i s,n as the M-server-side model when the M-server interacts with client n for the i -th iteration of model training at round t . Using the (sequential) gradient update rule of x t +1 s = x t s -η t ∑ N n =1 ∑ ˜ τ -1 i =0 a n g t,i s,n ({ x t,i c,n , x t,i s,n }) , we have

where the second equality is from ( a + b ) 2 = a 2 + 2 ab + b 2 and the last inequality is due to E [ ∇ x s f ({ x t,i c , x t,i s }) -g t,i s ] = 0 .

The first part in (54) is

where we use ∇ x s f ( { x c , x s } ) = ∑ N n =1 a n ∇ x s F n ( { x c , x s } ) . For (55), we have

where the first inequality applies triangle inequality. In the last inequality, we apply the bound of variance and expected squared norm for stochastic gradients in Assumption C.1.

Since F n ( x ) is S -smooth and µ -strongly convex, using Lemma C.3 we have

From Assumption C.1, the second part in (54) is bounded by

We now prove the convergence error. Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 s -x ∗ s ∥ ∥ 2 ] . We can rewrite (60) as:

where B 1 := 4 N ∑ N n =1 a 2 n ( 2 σ 2 n + G 2 ) and B := 192 S ∑ N n =1 a n ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β ˜ τ ( γ + t ) , i.e, η t ˜ τ 2 = β γ + t , where β = 2 µ , γ = 8 S µ -1 . It is easy to show that η t ≤ 1 2 S ˜ τ for all t . Next, we will prove that ∆ t +1 ≤ v γ + t +1 , where v = max { 4 B 1 µ 2 + 8 B 2 µ 3 ( γ +1) , ( γ +1)∆ 0 } . We prove this by induction. First, the definition of v ensures that it holds for t = -1 . Assume the conclusion holds for some t , it follows that

Hence, we have proven that ∆ t ≤ v γ + t , ∀ t . Therefore, we have

D.1.2 One-round Parallel Update for Client-Side Models

Under Assumptions C.1 and C.2, if η t ≤ 1 2 Sτ , in round t , Lemma D.1 gives

Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] . We can rewrite (70) as:

where B 1 := 4 N ∑ N n =1 a 2 n ( 2 σ 2 n + G 2 ) and B 2 := 192 S ∑ N n =1 a n ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β τ ( γ + t ) , i.e, η t τ 2 = β γ + t , where β = 2 µ , γ = 8 S µ -1 . It is easy to show that η t ≤ 1 2 Sτ for all t . For v = max { 4 B 1 µ 2 + 8 B 2 µ 3 ( γ +1) , ( γ +1)∆ 0 } , we can prove that ∆ t ≤ v γ + t , ∀ t . Therefore, we have

D.1.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (69) and (72) using Proposition 3.5 by setting η t ≤ 1 2 S max { τ, ˜ τ } . We have

D.2 General convex case for SFL-V1

D.2.1 One-round Parallel Update for M-Server-Side Model

By Lemma D.1 with µ = 0 and η t ≤ 1 2 S ˜ τ , we have

D.2.2 One-round Parallel Update for Client-Side Models

By Lemma D.1 with µ = 0 and η t ≤ 1 2 Sτ , we have

D.2.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (74) and (75) as follows

Then, we can obtain the relation between E [ ∥ ∥ x t +1 -x ∗ ∥ ∥ 2 ] and E [ ∥ x t -x ∗ ∥ 2 ] , which is related to E [ f ( x t ) -f ( x ∗ )] . Applying Lemma 8 in [17] and let τ min := min { ˜ τ, τ } and η t ≤ 1 2 S max { τ, ˜ τ } , we obtain the performance bound as

D.3 Non-convex case for SFL-V1

D.3.1 One-round Parallel Update for M-Server-Side Model

For the server, we have

where we apply Assumption C.1, ∇ x s f ( x t ) = ∑ N n =1 a n ∇ x s F n ( x t ) , and ⟨ a, b ⟩ ≤ a 2 + b 2 2 . By Lemma C.6 with η t ≤ 1 √ 8 S ˜ τ , we have

Thus, (78) becomes

Furthermore, we have

where the last line uses Assumption C.1 and E [ ∥ z ∥ 2 ] = ∥ E [ z ] ∥ 2 + E [ ∥ z -E [ z ] ∥ 2 ] for any random variable z .

By Lemma C.7 with η t ≤ 1 2 S ˜ τ , we have

Thus, (81) becomes

D.3.2 One-round Parallel Update for Client-Side Models

The analysis of the client-side model update is similar to the server. Thus, we have

D.3.3 Superposition of M-Server and Clients

Applying (80), (83), (85) and (84) into (36) in Proposition C.4 and define τ min ≜ min { τ, ˜ τ } , τ max ≜ max { τ, ˜ τ } , we have

where we first let η ≤ 16 Sτ max and then let η ≤ 8 SN τ 2 max τ min ∑ N n =1 a 2 n . We also use ∥∇ x f ( x ) ∥ = ∥∇ x c f ( x t ) ∥ 2 + ∥∇ x s f ( x t ) ∥ 2 . Rearranging the above we have

Taking expectation and averaging over all t , we have

E Proof of Theorem 3.7

In Sec. E.1, we prove the strongly convex case.

In Sec. E.2, we prove the general convex case.

In Sec. E.3, we prove the non-convex case.

E.1 Strongly convex case for SFL-V2

E.1.1 One-round Sequential Update for M-Server-Side Model

Lemma E.1. Under Assumptions C.1 and C.2, if η t ≤ 1 2 Sτ , in round t , the M-server-side model evolves as

We prove Lemma E.1 as follows.

Proof. We use x t,i s,n as the M-server-side model when the M-server interacts with client n for the i -th iteration of model training at round t . Using the (sequential) gradient update rule of x t +1 s = x t s -η t ∑ N n =1 ∑ τ -1 i =0 g t,i s ({ x t,i c,n , x t,i s,n }) , we have

where the first equality is from ( a + b ) 2 = a 2 + 2 ab + b 2 and the last inequality is due to E [ ∇ x s f ({ x t,i c , x t,i s }) -g t,i s ] = 0 .

The first part in (91) is

where we use ∇ x s f ( { x c , x s } ) = ∑ N n =1 ∇ x s F n ( { x c , x s } ) . For (92), we have

where the first inequality applies triangle inequality. In the last inequality, we apply the bound of variance and expected squared norm for stochastic gradients in Assumption C.1.

Since F n ( x ) is S -smooth and µ -strongly convex, using Lemma C.3 we have

From Assumption C.1, the second part in (91) is bounded by

Thus, (91) becomes

Using the above lemma, we can prove the convergence error. Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 s -x ∗ s ∥ ∥ 2 ] . We can rewrite (97) as:

where B 1 := 4 N ∑ N n =1 ( 2 σ 2 n + G 2 ) and B := 192 S ∑ N n =1 ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β Nτ ( γ s + t ) , i.e, Nη t τ 2 = β γ s + t , where β = 2 µ , γ s = 8 S Nµ -1 . It is easy to show that η t ≤ 1 2 Sτ for all t . Next, we will prove that ∆ t +1 ≤ v γ s + t +1 , where v = max { 4 B 1 µ 2 + 8 B 2 µ 3 ( γ s +1) , ( γ s +1)∆ 0 } . We prove this by induction. First, the definition of v ensures that it holds for t = -1 . Assume the conclusion holds for some t , it follows that

Hence, we have proven that ∆ t ≤ v γ s + t , ∀ t . Therefore, we have

E.1.2 One-round Parallel Update for Client-Side Models

Under Assumptions C.1 and C.2, if η t ≤ 1 2 Sτ , in round t , Lemma D.1 gives

Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] . We can rewrite (107) as:

where B 1 := 4 N ∑ N n =1 a 2 n ( 2 σ 2 n + G 2 ) and B 2 := 192 S ∑ N n =1 a n ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β τ ( γ c + t ) , i.e, η t τ 2 = β γ c + t , where β = 2 µ , γ c = 8 S µ -1 . It is easy to show that η t ≤ 1 2 Sτ for all t . For v = max { 4 B 1 µ 2 + 8 B 2 µ 3 ( γ c +1) , ( γ c +1)∆ 0 } , we can prove that ∆ t ≤ v γ c + t , ∀ t . Therefore, we have

E.1.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (106) and (109) using Proposition 3.5. For η t ≤ 1 2 Sτ and γ = 8 S µ -1 , we have

E.2 General convex case for SFL-V2

E.2.1 One-round Sequential Update for M-Server-Side Model

By Lemma E.1 with µ = 0 and η t ≤ 1 2 Sτ , we have

E.2.2 One-round Parallel Update for Client-Side Models

By Lemma D.1 with µ = 0 and η t ≤ 1 2 Sτ , we have

E.2.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (111) and (112) as follows

Then, we can obtain the relation between E [ ∥ ∥ x t +1 -x ∗ ∥ ∥ 2 ] and E [ ∥ x t -x ∗ ∥ 2 ] , which is related to E [ f ( x t ) -f ( x ∗ )] . Applying Lemma 8 in [17], we obtain the performance bound as

E.3 Non-convex case for SFL-V2

E.3.1 One-round Sequential Update for M-Server-Side Model

For the server, we have

where we apply Assumption C.1, ∇ x s f ( x t ) = ∑ N n =1 ∇ x s F n ( x t ) , and ⟨ a, b ⟩ ≤ a 2 + b 2 2 . By Lemma C.6 with η t ≤ 1 √ 8 Sτ , we have

Thus, (115) becomes

Furthermore, we have

where the last line uses Assumption C.1 and E [ ∥ z ∥ 2 ] = ∥ E [ z ] ∥ 2 + E [ ∥ z -E [ z ] ∥ 2 ] for any random variable z .

By Lemma C.7 with η t ≤ 1 2 Sτ , we have

Thus, (118) becomes

E.3.2 One-round Parallel Update for Client-Side Models

The analysis of the client-side model update is the same as the client's model update in version 1. Thus, we have

E.3.3 Superposition of M-Server and Clients

Applying (117), (120), (122) and (121) into (36) in Proposition C.4, we have

where we first let η t ≤ 1 16 Sτ and then let η t ≤ 1 8 SN 2 τ . We have applied ∑ N n =1 a 2 n ≤ N . Rearranging the above we have

Taking expectation and averaging over all t , we have

F Proof of Theorem 3.8

In Sec. F.1, we prove the strongly convex case.

In Sec. F.2, we prove the general convex case.

In Sec. F.3, we prove the non-convex case.

F.1 Strongly convex case for SFL-V1

F.1.1 One-round Parallel Update for M-Server-Side Model

We first bound the M-server-side model update in one round for full participation ( q n = 1 for all n ), and then compute the difference between full participation and partial participation ( q n < 1 for some n ). We denote I t n as a binary variable, taking 1 if client n participates in model training in round t , and 0 otherwise. Practically, I t n follows a Bernoulli distribution with an expectation of q n .

For full participation, Lemma D.1 gives

Considering that each client n participates in model training with a probability q n , we have

where we use E ∥ X -E X ∥ 2 ≤ E ∥ X ∥ 2 , E [ I t n ] = q n , and x t +1 s = x t s -η t ∑ n ∈P t ( q ) ∑ ˜ τ -1 i =0 a 2 n q n g t,i s,n ({ x t,i c,n , x t,i s,n }) .

Combining the above gives

Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 s -x ∗ s ∥ ∥ 2 ]

where B 1 := 4 N ∑ N n =1 a 2 n ( 2 σ 2 n + G 2 ) +4 NG 2 ∑ N n =1 a 2 n q n and B := 192 S ∑ N n =1 a n ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β ˜ τ ( γ s + t ) , i.e, η t ˜ τ 2 = β γ s + t , where β = 2 µ , γ s = 8 S µ -1 . It is easy to show that η t ≤ 1 2 S ˜ τ for all t . We can prove that ∆ t ≤ v γ s + t , ∀ t . Therefore, we have

F.1.2 One-round Parallel Update for Client-Side Models

Define x c t = ∑ N n =1 a n x t c,n , which represents the aggregating weights in round t for full participation. Using a similar derivation as the M-server side, we first bound the client-side model update in one round for full participation E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] and then bound the difference of client-side model parameters between full participation and partial participation E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] . The overall gradient update rule of clients in each training round is x t +1 c = x t c -η t ∑ n ∈P t ( q ) ∑ τ -1 i =0 a n q n g t,i c,n ({ x t,i c,n , x t,i s,n }) .

Under Assumptions C.1 and C.2, if η t ≤ 1 2 Sτ , in round t , Lemma D.1 gives

Considering that each client n participates in model training with a probability q n , we have

where we use E ∥ X -E X ∥ 2 ≤ E ∥ X ∥ 2 , E [ I t n ] = q n , and x t +1 c = x t c -η t ∑ n ∈P t ( q ) ∑ τ -1 i =0 a n q n g t,i c,n ({ x t,i c,n , x t,i s,n }) .

We obtain the client-side model parameter update in one round for partial participation by combining the two terms and we have

where we consider E [ f ( x t ) -f ( x ∗ )] ≥ 0 .

Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] . We can rewrite (163) as:

where B 1 := 4 N ∑ N n =1 a 2 n ( 2 σ 2 n + G 2 ) + 4 NG 2 ∑ N n =1 a 2 n q n and B 2 := 192 S ∑ N n =1 a n ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β τ ( γ c + t ) , i.e, η t τ 2 = β γ c + t , where β = 2 µ , γ c = 8 S µ -1 . It is easy to show that η t ≤ 1 2 Sτ for all t . For v = max { 4 B 1 µ 2 + 8 B 2 µ 3 ( γ c +1) , ( γ c +1)∆ 0 } , we can prove that ∆ t ≤ v γ c + t , ∀ t . Therefore, we have

F.1.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (130) and (135) using Proposition 3.5. For η t ≤ 1 2 S max { τ, ˜ τ } and γ = 8 S µ -1 , we have

F.2 General convex case for SFL-V1

F.2.1 One-round Parallel Update for M-Server-Side Model

By Lemma D.1 with µ = 0 and η t ≤ 1 2 S ˜ τ , we have

Considering that each client n participates in model training with a probability q n , we have

Thus, we have

F.2.2 One-round Parallel Update for Client-Side Models

By Lemma D.1 with µ = 0 and η t ≤ 1 2 Sτ , we have

Considering that each client n participates in model training with a probability q n , we have

Thus, we have

F.2.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (139) and (142) as follows

Then, we can obtain the relation between E [ ∥ ∥ x t +1 -x ∗ ∥ ∥ 2 ] and E [ ∥ x t -x ∗ ∥ 2 ] , which is related to E [ f ( x t ) -f ( x ∗ )] . Applying Lemma 8 in [17] and let τ min := min { ˜ τ, τ } , we obtain the performance bound as

F.3 Non-convex case for SFL-V1

F.3.1 One-round Parallel Update for M-Server-Side Model

For the server, we have

where we apply Assumption C.1, ∇ x s f ( x t ) = ∑ N n =1 a n ∇ x s F n ( x t ) , ⟨ a, b ⟩ ≤ a 2 + b 2 2 , and E [ I t n ] = q n .

By Lemma C.6 with η t ≤ 1 √ 8 S ˜ τ , we have

Thus, (145) becomes

Furthermore, we have

where the last line uses Assumption C.1 and E [ ∥ z ∥ 2 ] = ∥ E [ z ] ∥ 2 + E [ ∥ z -E [ z ] ∥ 2 ] for any random variable z .

By Lemma C.7 with η t ≤ 1 2 S ˜ τ , we have

Thus, (148) becomes

F.3.2 One-round Parallel Update for Client-Side Models

The analysis of the client-side model update is similar to the server. Thus, we have

F.3.3 Superposition of M-Server and Clients

where we first let η t ≤ 1 16 Sτ max and then let η t ≤ 1 8 SN τ 2 max τ min ∑ N n =1 a 2 n qn . We also use ∥∇ x f ( x t ) ∥ 2 = ∥∇ x c f ( x t ) ∥ 2 + ∥∇ x s f ( x t ) ∥ 2 . Rearranging the above we have

Taking expectation and averaging over all t , we have

G Proof of Theorem 3.9

In Sec. G.1, we prove the strongly convex case.

In Sec. G.2, we prove the general convex case.

In Sec. G.3, we prove the non-convex case.

G.1 Strongly convex case for SFL-V2

G.1.1 One-round Sequential Update for M-Server-Side Model

We first bound the M-server-side model update in one round for full participation ( q n = 1 for all n ), and then compute the difference between full participation and partial participation ( q n < 1 for some n ). We denote I t n as a binary variable, taking 1 if client n participates in model training in round t , and 0 otherwise.

For full participation, Lemma E.1 gives

Considering that each client n participates in model training with a probability q n , we have

where we use E ∥ X -E X ∥ 2 ≤ E ∥ X ∥ 2 , E [ I t n ] = q n , and x t +1 s = x t s -η t ∑ n ∈P t ( q ) ∑ τ -1 i =0 1 q n g t,i s,n ({ x t,i c,n , x t,i s,n }) .

Combining the above gives

Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 s -x ∗ s ∥ ∥ 2 ]

where B 1 := 4 N ∑ N n =1 ( 2 σ 2 n + G 2 ) +4 NG 2 ∑ N n =1 1 q n and B := 192 S ∑ N n =1 ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β Nτ ( γ s + t ) , i.e, Nη t τ 2 = β γ s + t , where β = 2 µ , γ s = 8 S Nµ -1 . It is easy to show that η t ≤ 1 2 Sτ for all t . We can prove that ∆ t ≤ v γ s + t , ∀ t . Therefore, we have

G.1.2 One-round Parallel Update for Client-Side Models

Define x c t = ∑ N n =1 a n x t c,n , which represents the aggregating weights in round t for full participation. Using a similar derivation as the M-server side, we first bound the client-side model update in one round for full participation E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] and then bound the difference of client-side model parameters between full participation and partial participation E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] . The overall gradient update rule of clients in each training round is x t +1 c = x t c -η t ∑ n ∈P t ( q ) ∑ τ -1 i =0 a n q n g t,i c,n ({ x t,i c,n , x t,i s,n }) .

Under Assumptions C.1 and C.2, if η t ≤ 1 2 Sτ , in round t , Lemma D.1 gives

Considering that each client n participates in model training with a probability q n , we have

where we use E ∥ X -E X ∥ 2 ≤ E ∥ X ∥ 2 , E [ I t n ] = q n , and x t +1 c = x t c -η t ∑ n ∈P t ( q ) ∑ τ -1 i =0 a n q n g t,i c,n ({ x t,i c,n , x t,i s,n }) .

We obtain the client-side model parameter update in one round for partial participation by combining the two terms and we have

Let ∆ t +1 ≜ E [ ∥ ∥ x t +1 c -x ∗ c ∥ ∥ 2 ] . We can rewrite (163) as:

where B 1 := 4 N ∑ N n =1 a 2 n ( 2 σ 2 n + G 2 ) + 4 NG 2 ∑ N n =1 a 2 n q n and B 2 := 192 S ∑ N n =1 a n ( 2 σ 2 n + G 2 ) .

Consider a diminishing stepsize η t = 2 β τ ( γ c + t ) , i.e, η t τ 2 = β γ c + t , where β = 2 µ , γ c = 8 S µ -1 . It is easy to show that η t ≤ 1 2 Sτ for all t . For v = max { 4 B 1 µ 2 + 8 B 2 µ 3 ( γ c +1) , ( γ c +1)∆ 0 } , we can prove that ∆ t ≤ v γ c + t , ∀ t . Therefore, we have

G.1.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (160) and (165) using Proposition 3.5. For η t ≤ 1 2 Sτ and γ = 8 S µ -1 , we have

G.2 General convex case for SFL-V2

G.2.1 One-round Sequential Update for M-Server-Side Model

By Lemma E.1 with µ = 0 and η t ≤ 1 2 Sτ , we have

Considering that each client n participates in model training with a probability q n , we have

Thus, we have

G.2.2 One-round Parallel Update for Client-Side Models

By Lemma D.1 with µ = 0 and η t ≤ 1 2 Sτ , we have

Considering that each client n participates in model training with a probability q n , we have

Thus, we have

G.2.3 Superposition of M-Server and Clients

We merge the M-server-side and client-side models in (169) and (172) as follows

Then, we can obtain the relation between E [ ∥ ∥ x t +1 -x ∗ ∥ ∥ 2 ] and E [ ∥ x t -x ∗ ∥ 2 ] , which is related to E [ f ( x t ) -f ( x ∗ )] . Applying Lemma 8 in [17], we obtain the performance bound as

G.3 Non-convex case for SFL-V2

G.3.1 One-round Sequential Update for M-Server-Side Model

For the server, we have

where we apply Assumption C.1, ∇ x s f ( x t ) = ∑ N n =1 ∇ x s F n ( x t ) , ⟨ a, b ⟩ ≤ a 2 + b 2 2 , and E [ I t n ] = q n .

By Lemma C.6 with η t ≤ 1 √ 8 Sτ , we have

Thus, (175) becomes

Furthermore, we have

where the last line uses Assumption C.1 and E [ ∥ z ∥ 2 ] = ∥ E [ z ] ∥ 2 + E [ ∥ z -E [ z ] ∥ 2 ] for any random variable z .

By Lemma C.7 with η t ≤ 1 2 Sτ , we have

Thus, (178) becomes

G.3.2 One-round Parallel Update for Client-Side Models

The analysis of the client-side model update is the same as the client's model update in version 1. Thus, we have

G.3.3 Superposition of M-Server and Clients

Applying (177), (180), (182) and (181) into (36) in Proposition C.4, we have

where we first let η t ≤ 1 16 Sτ and then let η t ≤ 1 8 SN 2 τ ∑ N n =1 1 qn . We have applied ∑ N n =1 a 2 n ≤ N . Rearranging the above we have

Taking expectation and averaging over all t , we have

H Comparative Analysis

H.1 Main technical results

We conclude the convergence results in our paper in Table 1.

H.2 Comparison of Bounds

We compare our derived bounds for SFL to other distributed approaches. For simplicity, we let a n = 1 /N in (1) and σ n = σ for all n in (6). The result are summarized in Table 2. Since different convergence theories make slightly different assumptions, we clarify them below.

In [33], σ 2 ∗ is the variance of the stochastic gradient at the optimum: E ζ n ∼D n [ ∥ g n ( x ∗ , ζ n ) -∇ F n ( x ∗ ) ∥ 2 ] ≤ σ 2 ∗ . In [16], Γ = f ∗ -∑ N n =1 F ∗ n /N characterizes the client heterogeneity. In [17], ϵ 2 ∗ characterizes the client heterogeneity at the optimum, similar to [12], i.e., 1 N ∑ N n =1 ||∇ F n ( x ∗ ) || 2 = ϵ 2 ∗ .

The key observation is that our derived bounds match the other distributed approaches in the order of T and they all achieve O (1 /T ) .

Furthermore, we will compare the convergence upper bounds of SFL to those of distributed SGD in [12] (with parameters p = 1 and ¯ ζ 2 = 0 ) as follows:

In the aforementioned table, we denote σ n = σ , define I err ≜ || x 0 -x ∗ || 2 , and represent F err ≜ f ( x 0 ) -f ∗ . The absolute constants and polylogarithmic factors are omitted for brevity. Our SFL algorithms show the same convergence rate as the distributed SGD.

H.3 Comparison of Communication and Computation Overheads

There have been are some papers discussing the overhead of SFL, e.g., [27, 4]. We mainly use the analysis from [27].

We start with the definitions. Let K represent the total number of clients involved, D denote the aggregate size of the data, and v indicate the size of the smashed layer. The rate of communication is given by R , while T fb signifies the duration required for a complete forward and backward propagation cycle on the entire model for a dataset of size D , applicable across various architectures. The time needed to aggregate the full model is expressed as T fedavg. The full model's size is denoted by | W | , and r reflects the proportion of the full model's size that is accessible to a client in SFL, specifically, | W C | = r | W | . The factor 2 r | W | in the communication per client arises from the necessity for clients to download and upload their model updates before and after the training process. These findings are encapsulated in Table 4. It is observed that as K escalates, the cumulative cost of training time tends to rise following the sequence: SFL-V2 being less than SFL-V1.

0

50

100

150

200

Training Round

0

20

40

60

80

100

SFL-V1

SFL-V2

FL

SL

FedProx

FedOpt

(a)

β

= 0

.

1

, N

= 10

.

0

50

100

150

200

Training Round

0

20

40

60

80

100

SFL-V1

SFL-V2

FL

SL

FedProx

FedOpt

(b)

β

= 0

.

1

, N

= 100

.

0

50

100

150

200

Training Round

0

20

40

60

80

100

E

=2

E

=5

E

=10

0

50

100

150

200

Training Round

0

20

40

60

80

100

E

=2

E

=5

E

=10

0

50

100

150

200

Training Round

0

20

40

60

80

100

E

=2

E

=5

E

=10

0

50

100

150

200

Training Round

0

20

40

60

80

100

E

=2

E

=5

E

=10

I Additional Experiments

I.1 More comparing methods

We compare the SFL methods with the benchmarks, i.e., FedProx [15] and FedOpt [19]. We have used the same hyperparameters, and trained the models on CIFAR-10. The results are provided in Figure 6. From the figures, we observe that SFL-V2 continues to be the best-performing algorithm. This is consistent with our observations in the main paper.

I.2 Impact of local iteration

We further study the impact of local epoch number E on the SFL performance. The results are reported in Fig. 7. We observe that SFL generally converges faster with a larger τ , demonstrating the benefit of SFL in practical distributed systems.

I.3 Results using loss metric

We further report the results using the loss metric. More specifically:

Impact of cut layer : The results are reported in Figs. 8 and 9.

Impact of data heterogeneity : The results are reported in Fig. 10.

Impact of partial participation : The results are reported in Fig. 11.

In general, we see similar (but opposite) trends with the observations in the main paper. That is, a higher accuracy is associated with a smaller loss. These results are again consistent with our theories.

I.4 Results on the impact of the position of cut layer.

We can observe from the results that the performance of SFL-V1 and SFL-V2 increases in L c . We look at the impact of the position of cut layer from the gradient perspective. We plot the gradient divergence in Fig. 12:

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0.0

0.2

0.4

0.6

0.8

1.0

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0.0

0.2

0.4

0.6

0.8

1.0

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

2

4

6

8

10

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

2

4

6

8

10

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

2

4

6

8

10

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

0

50

100

150

200

Training Round

0

2

4

6

8

10

Loss

Lc

=1

Lc

=2

Lc

=3

Lc

=4

We see for SFL-V1 and SFL-V2, the gradient divergence decreases as we choose a latter cut layer (a larger L c ). This means that the client drift issue is less severe and hence the performance increases. In addition, from a theoretical perspective, if we write ϵ 2 (i.e., upper bound of gradient divergence defined in Assumption 3.3) as a function of L c , we can see that the upper bounds of performance loss decrease in L c . This provides a theoretical angle that the performance of SFL-V1 and SFL-V2 increases in L c .

I.5 Results on FEMNIST dataset

We have conducted more simulations on a larger dataset FEMNIST. In particular, we consider N = 100 and train FL, SFL-V1, SFL-V2. Note that the data come from different sources and are heterogeneous across clients. The results are reported in Fig. 13.

We note that our key observation continues to hold. That is, SFL-V2 outperforms FL and SFL-V1 under-performs FL under heterogeneous data and a large number of clients.

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Loss

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Loss

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0.0

0.2

0.4

0.6

0.8

1.0

Loss

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0.0

0.2

0.4

0.6

0.8

1.0

Loss

=0.1

=0.5

=1

=

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Loss

q

=0.2

q

=0.5

q

=1

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Loss

q

=0.2

q

=0.5

q

=1

0

50

100

150

200

Training Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Loss

q

=0.2

q

=0.5

q

=1

0

50

100

150

200

Training Round

0.0

0.2

0.4

0.6

0.8

1.0

Loss

q

=0.2

q

=0.5

q

=1

0

100

200

300

400

500

Training Round

0

20

40

60

80

100

V1

V2

FL

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that we provide convergence analysis of split-federated learning (SFL) for strongly convex, general convex, and non-convex objectives on heterogeneous data in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: In Section 5, we claim that it is also important to theoretically analyze how the choice of the cut layer affects the SFL performance.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Section 3 presents the theoretical results with full set of assumptions. The proof of Proposition 3.5 is given in Appendix C.4. The complete proofs of Theorems 3.6-3.7 are given in Appendices D-E, respectively. Proofs of Theorems 3.6-3.7 are given in Appendices D-E, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 4.1 presents the setup of our experiments to reproduce the experimental results. We further provide our codes in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our codes in the supplementary material. We will provide open access to the data and code if the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We present the setup of our experiments, including the datasets and hyperparameters, in Section 4.1 and provide our codes in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This is a theory paper. Due to limit of computation resources and time, we were not able to run more experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have used one CPU and one GPU. Specifically, the CPU is Intel(R) Xeon(R) Gold 5320 CPU with 2.20GHz, and the GPU is A100-PCIE-80GB. Memory: 256GB Storage: 10TB. Each experiment curve takes about 10 hours to train.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have conformed to all aspects of code of ethics with this submission.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The goal of this work is to provide a comprehensive analysis on the convergence of SFL. Our theoretical results have two potential impacts. First, we provide a thorough understanding on the performance of SFL, which potentially guides the implementation of SFL (e.g., the choice between FL and SFL, the choice of hyper-parameters and cut layers).

Second, the convergence results can be used for modeling the training performance of SFL. Together with an effective modeling of communication and computation overheads of clients, researchers will be able to perform SFL system optimization. Due to the reduced clients' training loads in SFL, such a system optimization can potentially minimize the burden at clients (e.g., human mobile devices) while maintaining client privacy and satisfactory training performance.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks. We use open-access datasets and models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Our codes are based on the codes provided in [27], which was cited in the main paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSplit federated learning combines the advantage of split learning and federated learning. However, there is \na lack of theoretical analysis. This paper provides the first comprehensive analysis on split federated learning. The settings include strongly convex, general convex and non-convex. This paper discusses both the full participation and partial participation case. Both two versions SFL-v1 and SFL-v2 are analyzed. Moreover, some experiments are conducted to compare split federated learning with federated learning and split learning.\n\nSTRENGTHS:\nThis paper provides the first theoretical analysis of split federated learning. I think that this is an important problem. \n\nThe analysis is comprehensive. It includes strongly convex, general convex and non-convex case. It also discusses full client participation and partial client participation. Two variants of SFL are discussed, SFL v1 and SFL v2.\n\nThis paper has a long appendix, indicating that authors indeed have worked hard on this work.\n\nWEAKNESSES:\nI am impressed by the long appendix, which shows that the authors indeed work very hard on this problem. Nevertheless, I think that this paper is not ready for publication currently, due to many potential concerns.\n\n1. In paragraph 2 of introduction, this paper states that split learning leads to high latency, but this paper does not provide theoretical/experimental evidences.\n\n2. Inconsistency between theory and experiments. The theory only gives the convergence rate. It fails to explain the experiment phenomenon shown in this paper. In particular, the theoretical analysis does not reveal why SFL outperforms FL/SL under strong heterogeneity. \n\n3. Line 128: statements need to be improved. It is weird to say \"Based on x_s^{t,i}, based on which ... \"\n\n4. The assumptions need to be compared with existing works on federated learning.\n\nFor example, in Li et al. On the convergence of Fedavg on non-iid data, ICLR 2020, Assumption 3.3 is not required. Personally I also feel that the bound of heterogeneity by epsilon^2 may not be necessary. Hope that authors can explain more about why this assumption is necessary and how is the convergence rate affected if we remove this assumption.\n\n5. In eq.(10), gamma=8S/\\mu-1 need to be put later, after line 190. gamma is only used for the strongly convex case.\n\n6. In Theorem 3.5, I do not see lower bound on the learning rate. To ensure that (11) holds, I think that a lower bound of \\eta_t is needed. Merely requiring \\eta_t\\leq 1/(2Stau_max) may not be enough. Note that \\eta_t=0 also satisfies \\eta_t\\leq 1/(2S\\tau_{\\max}), then E[f(x_T)]-f^* will not converge to zero.\n\n7. I do not see any term including epsilon in strongly convex and general convex case. It is only included in the nonconvex case. Why the convergence is affected under nonconvex case but not convex case? Some intuitions are needed here.\n\n8. In Figure 4 and Figure 5, From the curves, the accuracy oscillates violently. Therefore I suggest to reduce the learning rate.\n\n9. List of inequalities after line 486. A^t is not defined. From the relationship above, one can only get\nA^{t,i}\\leq CA^{t,i-1}+B\\sum_{j=0}^{i-1} C^j.\n\n10. In line 520. First equality does not use (a+b)^2=a^2 +2ab+b^2. It just expands x_s^{t+1}. Perhaps the authors mean the second equality.\n\n11. Table 1 needs to be put in the main paper. The comparison is important.\n\n12. Line 828: Notation abuse of beta. In experiment section, it refers to the degree of heterogeneity.\nHere it refers to the fraction of full model accessible to clients.\n\nQUESTIONS:\n1.In eq.(17): It seems that partial participation significantly increases the error. In particular, $G^2/q_n$ is significantly larger than $G^2$. It is not the case for federated learning. In the ICLR paper, compare eq.(6) with eq.(4), the only difference is that B is changed to B+C. Therefore, the results seem to suggest that the Splitfed has significantly worse performance for partial client participation. Please provide some intuitive explanations.\n\n2. a_n is much smaller than 1, thus eq.(20) is significantly larger than eq.(17), indicating that SFL-V2 performs worse. This seems to be inconsistent with the experiment (which shows that SFL-V2 is better). My guess is that the current bound is not tight. Please explain.\n\n3. Are there any supports from theoretical analysis, proving that SFL indeed performs better than FL/SL under high heterogeneity?\n\nLIMITATIONS:\nI think that the authors do not summarize the limitations well. The authors have mentioned how the choice of the cut layer affects the SFL performance. However, I think that the major limitation is that the good performance under strong heterogeneity is not well supported by theoretical analysis.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSplit Federated Learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this work aims to fill this gap. The analysis of SFL can be more challenging than that of Federated Learning (FL), due to the potential dual-paced updates at the clients and the main server. This work provides a convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data.\n\nSTRENGTHS:\n1. Split federated learning is an interesting field in FL, however, existing work fail to provide some theoretical analysis for SFL, e.g., convergence rate. And this is the first work provide convergence analysis of SFL for strongly convex, general convex, and non-convex objectives on heterogeneous data.\n\n2. This work offers comprehensive discussion about the convergence rate of SFL under several settings and the theoretical contributions are excellent. Specifically, this work provides the convergence analysis for SFL with strongly-convex, convex, and non-convex objectives under the full participation and partial participation settings. Moreover, the assumptions in the theoretical analysis are mild and widely-used in the literature of FL.\n\n3. The presentation and organization of this work are good.\n\n4. Extensive experiments are conducted to evaluate the properties of SFL, e.g., the impact of cut layer, impact of data heterogeneity, and so on.\n\nWEAKNESSES:\nI believe this is a solid and interesting work, I also have some minor concerns as follows.\n\n1.\tThe section on Related Work is too brief, more discussion about the SFL needs to be included.\n\n2.\tI suggest that the author provide a table to summarize and list the convergence rate of SFL under different settings, such as convex, non-convex, full participation, and partial participation. In this way, the theoretical results can be clearer.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the convergence of split federated learning (SFL) on heterogeneous data and gives theoretical convergence rates under different assumptions on the convexity of the loss function and the participation schemes of the clients.\n\nSTRENGTHS:\nThe paper considers the problem comprehensively, including the assumptions on the convexity of the loss function (strongly convex, convex, and non-convex), the participation schemes of the clients (full, partial), and split schemes (soft-sharing: SFL-V1, hard-sharing: SFL-V2), and provides a very detailed theoretical analysis of the convergence rates under these different settings.\n\nWEAKNESSES:\nNumerical experiments are comparably weak since the baselines for comparison are FedAvg and SL. The latter is pure local training under the perspective of federated learning. The former is a very basic federated learning algorithm. Perhaps the authors could compare with more advanced federated learning algorithms, such as FedProx, FedOpt, etc.\n\nQUESTIONS:\n1. In Figure 4, it can be observed that the accuracies of the algorithms on CIFAR-10 and CIFAR-100 are very close (the latter only slightly lower). However, the accuracies on CIFAR-100 generally have an observable gap compared to CIFAR-10. Is this due to the small number of clients (only 10) in the experiments? If so, how would the results change if the number of clients is increased?\n\n2. In Figure 7, the losses of SFL-V2 (the hard-sharing scheme) almost tend to zero for the cutting layer = 1 or 2. I wonder if these curves are losses on the training set or the test set. If the losses are on the test set, the results are surprising, since even centralized training cannot achieve zero loss on the test set. Could the authors provide more insights into this phenomenon?\n\nLIMITATIONS:\nThe last section discusses the limitations and provides some future research directions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the first convergence analysis of the two variants of split federated learning under different set of hypotheses.\n\nSTRENGTHS:\nTo the best of my knowledge, as also stated by the authors, there is no theoretical analysis of split federated learning. This is a serious knowledge gap, given the potential practical interest of split FL. The paper proposes the first of such analysis under the classic set of hypotheses: L-smoothness and strong convexity, L-smoothness and convexity, non-convexity.\n\nWEAKNESSES:\nI was not able to check the proofs in their completeness, but there are some signs suggesting that the analysis may not have been carried out as carefully as required. First, it seems some hypotheses are missing in the statement of the main theoretical results. For example, convergence to the minimizer in presence of stochastic gradients is not possible unless the learning rate opportunely decreases over time, but this is not mentioned. Second, I have some concerns because 1) the verbal description of the studied algorithms is not very clear, and, more importantly, 2) even their mathematical descriptions are not consistent across the paper. Before moving to the specific comments, I want to stress that a certain vagueness in the actual operation of the algorithm is common to the original papers on split learning and split federated learning, but it is a less serious concern in their case because they do not provide theoretical guarantees.\nMy main concerns are about how models are aggregated. Now, the two aggregations presented in the main text (equation (3)) and in the algorithm formal presentations in appendix B (line 22 in algorithm 1) do not match. The first one presents an unbiasing effect, the second one not. More importantly, neither one nor the other can lead to asymptotic convergence under partial participation, because there would be a persistent noise due to the variability in the set of participating clients. By a first look at the proof, it seems the proofs do not consider neither of these aggregation rules but rather an aggregation at the level of the gradients (see page 15), which may converge (but I did not check), but there are still some potential problems for SFL-v3 (line 453) because the a_n are ignored. \n\nI have also some concerns about the comparison with FedAvg in the experimental section. My understanding is that the authors study a version of SplitFL where clients communicate after each batch. In this setting, SFL-v1 coincides with FedAvg if 1) \\tau=\\tilde{\\tau} (both the server and the clients average the models at the same time) and 2) FedAvg is allowed to transmit after each gradient computation (as it seems required for a fair comparison). Now, the results presented show different performance for FedAvg and SFL-v1. Unfortunately, there is no information reported about the specific setting (i.e., what are \\tau, \\tilde{\\tau} and how many local gradient steps FedAvg performs before communicating).\n\nThe authors have addressed my criticisms above and I have upgraded my score. It is required a significant review of the main text.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nA discussion of the limitations (e.g. the fact that the convergence results hold only when clients are allowed to communicate after each batch) is missing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theoretical convergence analysis for split federated learning (SFL) methods (SFL-V1 and SFL-V2) under full and partial client participation for strongly convex, generally convex and non-convex settings. These bounds help build fundamental understanding of the settings in which SFL, federated learning (FL) and split learning (SL) are best suited and where SFL suffers (e.g., in terms of data heterogeneity, number of clients, participation rate). The authors provide empirical results that validate their theoretical claims and showcase the performance tradeoffs between SFL, FL and SL.\n\nSTRENGTHS:\n* Well-written and well-structured.\n* Theoretical convergence analysis provided for both full and partial client participation across strongly convex, general convex and non-convex objectives.\n* Empirical validation of theoretical insights with ablations across datasets on data heterogeneity, number of clients and client participation rate. \n* Important and interesting findings as to the most applicable data settings for each algorithm.\n* Includes comparison of SL, FL and SFL in terms of accuracy, convergence speed, communication and computation costs.\n\nWEAKNESSES:\n* Empirical choice of L_c is not properly swept. Given that the highest value of L_c considered is found to yield the best performance, L_c should be increased until worse performance is observed to evaluate SFL and SL at the optimal choice of cut layer and understand at what point diminishing returns are observed from increasing L_c. Is it always better just to use FL if computational resources allow? What are the tradeoffs?\n* Lacking specifics of what FL and SL algorithms are used for comparison. Opportunity to consider other algorithms beyond the most basic implementations of FL and SL.\n* Plots are hard to read.\n\nQUESTIONS:\nWhere are there diminishing returns in increasing L_c? If a increasing L_c yields better performance, then why not just use FL? What tradeoffs should be considered when selecting L_c beyond just accuracy?\n\nLIMITATIONS:\nWhile there is no explicit limitations section in the paper, and such discussion could be expanded upon in the text, the authors do mention that the choice of how to set the cut layer (L_c) is unknown and left for future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-08-01"}