Convergence Analysis of Split Federated Learning on Heterogeneous Data
Authors: Pengchao Han
Chao Huang
Geng Tian
Ming Tang
Xin Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: split federated learning
distributed learning
convergence analysis
machine learning
Primary area: optimization
TL;DR: Convergence analysis of split federated learning

Abstract: Split federated learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL), due to the potential dual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data. The convergence rates are $O(1/T)$ and $O(1/\sqrt[3]{T})$, respectively, where $T$ denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and where some clients may be unavailable during training. Numerical experiments validate our theoretical results and show that SFL outperforms FL and split learning (SL) when data is highly heterogeneous across a large number of clients.

Convergence Analysis of Split Federated Learning on Heterogeneous Data
Pengchao Han
Chao Huang
Guangdong University of Technology, China hanpengchao@gdut.. edu.cn cn
Montclair State University, USA
huangch@montclair.ccm edu
Geng Tian
Southern University of Science and Technology. China 123324638mail sustech. edu. cn
Ming Tang
Xin Liu
Southern University of Science and Technology, China tangm3dsustech. edu. cn
University of California, Davis, USA sinliu@ucdvvis.c edu
Abstract
Split federated learning (SFL) is recent distributed approach for collaborative model training among multiple clients. In SFL, global model is typically split into two parts, where clients train one part in a parallel federated manner, and main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL.), due to the potential duual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives heterogeneous data. The convergence rates are O(1"T") and O(I/VT), respectively, where T denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and the scenario where some clients may be unavailable during training. Experimental experiments validate our theoretical results and show that SFL outperforms FL. and split learning (SL) when data is highly heterogeneous across large number of clients.
Introduction
1.1 Motivation
Federated learning (FL) allows distributed clients train global machine learning model collaboratively without sharing raw data. FL leverages the parallel computing capabilities of clients enhance model training efficiency. However, FL is usually computationally intensive. Clients need to train the entire global model multiple times, which can be infeasible for resource-constrained edge devices. This challenge is further exacerbated as the trend towards increasingly larger model architectures demands more substantial resources [ Moreover, FI. suffers from the client drift.
'Equal contribution.
*Corresponding author.
This work was partially supported by the National Natural Science Foundation of China (Grants 62202214 and 62401 161), Guangdong Basic and Applied Basic Research Foundation (Grants 2023A1515011 5515918899 and 1022A15151000 and USDA-000-67021-32855
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Layer
Layer
Fed Server
Main Server
Fed Server
Main Server
Clients
Clients
SFL-VI
SFL-V2
Figure 1: An illustration of SFL. framework, and there are major algorithms, i.e., SFL-V1 (left) and SFL-V2 (right) [27]. More discussions on SFL-V1 and SFL-V2 are given Sec. [
problem when clients' data distributions are heterogeneous, aka non-identically and independently distributed (non-IID), A large number of studies have proposed algorithms address the client drift
issue, e.g., [1 10 25
Split learning (SL) 28 is another distributed approach. By splitting the model across clients and main server, SL can substantially reduce the computational workload on edge devices. Moreover, recent studies in (34 [77 show that SL can outperform FL when data is highly heterogeneous. However, SL's sequential training among clients can lead to high latency in each training round and potential performance loss (e.g., caused by catastrophic forgetting), which impedes its practical applicability real-world distributed systems.
In light of above challenges, Thapa et. al in [27] proposed split federated learning (SFL) hybrid approach that synergizes the strengths of both FL and SL. SFL combines parallel training of FL with partial model training of SL. They proposed two major SFL algorithms: SFL-V1 and SFL-V2. An illustration of these SFL algorithms are shown in Fig. [ Specifically, the global model (to trained) is first split cut layer into two parts: client-side model and server-side model. Then, the clients are responsible for training only the client-side model under the coordination of aed server (similar to FL). Another server, known the main server, is tasked with training the server-side model by collaborating with the clients (similar to SL.. SFL aims to leverage parallel processing to reduce latency, while benefiting from the reduced computational workloads and enhanced data heterogeneity handling of SL.
Following 227, there has been emerging volume of empirical studies SFL. e.g., 12
3
23 8 31 5. However, a convergence analysis of SFL is missing the literature, and this paper aims to provide a comprehensive convergence analysis under different conditions. Convergence theory is crucial for understanding the learning performance of SFL, particularly in the context of heterogeneous data and partial participation scenarios. In practical distributed systems, clients prone to have different data distributions. Morcover, not all clients may be active or available at all times. These two issues significantly affect the learning performance of SFL. We aim provide convergence guarantees for SFL on heterogeneous data (under both full and partial participation). We further compare the results FL and SL, which provides insights into the practical deployment of various distributed approaches.
1.2 Related Work
Convergence theories of FL and SL. There many convergence results on FL. Most studies focus on data heterogeneity, e.g.. 30 16
10 12]. Some studies look at partial participation, e.g., SGD, e.g., [24 33 32 where [33]
35 29 26). There are also convergence results
argued that the key difference between FL and Mini- Batch SGD is the communication frequency.
To our best knowledge, there is only one recent study 17
discussing the convergence of SL. The major difference to SI. analysis lies in the sequential training manner across clients, while SFL. clients perform parallel training.

---

1.3 Challenges and Contributions
Challenges of SFL convergence analysis. When data is homogeneous (IID) across clients, the convergence theory in [12] (mainly developed for FL) can be applied to SFL. When data is het- erogeneous, however, the theory cannot be directly applied due to the client drift problem. The challenge is intensified with clients' partial participation, which induces bias in the training process. Despite that prior FL theories have handled data heterogeneity [16] and partial participation [29], SFL convergence analysis imposes unique challenges due to the dual-paced model aggregation and model updates at the client-side and server-side. More specifically.
Dual-paced model aggregation in SFL-VI: In SI the main server maintains one server-side model for each client, and it periodically aggregates the server-side models. When the main server aggregates its models at the same frequency as the clients, the analysis is the same to that of FL. However, FI. analysis cannot applied when aggregations occur at different frequencies, and challenging to analyze the impact of such discrepancy on SFI. convergence.
Dual-paced model updates in SFL-V2: In SFL-V2, the main server only maintains one version of server-side model. The clients update the client-side models in parallel manner while the main server updates the server-side model in sequential fashion. Hence, each client's local update depends on the randomness of the previous clients who have interacted with the main server. While [17] handled sequential client training, their theory cannot be applied to SFL-V2 as they did not consider the aggregation of client-side models. This makes our analysis more challenging than FL and SL.
Contributions. We summarize our contributions follows:
We provide the first comprehensive convergence analysis of SFL. The analysis is more challenging than prior FL analysis due to the dual-paced model aggregation and model updates. To this end, we derive a key decomposition result (Proposition (35) that enables us to analyze the convergence from the server-side and client-side separately.
Based on the decomposition result, we prove that the convergence guarantees of both SFL.- VI and SFL-V2 are O(1/T) for strongly convex objective and (1/ (() for general convex objective, where T denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and more practical scenarios where some clients may be unavailable during training.
We conduct simulations various datasets. We show that the results are consistent with our theories. We further show two surprising results: (i) SFL achieves a better performance when clients maintain larger portion of the global model; (ii) SFL-V2 outperforms FL and SL when clients have highly heterogeneous data and the number of client is large.
The rest of the paper is organized as follows. Sec. 2 formulates the SFL model. Sec. presents the convergence results for SFL. We conduct experiments in Sec. and conclude in Sec.
2 Problem Formulation
2.1 Model
We consider set of clients N 21, .N., where each client n E M has local private dataset D, of size D, = I- Suppose the global model parameterized by æ has L layers. In SFL, the global model is split the L.-th layer (i.e., the cut layer) into two segments: a client-side model a. (from the first layer to layer La) and a server-side model æ, (from layer L. + to layer L), where x = 2ii a.). Let Ba,,. denote the local client-side model of client n. The clients train models with the help of two servers: (i) fed server, which periodically aggregates clients' local models æe,n (similar to FL), and (ii) main server, who trains the server-side model x.. In this work, we consider two major SFL algorithms: SFL-VI and SFL-V2 [27). In SFL-VI, the main server maintains separate server-side model æa,n corresponding to each client n.. In comparison, in SFL-V2, the main server only maintains one model x...
Let F,, (z:; (x) denote the loss of model x over client n's mini- batch instance Gos which is randomly sampled from client n's 's dataset D,,. Let F, (x)) A RR,, [F, G.)) denote the expected loss of model over client n's dataset. The goal of SFL is to minimize the expected loss of the model
3

---

over the datasets of all clients:
min (() - Σ an Fn (2), x
where E [0,1] is the weight of client satisfying Ener on = 1. Typically, On Dn/ EweN Dw, where a client with a larger data size is assigned a larger weight [34].
2.2 Algorithm Description
We provide a brief description of SFL. Refer to Appendix B for more detailed discussion. SFL takes total number of I rounds to solve (T). At the beginning of each round clients download the recent global client-side model from the fed server, where the model is an aggregated version of the client-side models of the clients from the previous round t Each round t contains two stages:
Stage 1: model training. Clients and the main server train the full global model for iterations in each round. In each iteration i < T, there are three steps:
Step I: client forward propagation. Each client n samples mini-batch of data Gh from D,, computes the intermediate features (e.g., activation values at the cut layer) over its current model z.d and sends the activation to the main server. The clients perform forward propagation in parallel.
Step 2: main server training. Upon receiving the activation of each client n.
the main server computes the loss using the current server-side model æ It then computes the gradients over a!: update the model. It also computes the gradient over the activation at the cut layer, and sends it to client n.
SFL-VI:
SFL-V2: the main server computes the loss F > xf:")) based on which it then updates the server-side model vf. It also computes and sends the gradient over activation the cut layer to client n. Note that the main server sequentially interacts with the clients in a randomized order.
Step 3: client backward propagation. Receiving gradient at the cut layer, each client n computes the client-side gradient using the chain rule, and then updates its model x
Stage model aggregation. Model aggregation can occur for both client-side and server-side models. For the client side, after T iterations of model training (i.e., at the end of round i), each client sends its current client-side model the fed server. The fed server aggregates the clients" models (e.g., weighted averaging), which will be downloaded in the next round + 1:
1+1 E
Σ nEN
(2)
c,n-
For the server side, (i) in SFL-V1, after 7 iterations of training, the main server aggregates all server- side models. Note that i does not necessarily need to equal т. but when equality holds, SFL-V1 can be regarded as FL (despite the model splitting). (ii) In SFL-V2, no aggregation occurs since the main server only maintains one model.
2.3 Client Participation
We consider two cases: (i) full participation where all clients are available during training. This model the scenarios where clients are organizations companies who likely have sufficient computation and communication resources 77]: (ii) partial participation where some clients may be unavailable during training. This can model the cases where clients are edge devices (e.g.. mobile phones) that are usually resource-constraine and may be disconnected from the SFL process.
To model partial participation, we consider independent participation probabilities for each client, allowing for arbitrary and heterogeneous participation probabilities. Specifically, we use qn E [0, 1 to denote client n's participation level (or probability). and q (ym, E A/). If on = 1, client n participates in every round of SFL. with probability one. If (n < 1, client unavailable in some rounds. Denote pt (g) the set of participating clients in round t. In the presence of partial

---

participation, we need to modify 2
(and the potential server-side aggregation) to offset the incurred
bias:
On
In
NEP (q)
3 Convergence Analysis
We first make technical assumptions in Sec. 3.1 Then, we present key technical result in Sec. 3.2 to support the SFL convergence analysis. Finally, we provide the convergence results under full participation and partial participation in Sec. 3.3 and Sec. 3.4 respectively.
3.1 Assumptions
We start with some conventional assumptions for convergence analysis in the FL literature. Assumption 3.1. (S (S-Smoothness) Each client nt's 's loss function F, is S-smooth. That is, for all x. y E Rd
(yy & (x) + (x).))) - x) + 2 x|12.
(4)
The smoothness assumption holds for many loss functions in, for example, logistic regression. softmax classifier, and norm regularized linear regression [16].
Assumption 3.2. (Unbiased and bounded stochastic gradients with bounded variance) The stochastic gradients 9,,(-) of F.. (-) is unbiased with the variance bounded by of
Ean-DA D., ( (x, Cn) F. (æ),
(5)
E, IIgn (x,, Ga) F. (z=) S o
Assumption 3.3. (Bounded gradients) The expected squared norm of stochastic gradients is bounded by G².
Bg 19,, (z, Sn)11 G².
The value of on measures the level of stochasticity. Assumption 3.4. (Heterogeneity) There exists an e² such that the divergence between local and global gradients is bounded by e
|VF (x) Vf
(8)
larger è² indicates a larger degree of data heterogeneity.
3.2 Decomposition
As discussed in Sec. I.3 analyzing the performance bound of SFL can be more challenging than that of conventional FL counterparts due to the dual-paced model aggregation and model updates. To address this challenge, decompose the convergence analysis into the server-side and client-side updates, respectively. We give the decomposition below.
Proposition 3.5. (Convergence decomposition) Let x² [x=;; x;]] denote the optimal global model that minimizes [(.), and x A la=2' a²{ is the global model obtained after T rounds of SFL training. Under Assumption we have
S
E [f(=")] - f(x)){
(Ell - +||2+E||="" x;:11))
The proof is given in Appendix C.4 Proposition 3.5 is particularly useful. It shows that despite the challenging dual-paced updates, to bound the SFL performance gap, suffices to separately bound the gap at the server-side and client-side models. Note that our decomposition can be easily applied to other distributed approaches such as SI. addition, such a decomposition is not necessarily loose, our derived bounds for SFL achieve the same order as in FL (see Appendix H.2 for details).
5

---

3.3 Results under Full Participation
Built upon Proposition 3.3 we first present the convergence results under full participation. For convenience, define
per
y - 85/M 1, Tmin min
Tmax TT 7}.
(10)
and let of represent the learning rate at round t. Let f denotes the optimal global loss, i.c., f 2 f(x*), All results are obtained based on Assumptions 3.1 3.4 The convergence results for SFL-VI and SFL-V2 summarized in Theorems 3.6 and 3. respectively!
Theorem 3.6. SFL-VI: full participation)
j--strongly convex: Let Assumptions 3.1
3.3 hold, and nz #F(r+0] for client-side model and
n "i([+t] for server-side model,
8SN N n= (20 C2)
N 7685
m== Onp (20 G2) S(7 1)
E [[(x"))---
(11)
1 (
e (7 + (r
2(7 + T)
General convex: Let Assumptions
3.1 3.3 hold, and n/ 25 Tmaa
1
E (x")] fr - 2(T + +1)
$1 1ler
+
1
(72 +2+ N
Σ R=1
N
a (20 + +0)
2
T ( +1
(12)
24(72 + +
N
an (20 + 02)
+22
m (T + 1)
==1
Non-convex: Let Assumptions 3.1 3.2 and 3.4 hold, and n/ S min
Toin
169Tmax 8SN+2 max
ΣN 03
SNS((²+77) N
T-1 a (o + 6
T 1 (x')
(a')
Triin (1(x") -1)+
(11) (13)
T Trmin
t=0
n=1
Theorem 3.7. SFL-V2: full participation)
j--strongly convex: Let Assumptions 3.1
3.3 hold, and ry²
wF(r+e] for client-side model and
4
'/ = " for server-side model,
-({("")-s' SSNEN (a= +1) (20 +G+ 7685 N DN En=i(an (20 GG²)
5(7+ 1)++1)en
122 (T+
(T+T)
2(y+T)
(14)
General convex: Let Assumptions
3.1 3.3 hold, and 1/¹ s 25F
1/
STen 2T+1) T+1 ==1
1
Ner N
24ST Ter N
Bif
Σ
4 +)) (20 +G9
+
Σ on + +2) 2 +G3
T+1
n=1
(15)
Non-convex: Let Assumptions
3.1
3.2 and 3.4 hold, and n¹ < min { 1657
BSN=T
8NST N
T-1
E'E 1=
f (x')
(x) Tt (f -
+ 1) + 2)
Σ (n²)² (16)
T
T
==1
1=
FFllowing many cxisting works in FL (c.g.. 10
we consider E [f(x*)] f and
1. ETD V=/((" as the performance metries for (strongly) convex and non-conver objectives. respectively.

---

Proofs of Theorems 3.6 3.7 are given in Appendices D
respectively. We summarize the key
findings below.
Convergence rate. The convergence bounds both SFL-V1 and SFL-V2 achieve an order of O (T) on strongly convex (and non-convex) objectives. For general convex objectives, the convergence rate becomes / VT) Note that bounds match the existing bounds for FL and SL (in terms of the order of T) on heterogeneous data for strongly convex objectives. For a more detailed comparison, please refer Appendix H.2
Impact of data heterogeneity. The convergence bounds increase as the level of data heterogeneity increases. For example, in (13). the bound increases in e² (see Assumption 3.4 This means that SFL tends to perform worse when clients' data are more heterogeneous, which is a commonly observed phenomenon in distributed learning, e.g., FL..
Choice of learning rate. One should use a smaller learning rate when the number of local iteration increases. This bears a similar spirit to [6]. In addition, our results indicate that proper choice of constant learning rate suffices SFL convergence. would be an interesting direction to investigate whether diminishing learning rates are able to achieve faster convergence.
Comparison between SFL-V1 and SFL-V2. The convergence results between the two SFL versions are very similar, except that a (and a.)) in SFL-V1 are replaced by a (and an + 1) in SFL-V2. See (TT) and (14) for an inspection. We will show Sec. that SFL-VI and SFL-V2 achieve similar accuracy (except under highly heterogeneous data).
3.4 Results under Partial Participation
Now, we present the results under partial participation. Theorem 3.8. SFL-VI: partial participation)
p-strongly convex: Let Assumptions 3.1
3.3 hold, and ry²
4 /F((+t] for client-side model and
n pf( for server-side model,
E [f(x"))]ff SSNE " 220 G
7685 N
-- an 20 +G9
S(7 1)er
(17)
2 (Y + T)
10t (T+ +1)
2( +T)
General convex: Let Assumptions
3.7
3.3 hold, and 1/¹ S 25Tmax
STer
(f2
TouN N
Tour
G2
E [x fr
Σ a
20 + G
2(T
11
T (T + +1)
On
(18)
24( + +22)
N - (T Σ
an (20² + G²)
2
n=1
Non-convex: Let Assumptions
3.7
3.2
and 3.4 hold, and n/¹ < minf
Tmin
188Tmax 8SN-2
a2
max
(1) 8NS(72+77 N
T
Σ 7 V (x') ( (2')]l V.J
< Trmin
4
(o
+6
Σ 2m
(19)
TTmin
On
2=
mmm
Theorem 3.9. SFL-V2: partial participation)
3.3 hold, and n/¹ pF(y+0y for client-side model and u-strongly convex: Let Assumptions 3.1
n/ pp((+to for server-side model,
SSNEN (a +1)
G2
7685 M
- (an (1)+1))22 (20
G
5(y+1)rem
El (x -
(T+T)
1 (
2(7 + +T)
(20)
NNote that it might be counter-intuitive observe looser bounds on general convex objectives than non-convex objectives. This is associated with different performance metries used in analysis, e.g., see hand side of (12) and (13).
We also compared SFL to FL and SL in terms of communicatiom/compuaatooo overheads in Appendix H.3
7

---

100
Acournyy
Arrurwyy
60
Accurasy
60
40
8
40
20
8
20
Training Round
Traningg Round
Takling Round
Traiiing Round
(a) SFL-VI on CIFAR-I0. (b) SFL-V2 on CIFAR-10 10. (c) SFL-VI on CIFAR-100.(4 SFL-V2 on CIFAR-I00
Figure Impact of the choice of cut layer on SFL performance.
General convex: Let Assumptions 3.7 3.3 hold, and o/ S 1
2SF'
))
S 1
NI
G
E f (x
(a + 1)
20 +
2(T
1)
On
n=1
(21)
$
24 N
Σ
(an + n + G2)
2
T+1
Non-convex: Let Assumptions 3.1 3.2 and 3.4 hold, and of S min
16S7 SSN2t Σm
I' (x') (x')
Σ T V tm0
(2'
4 Tt
(f - -f+)+ SNST
N +1
(o + 2
T-1
Σ (77) (22) 1=
T
T
In
The proofs are given in Appendices G
Impact of partial participation. In practical cross-device settings, clients may not participate in all rounds of training, i.e., 4n < for some n. This brings an additional term G² /qu to the conver- gence bound (e.g., see (12) and (18)), meaning that partial participation worsens SFL performance. This is also observed in FL. literature (e.g., (29 and consistent with our experimental results.
Experimental Results
4.1 Setup
To simulate data heterogeneity, we adopt the widely used Dirichlet distribution 6 with controlling parameter 33. Here, a smaller 3 corresponds to a higher level of data heterogeneity across clients. We use ResNet-18 18, which contains four blocks, as the model structure and consider four types of model splitting represented by Le = {1, 2, 3, 4), where Le n means the model is split after the n-th residual block. We consider two major distributed approaches as the benchmark, i.e., FL (in particular FedAvg [ and SL (28). The learning rates for SFL-VI, SFL-V2, FL, and SL are set as 0.01. The batch- 200 rounds. Unless stated otherwise, we use N 10, B = 0.1, E = 5, where E is the number of local epochs for client-side model aggregation (i.e.. every E times of training performed over each client's dataset, their client-side models are aggregated at the fed server), and hence vanilla FL. The experiments a GPU (A 1A100-0CCEE-SGGB Our codes are provided in ltttss:/ htttp:://githuub..ooTIIIAAA com Cot.:///Ge//TAA///0BC htt.ap/sepppppptdddeeeeeeeaaaaaa
We conduct experiments on CIFAR-10 10 and CIFAR-100 100 13
size b, is 128, and we run experiments for
2 x E. We set T 7 for the fair comparison on CPU (Intel(R) Xeon(R) Gold 5320 2.20GHz) and
b,
Data
4.2 Impact of system parameters on SFL performance
Impact of cut layer. We first investigate how the choice of the cut layer Le affects the SFL performance. The results are reported in Fig. We observe that for both SFL-VI and SFL-V2,
More experiments on FEMNIST are given in Appendix L.5
8

---

100
Acourny
Arrurwy
Accuraxy
40
8 5
40
20
20
1-011
20
1003
Training Round
Traninng Round
Takinng Round
Training Round
(a) SFL-VI on CIFAR-IO. (b) S2 on CIFAR-I0 10. (c) SFL-VI on CIFAR-100.CO SFL-V2 on CIFAR R-00
Figure 3: Impact of data heterogeneity SFL performance.
1.00
100
80
80
Jccurnyy
60
Recunry
Iccurry
60
Accuracy
60
40
20
0 EL2
0-1
2-102
Q-1
1-1
0.2
100
150
200
50
100
150
200
100
150
200
100
150
200
Trairing Round
11 Traiiing Rasund
Taiting Hound
Training Round
(a) SFL-VI 1 on CIFAR-10. (b) SFL-V2 CIFAR-10. (c) SFL-VI CIFAR-100. (d) SFL-V2 on CIFAR-100.
Figure 4: Impact of client participation on SFL performance.
the performance increases in Le (i.e., clients have a larger proportion of the global model). This is associated with our empirical observation that the average client gradient variance gets smaller with Le. Intuitively, a smaller gradient variance implies a lower degree of the client drift issue, which leads to better algorithm performance Based on this observation, we use I 4 for SFL (and SI.) for the following experiments.
Impact of data heterogeneity. We study the impact of data heterogeneity on SFL performance, where we use 3 E {0.1, 0.5, 1, oo and 3 = oo means clients have IID data. The results are reported in Fig. 3 We observe that higher level of data heterogeneity (i.e., smaller () leads to slower algorithm convergence and a lower accuracy for both SFL-V1 and SFL-V2. The observation is consistent with our convergence bound, e.g., in (16), the performance bound increases in e?. Note that the negative impact of heterogeneity is commonly observed in distributed learning literature including FL 7 and SI. (21].
Impact of partial participation. We study the impact of client participation and 4. = 9 E (0.2, 0.5, 1) }, Vn. The results are reported in Fig. 4 We observe that lower level of participation leads to less stable convergence and also a smaller accuracy. This consistent with our convergence results, e.g., in (20), the bound decreases in clients' participation level gn. Partial participation is expected in practical cross-device scenarios where clients are resourced-constrained edge devices. is important to develop efficient algorithms well effective incentive mechanisms to encourage clients' participation in SFL.
4.3 Comparison among SFL, FL, and SL.
We now compare SFL to FL. and SL. We consider different combinations of data heterogeneity (0.1,0.5) and cohort sizes N (10, 50, 100). The results are reported in Fig. 5 When data is mildly heterogeneous (i.e., 3 = 0.5), SFL and FL have similar convergence rates and accuracy performance. Note that SL seems to under-perform SFL and FL. We think this is mainly due to the catastrophic forgetting issue, which has been observed in 21 2)
SFL outperforms FL and SL under highly heterogeneous data and a large client number. When data becomes more non-lID (i.c., B = 0.1), SFL-V2 tends to outperform FL and SL. The improvement becomes more significant as the cohort size gets larger. The bottleneck of FL is the client drift issue caused by data heterogeneity. The bottleneck of SL is associated with the catastrophic forgetting. SFL-V2 is a hybrid combination of FL and SL, which can lead better tradeoff between client drift and forgetting. By appropriately choosing the cut layer, SFL-V V2 outperforms
'See Appendix L.4 for more detailed discussions this point.

---

Acorray
Arrurwy
Accuraxy
8 8
40
5
20
20
Training Round
Traningg Round
Takinn Round
Training Round
(a) 3 = 0.5, N 10.
(b) 0.1, = 10.
(c) 3 = 0.1, = 50.
(d) 3 0.1, 100.
Figure 5: Performance comparison CIFAR-IO.
FL and SL. This observation also indicates that SFL-V2 can be more appealing solution than FL for practical cross-device systems, as it achieves a better performance while requiring smaller computation overheads from edge devices.
5 Conclusion
In this work, we provided the first comprehensive convergence analysis of SFL for strongly convex, general-convex, and non-convex objectives heterogeneous data. One key challenge the dual- paced model updates. We get around this issue by decomposing the performance gap of the global model into the client-side and server-side gaps. We further extend our analysis to the more practical scenario with partial client participation. Experimental experiments validate our theories and further show that SFL can outperform FL and SL under highly heterogeneous data and large client number. One limitation of our work is that our bounds for SFL achieve the same order (in terms of training rounds) as in FI., yet the experiments showed that SFL. outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL to be derived, which is important future work. For future work, one can apply our derived bounds to optimize SFL system performance, considering model accuracy, communication overhead, and computational workload of clients. It is also interesting to theoretically analyze how the choice of the cut layer affects the SFL performance.
10

---

References
[1] Ahmed M Abdelmoniem, Atal Narayan Sahu, Marco Canini, and Suhaib A Fahmy. Reft: Resource-efficient federated learning. In Proceedings of the Eighteenth European Conference on Computer Systems, pages 215-232, 2023.
[2] Yansong Gao, Minki Kim, Sharif Abuadbba, Yconjae Kim, Chandra Thapa, Kyuycon Kim, Seyit A Camtope, Hyoungshick Kim, and Surya Nepal. End-to-end evaluation of federated learning and split learning for internet of things. arXiv preprint arXin:2003.13376, 2020.
[3] Dong-Jun Han, Hasnain Irshad Bhatti, Jungmoon Lee, and Jaekyun Moon. Accelerating federated learning with split learning on locally generated losses. In ICML workshop federated learning for user privacy and data confidentiality, 2021.
[4] Dong-Jun Han, Yeon Kim, Minseok Choi, Christopher G Brinton, and Jaekyun Moon. Splitgp: Achieving both generalization and personalization in federated learning. Proc. of IEEE INFOCOM, 2023.
[5] Pengchao Han, Chao Huang, Xingyan Shi, Jianwei Huang, and Xin Liu. Incentivizing partici- pation in splitfed learning: Convergence analysis and model versioning. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS), pages 846-856, 2024.
[6] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:/909.06333 2019.
[7] Chao Huang, Shuqi Ke, and Xin Liu. Duopoly business competition in cross-silo federated learning. IEEE Transactions Network Science and Engineering, 2023.
[8] Chao Huang, Geng Tian, and Ming Tang. When minibatch sgd meets splitfed learning: Conver- gence analysis and performance evaluation. arXiv preprint arXiv:2.308. 1953, 2023.
[9] Yang Jiao, Kai Yang. Tiancheng Wu, Chengtao Jian, and Jianwei Huang. Provably convergent federated trilevel learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12928-12933 2024.
[10] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132-5143. PMLR, 2020.
[11] Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and PMLR, 2020.
Statistics, pages
[12] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pages 5381-5393. PMLR, 2020.
[13] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[14] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrasiive federated learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 107 10713-10722, 2021.
[15] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-44 4500 2020.
[16] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang. and Zhihua Zhang. On the convergence of fedavg on non-iid data. In Proc. of ICLR, 2020.
[17] Yipeng Li and Xinchen Lyu. Convergence analysis of sequential federated learning on het- erogeneous data. In shirty-seventh Conference on Neural Information Processing Systems, 2023.
11

---

[18] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication cenmumication-cffcceent efficient learning of deep networks from decentralized data, In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[19] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00233 00295, 2020.
[20] Sashank Reddi, Zachary Burr Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan, editors. Adaptive Federated Optimization, 2021.
[21] Jinglong Shen, Nan Cheng, Xiucheng Wang, Feng Lyu, Wenchao Xu, Zhi Liu, Khalid Al- dubaikhy, and Xuemin Shen. Ringstl: An adaptive split federated learning towards taming client heterogeneity. IEEE Transactions on Mobile Computing, 2023.
[22] Chamani Shiranthika, Zahra Hafezi Kafshgari, Parvaneh Saeedi, and Ivan V Bajić. Splitfed resilience to packet loss: Where to split, that is the question. In International Conference on Medical Image Computing and Computer-Assisteed Intervention, pages 367-377. Springer, 2023.
[23] Veronika Stephanie, Ibrahim Khalil, and Mohammed Atiquzzaman. Digital twin enabled asynchronous splitfed learning in e-healthcare systems. IEEE Journal on Selected Areas in Communications, 41(1 1):3600-3661. 2023.
[24] Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232. 2019.
[25] Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, and Chengqi Zhang. Federated learning on non-iid graphs via structural knowledge sharing. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 9953-9961, 2023.
[26] Ming Tang and Vincent WS Wong. Tackling system induced bias in federated learning: Stratification and convergence analysis. In Proc. of IEEE INFOCOM, pages 1-10, 2023.
[27] Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proc. of AAAI, volume 36, pages 8485-8493, 2022.
[28] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. ICLR Workshop on Al for Social Good, 2019.
[29] Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. Advances in Neural Information Processing Systems, 35:19124-19137, 2022.
[30] Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning resource constrained edge computing systems. IEEE journal on selected areas in communications, 3706): 2019.
[31] Dinah Waref and Mohammed Salem. Split federated learning for emotion detection. In 2022 4th Novel Intelligent and Leading Emerging Sciences Conference (NILES), pages 112-115. IEEE, 2022.
[32] Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. local sgd better than minibatch sgd? In International Conference on Machine Learning, pages 10334-10343. PMLR, 2020.
[33] Blake Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heteroge- neous distributed learning. Advances in Neural Information Processing Systems, 33:6281-673 2020.
[34] Wen Wu, Mushu Li, Kaige Qu, Conghao Zhou, Xuemin Shen, Weihua Zhuang, Xu Li, and Weisen Shi. Split learning over wireless networks: Parallel design and resource management. IEEE Journal on Selected Areas in Communications, 4054): 2023.
[35] Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in oon-ii federated learning. Proc. of ICLR, 2021.
12

---

A
Appendix supplemental material
We organize the entire appendix file as follows:
In Sec.
B
we provide detailed algorithmic descriptions.
In Sec.
C
we provide notations and some technical lemmas.
Sec.
C.I
we provide some notations
In Sec.
we recall SFL-VI and SFL-V2
In Sec.
C
recall the assumptions
In Sec.
C.4
we provide some useful technical lemmas together with their proofs
In Sec.
D
we prove Theorem 3.6 iee, convergence of SFL-V1 under full participation.
In Sec.
D.I
prove the strongly convex
In Sec.
D.2
we prove the general convex case
In Sec.
D.3
we prove the non-convex case
In Sec.
E
we prove Theorem 3.7 i.e. convergence of SFL-V2 under full participation.
In Sec.
E.1
we prove the strongly convex case
In Sec.
E.2
we prove the general convex case
Sec.
E.3
we prove the non-convex case
In Sec.
F
prove Theorem 3.8 i.e., convergence of SFL-V1 under partial participation.
Sec.
F.1
we prove the strongly convex case
In Sec.
F22
we prove the general convex case
Sec.
F3
we prove the non-convex case
In Sec.
G
we prove Theorem 3.9 i.e., convergence of SFL-V2 under partial participation.
Sec.
G.I
we prove the strongly convex case
In Sec.
G.2
we prove the general convex
In Sec.
G.3
prove the non-convex case
In Sec.
H
we SFL to other distributed approaches, i.e., FL, SL, and Mini-Batch SGD.
In Sec. H.2
we compare their convergence bounds
* In Sec.
HK
compare their overheads in terms of communication and computation
In Sec. I. we provide more experimental results.
13

---

B Algorithm description
For version the client-side model parameter and M-server-side model parameter aggregated every r and iterations, respectively. In iteration of round t, each client n samples mini-batch of data t from D,, computes the intermediate features h (h') (e.g., activation values at the cut layer) over its current model æ and sends h( (h:)) to the M-server. For each client n, the M-server computes the loss F, (hh(w c.n. St a based on w Let denote a gradient operator and W represents the gradient of F w.r.t. w. The M-server computes the M-server-side gradient giinn (asiis (hi)) = Va, Fr. Fh (h(aaan c.n ct.) s.m)) the gradient over the intermediate features (activations) at the cut layer pt. (x+t) to client n. Each client computes the client-side gradient gld (artd Sn based on r (w (L.) using the chain rule.
c,n sn (h)) = Vn (h(att (h:",, wii)) and sends r (wft 8, Chi))
8,8
For version 2, the client-side model is aggregated every T iterations, while the M-server trains only one version of the M-server-side model.
Algorithm 1: SFL-V1 under clients" partial participation
Input: F. T, and learning rale l/ Output: Global model xx (ar ax
Initialize {a=2 .²)):
for .,... 1)Tmax do
Determine participating client set pt N according to gn: Phase 1: model training.
each client r E Pp:
Sample mini-batch <h:;;
Send h (mfiii :l:") to the M-server;
the M-server:
Compute F. (h(eehh (l)") giin (arhi (h"). and
ri (arthi ch
Send rfi (atii to client n Pt;
wiit+ +- with - -nght (whi <hi);;
Compute gh (wli ctit);
11
1.441 +- Ten migen (m cli));
12
Phase 2: model aggregation. if 6% 0 then
14
each client n E pl:
L Send wt to the F-server;
16
the F-server:
x < -E Sa
each client n E N:
L c a;;;
20
if 8% 0 then
21
the M-server:
x, ++1 + Enert x
22
1.0 a.. <- x. Vn NN
14

---

Algorithm 2: SFL-V2 under clients' partial participation
Input: T, and learning, rate 1) Global model x²
Output:
, xx
Initialize =² (= .
for -.....,
do
Determine participating client set pt N according to gni
Phase 1: model training.
each client n E P
2 2
for
do Sample mini-batch ct
0
Send (w "h") to M-server;
the M-server:
Compute Fn (a (h"),, 2..") (af% (hi), and
r (arl'; <h");
Send Li (ari Sli to client n E P';
x. wfitt + xht 3 gi (xt' ith");
12
Compute g (miii (i)); 4.++ +- ti migen
13
14
15
the M-server:
L ++10 < =::;;
Phase 2: model aggregation. each client n E Pi: Send 2,, to the F-server;
17
18
19
F-server:
20
stt1 + EnEpi m
15

---

C Notations and technical lemmas
C.1 Notations
Recall that the objective of SFL is given by
N
min ((x) = Σ (nFa())
(23)
We define
2 and xz global model parameter on the clients and server sides, respectively.
Be,n and æs local forms of parameter on client and on the main server corresponding to client ni (in SFL-VI).
FVen (-) and VF (-): the gradients of Fn (-) over De and a,, respectively.
Ian (:) and 9s.n (-): the stochastic gradients of Fn (-) over æ. and æ,, respectively.
For convenience, we omit the notation for mini-batch training data when referring to stochastic gradients.
Further, we recall how SFL-VI and SFL-V2 update models below.
C.2 SFL-V1 and SFL-V2 model updates
Let On denote the participating probability of client and define := fg...... 2N We denote I' as binary variable, taking if client participates in model training round and otherwise. I follows Bernoulli distribution with an expectation of 9n- Denote Pp' (q) as the set of participating clients in round I.
Parameter update for SFL-VI:
Local training of client n: a &- xy 11,411 <- xx/. - -//g.i (art.f) ) x.t1 + 2...
on
Client-side global aggregation:
Full participation: a/++1 ar 11¹ EnEn an Ei Ihin (xtt:);
Partial participation: x! &- x¹ 11 Enepi(a)
an Vn
Ex geit (,
M-server-side model update:
Full participation: xt+1 f- x²i 1) EnEn On Σne 7--1
i= g (xta
Partial participation: xt+1 +- x¹ - 11 EnEP'(g)
an Ei= g (xf.1
an
Parameter update for SFL-V2:
Local training of client n: aill t- x m11t1
<- 2nd x - 1f gig (prtif) x.t1 +- 20..7
on
Client-side global aggregation:
Full participation: xf+1 x 11¹ EnEn an Ei glid (xti);
Partial participation: x. +- x² - Emepr(a) on
an El ghi (wr..))
M-server-side model update:
Full participation: az!+1 +- ar' 17¹ EnEn Et g (x.2.t)
Partial participation: att1 +- x¹ n¹ Enep:(g) gn E t.5 (xf
C.3 Assumptions
We further recall the following assumptions for clients' loss functions in the proof. Assumption C.I. For each client N:
16

---

The loss Fx (:) is S-smooth:
IV (a) V. (y)ll S S l yll Vv.
(24)
(n) S (x) + ( a) + 2 + 1 Va, 2 y E Rd.
(25)
The stochastic gradients of F. (-) are unbiased with the variance bounded by o
E g (x) V. (x).
(26)
|lx (x) - vn, (a)) s
(27)
The expected squared norm of stochastic gradients is bounded by G²: (x) s G².
E 99,
(28)
(Bounded gradient divergence) There exists constant O, such that the divergence between local and global gradients bounded by e::
IV (x) V (x)]l 2
(29)
Assumption C.2. For each client N:
The loss Fn (.) is j-strooglyy convex for some 2 0:
(v) 2 F. (x) (VF x) + #
x=12, Var y E Rd.
(30)
Here, we allow that μr O, referring this case of the general convex.
C.4 Technical Lemmas
Lemma C.3. [Lemma in []0]] The following holds for any S-smooth and p-strongly conver function h, and any x. y, in the domain of h;
(Vh())) y) 2 M(2) hly) +
11 11y =1|2 S1| x:
(31)
Proof of Proposition 3.5
(Convergence decomposition) Let x* [ denote the optimal global model that minimizes f(.). and x [ x!' is the global model obtained after T rounds of SFL training.
Proposition 3.5
Under Assumption we have
S (Ell=? - x11 + E - x²113).
E [(x²)] - f(x)){(++)))))))))))) v 2
(32)
Proof. Since Fz's 's are S-smooth, it easy to show that the global loss function f(-) is also S-smooth. Thus, we have
S
E [[(x)) - <<( =w", v )
+ - |1
x0"
1121
[||2 -x 113)
(33)
2
Since x 2 1o xx and x= - [ x;]. we have
|||0"" 113 E |- x 1
[x= x1
(34)
[[|[=]] m x=""
x: 1 = [|=2" x;
13]
+
[|
x;11.]
Substituting 34 into
333. we complete the proof.
Proposition C.4 (Decomposition in each round). Under Assumption C.1 we have
[ x [ - (x='
S < E V (x') x x²)] +
-
||
x 11
(35)
E (Vanf (x x ++1
||laott=
(36)
17

---

Proof. The proposition can be easily proved by the S-smoothness of f (-).
Lemma C.S. [Multiple iterations of local training in each round] Under Assumption C.1 if we let n V68+ and run client n's 'ss local model for T iteration continuousty in any round we have
EB
2²13]
s 112 (1²)² 2 (20 + G
(37)
1=0
Proof. Similar to Lemma 3 in 20). we have
E
n
x²'
E
1.i-1
n/
In
1.1-1
x 2²119]
$ E
=x'-nf
( Veen (I (x)(1-1) + VeFn ( af (z') + af (z')) [1]
(
$ 1+ 1
E a - x11
+ 3 + r) E |1 (g VaFn (x
|1 (v ( V. ( +3 || (VeFr (2* [²]
+3 + 7) E
1+ <1+
! :) E ( =11
+ +3 (1')
+3(1 + 7) (17)2 S2B || x111'
+ +3
" R (x*))
- + 6r (77)² 2 52
| x |2] + 67 (1 o 66 (7')²
V (2]
1+ 7
1+ 2
E
1 n
+ 67 (1') on 67 (1')
V (a')
1+
2
x² 11
+6T (1') o +6 67 (n¹)² 2 Vo (x') - gill
+B V g 1))
2
1+
E
+ 67 (n¹ (20%) + GG)
(38)
where we use Assumption C.1
(X (X S a1+ x2 + (1 + 2) y2 for some positive o, and
n¹ Vosr
Let
A E
|| - x $13)
B = 67 (1/)² (207 + G2)
2
C==1+
We have
A S CAA-11 +B
(39)
We can show that
1 S CA' +B
4 < CALL
+ +B < C
443 S CA
A
Σ 0i
j=0
18

---

Note that A' =
A
E
|
-
0. Accumulate the above for T iterations, we have
r-1
E
x²1
Σ B
Σ
C³
1=0
B
C
-1
B
Σ
(C' -
B
-1
C
C-1
C-1
C-1
i=
1=0
B
[+
2)
(40)
BB
2 2 -1)
(e--1
S 27r²
< 6r (17)² (20 + 69)
$ (1/)² 2 (20" + +
(41)
The first inequality is due to Σ -.v-l x² x"1 and the third line results from (1+ 2)² S e". Thus. we finish the proof.
Lemma C.6. [Multiple iterations of local training in each round] Under Assumption C.1 if we let and run client ni's is local model for T iteration continuously in any round i, we have
n V8S7
Σ B
2+ + 8T 8T x² ==1²|22r" 27² (8r (n')² oz + 88 (7))²
(42)
Proof.
E
/et1 x0'1 20(13]
E
11 g/ - 21]
fi- x² - r/' (/ Valn (x 10)
<
+
+V VaF..
Fn (x/) - vatn (xt) + VaFn (z') - of (x') + Var (x²) (z²))
E || 1 1 - 201²² + 8 ||, (gth V Fn (arth- )) [|²
+
+ 8- [|>l (V Fa (10 va (2) + 8TE [11'1 (V (2*) - Van (x+)) [x²)
+ 87 / (x*)) ||
<
[ ) E || - x0|]
+ 8T (n')² of - 8T 2 |
S²E
20|["] + 87 (n/)²²
+ 87 (11)² V / (1²
1+ 1+- +8r 6792 2 5²
E 1ela x' 2011 +87 (8)) (n') Vef(2
+ E
| fi- x0
+ 8T (n') of 87 ( ( (a'))
(43)
where we have applied Assumption C.I (X 2X) < (1+ X2 + (1 2) Y2 for some positive a, and n/ < V8S7
Let
Ar, : E || - |1²]
19

---

B = 87 (n')
o 8r 8 (n') Vof (x')
1==1+
We have
Ar, S CAt,-11+ +B
(44)
We show that
S
C
1=0
Note that A,
E
||
x²13
O. Accumulate the above for iterations, we have
= Σ
R ||
2-'1
1=0
1=0
1=0
- 2r
< 2T
(8T (n²))0²² 0² + 88 (11)² 2 + 8T (11) V (=]((=)))
(45)
where we use Ei x² = - and (1 -) S e". Therefore, we complete the proof.
Lemma C.7. [Multiple iterations of local gradient accumulation in each round] Under Assumption and run client n local model for t iteration continuously in any round t, we
C.I if we let n/ S 2Sr
have
Σ TE IIght - [/1] s <8r (n')²s'' S² (1Vv (2 + +o+) 0
(46)
Proof.
|1
$ E ||ght
9t (11-1 + gth 1,1-1
gh [|
$ (1+7)) E |19h gh 113
+
[+
E
g 4 g [1²]
7
(1++))
2/ 11
+ 1 +
1
E
9h 11-- - [/11"]
((1+7) (1/)² S'E
||9 1 -1
+ [1 (1+
E
119 4.5- 9/.17
S ((++)) (//)* 5'E
4- 9 + gh ||²]
1
} E | - -
$2(1+T) (v)') S²E 1
1 - ="ll" + 2 + (1/)² 2 s²B ||,8111
+ (1+ - E 11 - 9h [1]]
< 1+ 2
E |19h - o:11" +2(1+7) S²E ||9") [|
)
(47)
We define the following notation for simplicity:
Ar, = E ||9 t.
9f 11'
(48)
20

---

B 21++ (17) S || [|]
(49)
c: (1+ 7)
(50)
We have
Ar S CA
(51)
We can show that
Ass S Ee C³
1=0
Note that A, = E
|lg
g-ll""] = 0. For the second part, we have
94 - -B a²
E
2T
1=0
1=0
1=0
(n')² S |19 19
s 87 (71) SaE
|10:111
< < (1')² 82
V Pr (x*))) + o%.)
(52)
21

---

D Proof for Theorem 3.6
We organize the proof of Theorem 3.6 as follows:
Sec.
D.I
we prove the strongly convex case.
In Sec.
D.2
prove the general convex case.
In Sec.
D.3
we prove the non-convex case.
D.1
Strongly convex case for SFI-V1
D.1.1
One-round Parallel Update for M-Server-Side Model
Lemma D.I. Under Assumptions C.1 and
C.2
if n S 237 in round t, the M-server-side model
evolves as
B 1r - xx:11?
1 n/'
EE [x" x; x;1² -2n 2n [f (x') (ax))
(n7)²² (7)² N Σ
an (20 +
24S (7) (n7)
Σ an (20 + G
(53)
We prove Lemma D.I as follows.
Proof. We use whih as the M-server-side model when the M-server interacts with client n for the i-th iteration of model training at round t. Using the (sequential) gradient update rule of a041 x 1/" Em= Σ i ang'h (f axt))) }). have
E
1 1+1 -x;"l" x=1""
--1
+-1
E
x n
g
x
Σ 1=0
V
f x1 +
Va, f ((=)) ( Σ k=0
0
27
E
1
x, (f20 , [11i))
1=0
+ 2
x
V Jf (f ( x211))
Σ
V
- Eg
gf
2m0
0
2
F-1
f-1
+E
( 2
Σ
9
V n. f (f x.
1=0
1=0
V n f ( (ot 20. (mgri))
S E
x w
1=0
N +-1
#-1 Va.l
27
+ + E
Σ
ang
sm
tit
(f lid x
(54)
n=1
1=0
where the second equality is from VV (fat', a..i)) f.
(a + b)² a² + 2ab + b2 and the last inequality due to
The first part in [4 is
27
m (foli lit xo (
r xx
1=0
22

---

N +-1
I EE
[ x; 1|
+ (171² F. Σ Σ
V
I'n (fo a )1
1=
N サ-1
2 E
Σ
On (x= - x. V Fn ( (f
(55)
1=0
where we use V (f xz7)
= > an V Fn [Te. xt xz
For 13 we have
N
(1²) FN
V2, F' ( (f 2 22 1)1]²]
n=l
i=0
(7") FN
L || Fn (f 3bph)))
1=0
ghh (f
a 11
+E || (f 2eh })1]"
$ (7")
Σ a + G?)
(56)
where the first inequality applies triangle inequality. In the last inequality, we apply the bound of variance and expected squared norm for stochastic gradients in Assumption C.I
Since (x) is S-smooth and p-strongly convex, using Lemma
C.3
we have
27/²B E
On (ar x. Va, Fn (1 2. 2.tk)) 11)
==0
< 2n) Σ
ag E (Fr, (a') - Fn (x*)
"1"
S || x [1²)]
(57)
By Lemma C.3
we have
Σ
||a
12 Σ au (7)* (n7)² (20²
G?))
(58)
==
From Assumption C.I
the second part in 54 is bounded by
F-1
E
angae
Vn, n. I (fatt .o mn(it)))
==0
1.4
Vn, Fn (f 22.(1))
On
N
<
2 (7)2
(59)
Thus, by EN On 1. (54) becomes
1055 =
N
< ||10! - x=;1[²] + (17)2(77² N Σ a (o + G2)
(7)
n=1
23

---

N
- 2ny Σ
Bn (x') - fxx))
n=1
( >m=1 Og
x2'5 x;'1"² + 27
12
(20) (2
N
2
n=1
N
+ NN
Σ
an (77)²
2
on (7)2
n=1
1 T'
2 x3 [1] - -21/7B [f (z²) =f(x)]
E
2
+ (1')) 2 (7)² N Σ
2f (20 n + G9) + (1) (7/) Σ an (20 + G
(60)
We now prove the convergence error. Let A+1 ف || - x; |
We can rewrite (60) as:
A 1+1 S (1
7'FF
A' - 21 7 [f (x') - f (x*)),
2
N
N n20 + G²) + 245 (7) (7')²
+ (171)² (7) N
Σ
Σ (20 + G²)
n=1
==1
< (1 (1- n'vp")
A (nt)² (7)
4 B + (71)) (7)3
8 B2.
(61)
2
where B1 4N Em= o (20 + G9) and B 1925
N an (20 + G²).
Consider a diminishing stepsize nf = #(*+e)- i.e, n'F 3 2 7+1* where 3 = 2 7 = 85 1. It
23
is easy to show that 17' S 25F for all t. Next, we will prove that A++1 - 74241 where v =
4B 8Ba [ (7 + } We prove this by induction. First, the definition of e ensures that it holds for t= = -1. 1. Assume the conclusion holds for some t, it follows that
max
(1
n/F
(F)
(7'))
(7)
A 1--1 5
A +
B
+
B2
(62)
2
4
8
14
(n/)) (+)²
(19)3 (7)3
< 1
B1
B2
(63)
7+t)
7 +t t
4
8
++t-1
8'B,
3p-1
(x+1)² 0 +
(7 [7+
tx+
(7+ t)²
(64)
7+1-1
1BB1
8²B2
B1 -1 1
(4B1
10 (0(++1) 1) (7 + 1)4
8B2
(x+1)² ++
[7+
+
(T+
(v t)2 max
422
+
(65)
7+6-1
BBB
gB
8p 8p-1 1
182B1 +
33 B2
(7+ 1+
(T+ t)2
+
(T+
(r + 8)2
max
11p-1
(Bμ - 1) + 1) (7 + 1)4 0
(66)
+ (7 + t)2 V
(67)
(68)
7 ++++1
24

---

Hence, we have proven that A' S
V. vt. Therefore, we have 942
max
4B,
8BB (7+1)
(y + 1)
|| m
E - [| ~ v 7+1
7+
16N Σm= o (20 + G²)
1536S En= an (20 + G²)
(7 1)B
|| -x,1
12² (7+ +
n (7 (7 1)
7+
(69)
D.1.2 One-round Parallel Update for Client-Side Models
Under Assumptions C.1 and
C.2
if
r7
I 2ST in round t, Lemma D.1 gives
E
x=]l"]]
s (1 '''
IE
| x. 11
-21/TBB (α') (a*))
+ (1/)² (7) N
on (20 + GG² 24S (7) (7/)) Σ
an (20 G*)
(70)
Let A++1 A E
x 1+ 2ill"]]
We can rewrite 70 as:
A++l v 1
DTT
A + (17) (T)
B1 (n1)) (7) B2.
(71)
2
where B1 4N -N= a (20 + G²) and B, == 1928 Σn an (20 G2).
Consider diminishing stepsize 1/² [(++()
23
i.e,
n'I
where B == 2 8s 1. It is easy
2
7+t.
1
to show that n/' S ZSF for all t. For v max
1B,
+ [G++|] (7 + 1) A) }. we can prove that
4+z Vt. Therefore, we have
E || - 2:-1' 2-:1"]] = =1 7tt
max
48 1
(
8Bg
(7 + 1)E
|a2 x: =;]1?]}
7t
16N ΣN an (20
G2
1536S
EN an (20", + G2)
(r 1)E
1|06 - x:=1
+
12 (7 +
(7 1)
7+6
(72)
D.1.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in and 722 using Proposition 3.5 by setting
n I 7.77 We have
E [f(x)) - f(x")
S
< 2
[ [B|22 x2,1 + E 22
8SN EN = 0% (20 + G2)
768.22 > N 1 (r
On (20 G2)
S(y + 1)E
|| x= [=]
<
1 (x+T)
T
(7 + 1)
2(7 + T)
(73)
25

---

D.2
General convex case for SFL-V1
D.2.1
One-round Parallel Update for M-Server-Side Model
By Lemma D.1
with μ and n² S 257 we have
E || x;1l"]
E ||
==11²]
FnE [f (2')-
- -((x))
N
N a (20", + GO) + 24SF3 (7')
(17) N N
Σ
an (20 + G²)
(74)
n=1
D.2.2
One-round Parallel Update for Client-Side Models
By Lemma D.I with = 0 and rf I 257 we have
E || - x;;1²²]
1 E || 2:.1 =c10"]
2n TIE [I (x') - 8 [xx')]
N
(11") 2 N a (20 + G + 24S+ (71'
Σ
(20 G²).
(75)
D.2.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in 74 and
75 as follows
B | f1+ 1--1--r -x= 117 IE
[| | m; +IF ||a! -
E || -x; x=11²]
-21 [f (x') - (x*)]
N
+ (n7)² N Σ
an (20 + G²) + +24S73 (n/)² Σ
an (20 + GO)
x=1
==1
+E [105 x=11" [13]
- TE [f (x') - - fxx)]
N
+17)²² 2N N Σ
d (20 + G²) + 24S (n²)3 Σ an (20 + G²)
x=1
==1
E || -1²
- An/ min [ (x') (x*)]
(1r) N Σ (= (20 G²) + 24S (n1²3 3 Σ an (73 + T²) (20 G²). (76)
Then, obtain the relation between E / x || and E ||a' - x'117 which is related to E [ (æ*) f (x=)) (.))). Applying Lemma 8 in 11 and let Tmin min (7, and 17" 77
we obtain the performance bound as
E f 2 ) f (x²)
1
72
N
d (20 (20R
11000
x0 21
2
2
T T
min
$
2
7
|1250 - xr
s $11200 a
x=1
+
2
min
2
24S
Σ
an
(20 + G2)
T +
+
2(T + +1)
(77)
==1
26

---

D.3 Non-convex case for SFL-V1
D.3.1 One-round Parallel Update for M-Server-Side Model
For the server, we have
E (Va,, v. / (x') 20 5 -x'))
E V 2. / (x") xx
x/ + 1/ TV n (x') 7/ 2. (ad))
E [ V n./ / (x²) x
b+
xy + r VV L (x²)) - (Va,, (a²)) ///iVa,
a. (x9)1
N
+ +1/" nn (2')) ) - . || Vv, (z!))
V / (x'), ,E
t.t nn
im
Va S (z')
an V a, Fn (1 22t 1)
+ 2,, (x')
[a, (x'))²
==0
N サ-1
<
V f (x'),
ΣV
On Vx, Fn (f Li 2h 1) +n7
Σ
Σ
On (F (20))
n=1
7²7 V f (21)
N 1-1 Σ
1 +-1
an V Fn (z)) 1)
S 11
a (x') E
on
V x. Fr (f
+
Σ
Σ
7
n=1 1=0
n=1 1=0
7/ / (12
7'7 2
V'
V x. f (x)) (a')||²
+
E
ΣΣ
Σ
an V Fn (
ΣΣ
aa (2t)
27
1=0
n=l 1=0
- V 1 (1
N
27
n
2 V a. 5 (z')
+
E
ag
V x Fn
(( ((21)/22/1))) - Vz, Fu (zz")
27
1=0
7²7 || Vo.f (x (x')
Nn
N
27
+
Σ
V x, Fn ( (f 2
V Vz, Fr (2")
2
27
n=1
1=0
,/² | æ, 1 (x.) (x')|12
Niy'
52
Σ
N
2
E
|| - 22,1 2=[1"]]
(78)
2
2
n=1
1=0
where we apply Assumption
C.1
V u. f (xt)
=
Em= an Vz, Fn (x*), and (a,b) S 22422
2
By Lemma C.6 with 1/ <
V887. we have
E p
="11" 272 (87 ( 00 2 +8 2 Ve.. (=?) (79)
Thus, 78
becomes
E u. f xx')
x
n'²
Nnn' 52 N
V (z=))1
Σ
az 272 (87 (11))0 +87 (n))
( (z:))1²)
s
2
2
n=1
N
N
S 2
+ 8NN
(1²)
3 42
Σ
2
| ( 12
2
5233 Σ a (1 2 (0 + +2)
n=
n=l
(80)
27

---

Furthermore, we have
x ++1
2
SN (17')
N
Σ E
f.
an
2
==1
0
SN (17))
N
--1 Σ
E.E
f
2
s.m
1=1
SN (77))²
2
NV
2
IE
|19 11²)
2
1=0
SN )
2
Σ
E
|
gh,n + 96
2
SN (17))²
2
N
E
+ ||=))m111²))
2
n=1
1=0
SN (n7))
N
g +I E V Fn (z') + oo:l).
E
8.m
(81)
2
1=1
1=
where the last line uses Assumption C.1 variable
and
[[2]]]] = [2 + E - E[2]11]] for any random
By Lemma
C.7
with o7
25F. we have
ghin - -ganll S S87 $ (nn)²5² 5² (IVo,, Fn (x') 1² + oo.)
(82)
Thus,
81
becomes
E
1|00 1+
x4 [1
2
SN 2
2 7
N
87 (n7)2 52 (1Ve, (=')] +o2) + FE [1V (x)))
an
SN 2 7
N
f + 873 ()))
(I (=') + +o))
an
2
n=1
SN (17))² 2 +
N
7++ 8+3 (n')²
(/Ve,,F - Va, (x') + Va.l (2")) + +o?)
an
2
n=1
SN (1F) 2 7
an
8#3 (77)² 5²) (2/Va.) (a') 1|2 20 + +oa)
(83)
D.3.2 One-round Parallel Update for Client-Side Models
The analysis of the client-side model update is similar to the server. Thus, we have
EE (< v (x*) , x - -x-i-
n SN (77²) 33² Σ dg Va.l Va.((x)) (xt))] n=1
<
8N Σ a (n²)² (0 +
2
n=1
(84)
28

---

For n 1
1
28
S 13
|| | ++ 1--1-xx11 - [
2
2 SNN (n²)²
Σ an (T+ (1))²
(2 (2')\= +2e² + o2.)
(85)
2
D.3.3
Superposition of M-Server and Clients
Applying
80
83
85
and (84) into 36) in Proposition C.4 and define Tmin - min (T,
Tmax - max [T, 7 ), we have
[ x - 11 - f (z')
[< V x. I (z") , xx xtt1 x²']] + S 1 |10 - 2²21"]
E
E av (x*) aett =x,)]
8 E |
= 1P] 17' min [T. + +8N (11)) (max [T, 5²
2
N
Σ n=1
an ||aal (1
2
N
+ 82 82 (73 + +9)
(7") 2
(o +
n=1
SN (17))² max {T,
N
2 Qn
max F} - 8 (max {T, (1/)² 2 val (x') ||
n=1
SN (n7))²
Σ
+ 873 (n')² (e) (26² +
On
2
n=1
SN (n7)²²
2
N
7 + 873 (11)2 52) (26² + +
(86)
2
n=1
17 I'Tmin
+8N (7''
$2.
N
2 SSSNNN 2
Σed
N
a FTmxx + max ())) (²) / (0-911
Tmax
2
max
n=1
n=1
N + +8N1'52 52 (73 + 73)
(1") +2)
n=1
+ SN (11) 2 + 3 (112 52)
Σ
an
2
(2e 2 +o + $22
SN SN (11)²² (F+8773 (1)2' 5²)
Σ a (2e² 2 + on)
N 3
N
5
11 Finin +SN 2 MAX
2
2 (SN 3 SS 2
Σ
a + 8.5 N (1)) "
a
|| V (x))
2
n=1
n=1
1=
N Σ
N
+8N (n²)) 3 52
2. + 8N (71)) 3 S2732
Σ d
R=1
n=1
N
N SN (1)²²
N Σ a 1NS 74
N Σ
+SN (1)2 2 722
a
2
+
2 88S (1)
a n
2 2
n=1
n=1
n=1
==1
N
+8N (n/)) 3 2
O +
2
8N (1n)3 S2:3 2
Σd
a
n=1
N
N 2 Σ
N Σ a ANS 4N5 74
N
+SN (1)
7
2
SN (7))
a +8NS 717
n
2
n=1
1=1
n=1
n=1
29

---

<
T'Tmin
1
21 p
Tmax
Σ an (1+88 + 85² (18)² F
N
||Va (2)))
2
Tmin
n=1
N
1
5233 NS SN ( 4N5 (11)² 7d
+
o n
2
==1
+
(N + +NNSL3 (1')²>*)
2
* (1)) 8733 + 4NS (1/)**1 71
+
O 2
N
+ (NS) 72 + 8N (17) 3 5273 + &NSLL (11)*
n=1
$ 2
TTTmiin
1 11
2NN
max
N
n
2
1+ 1
2 32 )
Vef (x') |
Twin
n=1
NS (77)²² 72
1
1
1
N
2 0 + 2SN (7/)² ,2
1
1
Σ 2
+
2
2
64
2
64
n=1
N
N
+NS 72
1
1
0 2 + 2SN (17)) 72
[
Σ
+ 64
2
64
T/Tmiin
N
N
1
SNS p
'max
V (x') ( 27N5 (7² +72)
Σ
2
(o +
+2)
2
Tmin
n=1
n=1
'7TTmin
N
Voel (z')) + 2NS (n)
Σ n=1
da (o +c)),
(87)
4
where we first let 1/² S 1687mux and then let n/ <
We also use [Va (xt))? =
8SN max
2m az
miin
VV f (x*))| + 2 (²
Rearranging the above we have
N 7/' a/ (0')) < (1 -E [1 (a) + +NS (
Σ
no + +
Tinin
Tmin
(88)
Taking expectation and averaging over all t, we have
8NS z N
T
T Σ // [IVL [1)))()))1
4
(S (aro) - S*) +
Tmin
an
n
2
2
(1))² 2
TTmin
T
0=
t= 0
(89)
30

---

E
Proof of Theorem 3.7
In Sec.
E.1
we prove the strongly convex case.
In Sec.
E.2
we prove the general convex case.
In Sec.
E.3
we prove the non-convex case.
E.1
Strongly convex case for SFL-V2
E.1.1
One-round Sequential Update for M-Server-Side Model
Lemma E.1. Under Assumptions C.1 and C.2 evolves as
if I
1 25r'
in round t, the M-server-side model
|| 10 - 2;,1²]]
1
Nu/TT
E | m;1l²]
2n' TE [f (x²) fxx')]
2
(n') 7 N Σ
n20 + +²) 248- (17)² Σ (20 G²)
(90)
We prove Lemma E.1 as follows.
Proof. We use s.n as the M-server-side model when the M-server interacts with client n for the i-th iteration of model training round Using the (sequential) gradient update rule of at+1 x " E ElD g ((x a.in)) we have
1+1 x;; ||
E
E
x 1
g'
n
+
Σ Va, (1a2 x 1=0
a,
1=0
1=0
E
V u. I
(f ti xt11))
1=0
+ 2
V Va, (1 20 x0 21 Σ
Vnn (1a0t
Σ g
i=U
1=
i=0
+E
2 2
Σ
Va, (1 ar."
i=0
1=0
< E
x x
a. f (f x
1=0
N ΣΣ n=1
+ +(11)²B E
Va. 1=0
(f a..1)))
91)
Jan
where the first equality is from (a b)² aV.. ((=t. a.. 6. 0.
a² + 2ab 6² and the last inequality is due to
The first part in 91 is
27 V v (foth x (111)
T 8
1=0
31

---

| - 29
+ (171) T. Σ B V F'n (f 2 })1
N
2n E
[2_ - xy Va, Fn {
(92)
1=0
where we use Vn, ((aie) Fz3))
= Emm Va, Fn (1 tt,)).
For 92 we have
(11)² -N
E
V 2, Fn (1 mgnnn)) 3)1
n=1 i=0
r-1
(7') N
Σ
E
V Fn (f mgih))
gen (fa a 111
|| (frrin 2 111
N $ (T PN Σ (o
(93)
where the first inequality applies triangle inequality. In the last inequality, we apply the bound of variance and expected squared norm for stochastic gradients in Assumption C.I
Since Fu, (az) is S-smooth and p-strongly convex, using Lemma
C.3 we have
N
2
x
x Va, Fn ({ (Zrt x }))
1=0
< 2n
E [(Fn (x²) - Fn (x="
n=1 1=0
+ |
|
x1
s 1 x 1²)]
(94)
By Lemma C,5
we have
||
< 12
Σ 73 (7'') (20 G?).
(95)
R=1
E=0
From Assumption C.1
the second part in 91 is bounded by
N --1
E
n (( (H
n=1
i=U
7
E
gsh
V Pn (f a(.i)))
1=0
N
<
N
(96)
n2'
n=
Thus, 91)
becomes
++1
N + (n''² 2 Σ
|| 10
x; [|
(o + G2)
n=1
32

---

-21/TBB [f (z')= - f (x*)]
UNNTT'
N
| x + 201
12 _
ST) (1/)² (20 + G2)
N
N
(n')² 2 -22
1/ NTN
E [ x3]l"] -2n2 2n [s (x²) - -f f(x"))
<
1
2
N
220 + G2) + 245= (11)² 3 N
+ (11)) ?2N N Σ
Σ (20 + G2)
(97)
10
n=
Using the above lemma, we can prove the convergence enror. Let A E E || x; =;]""] We
can rewrite (77) as:
17
N'NTH
A 1+1
S
1
A' 27-2''' [f (x¹) - ff (xx)),
2
NV
+ (77)² N2N Σ
(20
+ ( (71)² Σ
(20 + G*),
n=1
n=1
1/ NTW
A' (n7)) 7
(17))² 73
1
B1 +
B2.
(98)
4
8
where B1 = 4N Em= (20 + G²) and B = 1925 Σm (20² + G
Consider a diminishing stepsize v/² = Nej i.e, Ny'r 2 7t where 3 2 7/ is easy show that '/² S 2ST for all t. Next, we will prove that A++1 < 7++8+I
85
N2
where
{ 4 + 8B, = max { 22 tI) (7+ + 1) We prove this by induction. First, the definition of ensures that holds for t 1. Assume the conclusion holds for some t, it follows that
A 1+1 < (1-
NoTTTA
2
72
(17)3
73
A
+
B
+
B2
(99)
2
4
43
1
(n1))
72
(17)3
3
B1
B2
(100)
+t
1/3 +t
4
8
7s 7t 1
BBB
B3
(7s 1122 U
(7, 1)2
(7s +13
(7/ + 1)22
(101)
7s 7611-1
B²B
[33
By
4B,
8B2
(7 + t)2
1 4+
1%/ + t)²
(7/ + +)3
(Yo + t)²
max
112
+
(7 1)
(7s + 1)44
(102)
7
BB
B
Bp-1 1
B³B₂
(7,41)2
(1s+t)² +
(1++
(7/ 1)2
max
13p-1-
+ (B 1)(7)
+ (7++1)4 0
(103)
$ 7s +1-1
(7+ + t)² V v
(104)
(105)
7/ + t + 1
33

---

Hence, we have proven that A'
vt. Therefore, we have
E ||10 - xxil"]] = =
max
11
8Bs
(% + 1)E
|| x; 1
To +
Ta +
16N Σn= (20 + G²)
15365 Em=1 (20 + G² n (Ys t) (7s + 1)
(7/ + +1B
|| - xx; 11
(106)
12 (7s +t)
No +
E.1.2
One-round Parallel Update for Client-Side Models
Under Assumptions
C.1 and
C.2
< 25F in round t, Lemma D.1 gives
E
x='l²]
T/
1 1
EE
x x 2;17
-21/ -2n/TB If (x') f (x')]
N
+ (17')² 72 N
an (20 + G²2 + 24ST (17))
3 Σ
an (20 + G²)
(107)
n=1
n=1
Let A++1 A E
1ol 1+1 2=;1²]]
We can rewrite (17 as:
1+1 < (1 n
(n71)² T2
(n')) 73
A
B1
B2-
(108)
2
4
8
where B1 := 4N N d (20" + G2) and B2 = 1925 En an (20 + G²).
Consider diminishing stepsize n/' +()+t))+ i.e, n'r 9
2/
tot where = / ?
88 1. It
easy to show that r/ < 2SF for all t. For v = max that A < Nott
4B + 1lo (% + 1) }. we can prove
8B,
VL. Therefore, we have
max
4B, n2
+
( 1): (%c + 1)E | - x; "
8Bg
E a 22:131 = A
No +t
To +t
(7o + 1)E 1| x; x;:l"
16N DN an (20 + G²) 1²2 (Te +))
1536S Σn (20 + G2)
pra (% + t) (7c + 1)
2e +
(109)
E.1.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in (106) and (109 using Proposition 3.5 For n" 28T and y 8s 1, we have
E [f(x²) f(x")
S
2 [B| - =1;1² + + -2
S IIE -000 -x="= |1
8SN Am= (a +1) 20 +G9
768. n= lan (20 G2)
+ +
2 (y + T)
13 (y + T) 11)
(110)
34

---

E.2
General convex case for SFL-V2
E.2.1
One-round Sequential Update for M-Server-Side Model
By Lemma E.1
with μ and nt 25T
have
E 8
++1 w"1
- E || | xz1 [1
2n TE [f f (x')]
(77) 7 N
Σ (20 + G²) + 24S+ (1') Σ
(20 G²)
(111)
n=1
E.2.2
One-round Parallel Update for Client-Side Models
By Lemma D.1 with
0 and r/ S 25F we have
E ||ot x=:1²]
E ||-6 =='1²]]
21 [I (xx') [(x)))
N
(7 2 a 220 + G²) 24S+ (7/))
Σ
Σ an (20 n + G²)
(112)
E.2.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in and (112) as follows
E
x +1
x
|1/ < E x
++1 <;ll? +E - ++1 x:
<
E
10' x3'1" 2 - 21 E [s (zt) - (x'))
+
E
| x [|]]
2ny rE [f (x') (x*)]
( N Σ
(a 1) ( + 69) + 245+ (n')
Σ
(an + +1) G?)
n=1
n=1
||as xx" [1]] 4n'7Rh [ (x') f (x')
E
+ +n ? ?
Σ
(a + (20 + G + 245+3 (7') Σ (an + 1) (20 : + G9).
(113)
Then, we can obtain the relation between E x
x
2
and E || ||-- 101 which is related to
E [ f (x*)] }] Applying Lemma 8 in
we obtain the performance bound as
E [ (x (x*)
1
| x 1
N Σ (a + 1) (20 + 69)
T +
$
N
(1 xx*
S a x.
+ 2
24S' Σ
(an + 1) (20 + (2)
(114)
T
2(T 1)
n=1
35

---

E.3 Non-convex case for SFL-V2
E.3.1 One-round Sequential Update for M-Server-Side Model
For the server, we have
E (V a. I (x')
++
x
8
< E V n. S (x')
x +1
x²= + 1) V 2. (a') - VV, a. s
< E V n. (x')
x=+11
=x' + n'' V 1 I (a')) (Va.J 1 (a')) /1/ n'TVVn, a. (x'))]
N
gomn + t,/TV0 I (a¹) ) ~ Vv.. / (x') (z²)|| ||
V I (z'). E
7)
N
V J (x))
V m. Fn (( ( 3)
+n/tVa )
7 / (z:3)1
< V 1 (a .B
n, Fa (1 an + 1/
Σ
Σ
V F' (at)
n=1 1=0
==1 i=0
T V f (2))
1ny
V f (z') EE
V x, Fr )) +
1
Σ
V a Fr (x')
==
n=1 ==
7/7 [ (12
< 2
1'
V f (z')
1| +
n E
Σ ===
N
N V a, Fn (( - ΣΣ
-
v Fy (z-)
2T
12:1 1 2m0
7/' V (x') (x
n'T
7 E
N T-1 Σ
<
Ve.f (x*) (x')||
+
(v Fn
(f=t)n ( Va. Fa (z'))
2
2T
n=1 1m0
n'T
Nn
T-
S
/ (xx) (x²)||
+
Σ
Σ
Vv, Fn (f ti 2 22 - V Fn (x*))
2
2r
n=1
1/' Ne.f (x*) )|(
5²
N
s.n - x [²]
<
+
(115)
2
2
12
im0
where we apply Assumption C.1
V x, I (x²)
= D. Va, Fa (xx'), and (a. S 2²_6²
2
By Lemma C.6 with 17 < V8Sr*
we have
8 x 2111"] s 2T (8T (n²)2 0z + +T (n')²² + 87 (x)))))) (116)
E
Thus, 115 becomes
E
[ V f. f (z") -x,)1_x()]]
|a,s S (x²¹)² 1/2 52 N
< 1/ T
> 2 (87(1)) 28 2 (1) 2 2 | (x231)
2
2
n=1
n'T
<
$1- 2
+ 8N2 (19)
|| f (2 + + S Σ (o
(o + +
(117)
36

---

Furthermore, we have
16
++1
"
SN (17')
Σ
2
SN (17))
Σ
9
2
SN (17')²
2
N
E
2
n=1
SN (n7))²²
N
ga.n + gsmn 1?
2
n=1 1=
SN (17))²
E
+IE IE (1)
2
m=1 i=0
SN (n7)²²
T-1
2, gemml 11]] + E
V Fn (z') o? oo.)
E
(118)
2
where the last line uses Assumption C.I variable
and [|z]]]] = || = - E[z]]]]] for any random
By Lemma
C.7 with 17 S 25r. we have
1 g/mm119 <872 (n)) 5² (IV Fn (x') 12 +
E
(119)
Thus,
118 becomes
||
2
SN (n')²² 2 T Σ
N
87 (1')2 52 (IV Fn (x')] + +oū) + TE V/ Fn
a=l
SN (
Σ
( 87 (7')² 5i) (1 (2+)]| oā)
SN (n7')² 2
T
8T (1')²s² Fa V (x.) + Va, (x')) + o
-
n=1
SN (n7))* T
N
++8 8 s²) (2)V 26 + + -z)
(120)
E.3.2 One-round Parallel Update for Client-Side Models
The analysis of the client-side model update the same as the client's model update in version Thus, we have
EE
[< V f (x').
x 1+1
- xx(i-
n/T
<
SN (
Va.l (
82 Σ a (n') (o +
2
n=1
n=1
(121)
37

---

For Forn' 1 28
1
2 b
s
|| x
++1 x''l?
2 SN
(77)²
7
N
a (T+ (1)
()) + +22² + ooh)
(122)
2
E.3.3 Superposition of M-Server and Clients
Applying (17). (20).
122 and 121 into (36 in Proposition C.4 we have
E [f ( []] f (z*)
+ S E
+ S B 100 1o:t1 =rll
J (w)) _- x
I
f w!!l +B[[V=. an (x)) e+1 s57
2
n²T + 8N2 (11)² |//n/ (x')) ||
2
N + S² 73
Σ (77) (a + 10 +2))
n=1
SN (77))² r
8T' {)))(²) 2 V/ (2')
+
2
n=1
SN (1')²
Σ
[a 11 T 8++8 8T' 873s ()))(²)) (2 + oz))
2
1/²T 2
+ &N8 ( SN + Σ
N
T + 873 (t) |Vas (a)))²
N
+ 8 22
Σ(7)) (a + 10
n=1
1 SN (11²)² 2 (T+
+1) +
+ 2
< ((
2 7
n'r
SSN
52 ST (17) (2'))
+8N (7/)² 52 Σ (a2 + +8N (1") 52 Σ (a +1)
N
+SN (1/)² ?62
--6
a 1) + 2 SN (1 2 2 Σ
(a + 1)02 2
n
n=1
n=1
N
N
+8NS8 (7')² 7422
(a 1) + 4 (17)* 74 >
(a + 1)0²
n=1
n=1
2 S 2 1122 (1+85n'r 7 + 852 (11)² ()))
--1't/ 1/'
|/v f (zz) ||
N
+ ( NS (1')² 2 ,2 + SN (n')²52,3 52,3 + ANS (1/)*² 71
Σ
(a + 1)0
N
+ +(NS (1/)² ,2 + 8N 8N (1/)² 52 + (1')** 71)
Σ
(ah + 1)e2
S n'T 1-2N2Sh Sin 7² イ
1 ) Voll (x*))))
2
38

---

+NS 72
1 + 2 64
1
Σ
(a
- 2SN - (7¹) 72
1
1
Σ
N
(a + 1)e²
2
64
n=
R=1
N
1 1 -1-4N2sh) Siy =)
$ 2
Val
(1')² Σ
(F
2 + (o + +
n=1
n'T Vaf (z") (2 (n')²
2
(an + 1) +,)
(123)
n=1
where we first let 1 S 165 and then let r/²
S
1
We have applied En a N.
8SN2j.
Rearranging the above we have
N
r V (2 - (f - [xf(")))) (N)
o +
(o + (124)
7
Taking expectation and averaging over all t, we have
4 s Tt (f - -
8NST Σ
(a
1) (o + E2) T-1
(n¹)² 2
(125)
39

---

F
Proof of Theorem 3.8
In Sec.
F.1
we prove the strongly convex case.
In Sec.
F2
we prove the general convex case.
In Sec.
we prove the non-convex case.
F.1 Strongly convex case for SFL.-V1
F.1.1 One-round Parallel Update for M-Server-Side Model
We first bound the M-server-side model update in one round for full participation (g.. 1 for all n), and then compute the difference between full participation and partial participation (g,. < for some n.). We denote I, as a binary variable, taking if client n participates in model training in round I, and otherwise. Practically, I, follows a Bernoulli distribution with an expectation of 9n.
For full participation, Lemma D. gives
E
x ++
T' I8 x 2;1 21/FFB [f (अ²) (xx))
s
(n/) (7)
N
an (20 + GZ) 24S (7) (n') Σ an (20 G2).
(126)
Considering that each client participates in model training with probability In. we have
E
t+1
xx. +11] [13]
E3
x ++1 x¹ + x x²¹ + 11]
E
[ ++ x=/11]]
an In
<
E
1.4 (f
Qn
|1 (fat
2. 111
< N
N (F) (1))
(127)
4n
where we use E -X
S 2
E
In.
and
x
x²
n EnEP!()) E qn gha
(f enn (..))
aid }).
Combining the above gives
x
x
[
=
||
x(+1 x141 -
T'
E 1/ 11:1
2
N
N a (20 - -0) + (7)3 (n/3 Σ
+ +1')) (7)² N Σ
au (20 + G2)
n=1
n=1
N
+N(7F² (nt) 62 Σ
(128)
4n
7=1
40

---

Let A+1 & E ||o 2.'1"]
We can rewrite (28) as:
A 2 (1- - F'
(17 (77))
(F)
B1 + (17)) (7)
A
B2.
(129)
4
where Bi == 4N Σn= / n G2) + +4NG2 Σ
ai 8m
and B = 1928 En On (20 + G²).
Consider diminishing stepsize n +d) i., /-
29
te where 3 = | 7o 88 - 1. It is
3
easy to show that 17" S 28F
for all L. We can prove that ' x
m VI. Therefore, we have
E || foo5==;ll²
m;11²]
= =
V
max
4B
+1)
8B,
(7s + 1)E
1 x=;l²)
7s
7s +
16N Σn= ag (20 + G2) + 16N
Σ
1536S Σn On (20 G2
qn
n.2 (7 + t)
1 (76 + t)
(76 + 1)
(7s + 1)
x x |1
(130)
7s +
F1122 One-round Parallel Update for Client-Side Models
>'a ag which represents the aggregating weights in round t for full par- ticipation. Using a similar derivation as the M-server side, we first bound the client-side - -=;l'==;'"" and then bound the dif- ference of client-side model parameters between full participation and partial participation || - a: " The overall gradient update rule of clients in each training round is x.
Define x
N
model update in one round for full participation E
2. T/ (n) E.=0
an gn.
Under Assumptions C.1
and C.2
if r7 I
1 in round t, Lemma D.1 gives
28T
E
==]1²]]
s 1 n'
IE ||0 xt -22 21'TTE (x') - ff(x())
N
+ (7/) (7) N Σ a (20 + G²) 24S (7) (77) Σ an (20 + G²)
(131)
R=1
Considering that each client participates in model training with probability ga. we have
E
x
x +111
E
|
x²= + t x 112
< E
||a
2t:113]
< E
Σ
N
"1
an
r-1 Σ
gei
(f 20..5
c,n
s,n })
N
< NT
(77
2
1 (f
n=1
1=
2
<T())²
(77)
(132)
qn
7m
41

---

where we use E n EnEp'(g) E
I E
E ET
In>
and
x
x
en (f s.
9n
We obtain the client-side model parameter update in one round for partial participation by combining the two terms and we have
2 ++1 x=11"²
||
e+
xet1
+ x²²t =x=1²²
=
T²
1
E
|| x=]1"]
2
+ (n')² (r) N Σ
2
(20 - + 245 (1')) Σ
an (20 + G²)
n=1
n=1
+ (n') 2
(133)
4n
where we consider (x²) - / (x:") 2 O.
Let A+ A E
||=2|| x=:l²]]
We can rewrite 163 as:
A+1 1 (1- TT'
A (7') (T)
(n') (T)
Bi
B2.
(134)
where B1 =
4N DN' a (20 + +2)
+ 4NG2 Σx a and B2
1925 Ea-1 On (20 + G.).
29 Consider diminishing stepsize 1/' +()++))
i.e,
n¹r
%ot where 8 = # 7c
9
88
1. It is
easy to show that r/' S 2SF for all t. For = max that A < tott VL. Therefore, we have
4B
+ tT. (%6 + ] we can prove
8B,
1
E | 2::1"] -A' V max { 4BB 1/2
{
+
(1 (76 + +1)E 1120 x x;13}}
SBg
Ve +
Ve +
16N Σm an (202 + G2) + 16NG2 M
15365 Σn On (20 G2)
122 (7 +1)
n² (To + 1)
(7e + 1)
(7e + 1)E 11000 xt'l??
(135)
Ye +
F.1.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in 130 and (135) using Proposition 3.5 For
n < 28 7.FF and y = 8S 1, we have
E [f(x') } f(x²"
S
< 2
[8| 20 x3112 + E xx
8SNEN m= 2 (202 + G 6²)
76852 >N On 220.+
S(y + 1)E
1/000 x=1
4n
112 (7 TT)
13 +
2(7 T'
(136)
42

---

F22
General convex case for SFL-V1
F.2.1
One-round Parallel Update for M-Server-Side Model
By Lemma D.1
with μ and rf S 257 have
E ||o||| w:'l²
- E || x 2=:1²]
2nj F (x') - fxx))
(17) N a (20 + G 2457 (1/) Σ an (20 G²).
(137)
Considering that each client n participates in model training with a probability gn. we have
N
E | ++1
x
SNP (7-)
(138)
In
Thus, we have
E
| 1-1
w;]l²]
E
-2n [f - fxx))
N a (20" + G2) 24SF3 (17) Σ 72
N
N
+
(1²)
N
an (20 + G²) + NF (1/)² 6
On
1=1
==
(139)
F.2.2
One-round Parallel Update for Client-Side Models
By Lemma D.1 with f 0 and n/ S 25F. we have
E || - ==;l?²]
I E || x.'1?
TT [f (x') - f((x))
N
N Σ
(n1 2N N Σ
a
(20 + G + 24ST (n''
an (20 + G²).
(140)
n=1
n=1
Considering that each client n participates in model training with a probability Jns we have
N
E x
x "
N
(141)
Jn
7=
Thus, we have
E | x='l?]
E
|| 20:133]
-2n' 2n'r (x') f (x*)
N
N
(7')
N Σ a (20 + G2) 24S+3 (1/)3 Σ au (20 + G²2 + N ++2)+N0
2 (1')
In
(142)
F.2.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in
139 and 142 as follows
43

---

-x01" x"1 1011 = + x +B || x -=:1==111 2311
S
||a0 2r {1]]
FB [ (x') (xx")
N Σ
(17)) 2 Σ
dn 2 n + G + 24SF (77)² Σ
an (20 + G²) -
+ N7 (7')² 6
On
==1
n=1
n=1
+E
2 xz 1
- 2n' TE f (x') - f [x'))
N
N Σ
+ +17)² P2N Σ
dg 2200 n + G²) + 245 (1/)² Σ an (20 - (7') 6
On
==1
==1
n=1
E || - x=1 [|]
4n min {T, [f (z') - [x")]
a
+11) N Σ 22 27) Σ an (0 (1)
G2 Σ
n= 9n
(143)
Then, we can obtain the relation between xx" ||] and E B|1=* x= |1] which is related
Eff (a") f [=))]. Applying Lemma 8 and let Tmin == min {o, J), we obtain the performance bound as
E [ (x"" ) (xx)
x 112 i
72 72
G2
N
20 + G2
2
min
On
T ++1
22
N
(I n
x=" 112
S a: 0 x.
+
2
2 min
245
an
220 n
+ G2)
T+1
+
2(T
(144)
n=1
44

---

F3 Non-convex case for SFL-V1
E.3.1 One-round Parallel Update for M-Server-Side Model
For the server, we have
E [(V / (x') 2.
S E
( V 2. f (x*)
x + 1/ VV a (x') - V/ (x+))
E ( V 2. f (x")
xx/1 x + + FV s. f (x')) - (Va,f (xx) v/ u. f (z'))]
+n a (01) ) 7 |V af (2
V a, f (z) E
On n
On
N 0=
a I
+ md ) - -1/5 VV (x-)](
V J (2). E
Vv, Fr (u f0 | 22thn)
9n
V S (x)), R
N
a
Va, Fa
(f (artiie 2 )) +ry"
I
V Fa (20)
On
On
n=1 1=
n=1
7/7 V f (12)
5n'F
V (x) E
N
On m
Vx, x,
Fn
(f c. 2 })
N
ag I
V x F' (x")
On
On
n=1 1=
m=1 ==0
7/7 / [12
1/7
f (x9) 112
N
an 12
It
V Vo, Fa (f
(2 s.n })
N f-1 Σ
anI 12
V の,
Fn (x))
2
On
In
1=0
n=1 x=0
7/7 of (2¹)
7-1
$
V v (x²) 112
n'
E
I I
x, Fy (f (f02tt 21t11) Vz. Fr (x²))
2
27
On
1²7 || x,, (x')1/²
No
N
E
Σ
V n Fr. ( (a 12 -
Pr (n()))))
2
27
In
7 2 Vz.f (z²)
S² S2
--1
E
d - =111²1]
(145)
4n
where we apply Assumption C.1 In.
a. f (x') =
En an Vn, (x*), (a, S =²40²
and E [[] =
2
By Lemma C.6
with 17'=
v8S7 we have
EE ti 2 =/] S 27 (87 (1')² 2 of + 87 (n')² 87 V (146)
1=0
Thus, 145 becomes
E (V x. f (z')= xrf+1 1--1-x() x.:o]
v (z' 122 S2
52
a 272
(87 (1²)² +8 (1/)² 2+87 (n¹)² |o.lf (2)
S 2
2
On
n=1
< 2
7
+ 2
N
Ve. (2t))1 + S2
Σ
N
c (17)² (0z + +
9n
In
==
1=
(147)
45

---

Furthermore, we have
x-'l"]
(n7²) SN N
Oz I
Σ E
gain
2
Gn
SN (n²)
10
9 a,m
f.
2
On
SN (n)) 7
a
E
9
2
2
9n
k,m
n=1
SN (n7))
N
9
g..n + gemnl 11²
2
n=1
SN (n1')² 7
2
+E |1s6.. 11²)
E
99.n
2
On
SN (n²) 7
+ E V Fn (z') + oo:l)
E
(148)
4n
where the last line uses Assumption C.1 variable
and [||]|?] = [| 12 BBIlz E[2]][]] for any random
By Lemma
C.7
with 17'
25F have
Σ
ghin - -9ttnll gimll"" S $18"(())² 5² (I
Fn (n(") (x') 2 o oz)).
(149)
Thus,
148
becomes
E
1+1
=²11]]
SN (n²)²²
N
a
(8+3 (n')² 5² (1Ve, (x)))) FB | Fn (2))) +o:l)
2
In
n=1
SNN (n²)²²
2
7
Og
2
7 + 873 (17) 5e) (1V=, Fo (12 +o?))
2
Qn
SNN (n²))
2
+ 873 (n²)* s²) (1 Fn (x') Va,l (x,( Va, (x0)) + +o?
2
On
SN
(7')²r
2
N
7 873 (1/) 8²) (2 (00))
(22 + 20² + +o?))
(150)
2
On
F33.2 One-round Parallel Update for Client-Side Models
The analysis of the client-side model update is similar to the server. Thus, we have
E V f (a') 2-1 =x(i|
n/T + 8N (n7²) 522 Σ n=
N
az
N 8N
V (z+)]1²
Σ (n²)² (0 + +2).
n
2
9n
Jn
n=1
(151)
46

---

For Fo 1 28
1
5
||| |ott - =021²]] 19
- E
2
SN (n=)² +
N
Σ (1+) s*) (2]lV a. (x+) 12 +22 + +oo) n=1
(152)
2
On
F.3.3 Superposition of M-Server and Clients
Applying 147 (550). 152) and [51 into 36 in Proposition C.4 and define Tmin - min {T, 7].
Tiax _ max {T, 7 we have
E f (x []] f (a')
E (V m.f (x))xz²)) ++1 - 2 + S 1 =2211 +B(Va.. af (x)) -.ott x22++ +
S EE Ille5*1 - x''l?
2
2
n' min [T,
+ 88 (11))* 3 (max (7, (7: S²
V (2((('))1 ||
2
qn
N
+ 5 (7a + +3)
(1")
(nn +2)
In
n=1
SN (77))
2 max {T, 7}
N
an
max {T, + 8 (max (T, (r')² / (x'))²
2
Gn
n=
SN 6792
T
N
2
(T + 87 (1)2 (2²²
SN (77) 2
7
N
2
(7+873 (1)² 55) +o?)
2
4n
2
n= qn
n=1
Tmiin
2
+8N (1))²
52
max
Σ
N
au
In
2
+
SN (7/)²
Tmax
On
2
Tmax + 8r max (19)25))
V (at))?
N
+ +8N1''² (73 + +79)
(17)²
2
(o +2
n=1 In
1 SN (1²)²² - T + 873 (7')² 52)
N
+
n
(2e² + Rn
4n
n=1
N
+ SN 2n²)²=
1
2 ; ( (7 (7²)² 52
(2e² + 2
In
n=1
<
T'Tminn
2
SSN (11²² max
In
+8N (1/)² 52
T
max
2 +85
On
N (7/) a
max
N
a
In
V (z/))²
N
N
+8N (17)3 3 5 3
Σ
+
SN
(17 3 52 2
In
Qn
n=1
B=1
N
N 22
+SN (17)² 2 7262 7
d
+
SN
1
(17) 2
0
In
2
4n
n=1
R=1
N
2
+8NS (17)*
2 (4)
4
+
SNS (1/)²
n
In
4n
n=1
R=1
+8N (1/)) 3 $223
N
a
n +
8N (n')² 3
5273 2
2
In
In
n=1
n=1
N
N
+SN (11)² 222
2 2722
SN (7)) 2 72
Zn
In
==
n=1
47

---

N
+8 (7))'
+ 4 (17)²*
2 74
Σ
In
On
n=1
n=
n
an (1+ + 852 (7')² )uuss) ) Voll (x))
T'anin
1- 1) 1
mmxx
(r/)²(uxxx)
2
Tmin
On
n=1
.G (ENS SN (1))) 4N5 (1') 71
+
Gn n
SNSL + (NS()T)*++BNN1)) (1))) ( (1/)* -4)
4n
*F (77) SN (17)* 5273 4NS (1')''1
N
Σ
+
0n
On
N
+ (NS 72 + &8N (77)* ' + 8NSL3 (11)***
n
2
4n
n=1
T'mmin
1
2NNSt
2
max
N
1+
2
1 + 32
(/2
V/alf (z') (z¹)
2
Tmin
qn
n=1
NN (1')² 72
1
+
on
2SN (11)² 72
(2
7 4 64
1
1
2
2
64
On
4n
+ (7') 72
+
N
2
o +
2SN (78)² ;2
(
1
+
1
2
2
2
64
On
4
64
4n
n=1
n=1
n
1-4N
mmxx
N
|| (z)) +2NS (79) 2 (t+77)
N
a
2
+E
2
Tmin
4n
9n
n=1
T'Tmnin
2
Vee (2'))
2NS (72 + 72)
(o +))
(153)
4
n=1 On
where we first let ny² <
165 Tmax and then 1 <
We also use VV [=)))
8SN Imox Tmin
Σn=1
VV (z*))l + || a, (='))²
Rearranging the above we have
1/ V/V (x')) < Tmin
(f(x9) -E E (ot+))) + (NS(("
(n)))
2
(o + +²)
Timin
On
n=1
(154)
Taking expectation and averaging over all 1, we have
8NN
N
T-1
ΣΣ E of (2' 0=
4
(# - --+)+ +
Tmin
a
0 + 22
(n²)².
Trmin
T
Jn
t=
(155)
48

---

G
Proof of Theorem 3.9
In Sec.
G.1
we prove the strongly convex case.
In Sec.
G.2
we prove the general convex case.
In Sec.
G.3
we prove the non-convex case.
G.1 Strongly convex case for SFL-V2
G.1.1 One-round Sequential Update for M-Server-Side Model
We first bound the M-server-side model update in round for full participation (g,. for all n), and then compute the difference between full participation and partial participation (gn < for some n). We denote I, binary variable, taking if client n participates in model training in round t, and O otherwise.
For full participation, Lemma
E. gives
Noy
E |r m=]1²] 217'TTE TE [s (x*) -f(x')
2
N
(1")
(20 + G2) + 24S+3 (1¹)3 3 Σ (20 + G2)
(156)
Considering that each client n participates in model training with probability In. we have
E
++1 x
x
+ x - 113
x ++ 22119
<
E
T-1
( 2 1)
< E
rj
g2.n ((
9n
N
NT
2
g (f
(12ii x2110) 1)
==1
1=0
S N (n1)) 2
(157)
#=1 4n
where we use E E -
<
E E|
E E[
In.
and
xf+1
x
' Enep'y) EED
gh
en s.it })-
vn
Combining the above gives
x
x
||ا| -
x(+1
x.!1 x |1
NH
E ||00 m;,17]
2
N
N (20 - Ge) + 24573 (ng) Σ
+ (n72² 2 2 Σ
(20² n
+ G²)
n=1
a=1
N
+ (7')²
(158)
gn
n=
49

---

Let A+1 & E ||o x='l²].
We can rewrite (18) as:
A++ < (1
NT
(1 (n))
72
B1 + (7')²² 73
!
8 B2.
(159)
2
where Bi = 4N - n + G²) + +NG
1
and B = 1925 >m (20 : + G²).
n= 4n
Consider diminishing stepsize r/" NT( ie.
28
Ng 9
2 7ott where 2
# 7s No
85
1. Is
easy to show that 1y' S 28+
for all L. We can prove that A S 7ott VL. Therefore, we have
max
[BL 112
+
SBg
+1 (7s + 1)
2 =;1²}
E || 1o:a==;ll"
w:ll²]
= =
V
7s 7+
+t
Ys +
16N NN
(20 G2)
16NG2 Σ
1536S Σn (20² + G
4n
122 (7s + ()
1B (76 + 1) (7s 1)
(7o + 1)EE
| x; [1]
(160)
Y +
G.1.2 One-round Parallel Update for Client-Side Models
Define _ = Exzi ag which represents the aggregating weights in round f for full par- ticipation. Using similar derivation as the M-server side, we first bound the client-side model update in one round for full participation E |<=+|==;|l"* x;]l"] and then bound the dif- ference of client-side model parameters between full participation and partial participation || -x:1== ==]²]] The overall gradient update rule of clients in each training round is x++1 E (n) Ei dn. glah (f=2) x }).
Under Assumptions
C.1 and C.2
if v <
25T in round I, Lemma D.1 gives
E
< 1
T'
EE | x: 2::13
2n' T [$ (a*) - f (xx)]
(17)² ,2 N
a (20 + +
24S+3
(n))) Σ an (20 G²)-
(161)
Considering that each client participates in model training with probability Jn. we have
E
x
1+1
x 1111
E
|
x²= + t x 113
< E
|| ||
2t'll3]
N
r-1 Σ
< E
Σ
"1
an
1.
(f
c,n
20..5
s,n })
N
< NT
(77
2
2
|| (f
2 }]]]
n=
0
S N7 (7')
6
(162)
4n
11
50

---

where we use E n EnEp'(g) Ex= 9n
I E
ET
E
In>
and
x
x
en (f 1.
We obtain the client-side model parameter update in one round for partial participation by combining the two terms and we have
e+1 c
x='l"]
|| - xet + x.t - =x²l²²
=
T''
|| x:² 1?]
E
2
(n')² ,2N Σ
On
2
(20 G + 245+ (n') Σ an (20 + G²)
n=1
n=1
+ (n')
(163)
yn
Let A++1 A E
x
1+ 2ill"]]
We can rewrite [63 as:
1+1 S
1
TH
A
( 72
Bi +
(nt)²²²
73
B2.
(164)
2
4
8
where B1 =
4N E 2 (20 + 62)
+ 4NG
Σ
n=1
a
gn
and
B2
=
1925 Σn=1 On (20 + 0²).
Consider a diminishing stepsize '/' easy show that r/' S 1 for all t. For max that A S goter Vt. Therefore, we have
28
i.e,
2
3 where 3= 3.2 7c
1. It is
+(+++)+
ret
4B. 122
+
ph 1) 17 + 1)40 J. we can prove
8Bs
28r
4B 11²
(7c+1) (7c + 1)B 1| x: 2:1²}
8Bs
max
+
E || - 2i'l"] 1
7o +1
Yo +1
16N 16N En on (20 G2)
+ 16NG B
a²
15365 ΣN
an (20 + G2) 1 (Te (% +1)
12 (%e +t)
(7e + 1)E 11020 x;'""
(165)
Ye +
G.1.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models in 160) and (655) using Proposition 3.5 For nf 28T and y 85 1, we have
P
E [f 2 )] - f(x²)
S
< 2 [B --o""" 12 + B1 xx:13))
8SN DN=, (a? + 1) 2 G2 G2
768S ΣN (a, + (2o + G²)
112 (r + T)
1²3 (r + T) +1)
S(7 + 1)R ||128 - a% I??
+
(166)
2(7+1)
51

---

G.2
General convex case for SFL-V2
G.2.1
One-round Sequential Update for M-Server-Side Model
By Lemma E.1 with p and n² 25r we have
E
x
++1
x;
E
||
x ==1l"]
-2n 22 TE (f(x') -f fxx)]
N
N
+ (7/)
72N
Σ
(20" + G²) + 24ST3 (1/)
Σ
(20 + G*).
(167)
Considering that each client n participates in model training with a probability In. we have
E
x 1+1
xtt1
S N (n1)) G2
(168)
4n
Thus, we have
x 2/. 11?
++
<
E
|| x²1 =0,1?]
-21 221/TE TE [f (x²) (a*)]
N
(7') 2
- N Σ
(2 + G²) + 24S (n')
E (2o +
+
N7² (7')²
G²
1
(169)
qn
n=1
G.2.2
One-round Parallel Update for Client-Side Models
By Lemma D.1 with μ 0 and 1/" S 25F have
E || - =x:11²
E ||02 =x:1²]] 2n EF (xx) (i)))s((s [xx')
N
+ (1') 2 N Σ a (20 + G 24S+ (77))
Σ an (20 n + G²)
(170)
#=1
Considering that each client n participates in model training with a probability In. we have
E | |+
- x.. x
N (7
62
(171)
Zn
Thus, we have
E || r-1 xz"1 x:'1[]]
E || xx:1²]
-2n rI [f (x') ff (x')]
N
+ +17)) 2
N Σ a (20 + G²) 24S+3 (n1)) 3 Σ au (20 n
G2) + + (n¹)² 62
In
(172)
G.2.3 Superposition of M-Server and Clients
We merge the M-server-side and client-side models 169 and 172 as follows
E +
E x0 ++ -x;
+E
x
52

---

E
|| 2 x1]l|]
2n/ TE [f (xt) f (x")
|| x:1 19]
TE [s (a') -1 (2*)]
N
N
(o + 1) (20 69) + 245T (1) Σ
(an + +1) (20 + G*)
N
a
+ N (1') -2 Σ
Yn
n=1
=E =B |||0*
x [1]]
4n'TTB E [f (x') - ff
+ (n7)² P Σ (a + 1) (20 G9) + 2157 (n') Σ (an +1) (20 + G22
+1
N+ (n')
(173)
4n
Then, we can obtain the relation between E x x= ||
and E || 10''-x11p
x²1]²
which is related to
E (x*) f (z*)). Applying Lemma 8 in 17 we obtain the performance bound as
B [ (f f (x")
N N
G
))
||000 ||=0 x= 1=
(an + +1) 20 + G2
On
+
n=1
N
$ /| xx"
112
S x x $12
+
248
(an + 1) (20 + G2)
(174)
2
T+1
2(T 1)
n=1
53

---

G.3 Non-convex case for SFL-V2
G.3.1 One-round Sequential Update for M-Server-Side Model
For the server, we have
a(va. f (x") xg -
S E ( V 2. f (x*)
x + 1/ VV a. (x') 77 Vv. (x'))
E V 2. f (x") xx+11
x + + TV sx f (x')) - (Va, (x 5/ V/,V 2. f (z'))|
L
V n, f (x*) E
+ w. 1 (2t) Vo.. (x') 1
qn=
N
V / (2) E
Vv, Fn (1 (
+ +n/'t vn (a')) (x²)
-n'r || Vi.. (a²
In
V 2 I (x*), E
Vz, a, Fn (farenn = + of Σ
Pr (at1)
On
On
n=1
n=1 1=0
T/T V (x
N E
S n'T V a (z') E
# v Fu
(f
v (z*)
i= On
On
- Ve, (1
1/
V (x5) (x=)|²=
+
7
N Σ
V nx
Fa
(f
(atiin }
N ΣΣ Pr (249
2
2T
Gn
==0 In
2=
7/7 / (2¹)
n'T
TTP E
N Σ
$ 2
V v. f (x²)
112
Fn ( - (Vo))-ve.. Fn (z¹))
2T
On
n'T V vv, (x')1²-
No
N
E
Σ Vv, Fr (1 22ihe) vn, Fn (z)))
2
2r
On
1 | sf I (z'))²
S²
| ==]
(175)
2
In
where we apply Assumption C.1 In.
V I (xt) = En' Va, Fn (2²),, (0,((a,))))) b) < 2²46²
2 and E [T.]
By Lemma C.6 with 17'=
we have
V8ST
E a
23 23/1]]
2T² (8r (n')2 87 (11)222 + 87
(1 IP) (176)
Thus, 175) becomes
E x, f xx') x -
n'T | If (z') ||
S²
Σ
27
8r (119) (17)202++88 +88 (1')2 2+87 (n/)² Va.. (=:))l²)
(n))²|Vm.)
2
n=1 qn
n/T + &NN (n7)
N
N (
3 3
Σ (1²)² 2 (o + +)
2
On
n=
n=
(177)
54

---

Furthermore, we have
S
||
SN (17)2 N
E
g 9,92
SN (n7))
N
1 E
Σ
9 8,72
On
SN (17))²
2
N
T
I
|
4.5
In
8,
R=1
SN (n)))²
N
T
E
gen + 11
2
In
x=1
SN (n7')²
T
E |lo0
ga [0] + +E (1..)ml11))
On
SN (n
E 119
gi.n ["] + E V a, Fn (x*) ) || + oa o
(178)
4n
where the last line uses Assumption C.I and E [|z]]]] = [| + + - E[z]||] for any random variable
By Lemma
C.7
with 17 S
1 25r. we have
878 <872(11' (n))2 5² (IV Fr (x') 12 + +
E
(179)
Thus,
178 becomes
s
1+1
x11 =11²]
x
2
SN (n')²
Σ
N
1
(87 (nt)² 52 (IV=, Bn (2')11 + to%) + T [1 (x)) o
2
On
n=l
SN (n')²²
N
1
87 (1')² )i (1V Fi (z') 1| + o.t)
T
+
2
On
SN (n²)²
2
N
1
T T (1-) (1))*5²) Fn (x') vn. (x²) Va.l /a, + +o?)
-
2
On
n=1
SN (17) T
2
N
1
( 8 (1))²5² s*) (2/Ve,/ + +o²))
(180)
2
Gn
G.3.2 One-round Parallel Update for Client-Side Models
The analysis of the client-side model update the same as the client's model update in version Thus, we have
E ( Var.f (x'), -xti|
1/' + 8N (n')²7336
Σ di On
Σ 2 (n²)²
V (x))]1
82
2 (o + +).
2
Jn
(181)
55

---

For n/ 1
1
28
5
| +
x
2
SN (n')² dn
N
T
T
(
8T
(21|va. (x+) 12 +22 + .
(182)
2
9n.
G.3.3 Superposition of M-Server and Clients
Applying
177). (80),
182 and
181 into (36) in Proposition C.4 we have
E f (a f (z')
E (V and I (x'), e+1 2.61'
+
S
E2
||
++1
=1 3 +E((V (x9) (x.)
x
+ S E
| 2+ 1a=tt=2211 231
2
2
n'T +8N (7")* 3 S²
N
1
V (2
On
n=1
N
+ +8N1 82 3
+1
1
(1')² 2 62)
On
x=1
SN (77))²
2
T
N
1
T + 87 (7')² 2 52) 2 Vol (x))
2
qn
n=1
SN (77)²²
N
2 +1
T 873 (11)² (26² + on)
2
Qn
n=1
n'T
+8N (7")* $2.3
+
Σ
N
1 SN (1¹)²² + Σ
N 1 (T + 873 (1))282) ) |V (at))
2
On
On
n=1
+ +8N1 52 73
1
(17)) 2 (o +
Gn
x=1
1 + SN (11)² 2- T T 873 (1') 52)
Σ
N
+1
(2e² + o
2
In
n=1
n + SN (n²)2 72
Σ
N
1
88 (n²)² 22,3 3
Σ
N
1
N (1t)
Σ (
|| (z*))1
2
n=1 In
In
n=1 on
n=1
N
+8N (7²)² 3 22 3
+1
o 8N
(1¹)²
3 52 2
+1
In
4n
n=1
N
N Σ
+SN (17)² 2 72(2 22?2
+1
2 -- SN (1²)²
4 +
+
2
In
Jn
1=
n=1
N
N
+8N53 (7')* 1 742
0
+1
4N (7¹)² 1 4
0
+1
+
On
In
On
n=1
n=1
n/T 1-2SN
N
1
(1+
+ 852 (11))
V (12
2
In
n=1
+ (1))* 8N (n') 52 4N5 + (1/)* ,/
Σ
a3
+ 1
0
In
==1
N
+ +(NS(1)2² 288 (11)²
88N
+ +NNSL3 ( ()))
a
6
In
n=1
56

---

N
+ ) ||Vas (x*))
1- 1-2NSy
1+ 2
2
n=1
+N (170)²
72
1
+
+
N
a
+1
2 +28 7/
+
+
1
N
2
6
2
2
64
On
4
64
4n
n=1
n=1
n'T
1
I-
Sn
N
Ve. (z')
2NS (1/)²
72
2
+1
+0)
2
5
4n
n=1
x=1
n'T (
a +1 1
+ 2NS (n7)² 72
(o +
(183)
4
qn
where we first let n 165 and then let n/ S 89N3, Σm
We have applied Σ og S N.
Rearranging the above we have
N 61)
n V f (3')) < 4
Σ
a
+1
(o
tE). (184)
4n
n=1
Taking expectation and averaging over all t, we have
N
T-1
+ 1=U
4 s T (a0) - -
8NST
+ 1
Σ
(r/)²
o +
T
On
1=0
(185)
57

---

H Comparative Analysis
H.1 Main technical results
We conclude the convergence results in our paper in Table |
Table
Performance upper bounds for different objectives (let Q := L Too
Scenario
Case
Method
Convergence result
SFL-V1
(N + G2) + 1.STen)
Strongly convex
SFL-V2
(N +?) + uSSTo
(S+
Full participation
SFL-VI
Noo'+6")
+( (Nlo
+ $ Iam
General convex
SFL-V2
(2
} 1
SFL-VI
NS
F
Non-convex
T
SFL-V2
N²
pore
SFL-VI
(N(o² + G Q)) + pS1
Strongly convex
SFL-V2
(N² (1 Q)) + pSSTer)
AS+
Partial participation
SFL-V1
(Nl NG++G"(44Q)
[M $ poor
General convex
N² 42
NO x= ) + & perr
SFL-V2
+
NS pe
SFL-VI
Non-convex
T s' por
T
SFL-V2
H.2
Comparison of Bounds
We compare our derived bounds for SFL other distributed approaches. For simplicity, let a, = 1/N in (1) and o, = a for all n in 66. The result are summarized in Table Since different convergence theories make slightly different assumptions, clarify them below.
In
a is the variance of the stochastic gradient at the optimum: Fi/N characterizes the client heterogeneity. [7], e characterizes the client heterogeneity at the optimum, similar to 12 ie., E. FV ]| e.
33
EgawD,
11 (xr", S..) -.)-VF (x (**))]
In f
Table 2: Performance upper bounds for strongly convex objectives with full client participation. Here, absolute constants and polylogarithmic factors are omitted. We further relax the upper bounds of SFL-V2 for an easier comparison.
Method
Performance upper bound
Mini-Batch SGD
33
KN+T
S(f(=")
exp
FF S
FL
16
8 #T+T a²
O2+-STNNN+2G STor
uN
10
nNTT
+
p²rT"
+ p²72 e exp
Sea
=uf
SL
+ N
So²
+ WaNT= + porr exp
-TT
uN+T
9
SFL
SFL-VI VI (Theorem
3.6
(N(o2 + G2) + pSST (N2 G2) HSSTE
A(S+uT)
SFL-V2 (Theorem 3.7
A(S+aT)
The key observation is that our derived bounds match the other distributed approaches the order of T' and they all achieve (1/TT).
58

---

Furthermore, we will compare the convergence upper bounds of SFL to those of distributed SGD in I2
(with parameters p 1 and ç² as follows:
Table 3: Performance upper bounds for different objectives with full client participation.
Case
Method
Convergence results
Distributed SGD
a² SI exp
#7
ANT S
Strongly convex
SFL-VI
(N(o²² + G²) pSSTem)
(S+
SFL-V2
(N2 G²²) + USI
MS+NT)
Distributed SGD
NTa + Ier N
General convex
SFL-VI
+(?)
+ $ _ $ Jerr
N² + (a -
SFL-V2
Distributed SGD
NT NSIo 4c")
+ F
7)
Non-convex
SFL-VI
pam
SFL-VV
N²
parr
In the aforementioned table, we denote on a, define f - || x 11 and represent F & fx=") f The absolute constants and polylogarithmic factors are omitted for brevity. Our SFL algorithms show the same convergence rate the distributed SGD.
H.3 Comparison of Communication and Computation Overheads
There have been are some papers discussing the overhead of SFL, e.g., 27 analysis from (27]
We mainly use the
We start with the definitions. Let K represent the total number of clients involved, D denote the aggregate size of the data, and indicate the size of the smashed layer. The rate of communication is given by R, while Tib signifies the duration required for a complete forward and backward propagation eycle the entire model for dataset of size D, applicable across various architectures. The time needed to aggregate the full model is expressed as Tiedavg. The full model's 's size denoted by [W), and reflects the proportion of the full model's size that is accessible to client in SFL, specifically, |Wel = W The factor 2r|W] in the communication per client arises from the necessity for clients download and upload their model updates before and after the training process. These findings encapsulated in Table It is observed that as escalates, the cumulative cost training time tends rise following the sequence: SFL.-V being less than SFL-VI.
Table 4: Communication and computation comparison between FL, SL, and SFL.
Method Communication per client Total communication
Total model training time
FL
2 W
2K [|]
Tbb + 2
+ Tredavg
R
SL
2D yu 2r W
2Dv + 2r 2rK(W
Tnb + 2Dv
+
2r K
R
SFL-VI
(2 yu 2r [r
2Dv + 2r
Tn RK
2Dv
+
W
Trodavg
R
SFL-V2
(2D
(K Y + W
2Dv + 2rK W
Tob +
2Dv
Tha
RK
R
59

---

100
SFLMI
FodOpt
Accuracy
Accuracy
60
60
SPL-V1
St.
SFL-V2
FedProx
150
%
150
Training Round
Training Round
(a) 8 0.1, 10.
(b) 0.1, 100.
Figure 6: Performance comparison CIFAR-10.
Acuracy
60
Accurry
60
Accuracy
60
60
40.
80
40
40
20
E-
6-
20
Em
Training Round
Traming Round
Tarinn Round
Training Round
(a) SFL-VI on CIFAR-10. (b) SFL.-V CIFAR-10. (c) SFL-V1 on CIFAR-100. (d) SFL. on CIFAR-100.
Figure 7: Impact of local iteration SFL performance.
I Additional Experiments
L.1 More comparing methods
We compare the SFL methods with the benchmarks, i.e., FedProx [3] and FedOpt 19 used the same hyperparameters, and trained the models CIFAR-10. The results provided in Figure From the figures, we observe that SFL.-V2 continues to be the best-performing algorithm. This is consistent with observations in the main paper.
We have
1.2 Impact of local iteration
We further study the impact of local epoch number E on the SFL performance. The results are reported in Fig. 7 We observe that SFL generally converges faster with larger demonstrating the benefit of SFL in practical distributed systems.
I.3 Results using loss metric
We further report the results using the loss metric. More specifically:
Impact of cut layer: The results reported in Figs. 8
and
g
Impact of data heterogeneity: The results are reported in Fig.
Impact of partial participation: The results are reported in Fig.
In general, we see similar (but opposite) trends with the observations in the main paper. That higher accuracy is associated with smaller loss. These results are again consistent with our theories.
L.4 Results on the impact of the position cut layer.
We can observe from the results that the performance of SFL-V1 and SFL-V2 increases in L... We look at the impact the position cut layer from the gradient perspective. We plot the gradient divergence in Fig. [2
60

---

0.3
0.2
0.
0.4
6.4
0.1
0.2
0.2
0.0g
0.0g
0.0g
0.0g
Training Round
Traningg Round
Takinng Round
Traiiing Round
(a) SFL-VI on CIFAR-IO. (b) V2 on CIFAR-I6 10. (c) SFL-VI V1 on CIFAR- 10TAR-100.C0 (d) SFL-V2 on CIFAR 1000
Figure 8: Impact of the choice of layer on SFL training loss.
10
10
2
Lose
50
150
150
200
50
100
150
200
50
100
150
200
1055
150
200
Training
Traning
Traminng
Training
(a) SFL-VI CIFAR-I0. (b) SFL-V2 CIFAR-IO. (c) SFL-VI CIFAR- (d) SFL-V2 CIFAR-100.
Figure 9: Impact of the choice of cut layer on SFL test loss.
We see for SFL-VI and SFL-V2, the gradient divergence decreases as we choose a latter cut layer (a larger L.). This means that the client drift issue is less severe and hence the performance increases. In addition, from a theoretical perspective, if we write e² (i.e., upper bound gradient divergence defined in Assumption 3.3) function of Le, we can see that the upper bounds of performance loss decrease in Le. This provides a theoretical angle that the performance of SFL-VI and SFL-V2 increases in Le-
1.5 Results on FEMNIST dataset
We have conducted more simulations on a larger dataset FEMNIST. In particular, we consider N = 100 and train FL., SFL-V1, SFL-V2. Note that the data come from different sources and are heterogeneous across clients. The results are reported Fig. 13
We note that our key observation continues to hold. That is, S2L-V2 outperforms FL and SFL-VI under-performs FL under heterogeneous data and large number of clients.
61

---

2w01
5-01
0-
0.6
0.8
0.4
0.0
6.6
0,3
0.3
0.4
6.6
0.2
0.2
0.1
0.1
0%
150
200
0.6g
50
100
200
0.8g
100
150
200
300
150
Training Round
Traning Round
Trkking Hound
Training Round
(a) SI on CIFAR-IO. SFL-V2 on CIFAR-10 10. (c) SFL-VI on CIFAR (d) SFL-V2 on CIFAR-100
Figure 10: Impact of data heterogeneity SFL training loss.
0.6
0.5
0.5
1.0
0::02
0::22
9-2
0.5
0.8
80.3
0.2
0.1
0.1
0.2)
0.0g
100
150
200
0.0
50
100
200
0.0g
50
100
150
200
50
100
150
200
Training
(a) SFL-VI on CIFAR-IO. (b) SFL-V2 CIFAR-10. (c) SFL-VI CIFAR-100. (d) SFL-V2 on CIFAR-100.
Figure 11: Impact of client participation on SFL training loss.
Avg. Gradient Var. across Clients
255
SR.VI SR-42
100
30
80
Accuracy
40
Layer
Training
Figure 12: Results on the impact of the position
Figure 13: Results on FEMNIST.
of cut layer.
62

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We claim that we provide convergence analysis split-federated learning (SFL.) for strongly convex, general convex, and non-convex objectives heterogeneous data the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section we claim that also important to theoretically analyze how the choice of the cut layer affects the SFL performance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
63

---

Answer: [Yes]
Justification: Section presents the theoretical results with full set assumptions. The proof of Proposition 3.5 is given in Appendix C.4 The complete proofs of Theorems 3.6 3.7 3.7.are are given in Appendices respectively. Proofs of Theorems 3.6 3.7 are given in
Appendices D respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4.1 presents the setup of our experiments reproduce the experimental results. We further provide our codes in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
64

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide our codes in the supplementary material. We will provide open access to the data and code if the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wwtpcc/pccpccpcceeeeeeeoocp httpc/ggguu.eeeeeeeuuuubiiiiiiii ieriiisssonoooii
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the setup our experiments, including the datasets and hyperpa- rameters, in Section 4.1 and provide codes in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This theory paper. Due to limit of computation resources and time, we were not able to run more experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
65

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have used one CPU and one GPU. Specifically, the CPU is Intel(R) Xeon(R) Gold 5320 CPU with 2.20GHz, and the GPU is A100-PCIE-80GBB Memory:
256GB Storage: 10TB. Each experiment curve takes about 10 hours to train.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have conformed to all aspects of code of ethics with this submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The goal of this work is to provide a comprehensive analysis on the convergence of SFL. Our theoretical results have two potential impacts. First, provide a thorough understanding on the performance SFL., which potentially guides the implementation of SFL (c.g., the choice between FL and SFL, the choice of hyper-parameee and cut layers).
Justification:
66

---

Second, the convergence results can be used for modeling the training performance of SFL. Together with an effective modeling of communication and computation overheads of clients, researchers will be able to perform SFL system optimization. Due to the reduced clients' training loads SFL,, such system optimization can potentially minimize the burden at clients (e.g., human mobile devices) while maintaining client privacy and satisfactory training performance.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks. We use open-access datasets and models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our codes are based on the codes provided in 27 which was cited in the main paper.
67

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
68

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
69


Reviews:
[{"review_id": "0b1K72uc42", "reviewer": "Reviewer_fmCP", "summary": "This paper provides theoretical convergence analysis for split federated learning (SFL) methods (SFL-V1 and SFL-V2) under full and partial client participation for strongly convex, generally convex and non-convex settings. These bounds help build fundamental understanding of the settings in which SFL, federated learning (FL) and split learning (SL) are best suited and where SFL suffers (e.g., in terms of data heterogeneity, number of clients, participation rate). The authors provide empirical results that validate their theoretical claims and showcase the performance tradeoffs between SFL, FL and SL.", "questions": "Where are there diminishing returns in increasing L_c? If a increasing L_c yields better performance, then why not just use FL? What tradeoffs should be considered when selecting L_c beyond just accuracy?", "limitations": "While there is no explicit limitations section in the paper, and such discussion could be expanded upon in the text, the authors do mention that the choice of how to set the cut layer (L_c) is unknown and left for future work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Well-written and well-structured.\n* Theoretical convergence analysis provided for both full and partial client participation across strongly convex, general convex and non-convex objectives.\n* Empirical validation of theoretical insights with ablations across datasets on data heterogeneity, number of clients and client participation rate. \n* Important and interesting findings as to the most applicable data settings for each algorithm.\n* Includes comparison of SL, FL and SFL in terms of accuracy, convergence speed, communication and computation costs.", "weaknesses": "* Empirical choice of L_c is not properly swept. Given that the highest value of L_c considered is found to yield the best performance, L_c should be increased until worse performance is observed to evaluate SFL and SL at the optimal choice of cut layer and understand at what point diminishing returns are observed from increasing L_c. Is it always better just to use FL if computational resources allow? What are the tradeoffs?\n* Lacking specifics of what FL and SL algorithms are used for comparison. Opportunity to consider other algorithms beyond the most basic implementations of FL and SL.\n* Plots are hard to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Our response to your comments is as follows.\n\n**1. Choice of $L_c$ and tradeoff**\n\nWe use the ResNet-18 model in our experiments, which contains four blocks. This means the candidate for $L_c$ is in {$1,2,3,4$}. Thus, $L_c=4$ is already the maximum $L_c$ that can be chosen in SFL with ResNet-18 model. Our experimental results in Figure 2 show that increasing $L_c$ leads to higher model accuracy for SFL. However, the model accuracy of FL is similar to SFL-V1, and both are lower than SFL-V2 under high data heterogeneity. This suggests that even if computational resources allow, it is not always better to use FL than SFL. The main reason is as follows. Under high data heterogeneity, FL suffers from the notorious client drift issue, while SL suffers from the catastrophic forgetting problem. SFL (in particular SFL-V2) combines the training logic of FL (at clients) and SL (at server), which can help mitigate the issues of client drift and forgetting. Hence, SFL can lead to a better performance than both FL and SL.\n\nOn the other hand, as suggested in Table 3 of the Appendix, SFL-V1 and SFL-V2 may have higher communication overheads than FL when the number of training data and smashed layer size are large. Therefore, there is a tradeoff between model accuracy and total training time in SFL. \n\n**2. Algorithm details** \n\nThe FL algorithm we use is FedAvg, and the SL algorithm we use is given in [Poirot et al. 2019]. We have also included more recent FL algorithms, i.e., FedProx and FedOpt, as additional comparison. The results are given as follows, which show that SFL-V2 outperforms FL and SL under high data heterogeneity. This is consistent with our main conclusion in the submission. \n\n[Poirot et al. 2019] Poirot M G, Vepakomma P, Chang K, et al. Split learning for collaborative deep learning in healthcare. NeurIPs, 2019.\n\n\n**3. Plot presentation** \n\nWe have adjusted the fonts and line sizes to make the plots more readable. \n\n\n**4. Limitation of our paper**\n\nOne limitation is that our bounds for SFL achieve the same order (in terms of training rounds) as in FL, yet the experiments showed that SFL outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL are to be derived, which is an important future work. Another limitation is that we did not include the theoretical analysis on how the choice of cut layer affects the convergence. We will add the more detailed discussions of limitations to the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WjY3QRpnp", "reviewer": "Reviewer_b5tv", "summary": "The paper proposes the first convergence analysis of the two variants of split federated learning under different set of hypotheses.", "questions": "See weaknesses", "limitations": "A discussion of the limitations (e.g. the fact that the convergence results hold only when clients are allowed to communicate after each batch) is missing.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "To the best of my knowledge, as also stated by the authors, there is no theoretical analysis of split federated learning. This is a serious knowledge gap, given the potential practical interest of split FL. The paper proposes the first of such analysis under the classic set of hypotheses: L-smoothness and strong convexity, L-smoothness and convexity, non-convexity.", "weaknesses": "I was not able to check the proofs in their completeness, but there are some signs suggesting that the analysis may not have been carried out as carefully as required. First, it seems some hypotheses are missing in the statement of the main theoretical results. For example, convergence to the minimizer in presence of stochastic gradients is not possible unless the learning rate opportunely decreases over time, but this is not mentioned. Second, I have some concerns because 1) the verbal description of the studied algorithms is not very clear, and, more importantly, 2) even their mathematical descriptions are not consistent across the paper. Before moving to the specific comments, I want to stress that a certain vagueness in the actual operation of the algorithm is common to the original papers on split learning and split federated learning, but it is a less serious concern in their case because they do not provide theoretical guarantees.\nMy main concerns are about how models are aggregated. Now, the two aggregations presented in the main text (equation (3)) and in the algorithm formal presentations in appendix B (line 22 in algorithm 1) do not match. The first one presents an unbiasing effect, the second one not. More importantly, neither one nor the other can lead to asymptotic convergence under partial participation, because there would be a persistent noise due to the variability in the set of participating clients. By a first look at the proof, it seems the proofs do not consider neither of these aggregation rules but rather an aggregation at the level of the gradients (see page 15), which may converge (but I did not check), but there are still some potential problems for SFL-v3 (line 453) because the a_n are ignored. \n\nI have also some concerns about the comparison with FedAvg in the experimental section. My understanding is that the authors study a version of SplitFL where clients communicate after each batch. In this setting, SFL-v1 coincides with FedAvg if 1) \\tau=\\tilde{\\tau} (both the server and the clients average the models at the same time) and 2) FedAvg is allowed to transmit after each gradient computation (as it seems required for a fair comparison). Now, the results presented show different performance for FedAvg and SFL-v1. Unfortunately, there is no information reported about the specific setting (i.e., what are \\tau, \\tilde{\\tau} and how many local gradient steps FedAvg performs before communicating).\n\nThe authors have addressed my criticisms above and I have upgraded my score. It is required a significant review of the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! Our response to your comments is as follows.\n\n**1. Diminishing stepsize for $\\mu$-strongly convex loss functions** \n\nWe use a diminishing stepsize of $\\frac{4}{\\mu\\tilde{\\tau}(\\gamma+t)}$ and update the conditions specific to $\\mu$-strongly convex loss functions. For these types of loss functions, Theorems 3.5 - 3.8 rely on the hypothesis that $\\eta^t = \\frac{4}{\\mu\\tilde{\\tau}(\\gamma+t)}$ for the client-side model, $\\eta^t = \\frac{4}{\\mu\\tau(\\gamma+t)}$ for the server-side model, and $\\eta^t \\leq \\frac{1}{2 S\\tau_{max}}$. When $\\gamma = \\frac{8S}{\\mu} - 1$, we have $\\frac{4}{\\mu\\tilde{\\tau}(\\gamma+t)} \\leq \\frac{1}{2 S\\tau_{max}}$ and $\\frac{4}{\\mu\\tau(\\gamma+t)} \\leq \\frac{1}{2 S\\tau_{max}}$.\n\nThus, the hypotheses for the learning rate in different cases are as follows:\n\n- $\\mu$-Strongly Convex: $\\eta^t = \\frac{4}{\\mu\\tilde{\\tau}(\\gamma+t)}$ for the client-side model, $\\eta^t = \\frac{4}{\\mu\\tau(\\gamma+t)}$ for the server-side model.\n\n- General Convex: $\\eta^t \\leq \\frac{1}{2 S\\tau_{max}}$.\n\n- Non-convex: $\\eta^t \\leq \\frac{1}{2 S\\tau_{max}}$.\n\nOur proofs in the appendix (lines 534, 543, 607, 616, 668, 687, 746, 764) support this analysis.\n\nIt is important to note that these hypotheses apply to both SFL-V1 and SFL-V2, under full and partial participation of clients. Additionally, $\\tilde{\\tau} = \\tau$ for SFL-V2. We will revise the presentation in our main paper accordingly. \n\n\n**2. Model aggregation** \n\n In SFL algorithms, the clients' local model parameters are aggregated according to equations (2) and (3) for full and partial participation, respectively. This process ensures unbiased model updates. Detailed model updates for both the server and clients in SFL-V1 and SFL-V2 can be found in Appendix C.2. Algorithms 1 and 2 describe the process for full participation only. We plan to revise these algorithms to include partial participation.\n\nIn our proof, we consider the features of model aggregation in different scenarios, as outlined in Appendix C.2 (page 15). Specifically, in line 453, which describes the model aggregation for SFL-V2 with partial participation on the server side, the parameter $a_n$ is not needed. This is because the main server updates the server-side model parameter sequentially with each client. \n\n**3. Performance comparison between FedAvg and SFL-V1**\n\nSFL characterizes dual-paced model aggregation. Specifically, \nin SFL-V1 and SFL-V2, clients communicate with the main server during every iteration (batch) for sequential model training. Additionally, clients communicate with the fed server every $\\tilde{\\tau} \\geq 1$ iterations (batches) for client-side model aggregation. On the main server side, in SFL-V1, the main server aggregates the server-side models every $\\tau \\geq 1$ iterations (batches). In contrast, in SFL-V2, the main server does not need to aggregate the server-side model because there is only one server-side model.\n\nIn our experiments, we set $\\tau = \\tilde{\\tau}$ to ensure a fair comparison with vanilla FL, i.e., FedAvg. Theoretically, the performance of SFL-V1 should be the same as FedAvg. However, our experiments show that FedAvg outperforms SFL-V1, especially when the data is non-iid distributed across clients, as shown in Fig. 5 (c) and (d).\n\nThe reason for this discrepancy lies in the optimizer used in our experiments. We added momentum to the SGD optimizer to stabilize and accelerate model training. In SFL-V1, there are two separate optimizers for the client-side and server-side models. In contrast, FedAvg uses only one optimizer, which provides better global momentum than SFL-V1. As a result, FedAvg performs slightly better than SFL-V1. This phenomenon is also noted in the original SFL paper [1].\n\nWe will revise the description of the experimental settings and results in our paper.\n\n[1] Thapa, C., Arachchige, P. C. M., Camtepe, S., \\& Sun, L. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022, Vol. 36, No. 8, pp. 8485-8493.\n\n\n**4. Limitation of our paper** \n\nOne limitation is that our bounds for SFL achieve the same order (in terms of training rounds) as in FL, yet the experiments showed that SFL outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL are to be derived, which is an important future work. Another limitation is that we did not include the theoretical analysis on how the choice of cut layer affects the convergence. We will add the more detailed discussions of limitations to the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SmeIoicp8W", "reviewer": "Reviewer_j2yY", "summary": "This paper analyzes the convergence of split federated learning (SFL) on heterogeneous data and gives theoretical convergence rates under different assumptions on the convexity of the loss function and the participation schemes of the clients.", "questions": "1. In Figure 4, it can be observed that the accuracies of the algorithms on CIFAR-10 and CIFAR-100 are very close (the latter only slightly lower). However, the accuracies on CIFAR-100 generally have an observable gap compared to CIFAR-10. Is this due to the small number of clients (only 10) in the experiments? If so, how would the results change if the number of clients is increased?\n\n2. In Figure 7, the losses of SFL-V2 (the hard-sharing scheme) almost tend to zero for the cutting layer = 1 or 2. I wonder if these curves are losses on the training set or the test set. If the losses are on the test set, the results are surprising, since even centralized training cannot achieve zero loss on the test set. Could the authors provide more insights into this phenomenon?", "limitations": "The last section discusses the limitations and provides some future research directions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper considers the problem comprehensively, including the assumptions on the convexity of the loss function (strongly convex, convex, and non-convex), the participation schemes of the clients (full, partial), and split schemes (soft-sharing: SFL-V1, hard-sharing: SFL-V2), and provides a very detailed theoretical analysis of the convergence rates under these different settings.", "weaknesses": "Numerical experiments are comparably weak since the baselines for comparison are FedAvg and SL. The latter is pure local training under the perspective of federated learning. The former is a very basic federated learning algorithm. Perhaps the authors could compare with more advanced federated learning algorithms, such as FedProx, FedOpt, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! Our response to your comments is as follows.\n\n**1. More comparing methods**\n\nWe have included the benchmarks, i.e., FedProx and FedAdam, as suggested. We have used the same hyper-parameters, and trained the models on both CIFAR-10 and CIFAR-100. The results are provided in Figure 1 in the [global rebuttal](https://openreview.net/forum?id=ud0RBkdBfE&noteId=kM9CF1FAdt). From the figures, we observe that SFL-V2 continues to be the best performing algorithm. This is consistent with our observations in the main paper. \n\n**2. Explanation on the experimental results of CIFAR-10 and CIFAR-100**\n\nYou are right. On CIFAR-100 we used 10 clients. Hence, after data sampling (with a Dirichlet parameter $\\beta=0.1$), each client still has relatively many classes (out of 100), leading to similar performance as in CIFAR-10. We have conducted more experiments with 100 clients on CIFAR-100 using $\\beta=0.1$, and the results are given in Figure 2 in the [global rebuttal](https://openreview.net/forum?id=ud0RBkdBfE&noteId=kM9CF1FAdt). The key observation is that the accuracy with 100 clients is generally lower than that with 10 clients, as expected. The reason is that each client has fewer types of classes, which can escalate the client drift issue during training. \n\n**3. Explanation on the experimental results of loss**\n\nThe loss plot is based on the training set rather than the test set, which is why, in certain cases, the losses approach zero.  We have also plotted the impact of the choice of cut layer on the SFL test loss, as shown in  Figure 3 in the [global rebuttal](https://openreview.net/forum?id=ud0RBkdBfE&noteId=kM9CF1FAdt). In this figure, we observe that the test loss is higher than the training loss, highlighting the generalization gap and the effect of cut layer selection on model performance.\n\n \n\n**4. Limitation and future work of our paper**\n\nOne limitation is that our bounds for SFL achieve the same order (in terms of training rounds) as in FL, yet the experiments showed that SFL outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL are to be derived, which is an important future work. Another limitation is that we did not include the theoretical analysis on how the choice of cut layer affects the convergence. \nFor future work, we can apply our derived bounds to optimize SFL system performance, considering model accuracy,  communication overhead, and computational workload of clients. \nWe will add more detailed discussions of limitations and future work to the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jRTsf5P1QA", "reviewer": "Reviewer_NwMP", "summary": "Split Federated Learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this work aims to fill this gap. The analysis of SFL can be more challenging than that of Federated Learning (FL), due to the potential dual-paced updates at the clients and the main server. This work provides a convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data.", "questions": "Please refer to the Weaknesses.", "limitations": "NA", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Split federated learning is an interesting field in FL, however, existing work fail to provide some theoretical analysis for SFL, e.g., convergence rate. And this is the first work provide convergence analysis of SFL for strongly convex, general convex, and non-convex objectives on heterogeneous data.\n\n2. This work offers comprehensive discussion about the convergence rate of SFL under several settings and the theoretical contributions are excellent. Specifically, this work provides the convergence analysis for SFL with strongly-convex, convex, and non-convex objectives under the full participation and partial participation settings. Moreover, the assumptions in the theoretical analysis are mild and widely-used in the literature of FL.\n\n3. The presentation and organization of this work are good.\n\n4. Extensive experiments are conducted to evaluate the properties of SFL, e.g., the impact of cut layer, impact of data heterogeneity, and so on.", "weaknesses": "I believe this is a solid and interesting work, I also have some minor concerns as follows.\n\n1.\tThe section on Related Work is too brief, more discussion about the SFL needs to be included.\n\n2.\tI suggest that the author provide a table to summarize and list the convergence rate of SFL under different settings, such as convex, non-convex, full participation, and partial participation. In this way, the theoretical results can be clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! Our response to your comments is as follows.\n\n**1. Related work** \n\nWe enriched the related work as follows. In particular, we have added more discussions on the references. \n\n There are many convergence results on FL.  Most studies focus on data heterogeneity. For example, [Li et al. 2020] proved the convergence of FedAvg on heterogeneous data. [Wang et al. 2019] gave the convergence of FL, which is used to design a controlling algorithm to minimize loss under budget constraint. [koloskova et al. 2020] provided a general framework of convergence analysis for decentralized learning.  Some studies look at partial participation. For example, [Yang et al. 2021] speeds up FL convergence in the presence of partial participation and data heterogeneity. [Wang and Ji 2022] gave a unified analysis for partial participation using probabilistic models. [Tang and Wong 2023] analyzed how to tackle system induced bias due to partial participation.  There are also convergence results on Mini-Batch SGD, where [Woodworth et al. 2020] argued that the key difference between FL and Mini-Batch SGD is the communication frequency. \n\n\nTo our best knowledge, there is only one recent study [Li and Lyu 2024] discussing the convergence of SL. The major difference to SL analysis lies in the sequential training manner across clients, while SFL clients perform parallel training.\n\n\n[li et al. 2020] Li X, Huang K, Yang W, et al. On the convergence of fedavg on non-iid data. ICLR, 2020.\n\n[Wang et al. 2019] Wang S, Tuor T, Salonidis T, et al. Adaptive federated learning in resource constrained edge computing systems[J]. IEEE journal on selected areas in communications, 2019, 37(6): 1205-1221.\n\n[Koloskova et al. 2020] Koloskova A, Loizou N, Boreiri S, et al. A unified theory of decentralized sgd with changing topology and local updates, ICML, PMLR, 2020: 5381-5393.\n\n[Yang et al. 2021]Yang H, Fang M, Liu J. Achieving linear speedup with partial worker participation in non-iid federated learning. ICLR 2021.\n\n[Wang and Ji 2022] Wang S, Ji M. A unified analysis of federated learning with arbitrary client participation. NeurIPs, 2022, 35: 19124-19137.\n\n[Tang and Wong 2023] Tang M, Wong V W S. Tackling system induced bias in federated learning: Stratification and convergence analysis. INFOCOM, 2023: 1-10.\n\n[Woodworth et al. 2020] Woodworth B E, Patel K K, Srebro N. Minibatch vs local sgd for heterogeneous distributed learning.  NeurIPs, 2020, 33: 6281-6292.\n\n[Li and Lyu 2024] Li Y, Lyu X. Convergence analysis of sequential federated learning on heterogeneous data. NeurIPs, 2024, 36.\n\n\n**2. Table of theoretical results** \n\nWe conclude the convergence results in our paper as follows ($Q:=\\sum_{n=1}^N\\frac{1}{q_n}$):\n\n| Scenario | Case |    Method    |  Convergence result  |\n|---|---|---|---|\n|Full participation| Strongly convex | SFL-V1| $\\frac{S}{\\mu(S+\\mu T)}( N(\\sigma^2 + G^2)+ \\mu S I^{\\rm err})$|\n|Full participation| Strongly convex | SFL-V2|$\\frac{S}{\\mu(S+\\mu T)}(N^2 (\\sigma^2+G^2)+ \\mu S I^{\\rm err})$|\n|Full participation| General  convex| SFL-V1|$(\\frac{N(\\sigma^2 + G^2)}{T})^{\\frac{1}{2}}+(\\frac{N(\\sigma^2 + G^2)}{T})^{\\frac{1}{3}}+\\frac{S}{T} I^{\\rm err} $|\n|Full participation| General  convex| SFL-V2 | $(\\frac{N^2(\\sigma^2 + G^2)}{T})^{\\frac{1}{2}}+(\\frac{N^2(\\sigma^2 + G^2)}{T})^{\\frac{1}{3}}+\\frac{S}{T} I^{\\rm err} $|\n|Full participation| Non-convex | SFL-V1 | $\\frac{NS(\\sigma^2 + \\epsilon^2)}{T}+\\frac{\\boldsymbol{F}^{err}}{T}$|\n|Full participation| Non-convex |  SFL-V2 | $\\frac{N^2S(\\sigma^2 + \\epsilon^2)}{T}+\\frac{\\boldsymbol{F}^{err}}{T}$|\n|Partial participation | Strongly convex | SFL-V1 |$\\frac{S}{\\mu(S+\\mu T)}( N(\\sigma^2 + G^2(1+Q))+ \\mu S I^{\\rm err})$|\n|Partial participation | Strongly convex | SFL-V2 | $\\frac{S}{\\mu(S+\\mu T)}(N^2 (\\sigma^2+G^2(1+Q))+ \\mu S I^{\\rm err})$|\n|Partial participation | General  convex | FL-V1 | $(\\frac{N(\\sigma^2 + G^2(1+Q))}{T})^{\\frac{1}{2}}+(\\frac{N(\\sigma^2 + G^2)}{T})^{\\frac{1}{3}}+\\frac{S}{T} I^{\\rm err} $| \n|Partial participation | General  convex | SFL-V2 | $(\\frac{N^2(\\sigma^2 + G^2(1+Q))}{T})^{\\frac{1}{2}}+(\\frac{N^2(\\sigma^2 + G^2)}{T})^{\\frac{1}{3}}+\\frac{S}{T} I^{\\rm err} $|\n|Partial participation | Non-convex |SFL-V1 | $\\frac{NS(\\sigma^2 + \\epsilon^2)Q}{T}+\\frac{\\boldsymbol{F}^{err}}{T}$|\n|Partial participation | Non-convex | SFL-V2 | $\\frac{N^2S(\\sigma^2 + \\epsilon^2)Q}{T}+\\frac{\\boldsymbol{F}^{err}}{T}$|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wZ93lospcA", "reviewer": "Reviewer_8Y1a", "summary": "Split federated learning combines the advantage of split learning and federated learning. However, there is \na lack of theoretical analysis. This paper provides the first comprehensive analysis on split federated learning. The settings include strongly convex, general convex and non-convex. This paper discusses both the full participation and partial participation case. Both two versions SFL-v1 and SFL-v2 are analyzed. Moreover, some experiments are conducted to compare split federated learning with federated learning and split learning.", "questions": "1.In eq.(17): It seems that partial participation significantly increases the error. In particular, $G^2/q_n$ is significantly larger than $G^2$. It is not the case for federated learning. In the ICLR paper, compare eq.(6) with eq.(4), the only difference is that B is changed to B+C. Therefore, the results seem to suggest that the Splitfed has significantly worse performance for partial client participation. Please provide some intuitive explanations.\n\n2. a_n is much smaller than 1, thus eq.(20) is significantly larger than eq.(17), indicating that SFL-V2 performs worse. This seems to be inconsistent with the experiment (which shows that SFL-V2 is better). My guess is that the current bound is not tight. Please explain.\n\n3. Are there any supports from theoretical analysis, proving that SFL indeed performs better than FL/SL under high heterogeneity?", "limitations": "I think that the authors do not summarize the limitations well. The authors have mentioned how the choice of the cut layer affects the SFL performance. However, I think that the major limitation is that the good performance under strong heterogeneity is not well supported by theoretical analysis.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper provides the first theoretical analysis of split federated learning. I think that this is an important problem. \n\nThe analysis is comprehensive. It includes strongly convex, general convex and non-convex case. It also discusses full client participation and partial client participation. Two variants of SFL are discussed, SFL v1 and SFL v2.\n\nThis paper has a long appendix, indicating that authors indeed have worked hard on this work.", "weaknesses": "I am impressed by the long appendix, which shows that the authors indeed work very hard on this problem. Nevertheless, I think that this paper is not ready for publication currently, due to many potential concerns.\n\n1. In paragraph 2 of introduction, this paper states that split learning leads to high latency, but this paper does not provide theoretical/experimental evidences.\n\n2. Inconsistency between theory and experiments. The theory only gives the convergence rate. It fails to explain the experiment phenomenon shown in this paper. In particular, the theoretical analysis does not reveal why SFL outperforms FL/SL under strong heterogeneity. \n\n3. Line 128: statements need to be improved. It is weird to say \"Based on x_s^{t,i}, based on which ... \"\n\n4. The assumptions need to be compared with existing works on federated learning.\n\nFor example, in Li et al. On the convergence of Fedavg on non-iid data, ICLR 2020, Assumption 3.3 is not required. Personally I also feel that the bound of heterogeneity by epsilon^2 may not be necessary. Hope that authors can explain more about why this assumption is necessary and how is the convergence rate affected if we remove this assumption.\n\n5. In eq.(10), gamma=8S/\\mu-1 need to be put later, after line 190. gamma is only used for the strongly convex case.\n\n6. In Theorem 3.5, I do not see lower bound on the learning rate. To ensure that (11) holds, I think that a lower bound of \\eta_t is needed. Merely requiring \\eta_t\\leq 1/(2Stau_max) may not be enough. Note that \\eta_t=0 also satisfies \\eta_t\\leq 1/(2S\\tau_{\\max}), then E[f(x_T)]-f^* will not converge to zero.\n\n7. I do not see any term including epsilon in strongly convex and general convex case. It is only included in the nonconvex case. Why the convergence is affected under nonconvex case but not convex case? Some intuitions are needed here.\n\n8. In Figure 4 and Figure 5, From the curves, the accuracy oscillates violently. Therefore I suggest to reduce the learning rate.\n\n9. List of inequalities after line 486. A^t is not defined. From the relationship above, one can only get\nA^{t,i}\\leq CA^{t,i-1}+B\\sum_{j=0}^{i-1} C^j.\n\n10. In line 520. First equality does not use (a+b)^2=a^2 +2ab+b^2. It just expands x_s^{t+1}. Perhaps the authors mean the second equality.\n\n11. Table 1 needs to be put in the main paper. The comparison is important.\n\n12. Line 828: Notation abuse of beta. In experiment section, it refers to the degree of heterogeneity.\nHere it refers to the fraction of full model accessible to clients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments! Our response to your comments is as follows. We use \"W*\" to tag responses to comments in the Weakness and \"Q*\" for responses to comments in the Questions.\n\n**1. Latency of SL and SFL (for W1)** \n\nSplit learning has higher latency than centralized learning due to the additional transmission of intermediate features and feature gradients between the two agents. In SFL, the communication between clients and the main server results in higher latency per training round compared to FL.\n\nWe compare the communication overhead and total model training time for FL, SL, and SFL in Table 3 of the appendix. \nThe total communication time of SFL is primarily determined by the size of model, the proportion of client-side model over the whole model, and the size of smashed layer. It is observed that SFL may induce higher latency when the data size and smashed layer size are large.\nWe will revise our paper to enhance its precision.\n\n**2. Theoretical results for SFL and FL comparison (for W2, Q3)**\n\nWe first explain the reason why SFL outperforms FL/SL under high data heterogeneity. Under high data heterogeneity, FL suffers from the notorious client drift issue, while SL suffers from the catastrophic forgetting problem. SFL (in particular SFL-V2) combines the training logic of FL (at clients) and SL (at server), which can help mitigate the issues of client drift and forgetting. Hence, SFL can lead to a better performance than both FL and SL. On the other hand, the theoretical comparison between SFL and FL/SL will serve as our future work. Through further characterizing the impact of cut layer on client drift and the degree of catastrophic forgetting in the theoretical analysis, we would be able to validate the aforementioned discussions and the theoretical improvement of SFL compared with FL/SL.\n\nThen, since our main focus is analyzing the convergence rate of SFL, we would like to summarize the observations that holds in both theoretical and empirical results. First, the performance of SFL degrades in data heterogeneity. In theoretical analysis, as $\\sigma_n^2$ or $\\epsilon_n^2$ increases, the convergence error increases. This is consistent with the empirical results that as $\\beta$ decreases, the accuracy of the trained model decreases. Second, from both the theoretical and empirical results, as the number of clients participating in each round (i.e., $q$) increases, the convergence error decreases. Third, from the empirical results, the accuracy of the trained model increases with the cut layer. This can be indirectly supported by our theoretical analysis. In particular, we empirically show that as the cut layer $L_c$ increases from 1 to 4, the maximum bound of the gradients across clients (i.e., $G^2$ defined in Eq. (7)) reduces (in particular, its values are 21.53, 19.86, 15.48, 12.81 for Lc = 1, 2, 3, 4, respectively). Based on the theoretical analysis, such a reduction of $G^2$ indicates the reduction of the convergence error and hence the improvement of the model accuracy.\n\n**3. Assumptions in different cases (for W4, W7)**\n\nThe Assumptions used under different cases for SFL-V1 and SFL-V2 are as follows:\n\n- $\\mu$-strongly convex: Assumptions 3.1 (smoothness) and 3.2 (Unbiased and bounded stochastic gradients with bounded variance);\n\n- General convex: Assumptions 3.1 (smoothness) and 3.2 (Unbiased and bounded stochastic gradients with bounded variance);\n\n- Non-convex:  Assumptions 3.1 (smoothness), 3.2 (6) and (7) (Unbiased and bounded stochastic gradients), and 3.3 (heterogeneity).\n\nThe above assumptions are widely applied in the literature. For instance, in the strongly convex case, the assumptions used in our paper are the same as those in Li et al. (2020). The heterogeneity assumption, captured by $\\epsilon^2$, is used only for the non-convex case, as in Jhunjhunwala et al. (2023) and Wang et al. (2023).\n\nThe difference in assumptions for strongly/general convex and non-convex cases is due to the form of convergence error we obtain. Specifically, we need to compute the model update in each local training with multiple iterations, denoted by $\\tilde{\\tau}$ and $\\tau$.\nFor strongly and general convex loss functions, we propose Lemma C.5, which directly provides the upper bound of the model parameter change over $\\tau$ iterations. However, for non-convex loss functions, we compute the relationships between the model updates and the global gradient of the loss functions, as described in Lemma C.6.\nThe upper bound of the squared norm of stochastic gradients, given by equation (7) in Assumption 3.2, and the upper bound of the divergence between local and global gradients, given by equation (8) in Assumption 3.3, are used to prove Lemmas C.5 and C.6, respectively.\n\n[Jhunjhunwala et al. 2023] D. Jhunjhunwala, S. Wang, and G. Joshi, “Fedexp: Speeding up federated averaging via extrapolation,”\nICLR, 2023.\n\n[Wang et al. 2023] S. Wang, J. Perazzone, M. Ji, and K. S. Chan, “Federated learning with flexible control,” IEEE Conference on Computer Communications, 2023.\n\n**4. Theoretical results on partial participation (for Q1)**\n\nOur paper considers a different partial participation mechanism compared to the ICLR paper, resulting in different convergence error bounds. Specifically, the ICLR paper assumes a constant number of participating clients in each training round, with clients randomly selected with equal probability. In contrast, our paper considers independent participation probabilities for each client, allowing for arbitrary and heterogeneous participation probabilities.\nTo prove the convergence error bound with clients' partial participation, we compute the difference between full and partial participation. In the future, we plan to improve these convergence bounds to make them tighter.\n\nDue to space limitations, we will add additional responses to your comments in Official Comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all of you for your time and comments on our paper. We have carefully revised our paper and prepared the responses. The added figures are in the attached file.

Meta-review: This paper studies the convergence of distributed split learning. 
A theoretical convergence analysis is presented and the performance of the method is empirically studied for various levels of heterogeneity.

The reviewers argued that the presentation of the results must be further improved for the final version:
- currently, too many details are delegated to the appendix. The main text should have a precise mathematical description of (at least one version of)  the algorithm.
- details on the numerical experiments are missing or not well explained. For example, the choice of the hyperparameters for the baseline methods (best practice would be to tune the parameters for each setting separately).

However, the reviewers concluded that the theoretical results presented in this paper are a step toward a better understanding of split learning and could inspire future research in this area.
Decision: Accept (poster)