Convergence Analysis of Split Federated Learning on Heterogeneous Data

Authors: pengchao  han
chao  huang
geng  tian
ming  tang
xin  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:15:44

Abstract: Split federated learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite the recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this paper aims to fill this gap. The analysis of SFL can be more challenging than that of federated learning (FL), due to the potential dual-paced updates at the clients and the main server. We provide convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data. The convergence rates are $O(1/T)$ and $O(1/\sqrt[3]{T})$, respectively, where $T$ denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and where some clients may be unavailable during training. Numerical experiments validate our theoretical results and show that SFL outperforms FL and split learning (SL) when data is highly heterogeneous across a large number of clients.

Main paper: 
 introduction 1.motivation: 
 Federated learning (FL) [18, 9] allows distributed clients to train a global machine learning model collaboratively without sharing raw data. FL leverages the parallel computing capabilities of clients to enhance model training efficiency. However, FL is usually computationally intensive. Clients need to train the entire global model multiple times, which can be infeasible for resource-constrained edge devices. This challenge is further exacerbated as the trend towards increasingly larger model architectures demands more substantial resources [1] . Moreover, FL suffers from the client drift Figure 1 : An illustration of SFL framework, and there are two major algorithms, i.e., SFL-V1 (left) and SFL-V2 (right) [27] . More discussions on SFL-V1 and SFL-V2 are given in Sec. 2.problem when clients' data distributions are heterogeneous, aka non-identically and independently distributed (non-IID). A large number of studies have proposed algorithms to address the client drift issue, e.g., [15, 10, 14, 25] .Split learning (SL) [28] is another distributed approach. By splitting the model across clients and a main server, SL can substantially reduce the computational workload on edge devices. Moreover, recent studies in [34, 17] show that SL can outperform FL when data is highly heterogeneous. However, SL's sequential training among clients can lead to high latency in each training round and potential performance loss (e.g., caused by catastrophic forgetting), which impedes its practical applicability in real-world distributed systems.In light of above challenges, Thapa et. al in [27] proposed split federated learning (SFL) as a hybrid approach that synergizes the strengths of both FL and SL. SFL combines parallel training of FL with partial model training of SL. They proposed two major SFL algorithms: SFL-V1 and SFL-V2. An illustration of these SFL algorithms are shown in Fig. 1 . Specifically, the global model (to be trained) is first split at a cut layer into two parts: a client-side model and a server-side model. Then, the clients are responsible for training only the client-side model under the coordination of a fed server (similar to FL). Another server, known as the main server, is tasked with training the server-side model by collaborating with the clients (similar to SL). SFL aims to leverage parallel processing to reduce latency, while benefiting from the reduced computational workloads and enhanced data heterogeneity handling of SL.Following [27] , there has been an emerging volume of empirical studies on SFL. e.g., [22, 21, 3, 23, 8, 31, 5] . However, a convergence analysis of SFL is missing in the literature, and this paper aims to provide a comprehensive convergence analysis under different conditions. Convergence theory is crucial for understanding the learning performance of SFL, particularly in the context of heterogeneous data and partial participation scenarios. In practical distributed systems, clients are prone to have different data distributions. Moreover, not all clients may be active or available at all times. These two issues can significantly affect the learning performance of SFL. We aim to provide convergence guarantees for SFL on heterogeneous data (under both full and partial participation). We further compare the results to FL and SL, which provides insights into the practical deployment of various distributed approaches.Convergence theories of FL and SL. There are many convergence results on FL. Most studies focus on data heterogeneity, e.g., [30, 16, 11, 10, 12] . Some studies look at partial participation, e.g., [35, 29, 26] . There are also convergence results on Mini-Batch SGD, e.g., [24, 33, 32] , where [33] argued that the key difference between FL and Mini-Batch SGD is the communication frequency.To our best knowledge, there is only one recent study [17] discussing the convergence of SL. The major difference to SL analysis lies in the sequential training manner across clients, while SFL clients perform parallel training.√ T ) for general convex objective, where T denotes the total number of rounds for SFL training. We further extend the analysis to non-convex objectives and more practical scenarios where some clients may be unavailable during training.• We conduct simulations on various datasets. We show that the results are consistent with our theories. We further show two surprising results: (i) SFL achieves a better performance when clients maintain a larger portion of the global model; (ii) SFL-V2 outperforms FL and SL when clients have highly heterogeneous data and the number of client is large.The rest of the paper is organized as follows. Sec. 2 formulates the SFL model. Sec. 3 presents the convergence results for SFL. We conduct experiments in Sec. 4 and conclude in Sec. 5.2 Problem Formulation The clients train models with the help of two servers: (i) fed server, which periodically aggregates clients' local models x c,n (similar to FL), and (ii) main server, who trains the server-side model x s . In this work, we consider two major SFL algorithms: SFL-V1 and SFL-V2 [27] . In SFL-V1, the main server maintains a separate server-side model x s,n corresponding to each client n. In comparison, in SFL-V2, the main server only maintains one model x s .Let F n (x; ζ n ) denote the loss of model x over client n's mini-batch instance ζ n , which is randomly sampled from client n's dataset D n . Let F n (x) ≜ E ζn∼Dn [F n (x; ζ n )] denote the expected loss of model x over client n's dataset. The goal of SFL is to minimize the expected loss of the model x over the datasets of all clients:EQUATIONwhere a n ∈ [0, 1] is the weight of client n satisfying n∈N a n = 1. Typically, a n = D n / n ′ ∈N D n ′ , where a client with a larger data size is assigned a larger weight [34] . 
algorithm description: 
 We provide a brief description of SFL. Refer to Appendix B for a more detailed discussion. SFL takes a total number of T rounds to solve (1) . At the beginning of each round t, clients download the recent global client-side model from the fed server, where the model is an aggregated version of the client-side models of the clients from the previous round t -1. Each round t contains two stages:Stage 1: model training. Clients and the main server train the full global model for τ iterations in each round. In each iteration i < τ , there are three steps:Step 1: client forward propagation. Each client n samples a mini-batch of data ζ t,i n from D n , computes the intermediate features (e.g., activation values at the cut layer) over its current model x t,i c,n , and sends the activation to the main server. The clients perform forward propagation in parallel.Step 2: main server training. Upon receiving the activation of each client n,• SFL-V1: the main server computes the loss using the current server-side model x t,i s,n . It then computes the gradients over x t,i s,n to update the model. It also computes the gradient over the activation at the cut layer, and sends it to client n.• SFL-V2: the main server computes the loss F n ( x t,i c,n , x t,i s ), based on which it then updates the server-side model x t,i s . It also computes and sends the gradient over activation at the cut layer to client n. Note that the main server sequentially interacts with the clients in a randomized order.Step 3: client backward propagation. Receiving gradient at the cut layer, each client n computes the client-side gradient using the chain rule, and then updates its model x t,i c,n . Stage 2: model aggregation. Model aggregation can occur for both client-side and server-side models. For the client side, after τ iterations of model training (i.e., at the end of round t), each client sends its current client-side model to the fed server. The fed server aggregates the clients' models (e.g., weighted averaging), which will be downloaded in the next round t + 1:EQUATIONFor the server side, (i) in SFL-V1, after τ iterations of training, the main server aggregates all serverside models. Note that τ does not necessarily need to equal τ , but when equality holds, SFL-V1 can be regarded as FL (despite the model splitting). (ii) In SFL-V2, no aggregation occurs since the main server only maintains one model.We consider two cases: (i) full participation where all clients are available during training. This can model the scenarios where clients are organizations or companies who likely have sufficient computation and communication resources [7] ; (ii) partial participation where some clients may be unavailable during training. This can model the cases where clients are edge devices (e.g., mobile phones) that are usually resource-constrained and may be disconnected from the SFL process.To model partial participation, we consider independent participation probabilities for each client, allowing for arbitrary and heterogeneous participation probabilities. Specifically, we use q n ∈ [0, 1] to denote client n's participation level (or probability), and q = (q n , n ∈ N ). If q n = 1, client n participates in every round of SFL with probability one. If q n < 1, client n is unavailable in some rounds. Denote P t (q) as the set of participating clients in round t. In the presence of partial participation, we need to modify (2) (and the potential server-side aggregation) to offset the incurred bias:x t+1 c ← n∈P t (q)a n q n x t,τ c,n .(3)We compare our derived bounds for SFL to other distributed approaches. For simplicity, we let a n = 1/N in (1) and σ n = σ for all n in (6) . The result are summarized in Table 2 . Since different convergence theories make slightly different assumptions, we clarify them below.In [33] ,σ 2 *is the variance of the stochastic gradient at the optimum:E ζn∼Dn ∥g n (x * , ζ n ) -∇F n (x * )∥ 2 ≤ σ 2 * .In [16] , Γ = f * -N n=1 F * n /N characterizes the client heterogeneity. In [17] , ϵ 2 * characterizes the client heterogeneity at the optimum, similar to [12] , i.e., 1 N N n=1 ||∇F n (x * )|| 2 = ϵ 2 * .σ 2 * µN τ T + S(f (x 0 )-f * ) µ exp -µT S FL [16] S µτ +T σ 2 +SΓN +N τ 2 G 2 µN + SI err [10] σ 2 µN τ T + Sσ 2 µ 2 τ T 2 + Sϵ 2 µ 2 T 2 + µI err exp -µT S SL [17] σ 2 µN τ T + Sσ 2 µ 2 N τ T 2 + Sϵ 2 * µ 2 N T 2 + µI err exp -µT S SFL SFL-V1 (Theorem 3.6) S µ(S+µT ) N (σ 2 + G 2 ) + µSI err SFL-V2 (Theorem 3.7) S µ(S+µT ) N 2 (σ 2 + G 2 ) + µSI errThe key observation is that our derived bounds match the other distributed approaches in the order of T and they all achieve O(1/T ).Furthermore, we will compare the convergence upper bounds of SFL to those of distributed SGD in [12] (with parameters p = 1 and ζ2 = 0) as follows:  
convergence analysis: 
 We first make technical assumptions in Sec. 3.1. Then, we present a key technical result in Sec. 3.2 to support the SFL convergence analysis. Finally, we provide the convergence results under full participation and partial participation in Sec. 3.3 and Sec. 3.4, respectively.We start with some conventional assumptions for convergence analysis in the FL literature.EQUATION)The smoothness assumption holds for many loss functions in, for example, logistic regression, softmax classifier, and l 2 -norm regularized linear regression [16] . Assumption 3.2. (Unbiased and bounded stochastic gradients with bounded variance) The stochastic gradientsEQUATIONEQUATIONAssumption 3.3. (Bounded gradients) The expected squared norm of stochastic gradients is bounded byEQUATION)The value of σ n measures the level of stochasticity. Assumption 3.4. (Heterogeneity) There exists an ϵ 2 such that the divergence between local and global gradients is bounded by ϵ 2 .EQUATIONA larger ϵ 2 indicates a larger degree of data heterogeneity.As discussed in Sec. 1.3, analyzing the performance bound of SFL can be more challenging than that of conventional FL counterparts due to the dual-paced model aggregation and model updates. To address this challenge, we decompose the convergence analysis into the server-side and client-side updates, respectively. We give the decomposition below. EQUATION)The proof is given in Appendix C.4. Proposition 3.5 is particularly useful. It shows that despite the challenging dual-paced updates, to bound the SFL performance gap, it suffices to separately bound the gap at the server-side and client-side models. Note that our decomposition can be easily applied to other distributed approaches such as SL. In addition, such a decomposition is not necessarily loose, as our derived bounds for SFL achieve the same order as in FL (see Appendix H.2 for details).Built upon Proposition 3.5, we first present the convergence results under full participation. For convenience, defineEQUATIONand let η t represent the learning rate at round t. Let f * denotes the optimal global loss, i. EQUATIONGeneral convex: Let Assumptions 3.1 -3.3 hold, andEQUATIONNon-convex: Let Assumptions 3.1, 3.2, and 3.4 hold, and EQUATIONEQUATION)General convex: Let Assumptions 3.1 -3.3 hold, andEQUATION)Non-convex: Let Assumptions 3.1, 3.2, and 3.4 hold, andEQUATION)1 Following many existing works in FL (e.g., [10] ), we consider E f (x T ) -f * and1 T T -1 t=0 η t E[∥∇xf (x t )∥ 2 ]as the performance metrics for (strongly) convex and non-convex objectives, respectively.Proofs of Theorems 3.6-3.7 are given in Appendices D-E, respectively. We summarize the key findings below.Convergence rate. The convergence bounds of both SFL-V1 and SFL-V2 achieve an order of O(1/T ) on strongly convex (and non-convex) objectives. For general convex objectives, the convergence rate becomes O(1/3 √ T ).2 Note that our bounds match the existing bounds for FL and SL (in terms of the order of T ) on heterogeneous data for strongly convex objectives. For a more detailed comparison, please refer to Appendix H.2. 3 Impact of data heterogeneity. The convergence bounds increase as the level of data heterogeneity increases. For example, in (13) , the bound increases in ϵ 2 (see Assumption 3.4) . This means that SFL tends to perform worse when clients' data are more heterogeneous, which is a commonly observed phenomenon in distributed learning, e.g., FL.Choice of learning rate. One should use a smaller learning rate when the number of local iteration τ increases. This bears a similar spirit to [16] . In addition, our results indicate that a proper choice of constant learning rate suffices for SFL convergence. It would be an interesting direction to investigate whether diminishing learning rates are able to achieve faster convergence.Comparison between SFL-V1 and SFL-V2. The convergence results between the two SFL versions are very similar, except that a 2 n (and a n ) in SFL-V1 are replaced by a 2 n + 1 (and a n + 1) in SFL-V2. See ( 11) and ( 14) for an inspection. We will show in Sec. 4 that SFL-V1 and SFL-V2 achieve similar accuracy (except under highly heterogeneous data).Now, we present the results under partial participation. EQUATIONGeneral convex: Let Assumptions 3.1 -3.3 hold, andEQUATIONNon-convex: Let Assumptions 3.1, 3.2, and 3.4 hold, andEQUATION)Theorem 3.9. ( SFL-V2: partial participation) µ-strongly convex: Let Assumptions 3.1 -3.3 hold, and η t = 4 µτ (γ+t) for client-side model and General convex: Let Assumptions 3.1 -3.3 hold, andEQUATIONEQUATIONEQUATION)Non-convex: Let Assumptions 3.1, 3.2, and 3.4 hold, andEQUATION)The proofs are given in Appendices F-G.Impact of partial participation. In practical cross-device settings, some clients may not participate in all rounds of training, i.e., q n < 1 for some n. This brings an additional term G 2 /q n to the convergence bound (e.g., see (12) and ( 18)), meaning that partial participation worsens SFL performance. This is also observed in FL literature (e.g., [29] ) and is consistent with our experimental results. 
setup: 
 We conduct experiments on CIFAR-10 and CIFAR-100 [13] . 4 To simulate data heterogeneity, we adopt the widely used Dirichlet distribution [6] with a controlling parameter β. Here, a smaller β corresponds to a higher level of data heterogeneity across clients. We use ResNet-18, which contains four blocks, as the model structure and consider four types of model splitting represented by L c = {1, 2, 3, 4}, where L c = n means the model is split after the n-th residual block. We consider two major distributed approaches as the benchmark, i.e., FL (in particular FedAvg [18] ) and SL [28] . The learning rates for SFL-V1, SFL-V2, FL, and SL are set as 0.01. The batchsize b s is 128, and we run experiments for T = 200 rounds. Unless stated otherwise, we use N = 10, β = 0.1, E = 5, where E is the number of local epochs for client-side model aggregation (i.e., every E times of training performed over each client's dataset, their client-side models are aggregated at the fed server), and hence τ = ⌈ Dn bs ⌉ × E. We set τ = τ for the fair comparison to vanilla FL. The experiments are run on a CPU (Intel(R) Xeon(R) Gold 5320 at 2.20GHz) and a GPU (A100-PCIE-80GB). Our codes are provided in https://github.com/TIANGeng708/ Convergence-Analysis-of-Split-Federated-Learning-on-Heterogeneous-Data.Impact of cut layer. We first investigate how the choice of the cut layer L c affects the SFL performance. The results are reported in Fig. 2 . We observe that for both SFL-V1 and SFL-V2, the performance increases in L c (i.e., clients have a larger proportion of the global model). This is associated with our empirical observation that the average client gradient variance gets smaller with L c . Intuitively, a smaller gradient variance implies a lower degree of the client drift issue, which leads to a better algorithm performance. 5 Based on this observation, we use L c = 4 for SFL (and SL) for the following experiments.Impact of data heterogeneity. We study the impact of data heterogeneity on SFL performance, where we use β ∈ {0.1, 0.5, 1, ∞}, and β = ∞ means clients have IID data. The results are reported in Fig. 3 . We observe that a higher level of data heterogeneity (i.e., a smaller β) leads to slower algorithm convergence and a lower accuracy for both SFL-V1 and SFL-V2. The observation is consistent with our convergence bound, e.g., in (16) , the performance bound increases in ϵ 2 . Note that the negative impact of heterogeneity is commonly observed in distributed learning literature including FL [7] and SL [21] .Impact of partial participation. We study the impact of client participation and let q n = q ∈ {0.2, 0.5, 1}, ∀n. The results are reported in Fig. 4 . We observe that a lower level of participation leads to less stable convergence and also a smaller accuracy. This is consistent with our convergence results, e.g., in (20) , the bound decreases in clients' participation level q n . Partial participation is expected in practical cross-device scenarios where clients are resourced-constrained edge devices. It is important to develop efficient algorithms as well as effective incentive mechanisms to encourage clients' participation in SFL.We now compare SFL to FL and SL. We consider different combinations of data heterogeneity β ∈ {0.1, 0.5} and cohort sizes N ∈ {10, 50, 100}. The results are reported in Fig. 5 . When data is mildly heterogeneous (i.e., β = 0.5), SFL and FL have similar convergence rates and accuracy performance. Note that SL seems to under-perform SFL and FL. We think this is mainly due to the catastrophic forgetting issue, which has been observed in [21, 2] .SFL outperforms FL and SL under highly heterogeneous data and a large client number.When data becomes more non-IID (i.e., β = 0.1), SFL-V2 tends to outperform FL and SL. The improvement becomes more significant as the cohort size gets larger. The bottleneck of FL is the client drift issue caused by data heterogeneity. The bottleneck of SL is associated with the catastrophic forgetting. SFL-V2 is a hybrid combination of FL and SL, which can lead to a better tradeoff between client drift and forgetting. By appropriately choosing the cut layer, SFL-V2 outperforms FL and SL. This observation also indicates that SFL-V2 can be a more appealing solution than FL for practical cross-device systems, as it achieves a better performance while requiring smaller computation overheads from edge devices. 
conclusion: 
 In this work, we provided the first comprehensive convergence analysis of SFL for strongly convex, general-convex, and non-convex objectives on heterogeneous data. One key challenge is the dualpaced model updates. We get around this issue by decomposing the performance gap of the global model into the client-side and server-side gaps. We further extend our analysis to the more practical scenario with partial client participation. Experimental experiments validate our theories and further show that SFL can outperform FL and SL under highly heterogeneous data and a large client number. One limitation of our work is that our bounds for SFL achieve the same order (in terms of training rounds) as in FL, yet the experiments showed that SFL outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL are to be derived, which is an important future work. For future work, one can apply our derived bounds to optimize SFL system performance, considering model accuracy, communication overhead, and computational workload of clients. It is also interesting to theoretically analyze how the choice of the cut layer affects the SFL performance. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We have used one CPU and one GPU. Specifically, the CPU is Intel(R) Xeon(R) Gold 5320 CPU with 2.20GHz, and the GPU is A100-PCIE-80GB. Memory: 256GB Storage: 10TB. Each experiment curve takes about 10 hours to train. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have conformed to all aspects of code of ethics with this submission. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The goal of this work is to provide a comprehensive analysis on the convergence of SFL. Our theoretical results have two potential impacts. First, we provide a thorough understanding on the performance of SFL, which potentially guides the implementation of SFL (e.g., the choice between FL and SFL, the choice of hyper-parameters and cut layers).Second, the convergence results can be used for modeling the training performance of SFL.Together with an effective modeling of communication and computation overheads of clients, researchers will be able to perform SFL system optimization. Due to the reduced clients' training loads in SFL, such a system optimization can potentially minimize the burden at clients (e.g., human mobile devices) while maintaining client privacy and satisfactory training performance.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks. We use open-access datasets and models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Our codes are based on the codes provided in [27] , which was cited in the main paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: We organize the entire appendix file as follows:In Sec. B, we provide detailed algorithmic descriptions.In Sec. C, we provide notations and some technical lemmas. For version 2, the client-side model is aggregated every τ iterations, while the M-server trains only one version of the M-server-side model.Input: τ, τ , T , and learning rate ηt Output: Global model x T = {x T c , x T s } 1 Initialize x 0 = {x 0 c , x 0 s }; 2 for i = 0, . . . , (T -1)Compute Fn(h(x t,i c,n ; ζ t,i n ), x t,i s ), g t,i s,n (x t,i s,n ; ζ t,i n ), and r t,i c,n (x t,i s,n ; ζ t,i n ); 9 Send r t,i c,n (x t,i s,n ; ζ t,i n ) to client n ∈ P t ; 10 x t,i+1 s,n ← x t,i s,n -ηtg t,i s,n (x t,i s,n ; ζ t,i n ); 11 Compute g t,i c,n (x t,i c,n ; ζ t,i n ); 12 x t,i+1 c,n ← x t,i c,n -ηtg t,i c,n (x t,i c,n ; ζ t,i n ); Phase 2: model aggregation.Input: τ, T , and learning rate ηt Output:Global model x T = {x T c , x T s } Initialize x 0 = {x 0 c , x 0 s }; for t = 0, . . . , T -1 doDetermine participating client set P t ⊆ N according to qn; Phase 1: model training.each client n ∈ P t :x t,0 c,n ← x t c ; for i = 0, . . . , τ -1 do 7 Sample a mini-batch ζ t,i n ; 8 Send h(x t,i c,n ; ζ t,i n ) to the M-server; 9 the M-server: 10 Compute Fn(h(x t,i c,n ; ζ t,i n ), x t,i s ), g t,i s,n (x t,i s ; ζ t,i n ), and r t,i c,n (x t,i s ; ζ t,i n ); 11 Send r t,i c,n (x t,i s ; ζ t,i n ) to client n ∈ P t ; 12 x t,i+1 s ← x t,i s -η t qn g t,i s,n (x t,i s ; ζ t,i n ); 13 Compute g t,i c,n (x t,i c,n ; ζ t,i n ); 14 x t,i+1 c,n ← x t,i c,n -ηtg t,i c,n (x t,i c,n ; ζ t,i n );the M-server:x t+1,0 s ← x t,τ s ; Phase 2: model aggregation. each client n ∈ P t : Send x t,τc,n to the F-server;the F-server:x t+1 c ← n∈P t an qn x t,τ c,n .Recall that the objective of SFL is given byEQUATIONWe define• x c and x s : global model parameter on the clients and server sides, respectively.• x c,n and x s,n : local forms of parameter on client n and on the main server corresponding to client n (in SFL-V1). • ∇F c,n (•) and ∇F s,n (•): the gradients of F n (•) over x c and x s , respectively. • g c,n (•) and g s,n (•): the stochastic gradients of F n (•) over x c and x s , respectively.For convenience, we omit the notation for mini-batch training data when referring to stochastic gradients.Further, we recall how SFL-V1 and SFL-V2 update models below.Let q n denote the participating probability of client n and define q := {q 1 , . . . , q N }. We denote I t n as a binary variable, taking 1 if client n participates in model training in round t, and 0 otherwise. I t n follows a Bernoulli distribution with an expectation of q n . Denote P t (q) as the set of participating clients in round t.Parameter update for SFL-V1:• Local training of client n: x t,0 c,n ← x t c , x t,i+1 c,n ← x t,i c,n -η t g t,i c,n x t,i c,n , x t+1 c,n ← x t,τ c,n ; • Client-side global aggregation:-Full participation:x t+1 c ← x t c -η t n∈N a n τ i=0 g t,i c,n x t,i c,n ; -Partial participation: x t+1 c ← x t c -η t n∈P t (q)an qn τ i=0 g t,i c,n x t,i c,n ; • M-server-side model update:-Full participation: x t+1 s ← x t s -η t n∈N a n τ -1 i=0 g t,i s,n x t,i s,n ; -Partial participation: x t+1 s ← x t s -η t n∈P t (q) an qn τ -1 i=0 g t,i s,n x t,i s,n .Parameter update for SFL-V2:• Local training of client n: x t,0 c,n ← x t c , x t,i+1 c,n ← x t,i c,n -η t g t,i c,n x t,i c,n , x t+1 c,n ← x t,τ c,n ; • Client-side global aggregation:-Full participation:x t+1 c ← x t c -η t n∈N a n τ i=0 g t,i c,n x t,i c,n ; -Partial participation: x t+1 c ← x t c -η t n∈P t (q)an qn τ i=0 g t,i c,n x t,i c,n ; • M-server-side model update:-Full participation: x t+1 s ← x t s -η t n∈N τ -1 i=0 g t,i s,n x t,i s,n ; -Partial participation: x t+1 s ← x t s -η t n∈P t (q) 1 qn τ -1 i=0 g t,i s,n x t,i s,n .We further recall the following assumptions for clients' loss functions in the proof. Assumption C.1. For each client n ∈ N :• The loss F n (•) is S-smooth:EQUATIONEQUATION• The stochastic gradients of F n (•) are unbiased with the variance bounded by σ 2 n :EQUATIONEQUATION• The expected squared norm of stochastic gradients is bounded by G 2 :EQUATION• (Bounded gradient divergence) There exists a constant ϵ > 0, such that the divergence between local and global gradients is bounded by ϵ 2 :EQUATION)Assumption C.2. For each client n ∈ N :• The loss F n (•) is µ-strongly convex for some µ ≥ 0:EQUATIONHere, we allow that µ = 0, referring to this case of the general convex.Lemma C.3. [Lemma 5 in [10] ] The following holds for any S-smooth and µ-strongly convex function h, and any x, y, z in the domain of h:EQUATION)Proof of Proposition 3.5 EQUATION)Proof. Since F n 's are S-smooth, it is easy to show that the global loss function f (•) is also S-smooth. Thus, we haveEQUATION) Since x T ≜ [x T c ; x T s ], and x * ≜ [x * c ; x * s ],we haveEQUATION)Substituting ( 34) into (33), we complete the proof.Proposition C.4 (Decomposition in each round). Under Assumption C.1, we haveEQUATION)Proof. The proposition can be easily proved by the S-smoothness of f (•).Lemma C. EQUATION)Proof. Similar to Lemma 3 in [20] , we haveEQUATION)where we use Assumption C.1,(X + Y ) 2 ≤ (1 + a) X 2 + 1 + 1 a Y 2for some positive a, andη t ≤ 1 √ 6Sτ . Let A t,i := E x t,i n -x t 2 B := 6τ η t 2 2σ 2 n + G 2 C := 1 + 2 τWe haveEQUATION)We can show thatA t,1 ≤ CA t + B A t,2 ≤ CA t,1 + B ≤ C 2 A t + CB + B A t,3 ≤ CA t,2 + B ≤ C 3 A t + C 2 B + CB + B . . . A t,i ≤ C i A t + B i-1 j=0 C j Note that A t := A t,0 = E ∥x t -x t ∥ 2 = 0.Accumulate the above for τ iterations, we haveEQUATION)The first inequality is due toN -1 i=0 x i = x N -1X-1 and the third line results from (1 + n x )x ≤ e n . Thus, we finish the proof. EQUATIONProof.E x t,i n -x t 2 ≤ E x t,i-1 n -η t g t,i-1 n -x t 2 ≤ E x t,i-1 n -x t -η t g t,i-1 n -∇ x F n x t,i-1 n +∇ x F n x t,i-1 n -∇ x F n x t + ∇ x F n x t -∇ x f x t + ∇ x f x t 2 ≤ 1 + 1 τ E x t,i-1 n -x t 2 + 8τ E η t g t,i-1 n -∇ x F n x t,i-1 n 2 + 8τ E η t ∇ x F n x t,i-1 n -∇ x F n x t 2 + 8τ E η t ∇ x F n x t -∇ x f x t 2 + 8τ η t ∇ x f x t 2 ≤ 1 + 1 τ E x t,i-1 n -x t 2 + 8τ η t 2 σ 2 n + 8τ η t 2 S 2 E x t,i-1 n -x t 2 + 8τ η t 2 ϵ 2 + 8τ η t 2 ∇ x f x t 2 ≤ 1+ 1 τ +8τ η t 2 S 2 E x t,i-1 n -x t 2 +8τ η t 2 σ 2 n +8τ η t 2 ϵ 2 +8τ η t 2 ∇ x f x t 2 ≤ 1 + 2 τ E x t,i-1 n -x t 2 + 8τ η t 2 σ 2 n + 8τ η t 2 ϵ 2 + 8τ η t 2 ∇ x f x t 2 (43)where we have applied Assumption C.1,(X + Y ) 2 ≤ (1 + a) X 2 + 1 + 1 a Y 2for some positive a, andη t ≤ 1 √ 8Sτ . Let A t,i := E x t,i n -x t 2 B := 8τ η t 2 σ 2 n + 8τ η t 2 ϵ 2 + 8τ η t 2 ∇ x f x t 2 C := 1 + 2 τWe haveEQUATION)We can show thatA t,i ≤ C i A t + B i-1 j=0 C j Note that A t = E ∥x t -x t ∥ 2 = 0.Accumulate the above for τ iterations, we haveτ -1 i=0 E x t,i n -x t 2 = τ -1 i=0 B i-1 j=0 C j ≤ 2τ 2 B ≤ 2τ 2 8τ η t 2 σ 2 n + 8τ η t 2 ϵ 2 + 8τ η t 2 ∇ x f x t 2 (45)where we use N -1 i=0 x i = x N -1 X-1 and (1 + n x ) x ≤ e n .EQUATIONProof.EQUATION)We define the following notation for simplicity:A t,i := E g t,i n -g t n 2(48)B := 2 (1 + τ ) η t 2 S 2 E g t n 2(49)C := 1 + 2 τ (50)We haveEQUATIONWe can show thatA t,i ≤ C i A t + B i-1 j=0 C j Note that A t = E ∥g t n -g t n ∥ 2 = 0.For the second part, we haveEQUATIOND Proof for Theorem 3.6We organize the proof of Theorem 3.6 as follows:• In Sec. D.1, we prove the strongly convex case. EQUATION)We prove Lemma D.1 as follows.Proof. We use x t,i s,n as the M-server-side model when the M-server interacts with client n for the i-th iteration of model training at round t. Using the (sequential) gradient update rule ofx t+1 s = x t s -η t N n=1 τ -1 i=0 a n g t,i s,n x t,i c,n , x t,i s,n, we haveEQUATION)where the second equality is from (a + b) 2 = a 2 + 2ab + b 2 and the last inequality is due toE ∇ xs f x t,i c , x t,i s -g t,i s = 0.The first part in ( 54) isEQUATIONwhere we use∇ xs f ({x c , x s }) = N n=1 a n ∇ xs F n ({x c , x s }).For (55), we haveEQUATION)where the first inequality applies triangle inequality. In the last inequality, we apply the bound of variance and expected squared norm for stochastic gradients in Assumption C.1.Since F n (x) is S-smooth and µ-strongly convex, using Lemma C.3 we haveEQUATION)By Lemma C.5, we haveEQUATION)From Assumption C.1, the second part in (54) is bounded byEQUATION)Thus, byEQUATION)We now prove the convergence error. Let∆ t+1 ≜ E x t+1 s -x * s 2 .We can rewrite (60) as:EQUATIONwhereB 1 := 4N N n=1 a 2 n 2σ 2 n + G 2 and B := 192S N n=1 a n 2σ 2 n + G 2 .Consider a diminishing stepsize η t = 2β τ (γ+t) , i.e, η t τ 2 = β γ+t , where β = 2 µ , γ = 8S µ -1. It is easy to show that η t ≤ 1 2S τ for all t. Next, we will prove that ∆ t+1 ≤ v γ+t+1 , where v = max 4B1 µ 2 + 8B2 µ 3 (γ+1) , (γ + 1)∆ 0 . We prove this by induction. First, the definition of v ensures that it holds for t = -1. Assume the conclusion holds for some t, it follows that∆ t+1 ≤ 1 - η t τ µ 2 ∆ t + (η t ) 2 (τ ) 2 4 B 1 + (η t ) 3 (τ ) 3 8 B 2 (62) ≤ 1 - µβ γ + t v γ + t + (η t ) 2 (τ ) 2 4 B 1 + (η t ) 3 (τ ) 3 8 B 2 (63) = γ + t -1 (γ + t) 2 v + β 2 B 1 (γ + t) 2 + β 3 B 2 (γ + t) 3 - βµ -1 (γ + t) 2 v (64) = γ + t -1 (γ + t) 2 v + β 2 B 1 (γ + t) 2 + β 3 B 2 (γ + t) 3 - βµ -1 (γ + t) 2 max 4B 1 µ 2 + 8B 2 µ 3 (γ + 1) , (γ + 1)∆ 0 (65) = γ + t -1 (γ + t) 2 v+ β 2 B 1 (γ + t) 2 + β 3 B 2 (γ + t) 3 - βµ -1 (γ + t) 2 max β 2 B 1 βµ -1 + β 3 B 2 (βµ -1)(γ + 1), (γ + 1)∆ 0 (66)EQUATIONHence, we have proven that ∆ t ≤ v γ+t , ∀t. Therefore, we have EQUATIONEQUATION) Let ∆ t+1 ≜ E x t+1 c -x * c 2 .We can rewrite (70) as:EQUATIONwhereB 1 := 4N N n=1 a 2 n 2σ 2 n + G 2 and B 2 := 192S N n=1 a n 2σ 2 n + G 2 .Consider a diminishing stepsize η t = 2β τ (γ+t) , i.e, η t τ 2 = β γ+t , where β = 2 µ , γ = 8S µ -1. It is easy to show that η t ≤ 1 2Sτ for all t. For v = max 4B1 µ 2 + 8B2 µ 3 (γ+1) , (γ + 1)∆ 0 , we can prove that ∆ t ≤ v γ+t , ∀t. Therefore, we haveEQUATIONWe merge the M-server-side and client-side models in (69) and (72) using Proposition 3.5 by settingEQUATIOND.2 General convex case for SFL-V1 D.2.1 One-round Parallel Update for M-Server-Side Model By Lemma D.1 with µ = 0 and η t ≤ 1 2S τ , we haveEQUATIONBy Lemma D.1 with µ = 0 and η t ≤ 1 2Sτ , we haveEQUATIONWe merge the M-server-side and client-side models in ( 74) and (75) as followsEQUATION)Then, we can obtain the relation between E x t+1 -x * 2 and E ∥x t -x * ∥ 2 , which is related toE [f (x t ) -f (x * )].Applying Lemma 8 in [17] and let τ min := min {τ , τ } and η t ≤ 1 2S max{τ,τ } , we obtain the performance bound asE f x T -f (x * ) ≤ 1 2 τ 2 + τ 2 τ 2 min N N n=1 a 2 n 2σ 2 n + G 2 1 2x 0 -x * 2 T + 11 2 + 1 2 τ 2 + τ 2 τ 2 min 24S N n=1 a n 2σ 2 n + G 2 1 3x 0 -x * 2 T + 1 For the server, we haveEQUATIONEQUATIONwhere we apply Assumption C.1,EQUATION)Thus, (78) becomesE ∇ xs f x t , x t+1 s -x t s ≤ - η t τ 2 ∇ xs f x t 2 + N η t S 2 2 N n=1 a 2 n 2τ 2 8τ η t 2 σ 2 n +8τ η t 2 ϵ 2 +8τ η t 2 ∇ xs f x t s 2 ≤ - η t τ 2 + 8N η t 3 τ 3 S 2 N n=1 a 2 n ∇ xs f x t 2 + 8N η t S 2 τ 3 N n=1 a 2 n η t 2 σ 2 n + ϵ 2 .(80) Furthermore, we haveEQUATIONwhere the last line uses Assumption C.1 andE ∥z∥ 2 = ∥E[z]∥ 2 + E[∥z -E[z]∥ 2 for any random variable z.By Lemma C.7 with η t ≤ 1 2S τ , we haveEQUATIONThus, (81) becomesEQUATIONThe analysis of the client-side model update is similar to the server. Thus, we haveEQUATIONEQUATIONApplying (80), ( 83), ( 85) and ( 84) into (36) in Proposition C.4 and define τ min ≜ min {τ, τ }, τ max ≜ max {τ, τ }, we haveEQUATION)where we first let η t ≤ 1 16Sτmax and then letη t ≤ 1 8SN τ 2 max τ min N n=1 a 2 n. We also use∥∇ x f (x t )∥ 2 = ∥∇ xc f (x t )∥ 2 + ∥∇ xs f (x t )∥ 2 .Rearranging the above we haveEQUATION)Taking expectation and averaging over all t, we haveEQUATION)E Proof of Theorem 3.7• In Sec. E.1, we prove the strongly convex case.• In Sec. E.2, we prove the general convex case. EQUATION)We prove Lemma E.1 as follows.Proof. We use x t,i s,n as the M-server-side model when the M-server interacts with client n for the i-th iteration of model training at round t. Using the (sequential) gradient update rule ofx t+1 s = x t s -η t N n=1 τ -1 i=0 g t,i s x t,i c,n , x t,i s,n, we haveEQUATION)where the first equality is from (a + b) 2 = a 2 + 2ab + b 2 and the last inequality is due toE ∇ xs f x t,i c , x t,i s -g t,i s = 0.The first part in ( 91) isEQUATIONwhere we use∇ xs f ({x c , x s }) = N n=1 ∇ xs F n ({x c , x s }).For (92), we haveEQUATION)where the first inequality applies triangle inequality. In the last inequality, we apply the bound of variance and expected squared norm for stochastic gradients in Assumption C.1.Since F n (x) is S-smooth and µ-strongly convex, using Lemma C.3 we haveEQUATION)By Lemma C.5, we haveEQUATIONFrom Assumption C.1, the second part in (91) is bounded byEQUATION)Thus, (91) becomesEQUATION)Using the above lemma, we can prove the convergence error. Let∆ t+1 ≜ E x t+1 s -x * s 2 .We can rewrite (97) as:EQUATION)whereB 1 := 4N N n=1 2σ 2 n + G 2 and B := 192S N n=1 2σ 2 n + G 2 .Consider a diminishing stepsize η t = 2β N τ (γs+t) , i.e, N η t τ 2 = β γs+t , where β = 2 µ , γ s = 8S N µ -1. It is easy to show that η t ≤ 1 2Sτ for all t. Next, we will prove that ∆ t+1 ≤ v γs+t+1 , where v = max 4B1 µ 2 + 8B2 µ 3 (γs+1) , (γ s + 1)∆ 0 . We prove this by induction. First, the definition of v ensures that it holds for t = -1. Assume the conclusion holds for some t, it follows thatEQUATIONHence, we have proven that ∆ t ≤ v γs+t , ∀t. Therefore, we have EQUATIONEQUATION) Let ∆ t+1 ≜ E x t+1 c -x * c 2 .We can rewrite (107) as:EQUATION)whereB 1 := 4N N n=1 a 2 n 2σ 2 n + G 2 and B 2 := 192S N n=1 a n 2σ 2 n + G 2 .Consider a diminishing stepsizeη t = 2β τ (γc+t) , i.e, η t τ 2 = β γc+t , where β = 2 µ , γ c = 8S µ -1.It is easy to show that η t ≤ 1 2Sτ for all t. For v = max 4B1 µ 2 + 8B2 µ 3 (γc+1) , (γ c + 1)∆ 0 , we can prove that ∆ t ≤ v γc+t , ∀t. Therefore, we haveEQUATIONWe merge the M-server-side and client-side models in (106) and (109) using Proposition 3.5. For η t ≤ 1 2Sτ and γ = 8S µ -1, we haveEQUATIONE.2 General convex case for SFL-V2By Lemma E.1 with µ = 0 and η t ≤ 1 2Sτ , we haveEQUATIONBy Lemma D.1 with µ = 0 and η t ≤ 1 2Sτ , we haveEQUATIONWe merge the M-server-side and client-side models in ( 111) and (112) as followsEQUATION)Then, we can obtain the relation between E x t+1 -x * 2 and E ∥x t -x * ∥ 2 , which is related toE [f (x t ) -f (x * )].Applying Lemma 8 in [17] , we obtain the performance bound asE f x T -f (x * ) ≤ 1 2 N N n=1 (a 2 n + 1) 2σ 2 n + G 2 1 2 x 0 -x * 2 T + 1 1 2 + 1 2 24S N n=1 (a n + 1) 2σ 2 n + G 2 1 3x 0 -x * 2 T + 1EQUATIONE.3 Non-convex case for SFL-V2For the server, we haveEQUATIONwhere we apply Assumption C.1,EQUATION)Thus, (115) becomesEQUATION)Furthermore, we haveEQUATIONwhere the last line uses Assumption C.1 andE ∥z∥ 2 = ∥E[z]∥ 2 + E[∥z -E[z]∥ 2 for any random variable z.By Lemma C.7 with η t ≤ 1 2Sτ , we haveEQUATIONThus, (118) becomesEQUATIONThe analysis of the client-side model update is the same as the client's model update in version 1. Thus, we haveEQUATIONEQUATIONApplying (117), ( 120), ( 122) and ( 121) into (36) in Proposition C.4, we haveEQUATION)where we first let η t ≤ 1 16Sτ and then let η t ≤ 1 8SN 2 τ . We have applied N n=1 a 2 n ≤ N . Rearranging the above we haveEQUATION)Taking expectation and averaging over all t, we haveEQUATION)F Proof of Theorem 3.8• In Sec. F.1, we prove the strongly convex case.• In Sec. F.2, we prove the general convex case.• In Sec. F.3, we prove the non-convex case. We first bound the M-server-side model update in one round for full participation (q n = 1 for all n), and then compute the difference between full participation and partial participation (q n < 1 for some n). We denote I t n as a binary variable, taking 1 if client n participates in model training in round t, and 0 otherwise. Practically, I t n follows a Bernoulli distribution with an expectation of q n . For full participation, Lemma D.1 givesEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATIONwhere we useE ∥X -EX∥ 2 ≤ E ∥X∥ 2 , E [I t n ] = q n , and x t+1 s = x t s - η t n∈P t (q) τ -1 i=0 a 2 n qn g t,i s,n x t,i c,n , x t,i s,n .Combining the above givesEQUATION) Let ∆ t+1 ≜ E x t+1 s -x * s 2 .We can rewrite (128) as:EQUATION)whereB 1 := 4N N n=1 a 2 n 2σ 2 n + G 2 + 4N G 2 N n=1 a 2 n qn and B := 192S N n=1 a n 2σ 2 n + G 2 .Consider a diminishing stepsize η t = 2β τ (γs+t) , i.e, η t τ 2 = β γs+t , where β = 2 µ , γ s = 8S µ -1. It is easy to show that η t ≤ 1 2S τ for all t. We can prove that ∆ t ≤ v γs+t , ∀t. Therefore, we have EQUATIONEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATION)where we useE ∥X -EX∥ 2 ≤ E ∥X∥ 2 , E [I t n ] = q n , and x t+1 c = x t c - η t n∈P t (q) τ -1 i=0 an qn g t,i c,n x t,i c,n , x t,i s,n.We obtain the client-side model parameter update in one round for partial participation by combining the two terms and we haveEQUATION)where we considerE [f (x t ) -f (x * )] ≥ 0. Let ∆ t+1 ≜ E x t+1 c -x * c 2 .We can rewrite (163) as:EQUATION)whereB 1 := 4N N n=1 a 2 n 2σ 2 n + G 2 + 4N G 2 N n=1 a 2 n qn and B 2 := 192S N n=1 a n 2σ 2 n + G 2 .Consider a diminishing stepsizeη t = 2β τ (γc+t) , i.e, η t τ 2 = β γc+t , where β = 2 µ , γ c = 8S µ -1. It is easy to show that η t ≤ 1 2Sτ for all t. For v = max 4B1 µ 2 + 8B2 µ 3 (γc+1), (γ c + 1)∆ 0 , we can prove that ∆ t ≤ v γc+t , ∀t. Therefore, we haveEQUATIONWe merge the M-server-side and client-side models in ( 130) and (135) using Proposition 3.5. ForEQUATIONF.2 General convex case for SFL-V1 F.2.1 One-round Parallel Update for M-Server-Side Model By Lemma D.1 with µ = 0 and η t ≤ 1 2S τ , we haveEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATION)Thus, we haveEQUATIONBy Lemma D.1 with µ = 0 and η t ≤ 1 2Sτ , we haveEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATION)Thus, we haveEQUATIONWe merge the M-server-side and client-side models in (139) and (142) as followsEQUATIONThen, we can obtain the relation betweenE x t+1 -x * 2 and E ∥x t -x * ∥ 2 , which is related to E [f (x t ) -f (x * )].Applying Lemma 8 in [17] and let τ min := min {τ , τ }, we obtain the performance bound asE f x T -f (x * ) ≤ 1 2 τ 2 + τ 2 τ 2 min N N n=1 a 2 n 2σ 2 n + G 2 + G 2 n q n 1 2 x 0 -x * 2 T + 1 1 2 + 1 2 τ 2 + τ 2 τ 2 min 24S N n=1 a n 2σ 2 n + G 2 1 3x 0 -x * 2 T + 1EQUATIONF.3 Non-convex case for SFL-V1 F.3.1 One-round Parallel Update for M-Server-Side ModelFor the server, we haveEQUATIONwhere we apply Assumption C.1,∇ xs f (x t ) = N n=1 a n ∇ xs F n (x t ), ⟨a, b⟩ ≤ a 2 +b 2 2, andEQUATION)Thus, (145) becomesEQUATIONFurthermore, we haveEQUATIONwhere the last line uses Assumption C.1 andE ∥z∥ 2 = ∥E[z]∥ 2 + E[∥z -E[z]∥ 2 for any random variable z.By Lemma C.7 with η t ≤ 1 2S τ , we haveEQUATIONThus, (148) becomesEQUATIONThe analysis of the client-side model update is similar to the server. Thus, we haveEQUATIONEQUATIONApplying (147), ( 150), ( 152) and ( 151) into (36) in Proposition C.4 and define τ min ≜ min {τ, τ }, τ max ≜ max {τ, τ } we haveEQUATIONwhere we first let η t ≤ 1 16Sτmax and then letη t ≤ 1 8SN τ 2 max τ min N n=1 a 2 n qn. We also use∥∇ x f (x t )∥ 2 = ∥∇ xc f (x t )∥ 2 + ∥∇ xs f (x t )∥ 2 .Rearranging the above we haveEQUATIONTaking expectation and averaging over all t, we haveEQUATIONG Proof of Theorem 3.9• In Sec. G.1, we prove the strongly convex case.• In Sec. G.2, we prove the general convex case.• In Sec. G.3, we prove the non-convex case.G.1 Strongly convex case for SFL-V2 G.1.1 One-round Sequential Update for M-Server-Side ModelWe first bound the M-server-side model update in one round for full participation (q n = 1 for all n), and then compute the difference between full participation and partial participation (q n < 1 for some n). We denote I t n as a binary variable, taking 1 if client n participates in model training in round t, and 0 otherwise.For full participation, Lemma E.1 givesEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATIONwhere we useE ∥X -EX∥ 2 ≤ E ∥X∥ 2 , E [I t n ] = q n , and x t+1 s = x t s - η t n∈P t (q) τ -1 i=0 1 qn g t,i s,n x t,i c,n , x t,i s,n.Combining the above givesEQUATION) Let ∆ t+1 ≜ E x t+1 s -x * s 2 .We can rewrite (158) as:EQUATION)whereB 1 := 4N N n=1 2σ 2 n + G 2 + 4N G 2 N n=1 1 qn and B := 192S N n=1 2σ 2 n + G 2 .Consider a diminishing stepsizeη t = 2β N τ (γs+t) , i.e, N η t τ 2 = β γs+t , where β = 2 µ , γ s = 8S N µ -1.It is easy to show that η t ≤ 1 2Sτ for all t. We can prove that ∆ t ≤ v γs+t , ∀t. Therefore, we have EQUATIONEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATIONwhere we useE ∥X -EX∥ 2 ≤ E ∥X∥ 2 , E [I t n ] = q n , and x t+1 c = x t c - η t n∈P t (q) τ -1 i=0 an qn g t,i c,n x t,i c,n , x t,i s,n.We obtain the client-side model parameter update in one round for partial participation by combining the two terms and we haveEQUATION) Let ∆ t+1 ≜ E x t+1 c -x * c 2 .We can rewrite (163) as:EQUATION)whereB 1 := 4N N n=1 a 2 n 2σ 2 n + G 2 + 4N G 2 N n=1 a 2 n qn and B 2 := 192S N n=1 a n 2σ 2 n + G 2 .Consider a diminishing stepsizeη t = 2β τ (γc+t) , i.e, η t τ 2 = β γc+t , where β = 2 µ , γ c = 8S µ -1.It is easy to show that η t ≤ 1 2Sτ for all t. For v = max 4B1 µ 2 + 8B2 µ 3 (γc+1) , (γ c + 1)∆ 0 , we can prove that ∆ t ≤ v γc+t , ∀t. Therefore, we haveEQUATIONWe merge the M-server-side and client-side models in (160) and (165) using Proposition 3.5. For η t ≤ 1 2Sτ and γ = 8S µ -1, we haveEQUATIONG.2 General convex case for version 2 G.2.1 One-round Sequential Update for M-Server-Side ModelBy Lemma E.1 with µ = 0 and η t ≤ 1 2Sτ , we haveEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATIONThus, we haveEQUATIONG.2.2 One-round Parallel Update for Client-Side Models By Lemma D.1 with µ = 0 and η t ≤ 1 2Sτ , we haveEQUATION)Considering that each client n participates in model training with a probability q n , we haveEQUATIONThus, we haveEQUATIONWe merge the M-server-side and client-side models in (169) and (172) as followsEQUATIONThen, we can obtain the relation between E x t+1 -x * 2 and E ∥x t -x * ∥ 2 , which is related toE [f (x t ) -f (x * )].Applying Lemma 8 in [17] , we obtain the performance bound asE f x T -f (x * ) ≤ 1 2 N N n=1 (a 2 n + 1) 2σ 2 n + G 2 + G 2 n q n 1 2x 0 -x * 2 T + 11 2 + 1 2 24S N n=1 (a n + 1) 2σ 2 n + G 2 1 3x 0 -x * 2 T + 1EQUATIONG.3 Non-convex case for version 2 G.3.1 One-round Sequential Update for M-Server-Side ModelFor the server, we haveEQUATIONwhere we apply Assumption C.1,∇ xs f (x t ) = N n=1 ∇ xs F n (x t ), ⟨a, b⟩ ≤ a 2 +b 2 2, andEQUATIONThus, (175) becomesE ∇ xs f x t , x t+1 s -x t s ≤ - η t τ 2 ∇ xs f x t 2 + N η t S 2 2 N n=1 1 q n 2τ 2 8τ η t 2 σ 2 n +8τ η t 2 ϵ 2 +8τ η t 2 ∇ xs f x t s 2 ≤ - η t τ 2 + 8N η t 3 τ 3 S 2 N n=1 1 q n ∇ xs f x t 2 + 8N η t S 2 τ 3 N n=1 1 q n η t 2 σ 2 n + ϵ 2 .(177) Furthermore, we haveEQUATIONwhere the last line uses Assumption C.1 andEQUATIONThus, (178) becomesEQUATIONThe analysis of the client-side model update is the same as the client's model update in version 1. Thus, we haveEQUATIONEQUATIONApplying (177), ( 180), ( 182) and ( 181) into (36) in Proposition C.4, we haveE f x t+1 -f x t ≤ E ∇ xc f x t ,x t+1 c -x t c + S 2 E x t+1 c -x t c 2 +E ∇ xs f x t , x t+1 s -x t s + S 2 E x t+1 s -x t s 2 ≤ - η t τ 2 + 8N η t 3 τ 3 S 2 N n=1 1 q n ∇ x f x t 2 + 8N η t S 2 τ 3 N n=1 a 2 n + 1 q n η t 2 σ 2 n + ϵ 2 + SN (η t ) 2 τ 2 N n=1 1 q n τ + 8τ 3 η t 2 S 2 2 ∇ x f x t 2 + SN (η t ) 2 τ 2 N n=1 a 2 n + 1 q n τ + 8τ 3 η t 2 S 2 2ϵ 2 + σ 2 n ≤ - η t τ 2 + 8N η t 3 S 2 τ 3 N n=1 1 q n + SN η t 2 τ N n=1 1 q n τ + 8τ 3 η t 2 S 2 ∇ x f x t 2 + 8N η t S 2 τ 3 N n=1 a 2 n + 1 q n η t 2 σ 2 n + ϵ 2 + 1 2 SN η t 2 τ τ + 8τ 3 η t 2 S 2 N n=1 a 2 n + 1 q n 2ϵ 2 + σ 2 n ≤ - η t τ 2 + SN η t 2 τ 2 N n=1 1 q n + 8N η t 3 S 2 τ 3 N n=1 1 q n + 8S 3 N η t 4 τ 4 N n=1 1 q n ∇ x f x t 2 + 8N η t 3 S 2 τ 3 N n=1 a 2 n + 1 q n σ 2 n + 8N η t 3 S 2 τ 3 ϵ 2 N n=1 a 2 n + 1 q n + SN η t 2 τ 2 ϵ 2 N n=1 a 2 n + 1 q n + 1 2 SN η t 2 τ 2 N n=1 a 2 n + 1 q n σ 2 n + 8N S 3 η t 4 τ 4 ϵ 2 N n=1 a 2 n + 1 q n + 4N S 3 η t 4 τ 4 N n=1 a 2 n + 1 q n σ 2 n ≤ - η t τ 2 1 -2SN η t τ 2 τ N n=1 1 q n 1 + 8Sη t τ + 8S 2 η t 2 τ 2 ∇ x f x t 2 + 1 2 N S η t 2 τ 2 + 8N η t 3 S 2 τ 3 + 4N S 3 η t 4 τ 4 N n=1 a 2 n + 1 q n σ 2 n + N S η t 2 τ 2 + 8N η t 3 S 2 τ 3 + 8N SL 3 η t 4 τ 4 N n=1 a 2 n + 1 q n ϵ 2We conclude the convergence results in our paper in Table 1 . (N (σ 2 + G 2 ) + µSI err ) SFL-V2 S µ(S+µT ) (N 2 (σ 2 + G 2 ) + µSI err ) General convex SFL-V1 ( N (σ 2 +G 2 ) T ) 1 2 + ( N (σ 2 +G 2 ) T ) 1 3 + S T I err SFL-V2 ( N 2 (σ 2 +G 2 ) T ) 1 2 + ( N 2 (σ 2 +G 2 ) T ) 1 3 + S T I err Non-convex SFL-V1 N S(σ 2 +ϵ 2 ) T + F err T SFL-V2 N 2 S(σ 2 +ϵ 2 ) T + F err T Partial participation Strongly convex SFL-V1 S µ(S+µT ) (N (σ 2 + G 2 (1 + Q)) + µSI err ) SFL-V2 S µ(S+µT ) (N 2 (σ 2 + G 2 (1 + Q)) + µSI err ) General convex SFL-V1 ( N (σ 2 +G 2 (1+Q)) T ) 1 2 + ( N (σ 2 +G 2 ) T ) 1 3 + S T I err SFL-V2 ( N 2 (σ 2 +G 2 (1+Q)) T ) 1 2 + ( N 2 (σ 2 +G 2 ) T ) 1 3 + S T I err Non-convex SFL-V1 N S(σ 2 +ϵ 2 )Q T + F err T SFL-V2 N 2 S(σ 2 +ϵ 2 )Q T + F err T H.Distributed SGDσ 2 µN T + SI err exp -µT S SFL-V1 S µ(S+µT ) (N (σ 2 + G 2 ) + µSI err ) SFL-V2 S µ(S+µT ) (N 2 (σ 2 + G 2 ) + µSI err ) General convex Distributed SGD ( σ 2 N T 2 + S T )I err SFL-V1 ( N (σ 2 +G 2 ) T ) 1 2 + ( N (σ 2 +G 2 ) T ) 1 3 + S T I err SFL-V2 ( N 2 (σ 2 +G 2 ) T ) 1 2 + ( N 2 (σ 2 +G 2 ) T ) 1 3 + S T I err Non-convex Distributed SGD ( σ 2 N T 2 + 1 T )SF err SFL-V1 N S(σ 2 +ϵ 2 ) T + F err T SFL-V2 N 2 S(σ 2 +ϵ 2 ) T + F err TIn the aforementioned table, we denote σ n = σ, define I err ≜ ||x 0 -x * || 2 , and represent F err ≜ f (x 0 ) -f * . The absolute constants and polylogarithmic factors are omitted for brevity. Our SFL algorithms show the same convergence rate as the distributed SGD.There have been are some papers discussing the overhead of SFL, e.g., [27, 4] . We mainly use the analysis from [27] .We 4 . It is observed that as K escalates, the cumulative cost of training time tends to rise following the sequence: SFL-V2 being less than SFL-V1. EQUATIONWe compare the SFL methods with the benchmarks, i.e., FedProx [15] and FedOpt [19] . We have used the same hyperparameters, and trained the models on CIFAR-10. The results are provided in Figure 6 . From the figures, we observe that SFL-V2 continues to be the best-performing algorithm. This is consistent with our observations in the main paper.We further study the impact of local epoch number E on the SFL performance. The results are reported in Fig. 7 . We observe that SFL generally converges faster with a larger τ , demonstrating the benefit of SFL in practical distributed systems.We further report the results using the loss metric. More specifically:• Impact of cut layer: The results are reported in Figs. 8 and 9 .• Impact of data heterogeneity: The results are reported in Fig. 10 .• Impact of partial participation: The results are reported in Fig. 11 .In general, we see similar (but opposite) trends with the observations in the main paper. That is, a higher accuracy is associated with a smaller loss. These results are again consistent with our theories.I.4 Results on the impact of the position of cut layer.We can observe from the results that the performance of SFL-V1 and SFL-V2 increases in L c . We look at the impact of the position of cut layer from the gradient perspective. We plot the gradient divergence in Fig. 12: • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Note that it might be counter-intuitive to observe looser bounds on general convex objectives than on non-convex objectives. This is associated with different performance metrics used in the analysis, e.g., see the left hand side of (12) and(13).We also compared SFL to FL and SL in terms of communication/computation overheads in Appendix H.3.More experiments on FEMNIST are given in Appendix I.5.See Appendix I.4 for more detailed discussions on this point.x t,0 s,n ← x t s , ∀n ∈ N ; 


Decision: This paper studies the convergence of distributed split learning. 
A theoretical convergence analysis is presented and the performance of the method is empirically studied for various levels of heterogeneity.

The reviewers argued that the presentation of the results must be further improved for the final version:
- currently, too many details are delegated to the appendix. The main text should have a precise mathematical description of (at least one version of)  the algorithm.
- details on the numerical experiments are missing or not well explained. For example, the choice of the hyperparameters for the baseline methods (best practice would be to tune the parameters for each setting separately).

However, the reviewers concluded that the theoretical results presented in this paper are a step toward a better understanding of split learning and could inspire future research in this area.

Reviews: review: paper_summary: paper_summary: This paper provides theoretical convergence analysis for split federated learning (SFL) methods (SFL-V1 and SFL-V2) under full and partial client participation for strongly convex, generally convex and non-convex settings. These bounds help build fundamental understanding of the settings in which SFL, federated learning (FL) and split learning (SL) are best suited and where SFL suffers (e.g., in terms of data heterogeneity, number of clients, participation rate). The authors provide empirical results that validate their theoretical claims and showcase the performance tradeoffs between SFL, FL and SL.
strength_weakness: strength_weakness: * Empirical choice of L_c is not properly swept. Given that the highest value of L_c considered is found to yield the best performance, L_c should be increased until worse performance is observed to evaluate SFL and SL at the optimal choice of cut layer and understand at what point diminishing returns are observed from increasing L_c. Is it always better just to use FL if computational resources allow? What are the tradeoffs?
* Lacking specifics of what FL and SL algorithms are used for comparison. Opportunity to consider other algorithms beyond the most basic implementations of FL and SL.
* Plots are hard to read.
questions: questions: Where are there diminishing returns in increasing L_c? If a increasing L_c yields better performance, then why not just use FL? What tradeoffs should be considered when selecting L_c beyond just accuracy?
limitations: limitations: While there is no explicit limitations section in the paper, and such discussion could be expanded upon in the text, the authors do mention that the choice of how to set the cut layer (L_c) is unknown and left for future work.
paper_summary: paper_summary: The paper proposes the first convergence analysis of the two variants of split federated learning under different set of hypotheses.
strength_weakness: strength_weakness: I was not able to check the proofs in their completeness, but there are some signs suggesting that the analysis may not have been carried out as carefully as required. First, it seems some hypotheses are missing in the statement of the main theoretical results. For example, convergence to the minimizer in presence of stochastic gradients is not possible unless the learning rate opportunely decreases over time, but this is not mentioned. Second, I have some concerns because 1) the verbal description of the studied algorithms is not very clear, and, more importantly, 2) even their mathematical descriptions are not consistent across the paper. Before moving to the specific comments, I want to stress that a certain vagueness in the actual operation of the algorithm is common to the original papers on split learning and split federated learning, but it is a less serious concern in their case because they do not provide theoretical guarantees.
My main concerns are about how models are aggregated. Now, the two aggregations presented in the main text (equation (3)) and in the algorithm formal presentations in appendix B (line 22 in algorithm 1) do not match. The first one presents an unbiasing effect, the second one not. More importantly, neither one nor the other can lead to asymptotic convergence under partial participation, because there would be a persistent noise due to the variability in the set of participating clients. By a first look at the proof, it seems the proofs do not consider neither of these aggregation rules but rather an aggregation at the level of the gradients (see page 15), which may converge (but I did not check), but there are still some potential problems for SFL-v3 (line 453) because the a_n are ignored. 

I have also some concerns about the comparison with FedAvg in the experimental section. My understanding is that the authors study a version of SplitFL where clients communicate after each batch. In this setting, SFL-v1 coincides with FedAvg if 1) \tau=\tilde{\tau} (both the server and the clients average the models at the same time) and 2) FedAvg is allowed to transmit after each gradient computation (as it seems required for a fair comparison). Now, the results presented show different performance for FedAvg and SFL-v1. Unfortunately, there is no information reported about the specific setting (i.e., what are \tau, \tilde{\tau} and how many local gradient steps FedAvg performs before communicating).

The authors have addressed my criticisms above and I have upgraded my score. It is required a significant review of the main text.
questions: questions: See weaknesses
limitations: limitations: A discussion of the limitations (e.g. the fact that the convergence results hold only when clients are allowed to communicate after each batch) is missing.
paper_summary: paper_summary: This paper analyzes the convergence of split federated learning (SFL) on heterogeneous data and gives theoretical convergence rates under different assumptions on the convexity of the loss function and the participation schemes of the clients.
strength_weakness: strength_weakness: Numerical experiments are comparably weak since the baselines for comparison are FedAvg and SL. The latter is pure local training under the perspective of federated learning. The former is a very basic federated learning algorithm. Perhaps the authors could compare with more advanced federated learning algorithms, such as FedProx, FedOpt, etc.
questions: questions: 1. In Figure 4, it can be observed that the accuracies of the algorithms on CIFAR-10 and CIFAR-100 are very close (the latter only slightly lower). However, the accuracies on CIFAR-100 generally have an observable gap compared to CIFAR-10. Is this due to the small number of clients (only 10) in the experiments? If so, how would the results change if the number of clients is increased?

2. In Figure 7, the losses of SFL-V2 (the hard-sharing scheme) almost tend to zero for the cutting layer = 1 or 2. I wonder if these curves are losses on the training set or the test set. If the losses are on the test set, the results are surprising, since even centralized training cannot achieve zero loss on the test set. Could the authors provide more insights into this phenomenon?
limitations: limitations: The last section discusses the limitations and provides some future research directions.
paper_summary: paper_summary: Split Federated Learning (SFL) is a recent distributed approach for collaborative model training among multiple clients. In SFL, a global model is typically split into two parts, where clients train one part in a parallel federated manner, and a main server trains the other. Despite recent research on SFL algorithm development, the convergence analysis of SFL is missing in the literature, and this work aims to fill this gap. The analysis of SFL can be more challenging than that of Federated Learning (FL), due to the potential dual-paced updates at the clients and the main server. This work provides a convergence analysis of SFL for strongly convex and general convex objectives on heterogeneous data.
strength_weakness: strength_weakness: I believe this is a solid and interesting work, I also have some minor concerns as follows.

1.	The section on Related Work is too brief, more discussion about the SFL needs to be included.

2.	I suggest that the author provide a table to summarize and list the convergence rate of SFL under different settings, such as convex, non-convex, full participation, and partial participation. In this way, the theoretical results can be clearer.
questions: questions: Please refer to the Weaknesses.
limitations: limitations: NA
paper_summary: paper_summary: Split federated learning combines the advantage of split learning and federated learning. However, there is 
a lack of theoretical analysis. This paper provides the first comprehensive analysis on split federated learning. The settings include strongly convex, general convex and non-convex. This paper discusses both the full participation and partial participation case. Both two versions SFL-v1 and SFL-v2 are analyzed. Moreover, some experiments are conducted to compare split federated learning with federated learning and split learning.
strength_weakness: strength_weakness: I am impressed by the long appendix, which shows that the authors indeed work very hard on this problem. Nevertheless, I think that this paper is not ready for publication currently, due to many potential concerns.

1. In paragraph 2 of introduction, this paper states that split learning leads to high latency, but this paper does not provide theoretical/experimental evidences.

2. Inconsistency between theory and experiments. The theory only gives the convergence rate. It fails to explain the experiment phenomenon shown in this paper. In particular, the theoretical analysis does not reveal why SFL outperforms FL/SL under strong heterogeneity. 

3. Line 128: statements need to be improved. It is weird to say "Based on x_s^{t,i}, based on which ... "

4. The assumptions need to be compared with existing works on federated learning.

For example, in Li et al. On the convergence of Fedavg on non-iid data, ICLR 2020, Assumption 3.3 is not required. Personally I also feel that the bound of heterogeneity by epsilon^2 may not be necessary. Hope that authors can explain more about why this assumption is necessary and how is the convergence rate affected if we remove this assumption.

5. In eq.(10), gamma=8S/\mu-1 need to be put later, after line 190. gamma is only used for the strongly convex case.

6. In Theorem 3.5, I do not see lower bound on the learning rate. To ensure that (11) holds, I think that a lower bound of \eta_t is needed. Merely requiring \eta_t\leq 1/(2Stau_max) may not be enough. Note that \eta_t=0 also satisfies \eta_t\leq 1/(2S\tau_{\max}), then E[f(x_T)]-f^* will not converge to zero.

7. I do not see any term including epsilon in strongly convex and general convex case. It is only included in the nonconvex case. Why the convergence is affected under nonconvex case but not convex case? Some intuitions are needed here.

8. In Figure 4 and Figure 5, From the curves, the accuracy oscillates violently. Therefore I suggest to reduce the learning rate.

9. List of inequalities after line 486. A^t is not defined. From the relationship above, one can only get
A^{t,i}\leq CA^{t,i-1}+B\sum_{j=0}^{i-1} C^j.

10. In line 520. First equality does not use (a+b)^2=a^2 +2ab+b^2. It just expands x_s^{t+1}. Perhaps the authors mean the second equality.

11. Table 1 needs to be put in the main paper. The comparison is important.

12. Line 828: Notation abuse of beta. In experiment section, it refers to the degree of heterogeneity.
Here it refers to the fraction of full model accessible to clients.
questions: questions: 1.In eq.(17): It seems that partial participation significantly increases the error. In particular, $G^2/q_n$ is significantly larger than $G^2$. It is not the case for federated learning. In the ICLR paper, compare eq.(6) with eq.(4), the only difference is that B is changed to B+C. Therefore, the results seem to suggest that the Splitfed has significantly worse performance for partial client participation. Please provide some intuitive explanations.

2. a_n is much smaller than 1, thus eq.(20) is significantly larger than eq.(17), indicating that SFL-V2 performs worse. This seems to be inconsistent with the experiment (which shows that SFL-V2 is better). My guess is that the current bound is not tight. Please explain.

3. Are there any supports from theoretical analysis, proving that SFL indeed performs better than FL/SL under high heterogeneity?
limitations: limitations: I think that the authors do not summarize the limitations well. The authors have mentioned how the choice of the cut layer affects the SFL performance. However, I think that the major limitation is that the good performance under strong heterogeneity is not well supported by theoretical analysis.

Comments: title: Additional response to comments
comment: Thank you for your reply. Our response is as follows:

Regarding the performance of FedOpt, due to time constraints, we only experimented with FedAdam (one of the algorithms in the FedOpt paper) using the same parameter settings as our original setup—specifically, learning rates of 1.0 for the server and 0.01 for the clients. Based on our experimental results, FedAvg outperforms FedOpt at these learning rates. This is consistent with the results in the FedOpt paper (see Figure 5 on Page 31 in the FedOpt paper). we will conduct hyper-parameter tuning for FedOpt in the final version of our paper.

Additionally, we will include the results of the test loss, such as those shown in Figure 3 of the global response PDF, in our paper. Thank you again for your valuable comments.
Two points on the numerical experiments:

1. I read the attached 1 page PDF file but surprisedly found that the FedOpt (FedAdam) performed almost the worst. This contradicts my previous research experience. You can check the FedOpt paper (https://arxiv.org/pdf/2003.00295). I re-implemented 10+ federated learning algorithms and conducted a comprehensive series of numerical experiments using computer vision datasets. The performance of these experiments (relative, not absolute) highly agrees with the FedOpt paper figures.

2. It's quite weird to draw performance curves only of the training set instead of the test set. It's more common to draw the test set or both.
Dear Reviewer j2yY,

Thanks for your reply. However, we did not see any comments of the AC and related replies. Maybe it is due to the visibility setting. Can you repeat your comments again? Thanks a lot!

Best regards,

Authors
See the reply to the comments of the AC.
Thank you very much for your quick response! 

We will enhance our main paper by emphasizing the contributions and simplifying the proof if the paper gets accepted. 
We will also improve our convergence results to make them more robust, as you suggested, in the future. Thanks again for your time and positive feedback on our paper.
Dear Area Chair and Senior Area Chair,

Thank you very much for handling our submission! We also appreciate the reviewers' efforts in reviewing this paper. As the author-reviewer discussion phase is ending very soon, we would like to bring up the following to your attention:


**1. The contributions of our paper**

We provide **the first comprehensive convergence analysis of SFL for different objective cases on heterogeneous data**, including strong convex, general convex, and non-convex objectives. We further consider the practical **partial participation of clients** for the convergence analysis. The experimental results reveal interesting insights on the impact of the cut layer on SFL and the superiority of SFL-V2 over SFL-V1 and FL. Overall, our paper provides a thorough understanding on the performance of SFL, which potentially guides the implementation and optimization of SFL (e.g., the choice between FL and SFL, the choice of hyper-parameters and cut layers).

**2. Our response to reviewers' comments**

The reviewers raised several concerns regarding the impact of the cut layer $L_c$ on performance, the assumptions and hypotheses in our proof, the details of the algorithms, the theoretical and experimental comparisons between Federated Learning (FL) and Split Federated Learning (SFL), and the limitations of our paper. We have made the following revisions in response to these concerns.

- From a theoretical perspective, we **clarify the assumptions and conditions for learning rates for different objectives**. We also give a qualitative discussion on why SFL-V2 outperforms SFL-V1, FL, and SL.

- Experimentally, we add **additional results using two benchmarks**, FedProx and FedAdam. We also include experiments on CIFAR-100 with more clients under varying data heterogeneity, showing that our key observations hold. We explain phenomena in our results, such as the loss fluctuation and the differences between FL and SFL-V1.

- Additionally, we explain the **choice of $Lc$ and tradeoffs in SFL** by considering the computation and communication overheads alongside model performance, detail our algorithms' model aggregation and partial participation methods, expand the related work section, and enhance clarity throughout the paper. Finally, we add more detailed discussions on **limitations and future work**.

- **Concerns about the inconsistency between theory and experiments**. The comment says that while SFL empirically outperforms FL/SL under high data heterogeneity (i.e., a small value of $\beta$), this improvement is not directly captured in the convergence results. To this end, one needs to improve the bounds for SFL and a potential approach is to improve the results for client-side and server-side models. This would require a different approach and effort, which is left to future work. 

If this paper gets accepted, we will include all the new results and discussions in the final version. Thank you again!

Very best regards,  

Authors
Thanks for your feedback. Since it is the first analysis on SFL, I think that it is reasonable that the analysis is not very complete, although I think that it would be better to let the theoretical analysis match your claims from experiments. Now I increase the soundness score to 3 and the overall rating to 5. 

As you said, I suggest to further emphasize in the paper that this is the first convergence analysis of SFL. Although this does not fully explain the advantage of SFL, the first analysis is still important. Another suggestion is to try your best to simplify the proof to reduce the length of the paper.
Thank you very much for the response. Thanks again for your time and positive feedback on our paper!
Thank you very much for the response! 

We will certainly revise our main paper according to the comments if it gets accepted.
Thanks again for your work and positive feedback on our paper!
Thank you very much for the response! Thanks again for your work and positive feedback on our paper.
Thank you for your feedback! We plan to enhance our convergence results to enable a theoretical comparison with FL/SL in the future. One potential approach is to improve results for the client-side and server-side models respectively. Moreover, to our best knowledge, including the data heterogeneity parameter $\beta$ in the convergence analysis is not a common practice in the literature. Our analysis follows typical assumptions in the FL community and uses the upper bound of gradient divergence to characterize data heterogeneity.

Again, we want to emphasize our primary contribution in this paper: presenting the first convergence analysis for SFL under different objectives and partial participation. The dual-paced model update mechanism in SFL adds complexity to the convergence analysis, making it a challenging yet significant achievement.

Thank you once again for your valuable comments.
Thank you for your response with an explanation of the split layer, added clarifications on algorithm details, and statement of limitations. This addresses my concerns.
I thank the authors for their answers. They confirm that there is too much important information which is relegated to the supplementary material. I trust that, if the paper gets accepted, the authors will make a significant revision of the main text to include the missing information (learning rates, client participation conditions, comparison with FedAvg) and increase my score to 5.
Thanks for your detailed response. My main concern is the inconsistency between theory and experiments. I think that authors have provided a good qualitative discussion on the reason why SFL outperforms FL/SL under high data heterogeneity. However, for a solid theoretical work, I think that it is necessary to theoretically justify that SFL outperforms FL/SL. For example, it would be better if the bound contains $\beta$ in your revision, and the dependence of $\beta$ of SFL is better than FL/SL. 

I will not strongly oppose the acceptance if it is the consensus.
Dear Reviewer b5tv,

We hope that our rebuttal above has addressed your concerns. As the author-reviewer discussion period is
ending very soon, please kindly let us know if you have further questions or comments. We will add all the new results and discussions to the final paper. Thank you very much!

Best regards,
Authors
Dear Reviewer 8Y1a,

We hope that our rebuttal above has addressed your concerns. As the author-reviewer discussion period is
ending very soon, please kindly let us know if you have further questions or comments. We will add all the new results and discussions to the final paper. Thank you very much!

Best regards,
Authors
Thanks for your reply. My questions have been well addressed with enriched related work and a conclusion of convergence results. I will keep the initial score.
Due to space limitations, we add additional responses to your comments as follows.

**5. Presentation improvement (for W3, W5, W9, W10, W11, W12)**

We have carefully refined the presentation of the entire paper mainly including the following:

- For line 128, it is revised as "SFL-V2: the main server computes the loss $F_n([\boldsymbol{x}_{c,n}^{t,i},\boldsymbol{x}_s^{t,i}])$ , based on which it then 
 updates the server-side model $\boldsymbol{x}_s^{t,i}$."
 
- We will revise the presentation of $\gamma$ in equation (10) to include it only in the strongly convex case.

- For line 487, we define $A^{t}:=A^{t,0}=\mathbb{E}\left[\left\Vert \boldsymbol{x}^{t,0}-\boldsymbol{x}^{t}\right\Vert^2\right]=0$. This is achieved by the fact that $\boldsymbol{x}^{t}=\boldsymbol{x}^{t,0}$ at the beginning of each training round $t$.

- For line 520, the second equality is due to $\left(a+b\right)^2=a^2+2ab+b^2$. 

- We will move Table 1 to the main paper and explain the comparison between Mini-batch SGD, FL, SL, and SFL.

- For line 828, we will redefine the notation to avoid confusion. Let $K$ represent the total number of clients involved and $D$ denotes the aggregate size of the data. Let $v$ indicate the size of the smashed layer.  The rate of communication is given by $R$.  $T_{\rm fb}$  signifies the duration required for a complete forward and backward propagation cycle on the entire model for a dataset of size $D$. The time needed to aggregate the full model is expressed as $T_{\text{fedavg}}$. The full model's size is denoted by $|W|$.  $r$ reflects the proportion of the full model’s size that is accessible to a client in SFL. The updated Table 3 is shown as follows.

|Method|Communication per client|Total communication|Total model training time|
|:-:|:-:|:-:|:-:|
|FL|$2\|W\|$ | $2K\|W\|$|$T_{\rm fb}+\frac{2\|W\|}{R}+T_{\rm {fedavg}}$|
|SL|$(\frac{2D}{K})v+2r\|W\|$ | $2Dv+2rK\|W\|$|$T_{\rm fb}+\frac{2Dv}{R} + \frac{2r\|W\|K}{R}$ |
|SFL-V1|$(\frac{2D}{K})v+2r\|W\|$|$2Dv+2r K\|W\|$|$T_{\rm fb}+\frac{2Dv}{RK}+\frac{2r\|W\|}{R}+T_{\text{fedavg}}$ |
|SFL-V2|$(\frac{2D}{K})v+2r\|W\|$|$2Dv+2r K\|W\|$|$T_{\rm fb}+\frac{2Dv}{RK}+\frac{2r\|W\|}{R}+\frac{T_{\text{fedavg}}}{2}$ |

**6. Learning rate of SFL (for W6)**

SFL and other deep learning algorithms, such as FL, require a positive learning rate to ensure the model is updated. Therefore, the lower bound of the learning rate is $\eta^t > 0$. We will update this condition in our main paper.

**7. Explanation of experimental results (for W8)**

The fluctuations in model accuracy observed in Figures 4 and 5 are due to clients' partial participation in SFL. In particular, clients are randomly sampled for training in each training round. Due to the data heterogeneity of the clients, the datasets of the sampled clients in each round may not represent those of the entire client population. Since the model is trained based on the local datasets of the sampled clients in each training round, the varying set of sampled clients makes the model accuracy fluctuate across rounds. This phenomenon has been discussed and verified in [Kairouz et al. 2021, Section 7.2] and [Tang et al. 2023]. Moreover, this explanation on the accuracy fluctuation can be supported by  Figure 4, where the fluctuations across training rounds become more pronounced at lower participation probabilities, $q$. On the other hand, regarding the learning rate,  we maintain the same learning rate across different participation probabilities to ensure a fair comparison. The learning rate was fine-tuned to achieve the best converged model accuracy under full participation. 

[Kairouz et al. 2021]  P. Kairouz and et al., ``Advances and open problems in federated learning", arXiv, Mar. 2021.

[Tang et al. 2023] M. Tang and V. W.S. Wong, “Tackling system induced bias in federated learning: Stratification and convergence analysis,” IEEE International Conference on Computer Communications (INFOCOM), May 2023. 

**8. Theoretical results comparison between SFL-V1 and SFL-V2 (for Q2)**

Our convergence bounds demonstrate the same convergence rate with respect to the training round $T$ for both SFL-V1 and SFL-V2. Despite $\alpha_n$, SFL-V1 and SFL-V2 have different bounds on the stochastic gradient variance 
$\sigma_n^2$. Specifically, SFL-V2 is more centralized than SFL-V1, resulting in a lower $\sigma_n^2$. In the future, we will improve the convergence error rate to make it tighter.

**9. Limitation of our paper (for Limitation)**

One limitation is that our bounds for SFL achieve the same order (in terms of training rounds) as in FL, yet the experiments showed that SFL outperforms FL under high heterogeneity. This is possibly due to that tighter bounds for SFL are to be derived, which is an important future work. Another limitation is that we did not include the theoretical analysis on how the choice of cut layer affects the convergence. We will add the more detailed discussions of limitations to the main paper.