Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approach
Authors: Dandan Liang
Jianing Zhang
Evan Chen
Zhe Li
Rui Li
Haibo Yang
Venue: NeurIPS 2025 poster ()
Keywords: Split Federated Learning
Unbalanced Updates
Zeroth-Order Optimization
Communication Efficiency
Primary area: optimization

Abstract: Split Federated Learning (SFL) enables scalable training on edge devices by combining the parallelism of Federated Learning (FL) with the computational offloading of Split Learning (SL). Despite its great success, SFL suffers significantly from the well-known straggler issue in distributed learning systems. This problem is exacerbated by the dependency between Split Server and clients: the Split Server side model update relies on receiving activations from clients. Such synchronization requirement introduces significant time latency, making straggler a critical bottleneck to the scalability and efficiency of the system. To mitigate this problem, we propose *MU-SplitFed*, a straggler-resilient SFL algorithm that decouples training progress from straggler delays via a simple yet effective unbalanced update mechanism. By enabling the server to perform $\tau$ local updates per client round, *MU-SplitFed* achieves convergence rate $\mathcal{O}(\sqrt{d/(\tau T)})$, showing a linear reduction in communication round by a factor of $\tau$. Experiments demonstrate that *MU-SplitFed* consistently outperforms baseline methods with the presence of stragglers and effectively mitigates their impact through adaptive tuning of $\tau$.

Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approach
Dandan Liang Rochester Institute of Technology Rochester, New York
Jianing Zhang Purdue University West Lafayette, Indiana zhan4670@purdue edu
Evan Chen Purdue University West Lafayette, Indiana chen4388@purdue edu
d159740rit.edu
Zhe Li Rochester Institute of Technology Rochester, New York 2140638rit. edu
Rui Li
Rochester Institute of Technology
Rochester, New York
rxlicsèrit. edu
Haibo Yang
Rochester Institute of Technology Rochester, New York
Abycis@rit. edu
Abstract
Split Federated Learning (SFL) enables scalable training edge devices by combining the parallelism of Federated Learning (FL) with the computational offloading of Split Learning (SL). Despite its great success, SFL suffers signifi- cantly from the well-known straggler issue in distributed learning systems. This problem is exacerbated by the dependency between Split Server and clients: the Split Server side model update relies on receiving activations from clients. Such synchronization requirement introduces significant time latency, making straggler a critical bottleneck to the scalability and efficiency of the system. To mitigate this problem, we propose TW-SplitFed, a straggler-
SFL algorithm in zeroth-order optimization that decouples training progress from straggler delays via simple yet effective unbalanced update mechanism. By enabling the server perform т local updates per client round, MU-Spl tFed achieves a convergence rate of ((vd/(rT)) for non-convex objectives, demonstrating linear speedup of T in communication rounds. Experiments demonstrate that MU-Splith Sted consis- tently outperforms baseline methods with the presence of stragglers and effectively mitigates their impact through adaptive tuning of 7. The code for this project available httts://githu/: com/ cow.Jonny-- http://///oonnmy222pt/tt---.1110 Mip/MU-SSSiiteee
Introduction
Split Federated Learning (SFL) [ integrates the strengths of Federated Learning (FL) and Split Learning (SL) 6), enabling efficient training
essource-oonstriine devices. FL offers parallel client updates but imposes heavy computation on edge devices (7). while SL reduces client load by offloading computation to the server but suffers from high latency due to its sequential nature. SFL balances these trade-offs,, making it a promising framework for scalable training, especially as model sizes grow. However, the relay-based training mechanism in SFL introduces synchronization bottlenecks due to stragglers: clients with the slowest computation or communication speeds delay the overall process (8.) Both global aggregation and client-side updates must wait for the slowest participant, limiting scalability [10]. This issue is a well-known bottleneck in distributed learning that
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

can severely degrade training efficiency [1)-14). This issue is further exacerbated by the increasing size of ML models and the limited computational capacity of edge devices (15)
To address this issue, existing works draw inspiration from straggler mitigation strategies in FL.. Adaptive splitting techniques [6] dynamically adjust the client-side cut layer based on network conditions enforce synchronization. However, this strategy requires the model architecture to expose layers with varying activation dimensions. In modern transformer-based models, where activation sizes are nearly uniform across layers, such flexibility absent, and shifting the cut therefore provides little benefit: the amount of data transmitted remains essentially constant, leading to persistent communication delays regardless of the split location. Another approach enables asynchronous updates by allowing the server to proceed with stale information [8]. While this reduces idle time, it exacerbates client drift under high data heterogeneity, harming model performance. Although these methods focus on reducing the straggler-ing indduedd latency, they often overlook more dominant factor contributing to training overhead: the total global communication round. As result, we investigate the following question: Can we efficiently alleviate the impact of stragglers in SFL by strategically reducing communication round?
We provide an affirmative answer in this paper, aiming to accelerate convergence under practical system heterogeneity and thereby reduce training time overhead. We propose MU-SplitFed a SFL framework that leverages unbalanced server-client updates to improve training efficiency by con- trolling communication frequency. Our approach exploits the computational advantage of powerful servers: instead of idly waiting for slow edge devices, the Split Server performs т optimization steps for each client-server server communication round, effectively accelerating the training process. To further ease memory and computation burdens on edge devices, we incorporate Beroth-Order (ZO) optimization on the client side, enabling training without backpropagation [17[18]. Beyond empirical performance, we provide a rigorous theoretical analysis showing that our method achieves linear speedup with respect to the server iteration т, without relying on strong assumptions. As a result, the total training time no longer affected by the speed of the slowest client. Our analysis also rigorously accounts for the variance introduced by ZO methods. Due to model splitting, obtaining tight convergence bounds for SFL is more challenging than for standard FL: existing theoretical results in parallel SFL (9] use stronger assumptions (e.g., bounded gradients) while failing to capture the acceleration from clients or local updates. contrast, our theoretical results not only reflect the acceleration from but also account for other factors such the number of clients.
We summarize our main contributions as follows:
A novel SFL framework: We propose MU-SplitFed a straggler- resiliet SFL framework that effectively reduces the communication round by leveraging unbalanced server-client updates. While other SFL methods suffer from server idleness due to stragglers, MU- WW-SplitFed enables the server to perform local updates during each client-server communication round. This effectively utilizes server-side computation and decouples total training time from the slowest client. By incorporating ZO optimization, our method further reduces resource usage on low-capacity edge devices (Sec. 3
Theoretical convergence with linear speedup: We provide a rigorous convergence analysis of MU-SplitFed. The convergence rate is O( ((vd((TT))) (d/(TT) for non-convex setting with the standard assumptions, showing the linear speedup w.r.t the server-side update т. Furthermore, our theory supports that the reduction the communication round allows the total training time to become independent of the straggler's speed, directly addressing major bottleneck in SFL (Sec. 4.2)
Insights into model partitioning and update alignment: We uncover a critical connection between the model splitting strategy and the unbalanced update ratio. Both our theoretical and empirical results demonstrate that aligning the server-side model depth with the value of 7 is essential for optimal convergence. A larger would benefit from more layers on the server side, thus accelerating convergence through more effective server-side computation (Sec. 4.I).
Empirical validation: We validate the effectiveness of MU-SplitFed through experiments on benchmark datasets. Beyond its advantage in reducing communication round, our method consistently outperforms baselines under high client heterogeneity, highlighting its practical feasibility for straggler mitigation in SFL (Sec.

---

Steps Unbalanced Updates
Clicat Side Model
M(FE))
during each Communication Round g
Server-Side Model
Flas Aui h((t))))) F (2 Au (Rem))
Client-Side Embeddings
6m
Informattion
(1) ZO estimation with fised client-oide emboddiing-
Server Aggregation during end each communication round
h(ZO3)
Jam 77 22 Mam
O,..., 7
Efficient local update utilizing Server Capacity
Total Communication Time:
Eatraggle
Eatrggeer furree
Fed Server
Clients
Split Server
Stragzher Training Time
Figure I: Overview of MU-SplitFes The global model is split at the cutting layers into two parts: client-side model ie and server-side model xg. Each client m trains its local copy z..m while the Split Server performs local updates on 2,,m using the latest embedding. without waiting for the client finish. At the end of each global round, the Fed Server aggregates all client-side models, and the Split Server averages all the server-side models to form the updated global model.
2 Background and Motivation
SFL Setup. We consider the parallel SFL. framework 11 which combines the model-splitting strategy of SL. (5) with the parallel client updates of FL [4] SFL, neural network partitioned at layer Le, assigning the first Le layers to the M clients "client-side model", parameterized by (Te.I....,, .)] ), and the remaining layers to the Split Server, which maintains M corresponding "server-side model" (Ie,ls...,, M.. J. The combined parameters for client are denoted I.m. (Jr.m J. Client m computes the embedding hm = (f fm. at the cut layer and sends it to the Split Server, which holds the label Vm and computes the loss:
F(rm: Em) = -(I,.., : Ym))
where 5.. Dm is the data sample as client input. The server computes gradient estimate and returns it to the client, which uses it to update both client-side and server-side parameters. The M client-server pairs collaboratively train global model. After each round of local training, the client-side models are aggregated by the Fed Server, while the server-side models are aggregated by the Split Server. The overall objective the SFL framework can be formulated as:
mina (x) = EM Wm/mmm (m)),
where fm(x)= Dmm EEED,, F(x;)) is the local loss function, and W.m is the weight of client m, with Wm [0, satisfying EM Wm 1.
ZO Optimization. Zeroth- Order Optimization (ZOO) is gradient-fre method, offering an al- ternative solution for scenarios where explicit gradient computation is impractical, expensive, or unreliable [T. ZOO has shown significant advantages memory saving because requires only forward passes 18). Since our goal is to improve training efficiency for edge devices with limited memory resources, we adopt ZOO to reduce even more memory consumption for our resource- constrained devices. In specific,
adopt Simultaneous Perturbation Stochastic Approximation (SPSA) [2 as our ZO gradient estimator. Let u be uniformly sampled from the Euclidean sphere v Vagd- for any function fxx Rd + and À >> 0, we define its ZO gradient estimator as:
g f()+ (alu))f((aa f(x= Au)
Challenges in Mitigating Stragglers SFL. The straggler problem a persistent bottleneck in distributed learning systems, where synchronous training requires coordinated updates across multiple agents [0 SFL, this issue is further exacerbated by the interdependence between clients and server. There two factors that contribute to this severity: the server must wait for all clients to transmit embeddings or gradients before continuing, making the system highly sensitive to the slowest participant; 2) the model is split across client and server, requiring frequent communication during both forward and backward passes. This tight coupling amplifies the impact of stragglers compared to traditional FL. where delays typically limited full model updates.
In FL., asynchronous updates have been proposed to mitigate such issues by decoupling client updates from global synchronization 23 27). However, these approaches insufficient for SFL,
3

---

as they only address global aggregation. In SFL, the straggler problem also arises from split-layer communication, a fundamental difference that makes asynchronous techniques in FL less effective when directly applied to SFL. Recent efforts in SFL have explored adaptive model partitioning to balance computation and communication delays | T6]. These methods are constrained by the network architecture and fail to address the core issue: the high communication frequency between clients and the server. As a result, none of the existing straggler solutions explicitly aim to reduce the number of communication between client and Split Server, which is the key problem to SFL's straggler-induced inefficiency. These limitations point to the need for a new framework that explicitly exploits SFL's structural properties to reduce communication frequency, thereby mitigating stragglers without sacrificing model performance.
3 Methodology
Building upon the aforementioned challenges, we propose MU-SplitFed to mitigate the straggler is- sue by jointly addressing memory inefficiency, computation imbalance, and communication overhead. By combining unbalanced update scheduling and zeroth-order optimization, our algorithm achieves robust and scalable performance tailored for resource-constrained edge devices.
Algorithm 1:
MU-SplitFed
Input:
Unbalanced update steps T, global communication rounds T, local learning rate on server side Y)s,
learning rate on client side T
[21 s. each global round t .,..... T do
Output: Global model x²
each client E 21, M) in parallel do Pull global model for initialization: z f x s.mm f x
/+ Phase 1:
Unbalanced Update on Split Server each client E {1, ................................ M) in parallel do Send embeddings f N to Split Server: each local iteration = O,... T
do
Compute zeroth-order gradient gh:hn according 5
Update Split Server model: r + rhi .y....
Compute zeroth-order info 8t.m according to (6) and send it back to the client;
Update client model: (+1 + x2.m ",,m
10
/* Phase 2: Model Aggregation Fed Server
Fed Server and Split Server updates according to 7 Fed Server broadcasts xf+1
1+ to all clients.
11
Training Procedures. MU-SplitFP integrates an unbalanced update strategy and ZO optimization into the SFL framework. The overall training process consists of two main phases: 1) Unbalanced ZO updates between clients and Split Server: A subset of clients communicates with their corresponding server-side models on the Split Server and performs local training using ZO optimization in an unbalanced update manner. 2) Federated Aggregation across M models: The Fed Server collects the updated model weights I.m for E [M] and applies the FedAvg strategy compute new global model. We detail both phases below and provide the full procedure in Algorithm T
Client Model Perturbation and Forwarding, At global round f, each activated client samples data point etn E Dm,- To perform ZO updates, the client perturbs its model parameters and computes the corresponding embeddings multiple times. First, the client computes the unperturbed embedding hf, = h=x'.,m ). and the perturbed embeddings:
14 = h( + mm gt stm); and h =h(x Xul St.):
where un. is the perturbation direction sampled according to Equation (3). is a smooth parameter, z is the client-side model at round t. We define H = [h/m h/ A/ }. The client then transmits H² to the server for computing the ZO gradient required for model updates.
Unbalanced Split Server Update. The transmission of embeddings follows an on-the-fly manner: each embedding is sent immediately after is computed. Unlike the client, which requires feedback from the server to proceed updates, the Split Server can compute Z0 gradients independently. To fully utilize the server's computational capacity, we introduce an unbalanced update mechanism, allowing the server to perform multiple updates using the unperturbed embedding hi Specifically,

---

instead of remaining idle, the server initiates multiple local updates using h while waiting for the full set of perturbed embeddings h and hn We denote 0, .,.... as the server update
round. At global round and server round i the server perturbs its model parameters and computes the corresponding 70 gradient differences
8% = F( + +uu him) - F( - -u s, hm))
(5)
where uhi is sampled according to (3), and x4. denotes the server-side model parameters for client m at global round and server update step i. The corresponding ZO gradient estimator is
&2im computed as: s.m 2A u where o'.,, denotes the loss difference obtained from the perturbed embeddings. The server-side model is updated iteratively over T local steps using the ZO oracle: a.m _ G i [0.
Zeroth-order Back Propagation and Client Update. After completing server-side local updates, it then computes the ZO loss differences required for client-side model updates:
_ Fl )
F( hm))
where each & is a scalar and incurs minimal communication overhead. These ZO differences
6%.m
u'.m- and update their sent back to the client. Clients compute their 70 estimates as G...m c.m 2A models via x/th z', - G
Global Aggregation. At the end of the global communication round I, once all activated local models Im fm F has completed their update, the Fed Server collects the updated parameters ce and performs model aggregation, while the Split Server also locally aggregates us.m and performs an update on Zg:
[+1 -
11g Σm Wm (=ttt) = and
x ++1
x 17g Em Wm (2tt,
(7)
where Wm denotes the aggregation weight for client m, in our algorithm we choose to set Wm 7g is the learning rate for global update. Then, the Fed Server broadcasts rf+1 I to all clients.
N
4 Convergence Analysis
In this section, we present rigorous convergence analysis of MU- Sp-SpietFed. Specifically, we want to quantify the effect of our unbalanced update mechanism on convergence. However, FL., this effect may be intertwined with other factors such as data and system heterogeneity. To isolate the influence of the unbalanced updates, we first analyze the single-client setting, which simplifies to a standard SL framework (Sec. [. Then, we propose our general result under SFL settings (Sec. 4.2). The complete proofs are deferred to Appendix C[ad
D Here, we first make some standard assumptions
that will facilitate our analysis 2
Assumption 4.1 (I-SSootth. The loss function ʃ is bounded from below, and is L-smooth, i.e.
Vr, y. V (z) ff(i)] <
Assumption 4.2 (Bounded Variance), The variance of the stochastic gradient w.r.t. the client and the server is upper-bounded by o² and o² Specifically, for Ay E Dm. V V. f E) V.. fx)) I of and v., (x;; V., f2 < o
4.1 Convergence Analysis for MU-Split
To analyze the impact of multiple server updates alone, we consider the special case where M denoted as MU-Split, which reduces to the SL setting. The convergence of MU-Split is established in the following theorem:
Theorem 4.1. Under Assumption 4.1 and 4.2 and let the server iteration number be T. If the
learning rates on client and server satisfy Te/ 7) === -n S min{ 6LL(7+2d,)) TeLrd, }. the sequence
of iterates generated by our MU-Sp Spli satisfies:
1 T EI
4F < mp + 16n ( 1)d,0? 8 Ld.
+ ( + 1/4) x2
(8)
'Here we slightly abuse the notation and denote F( F( Ma ) y.m)), where vin. is the label corresponding to data <t. 'The assumptions adopted in our analysis are standard and consistent with those commonly used in the distributed optimization literature 28 30]. We focus on the non-convex setting.
5

---

where ==
d. and d, represent the dimensions of the parameters on the client and server side, respectively; d de + d, the total number of parameters. the smooth parameters for ZO Oracle defined in (3) and o of are the upper bound of the gradient variance on client and server, respectively. 7 = 1e/TT = TJs is the unified learning rate.
To establish the theorem, the learning rate on server needs to shrink linearly with multiple update steps T, i.e. n 7/s- This requirement stems from the need to balance client and server progress: since the server performs r updates for each client update, a proportionally smaller server learning rate ensures synchronized convergence. The convergence bound in equation (8) contains five distinct terms, each capturing different aspects of the algorithm's 's behavior.
1 The first term, T represents the optimization error and decays as either the total number of communication rounds T or the server-side update frequency т increases. This rate matches the same rate typical ZO-SGD methods when 1, which generalizes the classical convergence rate without unbalanced update. It also highlights the benefit of unbalanced updates: increasing the number of server iterations per round leads to a faster reduction of this term. This demonstrates the improved convergence behavior enabled by unbalanced server updates.
The second and third terms quantify the error introduced by the variance of the stochastic gradient estimates on the server and client, respectively. Notably, those two terms scales up with the parameter т. This means that larger exacerbates the stochastic error, thus leading to high variance in the estimated gradient that hinders convergence performance. To keep these terms small, an inverse relationship between the Split training learning rate and server-side local steps should be satisfied, i.e.,
77 = 1) (1// Specifically, note that both the server-side and side variances are linearly amplified by r. This requires a sufficiently small 17 to offset the variance between two successive communication rounds to make the those error term in small. The intuitive explanation behind this is that when the server applies multiple consecutive updates using outdated client information, it introduces client drift and allows stochastic errors to accumulate progressively. Consequently, smaller step sizes are required to balance the impact of these accumulated error terms.
The last two terms, (1, + d/ d. capture emrors introduced by the zeroth-order gradient estimation. These terms are independent the learning rate choice and decrease as the smoothing parameter A decreases, indicating that more accurate ZO gradient estimation improves overall convergence.
We can further derive a convergence rate for all terms if certain conditions are met. Corollary 4.2. Based on Theorem 4.1 let the model split satisfies de = dd/T, d, = d - v d/t; let T S d, the smoothing parameter satisfies X2 S "TTT5/22L' and choose the unified learning rate as 1) S minf 61L(7+20.)) 16Lrd,' V )
Then we have the following convergence rate:
+ BUI
400F VIT
48Lydo VTT
9v
8L0² VT
VTT
Discussion. All dominant terms in equation 9 converge at the rate of d/dTT))) when we choose de = Vd/T and d, d - /d/T, where d de + de is the total number of parameters. This rate highlights a linear speedup term of The linear speedup is achieved when the client-side parameter dimension de scales as (d/ VF). This has direct implications for network architecture design in split learning systems. In particular, when the server has higher computational capacity, is beneficial to allocate fewer parameters to the client side, thereby placing the split closer to the input layer. That's being said, ZOO provides a natural mechanism for controlling stochastic variance through the cutting layer strategy. By connecting the cutting layer choice with multiple server updates steps т, the variance impact the client side is effectively reduced. This variance reduction occurs because fewer layers are processed on the client side, which inherently limits the accumulation of gradient estimation errors. This theoretical finding aligns with our empirical observations in the ablation study presented in Section 5
'To attain s accuracy for an algorithm, it needs ( communication rounds with a convergence rate O ( while needing O (()r)) rounds if the convergence rate is O VFT In this sense, one achieves linear speedup with respect

---

4.2 Convergence Analysis for MU-SplitFed
We further derive the following convergence result for MU-SplitFed under SFL with M clients. For the convergence analysis of MU-SplitFed under SFL, we further assume that the above two assumptions apply to fm for Vm E [M]] To quantify the data heterogeneity across clients, we make the following assumption Assumption 4.3 (Bounded Heterogeneity). ). For V'm E [M], the global variability of the local gradient is upper bounded: | fm (x) Vf()) 12 2.
data distribution:
Theorem 4.3. Under Assumption 44. 1o 4.3 consider a SFL framework with M clients, and let the server iteration number be т. If the learning rates on client and server satisfy ne/T 77s = 1) S
minf 120L7(142,.77) 12 Ldi }. the sequence of iterates generated by MU-Split satisfies:
1F T E V S Tnuet + 1
+ 4n Ld
M
/g +g + 2d.) + Ld
+ 24n //
+ -LT+88n²77 2n,n
L2
(10)
where F -==[((x)) }): de and d, represent the dimensions of the parameters on the client-side and server-side, respectively; À the smooth parameters for ZO Oracle defined in (3). and o;: og are the upper bound of the gradient variance on client and server. Additionally, '1g is the global learning rate for model aggregation, and e² quantifies data heterogeneity.
The first term and the last two terms are similar to MU-Split, which are attributed to model initial- ization and ZO optimization. Compared to traditional SFL, the presence of server iteration 7 is again observed on the denominator, which corresponds to our observation in MU-Split: convergence is accelerated by multiple server updates. The second and third terms correspond to the variance of the stochastic gradient estimator on the server and client, respectively. Again, both terms scales with the increase of r, which is consistent with MU-Split. In contrast to the analysis in MU-Split, the fourth and fifth terms are newly introduced account for data heterogeneity, and they are also observed in other Federated Learning literature. Notably, those two terms scales with the parameter 7. This means that larger 7 exacerbates the heterogeneity error thus leading increases client drift consequently. So, similar to SL, to offset the variance introduced by data heterogeneity and stochastic gradient estimation, sufficiently small n should be selected and decay linearly with т.
Corollary 4.4. Based on Theorem 4.3 if we further ensure that the neural network is cut such that de = vd/T,de = d - dd/:; let S d, let the smoothing parameter X2 S VTTES/L and choose learning rate 1j S min[ 120L-(1+2,/77) 12r Trvarh. F = =E([(x)) - [(xx)), }}. and we have the following bound:
M
1/ VTM. Define
4E. BIIl E(IV
ALVAF VITM
8yd((se +20"1 32Vd(3e+o
TT +
12e"+402 VTM
6V4
(11)
V+TM
Discussion. The first and second term converge at the rate O( ((Vd/(TTM)) Compared with MU-Split. the involvement of multiple clients M accelerates convergence through the increased number of participating clients. This property is particularly desirable in the federated setting, where large-scale parallelism can be leveraged to speed up training. In contrast, the third and final terms do not benefit from parallelism across clients. Nevertheless, their impact is mitigated by the faster convergence rate with respect to T, which decays faster than the dominant terms. The fourth term, which captures client heterogeneity and gradient variance at the client side, does not contain the acceleration factor. This further confirms that multiple local updates contribute to the acceleration of initial error and variance introduced by the server, while the client side does not benefit from it. More importantly, while the server- side learning rate decrease with t, the global learning rate amplifies by r. The intuition behind this is as follows: the server side uses stale information to update, smaller learning rate ensures that each server update remains close to the original model, preventing large deviations. However, smaller learning rates reduce the cumulative gradient step the server. To ensure a globally faster convergence rate, the global aggregation compensates for this by applying a slightly larger learning rate. Finally, the overall convergence rate is O ((/d/(rTM)) demonstrating that multiple local updates 7 and multiple clients M jointly accelerate convergence in SFL.
Straggler resilient communication time. The total communication time in SFL. is largely determined by the straggler, as all other parties must wait for the slowest client to complete its computation before
7

---

Table 1: Test accuracy on four datasets. We run each method for 100 epochs on Fashion-MNIST and 500 epochs on the others, and report the resulting accuracy the final epoch.
Dataset
GAS
Vanilla SplitFedi(( = 1)
Ours(r = 2)
Ours(r = 3)
Ours(r = 4)
CIFAR-I0 10
75.28
69.73
77.86
73.20
69.40
Pashion-MNIST
83.70
77.50
85.45
85.28
84.47
CINIC-10
57.80
51.96
59,50
55.75
52.43
CIFAR-100 25.33
16.58
32.16
24.64
22.38
proceeding to the next communication round. We first define three terms for further explanation: 1) Estraggler denotes the time delay of the straggler, 2) To represents the number of communication rounds required for convergence, and 3) server the server-side computation time for one local update. In parallel SFL settings, the required total delay caused by straggler can be represented as To sstagglers which mainly depends on the straggler and results in slow and unstable convergence.
In contrast, with unbalanced updates, if we let the server perform T = tstraggler/ server local iterations during each round. According to Corollary (4.4, this reduces the total number of communication rounds from To to Ti = To/T. Consequently, the total communication time becomes:
Ti Ismggler = To burnggler/T To bservers
(12)
which is now independent of the straggler time. This result highlights key advantage of MU-SplitFed: by appropriately choosing т, the system can effectively decouple overall training time from the performance of the slowest client.
Experiments
Experimental Setup. To evaluate the effectiveness of (U-SplitFedd we conduct experiments on four image classification benchmarks: MaShion-MNIST (3I), CINIC- (2). CIFAR-10, and CIFAR- 100 [33]. All experiments are carried out a node with 3 NVIDIA A 40GB GPUs. The model cut layer is denoted as Le. where Le = n means the model is split after the n-th block. For these tasks, we adopt the AlexNet architecture, assessing the framework's s ability mitigate the impact of stragglers. As AlexNet contains only 8 layers, it offers limited flexibility in exploring different splitting configurations. To further analyze the role of the unbalanced update ratio T in controlling communication round, we extend our study to a large language model (LLM), OPT- 1.3B (44, which has 24 transformer blocks and enables broader range of splitting strategies. We evaluate its performance on the SST-2 dataset (35) a binary sentiment classification task, to examine the applicability of #U-SplitFed in the LLM domain.
We compare MU-SplitF with vanilla SplitFed and GAS (8). recent SFL method that addresses stragglers via asynchronous updates. Vanilla SplitFed serves baseline without straggler mitigation strategy. To simulate the device heterogeneity, we follow the simulation design of [8] 12 In particular, we sample the computation time from an exponential distribution represent different computation capacities across different clients. In our experiment, we train 10 clients in total with 50% partial partitioning for each global aggregation. For fairness comparison, we modify both vanilla SplitFed and GAS to use ZO optimization. aligning them with MU-Spl SSSSppttFedd gradient- free design. Additionally, we evaluate the convergence performance w.r. to time unit of our simulation, providing direct measure of each method's performance to straggler-indus indueedddd delays.
Impact of 7 Selection. First, we investigate how the choice of 7 impacts the performance of our proposed MU-SplitFed. We compare the accuracy from the same global communication round across different methods: we pull the result of the 100th epoch for Mushion-MMIIST and choose the 500th epoch result for the rest three datasets. As shown in Table [] we compare the training accuracy with different values of the server iterations E {2. 4). Our method achieves the highest accuracy when = 2, demonstrating its effectiveness in reducing communication round. However, increasing r over 2 leads a noticeable drop accuracy. This observation aligns with our theoretical insights. Specifically, Corollary 4.2) suggests that the choice of is related to the parameter size of the client-
side submodel de = which is governed by the cut layer L.- Given the structure of AlexNet,
L. = 2 is the only split type satisfied this setting without violating the constraint L. 2 1. Thereby, 2 corresponds to the optimal choice of server steps given fixed cutting strategy. Consequently,
as r exceeds this value, the mismatch between r and splitting strategy contributed to the observed accuracy drop. Based on this insight, we use 2 for our method in the next experiment.
8

---

CIFA8-10 10
Feshion MINIST
CI
CIFAR 100
SPLv1
Spliified
Servuition Lime
Sinulation Line
Serdiitte lame
Figure 2: Performance Under Stragglers, where we for MU-SplitPed.
Performance under Straggler. In this subsection, we evaluate the resilience of MU-SplitFed straggler effects by comparing its convergence performance against baseline methods on four datasets. Here, we introduce random delays following an exponential distribution to emulate straggler-induced latency. Figure presents the accuracy over wall-clock time for all methods. Across all tasks, MU-SplitFed consistently achieves higher accuracy and in less time compared both vanilla SplitFed and GAS, highlighting its efficiency in mitigating grraglerrinducee delays. Notably, on both CIFAR-10 and more complex task CIFAR-100 100, Si-SSpiiFee main- tains a fast and stable convergence trend, while GAS exhibits slower convergence and less con- sistency. One possible reason for these scenarios is that GAS supports asynchronous updates, its activation generation step scales poorly with the increasing size of the label, introducing significant computational overhead that limits its efficiency in straggler-pron settings. In con- trast, MU- SW-SplitFed maintains lightweight computation on both server and client sides, which allows efficient parallelization and better utilization of system resources during straggler delays.
2
Interaction Between Cut Layer and Server Itera- tions. To fully explore the potential of our proposed unbalanced update in reducing the communication round, we fine-tune the OPT-1.3B that enables more types of model splitting. This allows us to more
Global Global Communicttoon Round
Cut Layer
600
L.
16
400
thoroughly explore how to jointly select 7 and cut layer L, to optimize communication efficiency. Fig-
ure 3 shows the total communication round required to attain 85% accuracy across different cut layers and values of т. For fixed cut layer (e.g. Le 4), increasing reduces communication round by up to 33%, confirming the benefit of unbalanced updates.
200
15
Se Ver Iteratinns
Figure 3: Interaction between cut layer Le and server iteration
More interestingly, there a clear trade-off emerging between and Le- When Le is fixed, increasing т initially improves convergence, but excessive server updates eventually lead diminishing or adverse effects. Conversely, when fixing т and tuning the cut layer, convergence consistently improves as The decreases, indicating deeper server-side model is beneficial for model performance. Moreover, the optimal value of shifts higher Le moves earlier the model. These trends confirm our theoretical insight in Remark 4.I to fully exploit server-side acceleration, the model partition must scale with the number of server iterations. The dashed gray curve illustrates this joint optimization trajectory, highlighting that coordinated tuning of Le and t yields the most communication
0330
Momoy Castees
8330
1000
Medelr
Nithod
Figure 4: Comparison of peak memory cost for different methods for fine-tuning I.I.M.
convergence.
Memory Efficiency. To evaluate the memory efficiency of our ZO- based framework in the context of LLM fine-tune ine-ttning, we compare the peak memory usage on the client side. Specifically, we compare our proposed MU-Splin heppiiFec Fed with FedAvg 4 and FedAvg with LoRA [36] (FedLoRA) for fine-tuning OPT-1.3B model on the SST-2 dataset. As illustrated Figure. 4 FedAvg incurs peak memory cost of 8.02 GB the client. Fedl o ooRA, which reduces memory usage by updating only low-rank adapter matrices, reduces this 5.64 GB. Despite these improvements, both FedAvg and FedLoRA still require substantial memory to store gradients and maintain the full model locally. In contrast,

---

MU-Spli Fee reduces the peak client-side memory footprint to just 1.05 GB. This is achieved by storing only a partial model on the client and leveraging 20 optimization, which eliminates the need to store gradient information during training, further contributing to its memory efficiency.
6 Related Work
Split Federated Learning. SFL [ is a powerful distributed learning framework that enables scalable training across resource-constrainge sharing raw data with the server, SFL provides memory-efficiett and privacy-preserving solution for seoource-constainee devices. Recent advances in SFL have addressed key challenges from different perspectives. To mitigate the communication bottleneck, Chen et al. [37] reduces communication frequency by proposing a loss threshold that determines when to exchange information between client and Split Server. Han et al. [3] employ different local loss functions on the client and server sides, thus reducing the gradient information transmission rounds. Other approaches apply quantization or sparsification techniques to reduce communication costs in each transaction round. For instance, (38) leverages Top-S sparsification for both forward embedding and backward gradient transmissions, while [39] introduces randomness for further enhancement. FedLite applies Top-K quantization to compress intermediate features 40). For privacy purposes, several methods tackle model inversion attacks. ResSFL [41] and NoPeek (4) achieve attacker- aware training by integrating inversion score regularization term. Moreover, other strategies apply differential privacy on intermediate embedding features to provide privacy guarantees against label leakage [43]. In heterogeneous settings, methods like SCALA 44 and GAS [8] introduce activation concatenation and centralized training to enhance robustness and accommodate for varying client capabilities. However, theoretical research for SFL is still insufficient.
edge devices. By model partitioning on the client side without
(45) provides the first convergence analysis for sequential SFL., while (2) proposes an efficient update mechanism using different synchronization frequencies on client and server with rigorous convergence analysis for both sequential and parallel SFL.
Existing Straggler Solutions. The straggler issue in FL has been well explored, with asynchronous updates emerging one of the most promising directions [TO). Yet, asynchronous methods rely on stale information to update, which can lead to performance degradation due to outdated or inconsistent model information. To address this, ASO-Fed (23) proposed dynamic learning rate adjustment mechanism tailored to each client's 's training progress to reduce the staleness effect from straggler. FedBuff (26) enables efficient training by using buffer to store information from faster clients. Based on that, CA2FI (27) enhances the performance on heterogeneous data by adaptively adjusting model updates based data property. Similarly, FedCompass (46) adopts resource-aware scheduling policy that prioritizes clients with high computation capacity, thus mitigating the impact of stragglers. FedASMU (47] employs dynamic model aggregation with adaptive model adjustment to mitigate the impact of stragglers. Yet, existing strategies regarding the straggler in SFL remain limited. [9] 16 reduce the time delay by employing adaptive splitting strategies to balance the arrival times of activations. GAS (8) propose an asynchronous SFL framework that utilizes an activation buffer to generate activations based on the degree of bias, thereby enhancing the robustness of the algorithm.
7 Conclusion and Limitations
We propose MU-Spli tFdd, a simple and effective framework for mitigating the straggler problem in Split Federated Learning by introducing unbalanced updates server-side. The simple yet efficient unbalanced update strategy enables faster training by reducing communication complexity, thereby mitigating delays caused by stragglers. Notably, both our theory and experiments show that increasing the unbalanced update ratio yields a linear reduction in communication frequency. When lstrggier/ /leeverr the total training time becomes independent of the straggler delay. Moreover, our analysis uncovers a key connection between the choice of the splitting layer and the optimal т, offering practical guidance for further system design. These findings suggest that MU-SplitFed is a promising solution for enabling scalable and efficient training on resource-constraiigg edge devices.
Our work also highlights the potential of applying SFL for fine-tuning task of LLM,, where memory efficiency is an impetus need. In L.LM setting, SFL offers& a natural fit: edge or local servers can serve as client-side device, while high- perforrefore cloud servers act the central server. Although framework demonstrates initial promise in this direction by solving the bottleneck in this realm, how fully realize the benefits of SFL for scalable LLM fine-tuning remains an open challenge and needs further investigation.
10

---

Acknowledgement
Research reported in this publication was supported by the National Institute Of General Medical Sci- ences of the National Institutes of Health under Award Numbers R16GM159671 and IR35GMI and the National Science Foundation under Award Number 2045804, The content is solely the respon- sibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
References
[1] C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun. "Splitfed: When federated learning meets split learning." in Proceedings of the AAAI conference on artificial intelligence, vol. 36, 2022, pp. 8485-8493.
[2] P. Han, C. Huang, G. Tian, M. Tang, and X. Liu, "Convergence analysis of split federated learning on heterogeneous data," Advances in Neural Information Processing Systems, vol. 37,
pp. 103 476-103 544, 2024.
[3] D.-J. Han, H. I. Bhatti. J. Lee, and J. Moon, "Accelerating federated learning with split learning on locally generated losses," in ICML 2021 workshop on federated learning for user privacy and data confidentiality, ICML Board, 2021.
[4] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. y.
y. Arcas, "Communication--fficeent learning of deep networks from decentralized data," in AISTATS, 2017.
[5] Vepakomma, O. R. Gupta, A. Dubey, and R. Raskar, "Split learning for health: Distributed deep learning without sharing raw patient data," arXiv preprint arXiv:1812.00566 2018.
[6] M. G. Poirot, Vepakomma, K. Chang, Kalpathy-Crann Cramer, R. Gupta, and R. Raskar, "Split learning for collaborative deep learning in healthcare," " arXiv preprint arXiv:1912.12111, 2019.
[7] Mo, M. Malekzadeh, S. Chatterjee, F. Kawsar, and A. Mathur, "Enhancing efficiency in multidevice federated learning through data selection," arXiv preprint arXiv:2211.04173 2022.
[8] Yang and Y. Liu, "Gas: Generative activation-aided aided asynchronous split federated learning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, 2025, pp. 21 956-21 964.
[9] D. Yan, M. Hu, Xia, Y. Yang, J. Xia, X. Xie, and M. Chen, "Have your cake and eat it too: Toward efficient and accurate split federated learning," arXiv preprint arXiv:231 131663, 2023.
[10] Wang, Z. Charles, Z. Xu, G. Joshi, B. McMahan, B. A. y Arcas, M. Al-Shedivat, G. Andrew, S. Avestimehr, K. Daly, D. Data, S. Diggavi, H. Eichner, A. Gadhikar, 7. Garrett, A. M. Girgis. F. Hanzely, A. Hard, C. Hc, S. Horvath, Huo, A. Ingerman, M. Jaggi, T. Javidi, P. Kairouz, S. Kale, S. P. Karimireddy. J. Konecny, S. Koyejo, T. Li, L. Liu, M. Mohri, H. Qi, J. Reddi, P. Richtarik, K. Singhal, V. Smith, M. Soltamolkotabi, W. Song, A. T. Suresh, S. U. Stich, A. Talwalkar, H. Wang, B. Woodworth, S. Wu, F. X. Yu, H. Yuan, M. Zaheer, M. Zhang, T. Zhang, C. Zheng, C. Zhu, and W. Zhu, "A field guide to federated optimization," 2021. [Online]. Available: https://aaii..rrg/b///00..00
[11] A. Hard, A. M. Girgis, H. Amid, S. Augenstein, I.. MeConnaughey, R. Mathews, and R. Anil, "I "Learning from straggler clients in federated learning," arXiv preprint arXiv:2403.09086, 2024.
[12] A. Reisizadeh, I. Tziotis, H. Hassani, A. Mokhtari, and R. Pedarsani, "Straggler-resilient feder- ated learning: Leveraging the interplay between statistical accuracy and system heterogeneity," IEEE Journal on Selected Areas in Information Theory, vol. 3, no. 2, pp. 197-205, 2022.
[13] Wang, P. Nair, and D. Mahajan, "Fluid: Mitigating stragglers in federated learning using invariant dropout," Advances in Neural Information Processing Systems, vol. 36, pp. 73 258- 73 273. 2023.
[14] J. Park, D.-J. Han, M. Choi, and J. Moon, "Sageflow: Robust federated learning against both stragglers and adversaries." Advances in neural information processing systems, vol. 34, pp. 840-851, 2021.
[15] Tu, L. Yang, and Cao, "Distributed machine learning in edge computing: Challenges, solutions and future directions." ACM Computing Surveys, vol. 57, no. 5, pp. 1-37, 2025.

---

[16] J. Shen, N. Cheng, X. Wang, F. Lyu, W. Xu, Z. Liu, K. Aldubaikhy, and X. Shen, "Ringsfl: An adaptive split federated learning towards taming client heterogeneity," IEEE Transactions on Mobile Computing, vol. 23, no. 5, pp. 5462-5478, 2023.
[17] S. Malladi, T. Gao. E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora, "Finc-tuning language models with just forward passes," Advances in Neural Information Processing Systems, vol. 36, pp. 53 038-33 075, 2023,
[18] Z. Li, B. Ying, Z. Liu, C. Dong, and H. Yang, "Achieving dimension-free communication in federated learning via zeroth-order optimization," in The Thirteenth International Conference on Learning Representations.
[19] S. Liu, P.-Y. Chen, B. Kailkhura, G. Zhang, A. O. Hero III, and P. K. Varshney, "A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications," IEEE Signal Processing Magazine, vol. 37, no. 5, pp. 43-54, 2020.
[20] H. Cai, Y. Lou, D. McKenzie, and W. Yin, "A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization," in International Conference on Machine Learning. PMLR, 2021, pp. 1193-1203.
[21] K. Nikolakakis, F. Haddadpour, D. Kalogerias, and A. Karbasi, "Black-box generalization: Stability of zeroth-order learning," Advances neural information processing systems, vol. 35, pp. 31 525-31 31 541, 2022.
[22] S. Ghadimi and G. Lan, "Stochastic first-and zeroth-order methods for nonconvex stochastic programming." SIAM journal on optimization, vol. 23, no. 4, pp. 2341-2368, 2013.
[23] Y. Chen, Y. Ning, M. Slawski, and H. Rangwala, "Asynchronous online federated learning for edge devices with non-iid data," in 2020 IEEE International Confereenn on Big Data (Big Data), 2020, pp. 15-24.
[24] C. Xie, S. Koyejo, and Gupta, "Asynchronous federated optimization," " arXiv preprint arXiv: 1et:::9934..444 2019.
[25] Y. Chen, X. Sun, and Jin, "Communication-effcce federated deep learning with layerwise asynchronous model update and temporally weighted aggregation." IEEE Transactions Neural Networks and Learning Systems, vol. 31, no. 10, p. 4238, Oct. 2020. [Online]. Available:
[26] J. Nguyen, K. Malik, H. Zhan, A. Yousefpour, M. Rabbat, M. Malek, and D. Huba, "Federated learning with buffered asynchronous aggregation," in International conference artificial intelligence and statistics. PMLR, 2022, pp. 3581-3607.
[27] Y. Wang, Y. Cao, Wu, R. Chen, and J. Chen, "Tackling the data heterogeneity in asynchronous federated learning with cached update calibration." in Federated learning and analytics in practice: algorithms,, systems, applications, and opportunities, 2023.
[28] H. Yang, M. Fang, and J. Liu, "Achieving linear speedup with partial worker participation in non-iid federated learning," International Conference on Learning Representations, 2021.
[29] E. Chen, J. Zhang, S. Wang. C. Liu, and C. G. Brinton, "Parameter tracking in federated learning with adaptive optimization." CoRR, 2025.
[30] J. Zhang, E. Chen, C. Liu, and C. G. Brinton, "Dpzv: Resource efficient zo optimization for differentially private vil," arXiv preprint arXiv:2502.20565 2025.
[31] H. Xiao, K. Rasul, and R. Vollgraf, "Fashion-m machine learning algorithms," arXiv preprint / 077477777777777 2017.
novel image dataset for benchmarking
[32] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, "Cinic-10 is not imagenet or cifar-10," arXiv preprint
2018.
[33] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images." " 2009.
[34] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., "Opt: Open pre-trained transformer language models," arXiv preprint 22X5:2205.01066. 2022.
12

---

[35] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, "Recursive deep models for semantic compositionality over sentiment treebank," " in Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 1631-1642.
[36] E. J. Hu, Y. Shen, P. Wallis, Z.. Allen-Zhu, Y. Li, S. Wang. L.. Wang, W. Chen et al., "Lora: Low-rank adaptation of large language models." ICLR, vol. 1, no. 2, p. 3, 2022.
[37] X. Chen, J. Li, and C. Chakrabarti, "Communication and computation reduction for split learning using asynchronous training," in 2021 IEEE Workshop on Signal Processing Systems (SiPS). IEEE, 2021, pp. 76-81.
[38] B. Yuan, S. Ge, and W. Xing, "A federated learning framework for healthcare iot devices," arXiv preprint arXiv:2005. 05083, 2020.
[39] F. Zheng, C. Chen, L. Lyu, and B. Yao, "Reducing communication for split learning by randomized top-k sparsification," arXiv preprint arXiv:2305. 18469, 2023.
[40] J. Wang, H. Qi, A. S. Rawat, S. Reddi, S. Waghmare, F. X. Yu, and G. Joshi, "Fedlite: A scalable approach for federated learning on resource-constrained clients," 2022. [Online]. Available: https://arxxvv.rrgabb//2200 11tp
[41] Li, S. Rakin, X. Chen, He, D. Fan, and C. Chakrabarti, "Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning," in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10 184-10 192.
[42] P. Vepakomma, A. Singh, O. Gupta, and R. Raskar, "Nopeek: Information leakage reduction to share activations in distributed deep learning." in 2020 International Conference Data Mining Workshops (ICDMW). IEEE, 2020, pp. 933-942.
[43] D. Xiao, C. Yang, and W. Wu, "Mixing activations and labels in distributed training for split learning," " IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 11, pp. 3165- 3177, 2022.
[44] Yang and Y. Liu, "Scala: Split federated learning with concatenated activations and logit adjustments," " 2024. [Online]. Available: https://axivvorr/bb////// 2ot5:///
[45] Y. Li and X. Lyu, "Convergence analysis of sequential split learning on heterogeneous data," arXiv preprint arXiv:2302.01633 2023.
[46] Z. Li, P. Chaturvedi, S. He, H. Chen, G. Singh, V. Kindratenko, E. A. Huerta, K. Kim, and R. Madduri, "Fedcompass: Efficient cross-silo federated learning on heterogeneous client devices using a computing power aware scheduler," 2024. [Online]. Available: https//arxiv..rr//b/2220..9.1
[47] J. Liu, J. Jia, Т. Che, C. Huo, J. Ren, Y. Zhou, H. Dai, and D. Dou, "Fedasmu: Efficient asynchronous federated learning with dynamic staleness-aware model update," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024, pp. 13 9000-33 908.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The goal of our work is solve the straggler issue under Split Federated setting. Due to the frequent communication between client and Split Server, leading to the training time overhead. To solve this problem, propose unbalanced update mechanism that presents advantage in controlling the communication frequency, consequently reduce the total time delay induced by straggler.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of this paper is placed at the end of the conclusion part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
14

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We claim that all the assumptions we make are minimal and common in distributed optimization literature. The proof details are attached in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We conduct comprehensive experiments to testify our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
15

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use open access dataset for evaluation the performance of our proposed framework. We will later put our code on github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce)
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setup and results are included in the paper, and the choice of hyperparameters is put on the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The advantage of our proposed method is very large, that surpass the baselines under straggler. To verify our theory we only need to observe the trend align with theory finds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention this at the beginning of the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
17

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///parrpsss.cc///////////a
Answer: [Yes]
Justification:
The research presented in the paper fully conforms to the NeurIPS Code of
Ethics. All experiments were conducted with integrity and transparency, ensuring fairness, accountability, and respect for privacy.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses such risks.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We clearly cite every assets that we use in experiment section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, wwpprswittchoeeeoooppppoppppp thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
19

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowd-sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowd-sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method our work does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
20

---

Please refer to our LLM policy httpss//Deerriigccc///oorrrrrres
for what should or should be described.
21

---

Appendix
Communication Benefits of Unbalanced Update
23
A.1
Dimension-Free 700 achieved by Unbalanced Updates
23
A.2
Comparable Analysis
23
Preliminaries
24
Notations
24
B.2 Assumptions
24
3 Technical Lemmas
24
Proof For MU-Split
25
Proof of main theorem
25
C.I.1
One-Round Update on Server Side
25
C.1.2
One-Round Update on Client Side
27
3
Server-Client Combination
28
C.1.4
Justification for Corollary 4.2
29
Important Lemmas
29
D Proof for for MU-SplitFed
32
D.1 Proof of main theorem
32
D.1.1 One-Round Update on Server Side
32
D.1.2 One-Round Update on Client Side
34
D.1.3
Server-Client Combination
35
D.1.4
Justification for Corollary 4.4
36
D.2
Important Lemmas
37
Additional Experiments
39
Choice of Hyperparameters
40
22

---

A Communication Benefits of Unbalanced Update
A.1 Dimension-Free ZOO achieved by Unbalanced Updates
As shown in Table 2 the proposed MU-SplitFed can further achieve dimension-fre ZOO with convergence rate (1/VT), when T - d. By appropriately scaling the unbalanced update factor т to match the model dimension d, the convergence rate becomes independent of d. This is particularly significant for ZOO. where the parameter dimension d often dominates the denominator of the convergence rate and thus slows down training the model size grows. Large models exacerbate this issue because the increased d the denominator hinders convergence and adds communication overhead. 1i-Spitteed mitigates this by exploiting unbalanced updates, which not only accelerates ZOO training but also reduces communication costs. Specifically, the convergence rate improves from O OV///T) to O(I/VT), meanwhile, the communication complexity reduces from (dde²²) to (() Compared with other dimension-free methods [17 18 which often rely on strong assumptions, e.g. low-rank assumption, that impractical in real-world scenarios, MU-SplitFed provides a more flexible way towards this end. By introducing unbalanced updates into ZOO, we effectively remove the dependency on without imposing additional assumptions, making the method significantly more feasible in practice.
A.2 Comparable Analysis
We analyze the communication costs of MU-Spll S--liiFteed under different choices of and compare against two existing theoretical baselines for SFL frameworks. SFL-V1, introduced (2). serves as the fundamental baseline for parallel SFL architectures using first-order optimization. Reference (45) provides rigorous convergence analysis with the perspective of SFL in sequential update manner. To systematically validate the benefits of unbalanced updates, we present results across different configurations: T = represents the balanced update scenario where client and server updates with equal frequency, providing insight into combining ZOO with traditional SFL; r > 1 corresponds to our proposed unbalanced update strategy; and - is the optimal case that т scales to the same order of dimensionality d. That being said, the convergence rate is longer dependent on d, thus achieving the dimension- free convergence rate.
Communication Advantage of Unbalanced Updates. Compared to balanced SFL with ZOO (r 1), our unbalanced update strategy (r > 1) demonstrates linear convergence acceleration with respect to т. This improvement translates directly to communication complexity, where provides linear communication cost reduction from (dd/Me²) to Od/TTMe)) Specifically, unbalanced updates reduce total communication overhead by decreasing the number communication rounds required for convergence. When т + d, we achieve a convergence rate of O ((///M that eliminates dependence on dimensionality d, resulting in dimension-free communication complexity of (1)Me²) Comparison with SFL-VV. To the best of our knowledge, SFL-VI provides the first theoretical anslysis for parallel SFL under bounded gradient, non-convex, and non-iid assumptions. However, their theoretical results exhibit acceleration with respect to either the number of clients M or local update steps. In contrast, our convergence rate demonstrates faster convergence as both the number of clients M increase under the more loose assumption, e.g. bounded variance, consequently requiring fewer communication rounds to reach an e-approximation solution.
Comparison with S2.-V2. Our method achieves comparable convergence rates to SFL-V2 [45) where K is the number of local updates. While multiple local updates K accelerate convergence
Method
Convergence Rate
SplitServer Comm. Cost
Assumptions
SFL-V1
2
0(1/V)) 0(L/TTMK)
O(K/e)
/c/nnnCc/non-iid
SFL-V2 45
O(K/M²2
C.vo/iiCnnoniiid
MU-SplitFed (r = 1)
O (V/TTTM)
(dd/Me2
MU-SplitFed (r > 1)
(ldd/TTM)
Old/TMe)
b.w/NNCCnon-iid
MU-SplitFed (r + d)
(V/1/TM)
O(1/M²)
Table 2: Comparison of Communication Complexity
23

---

in FL settings by reducing communication frequency. they impose additional communication costs when applied SFL architectures. As demonstrated in Table increasing local updates K actually increases the total communication cost for convergence in the SFL setting. This counterintuitive result stems from the relay-based update mechanism inherent in SFL., where local updates exacerbate communication overhead between clients and servers rather than reducing it. Conversely, our unbal- anced update parameter т facilitates convergence without requiring additional communication rounds, achieving linear communication cost reduction with respect to т. This fundamental architectural advantage establishes the superior communication efficiency of our unbalanced update strategy over existing theoretical result.
B
Preliminaries
B.1 Notations
Table Notations this paper
Notation
Meaning
d m, M t, T
Total model parameter dimension Index, total number of clients Index, total number of communication round Index, total number of perturbations Index, total number of server iterations Global model parameters in the 1-th round Client-side model parameters in the t-th round Server-side model parameters in the i-th iteration Data sample the f-th round for m-th client Stochastic Zeroth-order gradient for t-th round Stochastic Zeroth-order gradient for i-th iteration Zeroth-order gradient estimator for client Zeroth-order gradient estimator for server Local loss function for client m Global loss function for SL or SFL
p,
T
x!
zt
Sop
cm G
fm(() J6)
B.2 Assumptions
Assumption B.1 (I-Smooth). For Vm [M]. the loss function fm. is bounded from below, and is L-smooth, i.c. Vx, y. (x) (y)l < L||x - yl|. Assumption B.2 (Bounded variance). For Ym [M], the variance of the stochastic gradient w.r.t. the client and the server is upper-bounded by o² and o² Specifically, for Vg Dm.
V x. (m; { f=) of
V Fm (x; ) Vmm(z))] < Assumption B.3 (Bounded Heterogeneity). For Vm E [M]. the global variability of the local gradient upper bounded:
<
B.3 Technical Lemmas
Lemma B.1. Let g(z) be defined as in ( We define the smoothed functioon fx(x) =E.ff(x+ Av) where v uniformly sampled from the Euclidean ball vdBd {o E Rd |||||| S vdj. The following properties hold:
[x(x) differentiable and E, (xx Vaa(a).
i8 If f(:) is T-smooth, then we have that
[Vf(z) - -fxx=l < Ax8
(13)
24

---

and
I
E=]lox 2d 2d-
+ x2d..
(14)
2
Remark B.1. By 13) we immediately have
2 L7
x2d2
(15)
2
|V/() S 2 +
x2d
(16)
The dual-paced model aggregation and model update SFL presents more challenge convergence analysis compared to the analysis in traditional FL setting. To address this problem, we decompose the convergence analysis into client-side and server-side, respectively. The following lemma reveals this relationship.
Lemma B.2 (Decomposition). Let x' = [x m7]] denote the global model at the tth training rounds. By applying Assumption
B.7 we have:
[xf(x +1
fxx']]
E
(xx((x) x xt)] + ||x x211
L
L
VV, f(x')( r1t1 x!1]
Blll
2. +111++++1[(
2. f(x)
x(11
B1
21
Kg
R22
R4
(17)
C Proof For MU-Split
C.1 Proof of main theorem
We now prove the main theorem of MU-Split, and defer all important lemmas to Appendix first restate the main theorem below.
C.2
We
Theorem C.1. Under Assumption B. 7 and B.2 and let the server iteration number be t. If the learning rates satisfy Me/T 1)s 1 min 641(7+2d.)) ToLTd }, the sequence of iterates generated by MU-Split satisfies:
El
E
Blf(x") f(x3) + 16n 1)d,0²
n Lyr + (1² 1/4)x d; 12 (18)
C.1.1 One-Round Update on Server Side
For Ks:
B[(V,, n, [xx x x1
-E v. fxxx),
G. (xt+':
-E(VV n. f(x'),
Σ
(v, +
1s
1=0
=
-EK v f(x'),
v
Σ
Va, - -Va, - - f(x))
VF
1=0
V Vz. - - vo..
nsT
B N.V.z.J6. f(x1)|
+ 17
2
2r
r-1
E
Σ
- 7 )|12]
2r
1=
25

---

Blll V x, (x))]
+
7/ E 2r
Σ
Vx x. phit
Ve, + V., f V
E
A
2r
i=0
< T
Blll
V f(xx (xx
||
Th
Σ
V B.
+ 2e E
V nx Jm
Vx, f1l)
2
1
11 E
2T
N7
B|[l x,
fxx
||
+ 175
E
V.,
2
+ 1/3
Σ
E
V
م Vz
2T
i=0
$
17
Bl| f
||]
T + 7
The
4
E
211
17 E 2T
V
2
X
1=0
A
T-1
N6 B[IV [ 4 T The E
VV., H
21 1,2 (8) + P v. (x* |1 + 412 d o -2
=161 + 7
B|
V f |||
4
2
22 + 2
NE E
Σ
(19)
2T
where we apply Lsmooth, Lemma B.1 (a, b) < substitute Lemma C.4fnto A1.
||a|2+1|01 2
|||||
and ||a + b12 < 2([1a|22 + [1611). and
For K2::
El|a - ==11 = n Σ Gx(x++
By 31
1=0
(1
V v, fx
119
Σ
El||G, - |1
Vz,
k=0
Similar to the proof in C.4 we substitute in 25 and 30 in order:
GI('''")))) V Vz. fx
B||l .., (71
26

---

4dg E [(x)) 4L2d,
Ellaa --1111]] 2d
12
P
P
P
2P
=1/11 + L²
VE[| (xx)) +
41.2/11
x2
2
4d= +x,((
dL.dd
E|ll E1l - +1 2d.0
?
122Pdd2
P
P
P
2P
Ard. p
471.'d
2rd.0²
Bill V fxx')
Σ
-
P
P
So
Td
TL
E 01|
fxx))
E11
EE
-1
P
P
4n T Td
T-1
B11 Σ
V 2, (x 113
P
LL2d
T ds
4m Td
(1
-x/113] S
Blll V [xxx
(xx))x )1()
P
+ I1ll
V fxx"
Further assume that 17 I 4L//d.//p* we have
16 rd,
rd
E| |1 21
f(')
4n Σ
V Va, fx (20)
p
C.1.2 One-Round Update on Client Side
For K3,, we have
E(Vz 2 (x'), x. -x0)
-E(V fxx),, G. g²)}
-E(V ( (x²): -Te (V (Va, fí - Vv. fx')) + vx.f(x')))
-E(V Th V se fx'), TTc V
nc f V {()) 7. V f(x' )1
The ETIV s. (xx') )11 + 24
Blll (V fV
V f') || 1 The V fill x.
2
2
s - 2
Ell V n. f(x 1|
7e 22
12
(21)
4
2
For K4:
E E1
Go
(x': =²)
- V 1 + 1 || V
Substituting 25 and 30 in order, we have
E G
f
4d
4L2d.
+ 2d
L2X2d0
1
Will V n. f (xt))
-
}
10
P
p
2P
Ade
4L
x 11 + 2d
1222'd
[ll V v.. f (xx) 1191
Bll|
P
P
P
2P
n2
L
f(x |19
4 tit - - 21 + - X2
AL
p
2
27

---

Ane
Bl| 1 - -x(]]]] + o
[x'))|1
p
p
So
4n
E|| B1|
211131
SM
V fx
2
20J f(xxxx [1
41 2 de
21 doo
20
(+1
x-d1 11 +
P
P
Lo
4n
(1
[B[1
6+1
x''
S
RR
E
V x
[xx )|
+
P
n.
P
P
Further assume
ne
and have
LV /d.//
8n de
4n.d.02
Blll
x'' V
fxll+
+
f [1
(22)
p
P
C.1.3 Server-Client Combination
We now substitute (19). (20).
into 17
Blf(x*+1) - fx')]
+ -
EV(V u. fxx'), v, 2111
E[|| +1
x.J fx'),,
E1l - =1'1
2
2
K2
K2
L
L3 + Pdd./P) - NaT
Elll V v. [(x))
77 T 2
2T L L
4
P
K,
Sn LTd
4 LTd,0²
E
E n,
[xx*)1
+ 2n
LI(I
Σ
(x
V
2T
P
P
i=0
t=0
R22
7/ [||] V (x))11
ne 1 2d
Te VRl
+
2
2
4
4n Ld
2n. Ld
+ n: LE Vo. f1
+
B N. f(x'))11+
p
P
(16) L(T + ds/P)
7
[[[[ x, [xx1)]1 +
L + 1)d,0²
2
P
+ m + 1/4)x2 d
4m Lde Me |Ell V a. f(xx)
Te L2X2
Ld.
4
P
(10)
I 4
TT El V n, f(x' 1|
4m T 12 + 1)d,0?
+ noT 12n (2
+1/4)x2
P
me "' V x.
||
Nc =2 +
2n Ld
(23)
4
where in (i) we applied (9
in (ii) we assume T7 S 7T.
1 index on terms of
ns assume 775 I Me S 2L to remove the term E El V
dhd and combine the terms of
IV.. fxx))1 and f(x))) 122 In (ii)), we let
P
7s S 6AL(TP+2d) 2d.)
And
P
me <
16Ld
28

---

To combine the squared norm of the server gradient V f and client gradient E V x..f f(.)
we define the universal step size 17 := T/.- and let t7: T7. Rearranging the terms in 23 we have
7T (
B f(x))l - [xf(x') - f(x*+1)] An7' L + 1)
4
P
+ (2n + 1/4)) + " 3, TL?X'0
2
P
< B[f(x)) - f(x TL + 1)d,o?
Ell Vxf
4
P
+ n +2 + - -122X 21 Ldooe
(24)
P
Take the average from t- 0o - 1 at both sides:
Σ
T
[ ((x²)) L + 1)dao²
P
t=
+ + + ? T +
P
n -با
[lf(xx) - f(x") 12 L + 1)dao?
+ 4T + 1/4)>2d 1222d0 ShY Idd
P
where in the last step we divided both sides by 4 Let P = 1, and we complete the proof.
C.1.4 Justification for Corollary 4.2
To further simplify the result and achieve the optimal convergence rate in Corollary 4.2 again, we assume 17 S _. We also optimize upon v) to get the convergence rate. Let 17 VdrT derive that
1
Σ
T
xx'')
4vd VTT
Blf(x") f(x")l )+
48Ld,03 VdrT +9L2X + L2 8VFLd
T t=0
VaT
Let de / and d, = d - d/ VF, and further let
2
Thus, have
4v7
48L
9Vd
8L
-
VTT +
VTT
VTT
VT
The convergence rate is seen to be O V-T
C.2 Important Lemmas
Lemma C.2 (Bounds on the variance of Zeroth-order Gradient). Under the same condition as Lemma B.I and consider the stochastic Zeroth-order Gradient, we can further bound the variance of the stochastic Zeroth-order Gradient by true gradient at the beginning of the local iteration and the local update distance.
4d P E 2.
4L2d. Elll (
(26, -V V., + N
f(x))1+
P
29

---

2d
1222d0
1 -El SeIIV,,,A'tE x
P 2P
(25)
proof:
We use multi-perturbation to calculate the Zeroth-Order Oracle:
& Ein Ds ( , where Sوi is the stochastic Zeroth-Order Oracle for one perturbation. Then, the X-smooth function is represented as Emp, lopp (x2, ; = [=, we have
By Lemma B.1
L2 x2
Eulloshi (2 o 24. (-)(1 2d, IV F(x 24: st)
2
Thus we have
E ElI - Vn, Vn, (x
11 (2 1/ Va, f (xt,
1 V
E Ellgg tit (at 2-91
pa
I
2d,
Flaa +)))))
2 X2(?
- V (x
p2
(E[
+ 03,) +
_
L2
- =1
S (x 21:11
2
p=1
1
L
Ed V f 2 + 2d +
d2
El[ fa 2 )19
(26)
P
2
The bound for the squared norm of the variance is:
V[, x 2112 =EllV 2, f x1:1) - f(," - V fz f(x' I|I
<2E| v., (1 24/1) Vz. || + 2B Vz,,
<2L B11 - 2 2E|1| V x, (x')]]
(27)
Substituting (7) into (6) and we finish the proof.
Lemma C.3 (Bounds on the norm of the Zeroth-order gradient estimator).
B[| Ghd (1 (1 1
4(d, + P 1)
(L2 + PP11) 1) B Bllla* -1 +
2d 1.222
+
(28)
P
P
P
2
proof:
follows that
E[IGG (f 2 12] = (16 - V Ve,
(29)
From Lemma B.I we have
L²
E V V (2))))))))))0011 V f(z 2 111
x2d2
2
L2
<2E|| V, x Vz, (xx) + Vz, fx')() +
X2
2
1
4 21d1) vz,f(x') |1 4B11/ ((())
X2
1.2
4E V fx')
x²
(30)
2
Then we can finish the proof by combining Lemma
C.2 and 33
30

---

Lemma C.4 (Bounds multiple update steps(Zeroth Order)). If S
JP (P
have
16 + d
8n 8
BII
+ 2
P
P
0
proof:
We first apply the update formula:
E Blllaga - x=[13]]
Σ
G ( (1
x/ )17)
f=
By the property of martingale difference sequence, we have
E1
Σ
G (w x. 112)
j=0
<2E||
V 2, f (zt - +1) (
Σ Gl 21)) Vz,
x 11
i-1 Σ
22
B||l 22, 2 +
BlI (+
x - +
(31)
1=0
1=
We thus have
E a
1
1=0
22
Ts
Bl|l V n. fx
2( x1)) +
BilGG ( V V2. fi x
i=0
j=0
22 2
E
V 1 (2 250
Σ
Bll (1g Ve, "f,
1=0
where the last inequality is by the following equations:
T-1 ,X, < PX,
i=
1=0
j=0
And
Σ 1=0
j=0 km
Substituting in (25)
1=0
P72
<21
Blll V x, fl (x), +(:))
P
k=0
4dl
4T2d
207.ds ds
+2 2 7
Ell n. 1 +
E
Elllo
-
p
P
P
2P
31

---

Further substitute in 30
11
1=0
PT
L2
22
x1 +
4E V v2,
x2d2.
T-1
4dg
4I2d
- x=111 + 207dd 1 x2
+211 T
E|| 11
Bl1
P
P
2P
8n, 3 d
2 + d./T) T-1
V ++.
B1ll ff - x1,1191
P
4n on
+ +
P
Rearranging the terms, we have
L1 ds/T)
Σ Bl1 x111211
P
S + da/T)
B B|[
121
+ +
P
P
where we moved the term B||aat
[ to the left in the last inequality. Let S
rd,/p'
we have the coefficient on the L.H.S larger than 3. Thus, we complete the proof.
D Proof for for MU-SplitFed
D.1 Proof of main theorem
We now prove the main theorem of MU-SplitFed, and defer the important lemmas to Appendix We re-state the theorem below: Theorem D.1. Under Assumption B.7 to B.3 consider a framework with M clients, and let the
D.2
server iteration number be т. If the learning rates on client and server satisfy na/T ns = 17 S
min
12rld, }, the sequence of iterates generated by MU-Split satisfies:
M
/120L
4
T EL
B V
E[s(x²) [(xx(( 24m (g Mg/M) + 2d.)
Tng
+ + LLg (1/7 + 8,22 2n
+
"2ngrLdde AnghTIde + x
(32)
M
M
Similar the proof of MU-Split, We begin by analyzing the update client and server side, respectively. By (17). we bound one-round update K1, X₂ on the server side, and Kz. Kr, on the client side.
D.1.1 One-Round Update on Server Side For KG:
EKV,, n, [(x' x
M
r-1
-E ((x*),
M
Σ
Σ
n
Gt.5 (i. et
s.m
32

---

M
r-1
=BK[V vV, fxx),,
ng
Σ
Σ
Smi - V2,f(x²) + Vx.
17
M- m=1 1=0
M
=EX 77 f(x'),
s - V2.Fin)
71 TB V x. f(x')) ||3]
M V
1=0
M
V s - Verffn) 11"
7g T
E1
V
7g E
Σ
2
2M
M Σ
Ig E
V 3. m.d
+ Rill V n. [xx*)|
2M
0=
M
V x, [(xx)
7/ E +
ΣΣ
V plii
x. m.d - Vanlmi phit + Va, pli
2
2M
M r-1
1g E
Σ
Σ ==
Vo,Smin Sui
2M2T
M T
S NoT
R
[ [xxx) (xx))1
+
7o M2T
V Vo.
Var pia
2
m.d
m==1 1=0
M r-1
M --1
+
?g 2
E
ΣΣ
V x. flii
Va, fm)
?/ E 2M
Σ
pt,i m.)
1m0
M r-1
I 7T
V
f(x
||
+
ng
E
plat
V Vo,
pt.i)
2
M
m=1 1=0
M т-1
M T-1
+ 77
Σ
R
plai Va, sth |1]
n7
EE
vx plas
M
2M
m=1 1=0
m==1 1=0
< No T E
V
+
7g r 122
Mg L
M Σ
E
2
M
M --1 Σ
ng E
2M27
x. m.d
1=0
M --1
V [xxx) + 79 2.2
Tg EF
ΣΣ V z A
S 2
2M
+ 1g9 1.2
2T3 + 2 || 119 + 273 P
+ +212+=12x22t)
P
= (22 173 + 2dd./ P)=
Tg
E
V 1(
11 x222
+ 24m T?2 + 2dd./
2
snani
M
+ 2m 3-3 1
2g
Ug 2M2T
pl. s.
P
m.)
where in the last step we use Lemma
D.3 for Al.
For K2:
MMG
E Bl|l
4+1
Ti 1 E
Σ
Gli
(xh1 Li et
M
1=0
33

---

Mg
le E
ΣΣ
m.a
2
T M
Σ
E
G
Va, Va, pl.
|2
M
imo
Substituting 34
and
40
in order, we have
M
11
E
G
V
V2, m.A
M
n
M
1
(6E[)) V., f(x))
+ 6E2 + 66121|11211
2 s,
11 + x2
M
2
6d:
dgddg
Ed,E²
2d
+ L222d0
[ill V 2.
f(x' || +
B1|l=o 2, - x 112
+
P
P
p
P
2P
M
6d
n2 fxx'x 13]+ + dL
EdgE²
2d
13ll
1)mml?]++
+
M
P
p
P
m=l
i=0
nigls
(dd
|| + e
p + 2d
E||
M
6ng
M
Bllk - m1
PM
2=0
We then use Lemma D.3 and assume that 1)s S
/P
follows that
L/24rd.
M
ΣΣ
E
Gh. a.m
V
s ||
2
M
m=
0=0
2
6dg
6d,e²
2d
2 (xx))1
M
p
p
M
(7
P) 241 P))
Mr 2
m=1
8 2
+ 211
P
2Td P) V f(x)) 6(72 + +2rd,/P)e+ T2I2X2 4Td
6((-
M
P
where in the last step we use the fact that 2 1.
D.1.2 One-Round Update on Client Side For Ka::
BV x
M
=E fxx'')
"19
Σ
Gri
no c,m (x <n)M
.M
m=1
n. f(x') -V.
799
Σ To (Va fmn fxx + V Vx. f(x*)))
M
m=1
Tla
-BlKyTobe 7g V N. f(x'),
Varlm. V Virfil)) no no ((.
M
M
[1
ng 2M
E||
(V (VaaSt - Von [12]
2
m=1
34

---

1g E
m.>
"Ib"..llIV x [xx)]
2M
Te
Blll f
Tg 2 d2
lg E 2M
V J
fo
2
For Ka:
2
M
B
1+1
x=]1
Te E
Σ
Gt Cr ex Sml
M
Th E
Vi M
M Σ
Veelm
E
(G Gemm
M
Te
2 2 M
M
E
V
Jmm
Σ
E
G
V
M
Substituting (34 and (40) in order, we have
M
1
E
|| c.m
V
Smm,ll
M
m=l
Σ
E
6de
V ((x))11
Edg
2d.
L2X2d0
M
P
P
P
2P
1 x2(0)
L2
(E||
V
n.
[xx))
+ 66
M
2
6d
+ 6d
2d.
1222'0
+
[ V (x*))]
+ 2P
P
P P
Te
(d
BlIV (x')) [1
6d
+ 2d.06
M
p
D.1.3 Server-Client Combination
Putting together:
|l - [(xx)
< 12 L2(T3 + Pdds/P)-
!g
LE fxx''
7g T X2 + 24 12 (T3 +rdd//Pe
2
8ngu1 T.. +
M T-1 Σ m=1 1=
n L M
M
+ 22 3
T/g//s E
V
E
P
22
m.
m,y
m=1
i=0
66(=2 + 2Td. v. (x)) 6(72 2rd./ PP)2 T. M
4Td,0² p
2
Te
Bill n.
fxx'(()) ||
ng d2
Mg E 2M2
2
ne m.
L
M
21
(3de
ed
d
+ 2M2
E
V
Ne
ft
m.>
M
P
B
V x. f( || +
P
+
P
m==1
Gngg (0 (4 + // + 2Tds/P)
E[|]v,,[(')113
E
2
+ 6n (4n.TL M,4TL+ + 2Tdd./P)e
+ +
+ 11
T1
nz MM) x2
P
35

---

+ 3ighic Ldc
31g Ld NgieLdo02
No | [x*))
MP
2
MP
MP
4
7g E n, f
+ L/ 2rde/P)e²
(4 L
4
An (21). 7 L + Mg M 0 + 11/T 811 + 2g MM) 22
P
3n Ldnt²
nonoLL>dd
BIIV 1(x)
To
4
MP
MP
4
where we assume
MP 12Lde
7 S
Tec
120L2(7 + 2rd./P)
and combine the terms.
To combine the squared norm of the server gradient ||| 2, F and client gradient E[I V., F11, define the universal step size : 17s and let 7/c T/. Rearranging the terms, we have
= {x - f(x' + 67g"7((477 + Mg/M)L(T² + +2e_ 4n, (2 ? I. + mg /g/gg//// o 1/7(1/T + 811² + // ?2122
4
P
BoTT Lddot
MP
MP
4
Taking average from = to T - 1 at both sides:
UT
E
VxI
+ -(xx
MoorLi4n///MMFFPF 2rdd///
4
1=0
4n (211T L + Mg Mr/Mordd.o + + + / 222266
32 1² Ldoo?
+ LonTL
MP
MP
4
1
V < 4
Σ 1=0
Ei1
B[f(x") - f(xx')] + ( + Pdc
T
Tn
16n L + ng/M) -1/T+
21 7
12nghfLdde
Ang
L2
(33)
MP
MP
where in the last step we divided both sides by DF Let P 1, and we complete the proof.
D.1.4 Justification for Corollary 4.4
The optimal convergence rate is achieved by optimizing 33 w.r.t 17 and ng. solving which gives "4 = VTM and 1) +L Since de, de is typically very large, and is relatively small, we can assume that - S dg. Thus, we have
4Lvd VTTM B[S(x") B[J(x") [(xT))+ 24(4/VT MI
T
TVT
16(2/VT + VF/VM)/L//VO22 + (1 8/dT + /V
TVT
12/F(d./Vaje' AVF(d./Vajo I.2220. VTM
VTM
Since d, T. M, are positive integers and L are typically large, we have that
(1 (1+8/dT++ 8/dT + + L2X2d < 1112>
36

---

Let d/d. = so that de = vd/T and dg = d - vd/T. and further let
X² 1
TTd5/212
Finally, we have
T
VTTM Exf(x") - -[(xx) + 8vd(3e2 + 200)
4LVd
32vd(3e² + o
El [
+
VTTM
7T
4(3c² + o0.) 6Vd
VTM
TT
We can conclude that, the overall convergence rate is O VrTM
vd
D.2 Important Lemmas
Lemma D.2 (Bounds on the variance of Zeroth-order Gradient). Under the same condition as Lemma B.1 and consider the stochastic Zeroth-ordes Gradient, can further bound the variance of the local stochastic Zeroth-order Gradient by global gradient at the beginning of the local iteration and the local update distance.
6dg S
6L
BlllG"''("h : V.,
[(x))
E
xzym [12]
P
8,
c
2d.0 ?
L2
P
P
2P
P
(34)
proof:
First notice that G* 5th) $ Ep=r (xip(x (xth et and Eup,st, (xt:((xx11) et Va, (x
By Lemma B.1
we have
1
Bullight ( 2d, IV F(xth:
X2
2
Thus we have
G 5th) Va, (((x))1)
ti (xx) - V & (x
p2
Elllg (xx)) - ア V v (1
p=1
2d, 2d| 2, F (x 5th |1 + 2
22
P Bill V n. (xc)
2
2
2d. v, 1
22
P BBll ar, ff (1
L
2d, 11 + 2d
+
2
BIIV
(35)
2
Now we bound the squared norm of the variance:
V 11: =B|| V Shit plit
Va. Va,lt - fx,)('') + v
<3E V pt. Vz, 1tll + 33Il 2t. - vx,f(x')) ||2] + 3E V
f(x
[|2]
<I1"2|112511 xtm 122] + 3E| V 1(*)) +322 32"
(36)
Substituting (36) into 35 and we finish the proof.
37

---

Lemma D.3 (Bounds multiple update steps). 1f I
JP
we have
+L1/2((P4d./7)
Bll s. --'mm
+ 12 - =
+ +
+
P
proof:
We first apply the update formula:
El| a.m
Σ
(nl)
Ell
Σ
G (2 11
By the property martingale difference sequence, we have
E[I
Σ
G 11
[
12
Bll|G% aa, Mmilll [12]
1=0
1-1
$ 2i
Bill V 1
11
+ 2
Σ Elll
G
(37)
5,m
j=0
5=0
We thus have
T-1
E o
11 2
(m/))
Bll R., p m.
12
E G 8,m
Vv fiud 112]
5=0
$ 2
B1| م 11
(7h))
T9 [ s,m
Vz. phit 1m..1 113).
(38)
where the last inequality is by the following equations:
ΣΣ
PX,
1=
1=0 1=m
j=0
And
+-1
<
Σ -X,
==
1=0 =m
1=0
Substituting in 34
E
2 m 17
5,m
PT2
<2(n))²
E
V p [1
P
6dg
6L d,
6d.c²
2d.0²
L2X'
+2(n()
Ell
V
+
Bll
E
+
P
P
p
P
2P
1=0
(39)
From Lemma B.I we have
E| V a. p [
38

---

12
22B| 2, 11:
2 22
L2
V2 - v + Vaysm [x, + Vn, |||]
+
X²de
2
<6E||V
V. -
6B V S
+ [,
[xx))|+++
-/222 X2
12
I d2
Vz. 6c + 6L =/m +
(40)
2
Substitute into (39
8.m
2mmmll
0=0
PT
12
<2(11))
[R[ x, f(x))1 91 + 67. 6T.
6R
-1, 66
22
P
2
6dg
dLdd
6d,edd
2d.07
I2APddg
+2
Ei|l V s. (xx)
E
==,,mll 11
P
P
P
P
2P
12(n))) ( ds/T
12(m5)2 + d./T)
T-1
E V
i
s'mmml 1!
P
P
s,m
12(m)) + d=/
2(2b)
+
+ (1:)
P
P
Rearranging the terms, we have
12(m)) 222 + d./T)
(1
B B11a22 am - -='amml' 11
P
(2( + + P| v
12) Pd
2 (1)) 23
P
where we moved the term E|| - x.,mm || to the left in the last inequality. Let nr s
(1//74(7 we have the coefficient on the L.H.S larger than } Thus, we complete the proof.
E Additional Experiments
To investigate the interplay between splitting strategy and unbalanced update frequency т, we conduct an ablation study examining various combinations of values and cutting layers using O.T-1.3B on the SST-2 dataset. To isolate the effects of our core mechanism from confounding factors inherent in federated settings, such as data heterogeneity and client variability, we employ a simplified MU-Split configuration with single client.
Table shows the total communication round required to attain 85% accuracy across different cut lay- ers and values of T. For a fixed cut layer (e.g. L. = 2), setting - = 4 reduces communication rounds by more than half compared the baseline without unbalanced updates. Crucially, our results reveal a clear trade-off between and Le. When L, is fixed, increasing 7 initially improves convergence, but excessive server updates eventually lead to diminishing or adverse effects. Conversely, when fixing 7 and tuning the cut layer, convergence consistently improves as Le decreases, indicating deeper server-side model is beneficial for model performance. Moreover, the optimal value of T shifts higher as L. moves earlier in the model. These trends confirm our theoretical insight in Section 4 to fully exploit server-side acceleration, the model partition must seale with the number of server iterations.
39

---

Table 5 presents the final accuracy after 1,500 training steps under different combinations of split layers and Consistent with the observations in Table for fixed split layer, increasing 7 initially improves the final accuracy but eventually leads to decline. However, unlike Table when varying the split layer, the highest accuracy consistently achieved with our theoretical analysis in Section 4 although larger can accelerate convergence, it does not necessarily yield smaller loss value, which is strongly connected to better final accuracy. practice, selecting appropriate values for and the split layer requires balancing multiple factors, including desired training time, target accuracy, and device memory constraints.
or
This pattern aligns
Table 4: Ablation study of influence of and cutting layer on communication rounds
Split Layer
2
3
4
T
5
38
19
18
4
18
16
22
20
33
23
22
26
22
32
12
22
32
25
29
32
16
21
29
28
40
36
Table 5: Ablation study of influence of 7 and cutting layer on final accuracy
Split Layer T
3
T 5
88.75
88.97
90.90
87.95
87.05
88.52
4
89.09
89.89
87.05
86.93
89.04
8
90.34
90.11
89.50
89.54
88.30
12
89.20
89.43
88.41
88.41
88.43
16
88.98
88.75
87.95
88.41
87.99
F
Choice of Hyperparameters
Table 6: Hyperparameters
PARAMETER
VALUE
EXPLANATION
fl
0.3
Global aggregation learning rate Server learning rate
Ts
0.01
Te
0.005
Client learning rate
0.005
Scale of perturbation for ZOO
B
32
Batch size
40


Reviews:
[{"review_id": "4HR1r5Ny2d", "reviewer": "Reviewer_yz22", "summary": "This paper proposes an MU-SplitFed method to address the straggler problem in Split Federated Learning. Unlike existing methods, the MU-SplitFed approach reduces communication rounds between the server and clients to mitigate the impact of stragglers. Furthermore, a Zeroth-Order optimization technique is introduced on the client side to enable training without back-propagation. Theoretical analysis demonstrates that MU-SplitFed achieves linear speedup. Experimental results validate the effectiveness of the proposed method.", "questions": "1.More experiments should be added. This paper focuses on the straggler problem, but the experiments and analysis under straggler scenario are limited. Only SplitFed and GAS are used as the baseline methods, other SOTA methods for straggler problem in related works should be compared. The experiments also should be conducted on more complex dataset and LLMs.\n\n2.The novelty of the proposed method is limited. Only an unbalanced update ratio /tau is introduced on the server-side, which is trivial.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.This paper has clearly described the proposed method.\n2.The analysis is sufficient.\nWeaknesses:\n1.The experiments should be improved;\n2. The novelty of this method is limited.", "was_revised": true, "final_justification": "The authors have done a good job in rebuttal. I raised my rating.", "rebuttal": "**Response to Reviewer's Novelty Concerns:**\n\nWe believe that our work identifies and addresses fundamental challenges in Split Federated Learning (SFL) that have not been properly analyzed in prior work.\n\n**The Critical Challenge in SFL:**\n\nThe goal of Split Federated Learning (SFL) is to reduce memory usage on resource-constrained edge devices by dividing the model between the client and the server. However, this model partitioning introduces additional communication overhead during each local update round. As a result, SFL faces a fundamentally different straggler problem: unlike traditional Federated Learning (FL), where stragglers only delay global aggregation, stragglers in SFL block **every local iteration,** significantly hindering the training progress.\n\nExisting work on addressing straggler issue in SFL remains limited[8,9,16]. Most approaches borrow straggler mitigation strategies from FL, primarily focusing on minimizing server waiting time. However, under the relay-based update mechanism in SFL, this strategy proves less effective, yielding only marginal improvements. These limitations highlight the need to revisit the straggler problem in SFL.\n\n**Our Novel Solution - MU-SplitFed:**\n\nAcknowledging the limitations of current work, we propose MU-SplitFed that combines unbalanced updates with ZOO to address straggler issue from a fundamentally different perspective-by effectively reducing the number of global communication rounds. For further improvement, we introduce an eager forward method that minimizes server waiting time in each local round. The effectiveness of this method in reducing time delay has been empirically validated in Table 4.\n\n**Contribution I: Solving Straggler Issues in SFL**\n\n1. **Improved Convergence with Sublinear Speedup:** Our MU-SplitFed provides sublinear speed-up regarding unbalanced update steps, parameterized by $τ.$ Specifically, we improve the convergence rate from $O(\\frac{\\sqrt{d}}{\\sqrt{T}})$ to $O(\\frac{\\sqrt{d}}{\\sqrt{\\tau T}})$, without relying on strong assumptions such as bounded gradients.\n2. **Theoretical Solution to Straggler Problem:** We provide direct theoretical insight into how our unbalanced strategy effectively solves the straggler issue under Corollary 4.8, addressing the fundamental communication bottleneck in SFL.\n3. **Practical Effectiveness:** Our extensive experimental results demonstrate the advantage of our proposed framework in both small models and LLMs. As indicated in supplementary Table 2, our proposed unbalanced update can reduce more than half of the communication rounds, demonstrating the effectiveness of our algorithm in practical application.\n\n**Contribution II: Additional Theoretical Insights**\n\n1. **Principled Model Splitting Strategy:** Our work reveals important theoretical interpretation regarding the choice of cutting position, which coordinates with our unbalanced update strategy. Our theory indicates that for a larger choice of $τ$, more layers should be placed on the server side. The dependency between the convergence rate of ZOO and parameter dimension $d$ allows us to build this connection, enabling our theory to provide guidance for hyperparameter selection and making our algorithm more explainable.\n2. **Dimension-Free ZOO:** Our unbalanced update strategy can achieve dimension-free ZOO without further assumptions. Based on Corollaries 4.4 and 4.8, by appropriately scaling the unbalanced update factor $τ$ to the same order as the model dimension $d$, the convergence rate becomes independent of $d$. This directly addresses a key limitation of traditional ZOO methods, where convergence rates typically deteriorate with increasing dimension.\n\nIn summary, MU-SplitFed introduces a **fundamentally new perspective on handling stragglers in SFL**. We believe our contributions, both theoretical and empirical, establish a strong case for novelty and impact in this underexplored space.\n\n**Regarding your doubt of “straggler scenario are limited”** \n\nWe need to clarify that introducing time delay is a widely adopted approach to simulate the impact of stragglers in distributed learning. This method is well-supported in prior literature [8–14]. To establish a more realistic model that reflects the randomness of clients' time delay, we consider an exponential distribution model, which has been widely used to capture the computation delay for distributed clusters\n\nHere are more works that use this methodologies:\n\na. “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity”\n\nb. \"Speeding up distributed machine learning using codes.\" \n\nc. \"Coded computation over heterogeneous clusters.”\n\n**Regarding your doubt about lack of comparison with other SOTA methods**\n\nThank you for your suggestion. However, we must clarify that related work addressing straggler issues specifically in the Split Federated Learning setting is extremely limited. We have identified only three recent works tackling straggler issues in SFL [8,9,16] and we provide detailed justification for our comparison choices below. \n[8]: is compared in our manuscript\n[16]: This paper proposes a ring-structured SFL framework, which fundamentally differs from the standard SFL architecture. We believe that comparing algorithms under different ML system structures would not provide meaningful insights. \n\n[9]: While this work addresses straggler issues in SFL, it has significant limitations that make it less effective than our approach. The paper proposes an adaptive splitting strategy that dynamically adjusts layer allocation between clients and server. Specifically, the model is split into parts A, B, and C, where B consists of m+n layers. The client computes A+m layers (while still need to contain A+B), the server computes n+C layers (while still need to contain B+C), and m and n are dynamically adjusted based on client computational capacity.\n\nHowever, we identify two fundamental limitations of this method:\n\n1. Additional computational overhead: The dynamic layer adjustment requires continuous monitoring and reallocation, introducing extra computation time on the server side.\n2. Limited improvement potential: The effectiveness is constrained by the fixed size of the overlap region B. Since B's size is fixed, the potential improvement is inherently bounded and comparatively small.\n\nIn contrast, our MU-SplitFed addresses these limitations through a fundamentally different approach: we directly reducing communication rounds rather than merely minimizing client computation time. **Our key insight is that the computation disparity between client and server is considerably large in SFL.** Instead of optimizing the already limited client-side computation time, our framework intelligently utilizes the server's idle time by applying multiple updates strategy. As demonstrated in Table 2, **our method achieves more than 50% reduction in communication rounds,** which is significantly more effective for overall training time reduction.\n\nIf you are aware of other SOTA methods for straggler issue in SFL, please feel free to recommend them to us. We would be happy to conduct additional experiments for further comparison.\n\n**Regarding your request about “The experiments also should be conducted on more complex dataset and LLMs.”**\n\nThank you for your suggestion. In response, we have conducted and included additional experiments across a variety of datasets and model architectures, as detailed below:\n\n1. **LLM Benchmarks with OPT-125M (Table 1).** We evaluated our proposed MU-SplitFed against SplitFed_V1 [2] using the OPT-125M model on multiple LLM benchmarks, including **SST-2, CB, WSC, WIC, RTE, and BoolQ**. With $L_c=2$ and $\\tau=2$, MU-SplitFed consistently outperforms SplitFed_V1 across all tasks, demonstrating its effectiveness under various language understanding challenges.\n2. **Ablation Study with OPT-1.3B (Table 2).** We conducted an ablation study on the SST-2 dataset using OPT-1.3B to assess the impact of $\\tau$ and $L_c$ in reducing the communication rounds. According to the result, for a large choice of $\\tau$, less portion of the model should be assigned to client side. This aligns with our theoretical analysis on the interplay between unbalanced update steps and splitting strategy.\n3. **Image Classification under Data Heterogeneity (Table 5).** To evaluate the performance of our proposed algorithm under data heterogeneity, we added experiments on CIFAR-10 and FashionMNIST under non-IID setting with high heterogeneity (shard=2). The result shows that MU-SplitFed outperforms the baseline, indicating the robustness of our designed algorithm under data heterogeneity.\n4. **Ablation Study on Update Strategy (OPT-1.3B, SST-2).** To investigate the reason of  performance boost, we compared both FO and ZO versions of MU-Split on the SST-2 dataset. Specifically, we implemented MU-Split in both its FO and ZO versions. As shown in Table 6,7, we observe consistent performance improvement in both FO and ZO versions, indicating that the **accuracy gain primarily stems from the unbalanced server update schedule**. This further confirms the effectiveness of our proposed unbalanced update strategy in enhancing both the convergence and accuracy.\n\nTable 1 Best accuracy after 1500 rounds (OPT125M, Zeroth Order)\n| Dataset | SplitFed_V1 (ZO version) | MU-SplitFed ($\\tau=2$) |\n| --- | --- | --- |\n| SST-2 | 83.99 | 84.70 |\n| CB | 72.49 | 74.67 |\n| WSC | 55.18 | 58.35 |\n| WIC | 53.25 | 54.01 |\n| RTE | 52.91 | 55.78 |\n| BoolQ | 61.64 | 62.46 |\n\n Table 2 Required rounds to reach 85% accuracy of MU-Split (OPT-1.3B, Zeroth Order)\n| Split Layer | $\\tau=1$ | $\\tau=2$ | $\\tau=3$ | $\\tau=4$ | $\\tau=5$ | $\\tau=6$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| 2 | 38 | 17 | 19 | **16** | 18 | 18 |\n| 4 | — | 18 | **16** | 22 | 20 | 33 |\n| 8 | — | 23 | **22** | 26 | 22 | 32 |\n| 12 | — | **22** | 32 | 25 | 29 | 32 |\n| 16 | — | **21** | 29 | 28 | 40 | 36 |"}, {"review_id": "u7oKrV9icj", "reviewer": "Reviewer_WuCT", "summary": "This paper propose a straggler-resilient SFL algorithm by incorporating zeroth-order optimization.", "questions": "**Questions**:\n\nI have checked the proofs of MU-Split (Appendix B). Some questions and suggestions are as follows:\n\n1. Lines 110-111.\n\n   > Unlike FL, SFL requires the Split Server to receive and process distinct activation vectors from each client, which are concatenated rather than aggregated [8].\n\n   This is inconsistent with the description in Figure 1 and Algorithm 1, where only aggregation (no concatenation) is performed.\n\n2. Equation (17). Should the last inequality be equality?\n\n3. The forth inequality of the equation between Lines 561 and 562. Are $f_\\lambda$ and $f_\\lambda^{t,i}$ equivalent?\n\n4. The third equality of Equation (22). Should $f_\\lambda^t (x_c^t, x_c^t)$ be $f_\\lambda^t(x_c^t,x_s^t)$?\n\n5. The first inequality of Equation (32). Does $\\mathbb{E}[X^2] = (\\mathbb{E}X)^2 + \\text{Var}(X)$ used here? I think it should use Jensen's inequality. Please see Appendix C.2 in the following paper.\n\n    Convergence Analysis of Sequential Federated Learning on Heterogeneous Data, NeurIPS, 2023.\n\n**Suggestions**:\n\n1. Assumption 4.1. In MU-SplitFed, there are multiple loss functions. It would be better to specify which function is assumed to be $L$-smooth clearly.\n\n2. Lines 175-176.\n\n   > It is also worth noting that our convergence analysis does not require convexity of the loss function, adding to the complexity of the proof.\n\n   In my opinion, this note is not necessary (even a little misleading). If the convexity assumption is given, we can often get tighter bounds under other milder assumptions. Besides, the convexity assumption enables the use of more advanced techniques. So, it is not appropriate to say that the analysis of the non-convex settings is more complex than the convex settings without further discussion.\n\n3. Theorem 4.7. It seems that Assumptions 4.3-4.5 are not explicitly provided.\n\n4. Figures. It seems that the figures are not in PDF format? I recommend using the PDF format, which will help the paper be more professional.\n\n5. References. There are some papers which are cited as preprint, but have been published. For example, [2] has been published in NeurIPS, [29] has been published in ICLR.\n\n6. It would be better to see equations as one part of the sentences. For example, for Lines 573-574, please add one comma after Equation (23) and make \"Where\" in lower case. It would be better to revise the other sentences accordingly, e.g., Lines 615, 634, 673. The omitted punctuations should be added after the equations.\n\n7. Some equations are too long.\n\n8. Some sentences are unclear.\n\n   > Line 567: Similar to the proof in B.4, we substitute in (26) and (31) in order:\n   >\n   > Line 617: Substituting in (26):\n   >\n   > Line 618: Further substitute in (31):\n\nFor the formatting problems, I think that this page ( https://www.jmlr.org/format/formatting-errors.html ) of JMLR will be helpful.\n\nIf the problems I raised above are wrong, please free free to point them out.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n1. This paper combines zeroth-order optimization and SFL.\n2. The proposed algorithm allows unbalanced server-client updates. This alleviates the impact of stragglers in SFL.\n3. The convergence analysis of the proposed algorithm is provided.\n4. This paper is well-written and well-organized.\n\n**Weaknesses**:\n\n1. Convergence analysis of SFL. There has been one paper [2] that has established the convergence guarantees of SFL (including both SFLv1 and SFLv2) for strongly convex, convex and non-convex settings. Thus, for SFL, I recommend comparing the convergence guarantees of this paper with [2]. Furthermore, given [2], the claim that \"existing theoretical results in SFL either assume convexity or bounded gradients (Lines 51-52)\" need to be revised.\n\n    [2] Convergence Analysis of Split Federated Learning on Heterogeneous Data, NeurIPS, 2024.\n\n2. Convergence analysis of zeroth-order optimization in FL. It would also be beneficial to compare the convergence guarantees of this paper for SFL with ZO methods for FL (e.g., [17,18] mentioned in Line 47).\n3. Increased communication. The communication may be increased in the following two aspects: (i) In Mu-SplitFed, the clients send $H_m^t = \\{h_m^t, h_m^{t+}, h_{m}^{t-}\\}$ to the server, while in vanilla SplitFed (SFLv1), the clients only send $h_m^t$. This increases the communication. (ii)  In vanilla SplitFed, the clients and the server can perform multiple local updates before the aggregation of the local models. However, in Mu-SplitFed, the clients perform only one local update and the server can perform multiple local updates before the aggregation of the local models. This may increase the communication.", "was_revised": true, "final_justification": "This paper proposes MU-SplitFed, which is equipped with the unbalanced updates and zeroth order optimization.\n\nThe main concern is focused on the tightness of the bounds (i.e., $O(\\eta^2)$ terms).\n\nThis is mainly because the original SplitFedv1 (or SFL_v1) (in Thapa et al, (2022)) is equivalent to FedAvg in the absence of model partitioning. So, in my opinion, the bounds of SplitFed may be quite similar to those of FedAvg. However, this does not seem to be the case, based on the existing works (Han et al., 2024) and this paper.\n\nThis inconsistency makes me not really convinced of the tightness of the bounds provided in this paper.\n\nHowever, it is also unfair for me to reject this paper based on this concern given my unfamiliarity with the existing works in split learning and zeroth order optimization.\n\nI hope the authors will reflect on the reasons behind the inconsistency and discuss them in detail in the revised version.\n\nAs a consequence, I decide to increase the rating to 4, and decrease the confidence to 3.", "rebuttal": "We really appreciate the time and effort you dedicated to reviewing our work and providing such valuable insights! To facilitate a direct comparison, we have added **Table 3**, which includes the convergence rates and  communication complexity of our proposed MU-SplitFed alongside the SplitFed algorithms in [2] and other FedZO methods.\n\nTable3 Convergence/Communication Complexity Comparison Result\n\n| **Method** | **Convergence Rate** | **Communication Complexity** |\n| --- | --- | --- |\n| FedAvg(FO)[29] | $O(\\frac{F+1}{\\sqrt{MKT}})$ | $O(\\frac{F^2}{MK\\epsilon^2})$ |\n| SFL-V1(FO)[2] | $O(\\frac{F}{\\sqrt{T}})$ | $O(\\frac{F^2}{\\epsilon^2})$ |\n| **Ours: MU-SplitFed(ZO)** | $O(\\frac{\\sqrt{d}(F+1)}{\\sqrt{MT\\tau}})$ | $O(\\frac{d F^2}{M\\tau\\epsilon^2})$ |\n| FedZO[a] | $O(\\frac{\\sqrt{d}F}{\\sqrt{MKT}})$ | $O(\\frac{dF^2}{MK\\epsilon^2})$ |\n| DecomFL(ZO)[18] | $O(\\frac{\\sqrt{d}(F+1)}{\\sqrt{MKT}})$ | $O(\\frac{dF^2}{MK\\epsilon^2})$ |\n\n**Weakness 1:**\n\nWe initially avoided a direct rate comparison with [2] because their analysis is grounded in **first-order optimization (FOO)**, while our work is based on **zeroth-order optimization (ZOO)**. Given the fundamentally different gradient estimation paradigms, we considered the comparisons less direct. However, we agree that contrasting them will offer more valuable insights, and Table 3 now addresses this.\n\nWhile [2] provides an important theoretical foundation for SFL, we note two key limitations in their analysis:\n1. Conflict in the Strongly Convex Case: The results in [2] for the strongly convex setting rely on the assumption of bounded gradients. However, this assumption contradicts strongly convex, making the conclusions theoretically questionable.\n2. Loose Bounds and Strong Assumption in the Non-Convex Case: Although [2] does present convergence results for the non-convex case, it relies on **stronger assumptions** (e.g., bounded gradients), whereas we use **bounded variance**. Despite using milder assumptions, our analysis achieves a sublinear speed-up w.r.t $\\tau$, attributed to our use of more advanced techniques such as properties of martingale difference sequence and tighter equality-based estimations. Interestingly, [2] allows for multiple local updates in each round that should intuitively accelerate convergence. However, their theoretical bounds do not reflect this benefit, suggesting room for tightening.\n\nIn contrast, our analysis reveals that both increasing $\\tau$ and involving more clients contribute to convergence rate. This reflects a more refined understanding of the interplay between local computation and global coordination in SFL. \n\nLastly, thank you for catching our inaccurate phrasing in Lines 51–52. We will revise that section to more precisely reflect the limitations of [2] in the updated version of our paper.\n\n**Weakness 2:**\n\nWe have now included a detailed comparison of both convergence rates and communication complexities between our proposed method and existing ZOO approaches in federated learning, including one of the earliest works combining ZO with FL [a], and DecomFL [18]. The results are also presented in Table 3.\n\nFrom this comparison, we observe that our algorithm achieves a convergence rate comparable to those of existing FedZO methods. The key distinction lies in how acceleration is achieved:\n\nWhile FedZO and DecomFL rely on increasing the number of local update steps $K$ to accelerate convergence, our method introduces a novel acceleration mechanism via our **unbalanced update strategy** parameterized by $\\tau$.\n\nTo isolate the effect of $\\tau$, we consider the setting where $K=1$ across all methods. In this scenario, our algorithm exhibits a **sublinear speed-up w.r.t $\\tau$**, highlighting the role of unbalanced updates in improving convergence. Importantly, our method does **not impose any restrictive assumptions** that would hinder generalization. This allows our framework to naturally extend to $K > 1$, enabling multiple local updates and potentially further enhancing convergence.\n\nIn summary, this comparison highlights the advantage of our framework in exploiting the unbalanced update ratio $\\tau$ to improve convergence, while remaining compatible with and extensible to broader settings explored in the FedZO literature.\n\n**Weakness 3:** \n\n**Regarding your increased communication concern (i):**\n\nWe would like to clarify two key facts:\n\n1. **Per-round backward communication is reduced in MU-SplitFed.** In our proposed algorithm, during the forward pass, the client transmits $H_m^t=h^t, h^{t+},h^{t-}$, which involves three vectors. However, in the backward pass, the client only transmits a **scalar** value $\\delta_{c,m}^t$ as defined in Eq. (6). In contrast, in SplitFed-V1 [2], the backward pass requires transmitting the the partial gradient w.r.t. $h^t$. As such, although our forward pass involves slightly more communication, the backward communication is **significantly lighter**, making the total communication per round **comparable**.\n2. **Our total communication complexity is theoretically lower compared to [2].** As shown in Table1, our method exhibits a **lower overall communication complexity** despite slightly higher per-round communication. This is due to our **unbalanced update strategy**, which plays an important role in reducing the number of global communication rounds. In comparison, the method in [2] incurs a bad communication complexity even fail to reveal the accelerating of $K$ and $M$. Thus, our approach achieves a **reduction in total communication cost** over the entire training process.\n\n**Regarding your question(ii) about client local update steps:**\n\n1. Simplifying the analysis with $K=1$. To make our unbalanced update strategy easier to understand and analyze, we set $K=1$ in our current framework. However, our method is **not limited to this setting**. It can be naturally extended to support $K>1$. In such cases, during each local $K=j$ round, the client and server would still apply our proposed unbalanced update strategy. After completing $n$ local rounds, the client and server would perform global aggregation via FedServer. This generalization preserves the structure of our method while achieving accelerated convergence.\n2. No increase in communication complexity. Even with $K=1$, our method **does not suffer from increased communication cost**. As shown in Table 3, the overall communication complexity remains comparable to other FedZO methods. This is because our server-side update parameter **$\\tau$** plays a similar role to $K$ in reducing the frequency of global communication. Furthermore, extending our method to support $K>1$ is expected to **further improve both convergence rate and communication efficiency**.\n3. Lower communication cost for model aggregation. We understand that your concern is because global communication costs are comparably higher than local communication costs. However, our algorithm is designed to **substantially reduce the communication cost of model aggregation**. In particular, it allows us to allocate only a **small portion of the model to the client** (typically we choose $L_c=2$). This is supported by our theoretical analysis: by tuning $\\tau$, we can effectively minimize the client-side model size. Additionally, experimental results indicates that our unbalanced update strategy also lead to performance boost(see Supplementary Table 6, 7). In other words, our approach enables the use of a smaller client-side model while still maintaining competitive accuracy, thereby reducing the overall cost of global aggregation. In contrast, Han et al. [2] suggest that a larger portion of the model should be put on client to achieve decent accuracy, which inherently requires higher global communication cost.\n\n**Response to Questions:**\n\nQ1: Thank you for pointing out this inconsistency, and we apologize for the misleading description. The reason we originally referred to concatenation is that certain SFL designs [8,b] use concatenation of activation vectors instead of aggregation. However, we acknowledge that our current implementation follows the aggregation-based approach. To accurately represent our design, we will revise the sentence as follows in the updated version:\n“The straggler issue only delays the global aggregation in traditional FL. In contrast, in SFL, it also impacts local updates due to SFL's relay-based update mechanism. This motivates us to revisit straggler solutions tailored for SFL.”\n\nQ2: Thank you for your comment. To achieve a tighter bound, we apply an equality in the derivation. However, since the final result is expressed as “less than or equal to,” using the inequality is **technically not incorrect**. \n\nQ3: $f_\\lambda^t$ and $f_\\lambda^{t, i}$ are not equivalent. Specifically, $f_\\lambda^t$ denotes the loss function at the t-th global training round, while $f_\\lambda^{t, i}$ denotes the loss function at the t-th global training round and i-th server iteration. We will clarify this notation more explicitly in the revised version.\n\nQ4: Thank you for catching this typo, we appreciate your careful reading. We will correct it in the final version.\n\nQ5: Thank you for your comment. We would like to clarify that we did not apply the equation you mentioned in deriving the first inequality of Equation (32). Instead, our analysis follows the **martingale-based approach** under conditional expectation, consistent with the second situation in ***Lemma 4*** of **SCAFFOLD** paper[c]. However, we acknowledge that a scaling factor of 2 was omitted in front of the inequality, and we will correct this in the revised version.\n\nWe will follow your suggestions to refine our work in our future version. \n\nSupplement references:\n\na. “Communication-efficient stochastic zeroth-order optimization for federated learning” \n\nb. “SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments” \n\nc. “SCAFFOLD: Stochastic Controlled Averaging for Federated Learning”"}, {"review_id": "RhInOP7DBl", "reviewer": "Reviewer_PKLf", "summary": "This work proposed an algorithm to solve the straggler issue in split federated learning. Meanwhile, a zeroth-order based model update approach was proposed to reduce the memory consumption and load at the client side. The authors provided theoretical analysis on the convergence of the proposed algorithm and evaluate its performance using experiments.", "questions": "Please see above weakness.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n- This paper is well-written.\n- The incorporation of zeroth-order optimization is valid and interesting.\n- The theoretical analysis is solid.\n- The experimental results validate that the proposed approach can achieve a better performance than baselines.\n\nWeakness:\n- Although I understand the proposed algorithm, it is hard to see how the zeroth-order optimization addresses the straggler issue. Is it related to only when and how many rounds the updates are performance? If this is the case, the contribution on the algorithm design might be minor, as conventional zeroth-order optimization was used. \n- Zeroth-order optimization usually imposes multiple rounds of perturbations and hence increase the duration of each round. How is this problem solved in this work?\n- Is non-iid issue considered in this work? Straggler is an issue only when non-iid data is considered; otherwise, asynchronous aggregation does not impose any model accuracy degradation. As suggested in the unbalanced server update, different models may experience different number updates. Will the non-iid data be a problem? \n- Zeroth-order optimization is essentially an approximation. It is surprised to see the algorithm experience accuracy improvement. An ablation study may be needed to see if the zeroth-order optimization leads to any degradation, i.e., if the performance improvement comes from the schedule design (i.e., multiple updates) or the zeroth-order optimization. If it is from the schedule design, I am not sure about the necessity and significance of introducing the zeroth-order optimization.", "was_revised": true, "final_justification": "The motivation and main idea of this work appear to be promising. Despite this, there might be some limitations regarding the theoretical analysis on how straggler affects the convergence. Also, although the ZOO is a promising approach for reducing the memory of clients, some prior work have incorporated it into federated learning, so the main contribution of this work should be related to analyzing the impact of the stragglers on gradient estimation in ZOO. Thus, I would choose to maintain the score.", "rebuttal": "Thank you so much for your thoughtful feedback and for recognizing the strengths of our theoretical work. Below, we address your main concern regarding **the necessity of incorporating ZOO with our unbalanced update strategy**.\n\n**On the Role of ZOO in Addressing Straggler Issues:**\n\n1. **Theoretical dependency of ZOO in solving straggler issue.** Our straggler resilience result is derived from the sublinear speed-up convergence rate presented in Corollaries 4.4 and 4.8. Achieving this sublinear speed-up w.r.t the unbalanced update factor $\\tau$ requires a coordinated cutting strategy, where the dimension split satisfies $d_c = \\sqrt{\\frac{d}{\\tau}}$ and $d_s = d - \\sqrt{\\frac{d}{\\tau}}$. And this conclusion is only validated under ZOO, **since only the convergence rate of ZOO is dependent on dimension** $d$. This implies that the mitigation of straggler issue is **not merely due to multiple updates**, **but is tied to the incorporation of ZOO.**\n2. **Practical suitability of ZOO for straggler settings.** ZOO also offers practical advantages in straggler scenarios, particularly for memory-constrained edge devices. Compared to FOO, **ZOO can significantly reduce memory usage**[17, 18]. This aligns well with the realities of resource-constrained edge devices, where memory and computation overhead can introduce time delay that further exacerbates the straggler problem[a,b,c].\n\n**More interesting findings uncovered by incorporating ZOO with Unbalanced Update:**\n\n1. **ZOO enables theoretical interpretation between cutting strategy and unbalanced update.**  A key novelty of our work lies in providing a theoretical explanation for how the choice of splitting layer coordinates with unbalanced updates $\\tau$ affects convergence rate. Compared to FOO, the convergence rate of ZOO is related to the model dimensionality $d$, which helps us link the convergence performance to the cutting layer position. Specifically, adjusting the splitting layer allows us to optimize convergence behavior, which provides a principled guideline for choosing the cutting layer in practice. In contrast, such a theoretical connection between the splitting layer and convergence behavior is difficult to uncover in the FOO setting without the dimension dependency of ZOO convergence rate. \n2. **Dimension-Free ZOO achieved by Unbalanced Updates.**\nOur work is able to eliminate the dimension dependence in ZOO by employing unbalanced updates. Based on Corollaries 4.4 and 4.8, by appropriately scaling the unbalanced update factor $\\tau$ to the same order as the model dimension dimension $d$, the convergence rate becomes independent of $d$. This directly addresses a key limitation of traditional ZOO methods, where convergence rates typically deteriorate with increasing dimension.\n\n**Regarding your question about whether our algorithm design is just simply combines ZOO with Unbalanced updates.** \n\nFirstly, we would like to clarify that our algorithm does not merely apply conventional ZOO within SFL, but rather introduces a tailored design that explicitly addresses the straggler issue by reducing synchronization delays through a novel **Eager forward** transmission scheme. \n\nIn traditional ZOO, updates often require waiting for all perturbations to be computed before proceeding. We refer to it as the “Lazy” forward. **In contrast, our “Eager” forward method follows a sequential update process that effectively reduces waiting time.** Specifically, during each forward pass on the client, parameter-perturbed embeddings are immediately transmitted to the server as they are computed, without waiting for the full set of parameter-perturbed embeddings ($h, h^+, h^-$) to complete. For instance, the unperturbed embedding $h$ is first computed and sent to the server, which can immediately begin performing $\\tau$ local updates with $h$ as input, while the clients continue computing. \n\nTo have a more direct observation about our eager methods in minimizing waiting time, we conduct experiment using OPT1.3B on SST-2, and calculate the required time using different forwarding manner. As shown in Table 4, where we set $\\tau=10$, our eager scheme **reduces the overall forward time by nearly half.** This confirms the effectiveness of our eager update manner in minimizing synchronization delays by utilizing the overlapping computation time. \n\nTable 4 Required Time for Different Forward Manner  \n| Dataset | Client Forward Time (ms) | Server Forward Time (ms) | Eager Forward(ms) | Lazy Forward(ms) |\n| --- | --- | --- | --- | --- |\n| SST-2 | 61.18 | 5.41 | 673 | 1214 |\n\n> Zeroth-order optimization usually imposes multiple rounds of perturbations and hence increase the duration of each round. How is this problem solved in this work?\n> \n\nTowards this question, we believe that our **Eager Forward** design directly addresses this issue by performing **sequential transmission of parameter-perturbed embeddings**. This eliminates the need to wait for all perturbations to complete before the server can begin updates, thus significantly reducing time delay caused by multiple perturbations.\n\nMoreover, from a theoretical perspective, multiple perturbations lead to **faster convergence rates** w.r.t  $p$ [18]. Although applying multiple perturbations can introduce time delay for each local round, it can accelerate global convergence, thereby helping to mitigate the straggler issue to some extent.\n\n**Regarding non-iid issue:**\n\n**In this paper, we do not focus on the non-IID issue**. Non-IID data is often used to model data heterogeneity, which is indeed an important challenge in federated learning. **However, our focus is on the straggler problem, which stems from system heterogeneity** [8-14]. This refers to disparities in computational capabilities and network latency among clients, as we stated in the first paragraph of our paper. \n\nWe introduce variable time delays across all the clients which is a widely adopted approach to simulate the impact of stragglers in distributed learning system. This method is well-supported in prior literature [8–14]. To establish a more realistic model that reflects the randomness of clients' time delay, we consider an exponential distribution model, which has been widely used to capture the computation delay for distributed clusters [d,e,f].\n\nAs we acknowledge the importance of data heterogeneity in federated learning. To evaluate the robustness of our method under **non-IID data**, we incorporate a high data heterogeneity setup with shard=2.\n\nAs shown in Table 5, although all methods experience some performance degradation under non-IID settings, our proposed algorithm continuously outperforms the baseline GAS[8], which is specifically designed for non-IID scenarios. This suggests that our method is not only effective in mitigating straggler effects but also exhibits robustness to data heterogeneity.\n\nTable 5 Impact of data heterogeneity on SFL performance\n| Task | GAS | MU-SplitFed |\n| --- | --- | --- |\n| CIFAR-10(homogeneous) | 75.28 | 77.86 |\n| CIFAR-10(shard=2) | 66.23 | 66.83 |\n| Fashion-MNIST(homogeneous) | 83.70 | 85.45 |\n| Fashion-MNIST(shard=2) | 75.10 | 76.27 |\n\n**Questions about performance boost:**\n\nTo avoid any misunderstanding, we would like to first clarify that, as stated in Line 276, **we reimplemented all baselines using ZOO** to ensure a fair comparison across methods.\n\nTo address your concern, we conducted an ablation study on the SST-2 dataset using OPT-1.3B. Specifically, we implemented MU-Split in both its FO and ZO versions. This simplified setup includes only one client and one server, isolating the impact of the unbalanced update strategy. As shown in Table 6,7, we observe consistent performance improvement in both FO and ZO versions, indicating that the **accuracy gain primarily stems from the unbalanced server update schedule**. This further confirms the effectiveness of our proposed unbalanced update strategy in enhancing convergence and accuracy.\n\nTable 6  Best accuracy after 100 rounds of MU-Split(OPT-1.3B, Zeroth Order)\n| Split Layer | $τ=1$ | $τ=2$ | $τ=3$ | $τ=4$ | $τ=5$ | $τ=6$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| 2 | 88.75 | 88.97 | **90.90** | 87.95 | 87.05 | 88.52 |\n| 4 | — | 89.09  | **89.89**  | 87.05  | 86.93  | 89.04 |\n| 8 | — | **90.34**  | 90.11  | 89.50 | 89.54 | 88.30 |\n| 12 | — | 89.20   | **89.43**  | 88.41  | 88.41 | 88.34 |\n| 16 | — | **88.98** | 88.75  | 87.95  | 88.41 | 87.99 |\n\nTable 7  Best accuracy after 100 rounds of MU-Split(OPT-1.3B, First Order)\n| Split Layer | $τ=1$ | $τ=2$ | $τ=3$ | $τ=4$ | $τ=5$ | $τ=6$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| 2 | 88.06 | **91.82** | 88.75 | 88.07 | 87.73 | 88.30 |\n| 4 | — | 89.43 | 86.7 | 89.89 | **90.23** | 86.59 |\n| 8 | — | **89.89** | 87.61 | 84.20 | **89.89** | 88.86 |\n| 12 | — | 87.95 | 86.82 | 87.84 | **89.66** | 86.70 |\n| 16 | — | 87.84 | 88.64 | 86.70 | **89.77** | 88.86 |\n\na. “Thinking Forward: Memory-Efficient Federated Finetuning of Language Models”\n\nb. “Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training”\n\nc. “Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting”\n\nd. “Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity”\n\ne. \"Speeding up distributed machine learning using codes.\" \n\nf. \"Coded computation over heterogeneous clusters.”"}, {"review_id": "TXBUQlXcFO", "reviewer": "Reviewer_QcJK", "summary": "With the growing attention to the importance of federated learning (FL), a hybrid approach called Split Federated Learning (SFL) has been proposed to overcome the limitations of traditional FL and Split Learning (SL). However, even in SFL, overcoming the heterogeneity among clients and enabling efficient training remains a challenging task.\nThis paper proposes a method to enhance the efficiency of SFL by allowing the server to compensate for the slower clients' performance. Specifically, the server supports clients with limited computational resources, thus addressing the imbalance among heterogeneous clients.\nAdditionally, by incorporating ZO optimization, this paper presents a memory-efficient design suitable for SFL frameworks.", "questions": "There seems to be a reason for using perturbed parameters for gradient estimation on the server side and perturbed embeddings on the client side. Is it not possible to unify the gradient estimation method to perturbed parameters or perturbed embeddings for both the server and the clients? Alternatively, could they be reversed?\n\nIn the sentence, \nwhere delta_{...} denotes the loss difference obtained from the perturbed embeddings.\nappears to include a typo. It seems to be perturbed parameters.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "To address the straggler problem among clients, this paper proposes a method that effectively utilizes the server. It addresses memory constraints on the client side through ZO optimization. The advantages of the proposed method are validated through comparisons with recent works, and its convergence is supported by theoretical analysis. From this reviewer's perspective, the proposed method presents a novel contribution.", "was_revised": true, "final_justification": "This reviewer believes this paper has dealt with an important topic related to Split Federated Learning and has addressed it appropriately. The authors have responded to this reviewer’s questions, which has clarified my understanding. The concerns and opinions of the other reviewers have also been taken into consideration. This reviewer would like to maintain the positive rating.", "rebuttal": "Thank you so much for your encouragement and recognition of our theoretical contribution! We're happy that our research has raised your interest. We will further refine the detail in our work according to suggestions.\n\n**Clarification on \"perturbed parameters and embeddings\".**   \n\nThe \"perturbed embedding\" $h^{t+,-}$ refers to \"parameter-perturbed embedding\", which is obtained by perturbing the **client-side parameters** during the forward pass. \n\nFor server-side updates, we take $h^t$ as input when perturbing the **server-side parameters** to obtain ZO gradient estimates. Meanwhile, $h^{t+,-}$ is used on server-side to calculate $\\delta^t$(Eq. 6), which will be sent back for client-side gradient estimation and model updates. That being said, in our algorithm design, both client and server follow the same unified ZO update protocol.\n\nThis also helps explain our statement that “$\\delta^t$ obtained from the perturbed embeddings”, since $\\delta^t$ is obtained by using $h^{t+,-}$ as input. This information is used for client-side updates and does not require perturbing the server-side parameters.\n\nHope our explanation addresses your question. Thank you again for your careful review!"}]

Author rebuttal:

Meta-review: This paper proposes a straggler-resistant split FL method using an unbalanced update mechanism. Zeroth-order optimization enables efficient training in SFL settings. A theoretical proof for linear speedup is also provided. On the other hand, questions remain regarding how the presence of stragglers interacts with data heterogeneity. Overall, the merits outweigh the shortcomings.
Decision: Accept (poster)