FedEL: Federated Elastic Learning for Heterogeneous Devices
Authors: Letian Zhang
Bo Chen
Jieming Bian
Lei Wang
Jie Xu
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Heterogeneous Devices
Primary area: deep_learning

Abstract: Federated learning (FL) enables distributed devices to collaboratively train machine learning (ML) models while maintaining data privacy. However, the heterogeneous hardware capabilities of participating devices often result in significant training delays, as straggler clients with limited resources prolong the aggregation process. Existing solutions such as client selection, asynchronous FL, and partial training partially address these challenges but encounter issues such as reduced accuracy, stale updates, and compromised model performance due to inconsistent training contributions.
To overcome these limitations, we propose FedEL, a federated elastic learning framework that enhances training efficiency while maintaining model accuracy. FedEL introduces a novel window-based training process, sliding the window to locate the training part of the model
and dynamically selecting important tensors for training within a coordinated runtime budget. This approach ensures progressive and balanced training across all clients, including stragglers. Additionally, FedEL employs a tensor importance adjustment module, harmonizing local and global tensor importance to mitigate biases caused by data heterogeneity. The experiment results shows that FedEL achieves up to 3.87× improvement in time-to-accuracy compared to baselines while maintaining or exceeding final test accuracy.

FedEL: Federated Elastic Learning for Heterogeneous Devices
Letian Zhang
Bo Chen
Middle Tennessee State University Murfreesboro, TN 37132 letian. zhang@mtsu.edd
Middle Tennessee State University Murfreesboro, TN 37132
bc7b@mtmail mts..
Jieming Bian
Lei Wang
Jie Xu
University Florida Gainesville, FL 32611 jieming. bian@ufl.ec. edu
University of Florida Gainesville, FL 32611 leivang1@ufl.co edu
University of Florida Gainesville, 32611 jie.xu0ufl. xu@ufl. edu
Abstract
Federated learning (FL) enables distributed devices to collaboratively train machine learning models while maintaining data privacy. However, the heterogeneous hard- ware capabilities of devices often result in significant training delays, as straggler clients with limited resources prolong the aggregation process. Existing solutions such as client selection, asynchronous FL, and partial training partially address these challenges but encounter issues such reduced accuracy, stale updates, and compromised model performance due to inconsistent training contributions. To overcome these limitations, we propose FedEL., federated elastic learning frame- work that enhances training efficiency while maintaining model accuracy. FedEL introduces novel window-based training process, sliding the window locate the training part of the model and dynamically selecting important tensors for training within a coordinated runtime budget. This approach ensures progressive and balanced training across all clients, including stragglers. Additionally, FedEL employs a tensor importance adjustment module, harmonizing local and global tensor importance to mitigate biases caused by data heterogeneity. The experiment results show that FedEL achieves up 3.87x improvement compared to baselines while maintaining or exceeding final test accuracy.
Introduction
Federated learning (FI.) is privacy- seeeacy-presevvng machine learning paradigm where distributed clients, such as mobile devices and IoT systems, collaboratively train a global model while keeping their data local. Typically, FL involves devices performing local model training and sharing parameters with central server for global model updates. However, heterogeneous hardware capabilities among devices lead to "straggler", or slower clients, causing significant training delays as the server must wait for their updates. This challenge hinders the scalability of FL, particularly in large-scale cross-device scenarios.
Status Quo and Their Limitations. To address computational constraints, existing solutions fall into three main categories: client selection, asynchronous FI., and partial training. Client Selection (Figure [] top-left). Selecting a subset of devices for training based on specific criteria can mitigate delays. However, significant differences in clients' data distributions often leave stragglers underrepresented, reducing the global model's accuracy. Asynchronous FL (Figure [ top-right). This approach decou- ples local training from global aggregation, allowing stragglers train independently. While this
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

reduces delays, the global model often relies on faster clients, leaving stragglers' contributions infre- quent and potentially outdated, which may harm convergence [44] Partial Training (Figure FFiuueeJbottoo left). Techniques like width and depth scaling adjust the model architecture to accommodate varying resources. Width scaling resizes convolutional layers, risking channel mismatches during aggregation [1]. while depth scaling limits training to early layers, leading to suboptimal task-specific features and reduced performance [42]. These limitations highlight the need for a novel training paradigm to overcome resource heterogeneity and enable peger-errfrmance FL in real-world deployments. ElasticTrrainer [4] introduces method for selecting important deep neural network (DNN) tensors to meet runtime training re- quirements single device. By focusing on these key tensors, ElasticTrrainer acceler- ates training. When applied FI., it offers a potential solution for addressing strag- glers by allowing each client select im- portant tensors based on its hardware capa- bilities under unified runtime constraint. This ensures that all clients complete lo- cal training within similar timeframe, en- abling synchronized global model aggrega- tion. However, directly deploying Elastic- Trainer in FI. scenarios has two limitations: Limitation Limited Training Scope on Slower Clients: Due to the chained rule of DNN backward propagation, unselected tensors still compute gradients to propa- gate updates to the selected tensors. This constrains the selected tensors slower devices to the back-end of the DNN, reduc-
Neural Network
Untrainable
Trainable
Ciient Selection
Asynchronous EL
Round
Round
Round
Round
Slow Fest
Fast
Slow
Training lime
Training Time
Fast Client
Client
Client ...
Client
Partial Training
FedEL
Round 1
Round 2
Round
Round 2
Small DNN
Small DNN
Slew
a
Training Time
Treining Time
Fast Client
Slow Client
Figure 1: Existing works FedEL.
ing training the front-end feature extraction layers and degrading FL accuracy, especially with heterogeneous data distributions. Limitation## Exacerbated Local Model Drift: Variations in data distribution cause significant differences in tensor importance across clients. Training only the important tensors amplifies local model drift, where client models diverge from the global model, further reducing accuracy.
Overview of the Proposed Approach. Motivated by the above limitations, we propose FedEL, a federated elastic learning framework that enhances federated training efficiency. To address the first limitation, we propose a window-based training approach that divides the DNN model into multiple blocks, ensuring that each part of the model is trained during FL rounds. Before training, we use a tensor timing profiler to measure the training time for each tensor, which is then aggregated into block-level training times. In each FL round, the window slides to include set of blocks based on the runtime budget and current training status. The sliding window process involves moving the front edge include deeper blocks and shrinking the end edge to exclude blocks that longer require training. Elastic' Trainer then modified to select important training tensors within the selected window, allowing straggler clients to progressively train the erucial tensors of the entire DNN model. To address the second limitation, we design a tensor importance adjustment module. At the start of each FL round, the client estimates the global model's tensor importance using the global models from the current and previous rounds, along with the learning rate. This global tensor importance used adjust the local tensor importance computed by ElasticTrainer, ensuring tensor selection considers both local and global data distributions.
Evaluation. We implement FedEL on both hardware testbed and software simulations. The hardware testbed consists of ten NVIDIA Jetson devices connected wirelessly to a server. To simulate large-scale scenarios, extend the setup with diverse client simulation. We evaluate FedEL using various DNN models and four real-world FL datasets across three key tasks: image classification, voice command recognition, and next-word prediction. Our results show that: (1) FedEL outperforms baselines on the time-to-accuras accuracy while final test accuracy is par with memory overhead and energy consumption during training compared to existing methods. (3) Ablation studies confirm the necessity and importance of each key component in FedEL's design.
Specifically. FedEL outperforms FedAvg by 3.87 in time-to-
higher than FedAvg. (2) FedEL reduces

---

2 Related Work
On-single-device training. Leveraging mobile and emhedded computing for DNN model training has gained attention [53]. Some studies reduce computation by quantizing or pruning gradient propagation
for certain neurons 10 36 [6]. Others use a two-stage paradigm. where the system prepares the
computing graph and generates a training plan before model training [33 45
39 |2 2 [4]. Our work
builds on ElasticTrainer [4,. which dynamically selects important tensors during training. However,
applying single-device methods directly to FL scenarios can be challenging due to heterogeneous
systems and data.
Heterogeneous federated learning. To address the challenges posed by low-end devices in FL, three main training methodologies have been proposed: (1) client selection, (2) asynchronous FL,
and (3) partial training. Client selection methods 221 25 6 42 evaluate the utility of each client and select a subset to participate in FL rounds. For example, PyramidFL. (25 ranks clients based on utility. However, when slower clients have unique data, they may be infrequently selected, leading to accuracy loss [26]. Asynchronous FL methods [50 52 30 (28] allow the global model to be updated as soon as local models are received, bypassing slower clients. TimelyFL (50] adjusts workloads based on client resources, increasing participation. However, this may lead to slower convergence and accuracy issues, as model updates may arrive at different times, causing inconsistencies, particularly with heterogeneous devices and data [44]. Partial training involves training part of the model by
34 41]. HeteroFL 7 scales convolutional layers match devices' available training time. Similar methods include Federated Dropout [5] and FjORD [12), but these can disrupt model architecture and degrade performance. DepthFL. [8] customizes models based on client resource constraints, but the global model size is limited by the device with the largest memory. Unlike existing methods, FedEL ensures all clients participate in FL rounds, allowing clients with varying speeds to complete training of the full DNN model by sliding their windows.
scaling its width depth 7 5 12
18
3 Background and Motivation
To help better understand our design of FedEL, we first introduce how the ElasticTrainer can speed up on-device DNN training with a small accuracy loss. Afterwards, we show the issues of using ElasticTrainer directly in the heterogeneous federated learning framework, hence motivating our federated elastic selection of the trainable DNN portion at runtime.
The tensor selection problem in ElasticTrainer optimization problem:
14 is formulated as a constrained
max A I, s.l. Tfw + Too (A) < Tek-
Here, A is a binary mask representing the selected tensors. I is the importance of tensors. Trw is the fixed forward propagation time, independent of tensor selection. Thw (A) represents the backward propagation time, which depends on the selected tensors involved in gradient computation. The sum Tyw + Tow (A) is the estimated training time constrained to user-defined runtime threshold Ten aimed at accelerating training. For example, setting Th to 50% of the full model training time implies reducing the training time to half that of full model training. ElastieTrainer consists of two modules: the tensor timing profiler and the tensor importance evaluator. The tensor timing profiler creates an offline tensor-level backward computation time graph, preserving the execution order of all tensors during backward propagation, from the output to the input layer. In the online training phase, the start of each fixed interval, the tensor importance evaluator evaluates the importance of all tensors I. ElasticTraine then uses dynamic programming to solve the optimization problem ([). freezes unselected tensors, and trains only the selected tensors during each interval.
Federated Learning with Elastic In FL., diverse hardware capabilities lead to significant variations in local training times across clients. By employing ElasticTraine with a uniform runtime threshold Tea across all clients, it becomes theoretically feasible for all clients to participate in cach FL round, ensuring consistent training times regardless of hardware differences. Consider an FL setup with N clients, starting from the same initial model. In each FL round r, the central server distributes its current model to all clients. Each client trains the model its local data using ElasticTraine with the uniform Te,, then sends its model update wn,r to the server. The server aggregates the updates W,+1 E C.. @ Warr, yielding the global model for next round r + 1, where (c, (t)) = (t))
(A)( denotes the k-th tensor selection of mask A, (t) at training
3

---

round r. The updated global model w,+1 is broadcast to all clients for the next round. This process iterates until a predefined maximum number of training rounds is reached.
To validate this approach, we design the follow-
ing experiment. System Platform. We design FL system with 10 client devices, consisting of
Wiy Cluck Time (1en1
Nodel Traniin
80
80%
NVIDIA Jetson Xavier NX kits (Xavier) [2]
focuray
and 5 NVIDIA Jetson Orin kits (Orin) M. All
10
40%
devices connect to PC via WiFi, with Orin of-
18
20%
-D- Full Hacel Trankng
fering superior computational performance com- pared to Xavier. Dataset and Model. We fo-
20%
-
Elaxsd
Device
cus on an image classification task using the CIFARIO dataset [19] on VGG16 model [35] implemented within the FedAvg framework [31.
(a) Training time
(b) Accuracy
Figure 2: Average training time per FL round and training accuracy evolution of FedAvg with full model training and FedAvg with ElasticTrrainer
FlasticTrainer [4] is used for local training, and
the dataset is partitioned non-iid using a Dirich- let distribution a = 0.1) [46]. Training Setup.
The runtime threshold TeA is set based on the full model training time of the faster Orin devices, ensuring all clients complete local training within a similar timeframe.
Limitations of FL with ElasticTraiine Figure 2a illustrates the average training time per FL round on Xavier and Orin using FedAvg with full model training and FedAvg with ElasticTrrainer. Due to the disparity in computational performance between Xavier and Orin, Xavier's s training time per round is nearly twice as long Orin's when using FedAvg with full model training. Consequently, Orin clients must wait for Xavier clients to complete their training before responding to the central server, leading to longer idle times for the faster Orin clients. Figure 2a also demonstrates that FedAvg with ElasticTrainer reduces this imbalance, enabling both Xavier and Orin complete each round of training in roughly the same time. However, as shown in Figure 2b the accuracy of FedAvg with Elastic'Train In the following sections, we explore in more detail how the direct deployment of ElasticTrrang in FL underutilizes data and training efficiency. These insights are foundational to the design of FedEL.
Forward pass Backward pass
tensor
twi Tliee of weight update
'g. Time of computing gradient
Latency when only training layer 2 and
tw
Input
Pred.
3
4
Loss
Label
Figure 3: Tensor selection in Elastic
is 40.03% lower compared to FL with full model training.
Li Limited Training Scope on Slower Clients. ElasticTTrainer identifies the most important tensors under specified train-
Xavir:
ing time threshold Tek. However, the tensor selection process is not straightforward due to the dependencies inherent in backward propa- gation. Even if tensor is not selected, it must compute and pass gradients to previous tensors, contributing the total training time. For ex- ample, as illustrated in Figure 3 the backward propagation time comprises two components:
Xider
Xnder
Kaver 2-
Koder
Xavie
Xaver
X
60
00
onn
GHLL
Onn.3
Orn.A
Om
Om
62
15
20
Taraer
Figure 4: Tensor selec- tion in Xavier's model and Orin's 's model.
Figure 5: Tensor impor-
tance of ten-device FL and central training.
(1) Gradient Computation Time lg: Time spent calculating the gradient of the current tensor to pass to the previous tensor. (2) Weight Update Time tw: Time spent updating the tensor's weights using gradients from the subsequent tensor. If tensors 2 and 4 are selected, the total training time includes both selected and unselected tensors, calculated as t5 + t + ty + t f2 ElasticTrainer employs dynamic programming approach, starting from the last tensor and selecting important tensors in reverse order until the accumulated weight update and gradient computation time reaches Tih.
demonstrates tensor selection across Xavier and Orin clients during one FL round. While Orin clients (faster devices) can train nearly all tensors" Xavier clients (slower devices) tend focus training on the back part of the DNN model. This leaves the front feature extractor layers largely untrained due to the same To being applied across all devices. In FL settings with non-fid data, this
Figure 4
Unselected tensors Orin clients result from Traitics computational cost optimization process.

---

imbalance becomes critical. Xavier clients' untrained feature extractor layers fail to adequately learn essential features, weakening the global model's ' ability to extract meaningful features. Consequently, the overall accuracy of the system degrades.
2i Exacerbated Local Model Drift. ElasticTrrainer is optimized for centralized training, where all data resides on single device. However, FL involves distributed training, and recent studies have highlighted the local model drift challenge arising from non-iid data distribution among clients. Non-iid data can bias tensor importance evaluation, as local models trained on heterogeneous client datasets reflect varying data distributions. Figure [ compares tensor importance across ten FL clients and centralized training. In FL, tensor importance differs significantly between clients and also the centralized training due to non-iid data. Elastic Trainer'' selective training exacerbates this bias by freezing unselected tensors, intensifying local model drift. As result, when the central server aggregates these biased local models, the global model accuracy suffers compared to full model training in FL..
37 48 49
4
FedEL Design
4.1 Sliding Window Training
To address Limitation 1, we propose dividing the DNN into multiple blocks and utilizing a window- based scheme that ensures every part of the DNN model has the opportunity be trained during the FL local training rounds. Specifically, the DNN model is partitioned into B blocks, denoted as [0,,02.... ........... ..]. based on its original architecture. Each block may consist of one or more layers, preserving the inherent structural integrity of the model. For instance, in VGG16, which follows a chain-like architecture, each layer can be treated as separate block. In contrast, ResNet50 contains residual structures, so each residual structure can be considered block, while other layers outside these structures can also be treated as individual blocks.
Offline Tensor Time Profiling, Before initiating the online training process, each client uses the tensor timing profiler in ElasticTTrainer to measure the training time for each tensor. This offline tensor-level timing data is then aggregated into block-wise training times by summing the training times of all tensors within each block. Assume block b contains a set K of tensors. The training time T of block b is computed as: T = Σ (tg + th),, where to is the time of computing the gradient, and t is the time of updating weights for each tensor ls K
Online Window-Based Training Using the block- wise training time file, we first initialize the start-
ing window. The initial window begins with the
Training round traiin window freeze other
Lass
first block, 0, and progressively includes subse-
Window
quent blocks until the cumulative training time just exceeds the user- user-specified runtime threshold Tik- Specifically, the initial window consists of Oo (0, Jm). where (I).......
Input
T < Tes and Eoe(1.....) T 2 Tih- At each FL round, the win- dow slides, and ElasticTrainer is applied to train the corresponding portion of the DNN model. Over time, this approach ensures that the entire model is trained, enabling complete feature extraction from the data. However, as highlighted in Limitation #1, the blocks outside the current window still require time com- pute gradients and pass them to the blocks within the window. This dependency means the original output layer cannot serve as the final output for each window. To address this, lightweight output layer is attached to the last layer of the window, acting as an early exit. This ensures independent training for each window and facilitates the completion of the window-basee based training process. Example. Figure 6 illustrates the window-based training process with early exits in FL. In round r, Window 1, comprising blocks 1, 2, and 3. selected for training, while the remaining blocks are frozen. The early exit of Window I serves as the output layer. Inputs are forwarded through Window to generate predictions, which are used to compute the loss gradient. Backward propagation updates only the weights in Window 1, with other blocks entirely
22
0.3
0.6
Training round
1:train window 11002000 othor blocks
Losa
Window
0.6
Figure 6: Overview of window-bal based training in FedEL.
5

---

excluded from forward and backward propagation. This approach applies Elastic Trainer to Window significantly reducing training time. After round r. only Window I's updated weights are sent to the global server for aggregation, and the updated global model is broadcast to all clients for the next round. In round r + 1, Window shifts to Window 2, now consisting of blocks 3, 4, and 5. These blocks are trained while others remain frozen. The early exit of Window 2 acts as the output layer. Inputs are forwarded through blocks 1-5 to produce predictions, but only the weights in blocks 3, 4, and 5 are updated during backward propagation. Blocks after block 5 remain frozen, while blocks I and 2 participate forward propagation to pass intermediate results to Window GlasticTraanee continues to optimize training within Window 2.
This iterative and cyclical training process ensures consistent training time across all clients while allowing all parts of the DNN model to be trained, preserving model accuracy.
4.1.1 Sliding Window
We assume that the window has two boundaries: the Front Edge and the End Edge. All blocks between these edges form the current training window. At the beginning of each FL round, clients slide the window to determine the portion of the DNN model to train based on their training progress.
Window et round
Window
Window at roundr
atoundd
bddge
Front
odge
Front edgo atr+1
Irital window
End
Front ee
End bdge
Endedge
adone
77
+
7
T'
TT
1
2
3
4
5
6
7
8
19
1
3
5
6
7
8
2
4
6
7
8
9
+ 2T
(a) Front edge movement will con- tain blocks that have accumulated training time just above Teh.
(b) Front edge movement will re- vert to the initial window when it
(c) End edge movement will cull out blocks once there are no impor- tant tensors selected from them.
touches the end of DNN model.
Figure 7: Front edge movement and end edge movement in window sliding.
Front Edge Movement. As illustrated in Figure 7a the front edge moves forward to include deeper blocks of the DNN model. Each movement adds blocks whose cumulative training time slightly exceeds the user-defined runtime threshold Tı.- For instance, in Figure 7a when training round begins, the front edge (orange line) shifts cumulative training time of blocks 5, 6, 7, and 8 meets or exceeds Tek, while the cumulative time for blocks 5, 6, and 7 is below T.k. If the front edge reaches the end of the DNN model and the cumulative training time of newly added blocks is still below Tih this is also considered a front edge movement. Once the front edge reaches the model's end, as shown in Figure 7b) but FI. training is not yet complete, the window resets to the initial window for the next round.
deeper position (orange dashed line). Here, the
End Edge Movement. The end edge moves to shrink the training window and freeze blocks that no longer require training. This movement depends on the current training status. If blocks at the window's 's end are not selected in the previous FL round, the end edge excludes them from the window. This adjustment occurs for two reasons: either the window is too large, preventing ElasticTrainer from selecting important tensors within the threshold Ta. or Elastic ElasticTrainer determines no important tensors exist in those blocks. For example, shown in Figure 7c if blocks 2 and contain no important tensors during training round r, the end edge will shift to block in the next round.
4.1.2 Insert ElasticTrainee into Windows
To integrate ElasticTrainne into windows, we adapt its tensor selection module. In its original form, the module uses dynamic programming to identify the optimal set of important tensors for local training, starting from the last tensor and proceeding until the accumulated training delay, including weight update time and gradient computation time, reaches the runtime threshold Teh- Our modification adjusts the starting point of dynamic programming to begin at the tensor corresponding to the last layer within the current window. Additionally, we introduce a new base case: if a tensor lies outside the window's range, the dynamic programming process halts and returns the selected important tensors. This adjustment allows FlasticTrainer to be seamlessly applied to window based training, ensuring efficient and targeted training within each window.

---

4.2 Tensor Importance Adjustment
In Limitation #2, the tensor importance estimated by ElasticTrainer is biased due to the heterogeneous data distribution across clients. To address this bias, we propose strategy that leverages the global model after aggregation to compute tensor importance. This global tensor importance is then used in the subsequent local training round to adjust the tensor importance at the client side, thereby improving training efficiency. After collecting the locally trained models from all connected clients, the server aggregates these models to next round global model 11,+1. The aggregated global model then broadcast back the clients for the next round of training. Elastic'Trainer calculates tensor importance as Sru
0L A w. where the loss gradient multiplied by the tensor update. Upon receiving the updated global model, clients compute the tensor importance of the global model using the formula:
1 10+1- (Wr+1 w,) [w)+ Here, Y/n is the learning rate for client n, M 7p estimates the global model's loss gradient, and w ,,+1 - w, represents the tensor updates in the global model. The global tensor importance I is then used to adjust the local tensor importance for each client as follows: 1, = 3 In.++ + (1 (0-3) 3) I where 3 E [0, 1] is a balancing parameter that determines the weighting between local and global importance. This adjustment ensures that local tensor importance aligns better with global priorities, thus improving the overall training accuracy of
1/-
the model.
Due to page limitations, the complete algorithm and the theoretical convergence analysis of the proposed method are provided in Appendices A and B
5 Evaluation
5.1 Experiment Sctup
Datasets, Models, and Tasks. To demonstrate FedEL's effectiveness across tasks, datasets, and
models, we evaluate FedEL on four real-world datasets designed for FL applications at different scales. Image Classification. VGG16 (35 model on CIFARIO dataset [19] and Tiny ImageNet
dataset (23). Speech Recognition. ResNet50 [ model on Google command speech dataset [40]. Natural Language Processing. Lightweight Albert (22) model on Reddit dataset [32]. To follow the realistic non-iid data in FL scenarios, we partition the datasets into different clusters using a Dirichlet distribution with equals 0.1. The Reddit datasets inherently exhibits non-iid characteristics.
Baselines. The following baselines are adopted for evaluation purposes: (1) FedAvg 31
(2)
ElasticTrriine [4] (3) HeteroFL. 77. (4) DepthFL [18. (5) PyramidFL. [23]. (6) TimelyFL [50] (7)
FIARSE [4]. Detailed descriptions of these baseline methods are provided in the Appendix.
FL Setup. To evaluate FedEL's effectiveness, we conduct experiments in two scenarios: a small-scale practical edge device setup and a large-scale simulation. Small-scale Practical Edge Device Scenario: FedEL is deployed on ten heterogeneous edge devices, comprising five NVIDIA Jetson Xavier NX kits (Xavier) [2] and five NVIDIA Jetson Orin kits (Orin) (1) connected via WiFi to a central PC. Due to the limited number of devices, we evaluate performance using only the CIFAR10 10 dataset. Large-scale Simulation Scenario: To simulate larger environment, we use tensor timing profiees generated by ElasticTTran Trainer's 's offline tensor profiler on Orin as a baseline. From this profilng data, we simulate four types of heterogeneous devices with scaled tensor training times, including devices matching the baseline profiling time, devices with 1/2 of the profiling time, devices with 1/3 of the profiling time, devices with 1/4 of the profiling time. total of 100 clients are simulated, with each randomly assigned device type and corresponding processing time. This simulation is conducted on a PC equipped with an NVIDIA 3090 GPU. For fair comparisons with baseline methods, unless stated otherwise, the runtime threshold Tú, is set to the full model training time of the fastest device, and the balance parameter 33 is fixed at 0.6.
5.2 End-to-End Performance
FedEL accelerates training while maintaining high accuracy. Table I summarizes the final accuracy and wall-clock training time of FedEL and its baselines. FedEL consistently outperforms baselines under the same training rounds. Below is detailed analysis of the results: FedAvg: FedEL achieves comparable accuracy to FedAvg, which trains the full model, but reduces wall-clock training time by 1.87 x x. This efficiency arises because FedAvg waits for slower clients to complete
7

---

Table 1: Comparison of FedEL with baselines on time-to- tw-.ccaaccccc.....
Image Classif.
Speech Recng.
Device
DeVievioes
Devicisees
Aoc.
Time
Speodup
Aec.
Time
Spaclup
Aue.
Time
Spcedup
Perp
Time
Sposlup
Fedlvgg 51
56.13%
119.85
NA
33.76%
563.1h
NIA
58.04%
709.88
NA
77.48
546.4h
NAA
Elastic Trunner [4]
40.03%
64.86
1.3X
27.65%
138.00
3.55x
37,56%
184.36
3.81x
81.02
176.2h
3.10x
HeteroF 7
53.44%
80.15
1.49x
30.56%
248.22
2.26%
51.47%
265.96
2.00%
50.11
206.1
2.60x
DepthF1 [x]
89%
77.36
1.54x
34.14%
198.3h 3h
2,83
54.23%
207.4h
3.42x
78.08
212.4h
2.57
Pyramial L [5]
56.24%
115.75
1.88x
34.70%
497.4h
1.13%
58.12%
587.4b
1.21x
77.58
418.2h
1.31x
TimelyH-. [X]
53.74%
66.3h
1.81x
33.53%
198 1h
2.84%
56.49%
193.2h
3.67x
80.91
177 69
3.07%
FLARSE
36.48%
33.98%
191.50
2
3.58X
191.0h
2,80x
FodHl.
56.51%
63.86
1.87
34.96%
156.80
3.59%
58.26%
183.36
3.87
77.23
174.5h
3.13%
training, whereas FedEL dynamically selects portions of the DNN for slower clients, enabling all clients to complete local training roughly the same time. ElasticTrainer: While Trainer speeds up training by up to 3.84x compared to FedAvg, it sacrifices over 28.6% accuracy across four datasets. As noted in Section 3 ElasticTrriine'' focus on selecting important tensors only from the back of the DNN on slower clients limits global model feature extraction. FedEL addresses this limitation, achieving 1% 2% faster training time than ElasticTrraine by leveraging window sliding to reduce tensor selection overhead, while maintaining high accuracy. HeteroFL: FedEL improves accuracy by 5.7% 14.4% compared to HeteroFL. The uneven scaling of convolutional layers in HeteroFL compromises parameter training and degrades the model's architecture [8.. Furthermore, HeteroFL requires complex global aggregation for mismatched parameters, increasing training time. DepthEL: DepthFL partitions models into sub-models for slower clients and uses self-distillatie for knowledge transfer. However, its slower training and reliance on training only the front layers of the DNN for slower clients weaken the global model's s ability to Iearn from their data. FedEL outperforms DepthFL. with up to 7.1% higher accuracy. PyramaidFL:: PyramaidFL. synchronizes fast and slow clients by allowing fast clients to train for more epochs, accelerating convergence but not redueing total training time. FedEL achieves 1%%2% higher accuracy than PyramaidFL by ensuring balanced participation of slower clients. TimelyFL: FedEL achieves up to 5% higher accuracy compared to TimelyFL. The heterogeneity-awanl asynchronous approach in TimelyFL reduces participation rates for slower clients, leading to accuracy loss in heterogeneous data environments. FedEL, by contrast, ensures balanced participation across clients, preserving accuracy. FIARSE: FIARSE does not account for the dependency of backward gradient propagation. Specifically, its output layer is fixed as the last layer of the network structure. This results in the unselected tensors in FIARSE need to compute and propagate gradients previously selected tensors.
FedEL reduces the memory and energy consumption. Figures X and | compare FedEL with baselines in terms of mem- ory usage, power consumption, and energy consumption, as measured using the Jetson Power GUI on Xavier and Orin devices. Since the differences in measurements be- tween the two devices are negligible, we present the averaged results to save space. As shown Figure X FedEL reduces mem- ory usage by up to 32.7% compared to Fe- dAvg. This improvement stems from train- ing only portion of the DNN model while freezing unselected layers and tensors, which minimizes memory allocation required for gradient backpropagation. In Figure 9 we observe little variation in power consumption across methods, as both Orin and Xavier operate at full power when their GPUs are active. However, for the same set of computational tasks, FedEL significantly reduces energy consumption. FedEL achieves average reduction of 49.59% in total energy usage compared to FedAvg, primarily because it completes training in nearly half the time required by FedAvg.
Menorry (00)
Eroug
Fesatr Elassi Hetiont.
Toner Ayrensa Pi4ese Fiatt
Palevo
Tonay
Pansest
AMesee
Figure 8: Memory over- head.
Figure 9: Power/energy
consumption.
FedEL can adaptively select important tensors. FedEL's performance is driven by its dynamic sliding-winde window mechanism and elastic tensor selection at runtime. We analyze these adaptive behaviors using large-scale 100-device scenario with the Tiny ImageNet dataset. Figure TT] showcases representative devices from each of the four device types. As observed, the number of windows required to train the full model varies across devices due to their differing computational capabilities. Within each window, tensor selection is dynamically adjusted based on importance, For instance, if tensor at the front is critical for model performance, FedEL can adaptively skip updating
8

---

(a) Orin
(b) 1/2 Orin
(c) 1/3 Orin
(d) 1/4 Orin
Figure 10: Tensor selections during different FL rounds
few less important tensors (with higher indices) to maintain the desired training speedup while preserving model effectiveness.
5.3 Ablation
We analyze how parameter settingss influence FedEL using the small-scale practical 10-device scenario with the CIFARIO 10 dataset for image classification. Additional results can be found in the Appendix.
Impact of balancing parameter B. The balancing parameter 39 in FedEL determines the weighting between local and global tensor importance during adjustment. Figure [ how varying affects mine-to-accurace performance. A larger B overemphasizes local tensor importance, reducing the influence of global model variations, while a smaller focuses solely on global variations, neglecting local data heterogeneity. As shown in Figure [1] when 3 1 (fully local focus) 3 0 (fully global focus), FedEL's accuracy falls below that of FedAvg. In contrast, moderate values of 3 (e.g., 0.4) outperform FedAvg by balaneing local data heterogeneity with global model
3= 0.6 variations. This balance allows FedEL to effectively capture both local and global tensor importance, enhancing accuracy.
Impact of runtime threshold Tap To en- sure a fair comparison with other baselines, we set the training time threshold Te equal to the full model training time of the Orin (i.e., Torin). We vary Tth to examine its impact on FedEL's performance, with the experiment stopping once the training time reaches the predefined value. As shown in Figure 12 a smaller Tia slows down convergence. This is because slow clients must train the entire model, leading to more diding-windoo movements, while fast clients also perform additional window sliding, increasing the overall training time and reducing efficiency.
00%
00%
50%
Aoansay
Aczurwty
80%
40%
Fashky
-Q- Festug
3 20-3-00 0.6
T, 8.5T
--- A
0.8
30%
O.7T
8-
3-1
- B-
Tam
50
100
60
100
Walldaak lime (sour)
Willidock drm (uaat
Figure 11: Impact of B. Figure 12: Impact of
Teh-
Sliding Window. The sliding window sists of two processes: the front edge move- ment and the end edge movement, which define the window size and the range of se- lected important tensors. In each FL round, the front edge includes blocks with accu- mulated training time just above the run- time threshold Tek As shown in Figure [2] reducing Tea slows convergence, as more rounds are required to train the full model. The end edge movement reduces the win- dow size by excluding unselected blocks.
60%
FedEL-C
50%
Fedkug
FweE
50
100
Wall clock time (hour)
Figure 13: accuracy of FedAvg, selection illustration in FedEL-C and FedEI.. FedEL-C and FedEL.
Time-to- Figure 14: Tensor
To assess effectiveness, we compare with scenario where the end edge directly moved the current front edge (FedEL-C). As shown in Figure 13 FedEL-CC results in lower accuracy than FedEL. The tensor selection examples in Figure [4 explain this: FedEL-C treats each window independently and does not adjust training tensors between consecutive windows, leading to accuracy degradation.

---

Conclusion
We introduced FedEL, a progressive training approach address client heterogeneity in FL. To overcome the limitations of directly selecting important tensors, we propose two innovations: sliding- window training and local tensor importance adjustment. Sliding-window training enables FedEL to train the full DNN model by adjusting the front and end edges of the training window. Local tensor importance adjustment selects important tensors based on both local client data and global data importance. The results show that FedEL reduces wall clock training time (speeding up by 1.87x to 3.87 while achieving comparable or better accuracy and perplexity across various FL applications and DNN models.
Acknowledgments
The work of Letian Zhang and Bo Chen is partially supported by NSF under grant 2348279 and also supported by MTSU Stark Land project. The work of Jieming Bian, Lei Wang and Jie Xu is partially supported NSF under grants 2433886, 2505381 and 2515982.
References
[1] Nvidia jetson
orin.
get-started- jetson Drin-nano- devkit
[2] Nvidia jetson xavier nx. get-started- jetson xavier nx ex-devkit
[3] Alam, S., Liu, L., Yan, M., Zhang, M.: Fedrolex: Model-heterogennous federated learning with rolling sub-mode extraction. Advances in neural information processing systems 35, 29677-29690 (2022)
[4] Alistarh, D., Grubic, D., Li, J., Tomioka, R., Vojnovic, M.: Qsgd: Communication-eficn sgd via gradient quantization and encoding. Advances in neural information processing systems 30 (2017)
[5] Caldas, S., Konečny, J., McMahan, H.B., Talwalkar, A.: Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv: 1 812.07210 (2018)
[6] Cho, YJJ., Wang, J., Joshi, G.: Towards understanding biased client selection in federated learning. In: International Conference on Artificial Intelligence and Statistics. pp. 10351-10375. PMLR (2022)
[7] Diao, E., Ding. J., Tarokh, V.: Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01266 (2020)
[8] Fang, Y., Loparo, K.A., Feng, X.: Inequalities for the trace of matrix product. IEEE Transactions on Automatic Control (92),, 2489-2490 (1994)
[9] Gim, I., Ko, J.: Memory- dnn training mobile devices. In: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. pp. 464-474 (2022)
[10] Goli, N., Aamodt, T.M.:: Resprop: Reuse sparsified backpropagation. In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pp. 1548-1558 (2020)
[11] He, K., Zhang, X., Ren, S., Sun, Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[12] Horvath, S., Laskaridiss, S., Almeida, M., Leontiadis, I., Venieris, S., Lane, N.: Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout, Advances in Neural Information Processing Systems 34, 12876-12888 (2021)
10

---

[13] Hu, X., Chen, Z., Feng, C., Min, G., Quek, T.Q., Yang, H.H.: Sparsified random partial model update for personalized federated learning. IEEE Transactions on Mobile Computing (2024)
[14] Huang, K., Yang. B., Gao, W.: Elastictrainer: Speeding up on-device training with runtime elastic tensor selection. In: Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services. pp. 56 69 (2023)
[15] Ilhan, F., Su, G., Liu, L.: Scalefl: aesource-adaptive federated learning with heterogeneous clients. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 14532-24544 (2023)
[16] Jacob, B., Kligys, S., Chen, B.. Zhu, M., Tang, M.. Howard, A., Adam, H., Kalenichenko, D.: Quantization and training of neural networks for efficient snnger-arithmeticoonly inference. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2704-2715 (2018)
[17] Jiang, Y., Wang, S.. Valls, V., Ko, B.J., Lee, W.H., Leung, K.K., Tassiulas, L.: Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems 34(12), 10374-10386 (2022)
[18] Kim, M., Yu, S., Kim, S., Moon, S.M.: Depthfl: Depthwise federated learning for heterogeneous clients. In: The Eleventh International Conference on Learning Representations (2023)
[19]
Krizhevsky, A., Hinton, G., al.: Learning multiple layers of features from tiny images (2009)
[20] Lai, F., Dai, Y., Singapuram, S., Liu, Zhu, X., Madhyastha, H., Chowdhury, M.: Fedscale: Benchmarking model and system performance of federated learning at scale, In: International conference on machine learning. pp. 11814-11827. PMLR (2022)
[21] Lai, F., Zhu, X., Madhyastha, H.V., Chowdhury, M.: Oort: Efficient federated learning via guided participant selection. In: 15th [USENIX} Symposium on Operating Systems Design and Implementation ((OSDI) 21). pp. 19-35 (2021)
[22] Lan, Z.: Albert: lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1 1942 (2019)
[23] Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), (2015)
[24] Lee, R., Fernandez-Marquee J., Hu, S.X., Li, D., Laskaridis. S., Hospedales, T.. HuszÀ "r, F., Lane, N.D., et al.: Recurrent early exits for federated learning with heterogeneous clients, arXiv preprint arXiv:2405.144: (2024)
[25] Li, C., Zeng, X., Zhang, M., Cao, Z.: Pyramidfl: A fine-graines client selection framework for efficient federated learning. In: Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. pp. 158-171 (2022)
[26] Li, J., Chen, T., Teng, S.: A comprehensive survey client selection strategies in federated learning. Computer Networks 110663 (2024)
[27] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogencous networks. Proceedings of Machine learning and systems 2, 429 429-450 (2020)
[28] Liao, Y., Xu, Y., Xu, H., Chen, M., Wang, L., Qiao, C.: Asynchronous decentralized federated learning for heterogeneous devices. JEEE/ACM Transactions on Networking (2024)
[29] Lin, J., Zhu, L., Chen, W.M., Wang, W.C., Gan, C., Han, S.: On-device training under 256kb
memory, Advances in Neural Information Processing Systems 35, 22941-22954 (2022)
[30] Liu, J., Che, T., Zhou, Y., Jin, R., Dai, H., Dou, D., Valduriez, P.: Aedfl: efficient asynchronous decentralized federated learning with heterogeneous devices. Proceedings of the 2024 SIAM International Conference on Data Mining (SDM). pp. 833-841. SIAM (2024)
[31] McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficiee learning of deep networks from decentralized data, In: Artificial intelligence and statistics. pp- 1273-1282. PMLR (2017)

---

[32] Okon, E., Rachakonda, V., Hong, H.J., Callison-Burchh C., Lipoff, J.B.: Natural language processing of reddit data to evaluate dermatology patient experiences and therapeutics. Journal of the American Academy of Dermatology 83(3), 803-808 (2020)
[33] Patil, S.G., Jain, P., Dutta, P., Stoica, I., Gonzalez, J.: Poet: Training neural networks on tiny devices with integrated rematerialization and paging. In: International Conference on Machine
Learning- pp. 17583. PMLR (2022)
[34] Setayesh, M., Li, X., Wong, V.W.: Perfedmask: Personalized federated learning with optimized masking vectors. In: The Fleventh International Conference on Learning Representations (2023)
[35] Simonyan, K. Zisserman, A.: Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 14cKv:1409.111 (2014)
[36] Sun, X., Ren, X., Ma, S., Wang, H.: meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In: International Conference on Machine Learning, pp. 3299-3308 PMLR (2017)
[37] Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q.. Jiang. J., Zhang, C.: Fedproto: Federated prototype learning across heterogeneous clients. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 8432-8440 (2022)
[38] Wang. J., Liu, Q., Liang, H., Joshi, G., Poor, H.V.: Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems 33, 761 1-11-7623 (2020)
[39] Wang, Q., Xu, M., Jin, C., Dong, X., Yuan, J., Jin, X., Huang, G., Liu, Y., Liu, X.: Melon: Breaking the memory wall for resource-efficient on-device machine learning. In: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, pp.
450-463 (2022)
[40] Warden, P.: Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint 180400320 (2018)
[41] Wu, F., Wang. X., Wang, Y., Liu, T., Su, L., Gao, J.: Fiarse: Model--heterogeneeoss federated learning via importance-awar submodel extraction. arXiv preprint arXiv:2407.19389 (2024)
[42] Wu, Y., Li, L., Tian, C., Chang, T., Lin, C., Wang, C., Xu, C.Z.: Heterogeneiit-aware memory ef- ficient federated learning via progressive layer freezing. In: 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS). pp. 1-10. IEEE (2024)
[43] Xie, K., Lu, S., Wang, M., Wang, Z.: Elbert: Fast albert with confidence-window based early exit. In: ICASSP 2021-222220 1221 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7713-7717. IEEE (2021)
[44] Xu, C., Qu, Y., Xiang, Y., Gao, L.: Asynchronous federated learning on heterogeneous devices: survey, Computer Science Review 50, 100595 (2023)
[45] Xu, D., Xu, M., Wang, Q., Wang, S., Ma, Y., Huang, K., Huang, G., Jin, X., Liu, X.: Mandheling: Mixed-precision on-device dnn training with dsp offloading. In: Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. pp. 214-227 (2022)
[46] Xu, J., Chen, Z., Quek, T.Q.. Chong, K.F.E.: Fedcorr: Multi-stage federated learning for label noise correction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10184-10193 (2022)
[47] Xu, Z., Yu, F., Xiong, J., Chen, X.: Helios: Heterogeneity-awanl federated learning with dynamically balanced collaboration. In: 2021 58th ACM/IERÉ Design Automation Conference (DAC). pp. 997-1002. IEEE (2021)
[48] Ye, R., Ni, 7.., Xu, C., Wang, J., Chen, S., Eldar, Y.C.: Fedfm: Anchor-based feature matching for data heterogeneity in federated learning. IEEE Transactions on Signal Processing (2023)
12

---

[49] Ye, R., Xu, M., Wang, J., Xu, C., Chen, S., Wang, Y: Feddisco: Federated learning with discrepancy-awee collaboration. In: International Conference on Machine Learning, pp. 39879- 39902. PMLR (2023)
[50] Zhang, T., Gao, L., Lee, S., Zhang, M., Avestimehr, S.: Timelyfl: Heterogeneity-wwar asynchronous federated learning with adaptive partial training. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5064-5073 (2023)
[51] Zhou, H., Lan, T., Venkataramani, G.P., Ding, W.: Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction. Advances in Neural Information Processing Systems 36, 25991-26003 (2023)
[52] Zhou, Y., Pang, X., Wang, Z., Hu, J., Sun, P., Ren, K.: Towards efficient asynchronous federated learning in heterogeneous edge environments. In: IEEE INFOCOM 2024- 122- Conference on Computer Communications, pp. 2448-2457. IEEE (2024)
[53] Zhu, S., Voigt, T., Rahimian, Ko. On-device training: first overview on existing systems. ACM Transactions on Sensor Networks 20(6), 1-39 (2024)
13

---

A The Algorithm of FedEL
In this paper, we introduce the sliding window training address the first limitation and tensor importance adjustment to overcome the second limitation. We present a comprehensive window-based important tensor selection scheme implemented by FedEL, as outlined in Algorithm [ Specifically, prior to the FL process, each client performs offline tensor time profiling for the DNN model (Lines 3-5), which is done only once. each online FL round, once the client receives the broadcasted global model. it evaluates the tensor importance for the current global model (Line 8), calculates the global tensor importance (Line 9), and adjusts the local tensor importance accordingly (Line 10). the
Based the previous round's training status, FedEL then slides resets the window entire DNN model is trained (Line 11). Once the window fixed, ElasticTTraine is applied within the window to select important tensors, freeze unselected ones, and train only the selected tensors (Lines 12-13). Finally, the server aggregates the models from all clients and broadcasts the updated global model for the next FL round.
Algorithm 1 FedEL
1: Input: Client N, training time threshold Tik. balance parameter 3, DNN model w.
2: Output: Trained model.
Offline and only once
3: for Each client n: do
4:
5: end for
Online
6: for Each FL round r: do
7:
for Each client n: do
8:
I n.r = #Cnsoriporraceeevluuuttooto.
Tensor Importance adjustment
9:
1 GetGlobalTensorri TeeCiboaTTessormmpprtacccuu,,,, w,-1,
n.r = AdjustLocall wew.a/////eeommmppaannn....r. 1
> Window Sliding
θ n.r Slide w., Teh 2n,r-1)
> Elastic Training
An,r = SelectlmportantTeesoor On,r Tths n.))
13:
Wn,r = TrainImportant/eesorr An.r. W,)
end for
15:
W,-1 Aggregate(wx.r)
Server side
16: end for
B Detailed Datasets and Baselines
Baselines. The following baselines are adopted for evaluation purposes:
(1) FedAvg 31 is the classic generic FL algorithm without accounting for system heterogeneity. Each client trains the same full DNN model.
(2) ElasticTrainer 14 directly deployed into the local training clients of FedAvg framework.
(3) HeteroFI. 7 facilitates training across heterogeneous devices by scaling the channels of convo- lutional layers cater to diverse computation constraints.
(4) DepthFL 18 segments the model into sub-models of varying depths, distributing them to clients according to their computing capabilities.
(5) PyramidFL
aims to enhance time-to-ae to-accuracy by considering both data and system hetero- geneity during binary client selection.
(6) TimelyFL 50 is heterogeneity-awwan asynchronous FL framework with adaptive partial
training.
(7) FIARSE 32
dynamically masks the unimportant layers with adaptive partial training.
14

---

Datasets, Models, and Tasks. To demonstrate FedEL's effectiveness across tasks, datasets, and ML models, we evaluate FedEL on four real-world datasets designed for FL applications at different scales. To follow the realistic non-iid data in FL scenarios, we partition the datasets into different clusters using a Dirichlet distribution with a equals 0.1.
Image Classification. The CIFAR10 dataset [19] consists of 60,000 colored images in 10 classes. The Tiny ImageNet dataset [23] contains 100000 images of 200 classes colored images. We evaluate the dataset with VGGI6 (35 model.
Speech Recognition. The Google Command speech dataset 40 covers 105,829 audio commands recordings. The data set is composed of 35 common words from the everyday vocabulary, such as "Yes", "No". "Up", and "Down". We evaluate the datasets with ResNet50 model for a 35-class keyword spotting task.
Natural Language Processing, Reddit 33 consists of comments from 1,660,820 users in the Reddit forum. In this dataset, we filter the users with less than 20 words in total and restrict to the 30k most frequently used words, as the same settings in the previous work (20)- Then, we fine turn the lightweight Albert [22] model for the next-word- next-word-rrediction task. The performance is evaluated by the perplexity loss, which lower is better. It's worth noting that Reddit datasets inherently exhibits
non-iid characteristies. We follow 43
generate the blocks of the lightweight Albert model.
50%
80%
60%
20000
Peskup
50%
Hole
28%
80%
Foddig
2606
30%
10%
200000000
0.4
0-04
2050
60
103
200
400
600
300
400
600
800
200
400
800
Whl- Slock tine thoun
Whl--lock Smo Doun)
Wall clock time thourt
Wal book the thour)
(a) CIFARIO
(b) Tiny ImageNet
(c) Google Speech
(d) Reddit
Figure 15: Impact of balancing parameter / on four tasks.
60%
0.5T + T,
Fasiivg
50%
80%
A rouu
80%
Fodfog
Fadtfing
ave
20%
10%
0.77
40%
Toss
30%
50
100
200
420
600
2001
400
200
000
200
400
600
Wall-coss time phour)
Wall dock trm (hou)
Wal--ook time (rou))
Wilk time (hour)
(a) CIFARIO
(b) Tiny ImageNet
(c) Google Specch
(d) Reddit
Figure 16: Impact of runtime threshold Toh. on four tasks.
80%
000000
60%
-C- FedAug
FidEL-C -edEL
50%
Acourayy
60%
Accunec
Foaling LeC
10%
Fosheg well
40%
Fedhip FodEL
900-00000
50
3CD
200
400
900
200
400
600
800
200
400
500
Woll clook time (hour)
Wall clock din (hou))
Wall clook Sime thout
Wall dlock time (nour)
(a) CIFARIO
(b) Tiny ImageNet
(c) Google Speech
(d) Reddit
Figure 17: Time-to-aceuracy FedAvg, FedEL-C and FedEL on four tasks.
B.1 Ablation
In the Ablation section of the main paper, analyze the effect of parameter settings on FedEL using the CIFAR1O dataset. Here, we show the remaining ablation results for other three tasks in a large 100-device scenario.
Impact of balancing parameter 3. Figure 15 illustrate the impact of varying 3 on time-to- -oacurrcy performance across the Tiny ImageNet, Google Speech, and Reddit datasets. In FedEL, the balancing
15

---

parameter 3 controls the trade-off between local and global tensor importance during adjustment. larger places greater emphasis on local tensor importance, reducing the influence of global model variations. Conversely, smaller 3 prioritizes global variations while neglecting local data heterogeneity. When 3 (fully local) or /3 (fully global), FedEL achieves lower accuracy than FedAvg. However, with moderate values (33 = 0.4 or 3 0.6), FedEL outperforms FedAvg by effectively balancing local heterogeneity with global model updates. This balance enables FedEL to capture both local and global tensor importance, leading to improved accuracy.
Impact of runtime threshold Te.. Figure 16 illustrates how varying the runtime threshold Tea affects performance across three additional tasks In a 100-device scenario. To ensure fair comparison with baseline methods, we Ta equal the full model training time on fastest device. We then vary Ta to analyze its impact, stopping the experiment once the total training time reaches the predefined limit. As shown in Figure [6] a smaller Tıa slows convergence. This occurs because slow clients must train the entire model, requiring more sliding-window movements and fast clients also perform additional window sliding, increasing overall training time and reducing efficiency.
The sliding window operates through two processes. Front edge movement: Expands the window by including blocks until their cumulative training time slightly exceeds To.- End edge movement: Shrinks the window by excluding unselected blocks. As shown in Figure 17 reducing To, results in slower convergence, as more rounds are required to train the full model. To evaluate the effectiveness of end edge movement, we compare it with a variant called FedEL-C. where the end edge is immediately shifted to the current front edge. Figure I7 shows that FedEL-C leads to lower accuracy than FedEl,, highlighting the importance of gradual end edge adjustments for maintaining model performance. This is because FedEL--C treats each window independently and does not adjust training tensors between consecutive windows, leading to accuracy degradation.
Sliding Window,
B.2 Important Tensor Selection
In the main paper, we demonstrated tensor selection in a large-scale scenario with 100 devices, using the VGG16 model the Tiny ImageNet dataset. Here, present tensor selection results for additional tasks. Figure 18 illustrates tensor selection on VGG16 with the CIFAR1O dataset for representative Orin and Xavier devices. Figure 19 shows results for ResNet50 the Google speech dataset, using representative devices from each of the four device types. Figure 20 presents tensor selection for fine-tuning the Albert model the Reddit dataset. Specifically, we freeze the pre-trained albert-base-ve v2 model and train only the newly added output layers. As observed, the number of windows required to train the full model varies across devices due their differing computational capabilities. Within each window, tensor selection is dynamically adjusted based on importance. For instance, if tensor in an earlier layer critical for model performance, FedEL can adaptively skip updating certain less important tensors (with higher indices). This ensures an optimal balance between training speedup and model effectiveness.
FL Round
Round
4
4
8-
9
10
15
25
à
10
20
25
Tensor
Tensor
(a) Orin
(b) Xavier
Figure 18: Tensor selection of CIFAR10 dataset.
16

---

FL Round
FL Round
3
5
10
30
60
90
120
150
180
210
30
60
90
120
150
180
210
Tensor
Tensor
(a) Orin
(b) 1/2 Orin
0
0
1
2
2
FL Round
3
FL Round
3
4
5
5
6
7
7
8
0
30
60
90
120
150
180
210
30
60
90
120
150
180
210
Tensor
Tensor
(c) 1/3 Orin
(d) 1/4 Orin
Figure 19: Tensor selection of Google Speech dataset.
0
1
2
2
FL Round
3
FL Round
9
4
A
5
5
6
6
7
8
30
60
90
120
30
60
90
120
Tensor
Tensor
(a) Orin
(b) 1/2 Orin
0
0
1
2
2
FL Round
FL Round
3
4
5-
5
7
8
8
0
30
60
90
120
30
60
90
120
Tensor
Tensor
(c) 1/3 Orin
(d) 1/4 Orin
Figure 20: Tensor selection Reddit dataset.
17

---

B.3 How much does the training time deviate from the target time Ti
The differences in model architectures contribute to deviations between FedEL's training time and Tik The table presents the per-round average training time of FedEL compared to T As observed, for convolutional networks (i.e., VGG16 and ResNe150), the deviation ranges from 3.2% 6.8%, whereas for the LLM model (i.e., Albert), the deviation is 18.9%. Despite these variations, FedEL significantly accelerates training compared to FedAvg full-model training, achieving 1.87x speedup small-scale practical edge device scenario and a 3.13x 3.87 speedup large-scale simulation scenario.
Table 2: Deviation between the training time and Tik-
CIFARIO
Tiny ImageNet
Google speech
Reddit
FedEL Ta
38. 38.2min
45. min
54.9min
48.6min
36.0min
42.2min
53.2min
40.9min
Difference
6.1%
6.8%
3.2%
18.9%
FedAvg 71.8min
161.9min
212.9min
152.1
Speedup
1.87x
3.59x
3.87
3.13x x
B.4 FedEL with particular algorithms which try to address any data non-IIDness.
To assess FedEL's compatibility with aggregation algorithms beyond FedAvg, we integrated it with FedProx (27] and FedNova (38), both designed for non-llI data scenarios. Following their official implementations, we modified local updates and global aggregation to incorporate FedEL's adaptive tensor selection.
The table below compares the performance of FedProx/FedNova with and without FedEL on CIFAR10 dataset. As shown, FedEL. is not restricted to FedAvg; it can be seamlessly integrated into other FL aggregation methods, leveraging their advantages while mitigating their limitations, particularly in heterogeneous device environments.
Table 3: Time-to-accuracy for combining FedProx and FedNova with our FedEL.
Method
Acc
Time
Speedup
FedProx 56.1%
82.3h
N/A
FedProx + FedEI.
56.6%
45.4h
1.81x
FedNova 66.3%
84.7h
N/A
FedNova FedEL
66.1%
47.8h
1.77x
B.5 Statistical Comparison.
To confirm the significance of accuracy improvements, provide detailed statistical analysis, including confidence intervals. As shown in our box plot Figure 2I the confidence intervals indicate that method maintains exceeds accuracy with statistically significant improvements baseline methods.
Asueso m)
Papporty
Hebwort
Destet
Prnmort
Timirt
Pwel
reddop
Pyonestz
Toudet
Featt
Featoy
Hearort
Prunsnt
Tiwerr
Festt
Devirt
Promert
(a) CIFARIO
(b) Tiny ImageNet
(c) Google Speech
(d) Reddit
Figure 21: Accuracy statistical comparison.
18

---

Table The value of the O1 term in the theoretical convergence upper bound for FedEL is analyzed for both cases: with and without rollback.
Method
O₁ mean
O1 std
Rollback
63.06
8.62
Not Rollback
78.18
2.62
Table 5: Partition rate 25%
Image Classif.
Speech Recog.
NLP
Method
Acc. / Time / Speedup
Acc. / Time / Speedup
Acc. / Timme Speedup
FedAvg
33.76% 563.1h N/A
58.04% 709.8h N/A
77.48 / 54 /N/A
ElaiticTrainer
27.65% 158.6h 3.55x
47.96% / 184.3h 3.84x
81.02 176.2h 3.10x
HeteroFL
30.56% / 248.2h / 2.26x
51.47% / 265.9h / 2.66x
80.11 / 206. / 2.65x
DepthFL
34.14% / 198.3h 2.83x
54.23% / 207.4h 3.42x
78.08 / 212.4h / 2.57x
PyramidFL.
34.70% /497.4h 1.13x
58.12% 587.4h 1,21x
77.68 / /188.2h / 1,31x
TimelyFL
33.53% 198.th 2.84x
56.49% 193.2h 3.67x
3,07x
FIARSE
33.98%
191.5h 2.94x
58.13% / 198.2h 3,58x
77.31 / 191.0h 2.86x
FedEL
34.96% 156.8h 3.59x
58.26% 183.3h 3.87x
77.23 174.5h 3.13x
B.6
Does the method rolling back blocks if necessary?
The rollback mechanism in sliding window training ensures that earlier layers can be retrained, allowing the model to refine learned representations rather than reinforcing suboptimal updates. This particularly beneficial because deeper layers rely on feature representations from earlier layers. If earlier layers contain suboptimal representations, they can propagate errors throughout the network. By rolling back, the model can correct these errors and improve generalization, leading to more stable and effective learning.
In the convergence theorem of FedEL (Appendix E tensor selection introduces an additional bias term O1. To analyze the impact of rollback, we designed two training scenarios:
1. Sliding window training with rollback, where layers can be revisited and updated.
Sliding window training without rollback, where the window shifts forward after fixed number of rounds without revisiting earlier layers.
Table presents the statistical values of the bias term 01 for both cases. As shown, the average value of 01 is smaller when rollback is allowed, compared to when it is not. This provides theoretical
evidence that rolling back layers reduces the upper bound of convergence, leading more stable and efficient training.
C The Improvements and Clarifications from NeurIPS Rebuttal
C.1 Client Partitioning Scripts
CIFAR-10 (10-client hardware deployment): We use full participation, where all 10 NVIDIA devices join every training round. This setting reflects small-scale real-world deployment with stable device availability.
Nettee, Google Speech Commands, and Reddit (100-client simulation): We adopt partial participation, where 25 clients are randomly selected out of 100 in each round (i.e., 25% participation rate). This follows common practice in large-scale FL simulations and models realistic device availability constraints.
To evaluate the impact lower participation, we conducted experiments with a 10% participation rate. As shown in Table 5nd 6 all methods experienced slower convergence, leading to longer training time while maintaining similar accuracy. However, FedEI. consistently achieves the highest accuracy and best efficiency across all tasks, confirming its advantage under sparse client participation.
19

---

Table 6: Partition rate 10%
Image Classif.
Speech Recog.
NLP
Method
Acc. / Time Speedup
Acc. / Time Speedup
Acc. / /Tmee / Speedup
FedAvg
33.75% / 782.4h N/A
58.01% 987.3h N/A
77.45 764.2h N/A
ElaiticTrainer
27.62% / 220..4 3.55x
47.94% / 255.3h 3,87x
81.01 / 222.0h 0h 3.29x
HeteroFL
30.52% / 344.8h 2.27x
51.45% / 392.2h 2.52x
80.09 288.7h 2.65x
DepthFL
34.12% / 275.7h 2.84x
54.21% / 310.5h 3.18x
78.07 / 298 9h 2.56x
PyramidFL.
34.68% 7711.3 1,10x
58.10% / 805.1h 1h/ 1,23x
77.66 / 628.3h 1.22x
TimelyFL
33.50% 278.4h 2
56.45% / 267.2h 3.69x
80.89 / 233.3h 3.28x
FIARSE
33.96% / 269.2h 2.91x
58.11% / 271.4h 3.64x
77.28 251.6h 3.04x
FedEL
34.94% / /
58.24% / 257.0h 3.96x
77.21% / 226.0h 3.38x
Table 7: Performance under different Jetson power modes.
Method
Acc.
Time
FedAvg
58.21 %
550.2h
FedEL
59.35%
174.3h
C.2 Jetson Power Modes
In our experiments, Jetson devices ran in MAXN mode to ensure consistency. We reran experiments under varied power settings (10W, 15W, MAXN). We measured training times under each mode and used this variability in the FedEL scheduling process. As shown in the Table 7 FedEL still outperforms FedAvg in both accuracy and training time. These results confirm that FedEL remains effective even in more diverse and realistic hardware environments.
C.3 Communication and System Overhead
To clarify, our method does not add extra communication cost-in fact, reduces it compared to FedAvg. This is because FedEL only uploads selected important tensors rather than the full model. As shown the table, FedEL. results in: 1. Lower communication time per round than FedAvg. Communication taking up only small part of the total training time. We also measured the runtime of FedEL's system modules (e.g., sliding window, tensor importance update, and selection). As shown in Table the added overhead is minimal and has negligible effect on overall training time.
C.4 Discussing potential extensions of our work to handle communication heterogeneity and integration with differential privacy.
Our study focuses computational heterogeneity, dominant bottleneck in mobile edge environ- ments. In practice, training time on mobile devices far exceeds communication time, especially with modern 5G/WiFi networks. For example, ResNet50 ( 97.7 MB) takes 0.28-1.3 minutes to transmit over 10-45 Mbps uplink (e.g., AT&T, 2024), while training on an NVIDIA Jetson Xavier takes 38.3 minutes, making communication relatively negligible in our setting. Nonetheless, our method can incorporate communication heterogeneity. During offline profiling, we estimate tensor sizes. client bandwidth is known, the tensor selection can be extended to:
max A
I, s.l. Tym: + Thmo (A) + (A) S Tik-
where (A) estimates transmission time. This enables FedEL to jointly optimize computation and communication under unified latency constraint. Our method is fully compatible with privacy- preserving techniques like Differential Privacy (DP) and Secure Aggregation (SA), as FedEL operates at the system level and does not modify interfere with model eneryption privatization.
C.5 Detailing how approach can be adapted for transformer models.
As detailed in Section B our Supplementary Material, our implementation for Albert follows the block design from 43), placing early exits after each encoder stack. This supports
20

---

Table System overhead.
Method
Communication 2.45 min (3.2%)
Tensor processing N/A 0.97 min (2.4%)
Average round time
FedAvg
75.43 min
FedEL
1.09 min (2.7%)
40.34 min
variable-depth execution and is fully compatible with sliding window mechanism. Regarding shared-weight layers, such as embeddings and grouped attention heads, we handle them as follows: (I) Shared Embeddings: We profile the shared tensor single unit. Backward time is traced from the output back to the shared embedding, reflecting its impact on both input and output. The weight update time is counted only once since the tensor updated
per iteration.
(2) Grouped Attention Heads: For attention layers with multiple trainable projections (query, key, value, output), treat each as separate tensor and profile their backward time individually. If some projections are shared, we track their computation cost jointly during backpropagation.
C.6 More Related Works
Recurrent Early Exits (ReeFL) 24 ReeFL enables early exits in LLMs during FL. Our method FedEL can work with ReeFL by applying tensor selection inside each early-exit block. We combined both by inserting exits into transformer layers and applying FedEL within each. As shown below, FedEL+RecFL reduces training time while maintaining accuracy, confirming that the two approaches are complementary.
ScaleFL 13 ScaleFL adapts model width/depth based device profiles, but uses static selection via a meta-scheduler. In contrast, FedEL. dynamically selects tensors in each round based on importance, allowing finer control and better adaptability.
NeurlPS 2023 (51) offers convergence guarantees but applies global pruning without local data or neuron importance awareness, limiting its adaptability at the client level.
TNNLS 2022
17 supports distributed pruning but incurs significant communication overhead due to the need transmit importance scores and pruning decisions each round.
DAC 2021 (Helios) 47 enables dynamic model adaptation but requires frequent communication and additional optimization overhead, especially for straggler mitigation.
TMC 2024 [33 does not perform partial training; instead, it trains the full model and uploads only subset of parameters, which is orthogonal to our focus on efficient local training. In contrast, FedEL performs lightweight, local tensor selection using an importance-basee mecha- nism, incurs no extra communication, supports fine one-grained dynamic adaptation, and avoids heavy offimization-offrrinnn practical and scalable solution to device heterogeneity.
D Limitation
To evaluate the effectiveness of FedEL, we conduct experiments in two settings: small-scale practical setup using real edge devices, and a large-scale simulation. Due to hardware limitations, the practical setup includes only two types of edge devices. The large-scale simulation then designed based on system measurements collected from these two devices. While this approach demonstrates promising results, may face challenges when sealed to real-world environments with more extreme heterogeneity in client computational resources. Additionally, this work does not account for variations client network bandwidth, which we plan to explore future work.
E Convergence Theorem.
We consider one server and N edge devices. Each device n E W (1, (1, has its own set of local data samples D,, In a supervised learning setting each device aims to find a learning model O, E Rd, where de denote the dimensions of the model. A mask A,, E {0, 1] is selected for each device n E N based on the FlasticTrainer. During local update, each device n only updates those parameters in the global model that correspond to non-zero values of the masking vector Am.
21

---

Let " denote the local model of device n at the beginning of local update iteration i in training round The local model of device n is updated using SGD as follows:
(1) - w ((()) A. VJ (w,, (1), (O), I.....
where @ denotes the element- wiiee product, v7(1) is the learning rate, (-) is the loss function and (t) is the local batch sample chosen uniformly at random from the local dataset. After performing local update iterations, each device sends its final model to the server.
w. n. = w)(() - -()))))()))))))))) O
(1),
(4)
In the aggregation step, we consider that the server aggregates the received final local models by taking the masking vectors of the devices into account. The global model for the next communication round can thus be determined through stable aggregation of unfrozen parameters, as follows:
w 1)
Σ Cn(t) @ (t) nEN
(5)
where (c.(tt)k
(,,)(A, denotes the k-th tensor selection of mask (t) at training round t. Using (t)) in indicates that the server only aggregates the updated parameters from the participating devices.
(A.(0))
The analysis relies on the following assumptions, which are commonly used for obtaining the convergence rate of different FL algorithms the literature.
Assumption E.1. The function fw(w), n E N is L-smooth and satisfies:
f (ww, (t) 2f (t), fi), E N. 1, where f denotes the minimum value of fn (n))..
(6)
Assumption E.2. fn fw'",, (t), 6', (t)) is an unbiased stochastic gradient of function f.(w). The variance of the masked stochastic gradients is bounded for each device n E N. We have
(t) @ (wo" (t), by, (t)) - c, @ f (t)ll² < & n E N,, 1, I,.... Vt
(7)
Assumption E.3. The expected squared L2-norm of the masked stochastic gradients for the devices is uniformly bounded. We have
(t) O (t), (t)) S , n N, I,..,,,, Vt
(8)
Lemma E.4. The following inequality holds for any vectors and = E Rdd, for which there exists Q > 0 satisfying (ming (xr @ 2)xl S Q, and for any vector E Rd.
(z. ① 2) S max(y)s(x, + Q (dmax(y))
-E()e k=1
(9)
where (.,-) denotes the inner product operator Rdd.
Proof. Given vectors y. and we form diagonal matrices X, Y and Z, respectively, Note that wee can write (z, @ 2) as the form of the trace matrices X. and Z product, i.e., (2,8 0 Tr(XY Z). By using Theorem 3 in [8), we have the following inequality:
Tr(XYZ) < A (Y
Xx( -Tr(Y)
(10)
where A (Y) and Ax(X2) Z) are the largest eigenvalue of matrix Y and the smallest eigenvalue of matrix XZ, respectively. Since the considered matrices are diagonal, we have Å AY max (y)) and (X = ming (x @ z){- Hence, we have
(x,y02) < (ax(v)x(z, 2) + min(# x @ 2)x
dmmx(yy) m
Σ (v)&
(11)
A=1
max (v) x) + Q
dmax(yy) Σ (w)
(12)
Since max; (v)x - Ei_(v)) 0, by considering |ming(x @ 2)al Q, Lemma E.4
proved
using the second incquality.
22

---

We define the term (t) max; (t)( (a network. Note that in the full device participation scenario, N S (t) I 1, E N. Then, the following Theorem E.5 shows that the convergence bound of employing the masks to address the device heterogeneity issue FL.. Then, the following Theorem E.5 shows that employing the masks to address the device heterogeneity issue in FL leads to bias term in the convergence bound. However, does not affect the convergence rate, which similar to the observation in paper [41].
to quantify the degree of device heterogeneity in the
Theorem E.5. Under Assumptions 1-3, and for smooth and non-convex loss finctions, if the total number of communication rounds T is pre-defined and the learning rate ri() is smart enough such that ntt) 1 S LN77. we have
N
1 T
2 nFT (F(w F*) LN
B (VF(w)(()))
de > (c)(t)) +L2 (T 12(2--1)
+2
(t)
(13)
6
where V is a constant satisfying max (V fn (w", (t),) o (f))( V. F" = = F(w"),
where w is the global optimal weight. L. E and G are constants defined in Assumptions 1-3.
Proof. Considering the smoothness of fn(w),, E N, in each training round /: 2 1, we have
L
EF(w, + 1)) < (FF(w,(()) B(w (t + 1) w,(t), (t), V ((w Bllw, (t + 1)) (1))l
(14)
We first find an upper bound for + 2 as follows:
E. W (1 +
22)(1)] (a)
(1)E
Σ c.
V (w (!)
(t) E
ΣV
(t) O ( (t), () (t) 2 (w
n==
M,
N
+ (t)
Σ
(t) @ fn (t))
(15)
Mr
where equality (a) results from 4 and 5
Equality (b) is obtained via basic equality E||=||2
E||=- - |E + ||Ez] 12 for any random vector z. By using Assumption E.2 we have obtain an upper bound of M2 follows:
Mi E
Ct (f, ct, o (ww,
N
< N
[1 (2()) ct.(t) 2 fn (w,, (t))
m=
< 2
(16)
Sn
n=
23

---

By considering the convexity of || ||2 and by using (t) = max we can obtain an upper bound of M2 follows:
M2
O o fn(w,,t))
S N
cn((t 2 fn(w, (+))l
==1
< N
(t) fu (w (+))
(17)
By combining 15
16 and
we have the following inequality:
N
E| Wt [ wy w,(t)| ( N7
Σ a (t) ΣΣ (t) (l
(18)
Now, we aim to obtain an upper bound of B( (1+ (1) (F(w,(()) We have E(w, (t + 1) wy(t), VF(w,(t)))
E
Cl o VL
N
(t)
(t) (1(((
do Σ (cn()))
Et)E
(t) (1) ( (F(w + (())T
do
k=1
(4)
de
(t()
E
Su (1)) (1) + (1)=T
do7n(1)
(cn (t))
(19)
where equality (a) results from 44 and 5 Equality (b) follows from EV fn (1), by (t)) (n (t)). Inequality (c) holds by using Lemma E.4 Inequality (d) follows from 7n (1) 2 N.
To find an upper bound for -E + EN lows:
V (w (4)), (F())(()) >
we first represent as fol-
N
-E
fa W (f)), VF(w,(()) (F),
==1
N
N
B
1
E
Σ
(n (t))
fn
|| (t)
2
N
2
n=1
(20)
Then, E
I (V - (w, (w)))))) is bounded as follows:
E
(wn,
N
N
L
E
W (t) w. (t)]l
(21)
N
24

---

where inequality (a) results from the convexity of || 11a. Inequality (b) results from Assumption E.1 Now, we aim bound E| 20 (t) w (t)l for i 2, ..... By using (3)
have
E ( (1)l
=E
((t) ((t))An())) (1)
i-1 E
()/(t(i--1) j=1
V.. (1
-
(22)
where the last inequality results from Assumption E.3
By combining
21
and (2
we have
N
Σ
(Vnn( (0)) - (w
$ (1))((((i-))))
(23)
N
By combining 14
and
14
to
we have
L - N7) Σ
N
N
du
EF(w, (1 + 1)) < (F (1) +
22 + +[)[
Σ
dowr(!) - (cn(()))
n=1
n=1
k=1
n(tT
(V(W (t))l (+))) - 12 1)
T(T
12
n(t)
N LNT (t))l
(24)
Since rt() 1) S ,N we have last term
m(() Σ EL (N LNr)(())(t))
(t)))
By rearranging the terms in (24) we obtain
N
2 S (EF(W, EF(w,(t 1))) + LNT
E (V)(
2
T
n=1
L2 ((- 1)
2V (t) > (cn()))s
(25)
Finally, we multiply both sides of 255) by and sum over t = 1..... T. Then, Theorem E.5 is
concluded by considering that the first term on the right-hand side of (25) is telescoping series. We have
2
Σ
2 (EF (1)) (1+ 1))) 1TT (F(w, EF + +1))
NTT
t=0
< T
(w (1)) - F*)
(26)
where the last inequality is obtained by considering that EF + +1)) > F*
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
26

---

Justification:
Please see the limitation discussion in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "L "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peeiiiaiinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical convergence analysis provides the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper fully discloses all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper.
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: We consider to provide open access to the data and code the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc/puulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
28

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see our experiment setup main paper and supplemental material. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see the statistical comparison in Appendix B.5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All information about the computational resources is provided in the experi- mental setup section.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp////aaaaasssp////////aaaaaa cc/publlc/Bthiccsuueeel....
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research and not tied to particular applications, which has no negative societal impacts the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
30

---

Answer:
[NA]
Justification:
The models and datasets in our paper have no such risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we have cited the original papers for the code, data and models our paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com ddtasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: If the paper is accepted, we will consider providing open access to the data and code, but not now.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
31

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy hatoge//nearris.cc/ooofeeeeeeeee for what should or should not be described.
32


Reviews:
[{"review_id": "H9GEPUgzbq", "reviewer": "Reviewer_JWBy", "summary": "In brief, the authors proposed an algorithm FedEL, which applies Elastic Trainer (ET) into Federated Learning (FL) in order to solve the core problem of the core problem of FL. \nFederated Learning (FL) is a privacy-preserving machine learning method, involving devices performing local method training and sharing parameters with a central server with a global model update. Its core challenge is that heterogeneous device speeds force the server to wait for slow “straggler” clients to finish their local updates each round, causing significant delays and limiting scalability. \nThe existing algorithm can be divided into three categories: Client Selection, Asynchronous FL, and Partial Training. The limitation of each category is listed as below: \n1.\tClient Selection: Frequently excludes slower clients, causing their unique data distributions to be underrepresented and degrading global model accuracy. \n2.\tAsynchronous FL: Over-relies on fast clients’ frequent updates while slow clients’ contributions are delayed or stale, which can hinder convergence and reduce final performance.\n3.\tPartial Training: Width scaling leads to channel-mismatch during aggregation, and depth scaling trains only shallow layers, limiting the model’s ability to learn deep semantic features and harming task accuracy.\nTo propose a better algorithm, the authors consider applying Elastic Trainer (ET) algorithm into FL. ET is an algorithm to propose model training on a single device. Its strength is listed as below:\n1.\tControlled Training Acceleration: Uses a runtime threshold to limit training time and focus only on the most important tensors, achieving significant speedups. \n2.\tDynamic Tensor Selection: Automatically profiles and ranks tensor importance each iteration, updating only high-impact tensors while freezing the rest. \n3.\tMinimal Accuracy Loss: Delivers substantial speedups (up to ~3.8×) with only slight drops in final model accuracy. \n4.\tModular Design: Separates tensor timing and importance evaluation into distinct components, enabling easy integration into existing single‐device training pipelines.\nHowever, when applying ET to FL, 2 limitations are faced:\n1.\tStraggler Training Scope Limitation: On slow clients, tight time budgets force ET to update only the fast-to-train back-end layers, since it is natural that front layers are the feature-extraction stages and Back layers are the classification stages. This leaves the front-end feature extractors untrained and weakening the global model, particularly under non-IID data.\n2.\tExacerbated Local Model Drift: Diverse client data distributions lead to biased tensor importance estimates; training only locally important tensors amplifies these biases, causing client models to drift further from the global model and impairing convergence and final performance.\nMotivated by the limitations above, FedEL is proposed. To address each of these two limitations, FedEL is composed of two parts. \n1.\tWindow-Based Training: The DNN is partitioned into blocks; each round, a sliding window—based on the client’s time budget and past training—selects which blocks to train (expanding at the front and pruning at the back). Within this window, ET updates only the covered tensors, ensuring even slow clients gradually train all parts of the network.\n2.\tTensor Importance Adjustment: Clients compute a “global” tensor importance from the change in the global model (difference ÷ learning rate) and blend it with their local importance scores using weight β, ensuring tensor selection reflects both local data and global update needs.\nThrough evaluation of FedEL, three core findings are found.\n1.\tTime-to-Accuracy Improvement: FedEL achieves up to a 3.87× speedup in time-to-accuracy over FedAvg while matching or exceeding FedAvg’s final test accuracy.\n2.\tResource Efficiency: FedEL substantially reduces both memory overhead and energy consumption during training compared to existing methods.\n3.\tAblation Validation: Ablation studies confirm that each of FedEL’s core components (sliding-window training and tensor importance adjustment) is necessary and contributes meaningfully to its overall performance.", "questions": "1.\tReproducibility of the 100-Client Simulation\nQuestion: Can you specify the exact counts of clients in each speed tier (1×, ½×, ⅓×, ¼×) or provide the random seed/scripts used for allocation?\nWhy It Matters: Without this information, external researchers cannot precisely reconstruct large-scale experiments, and small differences in tier proportions can meaningfully affect convergence curves.\nIf provided, the Quality mark will raise.\nIf continue to refer only to “random allocation” without details, the Quality of this paper would stay at to 2, Fair since the experiment data may not be reproducible.\n2.\tQuantification of Communication and System Overhead\nQuestion: What is the additional communication cost (e.g., message size or frequency) and system-level complexity introduced by\na.\tsliding-window metadata\nb.\tglobal-local importance fusion\nWhy It Matters: Any gains in training time or accuracy could be offset by larger or more frequent messages, which matters in bandwidth-constrained settings.\nIf related data is provided, the Significance & Quality mark will raise.\n3.\tElaboration of Limitations and Future Directions\nQuestion: Can you expand the Conclusion to more explicitly discuss the known limitations of FedEL (e.g., only four speed tiers, need to tune β and Tₜₕ) and outline concrete next steps (e.g., adaptive β, dynamic tiering, integration with compression)?\nWhy It Matters: A clear articulation of where FedEL may fail and how to improve it will guide practitioners and strengthen confidence in your contributions.\nIf answered, the Quality mark will raise.", "limitations": "1.\tOnly four speed tiers.\n2.\tNeed to tune β and Tₜₕ.\n3.\tPotential Communication and System cost not discussed.\n4.\tConsidering more categories of tasks may be better, such as object detection, video classification, time series forecasting, recommender systems, medical image analysis.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This article proposed the FedEL algorithm, which adjusts the Elastic Trainer (ET) algorithm in order to apply it into Federated learning (FL). Its strength is listed as follows:\n\tWindow-based training approach solving the limitation of limited training scope on slower clients if ET directly. FedEL’s sliding-window training can be described as follows:\n\tBlock Partitioning\nSplit the DNN into contiguous “blocks” (e.g., residual units or layer groups).\n\tInitial Window Initialization\nStarting from the first block, accumulate each block’s offline-profiled training time until the sum first meets or exceeds the time budget, denoted by T_th, forming the initial window of m blocks.\n\tWindow-Scoped Tensor Selection\nWithin this window, run a knapsack-style DP under the same time budget to pick the most important tensors for forward/backward passes and updates; all other tensors remain frozen. The importance of each tensor is calculated by Tensor Importance Adjustment, as analyzed in (2)\n\tFront-Edge Advancement\nAfter each round, extend the window’s front edge by adding the next block(s) until the cumulative time again reaches T_th.\n\tBack-Edge Pruning\nRemove any trailing block in which no tensors were selected during that round, avoiding wasted computation.\n\tWindow Reset\nOnce the front edge reaches the final block, reset to the initial window position and repeat the push–pull process until convergence.\n\n\tTensor Importance Adjustment solves limitation of Exacerbated Local Model Drift, which is the Variations in data distribution cause significant differences in tensor importance across clients. The author uses a formula to calculate the importance of each tensor, where:\n\t I_(n,r+1) denoted the final tensor importance\n\t I_(n,r+1)^localdenoted the local importance of each tensor on client\n\t I^gdenoted the global importance of each tensor \n\t β is a weighting factor ∈ [0,1]\n\tThe experiment is well defined. FedEL shows the best accuracy and least run time among all considered algorithms, including FedAvg, ElasticTrainer (direct deployment), HeteroFL, DepthFL, PyramidFL, TimelyFL, FIARSE, FedEL. \nThe weakness of this article is listed as below.\n\tLack of Specific Allocation Ratios. \nFor the experiments using 10 devices, the authors clearly state that they use 5 NVIDIA Jetson Xavier NX and 5 NVIDIA Jetson Orin. However, for the experiments using 100 devices, the author only states that They throttled the Orin’s throughput to four tiers—100%, 50%, 33%, and 25% of its native speed—to simulate heterogeneous device classes, without clearly stating the number of each class. This limits the reliability of experiment data. \n\tIgnored Communication Heterogeneity\nIn both the 10- and 100-device experiments, the authors do not model variations in network bandwidth or latency among clients, omitting the impact of communication heterogeneity on training time and convergence, which can be significant in real-world federated deployments.\n\tNeed to tune β and T_th\nSince the decision depends on 2 constant, β and T_th, these two constants needs to be tuned while training. \n\tOverly Simplified Simulation Heterogeneity\nIn the 100 devices experiment, the authors only consider speed tiers (1×, ½×, ⅓×, ¼×), without consider other factors such as bandwidth, memory, and energy consumption. \n\tLimited Evaluation Tasks and Architectures\nExperiments are conducted only on image classification (VGG16, ResNet50), keyword spotting, and lightweight NLP, without validation on larger-scale Transformers, GNNs, or multimodal tasks.\n\tInsufficient Analysis of Communication and System Overhead\nAlthough memory and energy savings are reported, the additional communication overhead, system implementation complexity, and latency introduced by the sliding-window mechanism and importance fusion are not quantified, making it difficult to assess deployment costs in real network environments.\n\tOverly Concise Conclusion\nThe Conclusion section does not adequately discuss method limitations, future work, or potential application scenarios, nor does it integrate theoretical convergence analysis with experimental findings, reducing the paper’s completeness and persuasiveness.", "was_revised": true, "final_justification": "The author has responded to each of the weaknesses I stressed and provided related supported material and data. The author clearly stated their strength and research focus, explaining that their research focuses on the computational capability of edge devices. Hyperparameter analysis is stated as future work. \n\nGiven the limited originality of the work, I keep my original score.", "rebuttal": "**W1: Large-Scale Simulation Setup**\n\nThank you for your comment.\nWe acknowledge that the term “random allocation” may have been unclear. In our 100-client simulations, we uniformly assign 25 clients to each of four speed tiers, with clients randomly selected within each tier—consistent with DepthFL, HeteroFL, and FIARSE. We will include the client partitioning scripts in the final submission.\n\n---\n\n**W2: Communication Heterogeneity**\n\nWe thank the reviewer for this insightful comment.\nOur study focuses on computational heterogeneity, a dominant bottleneck in mobile edge environments. In practice, training time on mobile devices far exceeds communication time, especially with modern 5G/WiFi networks.\nFor example, ResNet-50 (~97.7 MB) takes 0.28–1.3 minutes to transmit over 10-45 Mbps uplink (e.g., AT&T, 2024), while training on an NVIDIA Jetson Xavier takes ~38.3 minutes, making communication relatively negligible in our setting.\n\nNonetheless, our method can incorporate communication heterogeneity. During offline profiling, we estimate tensor sizes. If client bandwidth is known, the tensor selection can be extended to:\n\n$\\max_{\\boldsymbol{A}} \\boldsymbol{A} \\cdot \\boldsymbol{I} ~~ s.t. T_{fw} + T_{bw}(\\boldsymbol{A}) + T_{tx}(\\boldsymbol{A}) \\le T_{th}$\n\nwhere $T_{tx}(A)$ estimates transmission time. This enables FedEL to jointly optimize computation and communication under a unified latency constraint.\n\nTo validate this, we simulated 100 clients across four bandwidth tiers (10, 20, 30, 45 Mbps, 25 clients each). We compared: (1) FedAvg, (2) FedEL, and (3) FedEL+Tx (our method with transmission-aware selection). FedEL+Tx consistently achieved comparable accuracy to FedEL with lower total training time, confirming its effectiveness.\n\n| Method   | Image Classif.  | Speech Recog.   | NLP            |\n|----------|-----------------|-----------------|----------------|\n|          | Acc. / Time     | Acc. / Time     | Acc. / Time    |\n| FedAvg   | 33.76% / 571.5h | 58.04% / 710.2h | 77.48 / 547.1h |\n| FedEL    | 34.96% / 157.5h | 58.26% / 184.2h | 77.23 / 175.0h |\n| FedEL+Tx | 34.98% / 156.8h | 58.24% / 183.3h | 77.22 / 174.5h |\n\n---\n\n**W3: $\\beta$ and $T_{th}$**\n\nThank you for your comment.\nWe fixed $\\beta$ and $T_{th}$ in our study and performed ablation (Sec. 5.3) to assess their impact. $\\beta$ balances global vs. local tensor sensitivity and affects accuracy. $T_{th}$ controls the time budget per round; larger values allow more updates, but increase computation cost.\n\nDeriving explicit analytical relationships between these parameters and accuracy/convergence is challenging due to the coupled effects of training and communication dynamics. That said, we agree adaptive tuning, e.g., via reinforcement learning or bandit control, offers promising potential, which we plan to explore in future work.\n\n---\n\n**W4: Simulation Heterogeneity**\n\nWe appreciate the reviewer’s thoughtful observation.\nIn this work, we focus on computational heterogeneity, as it is a major bottleneck in mobile FL and allows us to clearly study its impact. To ensure fair comparison, we also follow the setup used in prior work that only considers device speed differences.\n\nWe agree that communication, memory, and energy are also important. However, modeling all types of heterogeneity at once would make the problem more complex and shift the focus away from our main contribution.\n\nOur method is flexible and can be extended. As shown in our response to **W2**, we can include communication costs in the tensor selection process. Similarly, memory and energy limits can be added by updating the profiling step. Early results show our method still works well when communication is considered.\n\n---\n\n**W5: Tasks and Architectures**\n\nWe thank the reviewer for highlighting this important point. Our current experiments are focused on three representative FL tasks—image classification (VGG16), speech recognition (ResNet50), and lightweight NLP (Albert)—which were selected to align with realistic use cases of FL on mobile and edge devices. These models and tasks are commonly adopted in prior work (e.g., DepthFL, HeteroFL, and FIARSE), ensuring fair benchmarking under resource-constrained settings.\n\nWe fully agree that evaluating on larger-scale architectures (e.g., full-size Transformers), graph neural networks (GNNs), and multimodal tasks is a valuable direction. However, such extensions often require considerable training resources and pose challenges for practical FL deployment at the edge. That said, the underlying mechanisms in FedEL—namely sliding-window block scheduling and dynamic tensor selection—are model-agnostic and can be adapted to other architectures. For instance, our supplementary material (Section B) demonstrates how FedEL is applied to Transformer blocks in NLP.\nWe plan to extend our evaluation to more complex tasks and models (e.g., ViTs, GNNs, and vision-language models) in future work.\n\n---\n\n**W6: Communication and System Overhead**\n\nThank you for your comment.\nTo clarify, our method does not add extra communication cost—in fact, it reduces it compared to FedAvg. This is because FedEL only uploads selected important tensors rather than the full model. As shown in the table, FedEL results in: 1. Lower communication time per round than FedAvg. 2. Communication taking up only a small part of the total training time.\n\nWe also measured the runtime of FedEL's system modules (e.g., sliding window, tensor importance update, and selection). The added overhead is minimal and has negligible effect on overall training time. We will include these results in the final paper.\n\n| Method | Communication   | Tensor processing | Average round time |\n|--------|-----------------|-------------------|--------------------|\n| FedAvg | 2.45 min (3.2%) | N/A               | 75.43 min          |\n| FedEL  | 1.09 min (2.7%) | 0.97 min (2.4%)   | 40.34 min          |\n\n---\n\n**W7: Overly Concise Conclusion**\n\nThank you for the helpful feedback. Due to page limits, we kept the Conclusion in the main paper brief to focus on summarizing key contributions, methods, and results. A more detailed discussion is included in Section C of the Supplementary Material, where we further elaborate on the limitation and potential future directions."}, {"review_id": "jnKG1tJDkD", "reviewer": "Reviewer_LPLK", "summary": "This paper introduces FedEL, a framework for efficient federated training under system and data heterogeneity. Specifically, FedEL proposes a sliding window-based training scheme, within which the most important tensors are updated, subject to the device's computational budget. This is shown, across three models and tasks, to improve convergence rates compared to other heterogeneous federated solutions, while maintaining competitive accuracy to the baseline FedAvg baseline.", "questions": "* I would like to see how the proposed technique behaves under partial client participation and\n* I would be interested to see how the sliding window mechanism and important tensor selection method would deal with shared-weight layers, including grouped head attention or embedding layers.\n* One of the main motivations for FL training is data privacy. Therefore, it would be conducive to the reader to see how the technique behaves under Differential Privacy and Secure Aggregation schemes.\n* What are the energy and cooling modes used for the Jetson devices? Utilising this degree of freedom could create more classes of devices and thus more heterogeneous settings.\n* How does the technique compare to [a,b]?\n\n[a] Lee, R., Fernandez-Marques, J., Hu, S. X., Li, D., Laskaridis, S., Dudziak, Ł., ... & Lane, N. D. (2024). Recurrent early exits for federated learning with heterogeneous clients. arXiv preprint arXiv:2405.14791.  \n[b] Ilhan, F., Su, G., & Liu, L. (2023). Scalefl: Resource-adaptive federated learning with heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 24532-24541).", "limitations": "* The paper assumes a static performance of a specific device, which may be true for clients that run nothing concurrently and for burst workloads. However, this might not be the case under parallel workloads that share the same resources or throttling situations (DVFS).\n* It might have been preferable to use a wired connection with configurable bandwidth between the parameter server and the jetson devices, rather than shared WiFi, as currently there might be cross talk over the same communication channel.\n* It is unclear whether the authors have run their experiments more that once for robustness, since no error bars are shown.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* While the topic of system heterogeneity has been researched extensively, FedEL provides a relatively novel approach in selecting which parts of the model to train under computational constraints.\n* The authors have gone the extra mile and have run the federation on actual devices, manifesting the difference in computational dynamics of clients.\n* Additionally, I do welcome the quantification of memory and energy consumption in the evaluation of the FedEL method compared to the baselines.\n\n### Weaknesses\n\n* While the approach is conceptually interesting and the findings are empirically shown to work at small scale, the models and datasets used to do so are quite old and small scale.\n* The paper does not quantify the overhead of profiling or of the important tensor selection process. Moreover, the device dynamics may be fluctuating, especially under simultaneous workloads.\n* Wrt the setup, it is unclear from the paper whether the system has been evaluated under partial participation of clients (due to availability, rather than performance). Moreover, the papers assumes a uniform network channel, which might not be always the case.", "was_revised": true, "final_justification": "The authors run multiple additional experiments and hyperparameter setups to illustrate the strengths of their solution. I am happy to increase my score.", "rebuttal": "**W1: Model and Datasets**\n\nWe appreciate the reviewer’s comment regarding the scale and recency of the models and datasets used in our evaluation.\nOur choice of models (e.g., VGG16, ResNet50, Albert) and datasets (image classification, speech recognition, lightweight NLP) was guided by two primary considerations: (i) alignment with prior work in heterogeneous federated learning (e.g., DepthFL, HeteroFL, and FIARSE), and (ii) practical deployment on real edge devices (Jetson Orin, Xavier), where larger models such as full-scale Transformers or multimodal architectures are currently infeasible due to hardware constraints. This setup enables fair comparisons while maintaining realism in resource-constrained federated environments.\n\nThat said, we agree that demonstrating FedEL’s scalability to larger, more recent models is an important next step. We are currently extending it to support larger vision and language models, MobileBERT on the GLUE benchmark and ViT-tiny on the NICO++ dataset, running on more capable hardware platforms (Jetson AGX). Results below indicate that FedEL continues to provide consistent performance benefits over FedAvg, even as model complexity and task diversity increase. These findings validate the generalizability of our selective training framework beyond small-scale settings.\n\n| Method | MobileBERT   |   ViT-Tiny | \n|------------|--------------------|--------------|\n|              |     Acc. / Time  | Acc. / Time |\n| FedAvg | 82.4\\% / 410.7h | 78.6\\% / 472.1h  |\n|FedEL    | 83.9\\% / 167.5h | 80.1\\% / 188.2h    |\n\n---\n\n**W2: Overhead of Profiling and Tensor Selection**\n\nWe appreciate the reviewer’s attention to overhead analysis.\nAs shown in the first table, profiling during a single training epoch adds less than 1% overhead across all tested models. This step is done **offline**, before FL training begins, and its results are reused, so it introduces no runtime overhead during training.\n\n| NN model | Vgg   | ResNet | Albert |\n|----------|-------|--------|--------|\n| Overhead | 0.98% | 0.45%  | 0.61%  |\n\nFor the tensor selection step during each FL round, we measured a small runtime cost—only 2.4% of the total round time, as shown below. Communication also accounts for just 2.7%. These results confirm that FedEL has minimal overhead and is efficient enough for real-world use.\n\n|      | Communication   | Tensor processing | Average round time |\n|------|-----------------|-------------------|--------------------|\n| Time | 1.09 min (2.7%) | 0.97 min (2.4%)   | 40.34 min          |\n\n---\n\n**W3: Adapting to Device Dynamics**\n\nWe thank the reviewer for the valuable feedback. \nWhen runtime dynamics occur, profiling can be updated online with negligible cost (less than 1% overhead per epoch). This allows adaptive updates without impacting training efficiency. Dynamic re-profiling is a potential direction for future work.\n\n---\n\n**W4: Partial Client Participation**\n\nWe thank the reviewer for pointing out this clarification need. \nOur experiments do adopt partial client participation, which reflects realistic FL settings. Specifically, in each communication round, a random subset of clients is selected to participate, consistent with common practice in prior works such as FedAvg, HeteroFL, and FIARSE.\n\nWe will revise the manuscript to explicitly clarify this in the experimental setup. Additionally, FedEL's design—especially its local tensor selection and sliding window mechanism—naturally supports partial participation, as it makes no assumptions about client availability or full participation. Our reported results already reflect performance under this dynamic client selection.\n\n---\n\n**W5: Uniform Network Channel**\n\nWe thank the reviewer for raising this important point. \nWe isolate computational heterogeneity to evaluate our method fairly, following practices in HeteroFL and DepthFL. However, our approach naturally extends to communication heterogeneity.\n\nDuring offline profiling, we estimate tensor sizes and include communication time in the selection objective:\n\n$\\max_{\\boldsymbol{A}} \\boldsymbol{A} \\cdot \\boldsymbol{I} ~~s.t. T_{fw} + T_{bw}(\\boldsymbol{A}) + T_{tx}(\\boldsymbol{A}) \\le T_{th}$\n\nHere, $T_{tx}(\\boldsymbol{A})$ represents the estimated communication time of selected tensors. We conducted additional experiments simulating four bandwidth tiers (10, 20, 30, 45 Mbps) across 100 clients. We compared: (1) FedAvg (baseline), (2) FedEL (original), and (3) FedEL+Tx (FedEL with communication-aware tensor selection).\n\nResults show FedEL+Tx achieves similar or better accuracy while further reducing total training time, confirming the benefit of modeling communication heterogeneity:\n\n| Method   | Image Classif.  | Speech Recog.   | NLP            |\n|----------|-----------------|-----------------|----------------|\n|          | Acc. / Time     | Acc. / Time     | Acc. / Time    |\n| FedAvg   | 33.76% / 571.5h | 58.04% / 710.2h | 77.48 / 547.1h |\n| FedEL    | 34.96% / 157.5h | 58.26% / 184.2h | 77.23 / 175.0h |\n| FedEL+Tx | 34.98% / 156.8h | 58.24% / 183.3h | 77.22 / 174.5h |\n\n---\n\n**W6: Handling Shared-Weight Layers**\n\nWe thank the reviewer for this insightful question.\nAs detailed in Section B of our Supplementary Material, our implementation for Albert follows the block design from [39], placing early exits after each encoder-classifier stack. This supports variable-depth execution and is fully compatible with our sliding window mechanism.\n\nRegarding shared-weight layers, such as embeddings and grouped attention heads, we handle them as follows:\n\n(1) Shared Embeddings: We profile the shared tensor as a single unit. Backward time is traced from the output back to the shared embedding, reflecting its impact on both input and output. The weight update time is counted only once since the tensor is updated once per iteration.\n\n(2) Grouped Attention Heads: For attention layers with multiple trainable projections (query, key, value, output), we treat each as a separate tensor and profile their backward time individually. If some projections are shared, we track their computation cost jointly during backpropagation.\n\n---\n\n**W7: Compatibility with DP and Secure Aggregation**\n\nThank you for your comment.\nOur method is fully compatible with privacy-preserving techniques like Differential Privacy (DP) and Secure Aggregation (SA), as FedEL operates at the system level and does not modify or interfere with model encryption or privatization.\n\nTo evaluate this, we extended our experiments by integrating standard DP and SA into FedEL. The table below shows that both FedEL+DP and FedEL+SA achieve similar accuracy to vanilla FedEL. As expected, they incur some extra training time due to the added noise (DP) or encryption (SA).\n\n| Method   | Image Classif.  | Speech Recog.   | NLP            |\n|----------|-----------------|-----------------|----------------|\n|          | Acc. / Time     | Acc. / Time     | Acc. / Time    |\n| FedEL    | 34.96% / 157.5h | 58.26% / 184.2h | 77.23 / 175.0h |\n| FedEL+DP | 34.80% / 172.4h | 58.10% / 201.3h | 77.25 / 192.8h |\n| FedEL+SA | 34.95% / 170.2h | 58.24% / 197.5h | 77.24 / 188.6h |\n\n---\n\n**W8: Jetson Power Modes**\n\nWe thank the reviewer for this insightful suggestion. \nIn our experiments, Jetson devices ran in MAXN mode to ensure consistency. Inspired by reviewer feedback, we reran experiments under varied power settings (10W, 15W, MAXN). We measured training times under each mode and used this variability in the FedEL scheduling process. As shown in the table below, FedEL still outperforms FedAvg in both accuracy and training time. These results confirm that FedEL remains effective even in more diverse and realistic hardware environments.\n\n| Method | Acc.   | Time   |\n|--------|--------|--------|\n| FedAvg | 58.21% | 550.2h |\n| FedEL  | 59.35% | 174.3h |\n\n---\n\n**W9: Related Works**\n\nWe appreciate the reviewer’s suggestions and discuss two related works below. We will include both in the final version.\n\n[a] Recurrent Early Exits (ReeFL):\nReeFL enables early exits in LLMs during FL. Our method FedEL can work with ReeFL by applying tensor selection inside each early-exit block. We combined both by inserting exits into transformer layers and applying FedEL within each. As shown below, FedEL+ReeFL reduces training time while maintaining accuracy, confirming that the two approaches are complementary.\n\n[b] ScaleFL:\nScaleFL adapts model width/depth based on device profiles, but uses static selection via a meta-scheduler. In contrast, FedEL dynamically selects tensors in each round based on importance, allowing finer control and better adaptability.\n\nTo validate this, we compared FedEL and ScaleFL across three tasks. FedEL consistently achieves higher accuracy, as shown below.\n\n| Method      | Image Classif.  | Speech Recog.   | NLP            |\n|-------------|-----------------|-----------------|----------------|\n|             | Acc. / Time     | Acc. / Time     | Acc. / Time    |\n| ScaleFL     | 33.12% / 157.0h | 57.40% / 184.8h | 77.71 / 189.1h |\n| FedEL       | 34.96% / 157.5h | 58.26% / 184.2h | 77.23 / 175.0h |\n| FedEL+ReeFL | N/A / N/A       | N/A / N/A       | 77.20 / 159.4h |\n\n---\n\n**W10: Result Robustness**\n\nThank you for the comment. We did evaluate robustness and reported it in Section B.5 of the Supplementary Material. As shown in Figure 21 (box plot), each experiment was repeated five times. The results show that FedEL consistently achieves stable and higher accuracy than the baselines, with narrow confidence intervals. This confirms the reliability and robustness of our method across multiple runs."}, {"review_id": "VPZI13X4df", "reviewer": "Reviewer_RKje", "summary": "This paper addresses the issue of device heterogeneity in federated learning (FL). Specifically, heterogeneous devices often cause significant training delays, as straggler clients with limited resources prolong the aggregation process. To tackle this issue, the authors propose FedFL, an FL training framework that leverages a sliding window to locate the trainable parts of the model and dynamically select important tensors for training based on a given time budget.", "questions": "Please see the weakness section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The proposed method appears to achieve better convergence performance in terms of both accuracy and training time compared to previous works.\n\n2) The use of a sliding window to select tensors for local training is an interesting idea. The method also incorporates two strategies to ensure that most model blocks are able to participate in training throughout the entire FL process.\n\nWeaknesses:\n1) The discussion of drawbacks in existing work is unconvincing, especially regarding Partial Training. The authors overlook the fact that many existing studies (e.g., [1–4]) also leverage selective neuron training, rather than relying solely on depth/width scaling.\n\n[1] Every parameter matters: Ensuring the convergence of federated learning with dynamic heterogeneous models reduction. NeurIPS 2023.\n\n[2] Model pruning enables efficient federated learning on edge devices. IEEE TNNLS, 2022.\n\n[3] Helios: Heterogeneity-aware federated learning with dynamically balanced collaboration. DAC 2021.\n\n[4] Sparsified Random Partial Model Update for Personalized Federated Learning. IEEE TMC, 2024.\n\n2) Why is the model partitioned layer-wise? Does this mean some layers may be entirely skipped? Additionally, what is the partitioning strategy for transformer-based models? What is the granularity of model partitioning?\n\n3) The offline tensor time profiling model appears overly simplistic. Due to differences in computational workload and the size of intermediate data, each block may have significantly different training times. How is this diversity accounted for and applied across various models during the FL process?\n\n4) In practical FL scenarios, clients may dynamically join and leave the process. How does the proposed profiling and online window strategy adapt to this dynamic participation? Moreover, if a straggler has significantly lower computational capability, the selected tensors may be very few and under-trained—could their uploaded updates degrade the performance of the global model?\n\n5) The authors should share more details about how to realize that only the selected blocks are involved in local FL training.  Are any gating mechanisms or other techniques applied to enforce this selective training?\n\n6) Given a specific time budget, how is the combination of blocks selected for training? Is there an optimization to decide which blocks will participate?\n\n7) It seems that adjusting tensor importance introduces extra overhead at the beginning of each FL training round. Could you provide some quantitative analysis of this overhead to demonstrate its impact on overall efficiency?", "was_revised": true, "final_justification": "Based on the author's replies during the rebuttal period, I think they addressed most of my questions. I will keep my scores.", "rebuttal": "**W1: More Related Works**\n\nWe thank the reviewer for highlighting this important point. We have expanded our comparison with the cited works and summarize the key distinctions in the table below:\n\n| Method      | Extra Communication  | Uses Importance Measure   | Heavy Optimization  |\n|-------------|-----------------|-----------------|----------------|\n|[1] NeurIPS 2023 | Yes      |  No     | Yes  |\n|[2] TNNLS 2022   | Yes      | Yes     | Yes  |\n|[3] DAC 2021     | Yes      | Yes     | Yes  |\n|[4] TMC 2024     |Not partial training: trains full model, uploads partial model only |\n|FedEL (Ours)    | No       | Yes     | No   |\n\n[1] NeurIPS 2023 offers convergence guarantees but applies global pruning without local data or neuron importance awareness, limiting its adaptability at the client level.\n\n[2] TNNLS 2022 supports distributed pruning but incurs significant communication overhead due to the need to transmit importance scores and pruning decisions each round.\n\n[3] DAC 2021 (Helios) enables dynamic model adaptation but requires frequent communication and additional optimization overhead, especially for straggler mitigation.\n\n[4] TMC 2024 does not perform partial training; instead, it trains the full model and uploads only a subset of parameters, which is orthogonal to our focus on efficient local training.\n\nIn contrast, FedEL performs lightweight, local tensor selection using an importance-based mechanism, incurs no extra communication, supports fine-grained dynamic adaptation, and avoids heavy optimization—offering a practical and scalable solution to device heterogeneity.\n\nWe will add these to the related works in the final paper.\n\n---\n\n**W2: Model Partition**\n\nWe thank the reviewer for the insightful question. Our approach incorporates two key modules: (1) sliding window and (2) important tensor selection, which together enable multi-granularity model partitioning.\n\n1. Sliding Window (Layer-Level Slicing):\nThis is a coarse-grained strategy that slices the model at specific layers. As detailed in Section 4.1, only semantically meaningful layers (e.g., block or stage boundaries) are eligible slice points—not all layers. This ensures the model remains structurally sound and stable during training. The main goals of this module are to (i) enable compatibility with early-exit classifiers and (ii) limit the search space for finer-grained selection.\n\n2. Important Tensor Selection (Width-Level Slicing):\nWithin each layer or block selected by the sliding window, we further apply fine-grained slicing by selecting only important tensors for training. This allows each client to update only a subset of parameters, making the training process more efficient and adaptive to device capabilities.\n\n3. Transformer Models: \nAs noted in Section B of the supplementary material, we adopt the block design from the Albert model [39], placing early-exit classifiers after each encoder–classifier block (see [39], Figure 1). The sliding window works at the block level, enabling depth-wise flexibility, while tensor selection operates within each block to control the width of updates.\n\n---\n\n**W3: Offline Tensor Time Profiling**\n\nWe thank the reviewer for the comment. \nAs shown in the table below, our empirical results across diverse neural network architectures indicate that profiling performed over a single training epoch incurs less than 1\\% additional overhead, which aligns with observations in ElasticTrainer. This low cost enables us to periodically re-profile the system if needed—e.g., when data size or background workloads change—thus addressing concerns about time variance at the block level.\n\n\n| NN model | Vgg   | ResNet | Albert |\n|----------|-------|--------|--------|\n| Overhead | 0.98% | 0.45%  | 0.61%  |\n\n---\n\n**W4: Dynamic Participation and Handling Stragglers**\n\nWe thank the reviewer for highlighting this important point.\n\nFirst, our experiments use partial client participation, where a random subset of clients is selected in each round, based on availability—not device performance. This reflects standard FL practice and aligns with methods like DepthFL, HeteroFL, and FIARSE. FedEL runs on this selected subset each round. We will revise the main paper to make this clearer.\n\nSecond, we acknowledge that in cases of extreme heterogeneity, some straggler clients may contribute fewer updates, possibly under-training some tensors (as discussed in Section C of the supplementary material). To address this, one promising direction is client clustering—grouping clients by similar compute capacities. Each group can run FedEL independently, and asynchronous updates between groups can help balance training and reduce the impact of extreme stragglers.\n\n---\n\n**W5: Implementation of Block-Based Training**\n\nWe appreciate the reviewer’s question.\nTo enable training only on selected blocks, we use a simple and efficient layer-level masking approach based on the model structure.\n\nFor each local training round, we create a binary mask over model layers: Layers in the selected blocks get a mask value of 1.\nAll other layers get a mask value of 0.\n\nIn PyTorch, we apply this by setting each layer’s **requires_grad** attribute:\nIf mask = 1: requires_grad = True, so the layer is trainable.\nIf mask = 0: requires_grad = False, so the layer is frozen.\n\nThis approach avoids any changes to the model architecture or complex control logic. It is lightweight, flexible, and works well with our sliding window and tensor importance selection modules.\n\n---\n\n**W6: How to Select Blocks?**\n\nWe thank the reviewer for this insightful question.\nCurrently, the block selection in our method is not formulated as an explicit optimization problem. Instead, we use a practical heuristic guided by two components: offline tensor time profiling and a sliding window mechanism. Specifically, the model is first partitioned into blocks based on its computational structure, as determined through offline profiling. During each training round, a sliding window is used to define the active set of blocks:\nA new block is added to the window, and previous blocks may be removed based on their historical contribution to training effectiveness. This defines a dynamic candidate region, within which important tensors are then selected for training using our fine-grained selection module.\n\nWhile this heuristic works well in practice (as supported by our experimental results), we agree that formulating block selection as an optimization problem is a compelling direction for future research. One potential formulation could be derived from the term $O_1$ in our convergence analysis, enabling principled selection of blocks under resource or latency constraints.\n\n---\n\n**W7: Overhead of Adjusting Tensor Importance**\n\nWe appreciate the reviewer’s concern about potential overhead.\nAs shown in the table below, the total time spent on adjusting tensor importance—including the sliding window update, importance scoring and adjustment, and tensor selection—is minimal:\n\n| Tensor processing | Average round time |\n|--------|-----------------|\n 0.97 min (2.4%)   | 40.34 min          |\n\nThis represents only 2.4% of the total round time, which we consider negligible in the overall context of federated training. The lightweight design of our mechanism ensures scalability and practicality without compromising system efficiency."}, {"review_id": "0NG6V3Oq8F", "reviewer": "Reviewer_2Sji", "summary": "The paper proposes FedEL (Federated Elastic Learning), a training framework to mitigate the scenario of straggling clients due to participating devices having heterogeneous computation capabilities. FedEL builds upon a previously published method, Elastic Trainer and incorporates a couple of key adjustments, Sliding window training and Tensor importance adjustment to account for realistic scenarios as encountered in Federated Learning setups. Experiments show that FedEL outperforms other methods for different ML tasks", "questions": "1) Can the authors provide more information on The method relies on how the method will be able to handle if new devices join the setup in an online manner.\n\n2) Can the authors quantify what additional overheads that come with adding auxiliary heads\n\n3) Can the method be adapted to handle transformer based architectures\n\n4) Is the method able to handle scenarios where not all the devices are able to participate? In heterogeneous setups it is a possibility that some devices are unable to participate for many rounds", "limitations": "Statistical significance of the reported results have not been provided.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "***Strength***\n\n(1) The method is evaluated on a number of tasks and data modalities \n\n(2) Experiments have been conducted with both edge devices and simulations\n\n(3) Convergence analysis and mathematical guarantees have been provided for the proposed method\n\n***Weakness***\n\n(1) The method relies on profiling devices offline. It is not clear how the method would perform if new devices join the setup in an online manner.\n\n(2) In Figure 8 and Figure 9, the statistical significance of these results has not been captured. Thus it is challenging to assess the soundness of the method when assessing memory and power consumption\n\n(3) Since the sliding window requires inserting auxiliary heads, it’s not clear to me how many parameters are required here and how it's structure needs to be adjusted to account for tensors of different feature lengths\n\n(4) The neural network architecture for which the method has been evaluated are very similar. It will be interesting to see how the method performs on a commonly used transformer based architecture", "was_revised": true, "final_justification": "I have gone through the response the authors provided and feel satisfied to raise the score.", "rebuttal": "We appreciate the reviewer’s time and effort in providing feedback, though it appears that the comments may have been intended for a different submission. We welcome any relevant insights they can share regarding our paper."}]

Author rebuttal:

Meta-review: Thank you for your submission to the NeurIPS 2025. This paper proposes FedEL, a federated elastic learning framework that enhances training efficiency while maintaining model accuracy. The window-based training approach stands as a novel training paradigm to manage the stragglers. Although everyone has pointed out some merits, they also have raised some concerns, such as experimental setup, unclear comparison with existing partial training works etc. During the rebuttal period, the authors’ feedback has helped on clarifying the reviewers’ concerns. Thus, the average score 4.25 is both above the average levels. Based on the current reviews and closed-door reviewer discussions, every reviewer seems to be fine with an acceptance.
Decision: Accept (poster)