LLM at Network Edge: A Layer-wise Efficient Federated Fine-tuning Approach
Authors: Jinglong Shen
Nan Cheng
Wenchao Xu
Haozhao Wang
Yifan Guo
Jiajie Xu
Venue: NeurIPS 2025 poster ()
Keywords: federated learning
large language model
fine-tuning
Primary area: deep_learning

Abstract: Fine-tuning large language models (LLMs) poses significant computational burdens, especially in federated learning (FL) settings. We introduce Layer-wise Efficient Federated Fine-tuning (LEFF), a novel method designed to enhance the efficiency of FL fine-tuning while preserving model performance and minimizing client-side computational overhead. LEFF strategically selects layers for fine-tuning based on client computational capacity, thereby mitigating the straggler effect prevalent in heterogeneous environments. Furthermore, LEFF incorporates an importance-driven layer sampling mechanism, prioritizing layers with greater influence on model performance. Theoretical analysis demonstrates that LEFF achieves a convergence rate of $\mathcal{O}(1/\sqrt{T})$. Extensive experiments on diverse datasets demonstrate that LEFF attains superior computational efficiency and model performance compared to existing federated fine-tuning methods, particularly under heterogeneous conditions.

LLM at Network Edge: A Layer-wise Efficient Federated Fine-tuning Approach
Jinglong Shen', Nan Cheng', Wenchao Xu', Haozhao Wang', Yifan Guo', Jiajie Xu!
'chooo of Telecommuuications Engineering, Xidian University *Deaarmentt of Computing, The Hong Kong Polytechnic University School of Computer Science and Technology, Huazhong University of Science and Technology jlshen@stu. xidian edu. cn, dr nan. chan.cee@@eee org, wenchao. xu@polyu. edu.. hk, hz_ wz_wan@@hust. edu. cn, {guoyifan, xujiajie]@stu. xidian, edu. cn
Abstract
Fine-tuning large language models (LLMs) poses significant computational bur- dens, especially federated learning (FL) settings. We introduce Layer-wise Efficient Federated Fine-tuning (LEFF), novel method designed to enhance the efficiency of FL fine-tuning while preserving model performance and minimizing client-side computational overhead. LEFF strategically selects layers for fine- tuning based on client computational capacity, thereby mitigating the straggler effect prevalent in heterogeneous environments. Furthermore, LEFF incorpo- rates an importance-driver layer sampling mechanism, prioritizing layers with greater influence model performance. Theoretical analysis demonstrates that LEFF achieves convergence rate of O(1/VT). Extensive experiments diverse datasets demonstrate that LEFF attains superior computational efficiency and model performance compared to existing federated fine-tuning methods, particularly under heterogeneous conditions.
Introduction
Large Language Models (LLMs)) have exhibited remarkable capabilities various downstream tasks, including text generation Li et al. (2024), language translation Ranathunga et al. (2023). and question answering Yu et al. (2024). Their success is primarily attributed to their increasing model scale Baari et al. (2024), with contemporary models scaling from billions of parameters (e.g.. GPT-2) to hundreds of billions (e.g., GPT-4). While fine-tuning pre-trained L.LMs on task-specific data is the de facto approach for adaptation, the privacy-sensitive nature of user-generated data poses challenges to centralized collection. Federated learning (FL) offers a solution by enabling distributed model training without requiring data centralization McMahan et al. (2017); Huang al. (2024). In FL, models are fine-tuned directly on user devices, thereby preserving data privacy. However, the computational demands of fine-tuning such large models often surpass the capabilities of typical consumer devices. For instance, while typical consumer GPUs are often limited to 24GB graphics memory, fine-tuning GPT-3 model even with 16-bit precision requires approximately 326GB of memory.
Addressing the significant computational demands of LLMs, researchers have explored various mitigation strategies Han al. (2024). parameter-effcient fineituning (PEFT) methods, including Adapter Houlsby al. (2019), LoRA Hu et al. 2022), and prompt tuning Lester et al. (202) have
'Correspooding Author. II this paper, we focus on transformer-hased language models. For simplicity, we refer to transformer blocks as layers.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

emerged as promising approaches. These methods fine-tune a small number of additional or selected parameters while keeping the bulk of the pre-trained model frozen, thereby substantially reducing computational overhead. However, applying PEFT methods within FL. settings encounters several critical challenges. First, the con-independent and identically distributed (non-IIID) nature of client data is a primary concern Qi et al. (202)), often leading to substantial performance degradation Huang (ct al. (2023 2022). This degradation can be particularly pronounced in federated PEFT compared to full fine-tuning paradigms Babakniya al. (2023). Additionally, computational heterogeneity across participating devices impairs resource utilization efficiency due to the straggler effect, where overall training progress is constrained by the slowest clients Shhnn et al. (2024). While recent approaches, such as FedDSE, explore mub-model extraction based on neuron activation patterns for cosource-constranne FL Wang et al.] (2024a), maintaining model performance under non-IID data distributions remains a significant hurdle.
To address these challenges, we propose Layer-wise Efficient Federated Fine-tuning (LEFF), method that aims to preserve the efficacy of full full-parameter fine-tuning while significantly reducing computational overhead. LEFF enables clients to selectively fine-tune specific layers according to their computational capacity, while other layers remain frozen during local training. This flexibility allows resource-constrainee clients to fine-tune a reduced set of layers, thereby mitigating the straggler effect. Following local training, clients transmit only the updated parameters of their selected layers to the server. The server then performs layer-wiss aggregation to reconstruct the global model for the subsequent round. To further optimize the fine-tuning process, we introduce an importance based layer sampling strategy. This strategy dynamically adjusts the selection probability of each layer based on its contribution to overall fine-tuning performance. This approach ensures that more impactful layers are prioritized for updates. Our key contributions
Novel Architecture:
We introduce LEFF, federated fine-tuning framework designed to preserve full- uull-parameter fine-tuning efficacy while substantially mitigating client-side computational overhead.
Robustness to Data Heterogeneity: LEFF leverages full-parameter fine-tuning, enabling superior adaptation to non-lID data distributions across clients, outperforming conventional PEFT methods like LoRA.
Computational Heterogeneity Mitigation: Our framework accommodates computational heterogeneity by allowing clients to dynamically adjust their local training workload accord- ing to their available resources, thereby mitigating the straggler problem.
Adaptive Layer Prioritization: We develop an adaptive layer selection mechanism em- ploying an importance- sampling algorithm. This prioritizes updates for performance- critical layers, enhancing both training efficiency and model effectiveness.
Theoretical Guarantees: We provide rigorous convergence analysis, demonstrating that LEFF achieves a convergence rate of (1/vTT).
Empirical Validation: Extensive experiments demonstrate LEFF's capability to signifi- cantly reduce client-side computational burden while achieving superior fine-tuning perfor- mance under heterogeneous data and system conditions, outperforming existing state-of-the- art methods.
The remainder of this paper is organized as follows. Section 2 reviews the relevant literature on FL. and PEFT. Section Sotiioonndeaan details our proposed LEFF framework and its key components. Section 4 provides a convergence analysis. fection[]presentt comprehensive experimental results and analyses. Finally, Section 6 summarizes our findings and discusses future work.
Related Work
PEFT methods are essential for adapting LLMs by alleviating the prohibitive computational costs of full fine-tuning, Techniques have evolved from updating parameter subsets Zaken et al. (2021) and injecting trainable modules Houulsy et al. (2019) to latency-free approaches like LoRA Hu et al. (2022): ; Fu et al. (2022), which merges learned low-rank matrices into the original model. Further refinements, such as dynamically updating important layers as demonstrated by LISA Pan et al. (2024).
optimize efficiency by targeting model capacity more effectively. While established in centralized
2

---

settings, PEFT's application to FL is an emerging research area Zhuang et al. (2023); Yu et al. (2023);
Woisetschläger al.] (2024). Various PEFT strategies. including LoRA- based instruction tuning
Zhang et al. (2024), federated prompt optimization Guo et al. (2023 2024), and hackpropagation-free free
client fine-tuning Xu et al. (2023). have shown viability in Che et al. (2022 2023 ) [Sun et al.
(2022) Zhang et al. (2023) Fang et al. (2024); Legate et al. (2023); Wang et al. (2024b). However,
critical challenge is data heterogeneity across clients, which analyses confirm significantly degrades
PEFT performance compared to full fine-tuning Babakniya et al. (2023); Bai al. (2024); Cho et al.
(2023 a limitation even for specialized approaches like FedDAT in mu multimoda tasks Chen et al.
(2024) Concurrently, federated full-pan parameeterr tuning methods present their trade including
high client computation in gradient approximation Qin et al. (2024), unreduced peak memory load in comaate-oompesssion schemes Shu al. 2025), , and a lack of adaptabilityy in static cyclical updates Wang et al. 2024c). To address this performance degradation, we propose LEFF. Unlike prior work, LEFF facilitates decentralized, layer-wise adaptation of backbone models by enabling client-specific full-parameter updates to a selected subset of layers, aiming to effectively mitigate the adverse impacts of data heterogeneity while preserving computational efficiency.
Attention Layers:
Compressed Layer:
①D Layer Sampling
Model Compression
@ Model Aggregation
Pretmined Model
Pretrained Model
Pobobility
Sampling
0.1
02
(4.
16 by Aac
TI-HH
Layer Indeso:
(. (4.5,9)
Chent Model
Clinn Model
Cliem Model
DO -O----O---------
Stoad Uppltc
b Local Training
Local Training
X Local Training
Client
Client
Client
Figure 1: Overview of the LEFF framework. Each communication round comprises four key stages.
3 Methodology
3.1
System Architecture
FL clients in edge computing (e.g., mobile phones, personal computers) often possess limited computational resources. The increasing scale of LLMs intensifies this constraint, making their fine-tuning computationally burdensome. To mitigate this, introduce LEFF, a framework to reduce client-side training overhead while striving to maintain the benefits of full-parameter fine-tuning.
The LEFF framework (Figure I involves four steps per communication round. First, clients report their local fine-tuning capacity (number of layers L.). Based on these reports, the server calculates sampling probabilities and selects layer subset L, (of size L,) for each client C, to fine-tune. Second, to reduce client burden, the server compresses unselected layers L, via knowledge distillation, creating a customized local model. This model comprises client-specific fine-tunable layers (from Lj) and frozen compressed layers (derived from L, ). Third, clients fine-tune only their assigned layers; other compressed layers remain frozen. Finally, clients upload their fine-tuned layer parameters, and the server performs layer-wise aggregation to update the global model.
We consider standard FL setup: central server aggregates parameters from N clients (C,, 1...., N) performing local training on their datasets D, (size D. = |Di).. The global model O has L layers. For each client Ci, the server samples block of L; consecutive layers for fine-tuning,
3

---

denoted L;:
L: (Isturt: ..., l,ndd: s.t. lnnd start Ii--
(I)
The unselected layers 4; (4 1 < 1 l E Li) compressed by the server into 6 The resulting client-specific model 6; = (0 6 supports forward propagation. Client C, receives ; fine-tunes only e on D; to obtain updated parameters O (while 6 remain frozen), and uploads O The server aggregates these layer-specific parameters from clients update θg.
3.2 Layer Sampling
Sampling Probubility
At the start of each communication round, the server selects the layers for each client to fine-tune based on L, Considering that layers contribute differently to fine- tuning performance. we propose an importance-basse layer sampling method. This approach prioritizes lay- ers considered more crucial for model fine-tuning, thereby optimizing the fine-tuning process by concentrating com- putational resources on the most impactful model parts.
Softmax
Importance
Sum
Layer
Importanee
I
As discussed in Molchanov al. (2019), the importance of a neural parameter can be quantified by the change in the loss value when the parameter is introduced or removed:
Figure 2: Sampling probability for each block. Model layers are organized into blocks, and clients sample only one block per communication round for fine- tuning.
Im =
-
le.. lam=-)))²
(2)
where FF denotes the loss function, D represents the dataset, and On is the m-th parameter of the model O expansion form:
We further adopt its first-order Taylor
Ih = (gm
(3)
where Im the gradient of the m-th parameter. This approximation allows for estimating each param- eter's importance with only single forward and backward pass, significantly reducing computational complexity.
As illustrated in Figure Based on the importance scores of individual parameters, the importance score for each layer can be derived by summing the importance scores all parameters within that layer:
Ie = (0)
Σ (O)
Σ (gm.m )2
(4)
mEE
mee!
We define consecutive L; layers as a block, denoted as @ = ...,... Ah--i-1), with its impor- tance given by:
A Σ
Is:
I
(5)
1=k
To determine the sampling probability, these importance scores are normalized using the Softmax function. Thus, the sampling probability for each block during layer sampling for client C, is expressed as:
= Softmax (!Ig+ k ==
L L; 1})
(6)
The server utilizes P: to sample a block for client C, determining the set of layers C, that the client will fine-tune in the current communication round.
3.3 Model Compression

---

As illustrated in Figure 3 in each communication round, prior to client-side fine-tuning, the server compresses the layers not selected for client fine-tuning, denoted by 4; reduce computational overhead. Specifically, the server employs proxy dataset D proxy. sourced from publicly available data, for this knowledge distillation process. The
Mode!
MISI!
student model 65; maintains multi-layer attention ar- chitecture identical to that of the corresponding layers in the pre-trained global model. The number of layers in the student model is determined by compression rate If
KL Divergooee
Stace
the teacher model segment egi comprises Og layers,
Figure 3: Distillation process in LEFF The distillation loss comprises two ponents: the mean square loss of the hid- den state and the KL divergence of the attention matrix.
the student model's layer count 6 0 |r]]
Once the student model's architecture is defined, the server initiates compression via knowledge distillation. This distillation leverages two objectives: a Mean Squared Error (MSE) loss between the hidden states of the teacher and student models, and a Bullback-Leibler (KL) divergence between their attention matrices. Crucially, to prevent the student model from learning iesk-irrelevant information from Prrxxy, only the intermediate representations (hidden states and attention matrices) are used for distillation, not the task-specific output labels from D Prxxy. Specifically, for an input hidden state
sequence Hi, the teacher model segment o produces outputs (Ht, At) o (H,), while the student model G produces (Ha, A.) Ggi (H.). The distillation loss Edisill is then formulated as:
Editill (1 a) Enidden + +a aaaattttonnmmaaii matrix (1 MSE Ha) aAL, As),
(7)
where is a hyperparameter balaneing the two loss components. Following compression, the server integrates the student model G with the layers selected for fine-tuning, O to form the client-specific model θ, = ( 6 }. This model O is then dispatched to the i-th client.
3.4 Local Training
Upon receiving the model O from the server, client C, fine-tunes the selected layers Of while the
unselected layers , remain frozen to maintain the correct training context. This strategy enables clients to update only subset of parameters, with the compressed unselected parameters requiring minimal resources. As result, LEFF significantly reduces computational overhead on the client side, and adapts to the client's computational capabilities, thereby enhancing efficiency. After completing local training, client C, sends the updated model parameters O back the server for aggregation.
3.5 Model Aggregation
After receiving model parameters from all clients, the server aggregates them to construct compre- hensive global model. Since each client fine-tunes different components the model, we utilize a layer-wise aggregation approach, performing weighted averaging for each layer individually. Specif- ically, for a global model O, = {0) g. e O with layers, let S₁ represent the set of clients that fine-tuned the 1-th layer. The aggregated global model is expressed follows:
9, = 10 .
Dj EEPS ESS,, Dj iES,
Σ
1_ J
(8)
Once the server completes the aggregation and acquires the updated global model, the subsequent communication round commences. This process continues until the model converges or reaches predetermined number of iterations.
5

---

Convergence Analysis
4.1 Assumptions
Our convergence analysis of LEFF relies on the following assumptions. Assumptions 3 are standard in FL (Li et al. 2020); Assumption addresses complexities from LEFF's layer selection and model compression.
Assumption (L-Smoothness). Each client's local loss function F, R + R is L-smooth with respect the full model parameters θ. Thus, for constant L > 0, all θ O2 E Rd, and all clients i [
(.F((e)) V.. L| O2ll-
The global loss function F(O) N E(=)(()) is thus also L-smooth.
Assumption 2 (Bounded Variance). The variance of stochastic gradients computed by each client is bounded. Ler F.F; (+: ) be client i's stochastic gradient for sample & ~~ DD at parameters @ and (() ED (+;; {)] be the true local gradient. We assume constants of Z 0 such that for all θ E R.d and i [N]:
EE-D,II (I V.)(1))1 = o.
(10)
We define 0² maX o, This is standard in analyzing stochastic optimization, including FL. Assumption 3 (Bounded Heterogeneity). To account for data heterogeneity (non-IID data), assume the dissimilarity between true local and global gradients bounded. For a constant Ç² 2 and all @ Rd.
N Σ [TF 5
(11)
This is standard in FL analysis, especially for non-IID settings.
During local training in round t, client performs K steps of stochastic gradient descent, starting from 1,4,0 O At local step k E {O...., K - 1}. the client updates i,t,& using a stochastic gradient Gi.,k computed with respect to these active parameters:
Gi,cA- Vec, ii f E.,,K).
(12)
Here, =.(+)())) indicates the loss evaluated with active parameters O.; 11, conditioned on the fixed context 6.7 g.t using data sample &i,,, ~ D.
Let G. the average expected effective gradient computed by client over its K local steps in round t:
1 K-1
G.
Σ E
BEL (Gi,,k))
(13)
K
Let V Fec.F((Op.. denote the restriction of the full local gradient (F,(Og,)) to the subspace corre- sponding to the selected layers C..
Assumption 4 (Bounded Model Approximation Error). The expected squared norm difference between the average effective gradient G., and the corresponding part of the true local gradient Ve (0 (evaluated at the start-ofrround global model O,) is bounded. The expectation Rgyp,
laken over the randomness of layer selection for client (with selection distribution P.):
Ec,-p,
[[[G V.c.F.( < A
(14)
Furthermore, we assume the average of these client- error bounds across all clients is bounded, potentially depending on the round
A 5..
(15)
N
For simplicity certain analyses, one might further assume a uniform bound 2 such that A S 2 for all t.

---

4.2 Main Theorem
We present the key lemma leading our main convergence theorem.
Lemma 1 (Bound on Expected Squared Norm of Global Update), Let Assumptions 2 and hold. Let Ae be the set of clients selected in round t. The expected squared norm of the global model change in one communication round I. is bounded by:
E -100 -0,-0 0816] C3 (oR+ + +17(111)
(16)
where the expectation E[]] is taken over the randomness of client selection A,, layer selections and local stochastic gradients (Si,,,k Ass Aokkkk Here, C3 is constant that may depend on the total number of layers L, client sampling strategy, layer sampling probabilities p.. and aggregation weights.
Theorem 1 (Convergence of LEFF). Let Assumptions 7 2 3 and Ahold. If is chosen such that
n S Ca/(LC3 for sufficiently small C4 (where C3 is from Lemmal L is the smoothness constant and K is the number of local steps), then the LEFF algorithm satisfies:
from Assumption
C&(F(Og,u) g.0)
F*) F*) nKT Co
o²) K
BIITF(Og
A c²
+ Cconk + +22 (3)
T t=0
Vanishing torm
Evnor floor
(17)
where E[.] takes expectation over all randomness up to round I, T is the total number of commanica- tion rounds, 2 ΣT= - A7 is the average model approximation error over T rounds, F is lower bound for the global loss function F(O), and Cs, C9, Cio are positive constants depending problem parameters (L, o², (²), algorithm parameters (K), and constants from the assumptions and lemmas.
With an appropriately chosen decaying learning rate 7) = O(1/VT) (satisfying the condition in Theorem for sufficiently large T), the bound in Eq. (17) implies convergence rate where the average squared gradient norm diminishes towards an error floor. This leads to the following corollary regarding the minimum expected gradient norm:
Corollary 1 (Convergence Rate). Under the conditions of Theorem 7 if decaying learning rate 77 (LL/VT) is used, then LEFF achieves:
( +Cg A2 + 22
0²
min
E E||
(18)
OS
The convergence is guaranteed only up to error floor, primarily determined by the non-vanishing term Co + (2 + + (K) in Eq. (T8). This term quantifies the sources of residual error inherent in the federated optimization process with LEFF.
Trade-offs: The analysis reveals fundamental trade-off inherent LEFF. The framework enhances client-side efficiency (computation, memory, potentially comdeddmunication) through layer selection and context compression. However, this introduces the model approximation error 2 which contributes directly to the convergence eтror floor and may limit the achievable model accuracy. Consequently, the practical success of LEFF hinges on implementing layer selection and compression techniques that effectively minimize A while preserving the desired efficiency gains.
5 Evaluations
5.1
Experimental Setup
We conduct extensive experiments evaluate proposed method against established FL approaches. The evaluation framework incorporates baseline algorithms, pre-trained models, benchmark datasets, and detailed experimental configurations.
We benchmark our approach against FedAvg McMahan al. (017). which averages full model parameters, and three parameter-efffcient FL methods: FedBitFi Zaken al. (2021), updating only
7

---

bias terms: FedLoRA Hu et al. (202)), employing low-rank updates; and SLoRA Babakniya et al. (2023), utilizing server-side singular value decomposition (SVD) for enhanced stability and efficiency. Experiments are conducted using GPT-2 Medium Radford et al. (2019) for natural language generation (NLG) and DeBERTaV3 Base He
(202T) for natural language understanding (NIU). We evaluate on the GLUE benchmark Wang et al. (2019) (e.g., CoLA, MRPC, MNL.) for NLU tasks and the E2E NLG Challenge Novikova et al. (2017) for NLG from structured meaning representations.
Experiments were conducted for 20 communication rounds on the public datasets WebNLG Gardent al. ( and WNLI [ et (2019). To simulate heterogeneous (non-IID) data distributions, client data was partitioned using Diniclltt distribution with a concentration parameter (a) ranging from 0.05 50.0. Our FL simulations involved varying number of clients, each performing one local training epoch per communication round. During local training, clients fine-tuned their local models using the AdamW optimizer Cettieerrooshhhie Loshchilov, Hutter (2019) with learning rate of 1 x 10-5, All experiments were performed on a system with eight NVIDIA H100 GPUs. This experimental setup enables systematic evaluation of the proposed method's robustness against diverse data distributions and varying client participation levels.
5.2 Results
5.2.1
Comparison with Baselines
0.85
We evaluate LEFF on the GLUE benchmark across vary- ing data heterogeneity, controlled by the Dirichlet param-
0.80
eter E (0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0) (Figure 4 While FedAvg's aull-parameter fine-tuning ensures robust performance across all values, methods relying on par- tial parameter updates (e.g., FedBitFit FedLoRA, and SLoRA) exhibit limitations, particularly in highly hetero- geneous settings (low a). In contrast, LEFF employs layer-wise federated fine-tuning strategy, dynamically se- lecting layers based on client computational capabilities and importance sampling. This approach enables LEFF to achieve consistently strong performance, effectively addressing both data and system heterogeneity and outperforming baselines in critical heterogene- ity ranges. Notably, at high data heterogeneity (a = 0.05, Table []. FedAvg attains the highest
0.75
0.70
FedAvg
Average
FedBitFit
0.65
FedLoRA
0.60
SLORA
0.55
LEFF
10-1
10a
Dirichlet Alpha
Figure Average score of DeBERTaV3 on GLUE benchmark.
GLUE scores, with LEFF delivering closely comparable results and also demonstrating competitive performance on NLG tasks.
Table Test results of DeBERTaV3 (trained on GLUE tasks) and GPT2 (trained on E2E NLG task). The best result per task group is marked in underline, and the secondary is marked in bold.
DeBERTAV3 (GLUE Tasks)
GPT2 (EZE NLG Task)
Model
CoLA
SST-2
MRPC
STS-B
QOP
MNLI
QNLI
RTE
BLEU
NIST
METEOR
ROUGE
CIDEr
|| Avg.
2,2980
FedBiiFi
1.65
70.30
69.19
71.61
62.02
13:45
59.82
15.90
53.11
0.5570
7.3859
0.3413
0.5917
1.1059
2.0570
FoILORA
6.31
70.49
68.38
78.40
68.97
11.97
58.97
43.07
54.57
0,5402
7.0344
0.3625
0.5857
1.3712
1.9788
SLoRA
21.82
72.44
(0.00)
80.35
69.08
53
60.68
50.97
59
0.5700
7.9592
0.4001
0.6243
1.7322
2:2572
LFFF
0.5799
0.4064
0.6346
2.28055
5.2.2 Effect of Compression Ratio
Importance
Round Robin
122
Inpportince Round Aobin
0.2
Random
Compression Rabio
COLA 9072 MBRC STS- DOP
MULI ONU
ATE
RLEU
NE57
METEOR
ROUAGE
CIDEY
Figure 5:
Average scores of LEFF on GLUE benchmark and E2E NLG dataset different ent sampling method.
Figure 6: Scores of LEFF
Figure 7: Scores of LEFF
GLUE benchmark using differ-
E2E NLG dataset using differ- ent sampling method.
compression rates.

---

We analyze the sensitivity of LEFF's performance to the compression ratio Figure 5 shows that LEFF achieves superior performance with higher values of (i.e., lower compression levels where more parameters are retained). With higher the student model retains more parameters, enhancing its representational capacity to emulate the teacher model more effectively. This, in turn, provides more precise training context for the local model. Conversely, lower values of r (i.e., higher compression levels) indicate the student model retains fewer parameters, thereby constraining its representational capacity. This leads to less accurate teacher emulation and subsequently degrades LEFF's training efficacy and overall performance.
5.2.3 Effect of Client Scale
Table 2: Test result of DeBERTaV3
We evaluated model performance across a range of federated clients (8 to 40) on tasks from the GLUE benchmark and for NLG. Our results reveal systematic degradation in perfor- mance as the number of clients increases. As detailed in Ta- ble 2 for GLUE tasks, the robustness client scaling var- ied significantly across different metrics. For instance, SST-2 demonstrated minimal degradation, with performance decreas- ing by 3. 1,00 (from 94.30 to 91.38). In contrast, CoLA exhib- ited substantial sensitivity, with its score declining by 49.18% (from 61.51 to 31.26). Consequently, the average GLUE score dropped from 81.73 to 72.37. NLG tasks, presented in Table showed greater resilience. BLEU scores experienced marginal reduction of 1.94% (from 0.5762 to 0.5650), while CIDEr registered the largest relative decline at 5.16% (from 1.5174 1.4391). These findings suggest that while the model can maintain reasonable efficacy in federated settings with fewer clients, its performance is challenged when scaling to larger client populations.
trained under different client scales.
Number of Clients
16
24
32
40
CoLA
61.51
50.21
44.57
35.60
31.26
94.30
92.30
82.82
77.38
STS-B
86.84
85.90
85.12
80.39
79.35
QQP
88.35
87.32
86.76
86.32
86,00
MNLI
86.20
88.07
87.57
87.36
86.52
QNLI
RTE
60.01
59.33
57.16
52.71
50.90
Average
74.06
5.2.4 Effect of Sampling Methods
We evaluate LEFF employing three distinct sampling strategies: importance-basee round-robin (sequential block fine-tuming per communication round). and random (arbitrary block selection per round). As illustrated in Figure 6 and Figure the importance-based sampling strategy consistently outperforms the round-robin and ran- dom strategies across metrics from both the GLUE and E2E NLG benchmarks. On the GLUE benchmark, this performance advantage particularly pronounced on chal- lenging tasks such as CoLA. Similar trends are observed for MRPC and RTE, although the perfor- mance gap diminishes for tasks where all methods achieve high scores. For tasks within the E2E NLG benchmark, importance- based sampling demonstrates consistent superiority, yielding substantial improvements in METEOR and CIDEr scores, alongside modest gains in BLEU and NIST metrics. These results underscore its effectiveness across diverse natural language tasks.
Table 3: Test result of GPT-2 trained under different client scales.
Number Clients
24
32
40
BLEU
0,5762
0.5728
0.5726
0.5715
0.5650
NIST
7,4802
7.2763
7.2368
7,1747
6.8036
METEOR
0.3448
0.3401
0.3384
0.3352
0.3275
ROUGE
0.6056
0.6012
0.6033
0.6018
0,5981
CIDE:
1.1174
1.5139
1.5086
1,4988
1,4991
Average
2.1048
200155
20559
20034
1.9467
6 Conclusion
This paper introduced LEFF, a novel federated fine-tuning approach for LLMs that employs selective layer-wise fine-tuning to balance computational efficiency and model performance, achieving theoretical convergence rate of ((L/VT). Experimental evaluations on the GLUE benchmark and E2E NLG challenge demonstrate LEFF's performance is comparable to full fine-tuning and surpasses other parameter-efficient methods, highlighting its efficacy for resource-constrained, heterogeneous environments. Despite these strengths, LEFF's layer selection and compression introduce an in- herent efficiencc-fidelity trade-off and an approximation error (A that can cap performance, as indicated by our theory. Additionally, LEFF increases server-side computational load for tasks like layer importance calculation and specialized aggregation, and can incur substantial server-to-client communication for customized model components, particularly with lighter compression. Future research could mitigate these limitations by developing more sophisticated adaptive compression strategies, optimizing server-side operations, or exploring alternative proxy data utilization methods.

---

References
Babakniya Sara, Elkordy Ahmed Roushdy, Ezzeldin Yahya H, Liu Qingfeng, Song Kee-Bong, El- Khamy Mostafa, Avestimehr Salman. SLoRA: Federated parameter efficient fine-tuning of language models I/ arXiv preprint arXiv:2308.0653 2023.
Bahri Yasamun, Dyer Ethan, Kaplan Jared, Lee Jaehoon, Sharma Utkarsh. Explaining neural scaling laws // Proceedings of the National Academy of Sciences. 2024. 121, 27. e231 1878121.
Bai Jiamu, Chen Daoyuan, Qian Bingchen, Yao Liuyi, Li Yaliang. Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.
Chen Chaochao, Feng Xiaohua, Zhou Jun, Yin Jianwei, Zheng Xiaolin. Federated large language model: A position paper // arXiv preprint arXiv:2307 .08925. 2023.
Chen Haokun, Zhang Yao, Krompass Denis, Gu Jindong, Tresp Volker. FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning // Proceedings of the AAAI Conference on Artificial Intelligence. 38, 10. mar 2024. 11285-11293.
Chen Jinyu, Xu Wenchao, Guo Song, Wang Junxiao, Zhang Jie, Wang Haozhao. Fedtune: deep dive into efficient federated fine-tuning with pre-trained transformers // arXiv preprint arXiv:2211.08025. 2022.
Cho Yae Jee, Liu Luyang, Xu Zheng, Fahrezi Aldi, Barnes Matt, Joshi Gauri. Heterogeneous LoRA for Federated Fine-tuning of On-device Foundation Models // International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023. 2023.
Fang Zihan, Lin Zheng, Chen Zhe, Chen Xianhao, Gao Yue, Fang Yuguang. Automated fed- erated pipeline for parameter-efffcient fine-tuning of large language models // arXiv preprint arXiv:2404.06400 2024.
Fu Lele, Yang Jinghua, Chen Chuan, Zhang Chuanfu. Low-rank tensor approximation with local structure for multi-view intrinsic subspace clustering // Information Sciences. 2022. 606. 877-891.
Gardent Claire, Shimorina Anastasia, Narayan Shashi, Perez-Beltraching Laura. Creating Training Corpora for NLG Micro-Planners Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). 2017. 179-188.
Grattafior Aaron, Dubey Abhimanyu, Jauhri Abhinav, Pandey Abhinav, Kadian Abhishek. The Llama 3 Herd of Models. 2024.
Guo Tao, Guo Song, Wang Junxiao. PFedPrompt: Learning Personalized Prompt for Vision Linguu-Lan Models in Federated Learning // Proceedings of the ACM Web Conference 2023. 2023. 1374-1374. (WWW '23).
Guo Tao, Guo Song, Wang Junxiao, Tang Xueyang, Xu Wenchao. PromptFL: Let Federated Partici- pants Cooperatively Learn Prompts Instead of Models - Federated Learning in Age of Foundation Model // IEEE Transactions Mobile Computing. 2024. 23, 5. 1194
Han Zeyu, Gao Chao, Liu Jinyang, Zhang Sai Qian. Parameter--fffccient fine-tuning for large models: A comprehensive survey // arXiv preprint arXiv:2403.14608. 2024.
He Pengcheng, Gao Jianfeng, Chen Weizhu. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangle embedding sharing // arXiv preprint arXiv:2111.099 1..0543. 2021.
Hendrycks Dan, Burns Collin, Basart Steven, Zou Andy, Mazeika Mantas, Song Dawn, Steinhards Jacob. Measuring Massive Multitask Language Understanding // International Conference on Learning Representations. 2021.
Houlsby Neil, Giargiu Andrei, Jastrzebski Stanislaw, Morrone Bruna, De Laroussilhe Quentin, Gesmundo Andrea, Atlariyan Mona, Gelly Sylvain. Parameter-Efficient Transfer Learning for NLP Proceedings of the 36th International Conference on Machine Learning. 97. 2019. 2790-2799 (Proceedings of Machine Learning Research).
10

---

Hu Edward J., Shen Yelong, Wallis Phillip. Allen-Zhu Zeyuan, Li Yuanzhi, Wang Shean, Wang Lut, Chen Weizhu. LoRA: Low-Rank Adaptation of Large Language Models International Conference on Learning Representations. 2022.
Huang Wenke, Ye Mang, Du Bo. Learn From Others and Be Yourself in Heterogeneous Federated Learning // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). jun 2022. 10143-10153.
Huang Wenke, Ye Mang, Shi Zekun, Li He, Du Bo. Rethinking Federated Learning with Domain Shift: Prototype View // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023. 16312-16322
Huang Wenke, Ye Mang. Shi Zekun, Wan Guancheng, Li He, Du Bo, Yang Qiang. Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark // IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024. 46, 12. 9387-9406.
Karimireddy Sai Praneeth, Kale Satyen, Mohri Mehryar, Reddi Sashank, Stich Sebastian, Suresh Ananda Theertha. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning Proceedings of the 37th International Conference Machine Learning. 2020. 113333143.
Legate Gwen, Bernier Nicolas, Page-Caccia Lucas, Oyallon Edouard, Belilovsky Eugene. Guiding
The Last Layer in Federated Learning with Pre- Pre-Trained Models // Advances in Neural Information Processing Systems. 36. 2023. 69832-69888 -6848.
Lester Brian, Al-Rfou Rami, Constant Noah. The power of scale for parameter-efficient prompt tuning // arXiv preprint arXiv:2104.08691. 2021.
Li Junyi, Tang Tianyi, Zhao Wayne Xin, Nie Jian- Wen Ji-Rong. Pre-Trained Language Models for Text Generation: A Survey // ACM Computing Surveys. apr 2024. 56, 9.
Li Xiang, Huang Kaixuan, Yang Wenhao, Wang Shusen, Zhang Zhihua. On the Convergence of FedAvg on Non-IID Data. jun 2020.
Loshchilov lya, Hutter Frank. Decoupled Weight Decay Regularization, jan 2019.
McMahan Brendan, Moore Eider, Ramage Daniel, Hampson Seth, Areas Blaise Aguera y. Caamunication-Efffcieee Learning of Deep Networks from Decentralized Data Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 54. 2017. 1273-1282. (Proceedings of Machine Learning Research).
Molchanov Pavlo, Mallya Arun, Tyree Stephen, Frosio Iuri, Kautz Importance Estimation for Neural Network Pruning // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). jun 2019.
Novikova Jekaterina, Dušek Ondrej, Rieser Verena. The E2E Dataset: New Challenges for End- to-End Generation // Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2017. arXiv:1706.0922.
Pan Rui, Liu Xiang, Diao Shizhe, Pi Renjie, Zhang Jipeng. Han Chi, Zhang Tong. LISA: Layerwise Importance Sampling for Cemory-Efficiee Large Language Model Fine-Tuning arXiv preprint arXiv:2403.179. 2024.
Qi Zhuang, Meng Lei, Chen Zitan Hu Han, Lin Hui, Meng Xiangxu. Cross-silo prototypical calibration for federated learning with non-iid data // Proceedings of the 31st ACM International Conference on Multimedia. 2023. 3009-3107.
Qin Zhen, Chen Daoyuun, Qian Bingchen, Ding Bolin, Li Yaliang, Deng Shuiguang. Federated full-parameter tuning billion-sized language models with communication cost under 18 kilobytes // Proceedings of the 41st International Conference Machine Leamiing 2024. (ICMI.24).
Radford Alec, Wu Jeffrey, Child Rewon, Luan David, Amodei Dario, Sutskever Nya. Language models unsupervised multitask learners // OpenAl blog. 2019. 1, 8.

---

Ranathunga Surangika, Lee En-Shiun Annie, Prifti Skenduli Marjana, Shekhar Ravi, Alam Mehreen, Kaur Rishemjit. Neural Machine Translation for Low-resource Languages: Survey II ACM Computing Surveys. feb 2023. 55, 11.
Shen Jinglong, Cheng Nan, Wang Xiucheng, Lyu Feng, Xu Wenchao, Liu Zhi, Aldubaikhy Khalid, Shen Xuemin. RingSFL: An Adaptive Split Federated Learning Towards Taming Client Heterogeneity IEEE Transactions on Mobile Computing. 2024. 23, 5. 5462-5442 5478.
Shu Yao, Hu Wenyang, Ng See-Kiong. Low Bryan Kian Hsiang, Yu Fei Richard. Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models. 2025.
Sun Guangyu, Mendieta Matias, Yang Taojiannan, Chen Chen. Conquering the communication con- straints to enable large pre-trained models in federated learning // arXiv preprint arXivv2210.01708. 2022.
Wong Alex, Singh Amanpreet, Michael Julian, Hill Felix, Levy Omer, Bowman Samuel R. GLUE: A Multi-Tast Task Benchmark and Analysis Platform for Natural Language Understanding International Conference on Learning Representations, 2019.
Wang Haozhao, Jia Yabo, Zhang Meng, Hu Qinghao, Ren Hao, Sun Peng. Wen Yonggang, Zhang Tianwei. FedDSE: Distribution-awaln aware Sub-model Extraction for Federated Learning over Resource- constrained Devices // Proceedings of the ACM Web Conference 2024. 2024a. 2902-2912 (WWW '24).
Wang Haozhao, Zheng Peirong, Han Xingshuo, Xu Wenchao, Li Ruixuan, Zhang Tianwei. FedNLR: Federated Learning with Neuron-wise Learning Rates // Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024b. 3069-3080.
Wang Lin, Wang Zhichav, Tang Xiaoying. Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent. 2024c.
Wang Ziyao, Shen Zheyu, He Yexiao, Sun Guoheng, Wang Hongyi, Lyu Lingjuan, Li Ang. FLORA: Federated Fine-Tuning Large Language Models with Heterogeneous -aw-Rank Adaptations // The eiihtt- Annual Conference on Neural Information Processing Systems. 2024d.
Woisetschläger Herbert, Erben Alexander, Wang Shiqiang. Mayer Ruben, Jacobsen Hans-Arno. Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly // Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning. 2024. 39-50. (DEEM 24).
Xu Mengwei, Wu Yaozong, Cai Dongqi, Xiang, Wang Shangguang. Federated fine-tuning of billion-sized language models across mobile devices // arXiv preprint arXiv:2308.1 133444 2023.
Yu Fei, Zhang Hongbo, Tiwari Prayag, Wang Benyou. Natural Language Reasoning, A Survey // ACM Computing Surveys. may 2024. Just Accepted.
Yu Sixing, Muñoz J Pablo, Jannesari Ali. Federated foundation models: collaborative learning for large models // arXiv preprint arXiv:2305. 1414. 2023.
and
Zaken Elad Ben, Ravfogel Shauli, Goldberg Yoar. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models // arXiv preprint arXiv:2106. 10199. 2021.
Zhang Jianyi, Vahidian Saeed, Kuo Martin, Li Chunyuan, Zhang Ruiyi, Yu Tong, Wang Guoyin, Chen Yiran. Towards Building The FederatedGPT: Federated Instruction Tuning // ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2024. 6919.
Zhang Zhuo, Yang Yuanhang, Dai Yong, Qifan Wang, Yu Yue, Qu Lizhen, Xu Zenglin. FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models // Findings of the Association for Computational Linguistics: ACL 2023. 2023. 9963-9977
Zhuang Weiming, Chen Chen, Lyu Lingjuan. When foundation model meets federated learning: Motivations, challenges, and future directions // arXiv preprint arXiv:2306.15 15546. 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly outline the LEFF framework, its mecha- nisms for efficiency and performance preservation (e.g., layer selection, model compression), handling of data and system heterogeneity, the theoretical convergence rate claim, and the claim of superior empirical results. These aspects are subsequently detailed and supported in Sections 3 (Methodology). (Convergence Analysis), (Evaluations)
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The Conclusion (Section 6 explicitly discusses limitations, including the fffeiency-fideliig trade-off due layer selection and compression, the impact of approxi- mation eтror ( on performance, increased server-side computational load, and potential server-to-client communication overhead.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Section 4 (secciiically Subsection 4.1) lists the assumptions (Assumptions T for the theoretical analysis. The main theoretical results (Theorem [ and Corollary 1 are presented, and the paper states that proofs are provided in the Appendix (Sections A, B).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section (specifically Subsection 1) details the experimental setup, includ- ing baselines, models (GPT-2 Medium, DeBERTaV3 Base), datasets (GLUE, E2E NLG, WebNLG, WNLI), data heterogeneity settings (Dirichlet a)., number of clients, communi- cation rounds, and local epochs. Key components of LEFF like layer sampling and model compression are described Section 3 The provided information covers the core aspects needed to understand and attempt reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
14

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The paper uses public datasets (GLUE, E2E NLG, WebNLG, WNLI) and existing pre-trained models (GPT-2, DeBERTaV3), which are cited. However, it does not explicitly state that the code for the proposed LEFF method or the experimental scripts are publicly available, nor does it provide a link or instructions for accessing them.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ttppe.cc/uublic/guuuues/////// hwwpiiiliggguidsse/eeeeo Cooe//ooeeeeeeeeeeeeeeeeeee waweeSuumisssioon wwweiisssioooooooo ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section (Experimental Setup) specifies the models (GPT-2 Medium, De- BERTaV3 Base), datasets (GLUE, E2E NLG, WebNLG, WNLI), data partitioning (Dirichlet with a), optimizer (AdamW), learning rate
10 10-5), communication rounds (20), and lo- cal epochs (1). Key aspects of LEFF like layer sampling and model compression (including the analysis of compression ratio r) are described.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
15

---

7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The authors provide the standard deviation of the experimental results in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 5 (Experimental Setu) states that 'All experiments were performed system with eight NVIDIA H100 GPUs." This specifies the type and number of compute workers.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the https: ttticsshhtps:..
NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research focuses on algorithmic development for efficient federated fine- tuning of language models using publicly available datasets. The work aims to improve efficiency and manage resource constraints, with no apparent direct ethical concerns or violations of the NeurIPS Code of Ethics.
16

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
A discussion of broader impacts is provided in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper proposes new fine-tuning method (LEFF) and evaluates it using existing pre-trained models (GPT-2, DeBERTaV3) and public datasets. does not introduce release new models or datasets that would inherently pose high risk for misuse requiring specific safeguards developed by the authors.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
17

---

12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper properly cites the original sources for existing assets like pre-trained
models 2 et al. (2019), DeBERTaV3 He et al. (202T) and datasets (GLUE
Wang et al. (011). E2E NLG Novikova al 2017). WebNLG Gardent et al. (2017)..
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces new method (LEFF). The paper itself serves as docu- mentation for this method. No new datasets standalone models are introduced for release as distinct assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsourcing experiments or new research with human subjects; it utilizes existing public datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The research does not involve new studies with human subjects, so IRB
approval or discussion of participant risks is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
method (LEFF) for fine-tuning LLMs. LLMs are the subject of the study (e.g., GPT-2, DeBERTaV3 are fine-tuned), not tool used in important, original, or non-standard way to develop the core LEFF methodology itself (e.g., for algorithm design or proof generation).
Justification: The research focuses developing
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, non non-standart components.
Please refer to LLM policy /https://eeurip:: http:/ipp...c//oooeeeeeer//////0 for what should or should not be described.
19

---

A Proof of Lemmall 1
A.1 Bound on the Expected Local Gradient Norm
Proof. We use the standard variance decomposition ||||X| = E + |
Es [ I|]
Es | BelGs,, + ||E]GG,..All*.
(19)
By definition, Eg = Vec, F.(0 The first term is the variance of the stochastic gradient given the current local model O i,,, and the (potentially compressed) non-updated param-
eters 6 Assumption provides bound the variance of the gradient for the full model: [ <) (FF S of S oo? We assume this bound also applies to the variance the stochastic gradient computed only for the selected layers Li:
Eg |
Ee 0²²
(20)
For the second term, the squared norm of the expected gradient, we use the inequality ||a + b112 < 2||a|| + 2|
Vog F, M. 16
Vec, Fi(g,.)) VV,, F.00,)) (V LiN (g
(21)
2 V Vec. F. 16, Vic.F.(0
+ Vgc
The first term in 21 captures the deviation of the expected local gradient at step from the true local gradient (for layers L.) evaluated at the round's initial global model 0g,-- This de- viation arises from both the drift of the local model (0 vs (g.)) and the use of potentially
approximated parameters (6..). Assumption 4 bounds the average deviation Ec,sp, Ec,,p,I||s, Ve F(0g.))1) < A where G., = (1/K) E Eg[Gr,,s). While Assump- tion Applies to the average, analyses often rely on bounding the instantaneous deviation. We make simplifying assumption. that the expected instantancous deviation (over L,) is also related to the average approximation error bound Af. Specifically, we assume:
K steps:
Baaep
Von FiO 21. 16, F(0,.)) S C
(22)
for some constant
2 1.
Combining 20
21 and
22
and taking expectation over C, p;:
Ex. Ee
G
0² +
E
~ + Ec. 2 Von. Fi (1 -Vec.F(e..)) (oc.F((epi)(+2222 +2|| F.(y.))
(23)
00²² + 20, 2Bc. V
< cb 02 A E.c,
where = 2c, is a constant.
A.2 Bound on the Global Model Update Norm
Proof. The squared norm of the global model change is the sum of the squared norms of the changes each layer:
1.9. 01,.411--00 -0,
E
0! g.t
| p. 1
(24)
20

---

where 6',, denotes the parameters of the I-th layer of the global model at round t.
The global model update for layer round [+11 is given by the weighted average of the corresponding layer parameters from clients that updated this layer:
O
w L.K
(25)
g,++1
FES
where S = fi A, I L.) is the set of clients selected round t (A₂) that included layer in their
layer block Ci, w = Dz/(E Dj) are the aggregation weights (with Eies; w = 1, w 2 0), and ,,,,,K is the 1-th layer parameter of client after K local steps starting from 1 O 0
The local update process for layer on client (if I E L,) follows:
K
K-1
e , = 00,00 - 7 Σ Gh.,a =
0 ni
Σ
G
(26)
k==
where G is the component of the stochastic gradient Gi,t,k (computed by client local step using data sample fr,,,)) corresponding to layer I.
Therefore, the change the global model's 's /-th layer is:
@ g.t11 O¹ Σ ( VES
- Σ w
Σ k=0
G.
(27)
L.K
iES!
We want to bound the expected squared norm E[|| g. - 1. Let 81 = Or,,KK - -6',1
K k= Gr denote the total update applied by client i to layer (if E L,).
E 110 - 02 0/.11
Σ
(28)
iESt
The expectation E[-] is over all sources of randomness: A,. fiieE and (Er,c. beAA
Since || 2| is a convex function and Eies w = 1 with w 2 O, we can apply Jensen's inequality:
Σ o
w 18,.11
(29)
iES
iES
Taking the expectation over all randomness:
E
|
9,1+1
19]
< E
6% 11
(30)
JESt
We can rewrite the expectation using the law total expectation, conditioning first on the client and layer selections (Ar, Li), which determines the set St and weights wt:
S B.A.C.
Σ E
Ae, Lo
(31)
KES
where Eg|- Ar, L.] denotes the expectation the client and layer selections. Note that the condition S implies E A, and E C..
the stochasticity of local gradients {Er,. given
Now, we bound the inner term Eg .|l 2
Ar, L.] for E S):
K-1 Σ G
Eg [16%,,11 Acs C.] = Eg
1
Σ
Gl
i..k
A Li
= =
| Ls
k=0
(32)
21

---

Using the standard bound Blll Karimireddy et al. (2020))
Σ R=0
Xx
&
K
E R=
E
X
[ (often used in FL analysis
Eg
Σ Gha
Ar,Li
K
Σ
Eg
||
21 Ac: Ca]
(33)
Since ,r,,k is the I-th layer component of the full gradient Gi.tk we have 1.,, S |.....&ll'' 11? Thus:
Eg G ,.
|A C.] s Ee [[G 1
(34)
From Assumptions 2 and 4 the expected squared norm of the stochastic gradient Gs,ck computed by client at local step it can be bounded. common bound derived under these assumptions in (23 takes the form:
Eg [|GLLNIl As. C. S Cgrad (0- A (VF((,,))1))
(35)
where Cgrad is constant, and g, as reference point, absorbing dependencies the intermediate local models Oi,,,K into the constant Cgrad (this is a common simplification in FL analysis, valid for sufficiently small yR). Let X = = + A + (V.F( Then:
Ee [|G A Ri] < CgndXin
(36)
Substituting back:
K-1
Eg ||% 2
Ai, Li
S 1 K
Σ
(Cana = x22 Craat
(37)
k=0
Now, substitute this into the bound for B[[[O'
g, 0g,x133
E |10 0/,ell
< EAA.,
Σ wiini² 1² Cie) FESt
(38)
Σ iESt
Let Es;[]] denote the expectation E. The term s Eies; w(())] represents the expected weighted average over clients participating in the update for layer 1. Under assumptions of unbiased client sampling (e.g.. uniform random sampling of Ar clients) and potentially layer sampling (C.), this expectation can be related to the average over all clients. Assuming there exists a constant ca (which may depend on sampling strategy, e.g., A/N, and the distribution of weights w/) such that:
N
Est
Σ wfXis iES!
S cd
1 N
Σ
X
(39)
Then:
E ||0
g,0+1
6/,11 k Cgrad
1
Σ
A
+
(.,))
N
(40)
2²² K2 K2 CanaCCa
02
IFF
22

---

Now we use Assumption have the decomposition:
3 [Bounded Heterogeneity): N/ Ex= |VF(()) - V=(O)]| < < We
1 (V,)(Og.)) +
VF((
N
N
(). +1000==+++100100002=222030000010 V VVT VF((,,)))
[VF V=((O - +
+
N
+
VF( -FF(,),))
vF(O,.)) VFOO.)))))))))))
(41)
Since V.F(Og.)) = N E VFi the cross term is zero. Using Assumption
3
(42)
N
Substituting this into the bound for the layer update:
E | (0
S K2 Cgnace (o² + + c² VF(():))
(43)
Finally, sum over all layers Li...,L:
E |0 - 91
-
E |100 9,141 -%,.1
1=1
(44)
[n K (0² A3 (² [[VF(00))13]
Lxj Craud (o² + A ?² + |VF(),,))
Let Cg CGgraCaa This constant incorporates the number of layers L, the constant from the gradient bound Cyna. and the constant related to sampling and weighting ca. This yields the final result:
E
- -
I Csn2 K2 (0² + + c² (VF(9.)))))
(45)
This completes the proof.
B Proof of Main Theorem
Proof. this section, we present the convergence analysis for our algorithm. Our goal is to bound the average squared norm of the global gradient over T communication rounds.
We begin with the L-smoothness property the global objective function F, stated in Assumption By the definition of L-smoothness, often referred to as the Descent Lemma or Quadratic Upper Bound, we have:
Og.)) + 1 B[] (g.el13).
E[F(Og. S F(Og.)) + (eee))+E((((((CCC E(VF(())))
(46)
where the expectation [E[-] taken over all sources of randomness up to round t including client sampling (At.,, layer sampling (C, for i E At), and local stochastic gradient noise (Ei,,.))
The core of the analysis involves carefully bounding the inner product term E(( 0 Og,,)] and the quadratic term E[][O,,tt g,1+1 2). The bound for the quadratic term is typically established in a separate lemma (referred to as Lemma T). We focus on deriving the bound for the inner product term.
23

---

B.1 Bounding the Inner Product Term
Recall the global model update is constructed layer-wise:
O g,
Og = {0 g,,41
(47)
where for each layer
K-1
0 9,++1
6 -1
Σ
W
(48)
iES!
km0
Here, S = (i A. is the set of selected clients that update layer 1, w is the aggregation weight for client i on layer and G is the stochastic gradient computed by client i for layer I at local step k.
The inner product term can be expanded
over layers:
(((V(((g,)) 9,.41 gg,e)l = E
V Vi. (00
K-1
Σ
(VeiF((gg.)
-17 Σ
W
Σ
Grand
(49)
iES;
K-1
Σ n (VeI
9.1),
Σ
Σ
G
Gladd
FES
Let [g[-] denote the expectation over the local data samples 5, conditioned on the client selection K-1 k== E& (eGG,,,, (rr Li] be the average expected local
A, and layer selections Lj. Let G²,, k Σ gradient for layer computed by client (if E L.). Then KK1 Ee the expectation over & first, we get:
Ee(G i,t. Ars L.] KG Taking
EK = Σ BBAi (VeiF(0ge), Σ w
iESt
(50)
-K Σ qE (VoiF(Og.t)
Σ iESt
where the outer expectation E[-] is now over A, and Ci.
Standard techniques in FL analysis (e.g.. using algebraic identities like 2(a, b) ||||2 + ||b|2 ||a - b1l2 or Young's inequality, and carefully handling the expectations over sampling) allow us to bound the inner product. This involves decomposing the error terms and applying the assumptions. While the detailed derivation is intricate due to the layer selection and weighting scheme, it leads to bound of the following form (similar to analyses of FedAvg variants):
(51) 2, is the average model approximation error bound across clients round t. The constants C4 and C3 depend factors like the client sampling ratio (A/N), layer sampling probabilities (p,), and potentially the distribution of weights w/. The term CC represents the desired descent along the negative gradient direction, while the term Can <"+ captures the accumulated error from model approximation, heterogeneity, and residual stochastic gradient variance (scaled by 1/K K due to averaging K steps implicitly or explicitly in the derivation).
B((VFF(Og.)) Qg.t11 Og,)) -V where 43 + A
Csn + 2² +
B.2 Completing the Proof
We now substitute the inner product bound (51) and the bound on the quadratic term from Lemma 1 into the L-smoothness inequality (46). Lemmal I typically provides bound like:
BillO,,141 .0,,e113 < C'n ( + A + 2 [VF((,
(52)
24

---

where C3 is another constant derived in Lemma]
Substituting (51) and 52 into (46)
([ S F(O 9.1) Cn Csnk(() +
L
(53)
+
2
Rearranging the terms to group the gradient norm:
(Cink- 1.Cs x²²
B[F(0g,,41)) S g()
|VF((g,)))|
2
(54)
LC3s 2 -2+
+ +
To ensure convergence, we require the coefficient the gradient norm term to be positive. We choose
the local learning rate 7) sufficiently small such that C,nK L/an K2 0. standard choice is make the second term at most half of the first term, which holds if S
C. Under this condition:
LCaK'
LC3 R 2
LC3
C4 [CCg
C4 n
C4 nk
Cank
Cank >ank
nk Cank
(55)
2
Thus, the inequality becomes:
C4 $ C. [
(56)
+ Conn + + Cn + +2
where we define Cg = C55 and C- = LC3/2.
Rearranging to isolate the gradient term:
C4 [K]][ <F((,,)) (F(e))) (E(((),,1))
(57)
+ Conk ( + + + +
Summing this inequality over t 0,1 0.1..... 1:
C4
T-1
nK BilVFF(9,,)))) g0,)))
(F(O),) - -E[[(,))[[((
2
1=0
+2 O +2 {3)
(58)
The first term on the right-hand side telescoping sum:
T- (F(0,,)) E[F F(9g.0) (((0,))-B(F((0,.. -[FF(O,rr)
(59)
Assuming the objective function is bounded below by F", i.e. F(O) 2 F* for all θ we have F(O,,o) - E[F(O,,r)] < F(9 - FF.
Substituting this back and dividing by T:
Cank 2T
F(O,,)) F
Ell E
T
Σ
[Cnn 2 + K2 + + + {
T
(60)
25

---

Finally, dividing by Cank
yields the bound on the average squared gradient norm:
2
2(F(O,,0)
FT)
Σ
C
2Cs
T-1 Σ
o²
A
+
5²
(61)
C2
T
K
1=0
T-1 Σ
207
n
(02
+ (3)
C4
T
1=0
Let A 7 -
ΣT= A be the average model approximation enror over T rounds. Define constants Cs 2/C₁. Cg 2Cs/ CO,, and C10 = 2C7/C1. We arrive at the final convergence result:
1
Ca(F(O g.0) F*) nKT +C
0² 22+
||| F().)(()
A2
+ Cionk A (2)
T
K
Vanishing term
Error floor
(62)
This result shows that the average squared gradient norm converges to an error floor determined by the data heterogeneity (C²), stochastic gradient noise (a² /)), average model approximation error (A and the chosen learning rate (1)), at rate of (1/TT).
C Additional Experimental Results and A Analyses
This appendix provides supplementary materials and detailed results that support the claims made in the main paper. The content is organized to present further validation of LEFF's scalability, comprehensive analysis of client-side overhead, ablation studies key hyperparameters, and empirical validation our theoretical convergence claims.
C.1 Scalability to State-of-the-Arl Models and Benchmarks
To demonstrate the scalability and effectiveness of LEFF on contemporary large-scale architectures, we extend our evaluation to the Llama-3, Grattafioni et al. (2024) model using the challenging MMLU benchmark Hendrycks al. (021) This experiment includes highly relevant state-of-the-ant baselines FLoRA Wang et al. (2024d) and FoeXLoRA Bai et al. (2024). As shown in Table LEFF outperforms these recent methods by significant margin of 1.8 points the 5-shot MMLU average. This superior performance achieved while simultaneously reducing the peak client-side GPU memory by over 36%. These results confirm that LEFF's design is highly effective and its efficiency benefits are even more pronounced on larger models where client-side resources are the primary bottleneck.
Table 4: Comparison of LEFF with SOTA methods on the Llama-3.1 1-88B model using the MMLU benchmark (5-shot average accuracy).
Method
MMLU (5-shot Avg.)
Peak GPU Memory (Client)
FlexLoRA
55.7
46. 46.868 GB
FLoRA
55.2
46.868 GB
LEFF
57.5
29.881 GB
C.2 Analysis of Client-Side Computational Overhead
A detailed analysis of the computational overhead on client devices is crucial for evaluating the practical viability of any federated fine-tuning method. We present a comprehensive comparison of the number of trainable parameters and the peak GPU memory usage across various models and algorithms in T This analysis highlights LEFF's unique and superior efficiency profile. LEFF drastically reduces peak GPU memory for instance, achieving a 79% reduction GPT2-Large
26

---

compared to FedAvg and 66% compared to FedLoRA. This is because its architecture selectively loads only the necessary layers into GPU memory, whereas other PEFT methods must load the entire frozen model, resulting in much higher memory footprints. Concurrently, while achieving the lowest memory cost, LEFF deliberately retains more trainable parameters than methods like FedLoRRA or FedBitFit. This design maintains greater model expressiveness, which translates to superior task performance. Thus, LEFF strikes an optimal balance, delivering the minimal resource cost for deployment constrained devices while preserving enough capacity for high-accuracy fine-tuning.
Table Computational cost comparison of different models and algorithms. "OOM" indicates Out of Memory. For 1lama-3.1-8B. FedBitFit is not applicable as the model has no bias terms to train.
Model
Algorithm
Trainable Params
Peak Memory (GB)
DeBERTaV3 B-Base
FedAvg
85,648, 130
3.841
DeBERTaV3 3- Base
FedLoRA
1,340,930
2.644
DeBERTa B-Bae
FedBitFit
102,914
2.198
DeBERTiaV3-Bass
LEFF
7,681,538
2.136
DeBERTaaV3-Base
SLoRA
1,340,930
2.644
LeBaRgaV3-Large
FedAvg
303,363,074
9.361
LeBaRgav
FedLoRA
3,557.378
6.660
L-Baare
FedBitFit
272,386
5.488
DeBERTaV3-Large
LEFF
13,649.922
3.005
DeBERTaV3-
SLoRA
3,557,378
6.660
GPT2
FedAvg
85,056,000
3.358
GPT2
FedLoRA
811,008
2.476
GPT2
FedBitFit
102,144
2.206
GPT2
LEFF
7,089,408
1.719
GPT2
SL.RA
811,008
2.476
GPT2-Large
FedAvg
708_390,400
15.548
GPT2-Large
FedLoRA
4,055,040
9.739
GPT2-L.arge
FedBitFit
508,
8.318
GPT2-1.arge
LEFF
19,680,000
3.240
GPT2-Large
SLoRA
4.055,040
9.739
Llama-3. 3.1-aB
FedAvg
OOM
OOM
Llama-3.1 3.B
FedLoRA
20,971,520
46.86 46.868
Llama-3 3.1-BB
FedBitFit
Bias
No Bias
Llama-3. 3.1-B-3.11-B
LEFF
743.452,672
29.881
Llama-3.1188
SLoRA 20,971,520
46.868
C.3 Ablation Study on the Proxy Dataset
The choice of the proxy dataset for knowledge distillation is a key aspect of the LEFF framework. We conduct an ablation study to investigate the sensitivity of our method to this choice. Our distillation objective is designed to robust by matching intermediate functional representations (hidden states and attention matrices) rather than task-specific knowledge, which should reduce sensitivity to the proxy data's domain. The empirical results on the E2E NLG task, shown Table 6 confirm this hypothesis. Performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose datasets like WikiText- 103 and Open WebTet. The minimal fluctuation across all metrics demonstrates that LEFF is not reliant on perfectly-match corpus, validating its practical applicability in real-world scenarios where such data may not be readily available.
proxy
Table 6: Ablation study on the choice of proxy datasets for the E2E NLG task.
Proxy Dataset
BLEU
NIST
METEOR
ROUGE
CIDEr
WikiText- 103
0,5765
8.0012
0.4041
0.6310
1.7450
WebNLG
0.5799
8.0296
0.4064
0.6346
1.7521
Open WebText
0.5712
7.9688
0.4015
0.6259
1.7345
27

---

FedAvg
2.50
LEFF (Ours)
SLoRA
2,25
FedLoRA
Validation Loss
2.00
FedBitfit
1.75
1.50
1.25
1.00
0.75
2.5
5.0
7.5
12.5
15.0
17.5
20.0
Communication Rounds
Figure 8: The validation loss convergence curve of different methods under high data heterogeneity
(a 0.05).
C.4 Statistical Stability of Results
To ensure the reliability of our findings, we report the standard deviations of our main experimental results across three runs with different random seeds. The results, summarized in Table 7 show that the standard deviations are consistently small across all tasks and methods. This confirms that method is stable and the reported performance gains are consistent. LEFF exhibits stability comparable to or better than the baselines, substantiating the robustness of our conclusions.
Table 7: Standard deviations of test results for DeBERTaVe V3 (on GLUE tasks) and GPT2 (on the E2E
NLG task) across three runs with different random seeds.
DEBERTAY3 (GLUE Tasks)
GPT2 (E2E NLG Task)
Model
CoLA
SST-2
MRPC
STS-B
QQP
QNLI
BLEU
NIST
METBOR
ROUGE
CIDEr
FodAvg
1.52
0.71
0.88
0.45
0.31
0.53
0.62
1.21
0.0041
0.1512
0.0028
0.0033
0.0805
FedBitHi
1.89
0.95
1.02
0.82
0.55
0.98
0.77
0.0053
0.2144
0.0045
0.0049
0.1231
FedLoRA
1.72
0.88
1.15
0.75
0.49
1.05
0.81
1.68
0.0058
0.2516
0.0041
0.0051
0.1189
SLoRA
145
0.81
0.95
0.68
0.42
0.85
0.71
1.35
0.0049
0. 1832
0.0035
0.0039
0.0948
LEFF
1.8
0.75
0.91
0.51
0.35
0.62
0.68
1.25
0.0045
0.1604
0.0031
0.0035
0,0882
C.5
Empirical Convergence Analysis
Figure g provides an empirical validation of our theoretical convergence analysis presented in Section The validation loss curves for all compared methods plotted under a high data heterogeneity setting (a 0.05). The results show that LEFF converges stably and efficiently. Its final validation loss substantially closer to that of full fine-tuning (FedAvg) compared other PEFT methods like FedLoRA and SLoRA. This empirical evidence supports our theoretical analysis (Theorem I]. demonstrating that LEFF not only converges but also reaches a superior solution in challenging non-lll) environments.
D Broader Impacts
D.I Environmental Cost
The server-side computation in LEFF for layer importance calculation and knowledge distillation represents trade-off for enabling client-side efficiency. By making fine-tuning feasible numerous distributed, low-power edge devices, our approach can reduce the overall system's 's reliance on large, consistently -ntegy-intessvve data centers. This decentralization lowers the barrier to entry for
28

---

participation in model training and may lead to more distributed and potentially more energy- efficient computational footprint compared to fully centralized training paradigms.
D.2 Dual-Use Risks
Making LLM fine-tuning more efficient and accessible could lower the barrier for misuse. However, the federated architecture of LEFF provides a natural control and governance point the central server. This centralized aggregation step allows for the implementation of safeguards, such as client vetting, anomaly detection in model updates, and monitoring for malicious fine-tuning objectives. Such governance features are largely absent in purely decentralized or local fine-tuning scenarios, providing mechanism mitigate potential misuse.
29


Reviews:
[{"review_id": "KFLg93pv4T", "reviewer": "Reviewer_k7EF", "summary": "This paper introduces LEFF, a novel framework designed to efficiently fine-tune LLMs in FL settings, especially for resource-constrained edge devices. LEFF enables clients to fine-tune only selected layers of a model based on their computational capacity and leverages importance-based sampling to prioritize impactful layers. To reduce overhead, the server compresses unselected layers via knowledge distillation before sending a custom model to each client. The method incorporates a layer-wise aggregation strategy and is backed by theoretical convergence guarantees. Experimental results show that LEFF outperforms other FL fine-tuning approaches under heterogeneous data and system conditions while preserving performance close to full fine-tuning.", "questions": "1. What are the computational costs of calculating layer importance at each round, and is this feasible at scale?\n2. Have you considered applying LEFF to newer LLMs such as LLaMA-3 or Mistral? Would LEFF also works for them?\n3. Why were benchmarks like MMLU or MT-Bench not included, and how might LEFF perform on those tasks?", "limitations": "Compared to more recent works like (FLoRA)[https://neurips.cc/virtual/2024/poster/95025] (NeurIPS 2024), which evaluate on cutting-edge models and benchmarks, this paper primarily focuses on older architectures (e.g., GPT-2, DeBERTaV3) and traditional tasks (e.g., GLUE). This limits the ability to assess LEFF's applicability and competitiveness on SOTA models and evaluation standards such as MMLU or MT-Bench.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. LEFF significantly reduces client-side overhead by allowing selective layer-wise training, making it suitable for edge devices with limited resources.\n2. The method includes formal convergence guarantees, reinforcing its reliability and scalability.\n3. The approach effectively handles data and system heterogeneity using importance-based sampling and dynamic model customization.\n\nWeaknesses:\n1. Layer compression and partial updates can introduce approximation errors that cap model accuracy, especially in high-compression scenarios.\n2. The compression process relies on publicly available proxy datasets, which may not always be representative or available.\n3. The experiments do not include evaluations on more recent LLMs such as LLaMA3, nor do they incorporate  benchmarks like MMLU or MT-Bench, which limits the assessment of LEFF's effectiveness on state-of-the-art models and tasks.", "was_revised": true, "final_justification": "I would give this paper a borderline accept because it is a complete work with sound theory, sufficient experiments, and well-written manuscripts.", "rebuttal": "> *(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)*\n\nWe sincerely thank the reviewer for their insightful and constructive feedback, which has helped us to significantly strengthen our paper. We have grouped our responses thematically to address the core concerns raised about SOTA model applicability, approximation error, proxy data dependency, and computational cost.\n\n---\n\n# (W.3, Q.2, Q.3, L.1) Performance on SOTA Models (Llama-3.1-8B) & Benchmarks (MMLU)\n\nA central theme of your feedback (concerns **W.3, Q.2, Q.3, L.1**) was the need to evaluate LEFF on more recent models and challenging benchmarks to assess its competitiveness. We agree this is crucial and have conducted **new experiments** during the rebuttal period to address this directly.\n\nOur initial experiments on GPT-2/DeBERTaV3 were designed to rigorously validate LEFF’s core contributions to **efficiency and robustness in controlled, heterogeneous FL settings**, where tasks like GLUE allow for clear analysis of factors like label skew.\n\nTo prove LEFF's effectiveness on modern architectures, we have now evaluated it on **Llama-3.1-8B** using the **MMLU** benchmark. We included **FLoRA (NeurIPS 2024) [1] and FlexLoRA (NeurIPS 2024) [2]** as highly relevant SOTA baselines. The results are summarized below:\n\n| Method                  | MMLU (5-shot Avg.) | Peak GPU Memory (Client) |\n| ----------------------- | ------------------ | ------------------------ |\n| FlexLoRA (NeurIPS 2024) | 55.7               | 46.868 GB                |\n| FLoRA (NeurIPS 2024)    | 55.2               | 46.868 GB                |\n| **LEFF (Ours)**         | **57.5**           | **29.881 GB**            |\n\nAs shown, LEFF not only applies to SOTA models but excels, **outperforming recent NeurIPS 2024 methods FLoRA and FlexLoRA by 1.8-2.3 points on MMLU** while simultaneously **reducing client-side peak memory by over 36%**. These new results confirm that LEFF is highly competitive and its benefits are even more pronounced on larger models where client-side resources are the primary bottleneck. We will add these results to the appendix.\n\n[1] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations, NeurIPS 2024.\n\n[2] Federated fine-tuning of large language models under heterogeneous language tasks and client resources, NeurIPS 2024.\n\n---\n\n# (W.1) On Approximation Error and Model Accuracy\n\nRegarding your concern (**W.1**) that layer compression introduces approximation errors that cap accuracy, we agree that managing this error is critical. LEFF is designed from the ground up to actively model and minimize this trade-off.\n\n1. **System-Wide Robustness:** The \"high-compression scenario\" only applies to the most resource-constrained clients. As detailed in Sec. 3.1, LEFF is heterogeneous-aware. Higher-capacity clients perform updates with less (or no) compression, submitting high-fidelity gradients. During server-side aggregation, these high-quality updates **compensate for the higher-error gradients from weaker clients**, preventing the global model's accuracy from being capped by the worst-case devices. Our strong results in Table 2, where LEFF nearly matches full fine-tuning (FedAvg), empirically validate that this error is well-controlled.\n2. **Principled Error Minimization:** LEFF is not a random compression scheme. The **server** tailors a model for each client by using **importance-based sampling** (Sec. 3.2) to select critical layers for full-parameter updates, while using **knowledge distillation** (Sec. 3.3) to create compact, function-preserving representations of the rest. This principled approach ensures that clients perform local training with high-fidelity gradients, directly minimizing the approximation error $\\bar{\\Delta}^2$ from our convergence analysis (Thm. 1).\n\n---\n\n# (W.2) On the Role and Robustness of the Proxy Dataset\n\nWe thank the reviewer for raising the practical point (**W.2**) about the dependency on a proxy dataset. We have designed LEFF to be robust to this.\n\n1. **Robustness to Data Distribution:** Our method is robust to the choice of proxy data because we distill **functional representations**, not task-specific knowledge. As described in Section 3.3, our objective is to match the intermediate hidden states and attention matrices. This process preserves general representational capabilities and is far less sensitive to the specific domain of the input data.\n2. **Practical Availability:** Since our method only requires unlabeled text, readily available public corpora (e.g., C4, Wikipedia) are sufficient. In scenarios where external data is prohibited, the server can use the global model to **generate a synthetic proxy dataset**, making the framework entirely self-contained — a promising direction we are exploring.\n3. **New Ablation Study:** We empirically demonstrate LEFF’s robustness to the proxy dataset via a new ablation study on the E2E NLG task. As shown in the table, performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose WikiText-103 and OpenWebText. While the in-domain data yields a slight, expected performance gain, the minimal fluctuation across all metrics confirms that LEFF is not reliant on a perfectly-matched proxy corpus. This high degree of robustness validates its practical applicability. We will include this study in the appendix.\n\n| Proxy Dataset ($\\mathcal{D}_{\\text{proxy}}$​) | BLEU   | NIST   | METEOR | ROUGE | CIDEr  |\n| --------------------------------------------- | ------ | ------ | ------ | ------- | ------ |\n| WikiText-103                                  | 0.5765 | 8.0012 | 0.4041 | 0.6310  | 1.7450 |\n| WebNLG                                        | 0.5799 | 8.0296 | 0.4064 | 0.6346  | 1.7521 |\n| OpenWebText                                   | 0.5712 | 7.9688 | 0.4015 | 0.6259  | 1.7345 |\n\n---\n\n# (Q.1) Computational Cost of Layer Importance\n\nIn response to your question (**Q.1**), the computational cost of calculating layer importance is **minimal and highly scalable**.\n\nThe calculation uses a first-order Taylor approximation, requiring only a **single forward and backward pass** on the global model. This operation is performed **once per round on the central server**. Crucially, its cost is **independent of the number of clients** in the federated network. Therefore, the overhead remains constant and negligible as the system scales, posing no practical bottleneck.\n\n---\n\nOnce again, we thank the reviewer for their valuable and detailed feedback. We believe these clarifications and new experiments, which will be integrated into the final version, substantially strengthen our paper's contribution and address all raised concerns."}, {"review_id": "M1RgFeKxMe", "reviewer": "Reviewer_wsEs", "summary": "This work introduces a layer-wise fine-tuning strategy to mitigate the shortcomings of PEFT methods relative to full-parameter tuning in FL contexts with non-IID data, and enables clients to dynamically adjust their local training workload according to their available resources. The research problem is meaningful. However, this manuscript needs improvement in both the discussion of existing studies and the experimental setup.", "questions": "Please refer to Weaknesses.", "limitations": "Limitations provided in Conclusion. It is recommended to have a separated Section for these contents.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The research problem is meaningful for FL communities.\n2. This work proposes the theoretical analysis on convergence. \n### Weaknesses\n1. Figure 1 provides limited information. Given the space constraints of a top-tier conference like NeurIPS, it is expected that all figures present substantial and meaningful content. \n2. The current comparison methods are outdated. It is advisable to incorporate recent studies published within the last two years as comparative baselines, such as [1], [2], and [3].\n3. This study lacks a sufficient review of the existing literature. For example, since PEFT has limitations under non-IID settings compared to full-parameter tuning, it would be natural to consider prior attempts at applying full-parameter tuning in the context of FL, such as [3], [4] and [5]. Currently, the paper lacks discussion on this relevant line of work.\n4. This manuscript lacks an explanation of why the proposed layer-wise tuning method can address the limitations of PEFT methods compared to full-parameter tuning under non-IID conditions. In fact, layer-wise tuning appears to be a specific form of PEFT, since it similarly fine-tunes a subset of model parameters.\n5. The proposed approach relies on a proxy dataset, which is often difficult to obtain in real-world scenarios. The paper lacks an investigation into how the choice of proxy dataset affects the performance of the proposed method.\n6. The experiments were conducted with DeBERTaV3 and GPT-2. The scales of these models are too small by current standards. It is recommended to experiment with more recent models that have several billion parameters.\n7. It is suggested to provide convergence curves of the proposed methods together with the compared ones, to experimentally demonstrate the correctness of the theoretical results.\n\n[1] FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences, ATC 2024.\n\n[2] Federated fine-tuning of large language models under heterogeneous language tasks and client resources, NeurIPS 2024.\n\n[3] Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, ICML 2024.\n\n[4] Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models, arXiv 2409.\n\n[5] Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent, arXiv 2406.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns.", "rebuttal": "> *(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)*\n\nWe sincerely thank the reviewer for their thorough and constructive feedback. Your suggestions have been instrumental in helping us strengthen our paper's evaluation, clarify its core contributions, and improve its structure. We have grouped our responses to address the main themes of your review.\n\n---\n\n# (W.2, W.3, W.6) On SOTA Models, Baselines, and Literature Review\n\nYour feedback regarding the need for more recent baselines, SOTA models, and a broader literature review (**W.2, W.3, W.6**) is well-taken. We agree that this is essential for contextualizing our work.\n\nTo address this directly, we conducted **new experiments on Llama-3.1-8B with the MMLU benchmark**, including **FLoRA (NeurIPS 2024) [1] and FlexLoRA (NeurIPS 2024) [2]** as representative SOTA baselines. The results are below:\n\n| Method                  | MMLU (5-shot Avg.) | Peak GPU Memory (Client) |\n| ----------------------- | ------------------ | ------------------------ |\n| FlexLoRA (NeurIPS 2024) | 55.7               | 46.868 GB                |\n| FLoRA (NeurIPS 2024)    | 55.2               | 46.868 GB                |\n| **LEFF (Ours)**         | **57.5**           | **29.881 GB**            |\n\nThe results show LEFF **outperforms these recent NeurIPS 2024 baselines by 1.8-2.3 points** while **reducing client peak memory by over 36%**. This demonstrates that LEFF's principles are not only scalable but highly effective for modern, billion-parameter models.\n\nFurthermore, we have studied the suggested papers on federated full-parameter tuning ([3], [4], [5]) and will **add a detailed discussion to our Related Work section**. We will clarify LEFF's novelty by contrasting its approach:\n\n- Unlike ZOO-based methods [3] that approximate gradients, LEFF utilizes standard first-order optimization (backpropagation). This approach is significantly more computationally efficient on the client side, avoiding the high costs of multiple forward passes per step and typically leading to faster convergence.\n- In contrast to update-compression methods [4] which compute a full gradient update before compressing it, LEFF restricts the update scope a priori to a subset of layers. This design directly reduces the client's peak memory and computational load *during training* by confining backpropagation, rather than only saving communication bandwidth post-computation.\n- While cyclical methods [5] update model blocks in a sequential turn-based manner, LEFF employs a standard synchronous and parallel training protocol common in FL. Moreover, our dynamic, importance-based layer sampling enables more adaptive training by focusing resources on the most impactful parameters each round, instead of following a static, pre-determined cycle.\n\n[1] FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations, NeurIPS 2024.\n\n[2] Federated fine-tuning of large language models under heterogeneous language tasks and client resources, NeurIPS 2024.\n\n[3] Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, ICML 2024.\n\n[4] Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models, arXiv 2409.\n\n[5] Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent, arXiv 2406.\n\n---\n\n# (W.4) Clarifying LEFF's Advantage Over Conventional PEFT\n\nWe appreciate your insightful question (**W.4**) on why LEFF, as a form of PEFT, overcomes the limitations of other PEFT methods under non-IID conditions. The key distinction lies in the **nature and capacity of the parameters being tuned**.\n\n1. **Conventional PEFT** (e.g., LoRA) freezes the original model and injects a small number of **new, low-rank auxiliary parameters**. This fundamentally constrains model adaptation to a low-dimensional space, which is insufficient to capture the diverse data distributions across clients in non-IID settings [1].\n2. **LEFF**, in contrast, tunes the **original, full-rank parameters of entire layers**. While each client trains a subset of layers, these updates have the full expressive power of full-parameter tuning for those layers. This high-capacity adaptation is crucial for fitting the complex, client-specific data. From the global model's perspective, all parameters are gradually fine-tuned over the course of training. This principle is supported by recent findings in centralized tuning (e.g., LISA [2]), where layer-wise methods have also demonstrated superior performance over LoRA-style tuning.\n\nThe following table for DeBERTaV3-Large highlights this difference. LEFF tunes significantly more parameters than other PEFT methods, granting it greater expressive power, yet it achieves the lowest peak memory by only loading the necessary layers into GPU memory.\n\n|                          | FedAvg | FedLoRA | FedBitFit | **LEFF (Ours)** | SLoRA |\n| :----------------------- | :----- | :------ | :-------- | :-------------- | :---- |\n| Client Trainable Params  | 303M   | 3.5M    | 272K      | **13.6M**       | 3.5M  |\n| Client Peak GPU Mem (GB) | 9.361  | 6.660   | 5.488     | **3.005**       | 6.660 |\n\nThis design explains why LEFF's performance under high data heterogeneity is substantially better than conventional PEFT methods, as shown empirically in our Table 1 and Figure 5. We will add this clarification to the introduction.\n\n[1] SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models, NeurIPS 2023.\n\n[2] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS 2024.\n\n---\n\n# (W.5, W.7) Empirical Validation: Proxy Dataset and Convergence\n\nWe have conducted new analyses to address your concerns about the proxy dataset (**W.5**) and convergence curves (**W.7**).\n\n**Proxy Dataset Robustness:** To empirically validate our method's robustness, we conducted a **new ablation study** on the E2E NLG task by varying the proxy dataset.\n\n1. **Robustness to Data Distribution:** Our method is robust to the choice of proxy data because we distill **functional representations**, not task-specific knowledge. As described in Section 3.3, our objective is to match the intermediate hidden states and attention matrices. This process preserves general representational capabilities and is far less sensitive to the specific domain of the input data.\n2. **Practical Availability:** Since our method only requires unlabeled text, readily available public corpora (e.g., C4, Wikipedia) are sufficient. In scenarios where external data is prohibited, the server can use the global model to **generate a synthetic proxy dataset**, making the framework entirely self-contained — a promising direction we are exploring.\n3. **New Ablation Study:** We empirically demonstrate LEFF’s robustness to the proxy dataset via a new ablation study on the E2E NLG task. As shown in the table, performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose WikiText-103 and OpenWebText. While the in-domain data yields a slight, expected performance gain, the minimal fluctuation across all metrics confirms that LEFF is not reliant on a perfectly-matched proxy corpus. This high degree of robustness validates its practical applicability. We will include this study in the appendix.\n\n| Proxy Dataset ($\\mathcal{D}_{\\text{proxy}}$​) | BLEU   | NIST   | METEOR | ROUGE | CIDEr  |\n| --------------------------------------------- | ------ | ------ | ------ | ------- | ------ |\n| WikiText-103                                  | 0.5765 | 8.0012 | 0.4041 | 0.6310  | 1.7450 |\n| WebNLG                                        | 0.5799 | 8.0296 | 0.4064 | 0.6346  | 1.7521 |\n| OpenWebText                                   | 0.5712 | 7.9688 | 0.4015 | 0.6259  | 1.7345 |\n\n**Convergence Analysis:** We agree that showing convergence is vital. Due to format restrictions, we present the validation loss data below (under high heterogeneity, $\\alpha=0.05$) and will include the full plot in the final paper.\n\n| **Round** | **FedAvg** | **FedBitFit** | **FedLoRA** | **SLoRA** | **LEFF (Ours)** |\n| --------- | ---------- | ------------- | ----------- | --------- | --------------- |\n| 1         | 2.3732     | 2.2598        | 2.6286      | 2.5209    | 2.4223          |\n| 5         | 1.1091     | 1.1118        | 1.6372      | 1.3789    | 1.2227          |\n| 10        | 0.6797     | 1.0854        | 1.1147      | 0.9168    | 0.7670          |\n| 15        | 0.6019     | 1.1189        | 0.9427      | 0.8596    | 0.6416          |\n| 20        | 0.5962     | 1.0840        | 0.9576      | 0.8317    | 0.6515          |\n\nThis data empirically validates our theory (Thm. 1), showing that **LEFF converges stably and efficiently**, achieving a final loss much closer to full fine-tuning than other PEFT methods.\n\n---\n\n# (W.1, L.1) Paper Structure and Presentation\n\nFinally, we appreciate your suggestions for improving the paper's presentation (**W.1, L.1**).\n\n- We agree with your assessment of Figure 1. We will **remove it** and reallocate the space to our new experimental results, which provide stronger empirical evidence.\n- Following your advice, we will create a dedicated **\"Section 6: Limitations and Future Work\"** before the conclusion to improve clarity and transparency.\n\n---\n\nWe are grateful for your thorough review. We believe these extensive new experiments, clarifications, and structural changes directly address all your concerns and significantly strengthen the paper."}, {"review_id": "bpbXDcvlJi", "reviewer": "Reviewer_ctsn", "summary": "LEFF presents a novel layer-wise federated fine-tuning approach for LLMs at the edge, effectively addressing computational/data heterogeneity. The work is technically sound with strong empirical/theoretical contributions, though minor revisions would enhance reproducibility.", "questions": "1. Could you add standard deviations/confidence intervals to demonstrate result stability?\n2. What are the measured client-side trainable parameters and GPU memory costs compared to FedAvg?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- Innovative Methodology: Layer-wise adaptation with importance sampling and distillation-based compression elegantly solves straggler issues in heterogeneous FL.\n- Theoretical Rigor: $O(1/\\sqrt{T})$ convergence proof under standard FL assumptions (Theorem 1) is a key contribution.\n- Comprehensive Validation: Extensive tests on GLUE/E2E NLG across models (DeBERTaV3, GPT-2), heterogeneity levels (α=0.05-50), and client scales (8-40 devices) demonstrate superiority over baselines.\n- Practical Relevance: Reduces client compute while maintaining near-full-tuning accuracy – critical for edge deployment.\n\n**Weaknesses:**\n- Statistical Reporting: Lack of error bars/standard deviations (Tables 1-3) weakens empirical claims.\n- Overhead Analysis: Client computation costs and GPU memory pressure are unquantified.\n- Societal Impacts: No discussion of environmental costs from server operations or dual-use risks of efficient edge fine-tuning.", "was_revised": true, "final_justification": "Thanks for the detailed response, which has fully resolved my concerns. Although other reviewers rightly raised concerns about the evaluation scope (e.g., models and baselines) and proxy dataset, I believe the authors’ new SOTA experiments and robustness analysis have convincingly addressed these points. Given the paper’s significant technical innovation and now-strengthened validation, I will raise my score by +1 and recommend acceptance.", "rebuttal": "> *(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)*\n\nWe sincerely thank the reviewer for their valuable and constructive feedback. Your comments have helped us to significantly improve the empirical rigor and ethical considerations of our paper. We have grouped our responses to address your main points regarding empirical validation and societal impact.\n\n---\n\n# (W.1, W.2, Q.1, Q.2) On Empirical Rigor: Result Stability and Client-Side Costs\n\nA key theme of your feedback (**W.1, W.2, Q.1, Q.2**) was the need for more rigorous quantification of our results' stability and the client-side overhead. We agree this is crucial and have conducted detailed new analyses to address these points.\n\n**1. Statistical Stability of Results (W.1, Q.1)**\n\nTo demonstrate the stability of our findings, we have **re-run all experiments three times with different random seeds** and calculated the standard deviations for all metrics. The results confirm that our method is stable and the performance gains are consistent.\n\nDue to space constraints, we present a representative table of standard deviations below and will include the complete set of tables in a new Appendix. The key finding is that the **standard deviations are consistently small** across all tasks and methods, confirming the reliability of our claims. Our method, LEFF, shows stability comparable to or better than the baselines.\n\n| Model     | CoLA | SST-2 | MRPC | STS-B | QQP  | MNLI | QNLI | RTE  | BLEU   | NIST   | METEOR | ROUGE  | CIDEr  |\n| :-------- | :--- | :---- | :--- | :---- | :--- | :--- | :--- | :--- | :----- | :----- | :----- | :----- | :----- |\n| FedAvg    | 1.52 | 0.71  | 0.88 | 0.45  | 0.31 | 0.53 | 0.62 | 1.21 | 0.0041 | 0.1512 | 0.0028 | 0.0033 | 0.0805 |\n| FedBitFit | 1.89 | 0.95  | 1.02 | 0.82  | 0.55 | 0.98 | 0.77 | 1.53 | 0.0053 | 0.2144 | 0.0045 | 0.0049 | 0.1231 |\n| FedLoRA   | 1.72 | 0.88  | 1.15 | 0.75  | 0.49 | 1.05 | 0.81 | 1.68 | 0.0058 | 0.2516 | 0.0041 | 0.0051 | 0.1189 |\n| SLoRA     | 1.45 | 0.81  | 0.95 | 0.68  | 0.42 | 0.85 | 0.71 | 1.35 | 0.0049 | 0.1832 | 0.0035 | 0.0039 | 0.0948 |\n| LEFF      | 1.38 | 0.75  | 0.91 | 0.51  | 0.35 | 0.62 | 0.68 | 1.25 | 0.0045 | 0.1604 | 0.0031 | 0.0035 | 0.0882 |\n\n\n**2. Client-Side Overhead Analysis (W.2, Q.2)**\n\nWe have performed a detailed analysis of client-side trainable parameters and peak GPU memory usage across various models. The results highlight LEFF's unique and superior efficiency profile.\n\n| Model           | Algorithm | Trainable Params | Peak Memory (GB) |\n| :-------------- | :-------- | :--------------- | :--------------- |\n| DeBERTaV3-Base  | FedAvg    | 85,648,130       | 3.841            |\n| DeBERTaV3-Base  | FedLoRA   | 1,340,930        | 2.644            |\n| DeBERTaV3-Base  | FedBitFit | 102,914          | 2.198            |\n| DeBERTaV3-Base  | LEFF      | 7,681,538        | 2.136            |\n| DeBERTaV3-Base  | SLoRA     | 1,340,930        | 2.644            |\n| DeBERTaV3-Large | FedAvg    | 303,363,074      | 9.361            |\n| DeBERTaV3-Large | FedLoRA   | 3,557,378        | 6.660            |\n| DeBERTaV3-Large | FedBitFit | 272,386          | 5.488            |\n| DeBERTaV3-Large | LEFF      | 13,649,922       | 3.005            |\n| DeBERTaV3-Large | SLoRA     | 3,557,378        | 6.660            |\n| GPT2            | FedAvg    | 85,056,000       | 3.358            |\n| GPT2            | FedLoRA   | 811,008          | 2.476            |\n| GPT2            | FedBitFit | 102,144          | 2.206            |\n| GPT2            | LEFF      | 7,089,408        | 1.719            |\n| GPT2            | SLoRA     | 811,008          | 2.476            |\n| GPT2-Large      | FedAvg    | 708,390,400      | 15.548           |\n| GPT2-Large      | FedLoRA   | 4,055,040        | 9.739            |\n| GPT2-Large      | FedBitFit | 508,160          | 8.318            |\n| GPT2-Large      | LEFF      | 19,680,000       | 3.240            |\n| GPT2-Large      | SLoRA     | 4,055,040        | 9.739            |\n| Llama-3.1-8B    | FedAvg    | OOM              | OOM              |\n| Llama-3.1-8B    | FedLoRA   | 20,971,520       | 46.868           |\n| Llama-3.1-8B    | FedBitFit | No Bias          | No Bias          |\n| Llama-3.1-8B    | LEFF      | 743,452,672      | 29.881           |\n| Llama-3.1-8B    | SLoRA     | 20,971,520       | 46.868           |\n\nOur analysis reveals two key insights:\n\n- **Lowest GPU Memory Pressure:** LEFF drastically reduces peak GPU memory—for instance, achieving a **79% reduction on GPT2-Large compared to FedAvg and 66% compared to FedLoRA**. This is possible because LEFF's architecture selectively loads only the necessary layers into GPU memory, whereas other PEFT methods must load the entire frozen model, resulting in much higher memory footprints.\n- **Optimal Performance-Efficiency Trade-off:** LEFF deliberately retains more trainable parameters than methods like FedLoRA. This maintains greater model expressiveness, leading to superior task performance. Thus, LEFF strikes an optimal balance: it achieves the **lowest memory cost** for deployment on constrained devices while preserving enough capacity for **high-accuracy fine-tuning**.\n\nThese quantitative results confirm LEFF's superior design for practical, resource-constrained FL. We will integrate these findings into a new appendix.\n\n---\n\n# (W.3) On Societal Impacts\n\nWe thank you for raising the important ethical considerations of environmental cost and dual-use risks (**W.3**). We will add a dedicated section to discuss these.\n\n- **Environmental Cost:** We acknowledge the server-side computation in LEFF is a necessary trade-off. However, by enabling efficient fine-tuning on numerous distributed, low-power edge devices, our approach reduces the barrier to entry and can lessen the overall system's reliance on large, consistently energy-intensive data centers for training.\n- **Dual-Use Risks:** We recognize this concern. A key strength of LEFF's federated architecture is that the **central server** acts as a natural control point. This allows for the implementation of safeguards like client vetting, anomaly detection, and monitoring model updates to mitigate misuse—a governance feature absent in purely decentralized or local fine-tuning scenarios.\n\n---\n\nWe hope these detailed analyses and clarifications fully address your concerns. We are confident that incorporating these additions will significantly strengthen the paper."}, {"review_id": "fvohqBNolx", "reviewer": "Reviewer_dWpb", "summary": "This paper introduces Layer-wise Efficient Federated Fine-tuning. The core idea is to let each client fine-tune only a subset of the LLM's layers, with the number of layers chosen based on the client's computational capacity. To optimize this process, the server uses an importance-based sampling strategy to assign the most impactful layers to clients. For the layers that a client doe snot train, the server creates a compressed version using knowledge distillation. Experiments are extensive and results are promising.", "questions": "- Theoretical anaysis mainly state that the performance floor depends on how good the compression works, right?\n- How does the comp. cost at server scale as the number of clients increase above 40 (which is a low number for practcal settings)?\n- How does the proxy data used in knowledge distillation impact performance?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea of dynamically allocating layers to clients for finetuning has been explored before but compressing the rest is novel as far as I know.\n- Well-motivated and interesting problem.\n- Experiments are extensive and datasets/methods have variety, supporting the validation of the proposed approach.\n- Theoretical analysis is provided.\n\nWeaknesses:\n- Distillation to create compressed layers for each client every round seems computationally expensive.\n- Experiment results lack client-side efficiency metrics even though paper claims efficiency.\n- How the dataset used in knowledge distillation is constructed is critical and may not be practical as ideally it should be representative of client's dataset.", "was_revised": true, "final_justification": "My concerns/questions about the efficiency on server and client-side are addressed. Additional experimental results provided with more extensive analysis based on other reviewers' feedback are promising. I increase my recommendation from 4 to 5.", "rebuttal": "> *(For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and L.x (Limitations).)*\n\nWe thank the reviewer for their insightful questions, which have helped us to clarify the efficiency, scalability, and theoretical underpinnings of our work. We have grouped our responses thematically.\n\n---\n\n# (W.2) On Client-Side Efficiency: New Quantitative Results\n\nYou rightly pointed out the need for quantitative client-side efficiency metrics to substantiate our claims (**W.2**). To address this, we have conducted new experiments measuring peak GPU memory and the number of trainable parameters. The results below demonstrate LEFF's significant advantages.\n\n| Model           | Algorithm | Trainable Params | Peak Memory (GB) |\n| :-------------- | :-------- | :--------------- | :--------------- |\n| DeBERTaV3-Base  | FedAvg    | 85,648,130       | 3.841            |\n| DeBERTaV3-Base  | FedLoRA   | 1,340,930        | 2.644            |\n| DeBERTaV3-Base  | FedBitFit | 102,914          | 2.198            |\n| DeBERTaV3-Base  | LEFF      | 7,681,538        | 2.136            |\n| DeBERTaV3-Base  | SLoRA     | 1,340,930        | 2.644            |\n| DeBERTaV3-Large | FedAvg    | 303,363,074      | 9.361            |\n| DeBERTaV3-Large | FedLoRA   | 3,557,378        | 6.660            |\n| DeBERTaV3-Large | FedBitFit | 272,386          | 5.488            |\n| DeBERTaV3-Large | LEFF      | 13,649,922       | 3.005            |\n| DeBERTaV3-Large | SLoRA     | 3,557,378        | 6.660            |\n| GPT2            | FedAvg    | 85,056,000       | 3.358            |\n| GPT2            | FedLoRA   | 811,008          | 2.476            |\n| GPT2            | FedBitFit | 102,144          | 2.206            |\n| GPT2            | LEFF      | 7,089,408        | 1.719            |\n| GPT2            | SLoRA     | 811,008          | 2.476            |\n| GPT2-Large      | FedAvg    | 708,390,400      | 15.548           |\n| GPT2-Large      | FedLoRA   | 4,055,040        | 9.739            |\n| GPT2-Large      | FedBitFit | 508,160          | 8.318            |\n| GPT2-Large      | LEFF      | 19,680,000       | 3.240            |\n| GPT2-Large      | SLoRA     | 4,055,040        | 9.739            |\n| Llama-3.1-8B    | FedAvg    | OOM              | OOM              |\n| Llama-3.1-8B    | FedLoRA   | 20,971,520       | 46.868           |\n| Llama-3.1-8B    | FedBitFit | No Bias          | No Bias          |\n| Llama-3.1-8B    | LEFF      | 743,452,672      | 29.881           |\n| Llama-3.1-8B    | SLoRA     | 20,971,520       | 46.868           |\n\nThis new analysis reveals two key insights:\n\n1. **Lowest Memory Footprint:** LEFF drastically reduces peak GPU memory—for instance, achieving a **79% reduction on GPT2-Large compared to FedAvg and 66% compared to FedLoRA**. This is possible because LEFF's architecture selectively loads only the necessary layers into GPU memory. In contrast, other PEFT methods must load the entire frozen model, resulting in much higher memory footprints.\n2. **Optimal Efficiency-Performance Trade-off:** While achieving the lowest memory cost, LEFF deliberately retains more trainable parameters than methods like FedLoRA. This maintains greater model expressiveness, leading to superior task performance. LEFF thus strikes an optimal balance, delivering the **minimal resource cost** for client deployment while preserving enough capacity for **high-accuracy fine-tuning**.\n\nWe will add this detailed analysis to the appendix of our paper.\n\n---\n\n# (W.1, Q.2) On Server-Side Cost and Scalability\n\nWe appreciate your important questions regarding server-side computational costs and scalability (**W.1, Q.2**). Our framework is designed to be highly efficient and scalable.\n\n**1. Low Per-Round Distillation Cost (for W.1)**\n\nThe server-side knowledge distillation is computationally inexpensive, as its total workload is minimized by three key factors:\n\n- **Amortized Cost via Caching:** The server does not perform a separate distillation for every client. Instead, if multiple clients request the compression of the **exact same set of layers**, the server performs this task only **once**. The result is then cached and distributed to all relevant clients, drastically reducing the total number of required distillation processes per round.\n- **Incremental Fine-tuning (Warm Start):** The process is not training from scratch. The distilled \"student\" model from round `t-1` serves as a warm start for round `t`. Since the global \"teacher\" model evolves gradually, the student only needs a **brief fine-tuning** to catch up.\n- **Small Target Model:** By definition, the distillation operates on a \"student\" model that is significantly smaller than the full model (controlled by `r`), making each individual training process inherently fast.\n\n**2. High Scalability with Increasing Clients (for Q.2)**\n\nThe server's computational overhead scales efficiently for large-scale deployments, as neither of its main tasks grows linearly with the number of clients:\n\n- **Layer Importance Calculation:** This is performed **only once per round**, requiring a single forward and backward pass on the global model. Its cost is therefore **constant** regardless of the number of clients (e.g., 100 or 10,000).\n- **Distillation Workload:** As explained above, the caching mechanism **decouples the distillation workload from the client count**. The total cost is tied to the number of _unique compression requests_, which is significantly smaller than the total number of clients in a large-scale setting. This ensures the distillation overhead scales sub-linearly, making the entire framework highly scalable.\n\n---\n\n# (W.3, Q.1, Q.3) On the Proxy Dataset and Theoretical Guarantees\n\nYour questions about the proxy dataset (**W.3, Q.3**) and the interpretation of our theory (**Q.1**) are spot-on. These two aspects are deeply connected.\n\n**1. Proxy Dataset Robustness:** To empirically address your concern, we ran a **new ablation study** on the E2E NLG task, showing that LEFF is highly robust to the choice of proxy data.\n\n1. **Robustness to Data Distribution:** Our method is robust to the choice of proxy data because we distill **functional representations**, not task-specific knowledge. As described in Section 3.3, our objective is to match the intermediate hidden states and attention matrices. This process preserves general representational capabilities and is far less sensitive to the specific domain of the input data.\n2. **Practical Availability:** Since our method only requires unlabeled text, readily available public corpora (e.g., C4, Wikipedia) are sufficient. In scenarios where external data is prohibited, the server can use the global model to **generate a synthetic proxy dataset**, making the framework entirely self-contained — a promising direction we are exploring.\n3. **New Ablation Study:** We empirically demonstrate LEFF’s robustness to the proxy dataset via a new ablation study on the E2E NLG task. As shown in the table, performance remains remarkably stable across diverse corpora, from the in-domain WebNLG to general-purpose WikiText-103 and OpenWebText. While the in-domain data yields a slight, expected performance gain, the minimal fluctuation across all metrics confirms that LEFF is not reliant on a perfectly-matched proxy corpus. This high degree of robustness validates its practical applicability. We will include this study in the appendix.\n\n| Proxy Dataset ($\\mathcal{D}_{\\text{proxy}}$​) | BLEU   | NIST   | METEOR | ROUGE | CIDEr  |\n| --------------------------------------------- | ------ | ------ | ------ | ------- | ------ |\n| WikiText-103                                  | 0.5765 | 8.0012 | 0.4041 | 0.6310  | 1.7450 |\n| WebNLG                                        | 0.5799 | 8.0296 | 0.4064 | 0.6346  | 1.7521 |\n| OpenWebText                                   | 0.5712 | 7.9688 | 0.4015 | 0.6259  | 1.7345 |\n\n**2. Theoretical Interpretation:** Your interpretation of our theory (**Q.1**) is entirely correct. Our convergence bound (Thm. 1) explicitly shows that the performance floor depends on the compression quality, captured by the approximation error term $\\bar{\\Delta}^2$. This dependency is not a weakness but the **core, deliberate trade-off** our work models and manages.\n\nOur contribution lies in making this trade-off explicit and demonstrating empirically (e.g., Fig. 6) that we can manage it to achieve a state-of-the-art balance between efficiency and performance. Our main results (Table 2) confirm that despite this inherent dependency, LEFF's approach is highly effective, outperforming other PEFT methods in challenging FL settings.\n\n---\n\nWe hope these detailed quantitative results and clarifications have fully addressed your concerns. Thank you again for helping us improve the paper."}]

Author rebuttal:

Meta-review: This paper introduces a framework named LEFF for efficiently fine-tuning LLMs in FL settings, especially on resource-constrained edge devices. The proposed framework allows each client to fine-tune only selected layers based on its computational capacity, and employs a layer-wise aggregation strategy to ensure effective model updates. Experimental results demonstrate that LEFF achieves performance comparable to full fine-tuning and outperforms existing methods under heterogeneous conditions.

Reviewers express appreciation for the formal convergence guarantees and the extensive experiments in this paper. The author responses have addressed most of the major concerns, including those about the evaluation scope and the use of proxy datasets. Most reviewers give this paper positive ratings in light of its technical innovation, rigorous theory, and well-written manuscript. It would be beneficial to include the results and analysis provided during the author-reviewer discussion period in the revised paper.

Overall, the recommendation is to accept this paper.
Decision: Accept (poster)