Developmental Federated Tuning: A Cognitive-Inspired Paradigm for Efficient LLM Adaptation
Authors: Yebo Wu
Jingguang Li
Zhijiang Guo
Li Li
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Fine-Tuning
Large Language Models
Efficient Training
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: In this paper, we introduce Developmental Federated Tuning (DevFT), a resource-efficient approach inspired by cognitive development that progressively builds a powerful LLM from a compact foundation.

Abstract: Federated fine-tuning enables Large Language Models (LLMs) to adapt to downstream tasks while preserving data privacy, but its resource-intensive nature severely limits deployment on edge devices. In this paper, we introduce Developmental Federated Tuning (DevFT), a resource-efficient approach inspired by cognitive development that progressively builds a powerful LLM from a compact foundation. DevFT decomposes the fine-tuning process into developmental stages, each optimizing a submodel with increasing parameter capacity. Knowledge acquired in earlier stages is transferred to subsequent submodels, providing optimized initialization parameters that prevent convergence to local minima and accelerate training. This paradigm mirrors human learning, gradually constructing a comprehensive knowledge structure while refining existing skills. To efficiently build stage-specific submodels, DevFT introduces deconfliction-guided layer grouping and differential-based layer fusion to distill essential information and construct representative layers. Evaluations across multiple benchmarks demonstrate that DevFT significantly outperforms state-of-the-art methods, achieving up to $\textbf{4.59$\times$}$ faster convergence, $\textbf{10.67$\times$}$ reduction in communication overhead, and $\textbf{9.07}$\% average performance improvement, while maintaining compatibility with existing federated fine-tuning approaches.

Published as a conference paper at ICLR 2026
DEVELOPMENTAL
FEDERATED TUNING: A COGNITIVE-INSPIREI PARADIGM FOR EFFICIENT
LLM ADAPTATION
Yebo Wu'x, Jingguang Li Zhijiang Guo Li Li University of Macau HKKST 3HKUST (Guangzhou)
{yc37926, mc45005, 11ili]@um edu.mo, zhijiangguoehkust-tee edu. cn
ABSTRACT
Federated fine-tuning enables Large Language Models (LLMs) adapt down- stream tasks while preserving data privacy, but resource-intensive nature severcly limits deployment edge devices. In this paper, introduce Devel- opmental Federated Tuning (DEVFT), a resource-efficintt approach inspired by cognitive development that progressively builds powerful LLM from pact foundation. DEVFT decomposes the fine-tuning process into developmental stages, each optimizing submodel with increasing parameter capacity. Knowl- edge acquired in earlier stages is transferred to subsequent submodels, provid- ing optimized initialization parameters that prevent convergence local min- ima and accelerate training. This paradigm mirrors human learning, gradually constructing comprehensive knowledge structure while refining existing skills. To efficiently build stage-specifie submodels, DEVFT introduces deconfliction- guided layer grouping and differential-base layer fusion to distill essential in- formation and construct representative layers. Evaluations across multiple bench- marks demonstrate that DEVFT significantly outperforms state-of-the-an meth- ods, achieving up to 4.59x faster convergence, 10.67 reduction in communica- tion overhead, and 9.07 % average performance improvement, while maintaining compatibility with existing federated fine- fine-tuning approaches.
INTRODUCTION
Large Language Models (LLMs) exhibit exceptional capabilities across diverse domains Yuan ct al., 2024) Xu et al. 2025 2024b). While fine-tuning effectively adapts these models to specific tasks (Han et al. 2024). it demands substantial task-specific data. This data often resides privately edge devices, making centralized collection impractical. Federated fine-tuning (Zhang et al.) (0024)) offers privacy privacy-preserving alternative for collaborative adaptation. Nevertheless, deploying massive LLMs for federated fine-tuning on sisource-limited edge devices is challenging due to hardware and
communication constraints (Wu al. 2025a Tam al. 2024
Tian et al.
2024a 2022).
To address these challenges, researchers have pro-
415.2
posed various parametee-efffcieett federated fine-tuning ap-
400
TaylLsMA
HERT
proaches (Wu et al.) 2025c), with LoRA-based methods gar- nering significant attention due their efficiency and flex-
TFLOPs
300
LLaMMA2-18
1.1. 11.-MA--111 130
L12A563-411
223
ibility (Guo et al.) 2025) Tian et al.) 2024b). However, existing LoRA-based methods typically fine-tune LLMs end-to-end, which remains computationally prohibitive for edge devices compared to small language models such as BERT (Devlin et al.) 2019). Figure [ quantifies this gap by comparing the computational cost a single fine-tuning step across LLaMA (Touvron et al.) 2023) variants and BERT. Even the relatively compact TinyLLaMA (Zhang et al.
915.0
34.3
Figure 1: Computational overhead for one-step fine-tuning of different language models using LoRA.
"Equal Contribution. 1 "Corrsponndig Authors.

---

Published as a conference paper at ICLR 2026
Existing Works:
Ours:
Resource Efficiency Better Performance
Device Pool
Device Pool
High Resource
fDQ
Hard to Ophimize
ROUND:
STAGE:
Madel Copacity
Figure 2: Workflow comparison between existing works and DEVFT.
2024b) demands 9.3x more FLOPs than BERT, while LLaMA2-13B TTuvroo al. 2023) re- quires an overwhelming 415.2 TFLOPs, which is 112.2x that of BERT. Such substantial compu- tational requirements fundamentally challenge the practical deployment of federated fine-tuning on resource-constrained devices, even with current parameter-efficien techniques.
Inspired by human cognitive development (Bengio al., 2009 Sweller 2008) McArdle Wood- 2014), where learning unfolds progressively rather than instantaneously, we propose Develop- mental Federated Tuning (DEvFT), resource-efficient federated fine-tuning approach that allevi- ates computational burdens by gradually cultivating capable LLM from a compact foundation. As illustrated Figure 2 instead updating the full LLM throughout the entire federated fine-tuning process, DEvFT decomposes learning into sequence of developmental stages. Specifically, the learning journey begins with a compact submodel (analogous to a child), and upon mastering stage- specific competencies,
strategically expand the submodel capacity (mimicking human growth), while transferring the acquired knowledge to initialize the submodel of the next stage. This growth- and-transfer process repeats until the model reaches its target capacity (analogous to an adult).
This developmental paradigm, starting with compact models, offers several inherent advantages. Smaller models generally exhibit smoother loss landscapes, reducing the risk of convergence to poor local minima. Moreover, the knowledge distilled during early stages serves well-informed initialization for larger architectures, enhancing performance in subsequent stages. Compared to end-to-end L.M fine-tuning, DEVFT 's progressive capacity growth substantially accelerates fed- erated finettuning while lowering both computation and communication overheads. However, critical challenge lies in: How architect stage-specific submodels to ensure effective knowledge transfer across consecutive stages while optimizing overall performance?
To address this challenge, DEVFT introduces two novel techniques. The deconffiction-guidee layer grouping mechanism initially clusters layers based on parameter similarity, thereby grouping layers with minimal parameter conflicts together. Subsequently, the differential-based layer fusion strat- egy strategically distills and integrates the distinctive semantic information each layer through arithmetic operations, yielding representative layer for each group that encapsulates the group's collective knowledge and core functionality. These representative layers are then concatenated se- quentially construct the stage-specific submodel for federated fine-tuning. Due the functional homogeneity within groups, layers can directly inherit knowledge from their corresponding repre- sentative layers, thereby facilitating seamless knowledge transfer across developmental stages.
In order to empirically validate the effectiveness of DEVFT and its advantages, we conduct extensive experiments on multiple benchmarks. DEvFT significantly outperforms state-of-the-an methods, achieving up to 4.59x faster convergence, 10.67 x reduction in communication overhead, and 9.07% average performance improvement, while maintaining compatibility with existing approaches.
BACKGROUND AND MOTIVATION
2.1 EXISTING MARAMETER-EFTICHNN FEDERATED FINE-TUNING
Parameter-efffcient federated fine-tuning presents compelling strategy to mitigate resource de- mands in distributed learning by freezing most pre-trained model parameters and updating only a
small, task-specific subset (Wu al. 2025c). These methods generally fall into the following cate- gories. Prompt-basee techniques (Guo et al. 2023) Yang et al. 2023 [u et al. (024) utilize carefully
designed soft prompts guide model behavior without altering the pre-trained weights. Adapter-
2

---

Published as a conference paper at ICLR 2026
Step Stage Submedel Construction
Step Collaberative Optimization
Step Knowledge
Local Free
Laud FIL
Update the glozal
Deconfliction guiidd Loyer Grouping
Differential- besed Layer Fusion
Parc
Serves as the foundation for building next stage submedel
Tea-furing
Figure 3: Overview of DEVFT: The server first constructs the stage-specific submodel (step D), followed by collaborative optimization across edge devices (step @). After each stage. the acquired knowledge is employed to update the global model, which serves as the foundation for building the subsequent stage submodel (step ()).
based methods (Cai et al. 2022 Liu al. 2023 Li et al.
2022) insert lightweight adapter layers into the network, allowing for task adaptation with minimal modifications. Notably, LoR A-based approaches have garnered significant interest due to their effectiveness (Guo al. 2025).
2025b) introduce low-rank adaptations to weight updates, effectively preserving the expressiveness of the original model while significantly reducing the number of trainable parameters. To accommodate heterogeneous
LoRA-based methods (Wang et al. 2024 Sun et al. 2024 Wu et al.
sources, approaches like HETLoRA (Cho et al. 2024) and FlexLoRA (Bai et al. 2024) assign
varying LoRA ranks different devices. Fed-pilo (Zhang et al. 2024c) and Fed HeLLo (Zhang et al. 2025) optimize LoRA allocation through layer contribution quantification and resource -wsorcce-wware importance scoring. Moreover, FwdLLM (Xu al. 2023) and FedKSeed (Qin et al.) (023) employ zeroth-order optimization to mitigate resource consumption. Furthermore, FeDeRA Yann et al.) 2024) addresses data heterogeneity by initializing LoRA via singular value decomposition on pre- trained parameters. While these methods have shown promise in their respective domains, they often do fully tackle the fundamental issue of the substantial computational requirements imposed by end-to-end LLM fine-tuning. This persistent challenge motivates our proposed approach.
2.2 MOTIVATION FOR DEVELOPMENTAL FEDERATED TUNING
To address the persistent challenge of substantial computational burdens, propose different paradigm. Unlike existing works that update the LLM in an end-to-end manner, which can be resource-intensive, our approach progressively builds a capable model from a compact foundation. Drawing inspiration from human cognitive development (Bengio et al. 2009 Sweller 2008). where learning progresses incrementally rather than instantancously, we aim mitigate these computa- tional burdens by progressively cultivating more capable LLM from compact foundation.
Specifically, we decompose the fine-tume fine-tuning process into stages, each with multiple rounds, mim- icking different periods in human learning. The submodel capacity (i.e., the number of layers) at each stage is denoted as {Li, ; L2, .... ., Ls). forming a strictly monotonically increasing sequence where La, < L.,, for any 81 < 82. The final stage capacity Ls equals L, encompassing all layers of the LLM. Additionally, the knowledge acquired in each stage seamlessly transfers to the submodel of the subsequent stage, providing optimized initialization parameters. Compared to end-to-end fine- tuning, this developmental paradigm significantly reduces resource overhead for edge devices while achieving superior performance through smoother optimization trajectory. In this way, DEVFT enables participating devices to efficiently fine-tune an L-layer LLM for downstream tasks.
DEVELOPMENTAL FEDERATED TUNING (DEVFT)
3.1 OVERVIEW
Figure illlssrrate the overall framework of DEVFT, which proceeds through three key steps.
3

---

Published as a conference paper at ICLR 2026
Jj+ = + 8,
J 0, 0,
Layer
I Layer
0: (b) Layer Addition.
0
Layer
(a) Layer Vectors.
(c) Layer Subtraction.
Figure 4: An illustration of layer vectors and layer arithmetic operations.
Step @: Stage Submodel Construction. Prior to the commencement of each stage, the server constructs a stage-specific submodel. Specifically, the server utilizes the deconfliction-guidee layer grouping (DGLG) mechanism (Section 3.2 to cluster layers exhibiting minimal parameter con- flicts. Subsequently. the differential based layer fusion (DBLF) strategy (Section 3.3) is applied to integrate srrap information, generating a representative layer for each group. Finally, these representative layers are concatenated sequentially to assemble the stage-specific submodel.
Step @: Collaborative Optimization. Once the submodel is constructed, the federated fine- tuning process commences, wherein devices collaboratively train the submodel on their local data.
Upon completion of the current stage, the acquired knowledge is synchronized to update the global model and is seamlessly transferred to initialize the submodel for the subsequent stage (Section 3.4). This progressive model training process continues until the completion of the S-th stage.
Step 3: Knowledge Transfer.
3.2 DECONFLICTION-
LAYER GROUPING
As shown in Figure parameters of each layer can be represented as corresponding layer vectors, with varying degrees of parameter conflict across layers. When constructing representative layers for each group, strong parameter conflicts between layers can lead to severe information loss, as parameters with opposite signs may neutralize each other's unique contributions during the layer fusion process. This cancellation results in low-fidelity representative layer that fails to capture the distinct functions of the original layers. Therefore, ensuring high intra-group similarity is critical to minimize this loss and preserve representational fidelity. To achieve this, we propose deconfliction- guided layer grouping (DGLG) mechanism that clusters layers with minimal parameter conflicts into the same group to preserve their respective knowledge. Specifically, the server initially calculates inter-layer parameter similarity using Equation
(0,, 0z)
sim, 0,) ||0:1| ||,
where 0 and fy denote the parameters of layers and j. respectively, including their associated LoRA parameters. This computation yields layer similarity matrix W. where each entry wiy mea- sures the parameter similarity between layers and j. Higher similarity values indicate lower param- eter conflicts, suggesting these layers should be grouped together. Conversely, lower similarity val- ues signify more severe parameter confficts, necessitating the assignment of these layers to different groups. Based on the similarity matrix W, we construct a complete undirected graph = -V, E), where V {e1, 12, .... ) represents the set of layers and & = {sim(e,, 2y) lu, Uy V. Waj wxy denotes the set edges weighted by layer similarities. The objective is to partition graph G into L, non-overlapping groups (i for stage s, which can be formally expressed as:
Ixx
min (21182--
Σ
Σ cut(g gm), where cut(g gm)
Σ
Σ
Wpys
n magin
PEBe gEgm
s.t. Vm, n (1,2, m.
0gn 0 and
U = V.
To solve the optimization problem in Equation 2 diag(d1..... dd.), where d, Σド= my
we first construct the degree matrix D Wi denotes the of weights connected to vertex uy. We

---

Published as a conference paper at ICLR 2026
then compute the Laplacian matrix L D W and perform eigenvalue decomposition L to obtain the eigenvectors corresponding to the Lg smallest eigenvalues. These eigenvectors are stacked column by column to form the embedding matrix E E R Finally, k-means clustering applied to E to partition the vertex set into I, disjoint groups. This process can be formulated as:
(gi,..., gL, } k-means (E, L.). E -Vi,..... V
where D W. D diag
W1
Σ WLi
(3)
Lv = Aeve, VI (I, L.). S. A A S... AL::
where d and V, represent the 1-th eigenvalue and corresponding eigenvector of L. Through this deconffiction-guidee layer grouping mechanism, we can partition the L layers of the global model into L. groups (gn) where layers within each group exhibit minimal parameter conflicts.
3.3 DIFFERENTIAL BASSED LAYER FUSION
After obtaining the partitioned groups, construct representative layer for each group. To effec- tively synthesize these representative layers, we introduce the differential-based layer fusion (DBLF) strategy, which integrates layer information within each group through well-defined arithmetic oper- ations. As illustrated in Figure (b), the layer addition operation merges knowledge from two layers, producing a composite layer that encapsulates the semantic information of both source components. Figure Ce) shows the layer subtraction operation, which distills the unique information present in one layer relative to another. For any given layers and j. these operations are defined as follows:
Tj+i 0g + 0;: Ty-1 0g 0,.
(4)
where Ty+; and Tj denote the resulting parameter vectors after addition and subtraction operations, respectively. These operations enable fine-grained knowledge editing in the parameter space. A naive approach for intra-group information integration involves performing the addition operation on all layers. However, this introduces significant information redundancy, layers within the same group typically share similar functional characteristics. This redundancy limits the submodel's capability to capture diverse and meaningful representations.
To address this challenge, instead of indiscriminately merging all information, DBLF selectively integrates the unique semantic information of each layer. Specifically, it designates the first layer of each group as the anchor layer and computes the information differentials of other layers relative to this anchor layer through the layer subtraction operation. During layer fusion, only the informa- tion differentials are encapsulated into the anchor layer, thereby effectively preserving each layer's essential information while eliminating redundancy. This fusion process can be formulated as:
gB Oanchor + 3 Σ 1003-0amche Oanchor).
(5)
jERa
where /3 is a weighting factor, Oanchor represents the parameters of the anchor layer, and 0 stands for the representative layer of group ga, encapsulating the distinctive features of all layers within the group. These derived representative layers are then concatenated sequentially to construct the stage-specific submodel for federated fine-tuning.
3.4 KNOWLEDGE TRANSFER
Cross-stage knowledge transfer is critical for cultivating high-performance LLMs, analogous to hu- man cognition where new knowledge builds upon established foundations. At each stage, the ac- quired knowledge provides optimized initialization for the next-stage submodel, thereby accelerat- ing convergence and improving overall performance by avoiding poor local minima. Through strate- gie layer clustering and representative layer construction, the encoded knowledge in {0 h== can be directly utilized to update all layers within their respective groups {gr (step Figure The rationale is that functionally similar layers inherently exhibit similar parameter distributions and learning patterns. Notably, we only update the LoRA parameters of each layer. This transfer process yields an updated global model that serves the foundation for constructing the next-stage submodel, ensuring seamless knowledge inheritance across stages. A practical implementation ex- ample is detailed in Appendix A
5

---

Published as a conference paper at ICLR 2026
Performance evaluation of DEVFT against baseline methods on instruction tuning tasks. Bold and underlined values denote the best and second-best results, respectively.
Table 1:
Close-Ended Benchmark
Open-Ende Benchmark f
Method
TruthfulQA
MMLU
1FEval
BBH
Average
Vicuna
MT-1
MT-2
Average
LLaMA2-7B (INT4) (Tourron al.
2023
FedIT DoFIT
47.57
42.45
31.76
39.28
40.27
8.18
4.77
1.98
4.98
48.32
43.04
32.62
39.59
40.89
8.19
4.92
2.13
5.08
C2A
46.71
41.83
29.45
36.07
38.52
7.66
3.97
1.88
4.50
ProgFed
48.60
43.14
32.54
39.73
41.00
8.20
4.88
2.19
5.09
FLORA
47.76
42.64
32.08
39.25
40.43
8.21
4.85
2.02
5.03
FedSA-LoRA
48.24
42.91
32.71
39.36
40.81
8.26
5.09
2.31
5.22
DEVFT
50.28
44.15
33.97
40.93
42.33
8.41
5.76
2.92
5,70
LLaMA3.1 (INT4)
(Grattafiori et al. 2024
FedIT
48.07
63.31
47.32
62.69
55.35
8.89
6.54
5.03
6.82
DoFIT
49.12
65.17
51.66
65.21
57.79
9.01
6.72
5.22
6.98
C2A
48.99
63.76
46.10
61.85
55.18
8.74
6.67
4.98
6.80
ProgFed
53.12
66.77
54,55
66.03
60.12
9.07
6,85
5,08
7,00
FLoRA
50.23
64.95
50.47
64.93
57.65
8.96
6.75
5.11
6.94
FedSA-LoRA
53.29
66.87
56.17
67.56
60.97
9.03
6.92
7.12
DEVFT
55.23
68.42
62.29
71.04
64.25
9.18
7.63
6.57
7.79
LLaMA2-13B (INT4) (Touvron et al. 2023
FedIT
52.40
55.45
40.33
46.14
48.58
8.37
5.17
3.01
5.52
DoFIT
54.77
56.09
41.68
46.41
49.74
8.37
5.19
3.34
5,63
C2A
53.91
54.33
38.96
45.06
48.07
8.05
5.08
3.26
5.46
ProgFed
55.01
S7.38
42.13
46.36
50.22
8.38
5.28
3.07
5.58
FLoRA
54.26
56.23
41.49
46.32
49.58
8.40
5.22
3.15
5.59
RodSA-LoRt
55.73
57.51
43.21
46.91
50.84
8.49
5.39
3.45
5.78
DEVFT
57.19
58.74
46.45
48.70
52.77
8.67
6.18
4.52
6.46
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models and Datasets. Following OpenFedLLM (Ye et 2024). we evaluate DEVFT on three LLaMA-baser models: LLaMA2-7B (Touvron et al. 2023). LLaMA3.1-8B (Grattafiori et al.) 2024), and LLaMA2-15 13MB.BB...B... All models are fine-tuned on the Apaca-GPTTS dataset (Peng et al. 2023). and are evaluated on both dose-ended and open-ended benchmarks. Specifically, the close-ended benchmarks include TruthfulQA (Lin et al. 2022), MMLU (Hendrycks et al.) 2020). IFEval (Zhou al. 2023), and BBH (Suzgun et al. (0022. which assess the models' capabilities in honesty and truthfulness, knowledge coverage, instruction following, and reasoning, respectively. The open- ended benchmarks, including Vicuna-Bench (Chiang et al.) 2023) and MT-Bench (Zheng et al. 2024), evaluate the models' performance in multi-turn dialogue scenarios.
Implementation Details. The fine-tuning process is divided into four stages (S 4) for all models, with each stage's submodel receiving an equal number of federated fine-tuning rounds. The capacity of the submodels doubles at each stage. Specifically, for LLaMA2-7B and LLaMA3311888 the submodel capacities across the four stages are {4, 8, 16. 32), whereas for LLaMA2-13B, they are {5, 10, 20, 40). We the hyperparameter to 0. for LLaMA2-7B and LLaMA3.1 and 0.15 for LLaMA2-133 A2-BBB. Additional implementation details are provided in Appendix
4.2 BASELINES
Resource- Aisource-Unwware Methods. FedIT [Zhang et al. 2024a) integrates LoRA with FedAvg to enable
federated instruction tuning. DoFIT (uu al. (024a) employs specialized LoRA initialization and aggregation strategies to mitigate catastrophic forgetting. C2A (Kim et al.) (023) addresses data heterogeneity through hypernetwork that dynamically generates device-specific adapters.
Resource-Aware Methods. ProgFed (Wang et al. 2022) partitions the global model into blocks and gradually incorporates them for training. FLoRA (Wang et al. 2024) allocates different LoRA

---

Published as a conference paper at ICLR 2026
FeltT
DeffT
C24
FLORA
FodiMtokA
DoVFT
Training Time (h)
Training Time (h)
10.03
3.0
2.00
2,88
Training Time
3.0
2.56
2.60
1.19
2.0
2.0
6.07
6.75
4.58
1.0
1.0
2:25
0.0
2.0
(a) LLaMA2-7B.
(b) LLaMA3. 1-BB..
(c) LLaMA2-133.
Figure 5: Comparative analysis of cumulative local training time across different methods.
Fedff
parti
L24
btogiea
FLatA
bedBA-LORA
DOVFT
Communication (CBB)
20.0
20.12
Communication (GB)
45.0
11.85
35.0
14.0
Communication
14.0
8.0
8.0
6.21
15.0
5.30
2.0
5.0
$.220..05
(a) LLaMA2-7B.
(b) LLaMA3.1-8B.
(c)
LLaMA2-133.
Figure Comparative analysis of total communication overhead across different methods.
ranks to devices based on their resources. FedSA-LoRA (Guo et al. 2025) only shares matrices A
with the server reduce resource costs.
4.3 PERFORMANCE EVALUATION
Table presents the comprehensive performance comparison. DEVFT consistently outperforms baseline methods across all settings, demonstrating effectiveness and robustness.
1) Comparison with Resource-Unavarn Methods. Resource-unaware methods uniformly demon- strate inferior performance. On close-ended benchmarks, FedIT suffers average performance drops of 2.06%, 8.9%, and 4.9 LLaMA2-7B, LLaMA3.1-8B and LLaMA2-13B. respectively. For open-ended benchmarks, the gaps are 0.72, 0.97, and 0.94. This degradation primarily arises from noise introduced by FedIT's independent aggregation of matrices A and B. DoFIT achieves moder- ate gains through specialized initialization and aggregation strategies but still lags behind DEVFT, with gap of up 10.63% LLaMA3.1-81 Similarly. C2A performs notably with average performance drops of up to 9.07% and 1.2 in close-ended and open-ended benchmarks. respectively, underscoring the inherent limitations of adapter- based methods.
than DEVFT,
2) Comparison with Resource-Aware Methods. Resource-aware methods generally outperform resource-unaware counterparts, but still fall short DEVFT. ProgFed shows average performance degradation of 1.33% and 0.61 on LLaMA2-7B. 4.1%% and 0.79 on LLaMA3.1-23 1-B.BB.BBB and 2.55% and 0.88 on LLaMA2-138 for close-ended and open- open-ended benchmarks, respectively. FedSA-LoRA exhibits similar performance degradation patterns ProgFed, while FLORA demonstrates more significant performance deterioration. In particular, for close-ended benchmarks, FedSA-LORA shows average performance decrements ranging from 1.52% to 3.28% across these models, whereas FLoRA exhibits more substantial degradation, with decrements spanning from 1.9% to 6.6%. The superiority of DEvFT stems from its developmental paradigm, which progressively builds power- ful LLM from a compact foundation, effectively preventing convergence to local minima.
4.4 EFFICIENCY EVALUATION
In this section, we evaluate the efficiency DEVFT from both computational and communica- tion perspectives. Furthermore, present detailed analysis of training overhead across different developmental stages to understand how DEvFT effectively optimizes resource utilization.
Computation Efficiency. Instead of using floating- point operations per second (FLOPs) to evaluate computation efficiency, we employ wall-clock training time to provide more intuitive reflection of real-world deployment efficiency for each method. Specifically, measure the cumulative local

---

Published as a conference paper at ICLR 2026
Communication (MB)
Memory Usage (CB)
Traiiigg Time (4)
Foll DEYFT
Fell DEAIT
Fell
10
DaviTT
150
225
300
75
150
225
300
75
150
225
300
Round
Round
Round
Figure 7: Resource consumption analysis of a device per round: training time, communication overhead, and memory usage for LLaMA2-7B.
training time required for each method to reach convergence, with results shown in Figure [ Our experimental results demonstrate that DEVFT significantly accelerates model convergence across all model architectures. Notably, for LLaMA2-7B, DEVFT achieves up to x.59x speedup conver- gence time. This improvement can be attributed to the developmental training paradigm of DEVFT: early fine-tuning on compact submodels significantly reduces computational overhead, while cross- stage knowledge transfer further expedites convergence when scaling to larger submodels.
Communication Efficiency. Figure the total communication overhead required for each method to reach convergence. DEvFT consistently achieves convergence with minimal communi- cation costs across all experimental settings, reducing communication overhead by up to 10.67 x on LL.MA2-13B. This communication efficiency stems from the fact that DEVFT only transmits small number of LoRA parameters to the server during the initial S - stages.
Detailed Overhead Analysis. To gain deeper understanding of DEVFT's efficiency, Figure 7 re- ports the per-round resource consumption on each device for FedIT and DEVFT, including training time, communication overhead, and memory usage. FedIT consistently incurs high resource costs throughout fine-tuning. In contrast, DEVFT exhibits more efficient pattern, with resource require- ments gradually increasing the submodel capacity expands, thereby substantially reducing over- all training overhead. In the early stages, particularly the first stage, DEVFT achieves significant resource savings compared to FedIT, reducing per-round training time by 10.3 communication overhead by 4x, and memory usage by 4x. Intriguingly, we discover that fine-tuning the recon- structed models of DEVFT at each stage also yields acceleration compared to directly fine-tuning pre-trained models. For example, even in the fourth stage where the submodel grows to match the target model size, DEVFT still delivers a 1.44x speedup per round.
Table 2: Ablation study on different layer grouping strategies.
Table 3: Ablation study on different representa- tive layer construction methods.
Method
Ended Benehmark
Method
Close- Ende-EEded Benchmark
TruthruQQA
MMLU
[FEvvl
BBH
Average
TrachfuQA
MMLU
IFEval
BBH
Average
LLaMA2-7B (INT4)
[Touvron al. al..
2023
LLaMA2-7B (INT4)
Touvron
2023
DGLG
50.28
44.15
33.97
40.93
42.33
DBLF
50.28
44.15
33.97
40.93
42.33
RANDOM
47,89
42.09
29.18
38.15
39. 99.90 2.43)
O-ONE
16.75
40.13
26.38
37.62
37.72 (L 1.61)
EVEN
45/41
39,83
25.04
36.73
36.25
6.08)
SUM
48.15
42.91
30.69
39.84
40.00 (L 1.49)
LLaMA3.1-8 (INT4)
(Towsron et al. 2023
LLaMA31-8B (INT4)
Tanvion et al. 2025
DGLG
55.23
68.42
62.29
71.04
64.25
DBLF
$5.23
68.42
62.29
71.04
64.25
RANDOM
66.74
3.56)
R-ONE
47.51
57.33
50.21
58 38.09
53.29 10.96)
EVEN
62.50
50.01
70.03
(].766(6..4
58.47
68
61.20(L 3.05)
4.5 ABLATION STUDY
Effect of the Deconffiction-Guidee Layer Grouping Mechanism. To understand the significance of the deconfliction guided layer grouping (DGLG), we compare it with two baselines: random grouping (RANDOM) and even grouping (EVEN). Table 2 shows that DGLG consistently outper- forms both baselines across all settings. For LLaMA2-7B, RANDOM and EVEN incur average per- formance drops of 2.43% and 6.08%, respectively, relative to DGLG. Similar trends are observed on LLaMA3.1-8B where the degradations reach 3.56% and 6.49% respectively. These results demon- strate that DGLG effectively enhances the layer fusion process by clustering layers with minimal parameter conflicts into the same group, thereby preserving more informative representations.
8

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation of DEVFT's compatibility with existing methods
Method
Close-Ended Benchmark
Resource 1
TruthfulQA
MMLU
IFEval
BBH
Average
Time (h)
Comm. (GB)
LLaMA2-7B (INT4)
(Touvron et al. 2023)
FedIT
47.57
42.45
31.76
39.28
40.27
2.49
5.03 2.36 (x2.13)
DedIT+
49.86
43.87
33.65
40.79
42.04
1.77)
0.83
FedSA-LORA
48 24
42.91
32.71
39.36
40.81
2.38
2.52
YedSA-LoRA+DEVFT
50,42
44.57
40.92
41.36
44.32 (15,5)
0.72 (x3,31)
1.18 (x2.14)
LLaMA2-13B (INT4)
TTovrron et al. 2023]
FedTT
$2.40
55.45
40.33
46. 14
48.58
6.67
8.39
DEVFT
56.84
58.26
45.49
48.52
52.28 (1 3.70)
2.30 (x2.90)
3,93 (x2.13)
FedSA-LoRA
55.73
57.51
43.21
46.91 49.13
50.84
6.42
4.20
DEVF
57.61
59.25
47.63
53,41 (1 2.57)
2.19 (x2.93)
1.97 (x2.13)
Table 6: Performance analysis under varying submodel growth rates.
Table 5: Performance analysis of different ini- tial submodel capacities.
Growth Rate
Clase- Ended Benehmark
Initial Capucity TrathfuQAA
Close-Ended Benchmark
TruthtuQQA
MMLU
IFEval
BBH
Average
MMLU
IFEval
BBH
Averaage
LLaMA2-7B (INT4)
[Touvron et 20233
LLaMA3.--88 1-8B (INT4)
Touvron et all 2022)
50.28
44.15
33,97
40.93
42.33
$2.45
66.85
56.83
70.12
61.56 2.69)
47.96
42.56
29.87
38.79
39 39.801 2.53)
53.87
67.31
70.50
62.78(L
45.68
40.07
25.63
36.92
37.08 (:25)
$5.23
68.42
62.29
71.04
64.25
LLaMA2-13B (INT4)
Touvron al. [022]]
8 53.21
67.12
58.35
20.65
62.33 1.92)
16 51.08
65.89
54.12
70.01
60288 3.97)
57.19
58.74
46.45
48.70
52.77
32
48.79
64.49
49.75
69.33
58.09
56.78
56
46.77 ( 6.3
48.12
$2.33
26.78
37.45
41.17 ( 11.6)
Effect of the Differential-Basse Layer Fusion Strategy. To evaluate the effectiveness of the differential-basee layer fusion (DBLF), we compare it against two baselines: R-ONE, which ran- domly selects one layer from each group as the representative layer, and SUM, which directly per- forms the addition operation on all layers within each group to generate the representative layer. As shown in Table 3 DBLF consistently outperforms both baselines. On LLaMA2-7B. R- R-ONE and SUM incur average performance drops of 4.61 % and 1.43%, respectively, relative to DBLF. The per- formance gap widens on LLaMA3.1 1-8B, with degradations reaching 10.96% for R-ONE and 3.05% for SUM. These results confirm that DBLF can effectively capture and integrate the unique semantic information from layers within each group, leading superior representative layer construction.
4.6 ANALYSIS
Compatibility with Existing Methods. We further validate the compatibility of DEVFT with ex- isting approaches by integrating it with FedIT and FedSA-LoRA. Table shows that incorporating DEVFT consistently improves model performance and system efficiency. For example, combining DEVFT with FedIT I.LMMA-133B yields 3.7% average performance gain, 2.9x faster conver- gence, and 2.13 x reduction in communication overhead. Similar improvements are observed when integrating DEVFT with FedSA-LoRA. These results indicate that DEVFT functions as a general framework that seamlessly enhances existing methods while preserving their inherent strengths.
Impact of Initial Submodel Capacity. We also conduct experiments to investigate how the initial capacity of submodels influences the overall model performance. Specifically, we experiment with ALaMA3.1-8B and set different initial capacities 1122.488.16.22) while maintaining the same total training budget. The submodel capacity also doubles progressively until reaching the full model capacity. Table 5 shows that the model achieves optimal performance when the initial capacity is set to 4, while either smaller or larger initial capacities result in performance degradation. This phc- nomenon is analogous to human learning, where starting from either too early (infancy) or too late (adulthood) may lead to suboptimal outcomes due to premature delayed cognitive development.
We explore how different submodel growth rates affect over- all performance. Specifically, we experiment with diverse capacity scaling multipliers {2,4,8). For instance, a multiplier of indicates that the submodel capacity quadruples at each stage un- til reaching the full capacity. This generates capacity sequences of (4-+16->) 16-32) for LLaMA2-7B
Impact of Submodel Growth Rate.

---

Published as a conference paper at ICLR 2026
Table 7: Scalability and robustness analysis of DevFT on text classification tasks.
Evaluation Benchmark f
Method
YELP-P
AGNEWS
YAHOO
20NEWS
Average
BERT (Deliin et al.
2019
FedIT DEVFT
83.12
87.05
68.34 70.67
76.89
78.85
84.53
90.91
80.06
81.54 (12.69)
RoBERTa (Liu et al. 2019)
FedIT 82.93 84.02
87.86 90.27
68.21 71.35
77.32
79.08
DEVFT
79.83
81.37 (12.29)
and (5-20-~40} for LLaMA2-13B. Table demonstrates that higher growth rates significantly compromise model performance. For LLaMA2-7B, scaling multipliers of and 8 lead to average performance drops of 2.53% and 5.25%, respectively. The degradation is even more pronounced on LLaMA2-13B, with decreases of 6.3% and 11.6%. This deterioration can be attributed abrupt ca- pacity transitions, which may disrupt the construction of the knowledge structure. This phenomenon mirrors natural learning processes, where steady, incremental development typically yields better long-term outcomes compared to the aggressive pursuit of short-term performance gains.
Scalability and Robustness Analysis. assess DEVFT's scalability across varying device popu- lations and its robustness to data heterogeneity, we conduct additional experiments on text classifi- cation tasks partitioned via a Dirichlet distribution (a = 1). Our experimental setup spans wide range of scales, including 100 devices for 20NEWS (Lang 1995). 1,000) for AGNEWS (Zhang et al.) 2013) and YELP-P (Zhang et al. 2015), and up to 10,000 devices for YAHOO (Zhang et al. 2015). utilizing BERT (Devlin et al. 2019) and RoBERTa (Liu et al. 2019) global models. The results in Table 7 indicate that DEVFT consistently outperforms the baseline in these heterogeneous set- tings, achieving an average performance gain of up 2.69% on BERT. This success underscores the scalability and robustness of our method, which stems from our developmental training paradigm: by initiating training with smaller, less complex submodels, DEVFT effectively mitigates the risk of client drift and overfitting to local data distributions during the critical early phases.
FUTURE WORK
The developmental analogy underlying DEVFT be interpreted through distinct lenses: Capacity Curriculum (mirroring the physical development of the brain) and a Data Curriculum (mir- roring the progression from simple to complex knowledge) (Soviany al. 2022). In this work, DEVFT strategically prioritizes the Capacity Curriculum to address the immediate and prohibitive hardware bottlenecks prevalent recognize that integrating a Data Curriculum represents a logical and promising evolution of the framework. Future research could explore synergizing model-level growth with data-level currieu- lum learning- for instance, feeding simpler instructions to early-stage submodels while reserving complex reasoning tasks for the fully matured model-to fully realize the cognitive metaphor.
cessurce-constrained federated edge environments. However, we
6 CONCLUSION
In this paper, we introduce DEVFT, an innovative federated fine-tuning framework that substan- tially reduces the resource consumption of LLM adaptation through cognitive developmental train- ing. DEVFT decomposes the fine-tuning process into several developmental stages, where each stage adapts a submodel with increasing parameter capacity. To efficiently architect these stage- specific submodels, DEVFT integrates two key techniques: a deconfliction-uuddes layer grouping mechanism and a differential- based layer fusion strategy. Extensive evaluations across multiple benchmarks demonstrate that DEvFT achieves superior performance with significantly enhanced efficiency. Moreover, it maintains high compatibility with existing federated fine-tuning methods, offering robust and versatile enhancement to the current ecosystem.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported in part by the Science and Technology Development Fund of Macau (0107/2024/RIA2), Joint Science and Technology Research Project with Hong Kong and Macau in Key Areas of Nansha District's Science and Technology Plan 117720022-00100-01TT135 and the Multi-Year Research Grant of University of Macau (M1TGC-GGG223-0021---0------)) and -00-GGGGG222-00000-0TTTCCCC
REFERENCES
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. Advances Neural Information Processing Systems, 37:14457-14483, 2024.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009,
Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Fedadapter: Efficient federated learning for modern nip. arXiv preprint arXiv:2205.1011 10162, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
vvicuna/
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for fed- erated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 12903-12913, 2024.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org
abs/1810 04805)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiw:2407.21773, 2024.
Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. The Thirteenth International Confer- ence on Learning Representations, 2025.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfi: Let federated participants cooperatively learn prompts instead of models- federrted learning in age of foundation model. IEEE Transactions on Mobile Computing, 2023.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprini arXiv:2403. 0444003.4000. /4608, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0333 03300, 2020.
Ching-Y Yi Hung. Cheng-Hou Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in neural information processing systems, 32, 2019.
Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyun Park, and SangKeun Lee. Client-customized adaptation for parameter-officient federated learning. In Findings of the Association for Compu- fational Linguistics: ACL. 2023, pp. 1159-1172, 2023.
Ken Lang. Newsweeder: learning filter netnews. In Proceedings of the Twelfth International
Conference on International Conference on Machine Learning. ICML '95 pp. 331-339, San Fran- cisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc. ISBN 1558603778.
11

---

Published as a conference paper at ICLR 2026
Heju Li, Rui Wang, Jun Wu, and Wei Zhang. Federated edge learning via reconfigurable intelligent surface with one-bit quantization. In GLOBECOM 2022-2022 IEEE Global Communications Conference, pp. 1055-1060 IEEE, 2022.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang. and Zhihua Zhang. On the convergence of 2019.
fedavg on non-iid data. arXiv preprini
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10.186533v1/2022.2 www.ooo......ooo..22222.2 URL https:
1/aclanthology orq.oo////o...ooo. 2022. acl-long...... 229
Yi Liu, Xiaohan Bi, Lei Li, Sishuo Chen, Wenkai Yang, and Xu Sun. Communication effi- cient federated learning for multilingual neural machine translation with adapter. arXiv preprint arXiv:2305. 12449, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL httpst:::: /ttpss///aivv..o wttp/bog/g////////o/e 11wtabs//99.... 11692
John McArdle and Richard W Woodcock. Human cognitive abilities in theory and practice. Psychology Press, 2014.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficieent learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. Feder- ated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes. arXiv preprint vrXiv:23/2.06355 2023.
Petru Soviany, Radu Tudor lonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, 13066):1526-1565 2022.
Shangchao Su, Mingzhao Yang, Bin Li, and Xiangyang Xue. Federated adaptive prompt tuning for multi-domain collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15117-15125, 2024.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in learning. arXiv preprint arXiv:2403. 12313, 2024.
federated
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261,, 2022.
John Sweller. Human cognitive architecture. In Handbook of research on educational communica- tions and technology, pp. 369-381. Routledge, 2008
Kahou Tam, Chunlin Tian, Li Li, Haikai Zhao, and ChengZhong Xu. Fedhybrid: Breaking the memory wall of federated learning via hybrid tensor management. In Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems, pp. 394-408, 2024.
Chunlin Tian, Li Li, Zhan Shi, Jun Wang, and ChengZhong Xu. Harmony: Heterogeneity- hicrarchical management for federated learning system. In 2022 55th IEEE/ACM International Symposium Microarchitecture (MICRO), pp. 63 6-1-.... IEEE, 2022.
Chunlin Tian, Li Li, Kahou Tam, Yebo Wu, and Cheng-Zhong Xu. Breaking the memory wall for heterogeneous federated learning via model splitting. IEEE Transactions on Parallel and Distributed Systems, 2024a.
12

---

Published as a conference paper at ICLR 2026
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, arXiv preprint arXiv:2404.19245 2024b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2337.09228. 2023.
Hui-Po Wang, Sebastian Stich, Yang He, and Mario Fritz. Progfed: effective, communication, and computation efficient federated learning by progressive training. In International Conference on PMIR, 2022.
Machine Learning, pp.
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513 2024.
Wolf. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint 190.0:1110.03771, 2019.
Yebo Wu, Li Li, Chunlin Tian, and Chengzhong Xu. Breaking the memory wall for heterogeneous federated learning with progressive training. arXiv preprint arXiv:2404.13349 2024.
Yebo Wu, Jingguang Li, Zhijiang Guo, and Li Li. Elastic mixture of rank-wise experts for knowledge reuse federated fine-tuning. arXiv preprint arXiv:2512.00902, 2025a.
Yebo Wu, Jingguang Li, Chunlin Tian, Zhijiang Guo, and Li Li. Memory-efficient federated fine-
tuning of large language models via layer pruning. arXiv preprint arXiv:2508.17209 2025b.
Yebo Wu, Chunlin Tian, Jingguang Li, He Sun, Kahou Tam, Li Li, and Chengzhong Xu. A survey on federated fine-tuning of large language models. arXiv preprint arXiv:2503. 2025c.
Binqian Xu, Xiangbo Shu, Haiyang Mei, Zechen Bai, Basura Fernando, Mike Zheng Shou, and Jinhui Tang. Dofit: Domain-awar aware federated instruction tuning with alleviated catastrophic for- getting. In The eighth Annual Conference on Neural Information Processing Systems, 2024a.
Mengwei Xu, Dongqi Cai, Yaozong Wu, Xiang Li, and Shangguang Wang. Fwdllm: Efficient fedlim using forward gradient. arXiv preprint arXiv:2308. 13894, 2023.
Naen Xu, Changjiang Li, Tianyu Du, Minxi Li, Wenjie Luo, Jiacheng Liang, Yuyuan Li, Xuhong Zhang, Meng Han, Jianwei Yin, al. Copyrightmeter: Revisiting copyright protection in text-to- image models. arXiv preprint arXiv:241 13144, 2024b.
Naen Xu, Jinghuai Zhang, Changjiang Li, Zhi Chen, Chunyi Zhou, Qingming Li, Tianyu Du, and Shouling Ji. Videoeraser: Concept erasure in text-to-video diffusion models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 5965-5994, 2025.
Naen Xu, Hengyu An, Shuo Shi, Jinghuai Zhang, Chunyi Zhou, Changjiang Li, Tianyu Du, Zhihui Fu, Jun Wang, and Shouling Ji. When agents" nisremember" collectively: Exploring the mandela effect in lbsed multi-agent systems. arXiv preprint arXiv:2602.00422 2026a.
Naen Xu, Jinghuai Zhang, Ping He, Chunyi Zhou, Jun Wang. Zhihui Fu, Tianyu Du, Zhaoxiang Wang, and Shouling Ji. Fraudshield: Knowledge graph empowered defense for llms against fraud attacks. arXiv preprint 22X4:2200. 2026b.
Yuxuan Yan, Qianqian Yang, Shunpu Tang, and Zhiguo Shi. Federa: Efficient fine-tuning of language models in federated learning leveraging weight decomposition. arXiv preprint arXiv:2404.1884 2024.
Fu-En Yang, Chien Yi Wang. and Yu- Yu-Chiang Frank Wang. Efficient model personalization in feder- ated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19159-19168, 2023.
13

---

Published as a conference paper at ICLR 2026
Rui Ye, Wenhao Wang. Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via fed- erated learning. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6137-6147, 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing IIm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078 2024.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6915 6919. IEEE, 2024a.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.0283 02338, 2024b.
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL ELl1//500BBB1171333000888188-2-2
les/paperr22115/
Zikai Zhang, Rui Hu, Ping Liu, and Jiahao Xu. Fed-pilot: Optimizing lora allocation for efficient federated fine-tuning with heterogeneous clients. arXiv preprint arXiv:2410. 2024c.
Zikai Zhang, Ping Liu, Jiahao Xu, and Rui Hu. Fed-hello: Efficient federated foundation model fine-tuning with heterogeneous lora allocation. arXiv preprint arXiv: 12213, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gouzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiingg evaluation for large language models, arXiv preprint arXiv:2311.07911, 2023.
14

---

Published as a conference paper at ICLR 2026
A A CONCRETE EXAMPLE OF DEVFT IN PRACTICE
Assume the full model consists of six layers: [01, 0z. 03 04. 0s 06), and DEVFT proceeds in three stages with sub-model capacities: {2 6).
Stage (Capacity 2):
DGLG partitions the layers into two groups: [[01, 0s). 10.. 06H1
DBLF produces two representative layers: (01, 02 03) + 051; (04, 0s) + 982.
These two representative layers form the sub-model [d8t, y82] for the current stage.
After completing the current stage, perform knowledge transfer follows: ( 2,,3, 082 + 04,5,6.
All layers are updated, and the resulting global model then serves as the foundation for constructing the submodel in the next stage.
Stage (Capacity 4):
DGLG partitions the layers into four groups: [(0)], (02, 0z), (04, 06), [0a)]-
DBLF produces four representative layers: {01} + yea; ( Uea; (0s. 05} Jea; {6a] - 984.
These four representative layers form the sub-model [18t 06 05s 08.] for the current stage.
After completing the current stage, we perform knowledge transfer as follows: da 0 1962
2223 yaa 04.5; f 0g.
All layers are updated, and the resulting global model then serves as the foundation for constructing the submodel in the next stage.
Stage (Capacity 6):
The submodel now encompasses all six layers, reaching the model's full capacity and en- abling end-to-end fine-tuning of the global model to complete the training process.
This example serves as a simplified illustration to intuitively demonstrate the multi-stage group- ing concept. Empirically, we observe that neighboring layers often exhibit high similarity both function and parameters due to the hierarchical nature of information processing in Transformers. Consequently, these layers are frequently clustered together by DGLG. Thus, while DGLG operates independently of layer positions, this contiguous grouping serves representative and intuitive visualization of the actual process.
B ADDITIONAL IMPLEMENTATION DETAILS
Our DEVFT is implemented using PyTorch with the support of HuggingFace Transformers li- brary (Wolf 2019) for model and dataset management. Following the experimental setup of Open- FedLLM (Ye al. 2024), we randomly distribute the Alpaca-GPT4 dataset across 20 devices, with 10% of devices randomly sampled for participation in each training round. Each selected device performs 10 local training iterations with batch size of 16. The local fine-tuning process utilizes the AdamW optimizer coupled with cosine learning rate scheduler. We adopt a staged learning rate strategy, starting at le-6 and incrementing by factor of 10 at each subsequent stage until reaching le-4. Additionally, we exclusively apply LoRA to W 9 and W, matrices in the attention layers (Xu ctt al. 2026a b) and configure the LoRA module with a rank 32. The maximum sequence length set to 512 tokens (Ye et al. 2024). The total number of federated fine-tuning rounds is set to 300 for LLaMA2-7B and LLaMA3.1 -BB, and increases to 400 for LLaMA2-133 13B. Moreover, to improve computational efficiency. apply INT4 quantization (Ye et al. 2024) to all models and conduct experiments on single NVIDIA H800 GPU. To ensure the reliabiiity of our results, all experiments repeated multiple times, with the averaged values reported the final results.
15

---

Published as a conference paper at ICLR 2026
C THEORETICAL CONVERGENCE ANALYSIS
We analyze DEVFT under standard nonconvex FL assumptions (Li et al. 2019: Wang et al. 2022 Wu al. 2024). The analysis is aligned with the actual pipeline: each stage, the current full model is regrouped, DBLF constructs representative-layer initialization, the stage submodel is trained by FedAvg, and the trained submodel mapped back the full model by group-wise replication.
C.I PRELIMINARIES AND STAGE TRANSITION
Full objective. Let θ E Rd denote full-model trainable parameters (L layers). For client E [N]]
1 f()) N Σ f())),
f(( (et>D,(((;F00
fint : inf f()) 0c. @
(6)
Use block norn |0|| Σ/= 10
Stage-wise dynamic grouping. At stage [S], grouping is recomputed from the previous fused fuse is the given initialization):
full model (x-1) (for s 1, θ (0)
I1, (g.²) L. O (i
L.....6 Ls =.
Let m, == maX, gg and d, Lodaaer-
Stage-start DBLF construction (full - submodel). For each group gr (x) representative block
with anchor a(n). define
uo
(x)
0/v-11) + 8
Σ
fuse. (()))
B E (0, 1].
jER!
Stacking (10.)) -4 gives
(5) (5)
(5) (wo ...,,,...,...................... mo E Rdd.
uo
Stage embedding and objective. gn Ill. (u)) = un. Equivalently, E!L (u) A,u, where
Define replication embedding E. Rd. - Rd: for any j
AA.12 m.
(8)
Define stage objective
F. (. II,) =
Σ fa (u))
F (II) := :n
(u;
(9)
N
each ft is -mmoothh in θ, then Fa(; is (L.mo)-smooth. Let
L : L max
(10)
sE
every stage objective is L-smooth.
Stage-end map-back (submodel + full). After stage-s training obtains u/ define
ell.
(11)
fuse
This is pure replication (broadcast) and introduces extra fusion perturbation.
Federated protocol. Each round samples M = q.V clients (q E 1)) each selected client performs K local SGD steps (stepsize 7)), then server averages (FedAvg (MeMahan et al.) 2017))
16

---

Published as a conference paper at ICLR 2026
Assumptions.
Assumption C.1 (Smoothness). [V)(()) V Lyll0 L 0'1),
0. Assumption C.2 (Unbiased stochastic gradients, bounded variance). (b; (.((0) and ()) (+)] S o².
Assumption C.3 (Bounded heterogeneity).
E [V/( + G2, AO.
C.2 PER-STAGE CONVERGENCE
Theorem C.4 (Per-stage stationarity). Fix stage s and condition on grouping al Under Assump- tions C.I.C.3. run FedAvg Fa(;11,) for Ts rounds, K local steps each round, and M qN
participants each round. If ns S aLK then
T T, V I.II M.
2 F(Fc(u6 (F((0)),I) II,) FII.
L 0² qN + k)
1) NTKT,
(12)
where > 0 is an absolute constant.
Corollary C.5 (Per-stage O(1/ (T.)...))........ If
co KVT,
with co S L then
37.
T
(u (9) II) II, 50 Fa(ma
(s) II,) F (II,)
1
1
T,
E
VT,
xNN.TT
9N
+0
T,
t=U
(13)
C.3 DBLF CONSTRUCTION PERTURBATION BOUND
DBLF perturbation occurs stage start (Eq.
Lemma C.6 (Stage-start DBLF perturbation). Define the pre-construction intra-group diameter
gpre max max n j.keg!")
fuse,k
(14)
Then for any group (+) and any j gn
(e)
5 (1+ 3(1((1)) - -1)) 8p S (1 3m.))0m0
(15)
Let ((0)) = (.(l(((())) Then
|+6 |(إ-1ا <VL + 3mm
(16)
Proof. From Eq.
7
(0/0--1)/.))) fuse.a(m)
(0l
o((mm!)) + 3
Σ
(0/vm1) 1)
00,0
fuse..
kEg!"
Applying triangle inequality and the definition of 6 gives Eq. ations over L layers yields Eq. 16
15 Summing squared block devi-
C.4 FINAL-STAGE
GUARANTEE
At stage S, Ls = L, and thus Fs(:; IIs) = f(.). By construction,
e (8-1)
(17)
fuse
Also define the D-e.-crellcctedd start point
(65) ells (1)))
(S)
17

---

Published as a conference paper at ICLR 2026
Table 8: Assessing the generalization capability of DEVFT in centralized training scenarios.
Close-Ended Benchmark
Method
Time 4
TruthfuQQA
MMLU
IFEva
BBH
Average
LLaMA2-7B (INT4) (Touvron et al 2023
End-to-End DEYFT
48.36
43.27
32.64
39.54 42.25
40,95
1.77 0.55 (x3.22)
52.39
47.59
35,86
44.52 (1 3.57)
LLLMMA-13BB (INT4) (louvron al. 2023
End-to-End
54.68 59.38
57.32
42,51
46.98 49.95
50.37
4,28
DEVFT
61.82
48.57
54.93 (1 4.56)
1.41 (x3.04)
for which Lemma C.6 Livgives
| (8)
1 - S (1 8mg 3mg)
(18)
Using Ly smooohhnsss and Young's inequality, for any 0:
ʃ(0 + A) - f()) + - |1 a + L1 111? 20
(19)
2
Theorem C.7 (Final-stage convergence on full objective). Under Assumptions S for T' rounds with K local steps, participation rate q, and nıs S ALK
C.3
run stage
Then
|122 ) Sint)
Lus
+ t.t2ng
(20)
Ts t=
ns nkKTs
qN
Moreover, for any a > 0,
f(0(5)) - fint < ( fint) + 1 | a Ly
L 3mg)² ( (21)
2a
Proof. Eq. 20 is Theorem C.4
applied to stage S, where F's f. Eq. 21 follows from Eq.
19 with
θ (8) 6(5) O [ and Eq. 18
Corollary C.8 (Final-stage OLL/vTs) If cp with co + then
KVTs
1 Ts-1
f())5)) -ful finl
o
1 Ts
S o VTs
KVTs)
(22)
Ts
qN
In addition, Eq. OLL(I (2 JT.
27
shows an explicit DBLF construction
perturbation
term
Conclusion. The proof now matches the exact stage flow: regroup DBLF construct submodel stages. Per-stage guarantees are established on the corresponding stage objectives, and the final guarantee is stated on single full objective f, without telescoping across incompatible objectives.
FedAvg train + map back full model. repeated across
D MORE EXPERIMENTS
D.I GENERALIZABILITY ANALYSIS: CENTRALIZED TRAINING
We further evaluate the generalizability of DEVFT under centralized training. To this end, we conduct additional experiments using the same stage-wise training strategy, with end-to-end fine- tuning the baseline. As shown in Table DEVFTT consistently outperforms end-to-end fine-tuning across all benchmarks. On LLaMA2-78 DE improves performance by 4.03%, 4.32%, 3.22%, and 2.71% TruthfulQA, MMLU, IFEval, and BBH, respectively, while accelerating convergence by 3.22x. On the larger ALLMM2--3BB DEVFT maintains this advantage, yiclding an average performance gain of 4.56% and x speedup.
18

---

Published as a conference paper at ICLR 2026
Table 9:
Performance comparison under heterogeneous
resource constraints on LLaMA2-7B
(INT4).
Close-Ended Benchmark
Method
TruthfulQA
MMLU
IFEval
BBH
Average
FedIT
41.96
40,82
26.38
37.13
36.57 (-2.87)
DoFIT
43.74
41.53
28.71
38.45
38.11 (-1.33)
C2A
41.78
40.63
26.21
36.98
36.40 (-3.04)
ProgRed
45.04
41.67
29.26
38.51
38.62 (-0.82)
FLoRA
43.58
41.35
28.06
37.94
37.73 (-1.71)
FedSA-LoRA
45.27
41.77
29.60
38.62
38.82 (-0.62)
DEVFT
45,86
42.35
30.54
39.01
39.44
These gains arise from DEVFT 's developmental training paradigm, which incrementally expands model capacity during fine-tuning. Progressive scaling smooths the loss landscape and mitigates poor local minima, while compact early-stage submodels reduce computation. The knowledge learned at each stage is then reused to initialize subsequent larger models, thereby accelerating convergence. Overall, these results demonstrate the generalizability of DEVFT beyond the feder- ated setting, underscoring its potential as an efficient and scalable fine-tuning framework even under centralized training.
D.2 ROBUSTNESS UNDER HETEROGENEOUS RESOURCE CONSTRAINTS
To evaluate the robustness of DEVFT realistic edge environments, we simulate a heterogeneous setting where device memory budgets range from 3GB to 9GB. In this setup, resource- constraind devices (e.g., those with 3GB memory) participate exclusively in the early, less resource- resource-intensive developmental stages. Experimental results in Table 9 demonstrate that DEVFT outperforms all baselines, achieving an average performance improvement of up to 3.04%. This performance boost attributed to the inclusive nature of DEvFT: unlike baselines that exclude fow-memory devices entirely, DEvFT enables these devices to contribute their valuable local data to the model's founda- tional knowledge during the early stages.
D.3 COMPARISON WITH STATE-OF-THE ART LORA OPTIMIZATION METHODS
To further validate the effectiveness of DEVFT, we conduct a comprehensive comparison against recent state-of- the-ar LoRA optimization methods, including Fed-pilot (Zhang et al. 2024c). Fed- HeLLo (Zhang et al., 2025). FlexLoRA (Bai et al. 2024). and HETLoRÁ (Cho et al. 2024), using the LLaMA2-7B. As summarized in Table TO, DEVFT demonstrates superior pertormance across all evaluated benchmarks, achieving an average accuracy gain of up to 1.99% over baselines. We attribute this improvement to DEVFT's developmental training paradigm, which effectively navi- gates the optimization landscape discover superior convergence trajectory. It is important to note that these baselines primarily focus on optimizing LoRA module allocation or rank adaptation. In contrast, DEVFT targets the training process. Consequently, method is not mutually exclu- sive but rather complementary to these approaches. DEVFT can potentially combined with these LoR LooA-optinization techniques further enhance federated fine-tuning performance.
E DISCUSSION
Analysis of Peak Memory Efficiency. Peak GPU memory usage constitutes a pivotal constraint governing device eligibility in FL training. While DEVIT primarily architected to optimize cu- mulative system efficiency, it concurrently alleviates peak memory burdens. As evidenced in Fig- ure peak memory usage DEVFT's early phases substantially mitigated-rrduueel reduced by up to 4x relative baseline methods. Although peak memory in the final stage matches the baseline, the total resource savings over the entire training process are considerable. Furthermore. DEVFT remains orthogonal specific peak-memory reduction techniques, such as Fed-pilot (Zhang al.
19

---

Published as a conference paper at ICLR 2026
Table 10: Performance comparison with LoRA optimization methods on LLaMA2-7B (INT4).
Close-Ended Benchmark t
Method
TruthfulQA
MMLU
IFEval
BBH
Average
Fed-pilot
48.15
42.84
32.24
39.41
40.66 (-1.67)
Fed-HeLLLo
48.23
42.96
32.37
39.54
40.78 (-1.55)
FlexLoRA
47.83
42.72
32.15
39.36
40.52 (-1.81)
HETLoRA
47.71
42.58
31.96
39.12
40.34 (-1.99)
DEVFT
50.28
44.15
33.97
40.93
42.33
2024c). Since each DEVFT stage adheres to standard FL protocols, these optimization methods can be seamlessly integrated to further lower hardware barriers.
Adaptive Scheduling Strategy. In this work, employ globally fixed scheduling strategy. This design choice motivated by the need for controlled evaluation DEVFT 's core con- the developmental training paradigm, deconffiction-guul (DGLG), and differential-based layer fusion (DBLF) ensuring both reproducibility and fair com- parison against baselines. This approach aligns with established protocols in resource-efficiem. FL literature, such as ProgFed (Wang et al., 2022). Notably, even with this fixed schedule, DEVFT demonstrates significant performance gains, achieving up to 4.59 faster convergence and 10.67x greater communication efficiency (as illustrated in Figures S and 66. However, we recognize that adaptively scheduling each stage represents promising avenue for further optimization. As a gen- eral and extensible framework, DEvFT readily supports the integration of such adaptive mecha- nisms, which we leave for future exploration.
layer grouping
Distinction from Continual Learning. While the concept of dynamic model capacity has been explored in Continual Learning (CL) (Hung al. 2019). our objective differs fundamentally from this paradigm. The primary goal of CL is to mitigate catastrophic forgetting by expanding the model to accommodate sequence of distinct tasks. In contrast, DEVFT employs developmental growth resource-efficient training strategy for single downstream task within constrained federated environment. Rather than serving as a mechanism to preserve knowledge across different tasks, our approach leverages dynamic capacity specifically navigate hardware bottlenecks edge devices.
F THE USE OF LARGE LANGUAGE MODELS
A large language model used solely general-purpose tool for linguistic polishing (e.g., grammar, wording, and clarity). The LLM did not generate research ideas, design experiments, write substantive sections, produce analyze data, create code. All technical content, claims, and conclusions were authored and verified by the human authors, who take full responsibility for the manuscript. Suggested edits from the LLM inaccuracies, or fabricated statements. The LLM is not an author.
reviewed and post-edited to avoid plagiarism,
G LIMITATIONS
While our proposed DEvFT demonstrates superior performance, several limitations warrant ac- knowledgment. First, our current research primarily focuses on federated learning within a single organization. Extending our method to orossorrganzzational collaborative scenarios, where address- ing incentive mechanisms, trust establishment, and privacy concerns becomes paramount, represents significant yet valuable direction for future investigation. Second, although our approach substan- tially reduces computational requirements compared to traditional methods, the overall environmen- tal footprint of training LLMs remains considerable. Future work should more comprehensively quantify the carbon emission reductions achieved through our developmental paradigm and explore additional algorithmic and system-leve optimizations to further minimize environmental impact.
20


Reviews:
[{"review_id": "8bjF9nTp8V", "reviewer": "Reviewer_fj71", "summary": "This paper presents DEVFT, a federated fine-tuning framework that reduces LLM adaptation costs through cognitive developmental training by progressively expanding submodel capacities across stages. Leveraging deconfliction-guided layer grouping and differential-based layer fusion, DEVFT achieves efficient and effective fine-tuning across multiple benchmarks.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper’s main strength lies in its effective reduction of resource consumption while maintaining strong performance across benchmarks.", "weaknesses": "1. The intuition behind the use case of this developmental process is unclear. While the method seems to optimize overall efficiency, it lacks discussion on peak efficiency (e.g., throughput, maximum GPU memory limits), which is often more critical in practical federated learning scenarios. To improve fairness and applicability, results should also be evaluated under varying resource constraints.\n\n2. The paper lacks a clear system-level formulation of the FL process. Specifically, the grouping of parameters across clients is not well explained. In Equation (5), the meaning of θ is ambiguous and should be explicitly defined.\n\n3. The current setup appears to assume that all clients have devices capable of hosting the full model. Can the method be extended to cases where not all layers are trainable due to device limitations?\n\n4. It is also unclear how the method adapts to heterogeneous resources across clients. What strategies are employed for aggregation when clients train different subsets of layers?\n\n5. Please consider including loss and accuracy curves. I am uncertain about the number of local and global training rounds, as downstream fine-tuning of LLMs typically requires fewer rounds. Further justification of the experimental settings would be helpful.\n\n6. The choice of LoRA rank (e.g., 32) seems arbitrary. Since the optimal rank often depends on the downstream task, a fixed configuration might not generalize well. An ablation or explanation would strengthen this aspect.\n\n7. The anchor layer is set to the first layer by default. Is there a particular reason for this choice? Would selecting a different anchor (e.g., middle or final layers) affect the performance?\n\n8. The training rounds for each stage are fixed. Inspired by cognitive paradigms, one might argue that different stages (akin to learning phases) should have adaptive durations. For instance, adults may learn certain tasks faster than children and vice versa. Is the stage duration setting task-specific or task-agnostic?\n\n9. If “capacity” refers to the number of unfrozen layers, what does the stage-wise scaling actually control? For example, the assumption of linear memory increase may not hold. As shown in Fed-Pilot [1], memory cost tends to grow non-linearly due to activation reuse and other factors. The effect of the scaling setting on the resource is worth discussing.\n\n10. The implementation details of FedSA-LoRA as a baseline are unclear. In their original work, only the A matrix is shared across clients, and the server lacks a complete B matrix for global evaluation. How is this adapted in your framework for comparison?\n\n11. Important prior works are missing from the related work section. For example, Fed-Pilot [1] provides a memory-aware LoRA allocation strategy; Fed-HeLLo [2] introduces heuristic and Fisher Information-based layer selection; FlexLoRA [3] and HETLoRA [4] address task and resource heterogeneity. These works should be discussed and compared if necessary.\n\n12. Any generalizable findings from your experiments on the layer partition?\n\n13. Lastly, Equation (5) appears to be a straightforward weighted average, as in standard FedAvg.\n\n[1] Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clients. ArXiv 2024.\n\n[2] Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation. IEEE TNNLS 2025.\n\n[3] Federated fine-tuning of large language models under heterogeneous tasks and client resources. NeurIPS 2024.\n\n[4] Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models. EMNLP 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ufuKJDn1Hl", "reviewer": "Reviewer_4FbG", "summary": "This paper introduces DEVFT, a framework designed for resource-efficient federated fine-tuning of LLMs. DEVFT decomposes training into progressive stages of increasing model capacity, starting from a compact submodel. The method relies on two novel components: (1) **Deconfliction-Guided Layer Grouping (DGLG)**, which uses spectral clustering to group layers based on parameter similarity, and (2) **Differential-Based Layer Fusion (DBLF)**, which creates a representative layer for each group by fusing an anchor layer with parameter differentials. Knowledge is transferred between stages via LoRA parameters.", "questions": "1. **Layer adjacency constraint:** In Appendix B, each group's layers appear contiguous in the original model, but the clustering algorithm does not inherently enforce adjacency. How is this achieved or post-processed in implementation?\n2. **Fusion weighting:** When constructing representative layers, have the authors considered similarity-weighted averaging (e.g., weighted by inter-layer cosine similarity) instead of uniform $\\beta$-weighting? Would this further improve fusion fidelity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Novelty:** Introducing a developmental training framework to federated LLM tuning is creative and aligns with cognitive learning principles.\n- **Technical soundness:** The DGLG and DBLF modules are mathematically clear and empirically validated through detailed ablations.\n- **Practical benefits:** The approach significantly reduces communication and compute costs while improving accuracy on standard instruction-tuning tasks.\n- **Compatibility:** The method can be combined with existing frameworks such as FedIT and FedSA-LoRA, further improving efficiency.\n- **Reproducibility:** The authors provide source code, facilitating replication.", "weaknesses": "+ **Non-IID simulation:** Experiments are performed on datasets without explicit modeling of data heterogeneity, which is a core challenge in FL.\n+ **Resource accounting scope:** The reported communication overhead accounts only for uplink transmission of LoRA parameters, while downlink costs (e.g., transferring dense submodels between stages) and server-side overheads for clustering and layer fusion are not explicitly quantified.\n+ **Theory scope:** The convergence proof inherits classical FedAvg assumptions but does not analyze the approximation bias introduced by representative-layer fusion.\n+ **Hyperparameter sensitivity:** The method depends on several hyperparameters ($\\beta$, number of stages, initial capacity), yet sensitivity analysis is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OnTURf06ZY", "reviewer": "Reviewer_Sn7Y", "summary": "This work addresses the resource overhead in federated fine-tuning for LLMs at the edge devices and proposes a method called developmental federated tuning. This method decomposes the fine-tuning process into multiple developmental stages, with each stage optimizing a sub-model with increasing parameter capacity. Experimental results show that the proposed method outperforms several existing baselines in terms of communication overhead and convergence speed.", "questions": "1. The paper introduces its method design by drawing an analogy to human growth. However, the final method design mainly focuses on progressively adjusting the scale of trainable model parameters. In human growth, different stages may involve learning knowledge of varying complexity. Could the method be further optimized by perhaps adjusting the learning difficulty or task complexity at different stages? \n2. Please refer to the Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ This works aims at practical issues in federated instruction tuning on the edge.\n+ Figures are organized in a good shape.", "weaknesses": "1. Lack of investigation on existing work: Gradually introducing knowledge has already been demonstrated multiple times in research related to continual learning, such as in [1] (although [1] adjusts the model's capacity, which is not exactly the same as the technical approach in the current work). Additionally, the manuscript highlights the communication efficiency of this method in the introduction but does not discuss some existing works that optimize communication efficiency, such as [2] and [3].\n2. Lack of baseline: The currently referenced resource-aware methods are either not tailored for LLMs or have not been formally published. It is recommended to include highly relevant methods from top conferences or journals in the past two years as baselines. \n3. The relationship between this developmental tuning method and FL is unclear. The current method design may yield certain effects at the edge and seems to remain valid even without the distributed training architecture of FL. \n4. This work uses a scenario with only 20 clients, which is too few for the cross-device scenario targeted by this work. It is recommended to conduct evaluations in scenarios with a larger number of clients and greater data heterogeneity. \n5. The paper lacks theoretical analysis of convergence and does not provide a comparison of convergence curves, raising concerns about the method's convergence performance.\n\n[1] Compacting, Picking and Growing for Unforgetting Continual Learning. NeurIPS 2019. \n\n[2] Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes. ICML 2024.\n\n[3] FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences. ATC 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTylk4MjgH", "reviewer": "Reviewer_wUou", "summary": "This paper presents DEVFT, a federated fine-tuning framework designed to reduce the resource demands of large language model adaptation through cognitive developmental training. DEVFT progressively fine-tunes models across multiple developmental stages, each expanding parameter capacity. It employs a deconfliction-guided layer grouping mechanism and a differential-based layer fusion strategy to construct stage-specific submodels efficiently.", "questions": "1.\tDoes the framework assume identical or distinct data distributions across developmental stages, and how does this affect convergence?\n\n2.\tIs there reference to federated-specific parameter-efficient fine-tuning (pEFT) literature in Section 2.1, or are the cited works limited to centralized settings?\n\n3.\tDoes each developmental stage correspond to a single global model update, or are multiple federated communication rounds performed within a stage?\n\n4.\tIn the differential-based layer fusion process, how are non-linear interactions between layers preserved when performing linear subtraction-based fusion?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe addressed problem (how to design stage-specific submodels that facilitate progressive knowledge transfer while optimizing overall performance) is both practical and relevant to real-world federated learning scenarios.\n\n2.\tThe study is supported by comprehensive experimental evaluation across diverse benchmarks.", "weaknesses": "1.\tThe motivation could be strengthened. The framework assumes limited device resources only in the initial training phase, yet it remains unclear why smaller submodels are necessary if participating devices can accommodate full-model fine-tuning in the end. Furthermore, while the approach draws inspiration from human cognitive development, its applicability to heterogeneous resource settings warrants deeper justification—especially in cases where client resources evolve over time. I.e., for the cases that there are some devices who have more compute/memory resources available in the initial phase of the training and some clients who cannot accommodate larger models for the later half of the training.\n\n2.\tThe connection between LoRA/PEFT's limitations and federated-specific constraints is not clearly articulated, raising the question of whether the proposed solution addresses a genuinely federated challenge or a more general fine-tuning issue.\n\n3.\tThe layer grouping mechanism lacks empirical evidence for claims such as \"opposite signs neutralizing each other’s unique contributions,\" and the conceptual clarity of this section could be improved.\n\n4.\tThe assertion that “redundant layers limit representational diversity” conflicts with the subsequent need for a larger model, suggesting an inconsistency in the motivation for model expansion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)