Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Authors: Yangxu Liao
Wenke Huang
Guancheng Wan
Jian Liang
Bin Yang
Mang Ye
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Federated learning
Primary area: deep_learning->large_language_models
TL;DR: FedICU improves federated fine-tuning of large language models by splitting client updates into shared and unique parts and selectively uploading only important changes to boost generalization and efficiency.

Abstract: Federated learning provides an efficient privacy-preserving distributed training framework for large language models, addressing the growing scarcity of publicly available training data while enabling the utilization of private datasets. While integrating large language model fine-tuning with federated learning emerges as a promising research direction, researchers pay limited attention to non-IID instruction-following scenarios. Our key insight is decomposing client updates into consensus and divergence components, enabling the model to maintain core capabilities while adapting to domain-specific knowledge. We propose a novel federated learning framework called **FedICU** (Splitting with **I**mportan**C**e-aware **U**pdating for Heterogeneous **Fed**erated Learning with Large Language Models), which introduces an aggregation mechanism that dynamically balances these components based on their contribution to global model performance, while implementing an importance-aware parameter updating strategy to prevent catastrophic forgetting and domain overfitting. Extensive experiments across diverse domains demonstrate that FedICU significantly outperforms existing federated learning approaches in terms of both generalization performance and domain adaptation. Our code is available at https://github.com/liaosunny123/FedICU.

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Yangxu Liao 12 Wenke Huang 12
Guancheng Wan 2
Jian Liang' 2
Bin Yang?
Mang Ye
Abstract
(a) Different Downstream Instructions Disturbance
Medical
Datesees
Federated learning provides an efficient privacy- preserving distributed training framework for large language models, addressing the growing scarcity of publicly available training data while enabling the utilization of private datasets. While integrating large language model fine-tuning with federated learning emerges promising re- search direction, researchers pay limited atten- tion to non-IID instruction-following scenarios. Our key insight is decomposing client updates into consensus and divergence components, en- abling the model maintain core capabilities while adapting domain-specific knowledge. We propose novel federated learning framework called CedICU (Splitting with ImportanCe-awan Updating for Heterogeneous Federated Learning with Large Language Models), which introduces an aggregation mechanism that dynamically bal- ances these components based on their contri- bution global model performance, while im- plementing importance-award parameter up- dating strategy to prevent catastrophic forget- ting and domain overfitting. Extensive experi- ments across diverse domains demonstrate that FedICU significantly outperforms existing feder- ated learning approaches in terms both general- ization performance and domain adaptation. Our code is available at https:/ /gt..b...c.. com/ li liaosunny 23 /FFeIICU.
Medical
Interference
Different Update
Knowledge Catastrophic Forgetting
@: can salve Tank
SET
LoRA
solve Yask
Forget
Iterative Train
Figure Motivation. In the current process of Large Lan- guage Model Supervised Fine-tuning, the following issues exist: (a) Heterogeneous client distributions and varying instruction quality in Supervised Fine-tuning impair global model performance. (b) Ineffective preservation of general capabilities during federated training of large language mod- els fine-tuning.
1. Introduction
Federated learning is collaborative paradigm, allowing multiple clients to train shared global model collabora- tively without privacy leakage (McMahan et al., 2017b; Reddi et al., 2020; McMahan et al., 2017a; Huang et al., 2022; 2023c). Large language models (LLMs) (Achiam et al., 2023; Ouyang et al., 2022: Touvron et al., 2023; Devine, 2024; Liu et al., 2023; Wang et al., 2024) emerge as a transformative technology across main fields recent years (Imani et al., 2023; Didolkar al., 2024; Chen et al., 2023; Fang al., 2025a; Jiang et al., 2025). While trained on large public datasets, these large language models demon- strate sigmificant success in solving general problems across various domains. However, the scarcity of high-quality public training data becomes a critical bottleneck for large language model development (Kaddour al., 2023), with predictions suggesting the exhaustion of such data by 2026 (Villalobos et al., 2022). Although existing approaches attempt to address this challenge through datasets combi- nation (Wang et al., 2023) or self-generate data (Wang et al., 2022), these methods often fall short - the former is limited by data availability (Kaplan et al., 2020), while the latter risks model degradation (Alemohammad et al., 2023;
'Equal contribution 'Guangdong Laboratory Artificial In- telligence and Digital Economy (SZ), Shenzhen, China 'National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China. Correspon- dence to: Mang Ye <yemang @ whu.edu.en..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Muennighoff et al., 2023). Although substantial private datasets exist, their direct utilization is frequently restricted by privacy concerns. Notable large language models, such as BloombergGPT (Wu et al., 2023), FinGPT (Yang et al., 2023b) and etc., successfully leverage private datasets, high- lighting the potential value of these resources. This situation presents an opportunity for federated learning, which en- ables the utilization of private, high-quality datasets while maintaining privacy (Li et al., 2020), potentially offering solution to the ongoing challenge of data scarcity in large language model development (Zhuang et al., 2023).
foundation model capabilities, and another that optimally captures domain-specifee characteristies. To investigate this hypothesis, we perform decomposition analysis on LoRA parameters and discover convergence factor, which express the general ability of the model, and divergence factor, which expresses the especial ability of the model, in the process of supervised fine-tuning of LLM on the federated learning. Based on this observation, we raise the follow- ing question: 1) How can we ensure effective instruction learning during sederated fine-tuning on the heteroge- neous instruction preferences and varying quality levels? Furthermore, in terms of client-side updates, we find that extracting local features during client training is crucial for the global model. An effective training method should bet- ter capture features from datasets, perform sparse updates to the global model to preserve the base model ability and improve communication efficiency. However, in the current method, the ofate-of-the-art model algorithm found by Ye et al. (2024) simply uploads all the LoRA parameters. This indiscriminate parameter aggregation can result in the global model overfitting to specific client distributions while losing its general capabilities. Based on this, we further propose another question: 2) How can we preserve and transfer the general capabilities of LLMs while enabling them to adapt to domain-specific knowledge?
To address these challenges, numerous researchers develop diverse frameworks to train or fine-tune large language mod- els on geographically distributed private datasets (Li et al., 2024b; Fan et al., 2023; Xu et al., 2023b). In particular, researchers focus integrating the large language model Supervised Tunnig (SFT) module (Gunel et al., 2020; Ye et al., 2024) into the federated learning framework, such as FeederatedScop--LLM (Kuang et al., 2024), Shepherd (He et al., 2021) and OpenFedLLM (Ye et al., 2024). These studies utilize LoRA for fine-tuning the global model on geographically distributed datasets, enhancing the model capabilities while allowing it leverage knowledge from private datasets. However, these frameworks demonstrate excessive focus on classical federated learning algorithms, simply embedding the LLM supervised fine-tuning process within the federated learning framework without consider- ing the unique characteristics of language model training, such as the base ability and the domain -specffcc knowledge. This oversight potentially limits the effectiveness of feder- ated learning in scenarios where LLMs need to maintain their general capabilities while acquiring Nomain-specific knowledge. In a nutshell, the aforementioned discussions motivate us to rethink: How can we effectively incorporate intrinsic characteristics of the LLM, particularly the interac- tion between general and domain- domain-specifie knowledge, into federated fine-tuning frameworks?
To address the two issues mentioned above, this paper pro- poses an innovative framework to help the global model to absorb the instruction knowledge from heterogeneous clients, enhancing generalization. To address problem I, we propose the Consensus- Divverrggee Splitting, which com- bines consensus aggregation and divergence alignment, op- timizing the global aggregation process and improving the performance of the global model. For problem 2, during lo- cal updates, we introduce the Importance- Aware Updating, which focuses on uploading significantly altered parame- ters while disregarding minimal changes. The mask en- ables the global model accurately capture the direction of meaningful updates, addressing the problem of catastrophic forgetting and improving communication efficiency.
Preliminarily, numerous existing works perform LLM super- vised fine-tuning through FedAvg. In this paradigm, each local dataset operates an independent client that receives the LoRA parameters distributed from the global LLM in the server. These clients conduct training on their local datasets and transmit the updated gradients back to the global model. The global model then averages these gradient updates and iteratively repeats this process. However, when dealing with heterogeneous downstream instructions of different clients, simple averaging of client-side LoRA parameters may not achieve optimal performance, since the global LLM typi- cally exhibits bias toward specific local distributions rather than maintaining true global optimality. Considering that LLM training typically involves pre-training a foundation model followed by fine-tuning, we hypothesize the existence of two distinct factors: one that maximally preserves the
Our primary contribution in this paper can be summarized as following three points:
(1) We discover that L.LM fine-tuning can be decoupled into consensus factor and divergence factor, and identify their specific meaning in the SFT, which has practical significance in the federated large language model fine-tuning.
(2) We propose a novel framework for federated fine-tuning of large language models during the SFT stage. To address domain knowledge drift across clients, we adopt a hybrid optimization strategy that combines consensus- based aggre- gation with divergence alignment. This ensures balanced integration of knowledge from diverse downstream tasks. During the client uploading process, we focus making pa-
2

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
rameter updates effective and meaningful. Applying sparse updates to the global model, we preserve previously learned knowledge and maintain the model's generalization ability.
language models effectively addresses the critical challenge of leveraging private datasets while maintaining data privacy (Li et al., 2024b). Several recent works have made initial attempts in this direction. Fan et al. (2023) explored tradi- tional federated learning tasks for large language models fine-tuning, while Xu et al. (2023b) focused on improving computational efficiency on client devices. Additionally Kuang et al. (2024) investigated federated instruction tuning using FedAvg, and Ye et al. (2024) proposed framework encompassing both SFT and RLHF processes, combining large language models training processes with classical fed- erated learning algorithms using LoRA-base Abased fine-tuning, However, these existing methods exhibit several critical limitations: i) Some primarily focus on computational effi- ciency without addressing different downstream instructions heterogeneity in federated settings. ii) The interaction be- tween PEFT methods and federated learning remains poorly understood, especially in terms of parameter aggregation and convergence properties. To address these limitations, we propose a novel framework that specifically addresses these challenges by jointly considering the client upload process and global aggregation process in non-IlI settings. Through our careful design of the cunsensus-divergent mechanism and importance-awart parameter masking up- date strategy, our method not only achieves robust perfor- mance but also maintaining the efficiency benefits of LoRA- based fine-tuning.
(3) Through extensive experiments, we demonstrate that our framework achieves significant improvements in model generalization ability compared to existing approaches.
2. Related Work
2.1. Parameter Efficient Fine- Tine-Tuning for Large Language Models
The emergence of Large Language Models (LLMs) has demonstrated remarkable capabilities across various natural language processing tasks (Achiam et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Li et al., 2024a; Wang et al., 2024; Huang et al., 2023b; Bi et al., 2025a;c). However, adapting these models downstream tasks through con- ventional fine-tuning poses significant computational and resource challenges (Houlsby et al., 2019; Valipour et al., 2022; Bi et al., 2025b; Fang et al., 2025b; 2024). In re- sponse to these challenges, the development of Parameter Efficient Fine- Tuning (PEFT) methods has gained signifi- cant attention, as these methods aim to reduce computational overhead while maintaining model performance (Xu et al., 2023a). Current approaches can be categorized into three main directions: i) Adapter-based methods (Houlsby et al., 2019; Pfeiffer et al., 2020; He et al., 2021; Edalati et al., 2022) introduce additional neural network layers for task- specific adaptation. These methods insert trainable modules between the original transformer layers while keeping the pre-trained parameters frozen, thereby enabling efficient sarameter-sharing across tasks while maintaining model performance. ii) Prompting techniques (Petrov et al., 2023; Li & Liang, 2021) modify input sequences to guide model behavior through learnable or hard-coded prompts, effec- tively allowing models to leverage task specispecific knowledge without extensive fine-tuning of the base model parameters. iii) Low-rank adaptation approaches, particularly LoRA (Hu al., 2021), decompose weight updates into low-rank matrices (0' = 0 + BA), significantly reducing the number of trainable parameters while preserving the model ability to learn task task-specific adaptations. This approach is founded on the observation that the weight updates during fine-tuning often have a low intrinsic rank. While LoRA has become the dominant approach in SFT due to its efficiency, its effec- tiveness in distributed settings remains largely unexplored.
3. Methodology
3.1. Preliminaries
LoRA (Hu al., 2021) is employed to enhance the effi- ciency of fine-tuning by focusing the internal rank varia- tions that occur during parameter updates in the fine-tuning process. For fine-tuning a pre-trained model 0 E R LoRA keeps the pre-trained model parameter matrix frozen and utilizes two lower-rank matrices, 0A E æddrr and 0B R to represent the update A0. This process can be formulated as Equation (1):
6' 0++ + (0A x 0n).
(1)
A6
where r < (d, k) is the rank of the decomposition.
During training, 0 remains frozen, while 0 and 0B are updated. Initially, 0g is initialized using the Kaiming dis- tribution, and 0, set to a zero matrix, ensuring 0' = at the start, Any low-dimensional decomposition method can be applied for A0, as shown in Dettmers et al. (2023) and Rajabzadeh et al. (2024).
2.2. Federated Learning for Large Language Models
Federated learning has emerged as a promising paradigm for prisevi- collaborative training (Kairouz al., 2019; Wan et al., 2024; 2025; Yang et al., 2023a; Huang et al., 2023a; Wan et al., 2024). Its integration with large
In federated learning, multiple clients collaborate with a central server to train a global model on different datasets. In this framework, each client maintains its private dataset and performs local training, while the server coordinates the
3

---

Splitting with Importance-aware Updating for Heterogencous Federated Learning with Large Language Models
Server
3.2 Consensus-Divvrgenee Splitting
Consensus Aggregation
Consensus-Divergenee Split
Flatten & Norm
M M - (M, Mi-1)
N
Divergence Alignment
Cosine
Similarity
|M |
Vl. = V Vl-1
W (V W -
Inject
Distribution
Extract
3.3 Importance-Aware Updating
Calcutate Importancc
Copy
Momentum Importance Mask
Normalization
LoRA
General Speelfie
Train
Knowledge
Sparsely
Updating
Speefie
Updating
Momentum Weight
Client
Knowledge
Figure 2: Architecture illustration of the Splitting and Updating components. The key components shown the top (a) and bottom (b) of the image, with nodes of different classes marked in different colors. (a) Consensus- Consensus-Divergenne Splitting (Section 3.2) module splits the LoRA updates into consensus and divergence parts, applying consensus aggregation through delta averaging and divergence alignment using cosine similarity. (b) Importance-Aware Updating (Section 3.3) component computes every parameter importance based on the difference between the global model and the local model, selecting important knowledge from LLM clients for updates. Best viewed in color. Zoom in for details.
learning process through model aggregation. Specifically, during the i-th iteration, each client &: trains its local model 8% using its own dataset and sends the model updates to the server. The server then aggregates these local models to generate an updated global model 8', which is subsequently distributed back to the clients for the next round of train- ing, where client & obtains its next local model 0++1 as the formula below Equation (2):
where A x 0B represents the LoRA matrix split by A6.
However, two significant problems can arise with this train- ing method: (1) Naive parameter aggregation across clients can impair model convergence and degrade general capa- bilities, especially when dealing with heterogeneous down- stream instruction fine-tuning tasks (Guo al., 2024; Zhu et al., 2021; Cho al., 2024). (2) Not all knowledge ac- quired by clients from their private datasets contributes meaningfully to global model performance. Incorporating non-essential parameter updates can bias the global model toward suboptimal local minima. Moreover, indiscriminate parameter updates risk catastrophic forgetting, where the model loses previously acquired knowledge and suffers de- graded generalization capabilities (Dou al., 2024; Yang et al., 2024; Han et al., 2024; Zhu et al., 2024; Lee et al., 2022; Kim et al., 2024).
0
8i
Th n
where 0% is the model of k-thh client, ny is the size of the dataset for the k-th client, is the total dataset size, and N the number of clients.
In Fed LLM training, common approach to reduce com- munication costs is to utilize LoRA, which updates only subset of parameters instead of the entire model. In the FedAvg method with LoRA, the global model / is updated as follows Equation (3):
3.2. Consensus-Divergenee Splitting
A@t
Σ k= A
(3)
Motivation. Existing federated learning approaches for large language model supervised fine-tuning focus solely on aggregating client-side LoRA parameter matrices, failing to leverage interpretable nature of the LoRA or address the model drift that occurs when clients train on diverse instruction tasks (Zhu et al., 2021).
where A is the LoRA model of the global model at i-th round, and A6, represents the LoRA model of the k-th local client model i-th round.
can compute the global model by adding the 0a x On the frozen base model meaning that the global model o' after training is given by Equation (4):
Donsensus-Divergenee Splitting. To address the challenge of heterogeneous clients training on different downstream in- struction tasks in federated LLM supervised fine-tuning,
0' AA 0 x 0B.
(4)

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
consensus vector of client & iteration which captures the magnitude component of the decomposed LoRA parameters and M denote the global model consensus vector at itera- tion i. To aggregate these consensus vectors across clients, we compute the average change in updates relative to the previous global consensus state.
Similar
Different
(a) LoRA
(b) Divergence
(c) Consensus
Figure 3: Visualization of similarity matrices between different updating parameters of all the clients different layers are shown as follows: (a) Similarity of LoRA matrices without decomposi- tion, (b) Similarity of divergence matrices, and (c) Similarity of consensus matrices. The The results indicate that the differences between distributed LLM behavior become more pronounced after decomposition into consensus and divergence.
During local training, each client LLM extracts knowledge from its private dataset. We propose aggregate these updates by averaging the differences relative to the previ- ous global consensus, which ensures that new knowledge acquired by each client is incorporated uniformly into the global model, following the formula outlined below:
ΣN (M (MG -i
propose decomposing each client parameter matrix into mag- nitude and directional components inspired by Salimans & Kingma (2016). Our decomposition motivated by two key observations: (1) the magnitude of parameter updates often reflects general instruction capabilities across clients, while (2) the directional changes capture client-specific adapta- tions, which are specialized for specific downstream tasks. Formally, for the LoRA update matrix 6' E Rexd of client at i-th iteration, we split it using formula below, as Equa- tion (5):
M = M 1i-i
(6)
We can prove that this averaging strategy is optimal under certain conditions. Consider the consensus vector M of the k-th client at iteration i, which can be modeled as the formula below, Equation (7):
M = M + Ex.
where M represents the true global consensus and & ~ (0, I) denotes the noise term. For any linear unbiased estimator M = Σ ay-Mi satisfying Σ ag. = 1, its variance is given by Equation (8);
A K A II
AAG
1A0klle
(5)
Var(M) ΣX 0z0
(8)
divergenoe
Using the method of Lagrange multipliers, we can prove that the variance is minimized when aę = & for all A (see Appendix H). This theoretical foundation supports our practical approach of averaging the consensus vectors across clients to achieve optimal aggregation when the noise of the consensus is the same, which can be observed in Figure and proved in a experiment conducted in the Appendix F.
where || llc R" denotes the column-wise L2 norm that captures the scale of updates across the low-rank dimension and A6; ||00lle
represents the normalized directions.
To empirically validate our approach, we conduct experi- ments with four clients on non-IID datasets and perform aggregation using the FedAvg (McMahan et al., 2017b). We compute the similarity of the LoRA matrices across differ- ent clients and visualize the results using heat map, as shown in Figure.3. 3.
Ae denote the di- vergence vector of client l: at i-th iteration, which captures the divergence component of the decomposed LoRA param- eters and V denote the global model divergence vector at i-th iteration. To aggregate the divergence, we introduce an adaptive weighting scheme based on the pairwise cosine similarities between client updates.
Divergence Aggregation. Let V2
A0
Our analysis reveals that the client drift becomes more ap- parent after splitting parameter matrices into magnitude and direction components. Specifically, the differences between the clients are more pronounced in the Divergence Similar- ity Matrix, while the Consensus Similarity Matrix shows minimal variation between clients.
For each client divergence vector V we decompose it with LoRA down projection A E Rddr and LoRA up pro- jection Bj E R We perform vector-wise aggregation along their respective dimensions. Specifically, for the down projection matrix AL. we iterate over columns to obtain r- dimensional vectors, while for the up projection matrix B we iterate over rows to obtain d-dimensional vectors. For each vector slice, we compute the pairwise cosine similar- ities form a similarity matrix S p=xn following the formula as Equation (9):
Based on these observations above, we can draw two key conclusions: (1) The decomposition method provides better discrimination of client differences compared to the base LoRA method. (2) The decomposition effectively separates two distinct aspects of the model: the direction matrix cap- tures client-specific divergence characteristics, while the magnitude component reflects shared consensus patterns across heterogeneous clients.
Consensus Aggregation. Let M ||ABillc denote the
Spq cos(Up: Ug))
(9)
TT,, R
5

---

Splitting with Importance-aware Updating for Heterogencous Federated Learning with Large Language Models
where p,g q index the clients, and Up. Vg represent the corre-
Algorithm FedICU
sponding vector slices from clients p and
Input: Communication rounds T, number of clients N, local train- ing epochs E,, temperature т, learning rate 7, momentum coefficient /
We then compute each client k average similarity score across all other clients as Equation (10):
Output: Final global LoRA model parameters 0²
Initialize global LoRA model parameters 0 and client & local
8/
1 n-1 Ejye Skj.
(10)
model 0º = =
Initialize momentum buffers m = 0 for each client &
where j (1,,...,,)
for i = 1 to dooo
Client-side
The client contribution weights are determined through temperature-scaled softmax as Equation (11):
for cach client & E {I,..., N) in parallel do
for local epoch 1 to F do Update 0f, using local data
W/
exp(sx/T)
(xp(R)/T)
(11)
end
A0j < 0 0i // Equation (15)
Given the computed weights, we update the global diver- gence vector through a weighted aggregation of client up- dates relative the previous global state as Equation (12):
mi
- 3m 1- + (1
B)
á 16- 1-(100011
Equation (14)
alppr- 1144
G c
Equation (16)
o(lmn) 1 Ditt
V = Vi-1 + Ex= WA -V-
(12)
Glu) > Mol Equation (17)
W 0
otherwise
where V and Vi-1 denote the global divergence vectors at iterations i and - 1 respectively, V represents the diver- gence vector of client i at iteration wi is the computed weight for client k; and is the total number of clients.
of t W a (0 + mb) + (1 - --) @ 001
end
Server-side
for each client & do
Mj 1 lle Consensus of
Our weighting scheme ensures effective instruction learning during federated fine-tuning while accounting for client het- erogeneity. The approach balances consensus compatibility with client-specific update directions through the divergence weights, effectively mitigating knowledge drift while pre- serving diverse instruction knowledge across clients.
// Divergence Ve ll
end
M
+- Mi + t I [M 1D-MMM-) // Equation (6)
for each vector slice v Vi do
Spa f- cos Up. 11))
8k 6 N-I 14 Saz // Average similarity
Rationale for Weighting Our weighting scheme combines cosine similarity with temperature-scaled softmax, which can be theoretically formulated as the following optimiza- tion problem Equation (13):
expl>e/7) wk دn explay/t)
// Softmax weights
end
V Vi-1 1 + + uh (v - Vi-1) // Equation (12)
0' - M o
end return 0
min,. (W + T TE We log 32
(13)
where (L(w)p) represents the KL divergence between weight distribution w and uniform distribution p. and 5g denotes the average similarity score for client k.
3.3. Importance-Aware Updating
Motivation. Conventional approaches require uploading all local LoRA parameters after each training round. How- ever, during the SFT training process, the significance of parameter updates varies across different downstream tasks, with some parameters showing minimal significance. These unimportant parameters not only increase computational overhead during global aggregation but also can lead to catastrophic forgetting, degrading generalization perfor- mance (Dou et al., 2024; Yang et al., 2024; Han et al., 2024; Zhu et al., 2024: Liang et al., 2025; Huang et al., 2025). By selectively updating parameters for global model according to the importance, we can enhance model generalization during iterative SFT.
The first term encourages weight diversity, while the sec- ond term enforces similarity consistency, with temperature parameter controlling the trade-off between these objec- tives. This optimization formulation naturally leads to our softmax weighting scheme Equation (11). We choose cosine similarity as our metric due its scale-invariance and computational efficiency in high-dimensional spaces, making it particularly effective for divergence vectors in large language models.
Convergence Analysis. We provide theoretical guarantees for the convergence of our proposed algorithm. The details can be found in the Appendix G.
6

---

Splitting with Importance-aware Updating for Heterogencous Federated Learning with Large Language Models
Table I: Comparison with the state-of of the-ar methods, providing by framework (Ye et al., 2024) [Amiv24) for federated large language model tine-tuning, on different evaluation benchmark described in Appendix C. The best and second results are highlighted with bold and underline, respectively. Each value is accompanied by the improvement points compared to the benchmark model.
Generalization [Arxiv23]
Code [Arxiv21] Score
FIN. [Arxiv20]
Math [Arxiv2]]
Average Rank
Methods
MT-1
MT-2
MT-Final
Score
Score
Base
2.92
2,05
2.48
0.02
0.30
0.04
8
FedAvg (MeMahan al.)
5..8%
3.03147.8%
5..01%
0.05
0.35
0.05%
4,6
FedProx (Li al.)
4.29
3.30161.0%
3.791
1-00%%
0.28
0.131.22%
4.6
FedAvgM (Hsu al.)
2.95143.9% 5.82%%4.8%
50.8%
0.09 0-09%%50%
133.
0.101250%
4.0
Scaffold (Karimireddy et al.)
4.58 56.8%%
3.10151.2%
5..4%
0500%
0,30
3.021
3.4
FedAdam (Reddi et al.)
5..4%%
2.91142.00%
3.67
48.00%
0.09 4.50%
0,38 26.7%
0.121 3.1%
300%
4.2
FodYogi (Reddi et al.)
5..6%%
2.90141.5%
3.67 (6771.8.05 0%
0.06 1.061-300%$
0.38 122,7%
0,16 400%
4.6
Ours (FedICU)
65.4%
167.3%
66.3%
5000
0.35 16.7%
0.141350%
1.8
Importance-Aware Parameter Selection For client k at i-th iteration, where local parameters initialized from global parameters 0--1 and updated to 0f after training. We dual-metric importance
1.4
FedYog
1.3
introduce momentum-enhanee assessment mechanism that captures both the model gener- alization and specialization capabilities.
FedProx
1.2
FedAvgm
Loss
FedAdam
Inspired by parameter pruning methods (Han et al., 2015) and lottery ticket hypothesis (Frankle & Carbin, 2018), we propose that the magnitude of weights in the global model indicates the importance of individual parameters. We quan- tify the generalization importance through normalized and scaled weight magnitudes the training parameters from the global model from iteration i - 1, as shown below:
0.9
FedAvg
0.8
FedICD
0.7
Scaffold
0.5
Communication Epoch
I8 -1--(10''10)) "1), a(10' 1)+t
Figure 4: Different methods' training loss in Ye et al. (2024)
I = Sigmoid
(14)
benchmark of different epochs.
where (-) denotes the standard deviation, and μu()) com- putes the mean. We use the term e 10-6 6 ensure formula numerical stability.
first iteration, all local model parameters are uploaded to initialize the aggregation process.
{
1 if G > M[.]
To mitigate parameter oscillations between consecutive up- date rounds, we employ momentum-base based tracking of local model updates. The momentum term mi. for client at iteration is computed as:
Wl/
(17)
otherwise.
The sparse parameter update is implemented using Equa- tion (18). This selective update strategy prevents overfitting to the downstream instruction distribution of any specific client.
m 3m + +1 (1- 1)
(15)
where @ is the momentum coefficient, and mi represents the momentum term at iteration i.
0 0 W & (0i-++ +1 (1- o
(18)
where @ represents element-wise multiplication, and A is the model to be uploaded.
We then define the specialization importance G to quantify- ing the degree of local adaptation by measuring the relative magnitude of these momentum-smone ssoothhdd updates, normal- ized using statisties from the previous iteration:
Convergence Analysis. We provide theoretical guarantees for the convergence of proposed algorithm. The details can be found in the Appendix I.
[mn] (-(((((())))) a(mp - De
G = Sigmoid(
(16)
4. Experiment
During the client update process, parameters are selected for global model updates when their specialization importance exceeds their generalization importance. The parameter selection is determined by Equation (17). Note that in the
4.1. Experimental Setup
Datasets. Following Ye et al. (2024), we train our model on the four datasets Taori et al. (2023), Xiang Yue (2023),

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Chaudhary (2023) and Yang et al. (2023b). We provide details introduction in Appendix A.
Table 2: Ablation study two key components mentioned in the passage, Consensus- Consensus-Diverge Divergence Splitting (Splitring column in the table) and Important- Awaae Updating (Updating column in the table). The best results are highlighted in bold.
Framework Setup. We conduct experiments using Nous- Research -lama-2-7b-he model over 200 rounds. The model performance evaluated using the benchmark metrics pro- vided Ye et (2024).
Generalization
Updating Splitting
MT-1
MT-2
Final
4.58
3.03
3.80
Counterparts. We compare our framework FedICU with several state-of-the-an methods (Ye al., 2023) in the benchmark Ye et al. (2024): (1) Base Model without SFT. (2) FedAvg (McMahan et al., 2017b). (3) FedProx. (Li et al., 2018). (4) Scaffold (Karimireddy et al., 2019). FedAvg. (Hsu et al., 2019). (6) FedAdam (Reddi et al., 2020). (7) Fed (Reddi et al., 2020). We provide a details introduction of different methods in Appendix A.
4.55
3.25
3.90
x
4.70
3.41
4.06
4.83
3.43
4.13
4.1
Score
Score
3.9
Implement Details. We present experimental setup from three aspects:
Dataset Split: Each client is assigned one of the four domain-specific datasets deseribed in Appendix A, ran- domly sampling 5000 labeled examples for fine-tuning.
3.8
0.08
0.1
0.2
0.26
3.7
0.8
0.85
0.9
0.95
(a) Temperature
(b) Momentum
Figure 5: Klyper-paramete evaluation on MT-Bench, focusing (a) the temperature 7 in Equation (11) of Consensus-Divergenn Splitting and (b) the momentum coefficient 8 in Equation (15) of Importance- Aware Updating. Higher scores indicate better performance.
Training Setting: We follow the benchmark Ye et al. (2024) default hyperparameters' configuration to set up experiment environment. We provide details in- troduction in Appendix D. All experiments are repeated three times ensure statistical significance.
the results, demonstrating that both components contribute to performance improvements, with their combination yield- ing optimal results.
Evaluation Metrie: (1) Generalization: the first turn score from MT-Bench (Zheng al., 2023) (2) Contex- tual Understanding: the final score from MT-Bench. (3) Code: Human Eval (Chen et al., 2021) (4) Financial: MMLU benchmark (Hendrycks et al., 2020) (5) Math: The GSM&k benchmark (Cobbe et al., 2021). We provide a details introduction in Appendix
Hyperparameters Study. We analyze the sensitivity of the key hyperparameters of our method using MT-Beech, as shown in Figure 5. Results indicate that model perfor- mance decreases with increasing temperature above 0.1, though remaining relatively stable within certain range. The momentum factor shows minimal impact at higher val- ues, demonstrating the robustness of our approach. Based on these findings, we adopt a temperature of 0.1 and momentum factor = 0.9 in our experiments, as they perform best in the hyperparameters study.
Performance Comparison. Table shows the performance comparison, while Figure illustrates the training loss. Our results demonstrate FedICU superior performance in feder- ated L.LM training. Traditional methods such as FedAvg and FedProx struggle to effectively aggregate client consensus and align divergences, leading to the degradation of model performance. contrast, FedICU successfully maintains the model generalization capabilities under these conditions while achieving superior performance across diverse down- stream test sets. Specifically, our framework demonstrates significant improvement in generalization and consistently maintains an advantage across various specialized domains. We also conduct more experiment on the different model architecture in Appendix
Parameter Distribution in FL LLM. To demonstrate the rationale behind the Important- Aware Updating component, we conduct study on the importance of the covered param- eters shown in Table 3. The results show that Important- Aware Updating component physical significance. In the table, the parameter High-I remains stable while decreasing the parameter High-G proportions indicate better identifica- tion of core domain parameters. Declining overlap suggests that the Important-Aware Updating component has clearer functional partitioning, effectively separating general and domain-specites knowledge.
4.2. Diagnostic Experiments
A Ablation of Key Components. We evaluate the contribu- tion each component using MT-Bench across four diverse datasets with optimized hyperparameters. Table 2 presents
The Efficient the Binary Mask. We conduct an ablation study to investigate the impact of momentum based param-
8

---

Splitting with Importance-aware Updating for Heterogencous Federated Learning with Large Language Models
Table Parameter distribution in LLM. Overlap indicates im- portant parameters globally and locally, High-I represents parame- ters emphasizing general ability, and High-G represents parameters emphasizing domain capability.
erated large language model training, the underlying prin- ciples of our approach have broader implications. Future research could investigate the application of these concepts to other domains within federated learning, potentially lead- ing to new insights and methodological advances.
Class/Round
5
15
25
35
45
55
65
Overlap
0.24
0.16
0.13
0.12
0.11
0.10
0.08
High-I
0.48
0.46
0.46
0.45
0,45
0,44
0.44
6. Conclusion
High-G
0.50
0.40
0.38
0.38
0.37
0.36
0.35
In this paper, we introduce FedICU, framework that em- ploys consensus-divergene splitting for effective LLM fine- tuning in federated settings. Our method decouples LoRA updates into divergence and consensus components, en- abling fine-grained control over model updates. At the server level, our approach aligns divergence using cosine similarity and aggregates consensus, significantly improv- ing the global model generalization ability. At the client level, implement an importance-aware updating method that selectively updates parameters, preventing overfitting and preserving the global model generalization capabili- ties. Extensive experiments demonstrate the effectiveness and robustness of our approach. We believe this work pro- vides valuable insights for future research in federated large language model fine-tuning, particularly for heterogeneous downstream tasks.
Table 4: Experiment results of ablation study about Importance- Aware Update. MT-1 shows the model general capability, while MT-2 shows the model level of contextual understanding. MT- Final is the metric combining both of the two indicators.
Momentum
Smooth
MT-1
MT-2
MT--inall
4.59
3.20
3,90
x 4.60
3.33
3.97
/
4.65
3.37
4.01
/ (Ours)
X (Ours)
4.83
3.43
4.13
eter selection and continuous-valued parameter weighting on model generalization capability, as shown in the table below Table 4. In the Momentum column, we indicate whether we include the momentum component to smooth the mask construction process. In the Smooth column, we use smoothing mask approach, when G[u] > [(e] making Wlu] = min (1, + ; to filter uploaded parameters and their weights. The results validate that Important-Awant Updating component is a simple and effective method.
Glol
A Acknowledge
This research was financially supported by the Open Re- search Fund from Guangdong Laboratory of Artificial Intel- ligence and Digital Economy (SZ), under Grant No. GML- KF-24-10, the National Natural Science Foundation of China under Grant (62361 166629, 62176188, 62225113, 623B2080), the National Key Research and Development Program of China (2024YFC3308400), the Wuhan Uni- versity Undergraduate Innovation Research Fund Project, and the Hubei Postdoctoral Talent Introduction Program (2044HBBHDD070), The supercomputing system at the Supercomputing Center of Wuhan University supported the numerical calculations in this paper.
4.3. Communication Cost
Regarding FedICU, since LoRA splitting and merging can be performed locally on either the client or server, it doesn't 't introduce additional communication overhead compared to standard federated learning (Ye et al., 2025b;a). For the Importance-Aware Updating component, assuming there are K clients, each with N parameters, and in the mask built by the parameter importance selection upload compo- nent, the proportion that needs to be uploaded is a. So the communication cost of standard federated learning would be Cetd = O(K * N). while the communication cost of FedICU is Cimp o (ΣK N x au). where ag 1, which demonstrates the communication savings of FedICU.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5. Discussion and Limitation
(1) While our method leverages LoRA, which may not achieve the same performance levels as full parameter fine- tuning. Despite these trade-offs, LoRA significantly reduces computational requirements and enables rapid adaptation to diverse downstream tasks. Future work could focus on bridging this performance gap while maintaining LoRA computational advantages.
References
Achiam, J., Adler, S., Agarwal, S.,, Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint, 2023.
(2) Although our method is primarily developed for fed-
9

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Alemohammad, S., Casco-Rodriguez, J., Luzi, L.,, Hu- mayun, A. I,, Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative models go mad. arXiv preprint, 2023.
Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., al. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. In ACL, pp. 1932-1945, 2024.
Bi, J., Wang. Y., Chen, H., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Llava steering: Visual instruction tun- ing with 500x fewer parameters through modality linear representationnssteenni 2025a. URL https:: https://arxiv.. //rxxv.. arg/abs/2412.//
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint, 2024.
Edalati, A., Tahaei, M., Kobyzev, I., Nia, V. P., Clark, J. J., and Rezagholizadeh, M. Krona: Parameter efficient tun- ing with kronecker adapter. arXiv preprint, 2022.
Bi, J., Wang, Y. Yan, D., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection, 2025b. URL https: /arxiv..vvv org/ hbs//abs//t 2502. 12119.
Fan, T., Kang, Y., Ma, G., Chen, W., Wei, W., Fan, L., and Yang, Q. Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint, 2023.
Bi, J.. Yan, D., Wang. Y., Huang, W., Chen, H., Wan, G., Ye, M., Xiao, X., Schuetze, H., Tresp, V., and Ma, Y. Cot- kinetics: A theoretical modeling assessing Irm reasoning process, 2025c. URL https: /ttp/aaxvvv//.//////e..//aaaa..a org/ abs/ 2505. 13408.
Fang, J., Jiang, H., Wang, K., Ma, Y., Jie, S., Wang, X., He, X., and Chua, T.-S. Alphaedit: Null-space constrained knowledge editing for language models. ICLR, 2025a.
Chaudhary, S. Code alpaca: An instruction-following llama model for code generation. https: /tt::://gitttt com/ sahi 220114 codea paca, 2023.
Fang. X., Fang, W., Liu, D., Qu, X., Dong, Zhou, P., Li, R., Xu, Z., Chen, L.,, Zheng, P., et al. Not all inputs valid: Towards open-set video moment retrieval using language. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 28-37, 2024.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint, 2021.
Fang. X., Easwaran, A., Genest, B., and Suganthan, P. N. Your data is not perfect: Towards cross-domain out-of- distribution detection in clas-immallanced data. Expert Systems with Applications, 2025b.
Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Qian, C., Chan, C.-M., Qin, Y., Lu, Y., Xie, R., et al. Agentverse: Facili- tating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint, 2023.
Frankle, and Carbin, M.
The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint, 2018.
Cho, Y. J., Liu, L., Xu, Z., Fahrezi, A., and Joshi, G. Het- erogeneous lora for federated fine-tuning of on-device foundation models. In EMNLP, pp. 12903-12913, 2024.
Gunel, B., Du, Conneau, A., and Stoyanov, V. Supervised contrastive learning for pre-trained language model fine- tuning. arXiv preprint, 2020.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint, 2021.
Guo. Z., Zhang, Y., Zhang, Z., Xu, Z. and King, I. Fedlfe: Towards efficient federated multilingual mod- eling with lora-based language family clustering. In Find- ings @ NAACL, pp. 1519-1528, 2024.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: efficient finetuning of quantized 1lms. arXiv preprint, 52:3982-3992, 2023.
Han, J., Du, L.,, Du, H., Zhou, X., Wu, Y., Zheng, W., and Han, D. Slim: Let 11m learn more and forget less with soft lora and identity mixture, arXiv preprint. 2024.
Devine, P. Tagengo: A multilingual chat dataset. arXiv preprint, 2024.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. NeurlPS, 28, 2015.
Didolkar, A., Goyal, A., Ke, N. R., Guo, S., Valko, M., Lill- icrap, T., Rezende, D., Bengio, Y., Mozer, M., and Arora, S. Metacognitive capabilities of llm:: An exploration in mathematical problem solving. arXiv preprint, 2024.
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G. Towards unified view of parameter-effiin transfer learning, arXiv preprint, 2021.
10

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint, 2020.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. Scaffold: Stochastic con- trolled averaging for on-device federated learning. arXiv preprint, 2(6), 2019.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nip. In ICML, pp. 2790-2799. PMLR, 2019.
Kim, S., Jeong, M., Kim, S., Cho, S., Ahn, S., and Yun, S.- Y. Feddr+: Stabilizing dot-regression with global feature distillation for federated learning. arXiv preprint, 2024.
Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of mon-identical data distribution for federated visual clas- sification, arXiv preprint, 2019.
Kuang, W., Qian, B., Li, 7 Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., and Zhou, J. Federatedscope- Ilm: A comprehensive package for fine-tuning large language models in federated learning. In KDD, pp. 1270-5271, 2024.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.. and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint, 2021.
Lee, G., Jeong, M., Shin, Y., Bae, S., and Yun, S.-Y. Preser- vation of the global knowledge by not-true distillation in federated learning. NeurIPS, 35:38461-38474, 2022.
Huang, W., Ye, M., and Du, B. Learn from others and be yourself in heterogeneous federated learning. In CVPR, 2022.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F,, Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava-onevision: Easy visual task transfer. arXiv preprint, 2024a.
Huang, W., Wan, G., Ye, M., and Du, B. Federated graph semantic and structural learning. In Proceedings of the Thirty-Second International Joint Conference on Artifi- cial Intelligence, pp. 3830-3838, 2023a.
Li, L., Fan, Y., Tse, M., and Lin, K.-Y. A review of appli- cations in federated learning. Comput. Ind. Eng., 149: 106854, 2020.
Huang, W. Ye, M., Shi, Z., and Du, B. Generalizable hetcrogeneous federated cross- corsscoorool and instance similarity learning. TPAMI, 2023b.
Li, S., Ye, F., Fang, M., Zhao, J., Chan, Y.-H., Ngai, C.-H., and Voigt, T. Synergizing foundation models and federated learning: A survey. arXiv preprint, 2024b.
Huang, W., Ye, M., Shi, Z., Li, H., and Du, B. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023c.
Li, T., Sahu, A. K., Sanjabi, M., Zaheer, M., Talwalkar, A., and Smith, V. On the convergence of federated op- timization in heterogeneous networks. arXiv preprint, 2018.
Huang, W., Liang, J., Shi, Z., Zhu, D., Wan, G., Li, H., Du, B., Tao, D., and Ye, M. Learn from downstream and be yourself in multimodal large language model fine-tuning. In ICML, 2025.
Imani, S., Du, L., and Shrivastava, H. Mathprompter: Math- ematical reasoning using large language models. arXiv preprint, 2023.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint, 2021.
Liang, J., Huang, W., Wan, G., Yang, Q., and Ye, M. Lo- rasculpt: Sculpting lora for harmonizing general and spe- cialized knowledge in multimodal large language models. In CVPR, 2025.
Jiang, H., Fang, J., Zhang, N., Ma, G., Wan, M., Wang, X., He, X., and Chua, T.-s. Anyedit: Edit any knowledge encoded in language models. ICML, 2025.
Kaddour, J., Harris, J., Mozes, M., Bradley, H.,, Raileanu, R., and McHardy, R. Challenges and applications of large language models. arXiv preprint, 2023.
Liu, H. Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In CVPR, 2023.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G. Cummings, R., et al. Advances and open problems in federated learning. arXiv preprint, 2019.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and
y Arcas, B. A. Communication-effccinnn learning of deep networks from decentralized data. In AISTATS, pp. 1273- 1282, 2017a
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint, 2020.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and learning of deep
y Arcas, B. A. Communication-efficee networks from decentralized data. In AISTATS, pp. 1273- 1282. PMLR, 2017b.

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Neur/PS, volume 36, pp. 50358-50376. Curran Associates, Inc., Curran Associates, Inc., 2023.
Wan, G., Shi, Z., Huang, W., Zhang. G., Tao, D., and Ye, M. Energy-based backdoor defense against federated graph learning. In International Conference Learning Representations, 2025.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing language model's pereeption of the world at any resolution. arXiv preprint, 2024.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NeurlPS, 35:27730-27744, 2022.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint, 2022.
Petrov, A., Torr, P. H., and Bibi, A. When do prompting and prefix-tuning work? theory of capabilities and limitations. arXiv preprint, 2023.
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far camels go? exploring the state of instruction tuning on open resources. NeurIPS, 36: 74764-74786, 2023.
Pfeiffer, J., Kamath, A., Rückléé, A., Cho, K., and Gurevych, I. Adapterfusion: Non-destructive task composition for transfer learning, arXiv preprint, 2020.
Rajabzadeh, H., Valipour, M., Zhu, T., Tahaci, M., Kwon, J., Ghodsi, A., Chen, B., and Rezagholizadeh, M. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning. arXiv preprint, 2024.
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint, 2023.
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. Adaptive federated optimization. arXiv preprint, 2020.
Xiang Yue, Xingwei Qu, G. Z. Y.F. W. H. H. S. C. Mammoth: Building math generalist models through hy- brid instruction tuning. arXiv preprint. 2023.
Salimans, and Kingma, D. P. Weight normalization: A simple reparameterization accelerate training of deep neural networks. NeurlPS, 29, 2016.
Xu, L., Xie, H., Qin, S.-Z. J., Tao, X., and Wang, F. L. efffiiter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint, 2023a.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stan- ford alpaca: An instraction-following llama model, 2023. Available at: https:/ http:///gghhbb..t/h.te. ctw/:/tsuutccataeoooaaaooaoauu lab/ stanford_a paca.
Xu, M., Cai, D., Wu, Y., Li, X., and Wang, S. Fwdllm: Efficient fedllm using forward gradient. arXiv preprint, 2023b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint, 2023.
Yang, B. Chen, J., and Ye, M. Towards grand unified representation learning for unsupervised -nsbble-infrared person re-identification. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 11069-11079, October 2023a. Yang, H., Liu, X.-Y,, and Wang, C. D. Fingpt: Open-source financial large language models. In Finl.L@ 2023b.
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint, 2022.
Villalobos, P., Sevilla, J.. Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint, 2022.
Yang, Y., Li, X., Zhou, Z.,, Song, L., Wu, J., Nie, L., and Ghanem, B. Corda: Context-oriented decomposition adaptation of large language models. arXiv preprint, 2024.
Wan, G., Huang, W., and Ye, M. Federated graph learn- ing under domain shift with generalizable prototypes. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 15429-15444 2024.
Ye, M., Fang. X., Du, B., Yuen, P. C., and Tao, D. Hetero- geneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1 2023.
12

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
Ye, M., Rong, X., Huang, W., Du, B., Yu, N., and Tao, D. A survey of safety large vision-language models: Attacks, defenses and evaluations. arXiv preprint, 2025a.
A. Datasets
Taori et al. (2023): The dataset used for fine-tuning the Alpaca model. Alpaca is a dataset of 52.000 instructions and demonstrations generated by OpenAl text-davinci- 003 engine.
Ye, M., Shen, W., Du, B., Snezhko, E., Kovalev, V., and Yuen, P. C. Vertical federated learning for effectiveness, security, applicability: A survey. ACM Computing Sur- veys, 57 2025b.
Xiang Yue (2023): The dataset concerning the math field. Math instruct is compiled from 13 math rationale datasets, six of which are newly curated by this work. It uniquely focuses on the hybrid use of chain-of-thought (CoT) and program of thought (PoT) rationales, and ensures exten- sive coverage of diverse mathematical fields.
Ye, R., Wang. W., Chai, J., Li, D., Li, Z., Xu, Y., Du, Y., Wang, Y., and Chen, S. Openfedllm: Training large lan- guage models decentralized private data via federated learning. In KDD, pp. 6137-6147, 2024.
Zheng, L., Chiang, W.-L.,, Sheng. Y., Zhuang, S., Wu, Z., Zhuang. Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurlPS, 11146595-46662. 2023.
CodeAlpaca-205. The dataset concerning the code field. The 20K cnatruction-oollwinng data generated by the tech- niques Instrutt (Wang et al., 2022), with some modi- fications by author of the datasets.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic for- getting multi-modal large language models. In ICML, 2024.
FinGPT: The specialized financial datasets used in Fin- GPT (Yang et al., 2023b).
B. Counterparts
Zhu, H., Xu, J., Liu, S., and Jin, Federated learning on non-iid data: survey, Neurocomput. 465:371-390, 2021.
(1) Base Model without SFT.
(2) FedAvg (MeMahan al., 2017b). The standard feder- ated averaging algorithm, where updates from all clients are averaged at the server.
Zhuang, W., Chen, C., and Lyu, L. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint, 2023.
(3) FedProx (Li et al., 2018). An extension of FedAvg that introduces a proximal term to tackle heterogeneity across clients.
(4) Scaffold (Karimireddy al., 2019). A control variate- based method designed reduce the impact of client drift in federated learning with non-IID data.
(5) FedAvgM (Hsu et al., 2019). A cameenuumbbseee vari- ant of FedAvg, which integrates server-side momentum into the federated learning process.
(6) FedAdam (Reddi et al., 2020). federated version of the Adam optimizer. adapts the learning rates the server side using first and second-order moments of gradients, aim- ing to provide better performance in challenging federated settings.
(7) Yogi (Reddi et al., 2020). An adaptive federated optimization method that improves robustness and conver- gence federated learning, which effectively handles the challenges of non-IID data and system heterogeneity in federated settings.
C. Evaluation Metric
(1) Generalization: We use the first turn score from MT- Bench (Zheng et al., 2023) as the primary evaluation metric to assess the general performance of different models (Ye
13

---

Splitting with Importance-aware Updating for Heterogencous Federated Learning with Large Language Models
Table The performance of FedICU and other methods applied the Mistral-7B. MT-1 shows the model general capability, while MT-2 shows the model level of contextual understanding. MT- Final serves comprchensive metric that combines both of the aforementioned indicators.
Table The μ and 0 consensus parameters' delta update. μ rep- resents the mean of the consensus vector updates, and represents the fluctuations during the consensus update process.
Client
1,452974
1.53*10-1
Method
MT-1
MT-2
MT-Final
1,452972
1.54*10
Base
4.10
3.29
3.69
1,452974
1.56*10-1
FedAvg
5.48
3.80
4.64
1,452973
1.54#10-4
Scaffold
5.46
3.92
4.69
FedYogi
5.54
3.85
4.70
model architectures. The results of these supplementary ex- periments are in the Table 5, The results show that FedICU also demonstrates excellent performance across models with different architectures.
FedProx
5.50
3.83
4.67
FedAdam
5.56
3,91
4.73
FedAvgM
5.55
3.73
4.64
Ours
5.58
4.11
4.84
F. The experiment on consensus noise updating action.
al., 2024). This score is the most critical in the overall evaluation.
In Figure we use cosine similarity to support the assump- tion that "consensus noise" the same. To further support conduct experiment to explore the mean and variation of the consensus vector updates for each client under the condition independent updates across clients, as shown in the table Table 6. From the table, the similarity of μ and across different clients indicates that the consensus noise is consistent.
(2) Contextual Understanding: We use the final score from MT-Bench to evaluate the model ability to understand context. MT-BBecch comprises two turns of conversation, making it suitable for contextual testing.
(3) Code: We use Human Eval (Chen al., 2021) evalu- ate the model coding capabilities.
(4) Financial: We utilize the MMLU benchmark (Hendrycks et al., 2020) to evaluate the model financial knowledge, specifically selecting the finance-related do- mains for this assessment.
G. Splitting Convergence Analysis
Let F(0) denote the global objective function that measures the model performance across all clients, where 6 E Rd represents the global model parameters. We prove that our Consensus-Divergeen Splitring method converges under standard assumptions.
(5) Math: The GSM8k benchmark (Cobbe al., 2021) is used to test the model mathematical abilities.
D. Training Settings
Assumption G.1 (Smoothness). F(0) is L-smooth: for all 8,6' Rd:
For all experiments, we use the following hyperparameters settings:
VF - VF(O') - LI 0'|1
(19)
Learning rate: 5e
Assumption G.2 (Bounded Gradient). There exists G >> for all
0
Training rounds: 200
VVF(O)) S
(20)
FedProx proximal term μ.: 0.01
Assumption G.3 (Unbiased & Bounded-Variance Variance Gradient Estimates). During round i, each client produces update
Yegi and FedAdam:
A 8 0¹. The local gradient estimator is:
Server learning rate:
T: le -3
1
9h
A
(21)
FedAdam momentum coefficient 3: 0.9
17'
where 17' is the server step size. We assume:
E. More Experiments on Different Model Architecture
Unbiasedness:
We conduct supplementary experiments with Mistral-7B to verify that our method effective across broader range of
E
W
K 0 =
VF(0"..
(22)
14

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
where w 2 0 # are similarity-based weights.
H. Optimality of Averaging Strategy
Bounded variance: For 2² 2 0:
Here we prove that direct averaging consensus vectors yields the optimal linear unbiased estimator. We formalize the problem follows: Given the consensus vector Mj of
E VF(O')1 VP(0))
(23)
client & iteration i:
Assumption G.4 (Strong Convexity (Optional)). ju-strongly convex (fr 0): for all 0' Rd:
F(0) is
M = M + Ek.
(30)
F(0") 2 F()) + (F(()) 0)
H 110'' 011.. (24) 2
where M represents the true global consensus and &a NO, o21) denotes the noise term. For any linear unbi- ased estimator M = Σk=1 ag with the constraint k= ag 1, its variance is:
Note that Assumption G.4 is optional. If F is not strongly convex (e.g., just convex or nonconvex), we prove conver- gence to stationary point. If F is strongly convex, we show convergence to the unique global optimum.
Var
az
(31)
At iteration i, FedICU aggregates local models via:
k=1
N
To find the optimal weights og that minimize the variance, we use the Lagrange multiplier method. The Lagrangian is:
g
+
Σ
(0) 0¹) = wh
(25)
We employ cosine decay learning schedule:
ON, A)
aio + A
Ok 1). (32)
k=1
k=1
1 + cos( mi
7/mxx
Tiax
(26)
Taking partial derivatives:
Theorem G.5 (Unified Convergence of FedICU). Under Assumptions G.I-G.3, let {0'} be produced by (25). Then: For smooth possibly nonconvex F:
ÖL 20z0²
OL
2² +
ak
1 = 0. (33)
Oak
0x
From the first equation:
min osieT
(27)
VT
Or
(34)
20
For u-strongly convex F, FedICU converges to the unique global optimum 6 with rate depending μu, L, o².
This shows all og are equal. Combined with the constraint:
Proof. By L-smoothness:
No
(35)
L
Km
F(0++1)
-
100 8/1/2.
Therefore:
(28)
Using update rule and bounded-variance assumption:
k= 1,2, 1,2,.... N. N
(36)
T-1 0
min (II < 0 osieT
+
(7)
The second derivative:
E 17
1=0
(29)
8 L
20 >
(37)
When Σ ' R(VT) and Ei 0(1), we get (01 VT) rate. Under strong convexity, additional terms yield |0 1*1| 0.
da
confirms this is indeed minimum. This proof demonstrates that equal weights ag = x minimize the variance of the linear unbiased estimator, justifying our averaging strategy:
For finite-T training with cosine decay schedule, our anal- ysis shows that the model converges as the learning rate naturally decays near-zero the end of training.
M = Mi-1 + E - Mi-1)
(38)
N
15

---

Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models
I. Updating Convergence Analysis
For jeStroogly convex F, convergence to the unique global optimum 0* is achieved with rate depending on μ4, L, a² and 6.
We analyze the convergence of our Importance-Aware Up- dating mechanism. Following the same assumptions and notations in Appendix G, let 0 E 12 denote the global model parameters and F(0) be the global objective function:
Proof. By L-smoothness:
-0- T. 1 11- (49)
N F(O)
F(0) S F(0)-1 ((V
Fe(O),
(39)
N 1=1
Substituting the masked update:
where Fj is the local loss at client k.
For parameter selection and updates, we define importance metrics for each parameter 0:
0¹ -0-i-11
Σ (m) Ai).
(50)
00' |o]| 11100-11))
[u] Sigmoid
(40) The first term averages the gradient (by unbiased- ness), while the masking error A is bounded by 8² Follow- ing the analysis in Appendix G, this additional error term contributes only a constant factor to the convergence rate:
(10--11) +
| 1e( am(mm a(mp +e
G(v] Sigmoid
(41)
The binary mask W {0, 1 determines which parameters to update, the formula Equation (42):
T-1 +0
E (F) osKT
min
+0((8))
I i== 17
1=
1, 0, otherwise.
1, if Glu) π[.],
(51)
W[o]
(42)
With cosine decay schedule, when Li=o 17' (VVT) and E/-1(1/ = O(1), we achieve the O(1/vT) rate. For the strongly convex case, the additional error term similarly affects only the constants in the convergence rate to 0*
Each client uploads a masked model:
t = W @ + + (1 W) 00-11
(43)
These results show that our importance-aware parameter updating maintains convergence guarantees while allowing adaptive sparsification of updates.
We rewrite this update as:
0 = 0' 1 + W @ m4 = 0 + m Ai.
(44)
where A (1 - W) mt represents masked-out updates.
At iteration i, the server aggregates:
1 N
N E (m) - - (45)
Σ
0 0i-1
We employ cosine decay learning schedule as in the previous section:
1 + cos( TiNx
7max
(46)
Theorem L.I (Convergence). Under Assumptions G. I-G.3, let {0'} be generated by the masked update rule. If the masking error bounded:
N
ElI N
58.
(47)
1=1
Then for smooth F:
1
min
E
VF
0
(48)
0si<T
16


Reviews:
[{"review_id": "xFB0iYRNXi", "reviewer": "Reviewer_fzFX", "summary": "This paper introduces FedICU, a framework designed to enhance fine-tuning of Large Language Models in Heterogeneous Federated Learning. \n\nThe paper presents two core innovations:\n- **Consensus-Divergence Splitting**: A technique that decomposes client updates into **consensus (common capabilities)** and **divergence (domain-specific features)**. This allows the global model to **retain fundamental general knowledge** while **capturing client-specific information** effectively.\n- **Importance-Aware Updating**: A method that evaluates the importance of each parameter update to **reduce unnecessary updates**, **improve communication efficiency**, and **prevent catastrophic forgetting**.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fzFX:\n\nThank you for your valuable review of our paper. We hope our responses will address your concerns and improve our score.\n\n**Q1: Clarify how inactive parameter contribute to catastrophic forgetting.** (Claims And Evidence)\n\n**A1:** We apologize for the confusion in our original statement. In FedICU, we refer to inactive parameters, not inactive clients. In neural networks, only a small portion of parameters play crucial roles. To prevent clients focusing solely on local datasets and uploading all parameters that cause forgetting, FedICU calculates parameter importance and sparsely uploads parameters to balance the global model's general and domain capabilities. In the example, FedICU will upload parameters with more importance during training, balancing contributions from both clients. We conduct an experiment with results shown in the table below, demonstrating that FedICU is more stable than baseline methods and selects a substantial number of parameters from Client 1 for uploading rather than ignoring them.\n\n*Table: The accuracy of FedAvg and FedICU on CIFAR-10.*\n|Method/Round|78|79|80|81|82|83|84|\n|-|-|-|-|-|-|-|-|\n|FedAvg|41.37|42.38|42.54|41.66|42.48|40.13|40.84|\n|FedICU|45.57|46.08|46.29|42.42|46.25|46.54|46.75|\n\n*Table: The selection ratio in FedICU on CIFAR-10. The ratio means the proportion of parameters for upload. **It shows that FedICU will not ignore Client 1 for the similarity with the global model**.*\n|Client/Round|78|79|80|81|82|83|84|\n|-|-|-|-|-|-|-|-|\n|1|0.4853|0.4852|0.4866|0.4864|0.4866|0.4852|0.4855|\n|2|0.4850|0.4851|0.4849|0.4855|0.4849|0.4852|0.4857|\n\n**Q2: The physical meaning of Eq.14, 16 is not intuitive.** (Methods And Evaluation Criteria)\n\n**A2:** The internal logic of Eq. 14, 16 is that we define parameter importance based on the magnitude of updates[1, 2]. To compare client and global, we perform normalization and introduce momentum to make it stable, deriving Eq.14, 16. We add an experiment to show the distribution of importance parameters, with results shown in the table below.\n\n*Table: Parameter distribution in FL LLM. Overlap indicates important parameters globally and locally, High-I represents parameters emphasizing general ability, and High-G represents parameters emphasizing domain capability.*\n|Class/Round|5|15|25|35|45|55|65|\n|-|-|-|-|-|-|-|-|\n|Overlap|0.24|0.16|0.13|0.12|0.11|0.10|0.08|\n|High-I|0.48|0.46|0.46|0.45|0.45|0.44|0.44|\n|High-G|0.50|0.40|0.38|0.38|0.37|0.36|0.35|\n\nThe results show definition's physical significance. High-I remains stable while decreasing High-G proportions indicate better identification of core domain parameters. Declining overlap suggests clearer functional partitioning, effectively separating general and domain-specific knowledge.\n\n**Q3: The justification for Eq.17, 18 should be reinforced to strengthen validity.** (Methods And Evaluation Criteria)\n\n**A3:** Sparsely updating model is an approach to mitigate catastrophic forgetting [2, 3]. In FedICU, We construct masks and sparsely update models (Eq.17, 18) to mitigate forgetting. To further demonstrate the effectiveness, we conduct an experiment as shown in the table below. For Momentum, we test whether to incorporate momentum components, and for Smooth, whether to use binary or smooth updates based on the formula below. Results show our method achieves the best performance, demonstrating its effectiveness.\n\n*Formula: Smooth updates component.*\n$$\nW[v] = \n\\\\begin{cases} \n\\\\min{(1, \\frac{G[v]}{I[v] + G[v]})} & \\\\text{if } G[v] > I[v] \\\\\\\\\n0 & \\\\text{other}\n\\\\end{cases}\n$$\n\n*Table: Experiment results of ablation study. MT-1 shows the model's general capability, while MT-2 shows the model's level of contextual understanding. MT-Final is the metric combining both of the two indicators.*\n|Momentum|Smooth|MT-1|MT-2|MT-Final|\n|-|-|-|-|-|\n|❌|✅|4.59|3.20|3.90|\n|✅|❌|4.60|3.33|3.97|\n|✅|✅|4.65|3.37|4.01|\n|✅(Ours)|❌(Ours)|**4.83**|**3.43**|**4.13**|\n\n[1]Learning both weights and connections for efficient neural network.**NeurIPS, 2015**.  \n[2]A simple and effective pruning approach for large language models.**arXiv 2023**.  \n[3]Finding sparse, trainable neural networks.**arXiv 2018**.\n\n**Q4: Some essential references are not discussed.** (Essential References Not Discussed)\n\n**A4:** Thank you for suggesting additional references. We will incorporate these into the revised manuscript.\n\n**Q5: Some inconsistencies in the notation.** (Other Comments Or Suggestions)\n\n**A5:** Thank you for pointing out the unclear expressions.\n- For the second, the vectors are defined in Lines 252-261. Specifically, we perform vector-wise aggregation by processing $A_k^i$ column-by-column to obtain $r$-dimensional vectors $v$, while for $B_k^i$, we process it row-by-row.\n- For the third, $j \\in \\{1, 2, ..., n\\}$.\n- For the fifth, we use model gradients for mean and variance.\n  \nWe will correct all of them in the revised manuscript.", "claims_and_evidence": "The statement in **Lines 319-322**:  \n   *\"The significance of parameter updates varies across different downstream tasks, with some parameters showing minimal activation. These inactive parameters not only increase computational overhead during global aggregation but also can lead to catastrophic forgetting.\"*  \n\n   This claim is unclear.  \n\n   For example, consider a **CIFAR-10 classification scenario** where the central global model is trained to generalize across all classes:  \n\n   - **Client 1** has a well-balanced dataset covering all classes, resulting in minimal drift from the global model and smaller parameter updates.  \n   - **Client 2** only has samples from classes 0 and 1, causing **significant client drift** and larger parameter updates.  \n\n   In this situation, would **excluding Client 1** (which exhibits less parameter variation) actually **benefit the global model’s performance**? If so, could you clarify how inactive parameter updates contribute to catastrophic forgetting in this context?", "methods_and_evaluation_criteria": "1. The physical meaning of the importance measures defined in **Equations (14) and (16)** is not very intuitive. They appear to be defined arbitrarily rather than derived from a concrete motivation. It would be helpful to visualize their distributions in an **LLM FL setting** to demonstrate their necessity and relevance.  \n\n2. The justification for **Equations (17) and (18)** should be further reinforced with supporting evidence to strengthen their validity.", "theoretical_claims": "I noticed that the appendix contains theorems, but since there are no theoretical claims presented in the main text, and they do not appear to be central to the paper, I did not review them.", "experimental_designs_or_analyses": "I did not review the experimental designs or analyses in detail but briefly looked over the results.", "relation_to_broader_scientific_literature": "Unlike previous studies, this paper proposes a direct PEFT (Parameter-Efficient Fine-Tuning) method to address catastrophic forgetting, which makes it a novel contribution to the field.", "essential_references_not_discussed": "In **Line 193**, the paper states:  \n\n*\"Moreover, indiscriminate parameter updates risk catastrophic forgetting, where the model loses previously acquired knowledge and suffers degraded generalization capabilities.\"*  \n\nTo strengthen this claim, it would be helpful to cite works that have empirically demonstrated **catastrophic forgetting in FL**, particularly in **image classification** settings. The following references provide relevant insights:  \n\n- **[1]** Preservation of the global knowledge by not-true self knowledge distillation in federated learning, **NeurIPS 2022**.  \n- **[2]** Flashback: Understanding and Mitigating Forgetting in Federated Learning, **ArXiv 2024**.  \n- **[3]** FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning, **TMLR 2025**.  \n\nIncluding these references would help contextualize the discussion of **catastrophic forgetting in FL** beyond LLM applications and provide additional experimental evidence supporting the argument.", "other_strengths_and_weaknesses": "Apart from the points mentioned earlier, I have no additional comments.", "other_comments_or_suggestions": "There are inconsistencies in the notation throughout the paper, and the lack of detailed explanations for some metrics makes it very difficult to understand.\n\nThere are several inconsistencies and ambiguities in the notation throughout the paper:\n\n- In **Section 3**, the FL round index is denoted as **i**, but in the pseudo-code, it is written as **t ∈ [T]**. The notation should be consistent.  \n- In **Equation (9)**, **S_{p,q}** includes **v_p, v_q**, but it is unclear how these relate to **A_k^i** and **B_k^i**. Providing a clear explanation would improve clarity.  \n- In **Equation (10)**, the possible values for **j** should be explicitly stated, as the current notation is ambiguous and confusing.  \n- In **Equation (11)**, the temperature parameter is denoted as **T**, but in the pseudo-code, it is written as **τ (tau)**. It would be better to use a consistent notation throughout the paper.  \n- In **Equation (14)**, the normalization process lacks clarity regarding which samples were used to derive the **mean and variance**. Providing a precise description would enhance readability and understanding.", "questions_for_authors": "Apart from the points mentioned earlier, I have no additional comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzwVieqDQ1", "reviewer": "Reviewer_UEdC", "summary": "The paper proposes FedICU, a novel federated learning framework for large language models (LLMs) in heterogeneous settings. It decomposes client updates into consensus and divergence components. In the global aggregation phase, it balances these components based on their contribution to the global model performance. At the client level, it uses an importance - aware parameter updating strategy. Experiments across various domains show that FedICU outperforms existing federated learning approaches in generalization performance and domain adaptation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UEdC:\n\nThank you very much for your recognition of our paper and detailed suggestions. We will answer and explain the issues in detail below.\n\n**Q1: The method relies on LoRA, which may not achieve the same performance as full parameter fine-tuning.** (Other Strengths And Weaknesses)\n\n**A1:** Thank you very much for your suggestion. Using LoRA is a common method for fine-tuning large language models today, providing significant performance improvements at relatively low cost. In the future, we will further explore how to use LoRA more efficiently to achieve the ability to fine-tune nearly all parameters.\n\n**Q2: Consider using a more flexible thresholding method in the Importance-Aware Updating mechanism.** (Questions For Authors)\n\n**A2:** Thank you very much for your suggestions. We conduct a supplementary ablation study to investigate the impact of momentum-based parameter selection and continuous-valued parameter weighting on model generalization capability, as shown in the table below. In the Momentum column, we indicate whether we include the momentum component to smooth the mask construction process. In the Smooth column, we use a smoothing mask approach based on the formula below to filter uploaded parameters and their weights. The results validate that our binary component is a simple and effective method. We will add this supplementary experiment to the appendix of our revised manuscript to enhance the description of the effectiveness of our component.\n\n*Formula: Smooth updates component.*\n$$\nW[v] = \n\\\\begin{cases} \n\\\\min{(1, \\\\frac{G[v]}{I[v] + G[v]})} & \\\\text{if } G[v] > I[v] \\\\\\\\\n0 & \\\\text{other}\n\\\\end{cases}\n$$\n\n*Table: Experiment results of ablation study about Importance-Aware Update.*\n| Momentum |  Smooth  | MT-1 | MT-2 | MT-Final |\n| -------- | -------- | ---- | ---- | -------- |\n|    ❌    |   ✅      | 4.59 | 3.20 | 3.90     |\n|    ✅    |   ❌      | 4.60  | 3.33 | 3.97     |\n|    ✅    |   ✅      | 4.65 | 3.37 | 4.01     |\n|✅(Ours)|❌(Ours)|**4.83**|**3.43**|**4.13**|\n\n**Q3: How do you expect the performance of FedICU to scale when applied to larger models or different model architectures?** (Questions For Authors & Other Strengths And Weaknesses)\n\n**A3:** We greatly appreciate your suggestion. We've conducted supplementary experiments with Mistral-7B to verify that our method is effective across a broader range of model architectures. The results of these supplementary experiments are as follows. We will add this experiment to our revised manuscript to further support our approach.\n\n*Table: The performance of FedICU and other methods applied in the Mistral-7B. MT-1 shows the model's general capability, while MT-2 shows the model's level of contextual understanding. MT-Final serves as a comprehensive metric that combines both of the aforementioned indicators. **The results show that FedICU also demonstrates excellent performance across models with different architectures.***\n| Method    | MT-1 | MT-2 | MT-Final    | \n| --------  | ---- | ---- | ----------- | \n|  Base     | 4.10 | 3.29 | 3.69  | \n|  FedAvg   | 5.48 | 3.80 | 4.64  |\n|  Scaffold | 5.46 | 3.92 | 4.69  |\n|  FedYogi  | 5.54 | 3.85 | 4.70  |\n|  FedProx  | 5.50 | 3.83 | 4.67  | \n|  FedAdam  | 5.56 | 3.91 | 4.73  | \n|  FedAvgM  | 5.55 | 3.73 | 4.64  | \n|  Ours     | **5.58** | **4.11** | **4.84** |", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem. The Consensus - Divergence Splitting method decomposes LoRA updates in a way that effectively captures general and client-specific knowledge, which is crucial for handling heterogeneous downstream instructions. The Importance - Aware Updating method reduces computational overhead and prevents catastrophic forgetting. For evaluation, using metrics like generalization (first turn’s score from MT - Bench), contextual understanding (final score from MT - Bench), and domain - specific metrics (e.g., Human Eval for code, MMLU for finance, GSM8k for math) comprehensively assesses the model's performance in different aspects relevant to LLMs in federated learning.", "theoretical_claims": "The authors provide theoretical guarantees for the convergence of their proposed algorithms. For the Consensus - Divergence Splitting method, under standard assumptions such as smoothness, bounded gradient, and unbiased and bounded-variance gradient estimates, they prove convergence to a stationary point for non-strongly convex functions and to the unique global optimum for strongly convex functions. For the Importance - Aware Updating mechanism, they also show convergence under similar assumptions, given that the masking error is bounded. The proofs seem to be well-structured and based on established mathematical concepts in optimization theory.", "experimental_designs_or_analyses": "The experimental designs are reasonable. The authors train the model on four different datasets (Taori et al., 2023; Xiang Yue, 2023; CodeAlpaca - 20k; FinGPT) to simulate heterogeneous data distribution. They compare FedICU with multiple state - of - the - art methods in the same experimental setting. The hyperparameters are set following a benchmark, and all experiments are repeated three times to ensure statistical significance. The ablation study of key components and hyperparameter study further validate the effectiveness and sensitivity of the proposed framework.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to the broader scientific literature. The paper builds on existing works in parameter - efficient fine - tuning for LLMs, such as LoRA, and federated learning for LLMs. It addresses the limitations of previous approaches, like the lack of consideration for heterogeneous downstream instructions and poor understanding of the interaction between parameter - efficient fine - tuning and federated learning. By proposing FedICU, it extends the research in this area, offering a more effective way to fine-tune LLMs in federated learning.", "essential_references_not_discussed": "There do not seem to be any essential references not discussed in the paper.", "other_strengths_and_weaknesses": "A major strength of the paper is its innovative framework, which effectively addresses the challenges in heterogeneous federated learning for LLMs. The combination of Consensus - Divergence Splitting and Importance - Aware Updating is novel and shows significant performance improvement. However, a weakness is that the method relies on LoRA, which may not achieve the same performance as full - parameter fine - tuning. Also, the experimental results are mainly based on specific datasets and models, and the generalization to other scenarios may need further investigation.", "other_comments_or_suggestions": "Please refer to \"Other Strengths And Weaknesses\" box.", "questions_for_authors": "1. In the Importance - Aware Updating mechanism, the threshold for parameter selection (comparing generalization and specialization importance) is based on a binary decision. Have you considered using a more flexible thresholding method, and how would it affect the performance? A more flexible method might better balance the trade-off between preserving general knowledge and adapting to specific domains. If it leads to a significant improvement in performance, it could strengthen the proposed approach.\n2. The experiments are conducted on specific datasets and a single model (Llama-2-7b-hf). How do you expect the performance of FedICU to scale when applied to larger models or different model architectures? If FedICU can show consistent performance improvement across different model scales and architectures, it would greatly expand its applicability.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pO2pca2g6C", "reviewer": "Reviewer_UVcT", "summary": "This paper proposes FedICU to address client heterogeneity in FL. FedICU consists of two key components:  \n\n1. **Consensus-Divergence Splitting**: This method decomposes client updates into magnitude and direction, treating magnitude as consensus and direction as divergence. The two components are then aggregated separately.  \n2. **Importance-Aware Parameter Selection**: This technique selects the most important parameters for sparse updating, enhancing efficiency.  \n\nNumerical results demonstrate the superior performance of the proposed method compared to baseline approaches.\n\n\n## update after rebuttal.\n\nI remain unconvinced by the following points:\n(1) The rationale for using direction (cosine similarity) to measure differences in magnitude is unclear.\n(2) Whether the observation that \"client updates tend to have similar magnitudes but vary in direction\" sufficiently demonstrates that magnitudes represent consensus while directions indicate divergence remains unaddressed.\n(3) Whether the proposed method preserves knowledge in scenarios *where base models perform well* is still not adequately explained.\n\nHowever, I acknowledge and respect the efforts made by the authors, and I am increasing my rating to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UVcT:\n\nThank you very much for your time and review suggestions on our paper. We hope the following responses can address your concerns.\n\n**Q1：The claim regarding the relationship between magnitude-direction and consensus-divergence does not appear to be particularly strong.** (Claims And Evidence)\n\n**A1:** During model training, parameters are optimized toward both global and local optima, and direct aggregation leads to decreased general ability. We decompose parameters into consensus and divergence, where the former represents similar update behaviors across clients, while the latter represents domain-specific update characteristics. We find that during training, magnitude (consensus) updates remain relatively consistent, while directional (divergence) updates differ significantly. This shows that clients maintain general capabilities through consensus parameters and domain-specific capabilities through divergence parameters across downstream tasks.\nTo further support our claim, we conduct an experiment with results shown in the table below. We find that consensus are similar across clients, while divergence differ more significantly, supporting our argument. We will add this experiment to revised manuscript to further substantiate our claim.\n\n*Table: Similarity of consensus matrices.*\n|Clients|1|2|3|4| \n|-|-|-|-|-| \n|1|1|0.85|0.89|0.90|\n|2|0.85|1|0.85|0.91|\n|3|0.89|0.85|1|0.90|\n|4|0.90|0.91|0.90|1|\n\n*Table: Similarity of divergence matrices.*\n|Clients|1|2|3|4|\n|-|-|-|-|-|\n|1|1|0.23|0.49|0.04|\n|2|0.23|1|0.34|0.05|\n|3|0.49|0.34|1|0.11|\n|4|0.04|0.05|0.11|1|\n\n*Table: Similarity of unsplit matrices.*\n|Clients|1|2|3|4|\n|-|-|-|-|-|\n|1|1|0.32|0.54|0.05|\n|2|0.32|1|0.35|0.07|\n|3|0.54|0.35|1|0.14|\n|4|0.05|0.07|0.14|1|\n\n**Q2: Notation Confusion** (Methods And Evaluation Criteria)\n\n**A2:** We apologize for the inconsistent notation. \n- In Eq.1, $\\Delta\\theta \\in \\mathbb{R}^{d\\times k}$. \n- For the others, the dimension of LoRA should be corrected to $d$ x $k$. In Eq.9 part, we introduce $A$ and $B$ as its decomposition matrices, with dimensions $d$ x $r$ and $r$ x $k$ respectively.\n\n**Q3: Assumption in Eq. 7, 8 is not fully supported by Figure 3.** (Methods And Evaluation Criteria)\n\n**A3:** In Figure 3, we use cosine similarity to support the assumption that \"consensus noise\" is the same. To further support it, we conduct an experiment, as shown in the table below.\n\n*Table: the $\\mu$ and $\\theta$ of consensus parameters' update. $\\mu$ represents the mean of the consensus vector updates, and $\\theta$ represents the fluctuations during the update process.*\n|Client|$\\mu$|$\\theta$|\n|-|-|-|\n|1|37.44|1.06|\n|2|37.44|1.06|\n|3|37.44|1.06|\n|4|37.44|1.06|\n\nFrom the table, the similarity of $\\mu$ and $\\theta$ across different clients indicates that the consensus noise is consistent.\n\n**Q4：Need experiments to strengthen the claims in different aspects.** (Experimental Designs Or Analyses)\n\n**A4:** Thank you for your advice. We discuss our experiment and conduct more to further support our claims below.\n\n- Preservation of General Knowledge: Table 1 in paper details our model's performance. We measure FedICU on MT-Bench[1] to test general capabilities. The optimal performance of ours demonstrates that it can preserve the model's general knowledge to a certain extent.\n- Effectiveness of Sparse Update: According to related research [2, 3], sparse updates can help prevent catastrophic forgetting. Additionally, Table 2 in our paper shows that the model improves when including Sparse Update, proving the effectiveness.\n- Convergence Improvement: We conduct an experiment measuring the rounds to reach average loss, with the table below. We can find that FedICU optimizes convergence.\n- Handling Client Drift & Case Studies: In Table 1 of our paper, we test FedICU with other methods across various domains. FedICU achieves optimal average domain knowledge, meaning it effectively handles domain knowledge without excessive bias.\n\n*Table: Convergence rounds of FedICU and other methods.*\n|Method|FedAvg|FedProx|FedAvgM|Scaffold|FedAdam|FedYogi|FedICU|\n|-|-|-|-|-|-|-|-|\n|Round|53|43|45|38|37|39|34|\n\nWe will add this discussion and experiment in revised manuscript to support our claims.\n\n[1] Openfedllm: Training large language models on decentralized private data via federated learning. **KDD 2024.**  \n[2] A simple and effective pruning approach for large language models. **arXiv 2023.**  \n[3] LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models. **arXiv 2025.**\n\n**Q5: A key concern is whether FedICU could unintentionally expose user data to the server.** (Other Comments Or Suggestions)\n\n**A5:** FedICU follows FL protocols by only sharing parameter updates, not raw data. With Importance-Aware Update mechanism, FedICU can further reduce exposure risk by filtering parameters. Additionally FedICU follows standard FL logic and is compatible with other privacy-enhancing solutions.", "claims_and_evidence": "The claim regarding the relationship between magnitude-direction and consensus-divergence does not appear to be particularly strong. The authors have demonstrated that client updates tend to have similar magnitudes but vary in direction. However, it is unclear whether this observation is sufficient to substantiate their claim.", "methods_and_evaluation_criteria": "Some aspects of the method design are unclear:  \n\n1. **Notation Confusion**: The notation appears inconsistent. According to Eq. (1), we have \\(\\Delta \\theta \\in \\mathbb{R}^{d \\times k}\\). However, before Eq. (5), this changes to \\(\\theta_i \\in \\mathbb{R}^{r \\times d}\\), and before Eq. (9), it becomes \\(\\Delta \\theta_i^{k} \\in \\mathbb{R}^{d \\times d}\\). Could the authors clarify which part of the LoRA parameters is actually being updated?  \n\n2. **Assumption in Eq. (7)–(8)**: The design of Eq. (7)–(8) relies on the assumption that \"the noise of the consensus is the same.\" The authors state that this is supported by the results in Figure 3. However, they do not specify which metric is used for Figure 3. If cosine similarity is used, Figure 3 may not sufficiently support this assumption. Could the authors provide further clarification?", "theoretical_claims": "The convergence rates are given.", "experimental_designs_or_analyses": "Currently, only the final performance on domain knowledge is reported, which may not be sufficient to fully support the key motivation of this paper. To strengthen the claims, please consider including the following experiments:  \n\n1. **Preservation of General Knowledge**: Demonstrate that FedICU does not disrupt the general knowledge of LLMs, whereas other baselines do. This can be achieved by evaluating performance on benchmarks where the backbone LLMs typically perform well.  \n\n2. **Effectiveness of Sparse Gradients**: Show that sparse gradients effectively prevent catastrophic forgetting while maintaining generalization performance.  \n\n3. **Convergence Improvement**: Provide evidence that Consensus-Divergence Splitting accelerates convergence compared to the baselines.  \n\n4. **Handling Client Drift**: Illustrate how FedICU excels in managing varying levels of client drift.  \n\n5. **Case Studies on Client Drift**: Include case studies where client drift occurs, highlighting instances where baseline methods fail while FedICU performs well.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper is easy to follow.", "other_comments_or_suggestions": "Since LLMs are generative models, a key concern is whether FedICU could unintentionally expose user data when local LoRA updates are uploaded to the server. Could the authors clarify how FedICU prevents this risk?", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cdsDfbTeXX", "reviewer": "Reviewer_CCAN", "summary": "The manuscript addresses federated LLM fine-tuning, highlighting how existing methods often lead to catastrophic forgetting, diminishing the global model's generality, and failing to properly balance model updates across clients with different downstream task datasets. The authors propose the FedICU method, which cleverly enhances the global model's generalization capabilities through importance-aware parameter uploading and consensus-divergence splitting. Experimental results demonstrate the effectiveness of the authors' proposed framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer CCAN:\n\nThank you very much for your recognition of our work. Below I will provide detailed responses to your questions. Thank you for taking your valuable time to offer suggestions for our work.\n\n**Q1:Is the proposed framework orthogonal to other LoRA training methods?** (Questions For Authors)\n\n**A1:** Yes, our method can be divided into two components, applied to both client and server parts. \nOn the client side, we select important parameters based on generalization importance and special importance, and perform sparse updates on these important parameters. This is compatible with existing LoRA training methods since this process doesn't affect the training logic but simply adds a masking module. \nOn the server side, we split and aggregate LoRA parameters, which is equivalent to first splitting LoRA into two parameter matrices on the server, applying the same training logic to both matrices separately, and then aggregating them back into a single LoRA parameter matrix at the end. Therefore, it doesn't interfere with the LoRA training process either, as it can be considered as only modifying the input/output pipeline stage of the server's LoRA training. \nIn conclusion, our method is orthogonal to other methods for training LoRA.\n\n**Q2: Have the authors explored the communication efficiency benefits this might provide?** (Other Strengths And Weaknesses & Questions For Authors)\n\n**A2:** Thank you for your suggestions. Regarding FedICU, since LoRA splitting and merging can be performed locally on either the client or server, it doesn't introduce additional communication overhead compared to standard federated learning. For the Importance-Aware Updating component, we'll analyze it mathematically.\nAssuming there are $K$ clients, each with $N$ parameters, and in the mask built by the parameter importance selection upload component, the proportion that needs to be uploaded is $\\alpha$. So the communication cost of standard federated learning would be $C_{std} = O(K * N)$ , while the communication cost of our method is $C_{imp} = O\\left(\\sum_{k=1}^{K} N \\times \\alpha_k\\right), \\text{ where } \\alpha_k < 1$, which demonstrates the communication savings of our method.", "claims_and_evidence": "The claims made in this submission are well-supported by compelling evidence.\n1. The authors provide detailed theoretical foundations for both key components in the framework. What's more, the authors provide comprehensive and accurate convergence analysis to elucidate the correctness of the theory.\n2. The authors conducted training on heterogeneous datasets and provided complete and detailed test results. Experiments show that the proposed framework outperforms state-of-the-art methods on multiple test benchmarks, demonstrating significant performance improvements.", "methods_and_evaluation_criteria": "The lack of public datasets has made federated LLM fine-tuning an innovative direction. This paper effectively addresses the issue of decreased global model generalization capability during federated large model fine-tuning due to dataset heterogeneity, providing a novel solution for the field.", "theoretical_claims": "The paper provides theoretical analysis for both components of their framework. The proofs for convergence analysis (Sections E, F, and G in the appendix) establish the mathematical foundations for their approach.", "experimental_designs_or_analyses": "This paper’s experimental design is robust, comparing FedICU against the state-of-the-art methods across different evaluation criteria. The ablation studies effectively also isolate the contribution of each component.", "relation_to_broader_scientific_literature": "The authors focus on addressing the growing challenge of enhancing global model generalization when fine-tuning LLMs in distributed heterogeneous environments. This work is situated within the broader context of diminishing high-quality public training datasets for LLMs and the increasing adoption of federated learning approaches. By building on federated learning foundations, the paper specifically tackles the problems arising from heterogeneous data distributions across clients when using LoRA for LLM fine-tuning", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. Studies on combining federated learning with large language models are crucial given the growing scarcity of public training data and increasing demand for privacy-preserving distributed training. This work addresses a critical challenge in this emerging field by tackling non-IID instruction scenarios.\n2. The proposed consensus-divergence splitting mechanism and importance-aware updating strategy work synergistically to balance global model capabilities with domain adaptation. The theoretical foundation is solid with comprehensive convergence analysis, and the implementation is elegant with clear architectural design.\n3. The framework's approach to decomposing and balancing model updates has broader implications beyond LLMs. This framework offers new insights for federated learning in other domains where maintaining general capabilities while enabling specialization is crucial.\n\nWeakness:\n1. The paper lacks explanation regarding the communication efficiency of the proposed method.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Is the proposed framework orthogonal to other LoRA training methods? \n2. The Importance-aware Updating mechanism focuses on parameter selection based on importance. Have the authors explored the communication efficiency benefits this might provide?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers highlighted this paper's novelty, thorough analyses, and convincing experiments. The authors also provided reasonable answers to the doubts about the details of the proposed dataset and method, the authors further explain the proposed method and add more experiments to support it. Moreover, FedICU performs well across different model architectures, demonstrating its generalizability. The reviewers’ concerns were well-addressed. Therefore, the decision is to recommend Acceptance.
Decision: Accept (poster)