Towards Robust Parameter-Efficient Fine-Tuning for Federated Learning
Authors: Xiuwen Fang
Mang Ye
Venue: NeurIPS 2025 poster ()
Keywords: federated learning
label noise
parameter-efficient fine-tuning
Primary area: infrastructure

Abstract: Federated Learning enables collaborative training across decentralized edge devices while preserving data privacy. However, fine-tuning large-scale pre-trained models in federated learning is hampered by substantial communication overhead and client resource limitations. Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) reduce resource demands but suffer from aggregation discrepancies and heightened vulnerability to label noise, particularly in heterogeneous federated settings. In this paper, we introduce RFedLR, a robust federated PEFT framework designed to overcome these challenges. RFedLR integrates two key components: (1) Sensitivity-aware robust tuning, which identifies and selectively updates noise-sensitive parameters to bolster local robustness against label noise, and (2) Adaptive federated LoRA aggregation, which dynamically weights and aggregates LoRA updates based on their importance and stability to minimize bias and noise propagation. Comprehensive experimental validation shows RFedLR outperforms existing methods, achieving superior accuracy and robustness in noisy federated scenarios. Our code is available at: https://github.com/FangXiuwen/RFedLR

Towards Robust Parameter-Efficiet Fine- Fine-Tuning for Federated Learning
Xiuwen Fang', Mang Ye
1 School of Computer Science, Wuhan University, Wuhan, China 2 Taikang Center for Life and Medical Sciences, Wuhan University, Wuhan, China (fangxiuwen, yemang] @whu. edu.cn
Abstract
Federated Learning enables collaborative training across decentralized edge devices while preserving data privacy. However, fine-tuning large-scale scale pre-trained models in federated learning is hampered by substantial communication overhead and client resource limitations. Parameter-efficient fine-tuning methods like Low-Rank Adap- tation (LoRA) reduce resource demands but suffer from aggregation discrepancies and heightened vulnerability to label noise, particularly in heterogeneous federated settings. In this paper, we introduce RFedLR, robust federated PEFT framework designed to overcome these challenges. RFedLR integrates two key components: (1) Sensitivity-awar robust tuning, which identifies and selectively updates noise- sensitive parameters to bolster local robustness against label noise, and (2) Adaptive federated LoRA aggregation, which dynamically weights and aggregates LoRA updates based on their importance and stability to minimize bias and noise propaga- tion. Comprchensive experimental validation shows RFedLR outperforms existing methods, achieving superior accuracy and robustness in noisy federated scenarios. Our code is available at: ht https: / thub. com/E angxiuwen. Raaeee///edeeee RFedLR
Introduction
The proliferation of edge devices generates massive siloed data volumes, whose direct aggregation and utilization are impeded by significant privacy, regulatory, and transmission constraints [1]. To meet the challenge, Federated Learning (FL) [2] is distributed machine learning framework that facilitates collaborative model training across multiple decentralized clients without sharing sensitive private data [3-5]. In the classic FedAvg algorithm, clients train models locally on their private datasets and transmit updated parameters the server for weighted averaging.
Recently, large-scale Pre-Trained Models (PTMs) show impressive performance in various tasks [6]. Combining the knowledge of large models with the distributed nature of FL [7]. via client-side fine-tuning and server-side aggregation, offers improved performance and privacy guarantees [8-10]. However, applying Full Fine- Tunnin (FFT) to large-scale PTMs in FL faces significant challenges [1]]: (1) the substantial communication overhead incurred by transmitting large parameter updates across snource--onstraimg networks, and (2) the excessive storage and computational demands that exceed the capabilities of typical edge devices. Consequently, to facilitate FL. with large-scale models on resource-constrained edge devices, we adopt Low-Rank Adaptation (LoRA) [12], representative Parameter-Fificient Fine-Tuning (PEFT) technique, as the paradigm for client-side updates. LoRA drastically reduces trainable parameters by decomposing weight updates into low-rank matrices (Fig. I), making PTMs fine-tuning viable on edge devices.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Server
Challenge 2:
Clemn/Noisy
Data
A1%
AW;
AWG
AWW
Nodry Information
how
Trainable
Parameters
Noie propagation federated aggregatton
He We
Frozm Parameters
Challenge I:
AW
+
Ba
AA
(8 68) 64)
AWW
RA B&A + ABA +
Naive amplification
Client
* Wg
Client
Wg
Client
BB We's
Local noise amplification in LoRA fine-tuning
Figure 1: Illustration federated learning with parameter -fffccttr-efficintt fine- aie-tuning, where clients possess noisy datasets with different noise rate. Only a small number of trainable parameters are updated and transferred.
However, integrating LoRA into FL presents distinct challenges. Naive federated averaging of LoRA components, such as FedLR R [13] and SLoRA [14], leads to aggregation discrepancies because the average of low-rank decompositions is not equivalent the decomposition of the average of full updates (Eq. 3). This can result in suboptimal global models. Attempts to mitigate this, FFA-
LoRA [15] freezes the non-zero initialized low-rank matrix and only allows the zero-initialized low-rank matrix to be trained and aggregated. However, this rigidity may constrain the model expressiveness. RoLoRA [16] alternately updates between matrices across communication rounds, avoiding permanently freezing either matrix, but it still imposes a rigid update schedule. Critically, existing methods lack sufficient flexibility, potentially leading the neglect of valuable updates and suboptimal performance in heterogeneous FL [1].
Existing federated PEFT methods are based on the assumption that the clients have clean private datasets. However, the FL system involves numerous dis- tributed clients whose private data qual- ity varies significantly, often including noisy labels. Label noise in FL can stem from diverse sources, such as human ex- pertise limitations, cost compromises, or deliberate noise injection for privacy and fairness. The presence of label noise poses significant barrier to achieving robust performance in federated PEFT. FFT exhibits resilience to label noise owing to its expansive parameter space. The constrained parameterization of LoRA makes it more susceptible to overfitting noisy labels, as illustrated in Fig. 2. Furthermore, the LoRA decomposition itself can exacerbate bias from label noise. Label noise introduces bias terms in the two low-rank decomposition matrices. These biases are subsequently amplified through matrix multiplication, leading to a cascading effect that ultimately undermines model robustness (Eq. 4). Therefore, how to mitigate the negative effects of label noise during local LoRA fine-tuning becomes a key challenge.
56.80
83.59 K3.20
60
50
50
noly
(a) Accuracy federated FFT (b) Accuracy of federated LoRA
Figure 2: Comparison of federated fine-tuning methods under clean and noisy (20% pairflip noise) conditions. The average test accuracy (%) of all clients is shown. (a) Federated FFT shows stable performance with minor accuracy loss under noise. (b) Federated LoRA demonstrates greater susceptibility noise.
In FL., noisy updates from individual clients propagate through repeated aggregation, contaminating the global model and degrading system performance. Existing LoRA-basee aggregation strategies, constrained by inflexibility or noise sensitivity, fail to suppress this amplification while maintain- ing model expressiveness. Therefore, how to design robust and flexible LoRA-based federated aggregation method to prevent noise amplification in federated communication is an important issue.
In this paper, we propose RFedLR, a robust federated PEFT method under noisy scenarios, which consists of two stages: (1) To tackle the vulnerability of LoRA to label noise during local fine- tuning, we propose Sensitivity-awar Robust Tuning (SRT). SRT identifies and selectively updates noise-sensitive parameters during local training, thereby enhancing local model resilience to label noise while preserving stable features. (2) To overcome the aggregation discrepancies and noise amplification inherent in federated LoRA aggregation, we introduce Adaptive Federated LoRA Aggregation (AFLA). AFLA dynamically assesses the importance of local LoRA updates and assigns aggregation weights accordingly. This adaptive approach minimizes aggregation discrepancies and curtails noise propagation, leading to a more robust and accurate global model. The main contributions of this work are as follows:

---

We first study and address the problem of robust federated PEFT in the presence of label noise.
We propose RFedLR, a robust LoRA-based federated PEFT framework that leverages sensitivity- aware robust tuning and adaptive federated LoRA aggregation to mitigate the label noise effects.
Extensive experiments demonstrate that RFedLR outperforms State-Of-The-A (SOTA) methods, achieving higher accuracy and robustness in noisy federated learning scenarios.
2
Related Work
Efficiter-Effcciet Fine-Tuning, The computational and storage demands of FFT for large-scale models drive the development of PEFT methods, which update only a small subset of parameters or introduce limited trainable parameters. Adapter tuning [17] inserts small trainable adapter modules into each layer of a frozen pre-trained network. Prefix Tuning [18] and Prompt Tuning [19] append trainable continuous vectors to model inputs. BitFit [20] fine-tunes only the bias terms. Besides, LoRA [12]. proposed by Hu et al., is arguably the most widely adopted PEFT method to date. LoRA decomposes weight updates into low-rank matrices, significantly reducing trainable parameters without altering model architecture or input sequence length, thereby offering implementation simplicity and training stability. Furthermore, many variants of LoRA [21] are proposed. LoRA+ [22] employs an adjusted learning rate mechanism. DoRA [23] enhances tuning precision by decomposing weights into magnitude and direction components. LoRA-drop [24] and AdaLoRA [25] optimize adaptation through layer selection and dynamic rank allocation, respectively.
Parameter-E Efffcceent Fine-Tuning, in FL. Applying large-scale PTMs to FL is hindered by commu- nication costs and local resource constraints. Consequently, several works explore the integration of PEFT methods within federated frameworks [26-28]. FS-LLM [29] provides benchmark for feder- ated fine-tuning of large language models. Zhang et al. [13, 30] evaluate prominent PEFT methods in FL, including Adapter Tuning (FedAP), Prefix Tuning (FedPF), BitFit (FedBF), and LoRA (FedLR). The results show that FedLR achieves competitive results while saving resource costs. SLoRA [14] tackles data heterogeneity via data-driven initialization and sparse tuning in a LoRA-based FL setting. Cho et al. [31] enable varying LoRA ranks across clients, with the server adaptively aggregating and distributing modules using zero padding and truncation. FlexLoRA [32] facilitates dynamic LoRA rank adjustments and utilizes Singular Value Decomposition (SVD) for post-aggregation weight assignment. These LoRA-based federated PEFT methods directly integrate LoRA into the standard FedAvg paradigm, which relies on weighted averaging for parameter aggregation. However, such direct LoRA aggregation can lead to interference. To address this, FFA-LoRA [15] updates only the vero-initialized matrix, while RoLoRA [16] alternates freezing low-rank matrices. Existing LoRA-based federated PEFT methods [33] exhibit limitations. They suffer from biases caused by direct LoRA aggregation or are limited by rigid update strategies, which can lead to suboptimal training results. Crucially, existing methods show significant performance degradation in the presence of label noise, which is common problem in real-world FL deployments.
Label Noise Learning. Existing methods for label noise learning can be divided into four categories:
(I) Noise transition matrix estimation. These approaches [34, 35] correct training by estimating a noise transition matrix. Dual T [36] decomposes the matrix to reduce estimation errors. Li et [37] exploit label correlation without anchors. (2) Sample selection and reweighting. Such techniques [38- 41] identify reliable samples or adjust their weights during training. Han et al. [42, 43] train two networks to filter low-loss samples for cross-trains. DivideMix [44] leverages GMM and semi- supervised learning to separate noisy data. (3) Robust regularization. These methods [45] enhance model robustness via regularization schemes. Xia el al. [46] penalize non-critical parameters to prevent memorizing noisy labels. Menon et al. [47] propose a gradient clipping method based on composite loss to improve model robustness. (4) Robust loss functions. These functions [48-50] are designed for inherent noise tolerance. SCE [51] merges standard and reverse cross-entropy loss. Peer loss [52] uses predictions from peer network to guide the current network.
Label Noise Learning in FL. Several works develop methods mitigate label noise in FL. RHFL [53] studies the robust FL problem with label noise and heterogeneous model. Xu et al. [54] propose a multi-stage approach for label noise correction FL. Wu el al. [55] focus on addressing class imbalance and label noise heterogeneity in FI.. FedFixer [56] introduces a personalized model that cooperates with the global model to select clean samples. FedNed [57] uses the pseudo-label obtained by the global model to correct local training and performs aggregation through negative
3

---

distillation. While effective in conventional FL settings, these methods are not suitable for our federated PEFT environment. They usually assume access to sufficient local training resources or require massive data-level intervention, limiting their applicability in federated large-scale PTMs scenarios.
Motivation
3.1 Problem Setup and Notations
We consider standard FL setting for C-class image classification task involving clients. Each client C possesses a private dataset Dx = {( (et../")) )= with N₂ =| Dx |. and yf E 10 is the one-hot ground-truth label for image -..- However, in the noisy FL. scenario we focus on, the labels can be incorrect. Thus, each client Cą uses a noisy dataset D₃ = {(x 0. D where of is the potentially corrupted label for xç
The federated fine-tuning process comprises iterations of local fine-tuning and collaborative aggrega- tion, with T denoting the communication rounds. The central server maintains global pre-trained model with parameters Wo and global fine-tuning parameters AWg. The pre-trained model parame- ters Wo are pre-deployed on all client devices as W During the process, only A AW, is updated, transferred, and aggregated, keeping the full parameter set Wo frozen. In the local fine-tuning stage, clients receive the global fine-tuning parameters A W.. Each client Gg fine-tunes the local model with the private dataset, obtains updated fine-tuning parameters A We and sends them to the server. In the collaborative aggregation phase, the server aggregates the fine-tuning parameter updates received from the clients. The process can be represented
La Ca(AW
Σ
L(Wo + AW),00)
NE
(e,0))
(1)
N
AW = Σ k=1
AWW
N
where N = N₂ and E(.) the loss function. Intuitively, our objective to calculate an optimal set of local fine-tuning parameters AW = <AW,, AWx) to minimize the total loss of all the clients. The presence of noisy samples hinders local model convergence and destabilizes the global model upon aggregation due to error propagation. Therefore, is crucial investigate the noise-robust federated PEFT method.
3.2 LoRA in Noisy Federated Learning
We adopt a widely-adopter PEFT method LoRA for federated fine-tuning, which exhibits superior performance compared to other PEFT methods [13]. LoRA decomposes the weight update matrix AW m= in the pre-trained model into the product of two low-rank matrices mm and A E Brxn, ie., Wo AW = Wo + BA, where the rank r is significantly smaller than and (r << min(m, n)). This decomposition reduces the scale trainable parameters from O(m n) to O(r x (m + nt). In practice, the modified forward pass through the layer is expressed as
W = Wo + BA,
(2)
where a is scaling hyperparameter. Typically, A is randomly initialized, and is zero initialized, ensuring that AW is zero the beginning of fine-tuning, thus preserving the initial performance of the pre-trained model.
However, directly integrating LoRA into FL introduces a fundamental aggregation challenge [15]. In the FL context, Wo is pre-deployed to all clients and remains fixed, and only LoRA matrices A and B are transmitted to the server for aggregation. Ideally, federated aggregation should operate on the full low-rank update, iee. AW BA.. The theoretically ideal aggregated update shown in Eq. 3a. In contrast, the Vanilla FedAvg algorithm [2] separately averages the A and B matrices. The

---

Vederated A Aggregation
Matrix Imporiance Mensurement
duPg
B2
+w3
B2
AQ
Pg
A
Global Weights
Clean data
Local
Weights
= F/(5 ++)
W2
Parmeterr importance (Ea 10)
68,
A
Ab
A6
Client
Client
Client k
Robust Local Tuning
Nofiee Sousiiiivtyy Extimation
Clean dara
Diserepancy
[1.5t (0.5t
Sk Piy, Cx
Mi
Noisy
Local model
Gradiem
Fi
Noise sensilivity (Bg,
Mask
Client
Forward
W.
J.W
M₁
We
Scloctive backward
Illustration of RFedLR, which facilitates robust parameter-officient tine-tuning for federated Icarning. the local tuning phase, a sensitivity- aware robust tuning strategy is proposed handle local noise amplification (84.1). In the federated aggregation phase, design the adaptive federated LoRA aggregation to prevent noisy information propagation in communication ($4.2).
Figure 3:
actual aggregated update is shown Eq. 3b as follows
AW (BiA + B2A2 + ... + BxAk),
(3a)
K
1 1 ( +A1 ...+ +A..).
K (BI Bk)
(3b)
K
Consequently, discrepancy exists between the theoretically ideal aggregation (Eq. 3a) and the actual aggregation (Eq. 3b) that combine low-rank updates.
These challenges are compounded in noisy FL scenarios. The trainable parameter scale of LoRA significantly less than FFT. which may render models more susceptible to overfitting noisy labels. More critically. label noise introduces bias terms into both low-rank matrices A and B. The subsequent matrix multiplication non- aon-innerly amplifies this bias follows
AW = (B + (B) + 6A) = BA + B8A + 8BA + 8B5A,
(4)
where 5.A and 8B denote the bias terms induced by noise. The noisy impact is not merely additive, but an additional noise term BBA is generated by matrix multiplication, which amplifies the noise further exacerbated in FL due to the iterative propagation of noisy
effect. This degradation information across clients. Our experiments (Fig. 2) demonstrate that label noise substantially impairs the performance of LoRA in Fl., with the adverse impact intensifying over communication rounds. To address these compounded challenges, we propose FedLR, which enhances robustness through sensitivity-aware local tuning and adaptive federated LoRA aggregation. The framework of RFedl.R is described in Algorithm 1.
4 Proposed Method
4.1 Sensitivity-Awarc Robust Tuning
To mitigate the adverse effects of label noise in federated LoRA fine-tuning, we propose SRT, which enables robust local fine-tuning through selective parameter updates based on noise sensitivity analysis (Fig. 3). The key insight behind SRT is that model parameters exhibit different degrees of sensitivity to label noise, with some parameters being extremely susceptible to roise-induced perturbations while others remain relatively stable. The parameters with low sensitivity are relatively stable and close to the ideal state, making them less susceptible to label noise. Conversely, highly more susceptible noise and require careful fine-tuning. Therefore,
sensitive parameters identify and selectively update parameters based on their noise sensitivity, improving the model robustness to noisy labels.
Noise Sensitivity Estimation. To quantify the sensitivity of individual parameters to label noise, we leverage contrastive analysis between clean and noisy learning dynamics to identify parameters that
5

---

exhibit high susceptibility to moise-induced perturbations. Specifically, the server collects a mini- batch clean public proxy dataset D., and corresponding noisy dataset D,, generated by injecting random label noise into De. The proxy datasets are sent to all clients. For each client CR. we compute gradient vectors Vw, (WW,,, Dz) and f. (. (W5, D₂, of local model parameters We on clean
and noisy datasets, respectively. The magnitude of the gradient discrepancy between clean and noisy conditions for each parameter indexed by in the local model Wi can be expressed as:
S Wwa Wwi
(5)
The collection of all sensitivity scores forms sensitivity tensor Sg, which has the same dimensions as corresponding parameter tensor. The gradient magnitudes vary significantly across layers, and some layers that are sensitive to input changes may dominate the sensitivity estimation. To ensure comparability of gradient scales, we apply layer-wise min-max normalization to obtain the final sensitivity score. For parameter E layer l, the final normalized sensitivity score S is
s m mar - min (
St
where ( denotes the set of sensitivity scores for layer I. The parameters with high sensitivity scores show divergence in their optimization trajectories between noisy and clean data, indicating susceptibility to noise-induced biases. Conversely, a low score implies a more consistent update direction irrespective of label quality, signifying inherent stability.
Sensitivity-Guine -uidded Gradient Propagation. Based on the calculated parameter-level sensitivity scores, we selectively control gradient updates through binary mask M. Given [0, as the parameter keep ratio, it indicates the proportion of parameters allowed to be updated. For each parameter W the corresponding mask element M is determined as follows:
1.
$ 2 S' S S'
Mi =
0,
where S' is threshold set to select the parameters with the top proportion of sensitivity scores. controls the proportion of parameters that allow updates, and 7 1 indicates standard LoRA fine- tuning. This mechanism differentiates between highly noise-sensitive and relatively stable parameters. We find that prioritizing updates for the most sensitive parameters significantly enhances model robustness. These highly sensitive parameters require recalibration to adapt to noisy environments, allowing the model to extract robust patterns. Meanwhile, preserving stable parameters helps the model maintain foundational representational capacity and mitigates overfitting to noisy samples, creating an implicit regularization effect. Importantly, during forward propagation, all parameters remain active to preserve model representability. Only in backward propagation, the mask M affects the gradient flow by element-wise multiplication, similar to the conditional gradient dropout [58]. In round E [o, T), the local fine-tuning process on the private noisy dataset D₃ can formulated as:
AW = AW, 17V Lee (AW: D2) @ Mi.
(8)
where 77 is the learning rate, Lee denotes the cross-entropy loss and @ denotes element- multi- plication. This selective updating strategy improves the noise robustness of local fine-tuning and reduces the computational cost by limiting the amount of updated parameters.
4.2 Adaptive Federated LoRA Aggregation
While SRT improves the robustness of local fine-tuning, effectively aggregating LoRA updates under label noise remains challenging. Existing static or alternating matrix update schemes fail to capture dynamic parameter importance and stability. Our proposed AFLA addresses this by dynamically updating and aggregating LoRA matrices based on these properties (Fig. 3).
Matrix Importance Measurement. To discern the significance of matrices A and B within the local update AW;- = B& A₁ for each client ex. we propose a matrix importance measurement that considers both parameter information and noise sensitivity. We adopt the Fisher Information Matrix (FIM) calculate parameter contribution, which effectively estimates the information content of parameters [59, 60]. The parameters with higher Fisher values are considered more informative,

---

as they contribute more significantly to the model decisions. However, local noise may distort the gradient calculation, resulting in biased Fisher information estimate. To accurately evaluate the parameter contribution, we calculate the Fisher information value on the clean public proxy dataset D. instead of the local noisy dataset Dj., Since computing the exact FIM is computationally expensive, we adopt an empirical Fisher approximation. Specifically, the FIM measures the parameter contribution to the curvature of the loss function by calculating the square of the first-order derivative of the oog-liielihood function. The Fisher information for an individual parameter indexed by i the local model We is defined as:
Ölog L(WE, Do) Ow
We then integrate both noise sensitivity and parameter contribution into a hybrid importance score I. The matrix with lower noise sensitivity (Fq. 5) is more stable and reliable for federated aggregation. The hybrid importance score can be expressed as:
F
(10)
S c
where e = le - 8 ensures computational stability. To ensure the comparability of the importance score across layers, we apply the layer-wise min-max normalization to 1 to obtain zg. Without normalization, the inherent input-sensitive layer parameters will always have higher importance.
The overall importance scores for the update matrices B and A of client Cs. denoted as LB and IA respectively, are obtained by averaging the normalized parameter importance scores within each individual matrix:
LA
Σ JB. JEAL
JB.
1
PB.. Σ
(11)
A&
B&
JEB
Selective Weighted Aggregation. To mitigate aggregation bias and noise amplification in federated aggregation, we introduce a selective weighted aggregation strategy. This approach dynamically identifies the trainable matrix based on the matrix importance of the global model. Before each FL round f, we calculate the matrix importance JAg and TBg for the global model to determine the update matrix. The trainable matrix X' is selected as follows:
(A, if if > >
X²
(12)
B, otherwise
and the other LoRA matrix, denoted as Y The adaptive selection prioritizes the high-importance matrix while keeping the other matrix frozen. During the local fine-tuning phase, clients only fine-tune the selected matrix X This strategy concentrates computational resources on the most impactful parameter updates, minimizing aggregation bias and reducing noise propagation.
After the clients perform robust local fine-tuning (Sec. 4.1), recaleulate the updated trainable matrix importance score for clients to guide aggregation. In the federated aggregation phase, clients upload X to the server, and the server then aggregates these received low rank matrices via weighted average scheme. This scheme comprehensively considers both the trainable matrix importance and the local data scale of each client. Specifically, the federated aggregation process for the round t E [0, T can be formulated as:
X
Σ
(13)
1xt
N -=1 N
X
+ X)
Ixt
Di=1
where Wi is the weight for client eą and À balances client importance and data scale. Ix, denotes the importance score the selected trainable update matrix of client c in current round. By integrating data scale and parameter importance into the weighting scheme, we achieve efficient and robust knowledge transfer. By implicitly reducing the weight of contributions from potentially noisy or less informative updates, this approach effectively mitigates the detrimental impact of label noise in FL. Following aggregation, the server transmits only this single, updated global low-rank matrix X back to all participating clients. This download step only transmits a single matrix, thus halving the communication overhead.
7

---

5 Experiments
5.1 Experimental Setup
Datasets and Models. Following previous works [53, 61], we conduct extensive experiments on CIFAR-100 [62] dataset, which contains 60, 000 color images covering 100 classes. The FL setup involved K clients, each holding non-IID partition of CIFAR-100. The private data is partitioned among clients according to a Dirichlet distribution, with the degree of data heterogeneity controlled by the hyperparameter B. which is set to 0.5. We maintain a mini-batch of clean CIFAR- 100 dataset on the server as the public proxy dataset D. with Del = 256. Following existing visual PEFT works [63], we set an initial global model on the server, which adopts Vision Transformer- (ViT-B) [64] as the backbone architecture, initialized with ImageNet-21K [65] pre-trained weights, processing 224 x 224 pixel images with a 16 16 patch size.
Label Noise Settings. We investigate two common settings for label noise [42]. (1) pair flip (pairflip) noise, flipping the label to a pre-defined similar class. (2) symmetric flip (symflip) noise, randomly flipping the label to any other class with uniform probability. We experiment with noise rate μ of 0.2 and 0.4. Clients sample randomly from shuffled noisy CIFAR-100, so the noise rates of different clients might be inconsistent to simulate practical settings.
Comparison Methods. We compare RFedLR with SOTA methods to demonstrate its effectiveness in noisy FL setting. FedPF [13] applies prefix tuning in federated settings. FedBF [13] only trains and share bias parameters. FedAP [13] updates only small adapter modules inserted between model layers. FedLR [13] incorporates LoRA into FL. SLoRA [14] performs data-driven initialization and
sparse fine-tuning handle data heterogeneity in FL. FFA-LORA [15] only updates and aggregates the zero-initialized matrix in LoRA. RoLoRA [16] alternately freezes different low-rank matrices during communication rounds. FlexLoRA [32] implements weight redistribution through SVD. All methods were implemented within unified PyTorch framework for fair comparison.
Implementation Details. The FL system comprises K 5 clients and runs for T 40 com- munication rounds. Local fine-tuning is performed for one epoch per communication round. For optimization,
SGD with learning rate of 0.01, weight decay of 0.0001, momentum of 0.9 and a batch size of 256. LoRA decomposes weight updates into two low-rank matrices A and B with rank 4, and the scaling factor is set to 4. SRT employed keep ratio т 0.2. For AFLA, the balancing hyperparameter À is set to 0.4. Experiments are conducted on 4 NVIDIA RTX 3090 GPUs.
5.2 Ablation Study
As shown in Tab. 1, we evaluate the individual and combined effectiveness of all components under noise ratios of 0.2, 0.4, and noise types of pairflip and symflip. The baseline (first row in Tab. I) represents Vanilla FedAvg combined with LoRA.
Effectiveness of SRT. SRT mitigates local noise by selectively updating parameters based on their estimated noise sensitivity. The results in Tab. 1 show that enabling SRT alone yields notable im- provements over the baseline across all noise conditions. For μ = 0.2, the average accuracy increases from 75.28% to 81.63% (pairflip) and from 83.65% to 85.03% (symflip). At μ 0.4, SRT achieves 64.13% for pairflip noise and 84.62 84.62% for symflip noise, yielding improvements 7.51 % and 4.93%, respectively. The efficacy of SRT is pronounced at higher noise rates, affirming its effectiveness in countering local noise-induced induced perturbations. Besides, the impact of the hyperparameter 7 on SRT is further investigated through sensitivity analysis detailed in Appendix B.
Effectiveness of AFLA. AFLA adaptively freezes a single matrix and dynamically weights clients, aiming to minimize bias and noise amplification during global aggregation. We evaluate its contri- bution by comparing the baseline model against the model with only AFLA enabled. For μ. = 0.2. AFLA improves the baseline accuracy from 75.28% to 76.07% under pairflip noise and from 83.65% to 84.87% under symflip noise. At a 0.4, AFLA also shows significant improvements. The results demonstrate the ability of AFLA to refine aggregation by prioritizing stable and informative updates. As shown Appendix B.2, we implemented ablation experiments to analyze the respective contributions of the Fisher information component and the noise sensitivity component of the hybrid
8

---

Table I: Ablation study $5.2 of each component. The average test accuracy (%) of local models is demonstrated.
Trarude Pereeeeooraeee Purmmass
1,411
20
Components
0.2
0.4
6526
SRT
AFLA
Pairflip
Symflip
Pairflip
Symflip
0,394
75.28
83.65
56.62
79.69
2203
81.63
85.03
64.13
84.62
100
TMARIA RaS Hoolk 1i
/ 76.07
84.87
57.19
81.36
Figure Comparison trainable parameter (%) for various methods relative to FFT.
83.12
86.97
67.08
84.64
importance score AFLA. Furthermore, we conduct sensitivity analysis on À in Appendix B.1 to demonstrate the effectiveness of AFLA.
Combined Effectiveness of SRT and AFLA. The RFedLR framework integrates SRT and AFLA to achieve the highest performance in all settings. Especially at μt 0.4, accuracy under pairflip noise and symflip noise improves by 10.46% and 4.95% from baseline. The performance the full model surpasses the individual contributions of SKT and AFLA, revealing synergistic effect.
5.3 Comparison with SOTA Methods
We evaluate the performance of RFedLR L. against SOTA federated PEFT methods. As shown in Tabs. and 3, model performance is evaluated through the test accuracy local models, across different noise rates. LoRA-based based methods consistently outperform other PEFT approaches, especially under higher noise rates. This reinforces the suitability of LoRA as foundation for federated fine-tuning in noisy environments. RFedLR builds upon this foundation, leveraging SRT and AFLA achieve further improvements. RFedLR consistently outperformed all SOTA methods, demonstrating strong robustness noisy FL settings. The superior performance of RFedLR across both noise types and rates can be attributed to SRT and AFLA. SRT enhances local tuning by selectively updating parameters based on noise sensitivity, reducing overfitting noisy labels. Meanwhile, AFLA mitigates aggregation bias and noise propagation through adaptive matrix aggregation and dynamic weighting based on importance and stability, ensuring efficient and robust global updates. The larger performance gains under pairflip noise, especially at μ = 0.4, where RFedLR outperforms RoLoRA by 10.15%, suggest that RFedLR excels handling structured noise and the performance advantage is more pronounced at increased noise intensity.
All methods performed worse under pairflip noise than symflip, due to differing disruption mech- anisms. Symflip noise introduces uniform, less correlated errors, while pairflip creates structured, harder-to-resolve noise by flipping labels similar classes. Nonetheless, RFedLR achieves significant improvements under both pairflip and symflip noise. Under pairflip noise with = 0.4, RFedLR achieves an average accuracy of 67.08%, surpassing the existing best method RoLoRA by 10.15%. Under symflip noise with μ = 0.4, RFedLR achieves 84.64%, which is 3.13% higher than SLoRA.
In terms of parameter efficiency, RFedLR also demonstrates significant advantages, which is crucial for deployment on resource-constrainee edge devices. As illustrated in Fig. 4, RFedLR utilizes the fewest trainable parameters (0. 1754% relative Full Fine-Tuning) among all compared methods. This efficiency makes RFedLR well-suited for resource-constrained FL scenarios.
Beyond the main vision tasks, further validate the generalizability and robustness of our framework in additional experiments detailed Appendix B.3. To assess its versatility, we test RFedLR on the MNLI Natural Language Processing (NLP) task, where it outperforms other federated PEFT methods. Besides, we validate the scalability RFedLR in larger-scale cross-device scenarios (see Appendix B.4 for details). Furthermore, learning methods, by adapting their core boise-handling mechanisms to the PEFT setting for a fair comparison. As shown in Appendix B.5, RFedLR achieves competitive performance.
compare approach against SOTA federated noise
5.4 Analysis of the Proxy Dataset
Our method preserves the foundational privacy guarantee of federated learning, i.e., no private client data is ever shared. We follow the published federated learning research [66] in setting up proxy datasets, which can come from small public datasets relevant to the task, or from trusted participants in the federated learning system. In our work, require only mini-batch dataset serve this

---

Table 2: Comparison with the SOTA methods $5.3
when the noise μ 0.2.
Pairllip
Symilip
Mothod
W,
W
Ws
W4
W
Avg
W
Wa
W
Ws
Avg
FedPF [1.5]
67.32
67.98
75.45
76.10
74.25
FedBF [13]
75.87
74.74
73.47
75.26
74,56
74.78
83.74
82.68
82.35
83.67
84.17
83.32
FedAP [13]
75.58
75.79
73.83
76.93
75.28
75.48
83.02
82.20
81.46
83.01
84.38
FedLR [13]
77.09
73.85
73.50
75.98
75.97
75.28
84,50
83.03
82.87
83.94
83,90
83.65
SLoRA [14]
77.12
76.18
76.17
77.16
75.78
76.48
84.89
83.97
84.07
85.06
84.51
84.50
FEA LoRA [15]
73.68
72.68
72.05
74.65
72.94
73.20
81.09
80.59
79.05
80.94
80.34
80,40
RoLoRA [16]
76.62
74.88
74.42
76.48
76.22
73.73
85.21
83.95
83.69
84.86
85.03
84.55
FleLLoAA [32]
75.94
73.68
71.77
76.00
76.01
74.68
84.39
83.29
82.47
83.75
84.16
RFedLR
83,40
82,96
83,28
82,88
83,07
83,12
87.17
86.69
86,39
87,00
87,02
86,97
Table 3: Comparison with the SOTA methods $5.3
when the noise rate μ 0.4.
Pairllip
Symllip
Method
W,
2
W
W4
W
Avg
W₂
Wa
W
W
Avg
FedPF [13]
50.27
50.26
48.80
51.19
50.00
50.10
72.30
72.778
72.76
FedBFF [13]
55.76
55.00
53.46
56.24
59.30
55.95
79.11
77.69
76.98
78.30
80.43
78.50
[12]
54.20
52.81
57.40
77.89
77.15
78.69
78.62
FedL R [13]
56,91
55.91
56,88
57.56
55.83
56.62
79.31
79.33
79.13
79.22
81.46
79.69
SLoRA [14]
54.30
56.38
54.58
58.06
60.55
56.77
81.36
81.38
80.55
81.78
82.45
81.51
FFA-LoRA [15]
54.81
54.90
54.33
56.62
53.22
54.78
77.05
76.61
77.53
78.01
77.87
77.41
RoLoRA [16]
57.18
55.75
$7.39
58.20
56.13
56.93
80.95
81.00
80.79
81.53
82.27
81.31
HexLoRA [321
56.56
55.46
56.24
56.58
78.97
79.66
81.49
79.71
RFedLR
67.36
67.04
66.82
67,04
67.13
67.08
83.83
84,61
84,41
85.69
84.67
84.64
purpose. Crucially, the effectiveness of our method does not depend on the assumption that the distribution noise pattern of the proxy dataset must match to the local data.
Robustness Data Distribution Mismatch. In our setup, the proxy dataset is uniformly distributed mini-bateh randomly sampled from the CIFAR- 100 test set. However, the local data was explicitly configured to be non-III. As demonstrated by our results in Tabs. and 3, particularly the highly data heterogeneous environments (Appendix B.6), our method performs well, which indicates that distribution matching is unnecessary.
Robustness to Noise Type Mismatch. The noisy proxy dataset D,, is generated by injecting symflip noise into the clean dataset. Symflip noise represents general form of label noise that does not rely on any assumptions about class similarity. This ensures that our sensitivity analysis is robust and not biased towards a specific structured noise pattern. Tabs. 2 and 3 show that RFedLR still demonstrates remarkable effectiveness when clients have structured pairflpp noise. Besides, Tab. shows that the introduction of SRT module results in a significant accuracy improvement under pairflip noise with different noise rates. This shows that a mismatch between the proxy noise and the local true noise does not undermine the effectiveness of our gradient discrepancy analysis.
The proxy dataset serves as an objective benchmark for identifying parameters that are unstable to noise. This instability is general property of model parameters, rather than their response specific data distributions or noise patterns. Furthermore, the results in Appendix B.7 show that even using lask-irrelevant proxy datasets is still sufficient detect the most significant sensitive parameters.
Conclusion
In this paper, tackle the challenges of fine-tuning large-scale PTMs in FL under label noise scenar- ios. We propose RFedLRR RR, a robust PEFT framework including SRT and AFLA. SRT enhances local model robustness by selectively updating noise-sensitive parameters. AFLA mitigates aggregation discrepancies and curbs noise propagation via dynamic LoRA update aggregation. Comprchensive experiments conducted in noisy FL settings demonstrate that RFedLR achieves superior accuracy and robustness compared to SOTA methods, advancing the deployment of large-scale models in such environments. The primary limitations of RFedLR include its reliance on a mini-batch public proxy dataset for estimating noise sensitivity and matrix importance, dependency that may con- strain broader applicability. Furthermore, while RFedLR achieves substantial reductions in trainable parameters, the requisite computations for noise sensitivity estimation and importance measurement introduce additional resource demands. Future work will focus on developing proxy-data-rre and more computationally efficient solutions.
10

---

Acknowledgements. This work is partially supported by the National Key Research and Development Program of China (2024YFC3308400), National Natural Science Foundation of China under Grant (62361 166629, 621761888 188), Major Project of Science and Technology Innovation of Hubei Province (2024BCA003, 2025BFA0002..
References
[1] Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 2023.
[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiie learning of deep networks from decentralized data. In AISTATS, 2017.
[3] Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Jian Cao, and Haibing Guan. Gpft: Simultaneously learning global and personalized feature information for personalized federated learning. In ICCV, 2023.
[4] Yasar Abbas Ur Rehman, Yan Gao, Pedro Porto Buarque De Gusmão, Mina Alibeigi, Jiajun Shen, and Nicholas Lane. L-dawa: Layer-wise divergence aware weight aggregation in federated sule--uuer visual representation learning. In ICCV, 2023.
[5] Wenke Huang, Mang Ye, Zekun Shi, Guancheng Wan, He Li, Bo Du, and Qiang Yang. Federated learning for generalization, robustness, fairness: A survey and benchmark. IEEE TPAMI, 2024.
[6] Qu Yang, Mang Ye, and Dacheng Tao. Synergy of sight and semanties: visual intention understanding with clip. In ECCV, 2024.
[7] Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes. In ICMI,, 2024.
[8] Fu-En Yang, Chien- Yi Wang, and Yu-Chiang Frank Wang. Efficient model personalization in federated learning via client-specific prompt generation. In ICCV, 2023.
[9] Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, and Chen Chen. Fedperfix: Towards partial model personalization of vision transformers in federated learning. In ICCV, 2023.
[10] Rishub Tamirisa, Chulin Xie, Wenxuan Bao, Andy Zhou, Ron Arel, and Aviv Shamsian. Fedselect: Personalized federated learning with customized selection of parameters for fine- tuning. In CVPR, 2024.
[11] Chulin Xie, De-An Huang, Wenda Chu, Daguang Xu, Chaowei Xiao, Bo Li, and Anima Anandkumar, Perada: Parameter-efficien federated learning personalization with generalization guarantees. In CVPR, 2024.
[12] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[13] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficien tuning methods of pre- trained language models. In ACI,, 2023.
[14] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. In NeurlPS Workshop. 2023.
[15] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in federated learning, In ICLR, 2024.
[16] Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust federated finetuning of foundation models via alternating minimization of lora. In ICML, 2024.

---

[17] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nip. In ICMI,, 2019.
[18] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuuus prompts for generation. In ACL, 2021.
[19] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[20] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-basee masked language-models. In ACL, 2022.
[21] Songlin Chen, Weicheng Wang, Xiaoliang Chen, Peng Lu, Zaiyan Yang. and Yajun Du. Llama- lora neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains. Data Intelligence, 2024.
[22] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. In ICMI., 2024.
[23] Shih-Yang Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024.
[24] Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. Lora- drop: Efficient lora parameter pruning based on output evaluation. In ACL, 2025.
[25] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In ICLR, 2023.
[26] Panlong Wu, Kangshuo Li, Ting Wang. Yanjie Dong, Victor CM Leung, and Fangxin Wang. Fedfmsl: Federated learning of foundations models with sparsely activated lora. IEEE TMC, 2024.
[27] Liping Yi, Han Yu, Gang Wang. Xiaoguang Liu, and Xiaoxiao Li. pfedlora: hodeeehheteoogeneuuu personalized federated learning with lora tuning. arXiv preprint 1rXiv:2310.13283. 2023.
[28] Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. In SIGKDD, 2024.
[29] Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li. Bolin Ding, and Jingren Zhou. Federatedscope-IIn A comprehensive package for fine-tuning large language models in federated learming. In SIGKDD, 2024.
[30] Guangyu Sun, Umar Khalid, Matias Mendieta, Pu Wang, and Chen Chen. Exploring parameter- efficient fine-tuning to enable foundation models in federated learning. In IEEE BigDala, 2024.
[31] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In EMNLP, 2024.
[32] Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In NeurlPS, 2024.
[33] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang. Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In NeurlPS, 2024.
[34] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In ICML, 2021.
[35] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurIPS, 2019.
12

---

[36] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix label-noise learning. In NeurlPS, 2020.
[37] Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, and Tongliang Liu. Esti- mating noise transition matrix with label correlations for noisy multi-label learning. In NeurIPS, 2022.
[38] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks corrupted labels. In ICML, 2018.
[39] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
[40] Yang Lu, Yiliang Zhang, Bo Han, Yiu-ming Cheung, and Hanzi Wang. Label-noise learning with intrinsically long-tailed data. In ICCV, 2023.
[41] Yuyin Zhou, Xianhang Li, Fengze Liu, Qingyue Wei, Xuxi Chen, Lequan Yu, Cihang Xie, Matthew P Lungren, and Lei Xing. L.2b: Learning bootstrap robust models for combating label noise. In CVPR, 2024.
[42] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama. Co-teaching: robust training of deep neural networks with extremely noisy labels. In NeurlPS, 2018
[43] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICMI., 2019.
[44] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2020.
[45] Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. In NearlPS, 2021.
[46] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2020.
[47] Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In ICLR, 2020.
[48] Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018
[49] Ehsan Amid, Manfred K Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. In NeurIPS, 2019.
[50] Xingjun Ma, Hanxun Huang, Yisen Wang. Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In ICML, 2020.
[51] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019.
[52] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML, 2020.
[53] Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In CVPR, 2022.
[54] Jingyi Xu, Zihan Chen, Tony QS Quek, and Kai Fong Ernest Chong. Fedcor:: Multi-stage federated learning for label noise correction. In CVPR, 2022.
[55] Nannan Wu, Li Yu, Xuefeng Jiang, Kwang-Ting Cheng, and Zengqiang Yan. Fednoro: towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity. In IJCAI, 2023.
13

---

[56] Xinyuan Ji, Zhaowei Zhu, Wei Xi, Olga Gadyatskaya, Zilong Song, Yong Cai, and Yang Liu. Fedfixer: mitigating heterogeneous label noise in federated learning. In AAAI, 2024.
[57] Yang Lu, Lin Chen, Yonggang Zhang, Yiliang Zhang, Bo Han, Yiu-ming Cheung, and Hanzi Wang. Federated learning with extremely noisy clients via negative distillation. In AAAI, 2024.
[58] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. In EMNLP, 2021.
[59] Xiyuan Yang, Wenke Huang, and Mang Ye. Dynamic personalized federated learning with adaptive differential privacy. In NeurlPS, 2023.
[60] Wenke Huang, Mang Ye, Zekun Shi, Bo Du, and Dacheng Tao. Fisher calibration for backdoor- robust heterogeneous federated learning. In ECCV, 2024.
[61] Xiuwen Fang, Mang Ye, and Xiyuan Yang. Robust heterogeneous federated learning under data comuptiion. In ICCV, 2023.
[62] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
[63] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-wware visual parameter-efficient fine-tuning, In ICCV, 2023.
[64] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[65] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[66] Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In ICML, 2023.
[67] Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, and Cheng Wang. Fedpft: Federated proxy fine-tuning of foundation models. In IJCAI, 2024.
[68] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. NAACL-HLT, 2018.
[69] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[70] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Please refer Sec. 6.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: lease refer to Sec. 5.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification; We use publicly-accesseen dataset CIFAR-100 [62]. The code implementation is available via an online repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ /tte://nnpppppe cc/
public/ /ttic//gu/ssseeee /Cod/oooeuuomi ssi sisoooooicyyy cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https: / /nips. cc/
//tps.cc/publicc.o httpp/ublcccccc htte/pbblc//guuieeseeseeeee /uuuidesssseeeeeeigee Codee Submi ssnisssonn ooPollcng for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: Please refer Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are reproducible with low errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Sec. 5.1.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips....
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This research foundational, focusing on algorithmic improvements for federated learning. Potential positive societal impacts could arise from more robust and reliable privacy-preserving machine learning applications. We do not foresee direct negative societal impacts from the methodology itself, though any deployed machine learning system requires careful consideration broader implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
19

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 5.1.
Guidelines:
The answer NA means that the paper does not existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper sewieswiiin thoode com/ dataaaaaaa dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non- non-standard components.
Please refer to our LLM policy (https: httas:///eeeuue //nuunnusppp...// www.c///eneeeeceeeeeeccee /202/ LLM) for what should should not be described.
21

---

A The Algorithm of RFedLR
We summarize the Pseudocode of RFedLR in Algorithm 1.
Algorithm 1: Training process of RFedLR
Input:
Total communication rounds T,, number of clients K, private noisy dataset Dy for each client k. global public clean proxy dataset D, and its noisy version Dn, global pre-trained model parameters Wo. initial global LoRA matrices Ali B learning rate y).
Server Execute:
for each communication round ( = 0....... - do
Calculate importance scores LAg. Ing by Eqs. (10) and (11)
Select trainable matrix X with higher importance and the other is frozen matrix Y
for each client & = 1.,...., K in parallel do Send A/.- Bg De: Dn,, and the selection X to client k
x xxp LocalUpdate(AA) Big De. X')
end
Calculate aggregation weight wf and aggregate the received matrices X++1 by Eq. (13)
end
Local Update:
Function LocalUpdate( Ali, Bi. De, Dn: X
Receive Ab. Bg. De, Dn. and selection X from Server
Initialize local matrices Af. + Ab, B + B Let local model be W = Wo + B
Compute gradients on proxy data: Vw, C.W, D.) and Vw, (W Dn)
Calculate the normalized parameter sensitivity score Sr by Eqs. (5) and (6)
Generate binary mask M by Eq. (7)
Update only the trainable matrix X by Eq. (8)
Recalculate the importance score |x--1 for the updated matrix X!+1
return X!+1 [x--
B Additional Experimental Results
B.1 Hyperparameter Sensitivity Analysis
22
Actunccy CM
81,62
77
76,00
71.21
Figure 5: Parameter analysis for in the SRT module. The analysis is conducted under the 20% pairflip noise setting.
We include sensitivity analysis for the key hyperparameters 7 and X. For the parameter keep rate in SRT (Fig. 5), our analysis shows that performance peaks at our chosen value r = 0.2, achieving an accuracy of 83.12%. When is too low, the model plasticity is limited. When is too high, the regularization effect the selective update strategy is weakened and performance decline.
For the balancing hyperparameter À in AFLA (Tab. 4), our results show that the performance of RFedLR relatively stable for the selection of X. When À 0.4, RFedLR R achieves the highest test accuracy, which can effectively balance the contribution of data scale and matrix importance.
22

---

Table 4: Parameter analysis for module. The analysis is conducted under the 20% symflip noise setting.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Aec 86.22
85.50
85.46
86.97
85.34
85.84
85.42
86.06
B.2 Component Analysis of the Hybrid Importance Score AFLA
Table 5: Ablation study on the components of the hybrid importance score. The table compares the average test accuracy (%) of RFedLR (Ours) with variants that exclude Fisher information (w/o F) or noise sensitivity (w/o S) from the importance calculation in Eq. (10).
0.2
0.1
Method
Pairflip
Symflip
Pairfip
Symflip
Ours
83,12
86.97
67.08
84.44
w/o P
82.75
85.23
66.69
84.38
77.42
80.86
61.04
77.44
The hybrid importance combines FIM with noise sensitivity metric. We adopt the empirical FIM to quantify the parameter importance, as it is a classic and effective measure parameter impact model prediction. Parameters with high Fisher Information values most affect predictive ability. Although FIM can identify important parameters under ideal conditions, it ignores the parameter stability in noisy environments. Therefore, design the hybrid importance score I = (S cc to balance the contribution and reliability of parameters.
To validate this design choice, we supplement this with ablation experiments (Tab. 5) to analyze the respective contributions of and S to the final performance. We compare the RFedLR method that removes the Fisher information component ("w/o F," using 1 + E)) and the other that removes the noise sensitivity component ("w/o S," using / = F).
("Ours") with two variants:
The results confirm that both components contribute positively, justifying our combined formulation. The significant performance drop in the "w/o S" case confirms that accounting for noise sensitivity crucial for robust aggregation. Using Fisher information alone is insufficient and can be misleading noisy environments. The full method always outperforms the "w/o F case, which indicates that Fisher information further refines the importance score. The superiority of this hybrid score over using either Fisher information or noise sensitivity alone.
B.3 Effectiveness on NLP Tasks
Table 6: Comparison with SOTA methods on MNLI task from GLUE benchmark when the noise rate is 0.2 and noise type is symflip. We take the average test accuracy (%) of local models for demonstration.
Method
FedPF
FeaBF
FedAP
FeaLR
SLoRA
FEA-LORA
RoLoRA
FeeLoRA
RFedLR
67.64
69.46
71.2
To validate the gencralizability of RFedLR, we conduct experiments on NLP tasks. Following existing federated PEFT works [13, 15, 67], we perform an experiment on the MNLI [68] task from the GLUE [69] benchmark. We utilize Goberta-Bass [70] as the initial global model. The experimental setup mirrored our vision experiments, with clients holding non-IID data partitions and operating under 20% symflip label noise. As shown in the Tab. 6, RFedLR achieves accuracy of 71 11.42%, significantly outperforming all baseline methods. The results confirm that our framework effective for NLP tasks in noisy federated settings.
B.4 Performance in Large-Scale Scenarios
Table 7: Performance Comparison in large-scale federated setting (100 total clients, 10% participation rate)
with 40% pairflip noise. We take the average test accuracy of the local models for demonstration.
Method
PedLR
SLoRA
FFA-LORA
RoLoRA
FlexLoRA
RFedLR
Ace
59.04
53,91
58.74
58.75
54.17
59.61
Validating the scalability and generality of our method in a setting with larger number of clients is essential for demonstrating its practical applicability. Therefore, we conduct experiments to simulate more realistie, large-scale cross-device scenario. This experiment involves 100 total clients with 10% participation rate, meaning 10 clients sampled for training and aggregation in each
23

---

communication round. The results in Tab. 7 show that RFedLR framework maintains its performance advantage in large-scale systems.
B.5 Comparison with SOTA Federated Noise Learning Methods
Table 8: Performance comparison of RFedLR with SOTA robust federated learning methods under a noise rate
of 0.4. Values represent average test accuracy (%)
Type
FedCorr
RHFL
FedFixer
RFedLR
Pairflip
55.82
57.44
55.61
67.08
Symdlip
79.80
84.72
77.82
84.64
The experiments in Tabs. 2 and 3 focused on comparing RFedLR with other federated PEFT meth- ods. Additionally, We implement and evaluate robust federated learning methods, FedCorr [54], RHFL [53], FedFixer [56]. As these methods are for conventional full-parameter FL, we adapt their core noise-handling mechanisms to our federated PEFT setting for a fair comparison. The experiments are performed under a noise rate of 0.4. These results (Tab. 8) demonstrate that RFedL_R outperforms these robust FI. methods. We analyze that these methods cannot mitigate the aggregation bias inherent in federated LoRA, while our framework is designed with SRT to counter local noise amplification and AFLA to address aggregation bias and noise propagation.
B.6 Performance under Severe Data Heterogeneity
Table Comparison with SOTA methods under severe data heterogeneity (Dirichlet concentration parameter is 0.1) when the noise rate is 0.4. We take the average test accuracy (%) of local models for demonstration.
Type
FedPF
FedBF
FedAP
FedLR
SLoRA
FFA-LORA
RoLoRA
FleLLORA
RFedLR
Parttip
42.62
50.61
50.98
51.90
50.83
48.38
$2,06
51.75
62,56
Symilip
63,22
71.0
62.56
73.01
75.24
68.55
74.72
72.33
80.48
We have conduct the experiments in a highly non-IID setting by setting the Dirichlet concentration parameter to 0.1, with noise rate of 0.4. The results (Tab. 9) show that the performance advantage of RFedLR becomes even more pronounced in this severe data heterogeneous scenario. We analyze the advantages to be attributed to the AFLA mechanism. AFLA weights clients based on the amount of data, and the estimated importance and stability of their LoRA updates, ensuring robust aggregation results in highly non-lII) scenarios.
B.7 Robustness to Mismatched Proxy Datasets
Table 10: Performance comparison with the SOTA methods, where the local dataset is set to a subset of CIFAR-100. "RFedL and "RFedL "RFedL R(CIFAR-100) respectively indicate that we use subset of CIFAR-IO and CIFAR-100 as the proxy dataset. The best and second-best results are highlighted in bold.
0.2
0.4
Pairflip
Symflip
Pairflip
Symflip
FedPF
75.64
50.10
FedBF
74.78
83.12
55.95
78.50
FedAP
75.48
82.81
55.60
78.62
FedLR
75.28
83.65
56.62
79.69
SLoRA
76.48
84.50
56.77
81.51
FFA-LORA
73.20
80.40
54,78
77.41
RoLoRA
75.73
84.55
56.93
81.31
FlexLoRA
83.61
56.58
79.71
RFedLR (CIFAR-10)
76,70
84.54
57.22
81.96
RFveLRR (CHEAR-1OO) 100)
83.12
86.97
67.08
84.64
To validate the generalizability and adaptability of our proposed method, we supplement experiments by setting the proxy dataset to CIFAR-10, while the local dataset remain CIFAR-100, This setup introduces a significant mismatch in the label space and task complexity. The results in Tab. 10 show that the performance of RFedLR is reduced when using the mismatched proxy (CIFAR-10) compared to using the matched proxy (CIFAR-100) 100). However, even with this significant mismatch. RFedLR remains robust. RFedLR (CIFAR-10) still outperforms other state-of state-of-the of-the-art art methods in various scenarios. The purpose of using the proxy dataset to estimate the parameter sensitivity to gradient perturbations induced by label noise. While task-irrelevant proxy dataset may not perfectly capture parameter sensitivity, is still sufficient to detect the most significant instabilities, thereby identifying the parameters most susceptible noise-induced instability.


Reviews:
[{"review_id": "OSQXbtYMBQ", "reviewer": "Reviewer_MLqt", "summary": "This paper is the first to study the problem of robust federated PEFT in label noisy scenarios. LoRA is a common PEFT method that helps reduce costs but is highly sensitive to label noise and aggregation inconsistency in the federated setting. RFedLR addresses this problem by introducing two main components: sensitivity-aware robust tuning, which identifies and updates only the parameters most sensitive to noise during local tuning, and adaptive federated LoRA aggregation, which selectively aggregates LoRA updates based on their importance and stability. The paper evaluates RFedLR in different label noisy scenarios and shows that it significantly outperforms existing methods in accuracy and efficiency.", "questions": "The authors should provide additional resource consumption comparisons with other methods and sensitivity analysis experiments on all hyperparameters. Besides, the authors should correct the inconsistencies and typos in the paper (mentioned in weaknesses), especially the discrepancy between Eq.5 and its description.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. This paper explores the practical but underexplored problem of robust federated fine-tuning in noisy scenarios, providing a clear motivation for this research.\n\n2. The proposed method is well-designed and innovative. Clear illustrations and figures make the method easy to understand.\n\n3. The experimental setup is thorough and provides a convincing evaluation, including ablation studies, comparisons under different noise types and levels, and performance across clients.\n\nWeaknesses\n\n1. The calculation of gradient differences and Fisher information in RFedLR will introduce additional overhead. This may affect its adaptability on low-resource devices. How does the computation time or resource consumption of this method compare with the baseline method?\n\n2. The parameter keep rate \\tau in SRT is fixed to 0.2, and the balance hyperparameter \\lambda in AFLA is fixed to 0.4. There is a lack of discussion or sensitivity analysis on their selection, which may affect the generalizability of the method.\n\n3. Typos.\n- On page 5, in Eq. 5, the authors define noise sensitivity as the subtraction of two gradients, but previously described as \"the magnitude of the gradient discrepancy\". There is a difference between these two statements. Is the intention here to calculate the absolute value or norm of the difference?\n\n- For formula 10, it is recommended to specify whether the original sensitivity score or the normalized score is used.\n\n- The numerical format of Table 3 is inconsistent, please check.\n\n- There is a typo in line 171 on page 4, which should be \"the A and B matrices\".", "was_revised": true, "final_justification": "The authors’ rebuttal, particularly the new experiments, has resolved my initial concerns about the evaluation. This is a valuable contribution to federated graph learning, and I now recommend acceptance.", "rebuttal": "Dear Reviewer MLqt：\n\nThank you for your valuable feedback and for your time reviewing our work. We hope our responses below help clarify the issues.\n\n**W1: Computational Overhead**\n\nWe conduct a detailed analysis of the resource consumption of our proposed RFedLR compared to other baseline methods. As shown in Table R1, the results show that the additional memory cost of RFedLR is negligible. Specifically, its memory usage is 33642.74 MB, which is comparable to other baseline methods. Furthermore, RFedLR utilizes the fewest trainable parameters (0.1754%) and achieves the highest test accuracy among all compared methods.\n\nWe will add a table summarizing these findings to the appendix and discuss these results in the main paper.\n\n*Table R1: Comparison of trainable parameter(%) relative to FFT and memory cost(MB) for various methods.*\n| Methods  | Trainable Parameters (%) | Memory Cost (MB) |\n|:--------:|:-----------------------:|:----------------:|\n| FedPF    | 0.3042                  | 32475.57         |\n| FedBF    | 0.2085                  | 33005.22         |\n| FedAP    | 1.1413                  | 33107.69         |\n| FedLR    | 0.6107                  | 33633.73         |\n| SLoRA    | 0.6107                  | 46973.41         |\n| FFA-LoRA | 0.5221                  | 31858.00         |\n| RoLoRA   | 0.5249                  | 34144.27         |\n| FlexLoRA | 0.6107                  | 32713.07         |\n| RFedLR   | 0.1754                  | 33642.74         |\n\n**W2: Hyperparameter Sensitivity**\n\nWe include a sensitivity analysis for the key hyperparameters $\\tau$ and $\\lambda$. \nFor the parameter keep rate $\\tau$ in SRT (Table R2), our analysis shows that performance peaks at our chosen value of $\\tau=0.2$, achieving an accuracy of 83.12%. When $\\tau$ is too low, the model plasticity is limited. When $\\tau$ is too high, the regularization effect of the selective update strategy is weakened and performance decline.\n\n*Table R2: Parameter analysis for $\\tau$ in the SRT module. The analysis is conducted under the 20% pairflip noise setting.*\n| $\\tau$ | 0.1| 0.2| 0.3|0.4|0.5\n|:----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| Acc  | 71.31 | **83.12** | 81.62 | 77.89 | 76.08 |\n\nFor the balancing hyperparameter $\\lambda$ in AFLA (Table R3), our results show that the performance of RFedLR is relatively stable for the selection of $\\lambda$. When $\\lambda$ is 0.4, RFedLR achieves the highest test accuracy, which can effectively balance the contribution of data scale and matrix importance. \n\n*Table R3: Parameter analysis for $\\lambda$ in the AFLA module. The analysis is conducted under the 20% symflip noise setting.*\n| $\\lambda$ | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 \n|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| Acc   | 86.22 | 85.50 | 85.46 | **86.97** | 85.34 | 85.84 | 85.42 | 86.06 |\n\nWe will include these results and corresponding charts in the paper to improve the integrity of the paper.\n\n**W3: Manuscript Clarifications and Corrections**\n\nWe are grateful to the reviewer for the meticulous reading and for pointing out several areas for improvement in the manuscript. \n\nWe will correct all identified typos and inconsistencies. Specifically, we will revise the description of Eq. 5 and update the formula to $S_{k}^{i} = \\left| \\nabla_{W_{k}^{i}}L_{c} - \\nabla_{W_{k}^{i}}L_{n} \\right|$ to accurately reflect the calculation of the magnitude of the gradient discrepancy. For Eq.10, we need to clarify that the unnormalized sensitivity score ($S_k^i$) is used in the calculation of the hybrid importance score to avoid ambiguity. We will correct the inconsistent numerical formatting in Table 3 and the typos in the final version."}, {"review_id": "3jC2lrvjee", "reviewer": "Reviewer_N5Xd", "summary": "This work proposes a new federated PEFT framework, RFedLR, designed to improve global model performance when local clients possess label-noisy private datasets. RFedLR assumes the availability of a clean proxy dataset collected by the server, from which a synthetic label-noisy dataset is constructed. Both the clean and noisy datasets are distributed to the clients, who then compute the sensitivity of their local models by measuring the gradient discrepancy between models trained on the clean and noisy versions. Parameters identified as highly sensitive to label noise are deemed less reliable and are subsequently frozen during backpropagation. Furthermore, RFedLR incorporates an adaptive LoRA aggregation technique that accounts for both the data scale and parameter importance of each client, with the latter estimated using the Fisher Information Matrix. Experiments on the CIFAR-100 dataset demonstrate the effectiveness of the proposed RFedLR framework.", "questions": "1. What is the performance of clients who hold clean datasets? This is an important baseline that should be included, as it represents the theoretical upper bound of model performance.\n\n2. How does RFedLR perform in federated settings with greater data heterogeneity? For instance, if the Dirichlet concentration parameter $\\beta$ is increased to 0.1, does RFedLR lose its effectiveness? Furthermore, in scenarios where local data distributions are highly non-IID, is the small proxy dataset sufficient for accurately computing parameter sensitivity?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well-written, and the proposed methodology is clearly presented and easy to follow.  \n2. The motivation behind RFedLR is promising. Label noise is a naturally occurring issue in real-world datasets, and addressing it in the context of FL is both important and timely.  \n3. The authors conduct several experiments to demonstrate the effectiveness of the proposed RFedLR framework.\n\n**Weaknesses:**\n\n1. The technical novelty of the proposed approach appears somewhat limited. Leveraging gradient discrepancy to estimate parameter sensitivity to label noise is not a new idea and has been explored previously, particularly in literature related to backdoor attacks.  \n2. The experimental evaluation is narrow in scope. The study is restricted to the CIFAR-100 dataset, which is relatively small and stylized. It would strengthen the paper to include experiments on larger-scale datasets or consider the applicability of RFedLR to LLM fine-tuning tasks, as done in recent works.  \n3. The paper lacks sufficient details on how the server constructs the synthetic noisy dataset. For instance, what process is used to inject noise? The experiments mention two types of label noise, but it is unclear whether the server-generated noisy dataset adheres to these same noise patterns.  \n4. A related concern is that the server is unaware of the actual data distribution or noise type on each client. What happens if the server assumes pairwise flip noise while some clients actually follow symmetric label noise? This mismatch may undermine the effectiveness of the gradient discrepancy analysis.  \n5. The evaluated FL setting is highly cross-silo, involving only five clients. To validate the generality and scalability of RFedLR, it is important to evaluate it under cross-device settings with larger numbers of clients.", "was_revised": true, "final_justification": "The authors address most of my concerns in the rebuttal. As a result, I increased my score.", "rebuttal": "Dear Reviewer N5Xd：\n\nThank you for your insightful review and valuable feedback. We address your key concerns in detail below, aiming to clarify and demonstrate the effectiveness of our proposed approach.\n\n**W1: The Novelty of the paper**\n\nIn this paper, we investigate the novel problem of achieving robust federated PEFT in noisy environments. With the rapid growth of large-scale pre-trained models, this is a realistic and challenging problem. Compared to conventional robust federated learning, federated PEFT in noisy environments presents additional challenges. The low-rank factorization structure of LoRA can lead to noise amplification in noisy environments. Noisy perturbations generate new bias terms through matrix multiplication. Furthermore, federated LoRA aggregation suffers from inherent aggregation bias, exacerbated by noisy local updates, and this bias is propagated and amplified throughout the federated system.\n\nTo address these challenges, we propose RFedLR framework, which consists of two modules: (1) SRT selectively updates noise-sensitive parameters to enhance robustness at the local fine-tuning phase, and (2) AFLA introduces an adaptive aggregation scheme at the collaborative update phase to mitigate aggregation discrepancies and curtail noise propagation.\n\nPrior works, particularly in backdoor defense, have leverage gradient analysis to detect parameters affected by  the backdoor trigger. The backdoor literature aims to detect deterministic signals from a single static attack. In contrast, our approach aims to measure the parameter stability against random label noise. Its goal is not to defend against targeted attacks, but rather at achieving generalized robust regularization to prevent overfitting. Besides, some work prunes or permanently disables outlier parameters. Unlike these approaches, SRT maintains all parameters active during forward propagation to ensure the model representability is not compromised, while selectively updating noise-sensitive parameters during backpropagation. This conditional update mechanism is a more sophisticated robust training mechanism. Furthermore, the parameter sensitivity score is combined with Fisher information to form a hybrid importance score, based on which aggregate weights are assigned to the clients.\n\nIn summary, the novelty of our work lies in studying a new problem, analyzing its unique challenges, and proposing a new framework to address these challenges.\n\n**W2: Expansion of the experimental scope**\n\nAs our paper focuses on the challenges of fine-tuning large-scale Pre-trained Models (PTMs), we choose ViT-B (pre-trained on ImageNet-21K) as our backbone. Its fine-tuning on CIFAR-100 follows a standard and challenging benchmark from recent visual PEFT literature [1], ensuring our baseline comparisons are meaningful.\n\nTo demonstrate the broad applicability of our framework, we conduct experiments on a NLP fine-tuning task involving a language model (Roberta-Base model). We evaluate the performance of RFedLR on the MNLI task from GLUE benchmark with 20% symflip noise in a non-IID federated setting. The results are as follows: \n\n*Table R1: Comparison with the state-of-the art methods on MNLI task from GLUE benchmark when the noise rate is 0.2 and noise type is symflip. We take the average test accuracy of the local models for demonstration.*\n|Methods|FedPF|FedBF|FedAP|FedLR|SLoRA|FFA-LoRA|RoLoRA|FlexLoRA|RFedLR|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Acc|66.45|67.64|67.04|67.60|69.46|64.90|68.20|67.60|**71.42**|\n\nThe results show that RFedLR significantly outperforms existing federated PEFT methods in the language domain, confirming its applicability beyond image classification. We consider extending it to the LLM fine-tuning scenario in future work.\n\n**W3: The construction of the noisy proxy dataset**\n\nTo clarify, the noisy proxy dataset $D_n$ is generated by injecting symflip noise into the clean dataset $D_c$, which randomly flips a label to any other class with uniform probability. Symflip noise represents a general form of label noise that does not rely on any assumptions about class similarity. By evaluating parameter sensitivity against this high-entropy noise, we can identify parameters that are broadly unstable to any incorrect supervision signal. This ensures that our sensitivity analysis is robust and not biased towards a specific, structured noise pattern. The approach allows RFedLR to learn a more general measure of noise sensitivity, enhancing the overall robustness. \n\nThe design choice is validated by our experimental results (Tables 2&3), where RFedLR demonstrates superior performance under both symflip and pairflip noise settings, confirming the general applicability of the learned sensitivity measure. We will clarify this implementation detail in Section 4.1 of the revised manuscript.\n\n**W4: The construction of the noisy proxy dataset**\n\nThe effectiveness of our method does not depend on the assumption that the distribution or noise pattern of the proxy dataset must be match to the local data. \n\n1. Robustness to Data Distribution Mismatch: In our setup, the proxy dataset is a uniformly distributed mini-batch randomly sampled from the CIFAR-100 test set. However, the local data was explicitly configured to be non-IID. As demonstrated by our results, particularly in the highly data heterogeneous environments (Refer to Q2), our method performs well, which proves that distribution matching is unnecessary.\n2. Robustness to Noise Type Mismatch: As clarified in our response to W3, our framework utilizes symflip noise to generate noisy proxy data. Table 2&3 show that RFedLR still demonstrates remarkable effectiveness when clients have structured pairflip noise. Besides, Table 1 shows that the introduction of SRT module results in a significant accuracy improvement under pairflip noise with different noise rates. This shows that a mismatch between the proxy noise and the local true noise does not undermine the effectiveness of our gradient discrepancy analysis.\n\nThe proxy dataset is not used to simulate client local data, but to serve as a common, objective benchmark for identifying parameters that are unstable to noise. This instability is a general property of model parameters, rather than their response to specific data distributions or noise patterns. \n\n**W5: Effectiveness under cross-device settings**\n\nWe agree that validating the scalability and generality of our method in a setting with a larger number of clients is essential for demonstrating its practical applicability. To address this concern, we conduct experiment in a large-scale setting designed to simulate a cross-device federated learning scenario. This experiment involves 100 total clients with a 10% participation rate, meaning 10 clients are sampled for training and aggregation in each communication round. The results, presented in the table below, show that RFedLR outperforms other methods in this more challenging and realistic environment.\n\n*Table R1: Performance Comparison in a large-scale federated setting (100 total clients, 10% participation rate) with 40% pairflip noise. We take the average test accuracy of the local models for demonstration.*\n|Method|FedLR|SLoRA|FFA-LoRA|RoLoRA|FlexLoRA|RFedLR|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Acc|59.04|53.91|58.74|58.75|54.17|**59.61**|\n\nThis result demonstrates that the effectiveness of our proposed RFedLR framework is not limited to small-scale, cross-silo settings but scales successfully to systems with more clients. \n\n**Q1: Performance of clean clients**\n\nWe would like to clarify that this baseline is included in our analysis, presented in Figure 2 on page 2. The purpose of this figure is to visually demonstrate the gap in model performance between practical noisy the ideal clean scenarios. Specifically, Figure 2(b) shows the performance of Federated LoRA (FedLR), the baseline setting of our study. As can be seen from the figure, when all clients have clean datasets (shown by the gray dashed line), the final average accuracy of the model can reach 86.76%. When the client dataset contains noise (shown by the orange solid line), the accuracy drops to 75.28%. This 11.48% performance gap reveals the significant negative impact of label noise on Federated PEFT. This significant gap emphasizes the need to study a robust federated PEFT framework. Our proposed method RFedLR achieves an accuracy of 83.12% under the same noise conditions of the Figure 2. For clarity, we will add specific values to Figure 2 in the final version.\n\n**Q2: Performance under severe data heterogeneity**\n\nWe have conduct the experiments in a highly non-IID setting by setting the Dirichlet concentration parameter to 0.1, with a noise rate of 0.4. The results (Table R3) show that the performance advantage of RFedLR becomes even more pronounced in this severe data heterogeneous scenario. We analyze the advantages to be attributed to the AFLA mechanism. AFLA weights clients based on the amount of data, and the estimated importance and stability of their LoRA updates, ensuring robust aggregation results in highly non-IID scenarios.\n\n*Table R3: Comparison with the state-of-the art method under severe data heterogeneous scenario (Dirichlet concentration parameter is 0.1) when the noise rate is 0.4. We take the average test accuracy of the local models for demonstration.*\n|NoiseType|FedPF|FedBF|FedAP|FedLR|SLoRA|FFA-LoRA|RoLoRA|FlexLoRA|RFedLR|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Pairflip|42.62|50.61|50.98|51.90|50.83|48.38|52.06|51.75|**62.56**|\n|Symflip|63.22|71.70|62.56|73.01|75.24|68.55|74.72|72.33|**80.48**|\n\nThe small proxy dataset serves as a general benchmark for identifying which parameters are inherently unstable when exposed to label noise. This parameter-level property is largely independent of the specific data distribution.\n\n**References**\n\n[1] Sensitivity-aware visual parameter-efficient fine-tuning, in ICCV 2023."}, {"review_id": "1PfRkGVvUz", "reviewer": "Reviewer_yHga", "summary": "The paper tackles the challenge of LoRA training on noisy labels under the Federated Learning scenario. The authors do so in four steps. First, the parameters most susceptible to model drifts due to noisy labels are measured by comparing differences between a clean proxy dataset and its noisy variant derived by injecting random label noise to it. Then a threshold is defined which allows to heuristically control the proportion of parameters (i.e. how much of the sensitive parameters) to drop out during local backward propagation, implemented via masking matrices. Thirdly, Fisher Information Matrix is used to compute the contribution importance of the decomposed LoRA parameter matrices for each of the clients' locally trained model from the previous step. Lastly, at aggregation time, the LoRA parameter weights to use for aggregation are chosen based on the importance values from the previous step. The proposed method is called RFedLR.\n\nExperiments are performed with the Cifar100 and ImageNet 21k datasets on ViT models, and RFedLR is compared against other state-of-the-art and shown to significantly outperform them in the various training scenarios.", "questions": "1. Please describe, in detail, how/where a realistic proxy dataset and its subsequent noisy version may be derived from in order to make this solution more generalizable to practical scenarios.\n2. How do we determine the value for Tau?\n3. How and why where the FIM and the IM determined? I.e. what is the intuition behind their usage?\n4. How does this perform across highly data heterogeneous environments?\n5. Can this be adapted by other variations of LoRA such as AdaLora\n6. What are the privacy implications of sharing the Parameter Importance matrices?", "limitations": "There may be privacy issues involved, as with all topics related to FL in general, but I do not foresee any problems specific to this paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths - \n1. The problem tackled is an interesting one, few papers explore this specific problem.\n2. The various methods used for importance sampling at different stages is creatively applied.\n3. The results are significantly better than state-of-the-art.\n\nWeaknesses - \n1. The use of a clean pre-available proxy data and adding noise artificially to determine importance is not a good practical solution since it is a very strong assumption that such data will always exist and that the noise is of a similar distribution to the clients' baseline data. How can that be guaranteed? This may be a very fundamental problem since every step of the solution depends on this.\n2. The experimental section and discussion sections is lacking in diversity in terms of data heterogeneity (a fundamental concern for all FL systems) and systems (number of total clients, clients per round, datapoints per client). \n3. The state-of-the-art compared against are not designed to tackle this specific challenge so the numbers may not be fair. The state-of-the-art are designed (mostly) for tackling LoRA challenges under data heterogeneity, which this paper completely avoids. On the other hand, state-of-the-art that are relevant (e.g. FedFixer) are not compared against.\n4. While the individual steps are clearly described, how it all fits together is not. \n5. Privacy implications are not discussed at all.", "was_revised": true, "final_justification": "Concerns about applicability and generalizability still remain.", "rebuttal": "Dear Reviewer yHga：\n\nThank you for the valuable comments. We hope our responses address your concerns and provide a clearer understanding of our work.\n\n**W1&Q1: Practicality and assumption of the proxy dataset**\n\nWe acknowledge that requiring a clean proxy dataset is a significant consideration, as we mentioned in Section 6. We follow the proxy dataset settings in published federated learning work [1] to achieve robust federated learning.\n\nIn response to Q1, a proxy dataset can be sourced from small, publicly available datasets relevant to the task, or from a trusted, verifiable participant in FL systems. We agree that this is a constraint and will emphasize it in the limitations.\n\nIn response to W1, the effectiveness of our method does not require that the distribution or noise pattern of the proxy dataset must be similar to the local data. \n1. Robustness to Data Distribution Mismatch: In our setup, the proxy dataset is a uniformly distributed mini-batch randomly sampled from the CIFAR-100 test set. However, the local data was explicitly configured to be non-IID. Especially in the highly data heterogeneous environments (Refer to Q4), our method performs well, proving distribution matching is unnecessary.\n2. Robustness to Noise Type Mismatch: The server injects uniform symmetric flip (symflip) noise into the proxy data for its analysis. However, our method achieves its most significant performance gains when clients are affected by structured pairflip noise (Table 2&3). This demonstrates that our approach is effective even with mismatched noise types.\n\nThe proxy dataset is not used to simulate client local data, but as a common benchmark to identify parameters that are unstable to noise. This instability is a general property of model parameters, rather than their response to specific data distributions or noise patterns. \n\n**W2: Experimental diversity and system scale**\n\nTo address this, we conduct experiments to simulate a more realistic, large-scale cross-device scenario. In this setup, we use 100 total clients with a 10% participation rate (10 selected per round). The results show RFedLR framework maintains its performance advantage in large-scale systems.\n\n*Table R1: Performance Comparison in a large-scale federated setting (100 total clients, 10% participation rate) with 40% pairflip noise. We take the average test accuracy for demonstration.*\n|Method|FedLR|SLoRA|FFA-LoRA|RoLoRA|FlexLoRA|RFedLR|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Acc|59.04|53.91|58.74|58.75|54.17|**59.61**|\n\nFurthermore, to address the reviewer's concern about the diversity of data heterogeneity, we conduct experiments under a severe non-IID setting (the Dirichlet concentration parameter is 0.1), as detailed in our response to Q4 (Table R5). These results show that RFedLR is more effective in highly heterogeneous data distribution, demonstrating its robustness.\n\nThese new results provide a more comprehensive validation of our method, confirming that RFedLR is effective in diverse federated systems. \n\n**W3: Comparison of SOTA federated noise learning methods**\n\nThe experiments in Tables 2&3 focused on comparing RFedLR with other federated PEFT methods. As our work introduces a novel framework within the federated PEFT domain, we believe it is necessary to compare its performance with federated PEFT methods.\n\nWe agree that comparing with the most relevant state-of-the-art is essential. We implement and evaluate robust federated learning methods, FedCorr [2], RHFL [3], FedFixer [4]. \nAs these methods are for conventional full-parameter FL, we adapt their core noise-handling mechanisms to our federated PEFT setting for a fair comparison. The experiments are performed under a noise rate of 0.4. These results (Table R2) demonstrate that RFedLR outperforms these robust FL methods. We analyze that these methods cannot mitigate the aggregation bias inherent in federated LoRA, while our framework is designed with SRT to counter local noise amplification and AFLA to address aggregation bias and noise propagation.\n\n*Table R2: Performance comparison of RFedLR with SOTA robust federated learning methods under a noise rate of 0.4. Values represent average test accuracy (%).*\n|Noise Type|FedCorr|RHFL|FedFixer|RFedLR|\n|:-:|:-:|:-:|:-:|:-:|\n|Pairflip|55.82|57.44|55.61|67.08|\n|Symflip|79.80|84.72|77.82|84.64|\n\n**W4: Integration of each components**\n\nThe process for a single round of RFedLR is as follows:\n\n- Server-Side Preparation (AFLA): At the start of a round, the server selects the more important LoRA matrix (A or B) as trainable and distributes the global model to the participating clients.\n- Client-Side Local Tuning (SRT): Each client first uses the public proxy data to calculate a noise-sensitivity mask for the trainable LoRA matrix. During local training, this mask is used to selectively update only the most noise-sensitive parameters, enhancing local robustness.\n- Server-Side Aggregation (AFLA): After clients upload their updated matrices, the server calculates a hybrid aggregation weight for each client based on matrix importance and data size. It then performs a weighted average of these updates to create the new global model.\n\nTo further clarify this workflow, we will add the detailed pseudocode to the appendix of our paper.\n\n**W5&Q6: Privacy implications**\n\nWe would like to clarify a potential misunderstanding. Clients do not compute or share any “parameter importance matrices”. In each round, the client performs local training on the specified LoRA matrix and calculates an importance score. In the aggregation phase, each client only uploads an updated LoRA matrix and a scalar importance score. We believe that transmitting a scalar score computed on a public proxy dataset does not introduce substantial new privacy risks. We will discuss the privacy impact in the final version.\n\n**Q2: Sensitivity analysis for $\\tau$**\n\nThe parameter $\\tau$ controls the proportion of parameters updated in our SRT mechanism. To provide best performance, we conduct a sensitivity analysis. Our results show that performance peaks at our chosen value of $\\tau=0.2$. A smaller value overly constrains the model, while a larger value diminishes the regularization effect against noise. \n\n*Table R3: Parameter analysis for $\\tau$ in the SRT module. The analysis is conducted under the 20% pairflip noise setting.*\n|$\\tau$|0.1|0.2|0.3|0.4|0.5\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Acc|71.31|**83.12**|81.62|77.89|76.08|\n\n**Q3: The intuition behind FIM and IM**\n\nWe adopt the empirical FIM to quantify the parameter importance, as it is a classic and effective measure of parameter impact on model prediction. Parameters with high Fisher Information values most affect predictive ability. \n\nThe \"IM\" mentioned by the reviewer likely refers to our \"hybrid importance score\", denoted as $I$ in Eq.10, which combines FIM with our noise sensitivity metric. Although FIM can identify important parameters under ideal conditions, it ignores the parameter stability in noisy environments. Therefore, we design the hybrid importance score \"$I = F / (S + \\epsilon)$\" to balance the contribution and reliability of parameters. We conduct ablation experiments on these two components (Fisher information and Noise sensitivity) to justify the design.\n\nTo validate this design choice, our ablation study (Table R4) that empirically demonstrates the superiority of this hybrid score over using either Fisher Information or noise sensitivity alone. The results confirm that both components contribute positively, justifying our combined formulation.\n\n*Table R4: Ablation study for the hybrid importance score. The table compares the average test accuracy (%) of RFedLR (Ours) with variants that exclude Fisher information (w/o F) or noise sensitivity (w/o S) from the importance calculation in Eq.10*\n|Method|µ=0.2 (pairflip)|µ=0.2 (symflip)|µ=0.4 (pairflip)|µ=0.4 (symflip)| \n|:-:|:-:|:-:|:-:|:-:|\n|Ours|**83.12**|**86.97**|**67.08**|**84.64**| \n|w/o F|82.75|85.23|66.69|84.38| \n|w/o S|77.42|80.86|61.04|77.44|\n\n**Q4: Performance under highly data heterogeneous environments**\n\nWe conduct the experiments under highly data heterogeneous environments by setting the Dirichlet concentration parameter to 0.1, with a noise rate of 0.4. The results (Table R5) show that the performance advantage of RFedLR becomes even more pronounced in this severe data heterogeneous scenario. We attribute this to the AFLA mechanism, which weights clients based on the amount of data, and the estimated importance and stability of their LoRA updates, ensuring robust aggregation results in highly non-IID scenarios.\n\n*Table R5: Comparison with the state-of-the-art methods under severe data heterogeneous scenario (Dirichlet concentration parameter is 0.1) when the noise rate is 0.4. We take the average test accuracy for demonstration.*\n|NoiseType|FedPF|FedBF|FedAP|FedLR|SLoRA|FFA-LoRA|RoLoRA|FlexLoRA|RFedLR|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Pairflip|42.62|50.61|50.98|51.90|50.83|48.38|52.06|51.75|**62.56**|\n|Symflip|63.22|71.70|62.56|73.01|75.24|68.55|74.72|72.33|**80.48**|\n\n**Q5: Adaptation to LoRA variants**\n\nThe core modules SRT and AFLA in RFedLR focus on the LoRA matrix parameters and do not rely on their fixed rank or structure. Therefore, the RFedLR framework can be effectively combined with other LoRA variants. For example, AdaLoRA dynamically allocates optimal, heterogeneous matrix ranks to different layers, and then RFedLR performs robust local training and global aggregation of these allocated matrices.\n\n**References**\n\n[1] Revisiting Weighted Aggregation in Federated Learning with Neural Networks, in ICML 2023.\n\n[2] Fedcorr: Multi-stage federated learning for label noise correction, in CVPR 2022.\n\n[3] Robust federated learning with noisy and heterogeneous clients, in CVPR 2022.\n\n[4] FedFixer: Mitigating heterogeneous label noise in federated learning, in AAAI 2024."}, {"review_id": "PqtkLhB22W", "reviewer": "Reviewer_Egmb", "summary": "This paper proposes RFedLR, a robust parameter-efficient fine-tuning method for federated learning (FL) under label noise. \n\nThe method has two main components: \n- (1) Sensitivity-aware robust tuning (SRT), where each client estimates parameter-wise noise sensitivity using gradients from a clean proxy dataset and masks updates to the most noise-sensitive parameters; and \n- (2) Adaptive Federated LoRA Aggregation (AFLA), which aggregates LoRA parameters from clients using weights based on both client data size and empirical importance scores derived from the Fisher information. \n\nExperiments on CIFAR-100 with synthetic label noise show that RFedLR consistently outperforms several PEFT and LoRA-based FL baselines, while maintaining a low communication and training footprint.", "questions": "1. Could the proposed framework be extended to NLP tasks, such as LLM fine-tuning? What are the main challenges in applying it beyond image classification?\n\n2. The notations in Equations (5)–(7) are unclear—are these quantities matrices, vectors, or scalars? Including dimension annotations or explanations in the text would improve readability.\n\n3. The aggregation weights in AFLA (Eq. 13) appear intuitive but lack theoretical support. Could the authors provide either a toy example or empirical evidence to show how the weighting scheme helps mitigate noise amplification? Even a high-level, non-rigorous explanation would be helpful.\n\n4. Figure 2’s caption should clarify what is meant by \"federated LoRA.\" Based on the results, its performance under moderate noise does not appear significantly degraded, which seems to contradict statements made in the main text.\n\n5. Could the authors provide more justification for the design of the overall importance score in Eq. (10)? The formulation feels heuristic—some explanation or ablation would be useful to understand its effect.", "limitations": "The authors mention reliance on a small clean proxy set and acknowledge computational overhead for sensitivity estimation. \n\nIt would be helpful to further discuss limitations such as scalability to many clients, applicability to other tasks (e.g., NLP), and the potential privacy implications of using proxy gradients.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper addresses an important challenge in FL where label noise can severely degrade model performance, especially under parameter-efficient fine-tuning. \n\n2. The method is practical, requiring only a small clean proxy set, and shows strong empirical performance with well-designed ablations. \n\n3. The paper is generally clear and well-structured, though some notation is hard to follow.\n\n**Weaknesses**\n\n1. The connection to LLM fine-tuning, which motivates the work, is not followed through—no NLP experiments are provided. \n\n2. The notation for sensitivity scores (e.g., $S_k^I$) is inconsistent and confusing, alternating between scalars and matrices. \n\n3. The aggregation scheme in AFLA is heuristic and lacks theoretical justification or analysis. \n\n4. Lastly, reliance on a clean proxy set may be limiting in privacy-sensitive applications.", "was_revised": true, "final_justification": "All of my questions and concerns have been addressed. I would increase my points.", "rebuttal": "Dear Reviewer Egmb：\n\nThank you for your thoughtful review and for raising key concerns regarding our work. We hope the following responses will address your concerns and update the score.\n\n**W1&Q1: Effectiveness on NLP tasks**\n\nTo validate the generalizability of RFedLR, we conduct an additional experiment on NLP tasks. Following existing federated PEFT works [1][2][3], we perform an experiment on the MNLI task from the GLUE benchmark. We utilize Roberta-Base as the initial global model. The experimental setup mirrored our vision experiments, with clients holding non-IID data partitions and operating under a 20% symflip label noise. As shown in the Table R1, RFedLR achieves a accuracy of 71.42%, significantly outperforming all baseline methods. The results confirm that our framework is effective for NLP tasks in noisy federated settings.\n\n*Table R1: Comparison with state-of-the art methods on MNLI task from GLUE benchmark when the noise rate is 0.2 and noise type is symflip. We take the average test accuracy of the local models for demonstration.*\n| Methods | FedPF | FedBF | FedAP | FedLR | SLoRA | FFA-LoRA | RoLoRA | FlexLoRA | RFedLR    |\n|:-------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|:-----:|:------:|\n|   Acc  |  66.45 | 67.64 | 67.04 | 67.60 | 69.46 | 64.90    | 68.20  | 67.60    | **71.42** |\n\nDue to the time constraints of the rebuttal period, these NLP experiments were conducted for one representative task and noise setting. We will provide the full experimental results in the final version.\n\n**W2&Q2: Notational Clarity and Consistency**\n\nWe thank the reviewer for pointing out this ambiguity in our notation. $S_k^i$ in Eq.5 is a scalar representing the sensitivity of a single parameter. In Eq.6, the collection of these sensitivity scores forms a sensitivity tensor $S_k$, which has the same dimensions as its corresponding parameter tensor. Therefore, $M_k$ in Eq.7 is a binary mask tensor with the same shape. We will clarify these symbols and their dimensions in the final version.\n\n**W3&Q3: Empirical validation of the AFLA weighting scheme**\n\nTo provide the empirical evidence for our AFLA weighting scheme, we perform experiments in a controlled environment designed to validate its ability to mitigate noise amplification. We design a federated learning system with three clients. Client A is configured to be large but noisy (12,000 data points, 80% noise), Client B is configured to be small but clean (2,000 data points, clean), and Client C serves as a control (6,000 data points, 20% noise).\n\nAs demonstrated in the Table R2, the standard FedAvg approach, which relies solely on data size, naively assigns the highest weight to the noisy Client A. This fixed weighting leads to severe noise amplification from Client A, resulting in a poor final accuracy of only 29.54%. In contrast, our AFLA mechanism considers both importance scores and data size to give more comprehensive weight allocation. It identifies Client A as an unreliable participant and reduces its average aggregation weight to 0.35. Simultaneously, it raises the weight of the small but reliable Client B to 0.32. This dynamic re-weighting mitigate the noise propagation and significantly improves accuracy. \n\n*Table R2: Empirical validation of the AFLA weighting scheme in a controlled 3-client scenario. The weights in AFLA  are dynamic each round, we show the average weight across all rounds to summarize its effective influence.*\n| Method |The Weighting Basis| Weight of Client A| Weight of Client B|Weight of Client C| Acc(%) | \n|:---:|:---:|:---:|:---:|:---:|:---:|\n |FedAvg|Data Size|0.60 | 0.10 | 0.30 | 29.54 | \n |AFLA|Importance & Data Size|0.35 | 0.32 | 0.33| 38.87 | \n\n\n**W4: Discussion of privacy restrictions**\n\nWe agree that the reliance on a public proxy dataset can be a limitation. We have transparently acknowledged in the conclusion of our paper (Section 6). However, our method preserves the foundational privacy guarantee of federated learning, i.e., no private client data is ever shared. We follow the published federated learning research [4] in setting up proxy datasets, which can come from small public datasets relevant to the task, or from trusted participants in the federated learning system. In our work, we require only a mini-batch dataset to serve this purpose. Crucially, the proxy dataset does not need to be consistent with the local client data distribution, and the generated noise proxy dataset does not need to match locally specific noise patterns.\n\n**Q4: Clarification of Figure 2**\n\nWe will clarify the caption of Figure 2 to indicate that “Federated LoRA” refers to the baseline FedLR method, where FedAvg is directly applied to LoRA. In addition, the accuracy difference between clean and noisy environments for federated FFT reaches 5.04%. However, the accuracy difference between Federated LoRA (FedLR) in clean and noisy environments reaches 11.48%. The presence of label noise poses a significant barrier to the robust performance of federated LoRA (FedLR). We will add specific values to Figure 2 in the final version.\n\n**Q5: Ablation experiments for the importance score**\n\nFisher information (F) is used to approximate the importance of parameters on model decisions. Higher Fisher values indicate a greater influence of a parameter on the model output. The noise sensitivity score (S) measures the sensitivity of parameter updates to label noise. Lower sensitivity scores indicate greater stability and reliability. The formula $I = F / (S + \\epsilon)$ aims to prioritize parameters that have a high impact on the task and are robust to label noise.\n\nWe supplement this with ablation experiments (Table R3) to analyze the respective contributions of F and S to the final performance. We compare the RFedLR method (\"Ours\") with two variants: one that removes the Fisher information component (\"w/o F,\" using $I = 1/ (S + \\epsilon)$ ) and the other that removes the noise sensitivity component (\"w/o S,\" using $I = F$).\n\n*Table R3: Ablation study for the hybrid importance score. The table compares the average test accuracy (%) of RFedLR (Ours) with variants that exclude Fisher information (w/o F) or noise sensitivity (w/o S) from the importance calculation in Eq. (10)*\n| Method | µ=0.2 (pairflip) | µ=0.2 (symflip) | µ=0.4 (pairflip) | µ=0.4 (symflip) | \n| :--- | :---: | :---: | :---: | :---: |\n | Ours  | **83.12** | **86.97** | **67.08** | **84.64** | \n | w/o F| 82.75 | 85.23 | 66.69 | 84.38 | \n | w/o S| 77.42 | 80.86 | 61.04 | 77.44 |\n\nThe significant performance drop in the \"w/o S\" case confirms that accounting for noise sensitivity is crucial for robust aggregation. Using Fisher information alone is insufficient and can be misleading in noisy environments. The full method always outperforms the “w/o F” case, which indicates that Fisher information further refines the importance score.\n\n**References**\n\n[1] Improving LoRA in Privacy-preserving Federated Learning, in ICLR 2024.\n\n[2] FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models, in ACL 2023.\n\n[3] FedPFT: Federated Proxy Fine-Tuning of Foundation Models, in IJCAI 2024.\n\n[4] Revisiting Weighted Aggregation in Federated Learning with Neural Networks, in ICML 2023."}]

Author rebuttal:

Meta-review: This paper studies an important problem of robust federated lLoRA against label noises. The key idea is to introduce clean proxy datasets, train the models on both the clean and noisy datasets over the clients, compare their the gradient discrepancy to keep/ignore the parameters sensitive/insensitive to the noises. The reviewers point out several unsolved issues of the current version including the reliance on proxy datasets may raise serious privacy issues and limit its generalizability, tau tuning can lead to non-trivial performance drop, and insufficient experimental evaluation. I recommend the authors to take the reviewers' suggestions into account to further improve their work.
Decision: Accept (poster)