Dynamic Personalized Federated Learning with Adaptive Differential Privacy
Authors: Xiyuan Yang
Wenke Huang
Mang Ye
Venue: NeurIPS 2023 poster (2023)
Keywords: federated learning
differential privacy
personalization
TL;DR: A novel approach towards personalized federated learning under differential privacy protection.

Abstract: Personalized federated learning with differential privacy has been considered a feasible solution to address non-IID distribution of data and privacy leakage risks. However, current personalized federated learning methods suffer from inflexible personalization and convergence difficulties due to two main factors: 1) Firstly, we observe that the prevailing personalization methods mainly achieve this by personalizing a fixed portion of the model, which lacks flexibility. 2) Moreover, we further demonstrate that the default gradient calculation is sensitive to the widely-used clipping operations in differential privacy, resulting in difficulties in convergence. Considering that Fisher information values can serve as an effective measure for estimating the information content of parameters by reflecting the model sensitivity to parameters, we aim to leverage this property to address the aforementioned challenges. In this paper, we propose a novel federated learning method with Dynamic Fisher Personalization and Adaptive Constraint (FedDPA) to handle these challenges. Firstly, by using layer-wise Fisher information to measure the information content of local parameters, we retain local parameters with high Fisher values during the personalization process, which are considered informative, simultaneously prevent these parameters from noise perturbation. Secondly, we introduce an adaptive approach by applying differential constraint strategies to personalized parameters and shared parameters identified in the previous for better convergence.  Our method boosts performance through flexible personalization while mitigating the slow convergence caused by clipping operations. Experimental results on CIFAR-10, FEMNIST and SVHN dataset demonstrate the effectiveness of our approach in achieving better performance and robustness against clipping, under personalized federated learning with differential privacy.

Dynamic Personalized Federated Learning with Adaptive Differential Privacy
Xiyuan Yang!* Wenke Huang Mang Ye
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory Multimedia and Network Communication Engineering, School of Computer Science, Hubei Luojia Laboratory. Wuhan University, Wuhan, China.
(yangxiyuan, Nenkehuang, yemang) @whu. edu. cn httpa://github. wttpuut...o///iaaaaaaaaaa www/:/iyanyyngg///yyyaaaaa
Abstract
Personalized federated learning with differential privacy has been considered feasible solution to address non-IID distribution of data and privacy leakage risks. However, current personalized federated learning methods suffer from inflexible personalization and convergence difficulties due to two main factors: 1) Firstly, observe that the prevailing personalization methods mainly achieve this by personalizing a fixed portion of the model, which lacks flexibility. 2) Moreover, we further demonstrate that the default gradient calculation is sensitive the widely-used clipping operations in differential privacy, resulting in difficulties in convergence. Considering that Fisher information values can serve as effective measure for estimating the information content of parameters by reflecting the model sensitivity to parameters, we aim to leverage this property to address the aforementioned challenges. In this paper, we propose novel federated learning method with Dynamic Fisher Personalization and Adaptive Constraint (FedDPA) handle these challenges. Firstly, by using layer-wise Fisher information to measure the information content of local parameters, we retain local parameters with high Fisher values during the personalization process, which are considered informative, simultaneously prevent these parameters from noise perturbation. Secondly, we introduce an adaptive approach by applying differential constraint strategies personalized parameters and shared parameters identified in the previous for better convergence. Our method boosts performance through flexible personalization while mitigating the slow convergence caused by clipping operations. Experimental results CIFAR- 10, FEMNIST and SVHN dataset demonstrate the effectiveness of our approach in achieving better performance and robustness against clipping, under personalized federated learning with differential privacy.
1
Introduction
32 has shown great potential in facilitating collaborative machine learning across distributed devices. However, in real-world scenarios, local data normally present non-IID
Federated lcarning (FL)
26 (non-independent and identically distributed), posing data heterogeneity 37 in FL.. To address these challenges, various personalized federated learning (PFL) algorithms have emerged [39 2 30 28). demonstrating significant effectiveness by incorporating personalized components that cater to the local data characteristics of each client. Despite their success, PFL faces severe privacy leakage issues, as client-local training data may be reconstructed through inference attacks [1]
40
44
43
10
16
27
challenges [19
22
14
33
T8
42 during the collaborative updating phase. As
"Equal contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

result, it is essential to adopt user-level differential privacy (DP) techniques 12 8 41 38 to provide more stringent privacy protection PFL against potential privacy leaks, a.k.a, personalized federated learning with differential privacy.
Despite its pilot progress, personalized federated learning with differential privacy still faces numerous challenges, Most existing PFL approaches [30 28 3 2 primarily rely on strong prior assumptions to perform parameter partitioning, personalizing fixed portion of the model parameters and sharing the remaining ones. However, this approach lacks flexibility in parameter partitioning and cannot fully adapt to the diverse data characteristics of different clients, thereby affecting the personalized model performance. Besides, adding noise is an indispensable aspect in the differential privacy setting to ensure privacy protection in PFL. However, previous methods [8] 38 3 directly add noise to all the network parameters, which unavoidably causes performance degradation. Considering that partial personalized parameters might not rely on the feedback from the server, this provides potential to reduce the noise impact. In light of these limitations, an essential issue that has long been overlooked 1 how to develop a flexible personalization approach that enhances the learning capability of model while mitigating the impact of noise on the model.
In order achieve comprehensive user-level differential privacy, it is also crucial to implement a clipping operation targeting the L2 norm of each local update 22] 41. These updates are sent to the server upon the completion of local training, ensuring that their magnitudes are bounded by the clipping bound. However, traditional optimization methods [12 38) primarily focus on minimizing empirical risk, disregarding constraints on the L2 norms. Consequently, the local updates accumulated by gradients are susceptible to clipping, leading to distortion phenomena that can adversely affect model convergence. Existing methods for enhancing robustness N8 23) against clipping often apply uniform 1.2 regularization strength to all parameters. Such approaches neglect the inherent differences in parameter magnitudes, constraining the model learning capacity. This raises another crucial question: @ how can we devise dynamic constraint mechanism that maintains the learning capucity of the model as well as augments clipping robustness?
In response to we propose an adaptive method that dynamically determines the personalized parameter portion for each client, while simultaneously mitigating the influence of noise. Our approach is inspired by the Fisher information matrix, which refecct the contribution of parameters to the curvature of the loss function by computing the square of the first-order derivatives of log- likelihood function. This mathematical process can also be intuitively understood as the information content carried by the parameters. Following this concept, we further introduce layer-wise Fisher information to measure the information content of client parameters. Prior to client training, we retain essential parameters as personalized components, implementing personalization by preventing them from being overwritten by the corresponding noisy global parameters. By employing this adaptive personalization strategy, our method ensures that locally retained parameters remain unaffected by the noise introduced for privacy protection, effectively overcoming the challenges associated with inflexible personalization and the impact of noise.
To tackle the issue of local update robustness to different constraint strengths for different parameters. Taking into account the inherent differences in parameter magnitudes, we only apply L.2 norm regularization constraints to the updates of per- sonalized parameters selected on account of layer-wise Fisher information, which preserves more unique knowledge of local data, to maintain local knowledge effectively. While for the shared parameter updates, we ensure that their L2 norm values approach the clipping bound through a bounded regularization term, effectively mitigating the impact of clipping operation. By adopting these strategies, we enhance the robustness to clipping while addressing the limitations of uniform regularization successfully.
2 we introduce an adaptive method that determines
In this work, we propose a novel personalized federated learning method under differential privacy named FedDPA via Dynamic Fisher Personalization (DFP) and Adaptive Constraint (AC). Our main contributions are summarized as follows:
We propose dynamic personalization strategy using layer-wise Fisher information values, enabling flexible personalization and reduced noise impact.
We present an adaptive method for enhancing clipping robustness by adjusting constraint strengths for different parameters, balancing client-specific features and global knowledge while minimizing clipping effectt.
2

---

We conduct extensive experiments under multiple tasks and settings. Accompanied with set of ablative studies, promising results validate the efficacy of our proposed methods and indispensability of each module.
2 Related Works
2.1 Personalized Federated Learning
Federated Learning is distributed machine learning approach that allows data to remain on local devices while training a global model, with the FedAvg algorithm [32 as its most representative example. Despite its revolutionary approach, the inherent non-Independent and Identically Distributed
[3 1.] poses significant challenges, thus leading to the advent of personalized federated learning (PFL). In PFL, portion of the model is personalized to learn client-specific knowledge, while the remaining parts aim to capture the global patterns across all clients. The mainstream PFL methods include LG-FedAvg (28). FedPer (2), PPSGD 13, and FULR [5]. The LG-FedAvg [8 and FULR [3] methods both leverage a fixed local parameter to extract representation of the local data, thereby achieving personalization, but their static partitioning approach lacks flexibility in adapting to diverse data characteristics. Besides, the FedPer [2] FedBABU (35] and PPSGD [3] algorithms achieve personalization by retaining specific layers locally, which can better cater to local data characteristics, but suffer from potential inflexibility in model personalization. Additionally, another work (36) introduces FedSim and FedAlt, which analyze different update paradigms local and global parameters, contributing to the understanding of the trade-offss between local data specificity and global data commonality. These approaches, while effective to a certain extent, still bear limitations in their inflexible of personalization methods and the potential performance degradation due the direct training with noisy global parameters under DP mechanism. We introduce an innovative approach to personalized federated learning. Our method, based on layer-wise Fisher information values, allows for dynamic personalization that surpasses the inflexibility of fixed parameter partitioning while mitigating the noise impact.
(non-III) nature local data [19 [2 40 44
17
2.2 Differential Privacy
Differential Privacy (DP) provides substantial privacy in federated learning 12
38 25 24
by minimizing the impact of individual data changes computation outputs. This privacy level is quantified via (s, 8) didfffeerntta privacy. Smaller e and 8 offer stronger privacy but introduce more noise into the learning algorithm, which could degrade performance.
In federated learning, user-level DP achieved by a two-step process intrinsic to the DP mechanism: adding noise and clipping local updates before server transmission. The noise scale is calibrated according to the function sensitivity. Clipping, which bounds the impact of local update, further reinforces privacy. Although these essential steps ensure substantial privacy, they present challenges, such as performance degradation and slower convergence, due to the unavoidable noise and clipping operation. While user-level DP ensures robust privacy, it introduces challenges such as performance degradation from added noise and slower convergence due to clipping. These challenges have been addressed by several methods, including LUS [8) BLUR (8). DP-FedSAM [38], and PPSGD (3). LUS and BLUR employ sparsification and unified regularization techniques to mitigate the effects of added noise and enhance model convergence. DP-FedSA.M 38 utilizes the SAM optimizer to bolster parameter robustness against noise, aiming to identify more stable convergence points. Besides, PPSGD 3 capitalizes on personalization improve performance while maintaining privacy. Despite their promising solutions, these methods don't I sufficiently address the issues caused by noise and clipping in differential privacy. They failed to find the essence of the problem of noise perturbation, all parameters will be affected by noise during training, leading to the degradation of model performance. Furthermore, their uniform handling of clipping constraints fails adapt to the dynamic nature of local updates, which further hinders model learning and convergence. To tackle these challenges, we propose novel method that leverages layer-wise Fisher information to dynamically personalize and protect parameters of high information content from noise impact. Simultaneously, we introduce an adaptive regularization strategy to enforce differential constraints on personalized and shared parameters, enhancing the model learning capacity and robustness to clipping.
3

---

2.3 Fisher Information Matrix
The Fisher Information Matrix (29 a pivotal concept in statistical estimation theory, encapsulates the information an unknown parameter possesses about random distribution. In the realm of deep learning, FIM has been employed to study the curvature of loss functions, guide optimization and evaluate the information content of parameters [3 4 20). For instance, the Kronecker- Factord Approximate Curvature (K-FAC) method B1 utilizes a Kronecker product approximation to the FIM for more efficient natural gradient computations. The layer-wise relevance propagation method 4 uses the diagonal of the FIM to quantify the importance of features, thereby enhancing model interpretability. The Elastic Weight Consolidation algorithm [20 7 employs FIM to guard important parameters during the learning of new tasks, mitigating catastrophic forgetting. All these methods utilize the diagonal of the FIM for approximation, reducing computational complexity and facilitating more efficient learning processes. Although these methodologies have made significant strides, their application in personalized federated learning with differential privacy constraints, remains largely unexplored. These methods are not directly applicable to personalized federated learning, as they don't account for the unique challenges posed by non-IID data distribution and privacy constraints. Our novel approach integrates layer-wise Fisher information into personalized federated learning, leveraging it to dynamically determine the personalization parameters, thus protecting informative ones from noise perturbation induced by differential privacy. This strategy effectively minimizes the impact of noise, enhancing model performance in personalization. Additionally, we introduce an adaptive method that dynamically adjusts L2 norm constraints based on their fisher information value magnitudes. This method enhances the robustness to clipping operations, offering more flexible solution to balance between model learning and privacy protection under the PFL with differential privacy.
3 Methodology
3.1 Preliminary
Personalized Federated Learning. Personalized Federated Learning (PFL) extends Federated Learning (FL) by addressing the non-llD nature of real-world data distributions. In PFL, parameter vector of client model decomposed into a local part, u, that caters to the individual data peculiarities, and global part, v, which is universally shared.
Assume we have M clients, Each client possesses a unique private dataset, denoted by Dm (Ii, y) where Nm signifies the size of the local dataset of client m. The parameter vector w E Rh of model of client i separated into local and global parts, forming w;; = (v, uz), by personalization techniques.
The optimization problem in PFL defined as follows:
1 M
min {f(e, u1:m) = Σ f(l,,))) MMMaom
1m1
where U1m represents (u1...., Um) E (Rda) M and the fi(2, u,) == ED, (v, u,, Dz) is the expected risk of client i, where i
Nm- PFL use iteration of two steps solve this problem:
Local Update: During the local update phase, each client i adopts the latest global parameter of while retaining its local u/ from previous round to initialize the model w/ u(), and then performs Ftocal iterations of local updates to obtain new pa-
rameters wt [+1 (v/+), 1 Subsequently, they compute the local update Aot+1 !+1 by calculating the difference between t/ and vf.
Collaborative Update: In the collaborative update phase, all clients send their local updates Auft1 to the server. The server then averages these parameters to update the parameters ,f+1 n¹ + 1/ AM Aut+1, which are then distributed back to all clients for the next round of local updates. In our methods, since the shared part of parameters is selected dynamically and cannot be previously determined, all clients send their whole updates Awly, to server.

---

User-Level Differential Privacy. In the context of Personalized Federated Learning (PFL), it is essential to employ differential privacy techinque to prevent potential privacy leakage. One strong privacy-preserving mechanism used in this setting is Differential Privacy (DP).
Differential Privacy (DP) [12 41 is formal privacy framework that provides theoretical guarantees against the identification of private data in a dataset. In the DP framework, is known as the privacy budget, which quantifies the privacy protection level, and ô is the probability of the privacy guarantee being violated. A randomized algorithm M satisfies (e, 6)D if for any pair of adjacent datasets D and D' differing by only one record, and for any subset of outputs in the range of M, it holds that:
(D) S) 5 et Pr(M)(D)) E S] +
(2)
This property implies that the probability distribution of the output of M changes only slightly when a single record in the input dataset modified, thus protecting individual privacy.
User-Level Differential Privacy (DP) is extensively used as an application of DP mechanism in the context of personalized federated scenarios. User-Level DP refers to the privacy guarantee for the participation information of a single client in the learning process. Based on the aforementioned theory, user-level DP makes the global model updates indistinguishable whether particular client participates in the learning or not. This is achieved by implementing two crucial operations: clipping and noise addition.
Clipping is done to control the maximum contribution of a single client to the global update, ensuring that the DP property holds even in the presence of outliers. This is crucial for maintaining the privacy of individuals in the learning process. Besides, noise addition (typically sampled from a zero-mean Gaussian distribution) is performed satisfy the randomness requirement of DP, making harder to infer specific information about any individual client. The combined effect of these two operations is expressed as:
Awc = Aw aur/mar(L) | AWC,N Auc + N1 C² C'o0,Coo//MA /IMM))
C'
where Aw represents the original local update from the client, [M,] represents the available client in round t, C' is the clipping threshold that controls the maximum contribution of client the global update, Awc is the local update after clipping, AwC, is the final local update after clipping and adding noise, / is the Gaussian noise added to ensure DP, and is a noise multiplier computed by privacy accountant and composition mechanism with respect to and 8.
3.2 Dynamic Personalization Strategy
Motivation of Fisher Information. The motivation to use Fisher Information stems from its ability to quantify the amount of information that parameters can provide. Parameters that are more informative can represent the knowledge more effectively, playing a crucial role in the model prediction. Therefore, under the same noise perturbation, if the informative parameters are affected, it will lead to more severe degradation of model performance. In typical personalization work, at the beginning of the local update phase, clients directly place the global parameters in fixed positions into the network for training. This inflexible personalization approach doesn't 't consider the impact of noise on different parameters, which results in the model being more susceptible to noise interference. Therefore, we need to avoid noise affecting informative parameters. In light of this, our approach considers the hierarchical structure of neural networks and introduces layer-wise Fisher information. We measure layer-wise of each parameter Fisher information, which reflects the sensitivity of the model to the parameter changes, to optimize the model by preventing them from noise.
Construction of Dynamic Fisher Personalization. As discussed in 1 the current DP-PFI. methods struggle with inflexible personalization due to the fixed portion of the model used for personalization. To address this issue, we introduce a dynamic personalization strategy that leverages the Fisher Information to enhance the adaptability of the model the data distribution of individual clients as follows. We start at the beginning of local update phase of client at global epoch t, with its private dataset D, and parameters w. = (v--1, : u,4 remain from last epoch. We can get the empirical fisher value vector F; E ddd where d represents the num of parameters of w as good approximation of the diagonal of true fisher value for each parameter indexed by j in w 1-1 as:
@ alog L(ws, D.)
F(w))) =
Owiy
5

---

where the log L(w,, D,) represents log-likeliine likelihood function of wc,, Then, the layer-wise fisher value Fg for layer & by a layer-wise normalization for each parameters indexed by j in layer & as
Fa. - min (F. [ax Fin{Fk,]
FE.
(5)
Given the layer-wise Fisher values for each parameter, then generate binary masks, M1 and M2, for each parameter to conduct our dynamic parameter selection. each mask, a parameter entry set to if its corresponding Fisher value is greater than or equal threshold т, otherwise, is set to O. These masks are defined as follows for each parameter j:
1, if Fy 2 r and Malil S0,
if Pf 2T 11 otherwise
Mi
(6)
0, otherwise
where Mj and M2 are the entries in the first and second masks, respectively.
We then perform Hadamard product (element wist multiplication) between these masks and the parameters to select the appropriate parameters for personalization. Specifically, parameters with a larger Fisher value, which correspond to 1 I's in M1, are retained from the previous epoch, while the remaining parameters, which correspond to I's in M2, are replaced by the global parameters. This operation be expressed follows:
/ Mi O : + M2 O'
where O denotes the Hadamard product and w the global parameters downloaded from server. This approach ensures that more informative parameters (with larger Fisher values) are retained for personalized learning, while less informative ones are updated with the global parameters, mitigating the influence of noise.
Comparison with Analogous Methods. PPSGD employs a fixed layer personalization approach with an additive model to achieve personalization under differential privacy (DP). Meanwhile, the LUS mechanism { utilizes a mask during their sparsification process after local training. Although these methods have their merits, they are fundamentally different from our proposed strategy. In our work, we advocate for a dynamic personalization approach using mask, but crucially, this is applied prior to the training process. Our aim is to provide a more flexible and adaptive personalization strategy that can better cope with changes in the data distribution, noise level, and other dynamic factors affecting the model performance. Furthermore, we focus preserving the most informative parameters, those that significantly contribute to the model performance, while allowing less informative ones to be updated with global parameters, thereby reducing the impact of noise. This dynamic, pre-training personalization strategy provides a unique advantage over the static, post- post-training personalization of the aforementioned methods.
3.3 Adaptive Constraint
Motivation for Adaptive Constraint. The challenge of gradient distortion and convergence diffi- culties in DP-PFL arises from the clipping operation applied the L2 norm of each local update. Traditional methods 8 23 often apply a uniform regularization strength to all parameters, neglecting the inherent differences among parameter magnitudes. To address this issue, we propose an adaptive approach that determines different constraint strengths for different parameters, considering the inherent magnitude of each parameter. This aims to maintain the learning capacity of the model while also enhancing its robustness against clipping.
Construction of Adaptive Constraint. As discussed in the problem of gradient distortion and convergence difficulties in DP-PFL arises from the clipping operation applied to the L2 norm of each local update, our method is as follows. At the start of the local update phase, each client i global epoch has parameters w 1-1 (o'-1,ui -1) These parameters parameters o/ and personalized parameters u. identified by the previously introduced dynamic personalization strategy. We propose two different constraints for the local updates of these two sets of parameters.
divided into two sets: shared
For the personalized parameters, which contain unique knowledge from the local data, our aim is to retain this local knowledge as much possible. We therefore introduce regularization term to loss

---

function that constrains the L2 norm of the local update of the personalized parameters. The local update of personalized parameters u² is expressed as:
i n - v . Di),
(8)
where 1) is the learning rate, and V 1 is the gradient of the first loss function. The first loss function, L1, is calculated as the sum of the cross-entropy loss and the 1.2 norm of the difference between the current and the previous personalized parameters:
1 n1
A1 Iluu u 12.
L1
My log(ū;) + (1 - yj) log(1 ú5)l
(9)
2
where is the number of instances in the dataset D.. y, is the true label of instance j. A1 is hyper-parameter and ūy is the predicted label of instance j.
For the shared parameters, bound. By adopting such a strategy, we anticipate reducing the influence that the clipping operation exerts the local update. We therefore introduce bounded regularization term the loss function of the shared parameters. The local update of shared parameters of is expressed as:
strive to ensure that their L.2 norm values closely approach the clipping
of = =
C2 ,'', D:),
(10)
where VLe is the gradient of the second loss function. The second loss function, L2 is caleulated as the sum of the cross-entropy loss and the absolute difference between the L2 norm of the difference between the current and previous shared parameters and the clipping bound:
A2 I|||us of1l2 -Cl12
L2 =
1y log(ū;) + (1 - y/) log(1 - -05]] +
(11)
2
Here, n is the number of instances in the dataset Di. My is the true label of instance j. O3 is the predicted label of instance j. X2 also hyper-parameter and C the clipping bound.
By applying these adaptive constraints, our method effectively mitigates the impact of the clipping operation and boosts the performance of FedDPA. The two loss functions, L1 and L2, cater to the unique properties of personalized and shared parameters, respectively. L helps retain as much local knowledge possible, overcoming the slow convergence issue. Additionally, La ensures the shared parameters updates align with the clipping bound, thereby improving the model robustness against clipping. In both cases, the regularization strengths A₁ and X2 are determined adaptively, taking into account the inherent magnitude of each parameter. This novel consideration distinguishes our method from traditional approaches and contributes its superior performance.
3.4 Discussion and Limitation
Our analysis extra computation cost is as follows: Assuming a total of Eg Global Epochs, M clients, and P parameters for each model, during each client update process, the client undergoes S steps of parameter updates in training. In each round, our method introduces computation costs as follows: 1) Computing the Fisher information value, 2) Construction of the mask and 3) Calculating the initial parameters using masks. In these steps, as each step is parameter-speat brings additional computational cost of OPP). Regarding that we have to carry out these operations in each global epoch, we introduce a computational cost of OEE, M P) in total.
our method
Despite our extra computational cost, is noteworthy that our approach has relatively small overhead compared to the overall training process. This is because, the model training still has a considerable computational cost proportional to P, and all the training process would introduce OEE, . P.S). S). Generally, in a training process, the value of is in the tens or hundreds, so the computation cost introduced by our method will be an order of magnitude more less than the training expense. Extensive experiments have verified the consistent improvement over the counterparts.
Besides the core functionality of our model, it also shows potential for applications in other fields, such as feature selection in high-dimensional datasets, improving model robustness by focusing on critical parameters, and enhancing privacy protection by selective parameter sharing. Regarding the limitation of model homogeneity, our method assumes all clients share the same model structure, which may not always be the case in real-world scenarios. Heterogeneity in model structures across
7

---

Algorithm 1: The Proposed Method: FedDPA
Input:
Global epochs Eg. local epochs Et, participants number in the tth epoch m'. private data of the 2th client D; = (X,; Yi). global model parameters and client local parameters wi, \1, X2, learning rate
Local Update:
for 8 1,2, 2,,, m² do
Receive w'11 from Server
F; < (D;, : 1 by Eq. and Eq. (5)
M1 and M2. (Fi, by Eq.
w (u' ) t- Mi ①
+ M2. c w w using Eq.
for e = 1,2, F4 do
uf 4- "; - VV 1 (0'-1, at by L1 from
v-- C2 (0- ut) by Le from II
end
Auf (vi, u.) (0'-1 u 1
Auf + /wt/ max |
+ +N(00 C2 /2022/mmm) to ensure DP by Eq. 3
end
Server Execute:
Awi f Local Update(us w
for = 1, 1,2, ..., Ey do
w' <- w
21=1 Awf
for := 1,2, ..., m do
Send w² to th participant
end
end
clients can potentially limit the utility of our dynamic personalization strategy and adaptive constraints, differences in model could lead to incompatibilities, which is an area where further research needed to improve the versatility of our method. It should be noted that this limitation is not unique to our approach but is also shared by most of the aforementioned methods S
Experiments
We evaluate the performance of the proposed method and comparison methods through comprehensive experiments on two distinct datasets under various conditions. Specifically, we test the accuracy the FEMNIST and SVHN dataset across different privacy budgets denoted by e, and for the CIFAR-10, we adjust the a parameter to simulate varying degrees of non-IID distribution and assess accuracy accordingly. This wide-ranging experimental setup reflects practical scenarios and ensures a thorough assessment of our method's robustness and adaptability.
Data and Model. Our method is evaluated on classification tasks, FEMNIST (6, CIFAR-10 21 and SVHN [34, embodying real-world non-IID and grivacy-constraid scenarios. The FEMNIST is 62-class version of MNIST, CIFAR-10 10 is a widely-used image classification dataset, and the SVHN dataset it a digits classification dataset which is collected from street view. We use a simple CNN model with 2 convolution layers and 2 fully connected layers for FEMNIST and deeper model with 3 convolution layers and fully connected layers for CIFAR-10.
Comparison Methods. We compare the performance of our proposed method with several state- of-the-art methods in differential PPSGD (3, DP-FedAvs with LUS and BLUR [8], and DP-FedSAM 38).
federated learning, including DP-FedAvg [22]
Evaluation Metrics. The primary metric for evaluation the average accuracy, calculated uniquely for each client. Each client has their own personalized model trained on their individual dataset, and accuracy measured on these respective datasets. The average of these individual accuracies across all clients is then calculated to yield the final metric. For the FEMNIST and SVHN dataset, we evaluate this average accuracy under varying privacy budgets, underlining the advantage of method in mitigating noise impact. For the CIFAR-10 dataset, evaluate average accuracy

---

under different degrees of non-IID data partition, demonstrating the effectiveness of our dynamic personalization strategy in handling non-IlD data.
Implementation Details. For all dataset FEMNIST, CIFAR-10 10 and SVHN, we set the learning rate le-3 and optimize hyperparameters т, Å1, and A2 through grid search in {0.05, 0.1, 0.3, 0.5). We use the Rényi Differential Privacy (RDP) algorithm provided by Opacus as our privacy accountant. The value of 6, in (e, 8) -differennial privacy, is set to the reciprocal of the number of clients (i.e., 1/M). We use global epochs of 30 and 40, local epochs of and 4, and batch sizes of 16 and 64 for FEMNIST and CIFAR- 10, respectively. For CIFAR-I0, we partition the dataset into 10 subsets via Dirichlet distribution and use a fixed noise multiplier to ensure differential privacy. For FEMNIST, we selected the top 50 clients from the FEMNIST [6] division for training. All experiments were implemented in Python with PyTorch on an NVIDIA 3090 GPU.
Table 1: Comparison of Model Performance under Different Conditions. Refer Sec. 4.1
FEMNIST
SVHN
Methods
=
<=
8=8
<=16
DP RedAyg
72,80
73.26
54.76
57.65
BLUR LUS 8
72.80
74.23
74.58
75.16
53.76
57.52
56.90
58.57
PPSGD S
68.20
69.60
71.03
71.93
55,89
56.15
56.74
59.37
AAAM (38
73.13
73.94
74.54
74.66
53.04
52.84
54.03
$6.08
74.461133
77.27
77,42 13.84
76.99 1.83
58.78
62.63
63,661676
64.57
Table 2: Performance Comparison on CIFAR-10, Refer Sec. 4.1
Methods
a
a
a
100
IID
DP-FedAvg 12
42.61
58.89
60.03
60,97
BLUR+LUS g
48.97
58.70
60.50
61.12
PPSGD B
44.04
57.31
60.20
60.02
DP-FedSAM (38
45.07
56.15
58.87
60.77
FedDPA(Ours)
49.75 10.78
59.48 10.59
60.68 00188
61.55 10.43
4.1 Comparison with State-Of-the-Arl Methods
We provide comparison results with SOTA methods on two image classification tasks 21
6
Analysis Under Different Privacy Budget. As Tab I shows, method maintains high accuracy across various privacy budgets on the FEMNIST dataset, underlining its robustness and superior privacy-utility trade-off. The convergence plots Fig Ta and Fig [b] support our method's efficiency, demonstrating faster convergence and more effective loss minimization.
Analysis Under Different Degrees of Non-IID Data. Our method shows strong performance across varying non-III data partitions on the CIFAR-10 10 dataset, as seen on the right side of Tab 2 effectively handles non-IlD data challenges via our flexible personalization method.
26 3R-ei
15
10
(a) Accuracy convergence curve
(b) Loss convergence curve
Figure Comparison of Model Performance and Optimization Objective Value. See details 4.1

---

4.2 Diagnostic Experiments
To highlight the contribution of each component in our method towards the overall performance, we conduct a series of ablation experiments. Our proposed method comprises two integral components.
Dynamic Fisher Personalization (DFP). The DFP, described in Sec. 3.2 contributes to the improvement of accuracy across various privacy budgets. When DFP is employed (second row in the table), there is noticeable increase in accuracy for all privacy budgets compared to the base model (first row), validating the effectiveness of DFP in enhancing model performance.
Adaptive Constraint (AC). The AC strategy, as introduced in Sec. 3.3 is dependent DFP for its operation. Therefore, there no experiment featuring only the AC without the DFP. However, when both DFP and AC are incorporated (third row in the table), the model's accuracy further improves across all privacy budgets. This demonstrates that the AC strategy enhances the utility of DFP, contributing to overall superior Saivacy-utiliiy trade-off.
Table 3: Ablation Study with Different Privacy Budgets. Refer Sec. 4.2
Components c= 2
Accuracy on clients dataset
DFP
AC
8
16
71.92
72.80
73.26 75.02
72.18
74.56
74.38
75.33
14.26
(
77.42 12,93
76.99 1.66
The results in Table Blearlyy illustrate that both DFP and AC contribute significantly to the perfor- mance of the model under various privacy budgets. The combined use of both components provides the best results, reinforcing the effectiveness of our proposed method.
4.3 Hyperparameter Study
We vary the hyperparameters including À and T in our proposed methods to demonstrate how out model retains or change personalized portions DFP and the sensitivity of constraint strength in AC
Table Ablation Study on т. Refer Sec.
4.3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Proportion
100%
11.1%
4.5%
1.45%
0.97%
0.46%
0.15%
0.1%
0.1%
0,05%
0%
Accuracy
39.72
47.97
49.28
48.89
48.35
48.03
48.28
48.14
48.35
47.74
48.29
Table 5: Ablation Study on A Refer Sec 4.3
Table 6: Ablation Study on A2. Refer Sec, 4.3
A
0.05
0.1
0.3
0.5
A2
0
0.05
0.1
0.3
0.5
Accuracy 60.82
62.64
59.66
59.28
59.57
Accuracy
58.34
59.76
61.01
47.61
45.66
For T, observe that our proposed method is relatively stable, verifying the robustness of our proposed solution. For X, we achieve the best performance when x is around 0.05 and Xg is around 0.1. We apply this across multiple different datasets, and it results in consistent good performance.
5 Conclusion
In this paper, we delve into the challenges of inflexible personalization and convergence difficulties under differential privacy in personalized federated learning. We introduce FedDPA with two innovative components, Dynamic Fisher Personalization (DFP) and Adaptive Constraint (AC), that utilize fayer-wise Fisher information to dynamically personalize and adaptively constrain parameters, effectively addressing the aforementioned issues. The efficacy of our proposed methods has been thoroughly validated against many popular counterparts across various classification tasks. We are hopeful that this work will pave the way for future research on personalized federated learning with differential privacy.
Acknowledgment. This work is partially supported by National Natural Science Foundation of China under Grant (62176188), the Key Research and Development Program of Hubei Province (2021BAA) 187, 2022BCA009), Zhejiang lab 20222222FO,,,),,, CCa-eu Populus Grove Fund CCC-HHwwwiTC2222200) and the Special Fund of Hubei Luojia Laboratory (220100015).
10

---

References
[1] Shun-Ichi Amari. Natural gradient works efficiently in learning. NC, pages 251-276, 1998.
[2] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint /9/2.008// 2019.
[3] Alberto Bietti, Chen-Yre Yu Wei, Miroslav Dudik, John Langford, and Steven Wu, Personalization improves privacy-accuracy tradcoff's in federated learning. In ICML, pages 1945-1962, 2022.
[4] Alexander Binder, Grégoire Montavon, Sehastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN, pages 63-71, 2016.
[5] Duc Bui, Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, and Kang G Shin. Federated representation learning. arXiv preprint 1909: 1900:12559.. 2019.
[6] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li. Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benehmark for federated settings. arXiv preprint arXiv: 1012v01812.01097, 2018.
[7] Dengsheng Chen, Jie Hu, Vince Junkai Tan, Xiaoming Wei, and Enhua Wu. Elastic aggregation for federated optimization. In CVPR, pages 12187-12197, 2023.
[8] Anda Cheng, Peisong Wang, Xi Sheryl Zhang, and Jian Cheng. Differentially private federated learning with local regularization and sparsification. In CVPR, pages 10122-10131, 2022.
[9] Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In CVPR, pages 10072-10081, 2022.
[10] Xiuwen Fang. Mang Ye, and Xiyuan Yang. Robust heterogeneous federated learning under data corruption. In ICCV, pages 5020-5030, 2023.
[11] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM CCS, pages 1322 1233, 2015.
[12] Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv: 2017.
[13] Wenke Huang, Guancheng Wan, Mang Ye, and Bo Du. Federated graph semantic and structural learning. In IJCAI, pages 139-143, 2023.
[14] Wenke Huang, Mang Ye, and Bo Du Learn from others and be yourself in heterogeneous federated learning. In CVPR, pages 10143-10153, 2022.
[15] Wenke Huang, Mang Ye, Bo Du, and Xiang Gao. Few-shot model agnostic federated learning. In ACM MM, pages 7309-7316, 2022.
[16] Wenke Huang. Mang Ye, Zekun Shi, and Bo Du. Generalizable heterogeneous federated cross-correlation and instance similarity learning. TPAMI, 2023.
[17] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In CVPR, pages 16312 16322,, 2023.
[18] Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjcev Arora. Evaluating gradient inversion attacks and defenses in federated learning. NIPS, pages 7232-7241, 2021.
[19] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open prohlems in federated learning. FTMI., pages 1-210, 2021.
[20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka trabska-Barwinsske et al. Overcoming catastrophic forgetting in neural networks. NAC, pages 3521-3526 3526, 2017.
[21] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis. Department of Computer Science, University of Toronto, 2009.
[22] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE SPM, pages 50 60, 2020.

---

[23] Tian Li. Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. MLSys, pages 429-450, 2020.
[24] Tian Li, Manzil Zaheer, Ken Liu, Sashank J Reddi, Hugh Brendan McMahan, and Virginia Smith. Differentially private adaptive optimization with delayed preconditioners. In ICLR. 2022.
[25] Tian Li, Manzil Zaheer, Sashank Reddi, and Virginia Smith. Private adaptive optimization with side information. In ICML, pages 13086-13105, 2022.
[26] Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In ICML. pages 19767-19788, 2023.
[27] Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In ICCV, pages 5319-5329, 2023.
[28] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.00 00.552,,.. 2020.
[29] Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmakers. tutorial on fisher information. JMP, pages 40-55, 2017.
[30] Xiaosong Ma, Jie Zhang. Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In CVPR, pages 10092-10101, 2022.
[31] James Martens and Roger Grosse. Optimizing neural networks with kronceker-factored approximate curvature. In ICML, pages 2408-24113 2015.
[32] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficeenn learning of deep networks from decentralized data, In AISTATS, pages 1273-1282, 2017.
[33] Luca Melis, Congzheng Song. Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In IEEE SP, pages 691-706, 2019.
[34] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
[35] JAE HOON OH, Sangmook Kim, and Seyoung Yun. Fedbabu: Toward enhanced representation for federated image classification. In ICLR, 2022.
[36] Krishna Pillutla, Kshitiz Malik, Abdel-Rahmann Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Foderated learning with partial model personalization. In /CML, pages 17716-17758, 2022.
[37] Jiawei Shao, Yuchang Sun, Songze Li, and Jun Zhang. Dres-ft: Dropout-resilient secure federated learning for non-iid clients via secret data sharing. NIPS, pages 10533-10545, 2022.
[38] Yifan Shi, Yingqi Liu, Kang Wei, Li Shen, Xueqian Wang, and Dacheng Tao. Make landscape flatter in diffcrentially private federated learning. In CVPR, pages 24552 225622, 2023.
[39] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE TNNLS, 2022.
[40] Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, and Chengqi Zhang. Federated learning on non-iid graphs via structural knowledge sharing. In AAAI, 2023.
[41] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Hang Su, Bo Zhang, and H Vincent Poor. User-level privacy- preserving federated lcarning; Analysis and performance optimization. JEEE TMC, pages 3388-3401, 2021.
[42] Yuxin Wen, Jonas A Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for user data in large-batch federated leamning via gradient magnification. In JCMI., pages 23668-23688 2022.
[43] Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogencous federated learning: State-of the-art and research challenges. ACM CS, 2023.
[44] Y Zhao, M Li, L Lai, N Suda, D Civin, and V Chandra. Federated learning with non-iid data, arXiv preprint Xiv: 806.00000000 00582, 2018.
12


Reviews:
[{"review_id": "odJ3eV17lE", "reviewer": "Reviewer_dw3B", "summary": "This paper proposes a novel method for Dynamic Personalized Federated Learning with Adaptive Differential Privacy. The method addresses the challenges of inflexible personalization and convergence difficulties in current DP-PFL methods by leveraging layer-wise Fisher information and introducing an adaptive approach. The proposed method retains informative local parameters and prevents noise perturbation, resulting in better personalization performance and robustness against clipping. The experiments on CIFAR-10 and FEMNIST datasets demonstrate the effectiveness of the proposed method in achieving better personalization performance and robustness against clipping under differential privacy personalized federated learning.", "questions": "- Are the values of \\tau and \\lambda dataset-dependent, or are they independent of the specific datasets used in the experiments?\n- In scenarios where all local users have different datasets, how does this heterogeneity affect the performance and robustness of the federated learning approach proposed in the paper?\n- In Table 2, why does the combination of DFP and AC yield the highest values for \\epsilon = 8, whereas the highest performance of DFP and AC alone is achieved for \\epsilon = 16? What factors contribute to this discrepancy in performance for different privacy budgets?", "limitations": "Limitations are addressed. Explicit mention of ethical aspects are lacking, however they can be implicitly included in federated learning.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written, with clear organization.\n- The research questions are well-defined and relevant, backed by a comprehensive literature review.\n- The paper presents compelling results, demonstrating the effectiveness of the proposed approach \ncompared to state-of-the-art techniques in small-scale experiments.", "weaknesses": "- The way in which adaptive differential privacy is defined lacks clear motivation and appears to be a composition \nof not well-motivated components. The reasons for introducing losses in equations (9) and (8), \nas well as how these losses are combined to obtain the adaptive constraint in section 3.2.2, are not clearly elucidated.\n- The paper would benefit from running experiments in a larger context. While the results in small size benchmarks are promising, their applicability to real-world scenarios needs to be validated on more extensive datasets to observe the model's effectiveness at scale.\n- An ablation study on hyperparameters \\lambda and \\tau is necessary. Understanding how the model adapts or resists new parameter updates and how the two masks are computed based on the Fisher Information Matrix requires thorough investigation. The absence of such a study leaves questions about the robustness and sensitivity of the proposed approach unanswered.\n \nminors:\n- In Eq. 4, consider increasing the size of the brackets for better readability.\n- In Eq. 7, there are different symbols used for the global and local parameters of the network, u and v appear to be used inconsistently. This somewhat reduces clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dw3B: \n\nThanks for your insightful comments and questions. We sincerely hope our detailed response allows the reviewer to update the score.\n\n**Weaknesses:**\n\n**W1: Motivation of adaptive differential privacy and equations 8 and 9, Details about Section 3.2.2**\n\n**A1:** Current Differential Privacy operations require clipping the gradients in their magnitude. However, existing methods only focus on minimizing empirical risk and cannot handle the varying and inconsistent gradient norm, as we elaborate in our main manuscript (Line 51 in Page 2). As a result, they exhibit poor robustness to norm clipping, and the distortion caused by the clipping of the gradient can affect the model's convergence.  To address this issue, we develop an adaptive solution under the personalized framework.\n\nConsidering that the parameter 'u' represents vital knowledge learned from the client dataset and reflects its specific characteristics, this knowledge is indispensable in personalized federated learning. To retain this knowledge, consequently, we introduce a loss function that restricts the update size of the informative local parameter 'u' by adding a regularization term and also helps in mitigating the impact of clipping and achieving adaptation to differential privacy, as shown in equation (9). Equation (8) represents the process of updating the parameter 'u' using equation (9). \n\nFinally, the adaptive constraint is obtained by updating the 'u' parts of the parameters using equation (8) and updating the 'v' parts of the parameters using equation (11), as detailed in 8-Algorithm 1. Extensive experiments have verified this simple yet effective operation can consistently improve the performance under various settings.\n\n**W2: Additional real-world scenarios experiments**\n\n**A2:** We have expanded our experiments to include the Street View House Numbers (SVHN) dataset, which originates from real-world street view scenarios.\n\nTable: Comparison of model performance under different epsilons\n\n| Methods|ε = 2|ε = 4|ε = 6|ε = 8|\n|-|-|-|-|-|\n| DP-FedAvg|53.91|54.76|56.33|57.65|\n| BLUR+LUS|53.76|57.52|56.90|58.57|\n| PPSGD |55.89|56.15|56.74|59.37|\n| DP-FedSAM|53.04|52.84|54.03|56.08|\n| Ours |58.78|62.63|63.66|64.57|\n\nExperiments demonstrate that our method still surpasses other state-of-the-art approaches in a larger and more complex dataset from real-world scenarios, showcasing the effectiveness of our method in different scenarios.\nWe will include this experiment in the final version if accepted.\n\n**W3: Ablation study on hyperparameters $\\lambda$ and $\\tau$**\n\n**A3:** We have conducted the ablation experiments, and the results are presented in the following table:\n\nTable: Ablation study on $\\tau$\n\n|$\\tau$|0|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Proportion|100%|11.1%|4.5%|1.45%|0.97%|0.46%|0.15%|0.1%|0.1%|0.05%|0%|\n|Accuracy|39.72|47.97|49.28|48.89|48.35|48.03|48.28|48.14|48.35|47.74|48.29|\n\nFor $\\tau$, we observe that our proposed method is relatively stable, verifying the robustness of our proposed solution.\n\nTable: Ablation study on $\\lambda_1$\n\n|$\\lambda_1$|0|0.05|0.1|0.3|0.5|\n|-|-|-|-|-|-|\n|Accuracy|60.82|62.64|59.66|59.28|59.57|\n\nTable: Ablation study on $\\lambda_2$\n\n|$\\lambda_2$|0|0.05|0.1|0.3|0.5|\n|-|-|-|-|-|-|\n|Accuracy|58.34|59.76|61.01|47.61|45.66|\n\nFor $\\lambda$, we achieve the best performance when $\\lambda1$ is around 0.05 and $\\lambda2$ is around 0.1. We apply this selection across multiple different datasets, and it results in a consistent good performance. In the future, we will explore a more flexible solution for the selection of $\\lambda$. We will incorporate these experiments with above analysis into the final version if accepted\n\n**W4: Revise Eq. 4, Eq. 7**\n\n**A4**: Thanks for pointing out these issues. We will revise Eq. 4 and Eq. 7 to ensure clarity and consistency. \n\n**Questions:**\n\n**Q1: Whether the $\\tau$ and $\\lambda$ depends on the dataset**\n\n**A1:** Please refer to the ablation study table of $\\tau$, our proposed method is relatively stable when the value of $\\tau$ changes. Based on what our experiments have shown, generally speaking, a value of $\\tau$ around 0.2 can achieve good personalized effects, and continuing to expand its value does not cause serious performance degradation, verifying the robustness of our proposed solution.\n\nFor $\\lambda$, we find that choosing $\\lambda1$ to be around 0.05 and $\\lambda2$ to be around 0.1 across multiple datasets results in consistent good performance. This reflects the consistency of the hyperparameters across different datasets, demonstrating the effectiveness of our method.\n\n**Q2: How heterogeneity affects performance and robustness**\n\n**A2:** As data heterogeneity rises, the local data distributions become more biased towards the global distribution, resulting in more chaotic local updates. Consequently, the clipped local updates tend to align less with the original local updates, adding complexity to convergence.\n\nIn the experiments, we specifically explore the performance under varying degrees of non-IIDness, as presented in Table 1, as the degree of heterogeneity escalates ($\\alpha$ changing from 1 to 100, with IID representing infinite $\\alpha$), the model performance experiences degradation. This observation underscores that heightened heterogeneity can compromise both robustness and performance. However, despite the degradation in performance, compared with other baselines, the degradation on our method is minor, demonstrating the robustness of our method.\n\n**Q3: Reason why not having the highest score when $\\epsilon = 16$**\n\n**A3:** We believe that the observed variation in performance is due to statistical fluctuations that can occasionally occur in experiments. In order to address this, we repeated the experiments and obtained accuracy scores of 77.59%, 77.40%, and 78.31%, with an average of 77.76%. This average score represents the highest accuracy across all epsilon settings.\n"}, {"review_id": "n2A5DA5wQa", "reviewer": "Reviewer_imA4", "summary": "This paper proposes an adaptive method for Personalized Federated Learning with Differential Privacy (DP-PFL), which aims to address the inflexible personalization and convergence difficulties that DP-PFL suffers from. Firstly, layer-wise Fisher information is used to measure the information content of local parameters. Parameters with high Fisher values are retained locally and prevented from noise perturbation. Secondly, an adaptive approach by applying differential constraint strategies to personalized parameters and shared parameters identified in the previous step is proposed for better convergence. Experimental results on CIFAR-10 and FEMNIST datasets demonstrate the effectiveness of the approach in achieving better personalization performance and robustness against clipping.", "questions": "Except for the weak points, I have the following questions:\n\nQ1. Can you please provide more details on the mechanism of the approach, particularly how you modify the Fisher Information Matrix in statistical estimation theory to adapt to DP-PFL, and how it looks originally?\n\nQ2. In Section 3.1, the definition of Personalized Federated Learning is not described well, and the variables are not precisely expressed in the process of Local Update and Collaborative Update. Please review this section carefully and make the necessary corrections.\n\nQ3. The setting of epsilon is a little large, which could be impractical for real-world applications. Please provide more justifications.\n", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. A dynamic personalization strategy using layer-wise Fisher information values is proposed, enabling flexible personalization and reducing the impact of noise.\n\nS2. This paper presents an adaptive method for enhancing clipping robustness by adjusting constraint strengths for different parameters.\n\nS3. Extensive experiments under multiple tasks and settings, including an ablation study, are conducted to demonstrate the effectiveness and robustness of the approach.\n", "weaknesses": "W1. The paper lacks technical novelty. For example, this work mainly borrowed the idea from statistical estimation theory as a measurement to represent the amount of information content, and the other part of the proposed solution is straightforward.\n\nW2. The paper is sometimes self-conflicting. For example, in the introduction at Line 62, it is stated that the second-order derivative of the log-likelihood function is computed to get the Fisher value. However, in Equation 4, a square of the first-order derivative is computed instead, without adequate explanation.\n\nW3. The experiments are insufficient, and more experiments are suggested to demonstrate how the proportion of shared parameters changes with different settings of hyperparameters. Moreover, it is critical to show how the threshold affects the accuracy of the method.\n\nW4. There are some presentation issues and typos. For example, in Line 356, “…via our flexible personzalzation method” should be “…via our flexible personalization method”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer imA4:\n\nThanks for raising thoughtful questions and concerns. We sincerely hope this point-to-point response allows the reviewer to update the score.\n\n**Weaknesses:**\n\n**W1: Lacks technical novelty**\n\n**A1:** We did not merely adopt statistical estimation theory into our work. Rather, we approached it with unique insights and reasoning, tailoring it specifically for our application. Conscious alterations and improvements were implemented to ensure its effectiveness in our scenario. \n\nFor Personalization in FL, our novelty does not lie in the utilization of Fisher information. Existing personalization methods often depend on FIXED techniques by manually selecting personalized parts, which are inflexible under different data distributions. We provide a new angle to evaluate the information importance of parameters in the personalization process, which can adaptively select personalization parameters with a slight additional cost. Rather than simply repeating the Fisher calculation in traditional information field, we aligned it with the traits in deep neural networks, introducing a layer-wise Fisher calculation. This provides a more accurate selection of personalized parameters, finely tuned to the specific requirements of each layer. We believe this simple yet effective approach provides insights for the community.\n\nIn addition, we also design an adaptive Differential Privacy strategy, our novelty is reflected in designing different constraint strategies for parameters to make gradient adapt to the clipping operations required by differential privacy. Previous gradient computation methods are unsuitable for clipping operations, leading to difficulties in model convergence. We propose a new strategy that enables the gradient to adapt to the clipping operation, contributing to better convergence. Considering that Reviewer dw3B, EfaD, aCpX, and UFEt have all agreed that our method is a novel approach and our proposed method outperforms other state-of-the-art methods in experiments, we believe our research findings are worth sharing with the community.\n\n**W2: Equation 4 conflicts with the Introduction**\n\n**A2:** We are sorry for the misleading presentation. The fisher information value is actually defined as the square of the first-order derivative of the log-likelihood function as Eq.4 demonstrates.\nTo prevent misunderstandings, we will correct the relevant part of the Introduction in the final version if accepted.\n\n\n**W3: Additional experiments on $\\tau$**\n\n**A3:** We have extended our experiments as follows:\n\nTable: Ablation study on $\\tau$\n|$\\tau$|0|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Proportion|100%|11.1%|4.5%|1.45%|0.97%|0.46%|0.15%|0.1%|0.1%|0.05%|0%|\n|Accuracy|39.72|47.97|49.28|48.89|48.35|48.03|48.28|48.14|48.35|47.74|48.29|\n\nProportion is the proportion of personalization parameters. Experimental results show that the proportion of personalized parameters decreases as \\tau increases, and our method can achieve the best results when the value of \\tau is around 0.2. This demonstrates the effectiveness and indispensability of our personalization approach.\n\nGenerally, our proposed method is relatively robust against different $\\tau$s, verifying the effectiveness and indispensability of our personalization approach. We will incorporate these experiments into the final version if accepted.\n\n**W4: Presentation issues and typos**\n\n**A4:** We have corrected the error you noted on Line 356 and other presentation issues.\n\n**Questions:**\n\n**Q1: Rationale of mechanism and Fisher Information Matrix**\n\n**A1:** The original form of the Fisher Information Matrix is defined as:\n$$\nF=\\frac{1}{N} \\sum_{i=1}^N \\nabla \\log p(x_i | \\theta) \\nabla \\log p(x_i | \\theta)^T, \n$$\nConsidering the values outside the main diagonal of the matrix reflect the mixed information content between different parameters, which is not our concern, and the elements on the main diagonal uniquely correspond to the parameters, reflecting the sensitivity of the log-likelihood function of each parameter, we directly compute the values in the main diagonal to represent the information content corresponding to each parameter, as shown in Eq.4.\nIn the process of adapting to our algorithm, we did not simply apply this computation method to neural networks. Instead, considering traits in the neural network that distribution of Fisher Values of parameters in different layers is naturally different, we introduced the layer-wise Fisher Value, leading to a more precise personalization.\n\n**Q2: Revise Section 3.1 for clarity and precision**\n\n**A2:** We will make the following changes to address your comments:\n- We will improve the description of each client private dataset by explicitly stating the range of i as $i=1,..., N_m$. \n- We will redefine the \"Local Update\" part, it will delineate that each client adopts the latest global parameter $v^t$ while maintaining their local parameters $u_i^t$ at the beginning of local update round. \n- In the \"Collaborative Update\" part, we will stress that all clients transmit their global parameter updates $\\Delta v^{t+1}$ to the server.\n\nWe will incorporate these changes into the final version if accepted.\n\n**Q3: Justifications for a little large ε value**\n\n**A3:** Our initial selection of epsilon in (2, 4, 8, 16) was intentional, by selecting these exponentially growing values, we aim to showcase how well our method works across a broad range of ε. We also conduct additional experiments on SVHN dataset with epsilon in (2, 4, 6, 8). The results are in the table below:\n\n|Methods|ε = 2|ε = 4|ε = 6|ε = 8|\n|-|-|-|-|-|\n|DP-FedAvg|53.91|54.76|56.33|57.65|\n|BLUR+LUS|53.76|57.52|56.90|58.57|\n|PPSGD|55.89|56.15|56.74|59.37|\n|DP-FedSAM|53.04|52.84|54.03|56.08|\n|Ours|58.78|62.63|63.66|64.57|\n\nExperiments demonstrate that even under the setting of smaller epsilon, our method still surpasses other state-of-the-art approaches, showcasing the effectiveness of our method."}, {"review_id": "lPqT1ooKI2", "reviewer": "Reviewer_EfaD", "summary": "This paper proposes a dynamic personalization strategy using layer-wise Fisher information values, which enables flexible personalization and reduced noise impact. This paper further presents an adaptive method for enhancing clipping robustness by adjusting constraint strengths for different parameters, balancing client-specific features and global knowledge while minimizing clipping effects.", "questions": "1. Analysis on extra computation cost is expected. As mentioned in Sec 3.3, seems adding a computational overhead proportional to the number of parameters is huge, especially for deep models?\n2. For CIFAR-10, this paper evaluates average accuracy under different degrees of non-IID data partition, which seems unusual. What about the performance for different degrees of non-IIDness? \n3. What \"robust privacy\" means? seems robustness and privacy are two seperate topics?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is genrally well written.\n2. Proposed method seems novel. \n3. Experimental results under multiple tasks and settings seem promising.", "weaknesses": "1. Experiments are limited to simple datasets CIFAR-10 and FEMNIST datasets.\n2. Analysis on extra computation cost is expected. As mentioned in Sec 3.3, seems adding a computational overhead proportional to the number of parameters is huge, especially for deep models?\n3. For CIFAR-10, this paper evaluates average accuracy under different degrees of non-IID data partition, which seems unusual. What about the performance for different degrees of non-IIDness? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer EfaD: \n\nThank you for your thoughtful review and for raising key concerns regarding our work. We aim to address your concerns in our detailed responses below, hoping to provide clarity and demonstrate the effectiveness of our proposed approach.\n\n**Weaknesses:**\n\n**W1: Experiments are limited to simple datasets**\n\n**A1:** To enhance the credibility and robustness of our results, we have conducted experiments on the SVHN dataset, which is a larger and more complex dataset. The experiment results are as follows:\n\nTable: Comparison of model performance under different epsilons\n\n| Methods|ε = 2|ε = 4|ε = 6|ε = 8|\n|-|-|-|-|-|\n| DP-FedAvg|53.91|54.76|56.33|57.65|\n| BLUR+LUS|53.76|57.52|56.90|58.57|\n| PPSGD |55.89|56.15|56.74|59.37|\n| DP-FedSAM|53.04|52.84|54.03|56.08|\n| Ours |58.78|62.63|63.66|64.57|\n\nExperiments demonstrate that our method still surpasses other state-of-the-art approaches in a larger and more complex dataset, showcasing the effectiveness of our method in different scenarios.\n\nWe will incorporate this experiment into the final version if accepted.\n\n**W2: Analysis of extra computation cost**\n\n**A2:** Our analysis of extra computation cost is as follows:\n\nAssuming a total of N Global Epochs, M clients, and P parameters for each model, during each client update process, the client undergoes S steps of parameter updates. In each round, our method introduces the following extra computation costs:\n\n1. Computing the Fisher information value\n\n2. Construction of the mask\n\n3. Calculating the starting parameters based on the above results\n\nIn these three steps, considering that each step is parameter-specific, our method brings an additional computational cost of $\\mathcal{O}(3 \\cdot P)$, regarding that we have to carry out these operations once in each global epoch, we introduce an extra computational cost of $\\mathcal{O}(3 \\cdot E \\cdot M \\cdot N \\cdot P)$ over the entire training process.\n\nDespite our extra computational cost, it is noteworthy that our approach has a  **relatively small** overhead compared to the overall training process. This is because, in the entire update process, the model's training still has a computational cost proportional to P, and considering all the training process would introduce $\\mathcal{O}( E \\cdot M \\cdot N \\cdot P \\cdot S)$. Generally, in a training process, the value of S is in the tens or hundreds, so the computation cost introduced by our method will be **an order of magnitude** or more less than the training expense. Extensive experiments have verified the consistent improvement over the counterparts.\n\n**W3: Performance under different degrees of non-IIDness**\n\n**A3:** In our paper, we explore the performance under varying degrees of non-IIDness, as presented in Table 1, as the degree of heterogeneity escalates ($\\alpha$ changing from 1 to 100, with IID representing infinite $\\alpha$), the model performance experiences degradation. This observation underscores that heightened heterogeneity can compromise both robustness and performance.  However, compared with other baselines, the performance of degradation on our method is minor, demonstrating the robustness of our method, as verified in the following experiment\n\n| Method    | α = 0.1 | α = 0.3 | α = 0.5 |\n|-----------|---------|---------|---------|\n| DP-FedAvg | 24.65   | 30.07   | 38.43   |\n| BLUR+LUS  | 23.98   | 33.20   | 40.67   |\n| PPSGD     | 25.04   | 36.48   | 39.42   |\n| DP-FedSAM | 21.20   | 28.11   | 34.05   |\n| Ours      | 25.17   | 37.66   | 41.34   |\n\n**Questions:**\n\n**Q1: Analysis of extra computation cost**\n\n**A1:** As replied above.\n\n**Q2: Performance under different degrees of non-IIDness**\n\n**A2:** As replied above.\n\n**Q3: Meaning of “robust privacy”**\n\n**A3:** We use the \"robust privacy in Section 2.2 of our manuscript, we intended to emphasize the enhanced and strict privacy assurance offered by the Differential Privacy mechanism. This notion signifies a more fortified level of privacy protection compared to the conventional \"data-stay-local\" privacy concept associated with Federated Learning. \n\nWe will improve the clarity to avoid misleading understanding in the final version."}, {"review_id": "RktiO0gOop", "reviewer": "Reviewer_aCpX", "summary": "This study generally explores the problem of personalized federated learning with differential privacy. Specifically, the paper proposes two novel methods known as DFP and AC for enhancing the learning capability of a model while reducing the impact of noise, and improving robustness against clipping. At its core, the two methods encompass: 1) An adaptive scheme that selects parameters characterized by high Fisher values for personalization parts, and 2) different constraints on parameters, based on the information they carry. The experiments were conducted across multiple tasks and settings and have shown promising results, demonstrating the efficacy of the proposed method.", "questions": "1. I'm curious if the DFP technique still maintains its effectiveness when differential privacy constraints are absent. Could the authors elaborate on this and explain the underlying reasons?\n\n2. Considering the paper's emphasis on differential privacy for client-side protection, it provokes my consideration of the proposed method's scalability and interoperability with other privacy-preserving techniques. Could it be possible for the introduced approach could be seamlessly integrated with other strategies, such as secure aggregation, for enhanced privacy protection?", "limitations": "The discussion on limitations appears narrow, predominantly focusing on model heterogeneity. It could benefit from discussing further limitations and potential negative societal impact.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This study addresses a critical challenge in the field of federated learning and privacy preservation, particularly in the context of personalized federated learning with differential privacy. The proposed methods show promise in mitigating associated challenges, and offer a reasonable balance between model personalization, privacy protection, and the impact of noise, presenting a noteworthy contribution to the federated learning domain.\n\n2. The authors propose modifications to the client's local training procedure that could potentially enhance the robustness of the training results against differential privacy operations. This is an insightful contribution that seems to manage the trade-off between privacy and utility in federated learning, an important consideration in environments where privacy is paramount.\n\n3. The paper introduces the use of Fisher information values in the personalization process, which is a novel approach within the realm of federated learning. This offers a dynamic identification of personalized parts in each round, providing more flexibility compared to the current fixed portion personalization methods.\n\n4. The literature review in Section 2 of this paper is comprehensive. It provides a thorough overview of federated learning methods under differential privacy, all of which appear to be current and relevant. \n\n5. The experimental results in the study are promising. The proposed methods appear to perform better than the state-of-the-art methods under various settings. This, combined with the in-depth experimental section, suggests the potential robustness and efficacy of the proposed methods in personalized federated learning with differential privacy. \n", "weaknesses": "1. The analysis in Section 3.3 is not thorough enough when it comes to measuring complexity. The authors only focus on the number of parameters P and overlook important training factors like the size of the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aCpX:\nThank you very much for your recognition of our work and for raising thoughtful questions and concerns about our work. We have carefully considered each comment and provided responses. \n\n**Weaknesses:**\n\n**W1: Complete complexity analysis**\n\n**A1:** Our analysis of extra computation cost is as follows:\n\nAssuming a total of $N$ Global Epochs, $M$ clients, and $P$ parameters for each model, during each client update process, the client undergoes $S$ steps of parameter updates. In each round, our method introduces the following extra computation costs:\n\n- Computing the Fisher information value\n- Construction of the mask\n- Calculating the starting parameters based on the above results\n\nIn these three steps, considering that each step is parameter-specific, our method brings an additional computational cost of $\\mathcal{O}(3 \\cdot P)$, regarding that we have to carry out these operations once in each global epoch, we introduce an extra computational cost of $\\mathcal{O}(3 \\cdot E \\cdot M \\cdot N \\cdot P)$ over the entire training process.\n\nDespite our extra computational cost, it is noteworthy that our approach has a relatively **small overhead** compared to the overall training process. This is because, in the entire update process, the model's training still has a computational cost proportional to $P$, and considering all the training process would introduce $\\mathcal{O}( E \\cdot M \\cdot N \\cdot P \\cdot S)$. Generally, in a training process, the value of $S$ is in the tens or hundreds, so the computation cost introduced by our method will be an order of magnitude or more less than the training expense. Extensive experiments have verified the consistent improvement over the counterparts.\n\nWe will include this analysis into the final version of our work if accepted.\n\n**Questions:**\n\n**Q1: Effectiveness of our method under other conditions**\n\n**A1:** Our Dynamic Fisher Personalization (DFP) technique will retain its effectiveness even in a scenario without differential privacy protection. The effectiveness of this Personalization lies in retaining parts that have learned client-specific key knowledge within the client during the personalization process, which can simultaneously enhance the personalized effect while reducing the impact of noise on model training.\n\n**Q2: Feasibility of integration with other methods**\n\n**A2:** Considering that our method does not have any operations conflicting with encryption during the process of sending the gradient back to the server, it is feasible to integrate secure aggregation methods into our framework, demonstrating the scalability of our approach.\n"}, {"review_id": "ZU6noaGLRo", "reviewer": "Reviewer_UFEt", "summary": "This paper addresses the pressing issues of inflexibility and convergence difficulties in personalized federated learning under the influence of differential privacy constraints. The authors propose an innovative adaptive approach that prioritizes high fisher value local parameters during the personalization process while utilizing differential constraint techniques to enhance convergence. Rigorous empirical testing conducted on CIFAR-10 and EMNIST datasets corroborates the effectiveness of the proposed method.", "questions": "Could the authors  provide further insights into the motivation behind preserving parameters with high fisher values for personalization? How do these parameters enhance the personalization process in the federated learning context? ", "limitations": "The authors' discussion about potential adverse social impact is sparse. Inclusion of this aspect would enhance the overall analysis.\n\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors tackle an important problem in the realm of personalized federated learning. Given that current methodologies often rely heavily on fixed personalization elements, the ability to dynamically select parameters for personalization, as proposed in this paper, could potentially enrich the field.\n\n- The addressed problem also matters in the context of model training under differential privacy. While differential privacy can offer stricter privacy protection during model training, its inclusion can significantly interfere with the learning process. The proposed adaptive constraint method, by introducing an adaptive algorithm for differential privacy techniques, effectively addresses this critical issue.\n\n- The proposed technique of incorporating Fisher information into the personalization process is a novel approach. They select parameters based on Fisher information values, which introduces an innovative angle to the personalization process.\n\n- The experimentation in the paper is thorough, considering various influential factors. By conducting experiments under different settings on the CIFAR-10 and EMNIST datasets - such as varying degrees of non-iid data among clients and different privacy budget epsilon settings - the authors demonstrate the effectiveness of the proposed method.\n\n- The paper is clearly written and well-structured. The authors clearly split the problem they are solving into two parts, providing a clear and logical flow that facilitates understanding.\n\n", "weaknesses": "- Although the authors touch upon some limitations of their proposed methodology, a more comprehensive discussion on potential adverse societal impacts would have strengthened the paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UEFt:\n\nThank you very much for your affirmation of our work, as well as the insightful concerns and questions you have raised. We have carefully considered each comment and provided responses. \n\n**Weaknesses:**\n\n**W1: More discussion on potential adverse social impact**\n\n**A1:** Thank you for pointing this out, we have added the following discussion: \n\nTowards the proposed personalized federated learning approach, if personalization is not handled carefully, it might introduce biases that favor particular groups, which is an important adverse social impact. \n\nRegarding we propose a method that adapts to differential privacy, achieving good generalization while protecting user data privacy. In practical applications, care should be taken to strike a balance between privacy and utility, otherwise, it may lead to adverse social impacts such as inefficient use of data or data leakage. \n\nWe will incorporate this discussion into the final version if accepted.\n\n**Questions:**\n\n**Q1: Motivation for personalization strategy and working manner of our personalized approach**\n\n**A1:** The motivation behind preserving parameters with high Fisher values for personalization in our work stems from their ability to quantify the amount of information that these parameters can provide. More informative parameters represent knowledge more effectively and play a crucial role in model prediction. By employing layer-wise Fisher information to measure the information content of client parameters, we dynamically determine the personalized parameter portion for each client, ensuring that these informative parameters remain unaffected by noise introduced for privacy protection. \n\nIn the federated learning context, this approach enhances the adaptability of the model to individual clients' data distributions, effectively overcoming the challenges associated with inflexible personalization and the impact of noise, leading to improved personalized performance.\n"}]

Author rebuttal:

Meta-review: A new method is presented to address the challenges of personalization and privacy in FL by retaining informative local parameters while reducing the impact of noise, resulting in better personalization and robustness against clipping. Experiments are run on  CIFAR-10 and FEMNIST datasets and show the effectiveness of the proposed method.  Reviews are fairly consistent and positive.
Decision: Accept (poster)