SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\odot$  denotes the Hadamard product and  $w^{t - 1}$  is the global parameters downloaded from server. This approach ensures that more informative parameters (with larger Fisher values) are retained for personalized learning, while less informative ones are updated with the global parameters, mitigating the influence of noise.
Comparison with Analogous Methods. PPSGD [3] employs a fixed layer personalization approach with an additive model to achieve personalization under differential privacy (DP). Meanwhile, the LUS mechanism [8] utilizes a mask during their sparsification process after local training. Although these methods have their merits, they are fundamentally different from our proposed strategy. In our work, we advocate for a dynamic personalization approach using a mask, but crucially, this is applied prior to the training process. Our aim is to provide a more flexible and adaptive personalization strategy that can better cope with changes in the data distribution, noise level, and other dynamic factors affecting the model performance. Furthermore, we focus on preserving the most informative parameters, those that significantly contribute to the model performance, while allowing less informative ones to be updated with global parameters, thereby reducing the impact of noise. This dynamic, pre-training personalization strategy provides a unique advantage over the static, post-training personalization of the aforementioned methods.
3.3 Adaptive Constraint
Motivation for Adaptive Constraint. The challenge of gradient distortion and convergence difficulties in DP-PFL arises from the clipping operation applied to the L2 norm of each local update. Traditional methods[8, 23] often apply a uniform regularization strength to all parameters, neglecting the inherent differences among parameter magnitudes. To address this issue, we propose an adaptive approach that determines different constraint strengths for different parameters, considering the inherent magnitude of each parameter. This aims to maintain the learning capacity of the model while also enhancing its robustness against clipping.
**Construction of Adaptive Constraint.** As discussed in  $\mathbf{2}$ , the problem of gradient distortion and convergence difficulties in DP-PFL arises from the clipping operation applied to the L2 norm of each local update, our method is as follows. At the start of the local update phase, each client  $i$  at global epoch  $t$  has parameters  $w_{i}^{t-1} = (v^{t-1}, u_{i}^{t-1})$ . These parameters are divided into two sets: shared parameters  $v_{i}^{t-1}$  and personalized parameters  $u_{i}^{t-1}$ , identified by the previously introduced dynamic personalization strategy. We propose two different constraints for the local updates of these two sets of parameters.
For the personalized parameters, which contain unique knowledge from the local data, our aim is to retain this local knowledge as much as possible. We therefore introduce a regularization term to loss
function that constrains the L2 norm of the local update of the personalized parameters. The local update of personalized parameters  $u_{i}^{t}$  is expressed as:
$$
u _ {i} ^ {t} = u _ {i} ^ {t - 1} - \eta \nabla_ {u} \mathcal {L} _ {1} \left(v _ {i} ^ {t - 1}, u _ {i} ^ {t - 1}, D _ {i}\right), \tag {8}
$$
where  $\eta$  is the learning rate, and  $\nabla \mathcal{L}_1$  is the gradient of the first loss function. The first loss function,  $\mathcal{L}_1$ , is calculated as the sum of the cross-entropy loss and the L2 norm of the difference between the current and the previous personalized parameters:
$$
\mathcal {L} _ {1} = - \frac {1}{n} \sum_ {j = 1} ^ {n} \left[ y _ {j} \log \left(\hat {y} _ {j}\right) + \left(1 - y _ {j}\right) \log \left(1 - \hat {y} _ {j}\right) \right] + \frac {\lambda_ {1}}{2} \left| \left| u _ {i} - u _ {i} ^ {t} \right| \right| _ {2}, \tag {9}
$$
where  $n$  is the number of instances in the dataset  $D_{i}$ ,  $y_{j}$  is the true label of instance  $j$ ,  $\lambda_{1}$  is a hyper-parameter and  $\hat{y}_j$  is the predicted label of instance  $j$ .
For the shared parameters, we strive to ensure that their L2 norm values closely approach the clipping bound. By adopting such a strategy, we anticipate reducing the influence that the clipping operation exerts on the local update. We therefore introduce a bounded regularization term to the loss function of the shared parameters. The local update of shared parameters  $v_{i}^{t}$  is expressed as:
$$
v _ {i} ^ {t} = v _ {i} ^ {t - 1} - \eta \nabla_ {v} \mathcal {L} _ {2} \left(v _ {i} ^ {t - 1}, u _ {i} ^ {t}, D _ {i}\right), \tag {10}
$$
where  $\nabla \mathcal{L}_2$  is the gradient of the second loss function. The second loss function,  $\mathcal{L}_2$ , is calculated as the sum of the cross-entropy loss and the absolute difference between the L2 norm of the difference between the current and previous shared parameters and the clipping bound:
$$
\mathcal {L} _ {2} = - \frac {1}{n} \sum_ {j = 1} ^ {n} \left[ y _ {j} \log \left(\hat {y} _ {j}\right) + \left(1 - y _ {j}\right) \log \left(1 - \hat {y} _ {j}\right) \right] + \frac {\lambda_ {2}}{2} \left| \left| | v _ {i} - v _ {i} ^ {t} | | _ {2} - C \right| \right| _ {2}. \tag {11}
$$
Here,  $n$  is the number of instances in the dataset  $D_{i}$ ,  $y_{j}$  is the true label of instance  $j$ ,  $\hat{y}_j$  is the predicted label of instance  $j$ ,  $\lambda 2$  is also a hyper-parameter and  $C$  is the clipping bound.
By applying these adaptive constraints, our method effectively mitigates the impact of the clipping operation and boosts the performance of FedDPA. The two loss functions,  $\mathcal{L}_1$  and  $\mathcal{L}_2$ , cater to the unique properties of personalized and shared parameters, respectively.  $\mathcal{L}_1$  helps retain as much local knowledge as possible, overcoming the slow convergence issue. Additionally,  $\mathcal{L}_2$  ensures the shared parameters updates align with the clipping bound, thereby improving the model robustness against clipping. In both cases, the regularization strengths  $\lambda_1$  and  $\lambda_2$  are determined adaptively, taking into account the inherent magnitude of each parameter. This novel consideration distinguishes our method from traditional approaches and contributes to its superior performance.
3.4 Discussion and Limitation
Our analysis of extra computation cost is as follows: Assuming a total of  $E_{g}$  Global Epochs,  $M$  clients, and  $P$  parameters for each model, during each client update process, the client undergoes  $S$  steps of parameter updates in training. In each round, our method introduces computation costs as follows: 1) Computing the Fisher information value, 2) Construction of the mask and 3) Calculating the initial parameters using masks. In these steps, as each step is parameter-specific, our method brings an additional computational cost of  $\mathcal{O}(P)$ . Regarding that we have to carry out these operations in each global epoch, we introduce a computational cost of  $\mathcal{O}(E_g\cdot M\cdot P)$  in total.
Despite our extra computational cost, it is noteworthy that our approach has a relatively small overhead compared to the overall training process. This is because, the model training still has a considerable computational cost proportional to  $P$ , and all the training process would introduce  $\mathcal{O}(E_g \cdot M \cdot P \cdot S)$ . Generally, in a training process, the value of  $S$  is in the tens or hundreds, so the computation cost introduced by our method will be an order of magnitude or more less than the training expense. Extensive experiments have verified the consistent improvement over the counterparts.
Besides the core functionality of our model, it also shows potential for applications in other fields, such as feature selection in high-dimensional datasets, improving model robustness by focusing on critical parameters, and enhancing privacy protection by selective parameter sharing. Regarding the limitation of model homogeneity, our method assumes all clients share the same model structure, which may not always be the case in real-world scenarios. Heterogeneity in model structures across
Algorithm 1: The Proposed Method: FedDPA
Input: Global epochs  $E_{g}$  , local epochs  $E_{l}$  , participants number in the  $t^{th}$  epoch  $m^t$  , private data of the  $i^{th}$  client  $D_{i} = (X_{i},Y_{i})$  , global model parameters w and client local parameters  $w_{i}$  , hyper-parameter  $\lambda_1,\lambda 2,\tau$  , learning rate  $\eta$    
Local Update :   
for  $i = 1,2,\dots,m^{t}$  do   
Receive  $w^{t - 1}$  from Server   
 $\hat{F}_i\gets (D_i,w_i^{t - 1})$  by Eq. (4) and Eq. (5)   
 $M_{1i}$  and  $M_{2i}\leftarrow (\hat{F}_i,\tau)$  by Eq. (6)   
 $w_{i}^{t} = (v^{t - 1},u_{i}^{t - 1})\leftarrow M_{1i}\odot w_{i}^{t - 1} + M_{2i}\odot w^{t - 1}$  using Eq. (7)   
for  $e = 1,2,\ldots ,E_l$  do   
 $\begin{array}{r}u_i^t\gets u_i^{t - 1} - \eta \nabla_u\mathcal{L}_1(v^{t - 1},u_i^{t - 1})\text{by}\mathcal{L}_1\text{from(9)}\\ v_i^t\gets v^{t - 1} - \eta \nabla_v\mathcal{L}_2(v^{t - 1},u_i^t)\text{by}\mathcal{L}_2\text{from(11)} \end{array}$    
end   
 $\Delta w_{i}^{t}\gets (v_{i}^{t},u_{i}^{t}) - (v^{t - 1},u_{i}^{t - 1})$ $\Delta w_{i}^{t}\gets \Delta w_{i}^{t} / \max (1,\frac{\||\Delta w_{i}^{t}||_{2}}{C}) + \mathcal{N}(0,C^{2}\sigma^{2} / |m^{t}|)$  to ensure DP by Eq. (3)   
end   
Server Execute :   
 $\Delta w_{i}^{t}\gets$  Local Update(w  $t^{-1}$  )   
for  $t = 1,2,\dots,E_g$  do   
 $w^{t}\gets w^{t - 1} + \frac{1}{m^{t}}\sum_{i = 1}^{m^{t}}\Delta w_{i}^{t}$    
for  $i = 1,2,\dots,m^t$  do   
 $|$  Send  $w^t$  to  $i^{th}$  participant   
end
clients can potentially limit the utility of our dynamic personalization strategy and adaptive constraints, differences in model could lead to incompatibilities, which is an area where further research is needed to improve the versatility of our method. It should be noted that this limitation is not unique to our approach but is also shared by most of the aforementioned methods [8, 41, 38, 3].
We evaluate the performance of the proposed method and comparison methods through comprehensive experiments on two distinct datasets under various conditions. Specifically, we test the accuracy on the FEMNIST and SVHN dataset across different privacy budgets denoted by  $\epsilon$ , and for the CIFAR-10, we adjust the  $\alpha$  parameter to simulate varying degrees of non-IID distribution and assess accuracy accordingly. This wide-ranging experimental setup reflects practical scenarios and ensures a thorough assessment of our method's robustness and adaptability.
Data and Model. Our method is evaluated on two classification tasks, FEMNIST [6], CIFAR-10 [21] and SVHN [34], embodying real-world non-IID and privacy-constrained scenarios. The FEMNIST is a 62-class version of MNIST, CIFAR-10 is a widely-used image classification dataset, and the SVHN dataset it a digits classification dataset which is collected from street view. We use a simple CNN model with 2 convolution layers and 2 fully connected layers for FEMNIST and a deeper model with 3 convolution layers and 3 fully connected layers for CIFAR-10.
Comparison Methods. We compare the performance of our proposed method with several state-of-the-art methods in differential privacy-enabled federated learning, including DP-FedAvg [12], PPSGD [3], DP-FedAvg with LUS and BLUR [8], and DP-FedSAM [38].
Evaluation Metrics. The primary metric for evaluation is the average accuracy, calculated uniquely for each client. Each client has their own personalized model trained on their individual dataset, and accuracy is measured on these respective datasets. The average of these individual accuracies across all clients is then calculated to yield the final metric. For the FEMNIST and SVHN dataset, we evaluate this average accuracy under varying privacy budgets, underlining the advantage of our method in mitigating noise impact. For the CIFAR-10 dataset, we evaluate average accuracy
under different degrees of non-IID data partition, demonstrating the effectiveness of our dynamic personalization strategy in handling non-IID data.
Implementation Details. For all dataset FEMNIST, CIFAR-10 and SVHN, we set the learning rate to 1e-3 and optimize hyperparameters  $\tau$ ,  $\lambda_{1}$ , and  $\lambda_{2}$  through grid search in  $\{0.05, 0.1, 0.3, 0.5\}$ . We use the Renyi Differential Privacy (RDP) algorithm provided by Opacus as our privacy accountant. The value of  $\delta$ , in  $(\varepsilon, \delta)$ -differential privacy, is set to the reciprocal of the number of clients (i.e., 1/M). We use global epochs of 30 and 40, local epochs of 3 and 4, and batch sizes of 16 and 64 for FEMNIST and CIFAR-10, respectively. For CIFAR-10, we partition the dataset into 10 subsets via Dirichlet distribution and use a fixed noise multiplier to ensure differential privacy. For FEMNIST, we selected the top 50 clients from the FEMNIST [6] division for training. All experiments were implemented in Python with PyTorch on an NVIDIA 3090 GPU.
Table 1: Comparison of Model Performance under Different Conditions. Refer Sec. 4.1
Methods	FEMNIST	SVHN
ε=2	ε=4	ε=8	ε=16	ε=2	ε=4	ε=6	ε=8
DP-FedAvg [12]	71.92	72.80	73.26	75.02	53.91	54.76	56.33	57.65
BLUR+LUS [8]	72.80	74.23	74.58	75.16	53.76	57.52	56.90	58.57
PPSGD [3]	68.20	69.60	71.03	71.93	55.89	56.15	56.74	59.37
DP-FedSAM [38]	73.13	73.94	74.54	74.66	53.04	52.84	54.03	56.08
FedDPA(Ours)	74.46↑1.33	77.27↑3.04	77.42↑2.84	76.99↑1.83	58.78↑2.89	62.63↑5.11	63.66↑6.76	64.57↑5.29
Table 2: Performance Comparison on CIFAR-10. Refer Sec. 4.1
Methods	α = 1	α = 10	α = 100	IID
DP-FedAvg [12]	42.61	58.89	60.03	60.97
BLUR+LUS [8]	48.97	58.70	60.50	61.12
PPSGD [3]	44.04	57.31	60.20	60.02
DP-FedSAM [38]	45.07	56.15	58.87	60.77
FedDPA(Ours)	49.75↑0.78	59.48↑0.59	60.68↑0.18	61.55↑0.43
4.1 Comparison with State-Of-the-Art Methods
We provide comparison results with SOTA methods on two image classification tasks [21, 6].
Analysis Under Different Privacy Budget. As Tab.1 shows, our method maintains high accuracy across various privacy budgets on the FEMNIST dataset, underlining its robustness and superior privacy-utility trade-off. The convergence plots Fig.1a and Fig.1b support our method's efficiency, demonstrating faster convergence and more effective loss minimization.
Analysis Under Different Degrees of Non-IID Data. Our method shows strong performance across varying non-IID data partitions on the CIFAR-10 dataset, as seen on the right side of Tab.2. It effectively handles non-IID data challenges via our flexible personalization method.
(a) Accuracy convergence curve
(b) Loss convergence curve
Figure 1: Comparison of Model Performance and Optimization Objective Value. See details in 4.1
4.2 Diagnostic Experiments
To highlight the contribution of each component in our method towards the overall performance, we conduct a series of ablation experiments. Our proposed method comprises two integral components.
Dynamic Fisher Personalization (DFP). The DFP, as described in Sec. 3.2, contributes to the improvement of accuracy across various privacy budgets. When DFP is employed (second row in the table), there is a noticeable increase in accuracy for all privacy budgets compared to the base model (first row), validating the effectiveness of DFP in enhancing model performance.
Adaptive Constraint (AC). The AC strategy, as introduced in Sec. 3.3, is dependent on DFP for its operation. Therefore, there is no experiment featuring only the AC without the DFP. However, when both DFP and AC are incorporated (third row in the table), the model's accuracy further improves across all privacy budgets. This demonstrates that the AC strategy enhances the utility of DFP, contributing to an overall superior privacy-utility trade-off.
Table 3: Ablation Study with Different Privacy Budgets. Refer Sec. 4.2
Components	Accuracy on clients dataset
DFP	AC	ε = 2	ε = 4	ε = 8	ε = 16
71.92	72.80	73.26	75.02
✓	72.18	74.56	74.38	75.33
✓	✓	74.46↑2.28	77.27↑2.71	77.42↑2.93	76.99↑1.66
The results in Table 3 clearly illustrate that both DFP and AC contribute significantly to the performance of the model under various privacy budgets. The combined use of both components provides the best results, reinforcing the effectiveness of our proposed method.
4.3 Hyperparameter Study
We vary the hyperparameters including  $\lambda$  and  $\tau$  in our proposed methods to demonstrate how our model retains or change personalized portions in DFP and the sensitivity of constraint strength in AC.
Table 4: Ablation Study on  $\tau$  . Refer Sec. 4.3
τ	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
Proportion	100%	11.1%	4.5%	1.45%	0.97%	0.46%	0.15%	0.1%	0.1%	0.05%	0%
Accuracy	39.72	47.97	49.28	48.89	48.35	48.03	48.28	48.14	48.35	47.74	48.29
Table 5: Ablation Study on  ${\lambda }_{1}$  . Refer Sec.4.3
λ1	0	0.05	0.1	0.3	0.5
Accuracy	60.82	62.64	59.66	59.28	59.57
Table 6: Ablation Study on  ${\lambda }_{2}$  . Refer Sec. 4.3
λ2	0	0.05	0.1	0.3	0.5
Accuracy	58.34	59.76	61.01	47.61	45.66
For  $\tau$ , we observe that our proposed method is relatively stable, verifying the robustness of our proposed solution. For  $\lambda$ , we achieve the best performance when  $\lambda_{1}$  is around 0.05 and  $\lambda_{2}$  is around 0.1. We apply this across multiple different datasets, and it results in a consistent good performance.
In this paper, we delve into the challenges of inflexible personalization and convergence difficulties under differential privacy in personalized federated learning. We introduce FedDPA with two innovative components, Dynamic Fisher Personalization (DFP) and Adaptive Constraint (AC), that utilize layer-wise Fisher information to dynamically personalize and adaptively constrain parameters, effectively addressing the aforementioned issues. The efficacy of our proposed methods has been thoroughly validated against many popular counterparts across various classification tasks. We are hopeful that this work will pave the way for future research on personalized federated learning with differential privacy.
Acknowledgment. This work is partially supported by National Natural Science Foundation of China under Grant (62176188), the Key Research and Development Program of Hubei Province (2021BAA187, 2022BCA009), Zhejiang lab (NO.2022NF0AB01), CCF-Huawei Populus Grove Fund (CCF-HuaweiTC2022003), and the Special Fund of Hubei Luojia Laboratory (220100015).