SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After obtaining a set of candidate models  $\{\mathbf{c}_t^1,\dots ,\mathbf{c}_t^M\}$ , to optimize Eq. (2), we need to calculate the similarity between each client model  $\mathbf{w}_t^n$  and all the cadidates  $\{\mathbf{c}_t^1,\dots ,\mathbf{c}_t^M\}$  using the public data  $\mathcal{D}_p$ . However, this is non-trivial since  $\mathbf{c}_t^m$  is assembled by layers from different client models, which is not a complete model architecture. We have to stitch these layers together before using  $\mathbf{c}_t^m$ .
Layer Stitching. Assume that  $\mathbf{L}_{t,i}^n$  and  $\mathbf{L}_{t,j}^b$  are any two consecutive layers in the candidate model  $\mathbf{c}_t^m$ . Let  $d_{i}$  denote the output dimension of  $\mathbf{L}_{t,i}^n$  and  $d_{j}$  denote the input dimension of  $\mathbf{L}_{t,j}^b$ .  $d_{i}$  is usually not equal to  $d_{j}$ . To stitch these two layers, we follow existing work [34] by adding a nonlinear activation function  $\mathrm{ReLU}(\cdot)$  on top of a linear layer, i.e.,  $\mathrm{ReLU}(\mathbf{W}^\top \mathbf{X} + \mathbf{b})$ , where  $\mathbf{W} \in \mathbb{R}^{d_i \times d_j}$ ,  $\mathbf{b} \in \mathbb{R}^{d_j}$ , and  $\mathbf{X}$  represents the output data from the first layer. In such a way, we can obtain a stitched candidate  $\tilde{\mathbf{c}}_t^m$ . The reasons that we apply this simple layer as the stitch are twofold. On the one hand, even adding a simple linear layer between any two consecutive layers, the model will increase  $d_{i} * (d_{j} + 1)$  parameters. Since a candidate  $\mathbf{c}_t^m$  may contain several layers, if using more complicated layers as the stitch, the number of parameters will significantly increase, which makes the new candidate model hard to be trained. On the other hand, using a simple layer with a few parameters may be helpful for the new candidate model to maintain more information from the original models. This is of importance for the similarity calculation in the next step.
Similarity Calculation. We propose to use the cosine score  $\cos (\cdot ,\cdot)$  to calculate the similarity between a pair of models  $(\mathbf{w}_t^n,\tilde{\mathbf{c}}_t^m)$  as follows:
$$
\operatorname {s i m} \left(\mathbf {w} _ {t} ^ {n}, \mathbf {c} _ {t} ^ {m}; \mathcal {D} _ {p}\right) = \operatorname {s i m} \left(\mathbf {w} _ {t} ^ {n}, \tilde {\mathbf {c}} _ {t} ^ {m}; \mathcal {D} _ {p}\right) = \frac {1}{P} \sum_ {p = 1} ^ {P} \cos \left(\boldsymbol {\alpha} _ {t} ^ {n} (\mathbf {x} _ {p}), \boldsymbol {\alpha} _ {t} ^ {m} (\mathbf {x} _ {p})\right), \tag {5}
$$
where  $P$  denotes the number of data in the public dataset  $\mathcal{D}_p$  and  $\mathbf{x}_p$  is the  $p$ -th data in  $\mathcal{D}_p$ .  $\alpha_t^n (\mathbf{x}_p)$  and  $\alpha_{t}^{m}(\mathbf{x}_{p})$  are the logits output from models  $\mathbf{w}_t^n$  and  $\tilde{\mathbf{c}}_t^m$ , respectively. To obtain the logits, we
need to finetune  $\mathbf{w}_t^n$  and  $\tilde{\mathbf{c}}_t^m$  using  $\mathcal{D}_p$  first. In our design, we can use both labeled and unlabeled data to finetune models but with different loss functions. If  $\mathcal{D}_p$  is labeled, then we use the supervised cross-entropy (CE) loss to finetune the model. If  $\mathcal{D}_p$  is unlabeled, then we apply the self-supervised contrastive loss to finetune them following [38].
3.2 Client Update
The obtained personalized model  $\hat{\mathbf{w}}_t^n$  (i.e.,  $\mathbf{c}_t^*$  in Eq. (2)) will be distributed to the  $n$ -th client if it is selected in the next communication round  $t + 1$ .  $\hat{\mathbf{w}}_t^n$  is a reassembled model that carries external knowledge from other clients, but its network structure is different from the original  $\mathbf{w}_t^n$ . To incorporate the new knowledge without training  $\mathbf{w}_t^n$  from scratch, we propose to apply knowledge distillation on the client following [39].
Let  $\mathcal{D}_n = \left\{(\mathbf{x}_i^n,\mathbf{y}_i^n)\right\}$  denote the labeled data, where  $\mathbf{x}_i^n$  is the data feature and  $\mathbf{y}_i^n$  is the corresponding ground truth vector. The loss of training local model with knowledge distillation is defined as follows:
$$
\mathcal {J} _ {n} = \frac {1}{| \mathcal {D} _ {n} |} \sum_ {i = 1} ^ {| \mathcal {D} _ {n} |} \left[ \mathrm {C E} \left(\mathbf {w} _ {t} ^ {n} \left(\mathbf {x} _ {i} ^ {n}\right), \mathbf {y} _ {i} ^ {n}\right) + \lambda \mathrm {K L} \left(\boldsymbol {\alpha} _ {t} ^ {n} \left(\mathbf {x} _ {i} ^ {n}\right), \hat {\boldsymbol {\alpha}} _ {t} ^ {n} \left(\mathbf {x} _ {i} ^ {n}\right)\right) \right], \tag {6}
$$
where  $|\mathcal{D}_n|$  denotes the number of data in  $\mathcal{D}_n$ ,  $\mathbf{w}_t^n (\mathbf{x}_i^n)$  means the predicted label distribution,  $\lambda$  is a hyperparameter,  $\mathrm{KL}(\cdot ,\cdot)$  is the Kullback-Leibler divergence, and  $\alpha_{t}^{n}(\mathbf{x}_{i}^{n})$  and  $\hat{\alpha}_t^n (\mathbf{x}_i^n)$  are the logits from the local model  $\mathbf{w}_t^n$  and the downloaded personalized model  $\hat{\mathbf{w}}_t^n$ , respectively.
Datasets. We conduct experiments for the image classification task on MNIST, SVHN, and CIFAR-10 datasets under both IID and non-IID data distribution settings, respectively. We split the datasets into  $80\%$  for training and  $20\%$  for testing. During training, we randomly sample  $10\%$  training data to put in the server as  $\mathcal{D}_p$  and the remaining  $90\%$  to distribute to the clients. The training and testing datasets are randomly sampled for the IID setting. For the non-IID setting, each client randomly holds two classes of data. To test the personalization effectiveness, we sample the testing dataset following the label distribution as the training dataset. We also conduct experiments to test models using different public data in Section 4.4.
Baselines. We compare pFedHR with the baselines under two settings. (1) Heterogenous setting. In this setting, clients are allowed to have different model structures. The proposed pFedHR is general and can use both labeled and unlabeled public datasets to conduct heterogeneous model cooperation. To make fair comparisons, we use FedMD [17] and FedGH [22] as baselines when using the labeled public data, and FCCL [18] and FedKEMF [20] when testing the unlabeled public data. (2) Homogenous setting. In this setting, clients share an identical model structure. We use traditional and personalized FL models as baselines, including FedAvg [4], FedProx [2], Per-FedAvg [26], PFedMe [27], and PFedBayes [28].
4.2 Heterogenous Setting Evaluation
Small Number of Clients. Similar to existing work [18], to test the performance with a small number of clients, we set the client number  $N = 12$  and active client number  $B = 4$  in each communication round. We design 4 types of models with different structures and randomly assign each type of model to 3 clients. The Conv operation contains convolution, max pooling, batch normalization, and ReLu, and the FC layer contains fully connected mapping, ReLU, and dropout. We set the number of clusters  $K = 4$ . Then local training epoch and the server finetuning epoch are equal to 10 and 3, respectively. The public data and client data are from the same dataset.
Table 2 shows the experimental results for the heterogeneous setting using both labeled and unlabeled public data. We can observe that the proposed pFedHR achieves state-of-the-art performance on all datasets and settings. We also find the methods using labeled public datasets can boost the performance compared with unlabeled public ones in general, which aligns with our expectations and experiences.
Table 2: Performance comparison with baselines under the heterogeneous setting.
Public Data	Dataset	MNIST	SVHN	CIFAR-10
Model	IID	Non-IID	IID	Non-IID	IID	Non-IID
Labeled	FedMD [17]	93.08%	91.44%	81.55%	78.39%	68.22%	66.13%
FedGH [22]	94.10%	93.27%	81.94%	81.06%	72.69%	70.27%
pFedHR	94.55%	94.41%	83.68%	83.40%	73.88%	71.74%
Unlabeled	FedKEMF [20]	93.01%	91.66%	80.41%	79.33%	67.12%	66.93%
FCCL [18]	93.62%	92.88%	82.03%	79.75%	68.77%	66.49%
pFedHR	93.89%	93.76%	83.15%	80.24%	69.38%	68.01%
Large Number of Clients. We also test the performance of models with a large number of clients. When the number of active clients  $B$  is large, calculating layer pair-wise distance values using Eq. (3) will be highly time-consuming. To avoid this issue, a straightforward solution is to conduct FedAvg [4] for averaging the models with the same structures first and then do the function-driven layer grouping based on the averaged structures via Eq. (4). The following operations are the same as pFedHR. In this experiment, we set  $N = 100$  clients and the active number of clients  $B = 10$ . Other settings are the same as those in the small number client experiment.
Table 3 shows the results on the SVHN dataset for testing the proposed pFedHR for a large number of clients setting. The results show similar patterns as those listed in Table 2, where the proposed pFedHR achieves the best performance under IID and NonIID settings whether it uses labeled or unlabeled public datasets. Compared to the results on the SVHN dataset in Table 2, we can find that the performance of all the baselines and our models drops. Because the number of training data is fixed, allocating these
Table 3: Evaluation using a large number of clients on the SVHN dataset  $(N = 100)$
Public Data	Model	IID	Non-IID
Labeled	FedMD	78.16%	74.34%
FedGH	76.27%	72.78%
pFedHR	80.02%	77.63%
Unlabeled	FedKEAF	76.27%	74.61%
FCCL	75.03%	71.54%
pFedHR	78.98%	75.77%
data to 100 clients will make each client use fewer data for training, which leads to a performance drop. The results on both small and large numbers of clients clearly demonstrate the effectiveness of our model for addressing the heterogeneous model cooperation issue in federated learning.
4.3 Homogeneous Setting Evaluation
In this experiment, all the clients use the same model structure. We test the performance of our proposed pFedHR on representative models with the smallest (M1) and largest (M4) number of layers, compared with state-of-the-art homogeneous federated learning models. Except for the identical model structure, other experimental settings are the same as those used in the scenario of the small number of clients. Note that for pFedHR, we report the results using the labeled public data in this experiment. The results are shown in Table 4.
Table 4: Homogeneous model comparison with baselines.
Model	Dataset	MNIST	SVHN	CIFAR-10
Setting	IID	Non-IID	IID	Non-IID	IID	Non-IID
M1	FedAvg [4]	91.23%	90.04%	53.45%	51.33%	43.05%	33.39%
FedProx [2]	92.66%	92.47%	54.86%	53.09%	43.62%	35.06%
Per-FedAvg [26]	93.23%	93.04%	54.29%	52.04%	44.14%	42.02%
PFedMe [27]	93.57%	92.00%	55.01%	53.78%	45.01%	43.65%
PFedBayes [28]	94.39%	93.32%	58.49%	55.74%	46.12%	44.49%
pFedHR	94.26%	93.26%	61.72%	59.23%	54.38%	48.44%
M4	FedAvg [4]	94.24%	92.16%	83.26%	82.77%	67.68%	58.92%
FedProx [2]	94.22%	93.22%	84.72%	83.00%	71.24%	63.98%
Per-FedAvg [26]	95.77%	93.67%	85.99%	84.01%	79.56%	76.23%
PFedMe [27]	95.71%	94.02%	87.63%	85.33%	79.88%	77.56%
PFedBayes [28]	95.64%	93.23%	88.34%	86.28%	80.06%	77.93%
pFedHR	94.88%	93.77%	89.87%	87.94%	81.54%	79.45%
We can observe that using a simple model (M1) can make models achieve relatively high performance since the MNIST dataset is easy. However, for complicated datasets, i.e., SVHN and CIFAR-10, using a complex model structure (i.e., M4) is helpful for all approaches to improve their performance significantly. Our proposed pFedHR outperforms all baselines on these two datasets, even equipping with a simple model M1. Note that our model is proposed to address the heterogeneous model cooperation problem instead of the homogeneous personalization in FL. Thus, it is a practical
approach and can achieve personalization. Still, we also need to mention that it needs extra public data on the server, which is different from baselines.
4.4 Public Dataset Analysis
Sensitivity to the Public Data Selection. In the previous experiments, the public and client data are from the same dataset, i.e., having the same distribution. To validate the effect of using different public data during model learning for all baselines and our model, we conduct experiments by choosing public data from different datasets and report the results on the SVHN dataset. Other experimental settings are the same as those in the scenario of the small number of clients.
Figure 1 shows the experimental results for all approaches using labeled and unlabeled public datasets. We can observe that replacing the public data will make all approaches decrease performance. This is reasonable since the data distributions between public and client data are different. However, compared with baselines, the proposed pFedHR has the lowest performance drop. Even using other public data, pFedHR can achieve comparable or better performance with baselines using SVHN as the public data. This advantage stems from our model design. As described in Section 3.1.3, we keep more information from original client models by using a simple layer as the stitch. Besides, we aim to search for the most similar personalized candidate with a client model. We propose to calculate the average logits in Eq. (5) as the criteria. To obtain the logits, we do not need to finetune the models many times. In our experiments, we set the number of finetuning epochs as 3. This strategy can also help the model reduce the adverse impact of public data during model training.
Sensitivity to the Percentage of Public Data. Several factors can affect model performance. In this experiment, we aim to investigate whether the percentage of public data is a key factor in influencing performance change. Toward this end, we still use the small number of clients setting, i.e., the client and public data are from the SVHN dataset, but we adjust the percentage of public data. In the original experiment, we used  $10\%$  data as the public data. Now, we reduce this percentage to  $2\%$  and  $5\%$ . The results are shown in Figure 3. We can observe that with the increase in the percentage of public data, the performance of the proposed pFedHR also improves. These results align with
Figure 3: Performance change w.r.t. the percentage of public data.
our expectations since more public data used for finetuning can help pFedHR obtain more accurately matched personalized models, further enhancing the final accuracy.
4.5 Experiment results with Different Numbers of Clusters
In our model design, we need to group functional layers into  $K$  groups by optimizing Eq. (4). Where  $K$  is a predefined hyperparameter. In this experiment, we aim to investigate the performance influence with regard to  $K$ . In particular, we conduct the experiments on the SVHN dataset with 12 local clients, and the public data are also the SVHN data.
Figure 4 shows the results on both IID and Non-IID settings with labeled and unlabeled public data.  $X$ -axis represents the number of clusters, and  $Y$ -axis denotes the accuracy values. We can observe that with the increase of  $K$ , the performance will also increase. However, in the experiments, we do not recommend setting a large  $K$  since a trade-off balance exists between  $K$  and  $M$ , where  $M$  is the number of candidates automatically generated by Algorithm 1 in the main manuscript. If  $K$  is large, then  $M$  will be small due to Rule  $\mathcal{R}_4$ . In other words, a larger  $K$  may make the empty  $\mathcal{C}_t$  returned by Algorithm 1 in the main manuscript.
4.6 Layer Stitching Study
One of our major contributions is to develop a new layer stitching strategy to reduce the adverse impacts of introducing public data, even with different distributions from the client data. Our proposed strategy includes two aspects: (1) using a simple layer to stitch layers and (2) reducing the number of finetuning epochs. To validate the correctness of these assumptions, we conduct the following experiments on SVHN with 12 clients, where both client data and labeled public data are extracted from SVHN.
(a) Labeled Public Dataset
(b) Unlabeled Public Dataset
Figure 4: Results on different number of clusters  $K$ 's.
Stitching Layer Numbers. In this experiment, we add the complexity of layers for stitching. In our model design, we only use  $\mathrm{ReLU}(\mathbf{W}^\top \mathbf{X} + \mathbf{b})$ . Now, we increase the number of linear layers from 1 to 2 to 3. The results are depicted in Figure 5. We can observe that under both IID and Non-IID settings, the performance will decrease with the increase of the complexity of the stitching layers. These results demonstrate our assumption that more complex stitching layers will introduce more information about public data but reduce the personalized information of each client model maintained. Thus, using a simple layer to stitch layers is a reasonable choice.
Stitched Model Finetuning Numbers. We further explore the influence of the number of finetuning epochs on the stitched model. The results are shown in Figure 6. We can observe that increasing the number of finetuning epochs can also introduce more public data information and reduce model performance. Thus, setting a small number of finetuning epochs benefits keeping the model's performance.
Figure 5: Stitching layer number study.
Figure 6: Server finetuning number study.
4.7 Personalized Model Visualization
pFedHR can automatically and dynamically generate candidates for clients using the proposed heterogeneous model re-assembly techniques in Section 3.1.2. We visualize the generated models for a client at two different epochs ( $t$  and  $t'$ ) to make a comparison in Figure 7. We can observe that at epoch  $t$ , the layers are "[Conv2 from M1, Conv3 from M2, Conv4 from M3, Conv5 from M3, FC3 from M2]", which is significantly different from the model structure at epoch  $t'$ . These models automatically reassemble different layers from different models
Figure 7: Candidate models.
learned by the proposed pFedHR instead of using predefined structures or consensus information.
Model heterogeneity is a crucial and practical challenge in federated learning. While a few studies have addressed this problem, existing models still encounter various issues. To bridge this research gap, we propose a novel framework, named pFedHR, for personalized federated learning, focusing on solving the problem of heterogeneous model cooperation. The experimental results conducted on three datasets, under both IID and Non-IID settings, have verified the effectiveness of our proposed pFedHR framework in addressing the model heterogeneity issue in federated learning. The achieved state-of-the-art performance serves as evidence of the efficacy and practicality of our approach.
Acknowledgements This work is partially supported by the National Science Foundation under Grant No. 2212323 and 2238275.
[36] Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision, pages 73-91. Springer, 2022.
[37] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[38] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[39] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4320-4328, 2018.
Towards Personalized Federated Learning via Heterogeneous Model Reassembly (Appendix)
1 Pseudo-code of pFedHR
Algorithm 1 shows the pseudo-code of the proposed pFedHR model, which contains two main updates: the server update (lines 3-11) and the client update (lines 12 - 17). The details of reassembly candidate generation can be found in the main manuscript (Algorithm 1 on Page 5).
Algorithm 1: Algorithm Flow of pFedHR.
Input: Local data  $\mathcal{D}_1,\mathcal{D}_2,..,\mathcal{D}_N$  , the number of active clients  $B$  communication rounds  $T$  the number of local training epochs  $E_{c}$  , the number of fine-tuning epochs at the server  $E_{s}$  , cluster number  $K$    
1 for each communication round  $t = 1,2,\dots ,T$  do   
2 Randomly sample  $B$  active clients and their model parameters are denoted as  $\{\mathbf{w}_t^1,\mathbf{w}_t^2,\dots ,\mathbf{w}_t^B\}$  .   
3 Server Update   
4 for each  $n\in [1,\dots ,B]$  do   
5 Conduct layer-wise ecomposition on  $\mathbf{w}_t^n = [(\mathbf{L}_{t,1}^n,O_1^n),\dots ,(\mathbf{L}_{t,H}^b,O_E^n)];$    
6 end   
7 Conduct function-driven layer grouping with Eq. (4) to obtain  $\{\mathcal{G}_t^1,\mathcal{G}_t^2,\dots ,\mathcal{G}_t^K\}$    
8 Conduct reassembly candidate generation to obtain  $\{\mathbf{c}_t^1,\dots ,\mathbf{c}_t^M\}$  following  $\mathcal{R}_1,\mathcal{R}_2,\mathcal{R}_3$  , and  $\mathcal{R}_4$  .   
9 Conduct layer stitching on the generated candidates;   
10 Conduct similarity calculation on pairs of models with Eq. (5);   
11 Distribute the personalized models back to the clients accordingly.   
12 Client Update   
13 for each  $n\in [1,\dots ,B]$  do   
14 for each local epoch  $e$  from 1 to  $E_{c}$  do   
15 | Update  $\mathbf{w}_t^n$  with Eq. (6);   
16 end   
17 end   
18 Upload  $B$  models  $\{\mathbf{w}_t^1,\mathbf{w}_t^2,\dots ,\mathbf{w}_t^B\}$  back to the server;   
19 end
The proposed pFedHR is implemented in Pytorch 2.0.1 and runs on NVIDIA A100 with CUDA version 12.0 on a Ubuntu 20.04.6 LTS server. The hyperparameter  $\lambda$  in Eq. (6) is 0.2. We use Adam as the optimizer. The learning rate of the local client learning and the server fine-tuning learning rate are both equal to 0.001.
2.1 Model Details
In our experiments, we have 4 CNN models with different complexity. The details are shown as follows. In each convolutional NN sequential block, there are 1 convolutional layer, a max pooling layer, and a ReLU function.
M1: Cov1:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov2:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → FC1:{Linear → ReLU} → Dropout→ FC2:Linear.
M2: Cov1:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov2:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov3:{Conv2d (kernel size = 5) → ReLU} → FC1:{Linear → ReLU} → Dropout → FC2:Linear.
M3: Cov1:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov2:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov3:{Conv2d (kernel size = 5) → ReLU} → Cov4:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov4:{Conv2d (kernel size = 5) → ReLU} → FC1:{Linear → ReLU → Dropout} → FC2:{Linear → ReLU} → FC3:{Linear → ReLU} → FC4:Linear.
M4: Cov1:{Conv2d (kernel size = 5) → BatchNorm2d → ReLU} → Cov2:{Conv2d (kernel size = 3) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → Cov3:{Conv2d (kernel size = 3) → BatchNorm2d → ReLU} → Cov4:{Conv2d (kernel size = 5) → ReLU → MaxPool2D (kernel size = 2, stride = 2) → Dropout} → Cov5:{Conv2d (kernel size = 3) → BatchNorm2d → ReLU} → Cov6:{Conv2d (kernel size = 3) → ReLU → MaxPool2D (kernel size = 2, stride = 2)} → FC1:{Linear → ReLU → Dropout} → FC2:{Linear → ReLU} → FC3:{Linear → ReLU} → FC4:Linear.
2.2 Details of Rule  $\mathcal{R}_2$  (Operation Order)
We introduce the rule  $\mathcal{R}_2$  (operation order) in Line 11 of the algorithm in the main manuscript. We follow existing work [1, 2, 3, 4, 5] to define the following rules:
1. A CNN typically has convolutional layers, pooling layers, and fully connected layers, which requires the generated candidate to have these functional layers.
2. The typical operation order is convolution layers  $\rightarrow$  ReLU layers  $\rightarrow$  pooling layers  $\rightarrow$  fully connected layers.
2.3 Public Data Sensitivity Analysis
2.3.1 Experiment Results of Public Data Sensitivity on SVHN with 100 Clients