SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\mathcal {D}} (h ^ {g}) \leq \mathcal {L} _ {\tilde {\mathcal {D}} _ {i}} (h ^ {g}) + \sqrt {\frac {4}{m} (d \log \frac {2 e m}{d} + \log \frac {4}{\delta})} + d _ {\mathcal {H}} (\tilde {\mathcal {U}} _ {i} ^ {g}, \tilde {\mathcal {U}} ^ {g}) + \lambda_ {i},
$$
where  $\mathcal{L}_{\hat{\mathcal{D}}_i}$  is the empirical loss on  $\mathcal{D}_i$ ,  $e$  is the base of the natural logarithm, and  $d_{\mathcal{H}}(\cdot ,\cdot)$  is the  $\mathcal{H}$ -divergence between two distributions.  $\lambda_i\coloneqq \min_{h^g}\mathcal{L}_\mathcal{D}(h^g) + \mathcal{L}_{\mathcal{D}_i}(h^g)$ ,  $\tilde{\mathcal{U}}_i^g\subseteq \mathcal{Z}$ ,  $\tilde{\mathcal{U}}^g\subseteq \mathcal{Z}$ , and  $d_{\mathcal{H}}(\tilde{\mathcal{U}}_i^g,\tilde{\mathcal{U}}^g)\leq d_{\mathcal{H}}(\tilde{\mathcal{U}}_i,\tilde{\mathcal{U}})$ .  $\tilde{\mathcal{U}}_i^g$  and  $\tilde{\mathcal{U}}^g$  are the induced distributions of  $\mathcal{U}_i$  and  $\mathcal{U}$  under  $\mathcal{F}^g$ , respectively.  $\tilde{\mathcal{U}}_i$  and  $\tilde{\mathcal{U}}$  are the induced distributions of  $\mathcal{U}_i$  and  $\mathcal{U}$  under  $\mathcal{F}$ , respectively.  $\mathcal{F}$  is the feature extraction function in the original FedAvg without DBE.
As shown in Figure 2, given any  $\pmb{x}_i$  on client  $i$ , one can obtain  $\pmb{z}_i$  via  $\mathcal{F}$  in original FedAvg or obtain  $\pmb{z}_i^g$  via  $\mathcal{F}^g$  in FedAvg+DBE. With  $d_{\mathcal{H}}(\tilde{\mathcal{U}}_i^g,\tilde{\mathcal{U}}^g)\leq d_{\mathcal{H}}(\tilde{\mathcal{U}}_i,\tilde{\mathcal{U}})$  holds, we can achieve a lower generalization bound in local-to-global knowledge transfer than traditional FL, thus training a better global feature extractor to produce representations with higher quality over all labels. A small gap between the local domain and global domain in  $\mathcal{Z}$  promotes the knowledge transfer from clients to the server [82, 92, 94].
4.4.2 Global-To-Local Knowledge Transfer
The global-to-local knowledge transfer focuses on the assistance role of the global model parameters for facilitating local training, i.e., the transfer ability from  $\mathcal{D}$  to  $\mathcal{D}_i$ . After the client receives the global model and equips it with PRBM, for the second level of representation (i.e.,  $z_i$ ), we have
Corollary 2. Let  $\mathcal{D}_i, \mathcal{D}, \mathcal{F}^g$ , and  $\lambda_i$  defined as in Corollary 1. Given a translation transformation function  $PRBM: \mathcal{Z} \mapsto \mathcal{Z}$  that shares between  $\mathcal{D}_i$  and virtual  $\mathcal{D}$ , a random labeled sample of size  $m$  generated by applying  $\mathcal{F}'$  to a random sample from  $\mathcal{U}_i$  labeled according to  $c^*$ ,  $\mathcal{F}' = PRBM \circ \mathcal{F}^g: \mathcal{X} \mapsto \mathcal{Z}$ , then for every  $h' \in \mathcal{H}$ , with probability at least  $1 - \delta$ :
$$
\mathcal {L} _ {\mathcal {D} _ {i}} \left(h ^ {\prime}\right) \leq \mathcal {L} _ {\hat {\mathcal {D}}} \left(h ^ {\prime}\right) + \sqrt {\frac {4}{m} \left(d \log \frac {2 e m}{d} + \log \frac {4}{\delta}\right)} + d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} ^ {\prime}, \tilde {\mathcal {U}} _ {i} ^ {\prime}\right) + \lambda_ {i},
$$
where  $d_{\mathcal{H}}(\tilde{\mathcal{U}}', \tilde{\mathcal{U}}_i') = d_{\mathcal{H}}(\tilde{\mathcal{U}}^g, \tilde{\mathcal{U}}_i^g) \leq d_{\mathcal{H}}(\tilde{\mathcal{U}}, \tilde{\mathcal{U}}_i) = d_{\mathcal{H}}(\tilde{\mathcal{U}}_i, \tilde{\mathcal{U}})$ .  $\tilde{\mathcal{U}}'$  and  $\tilde{\mathcal{U}}_i'$  are the induced distributions of  $\mathcal{U}$  and  $\mathcal{U}_i$  under  $\mathcal{F}'$ , respectively.
Given  $\boldsymbol{x}_i$  on client  $i$ , we can obtain  $\boldsymbol{z}_i$  via  $\mathcal{F}'$  in FedAvg+DBE.  $h^g = h' \circ \mathrm{PRBM}$ , so PRBM does not influence the value of  $d_{\mathcal{H}}(\cdot, \cdot)$  for the pair of  $h^g$  and  $h'$  (see Appendix B.3), then we have  $d_{\mathcal{H}}(\tilde{\mathcal{U}}', \tilde{\mathcal{U}}_i') = d_{\mathcal{H}}(\tilde{\mathcal{U}}^g, \tilde{\mathcal{U}}_i^g)$ . The inequality  $d_{\mathcal{H}}(\tilde{\mathcal{U}}', \tilde{\mathcal{U}}_i') \leq d_{\mathcal{H}}(\tilde{\mathcal{U}}, \tilde{\mathcal{U}}_i)$  shows that the information aggregated on the server can be more easily transferred to clients with our proposed DBE than FedAvg. We train PRBM on the local loss and preserve it locally, so the local feature extractors can generate representations suitable for clients' personalized tasks. According to Corollary 1 and Corollary 2, adding DBE facilitates the bi-directional knowledge transfer in each iteration, gradually promoting global and local model learning as the number of iterations increases.
4.5 Negligible Additional Communication and Computation Overhead
DBE only modifies the local training, so the downloading, uploading, and aggregation processes in FedAvg are unaffected. In FedAvg+DBE, the communication overhead per iteration is the same as FedAvg but requires fewer iterations to converge (see Appendix D). Moreover, PRBM only introduces  $K$  additional trainable parameters, and the MSE value in the parameterless MR is computed for two representations of  $K$  dimension.  $K$  is the representation space dimension, typically a smaller value than the dimension of data inputs or model parameters [8, 83]. Thus, DBE introduces no additional communication overhead and negligible computation overhead for local training in any iteration.
4.6 Privacy-Preserving Discussion
Compared to FedAvg, using DBE requires client  $i$  to upload one client-specific mean  $\bar{z}_i^g$  (one  $K$ -dimensional vector) to the server only once before FL, which solely captures the magnitude of the
mean value for each feature dimension within the context of the given datasets and models. Thanks to this particular characteristic, as shown in Section 5.1.4, the performance of FedAvg+DBE can be minimally affected while enhancing its privacy-preserving capabilities by introducing proper Gaussian noise with a zero mean to  $\bar{z}_i^g$  during the initialization phase.
Datasets and models. Following prior FL approaches [14, 20, 45, 50, 56], we use four public datasets for classification problems in FL, including three CV datasets: Fashion-MNIST (FM-NIST) [77], Cifar100 [42], and Tiny-ImageNet (100K images with 200 labels) [19], as well as one NLP dataset: AG News [91]. For three CV datasets, we adopt the popular 4-layer CNN by default following FedAvg, which contains two convolution layers (denoted by CONV1 and CONV2) and two fully connected layers (denoted by FC1 and FC2). Besides, we also use a larger model ResNet-18 [29] on Tiny-ImageNet. For AG News, we use the famous text classification model fastText [36].
Statistically heterogeneous scenarios. There are two widely used approaches to construct statistically heterogeneous scenarios on public datasets: the pathological setting [56, 66] and practical setting [45, 50]. For the pathological setting, disjoint data with 2/10/20 labels for each client are sampled from 10/100/200 labels on FMNIST/Cifar100/Tiny-ImageNet with different data amounts. For the practical setting, we sample data from FMNIST, Cifar100, Tiny-ImageNet, and AG News based on the Dirichlet distribution [50] (denoted by  $Dir(\beta)$ ). Specifically, we allocate a  $q_{c,i}$  ( $q_{c,i} \sim Dir(\beta)$ ) proportion of samples with label  $c$  to client  $i$ , and we set  $\beta = 0.1/\beta = 1$  by default for CV/NLP tasks following previous FL approaches [50, 75].
Implementation Details. Following pFedMe and FedRoD, we have 20 clients and set client participating ratio  $\rho = 1$  by default unless otherwise stated. We measure the generic representation quality across clients and evaluate the MDL [65, 74] of representations over all class labels. To simulate the common FL scenario where data only exists on clients, we split the data among each client into two parts: a training set (75% data) and a test set (25% data). Following pFedMe, we evaluate pFL methods by averaging the results of personalized models on the test set of each client and evaluate traditional FL methods by averaging the results of the global model on each client. Following FedAvg, we set the batch size to 10 and the number of local epochs to 1, so the number of local SGD steps is  $\left\lfloor \frac{n_i}{10} \right\rfloor$  for client  $i$ . We run three trials for all methods until empirical convergence on each task and report the mean value. For more details and results (e.g., fine-tuning FedAvg on new participants and a real-world application), please refer to Appendix D.
5.1 Experimental Study for Adding DBE
5.1.1 How to Split the Model?
A model is split into a feature extractor and a classifier, but there are various ways for splitting, as each layer in a deep neural network (DNN) outputs a feature representation and feeds it into the next layer [8, 44] [8, 44]. We focus on inserting DBE between the feature extractor and the classifier, but which splitting way is the best for DBE? Here we answer this question by comparing the results regarding MDL and accuracy when the model is split at each layer in the popular 4-layer CNN. We show the MDL of the representation  $\boldsymbol{z}_i^g$  outputted by the prepositive layer of DBE (with underline here) and show MDL of  $\boldsymbol{z}_i$  for other layers. Low MDL and high accuracy indicate superior generalization ability and superior personalization ability, respectively.
In Table 1, the generic representation quality is improved at each layer for all splitting ways, which shows that no matter how the model is split, DBE can enhance the generalization ability of the global feature extractor. Among these splitting ways, assigning all FC layers to the classifier, i.e.,  $\mathrm{CONV2\rightarrow DBE\rightarrow FC1}$ , achieves almost the lowest MDL and highest accuracy. Meanwhile,  $\mathrm{FC1\rightarrow DBE}$ $\rightarrow \mathrm{FC2}$  can also achieve excellent performance with only  $4.73\%$  trainable parameters for DBE.
Since FedRep, FedRoD, and FedBABU choose the last FC layer as the classifier by default, we follow them for a fair comparison and insert DBE before the last FC layer (e.g.,  $\mathrm{FC1} \rightarrow \mathrm{DBE} \rightarrow \mathrm{FC2}$ ). In Table 1, our  $\mathrm{FC1} \rightarrow \mathrm{DBE} \rightarrow \mathrm{FC2}$  outperforms FedPer, FedRep, FedRoD, FedBABU, and FedAvg with lower MDL and higher accuracy. Since feature extractors in FedPer and FedRep are locally trained to cater to personalized classifiers, they extract representations with low quality.
Table 1: The MDL (bits, ↓) of layer-wise representations, test accuracy (\%, ↑), and the number of trainable parameters (↓) in PRBM when adding DBE to FedAvg on Tiny-ImageNet using 4-layer CNN in the practical setting. We also show corresponding results for the close pFL methods. For FedBABU, “[36.82]” indicates the test accuracy after post-FL fine-tuning for 10 local epochs.
Metrics	MDL	Accuracy	Param.
CONV1→CONV2	CONV2→FC1	FC1→FC2	Logits
FedPer [3]	5143	4574	3885	4169	33.84	—
FedRep [20]	5102	4237	3922	4244	37.27	—
FedRoD [14]	5063	4264	3783	3820	36.43	—
FedBABU [61]	5083	4181	3948	3849	16.86 [36.82]	—
Original (FedAvg)	5081	4151	3844	3895	19.46	0
CONV1→DBE→CONV2	4650 (-8.48%)	4105 (-1.11%)	3679 (-4.29%)	3756 (-3.57%)	21.81 (+2.35)	28800
CONV2→DBE→FC1	4348 (-14.43%)	3716 (-10.48%)	3463 (-9.91%)	3602 (-7.52%)	47.03 (+27.57)	10816
FC1→DBE→FC2	4608 (-9.31%)	3689 (-11.13%)	3625 (-5.70%)	3688 (-5.31%)	43.32 (+23.86)	512
5.1.2 Representation Bias Eliminated for the First Level of Representation
(a) FedAvg (B). (b) FedAvg (A). (c) +DBE  $(z_i^g, \mathrm{B})$ . (d) +DBE  $(z_i^g, \mathrm{A})$ . (e) +DBE  $(z_i, \mathrm{B})$ . (f) +DBE  $(z_i, \mathrm{A})$ .
Figure 3: t-SNE visualization for representations on Tiny-ImageNet (200 labels). "B" and "A" denote "before local training" and "after local training", respectively. We use color and shape to distinguish labels and clients, respectively. Best viewed in color and zoom-in.
We visualize the feature representations using t-SNE [73] in Figure 3. Compared to the representations outputted by the feature extractor in FedAvg,  $z_{i}^{g}$  in FedAvg+DBE is no longer biased to the local data domain of each client after local training. With the personalized translation transformation PRBM,  $z_{i}$  can fit the local domain of each client either before or after local training. According to Figure 3(b), Figure 3(e) and Figure 3(f),  $z_{i}$  in FedAvg+DBE can fit the local domain better than FedAvg.
5.1.3 Ablation Study for DBE
Table 2: The MDL (bits, ↓) and test accuracy (\%, ↑) when adding DBE to FedAvg on Tiny-ImageNet using 4-layer CNN and ResNet-18 in the practical setting.
Models	4-layer CNN	ResNet-18
Components	FedAvg	+MR	+PRBM	+DBE	FedAvg	+MR	+PRBM	+DBE
MDL	3844	3643	3699	3625	3560	3460	3471	3454
Accuracy	19.46	22.21	26.70	43.32	19.45	20.85	38.27	42.98
We further study the contribution of MR and PRBM in terms of generalization and personalization abilities by applying only one of them to FedAvg. From Table 2, we find that for 4-layer CNN and ResNet-18, +DBE gives a larger improvement in both MDL and accuracy than just using MR or PRBM, which suggests that MR and PRBM can boost each other in bi-directional knowledge transfer. The contribution of MR is greater than that of PRBM in improving the generalization ability in MDL, while +PRBM gains more accuracy improvement for personalization ability than MR.
5.1.4 Privacy-Preserving Ability
Following FedPAC [70], we add Gaussian noise to client-specific means  $\bar{z}_1^g, \dots, \bar{z}_N^g$  with a scale parameter  $(s)$  for the noise distribution and perturbation coefficient  $(q)$  for the noise. Adding the unbiased noise sampled from one distribution is beneficial for representation bias elimination and can further improve the performance of DBE to some extent, as shown in Table 3. Besides, adding too much noise can also bring an accuracy decrease. However, setting  $s = 0.05$  and  $q = 0.2$  is sufficient to ensure privacy protection according to FedPCL.
Table 3: The test accuracy  $(\% ,\uparrow)$  using FedAvg+DBE on TINY in the practical setting with noise.
q = 0.2	s = 0.05
Original	s = 0.05	s = 0.5	s = 1	s = 5	q = 0.1	q = 0.5	q = 0.8	q = 0.9
43.32	44.10	44.15	43.78	36.27	43.81	44.45	43.30	41.75
5.1.5 DBE Improves Other Traditional Federated Learning Methods
Table 4: The MDL (bits, ↓) and test accuracy (%, ↑) before and after adding DBE to traditional FL methods on Cifar100, Tiny-ImageNet, and AG News in the practical setting. TINY and TINY* represent using 4-layer CNN and ResNet-18 on Tiny-ImageNet, respectively.
Metrics	MDL	Accuracy
Datasets	Cifar100	TINY	TINY*	AG News	Cifar100	TINY	TINY*	AG News
SCAFFOLD [38]	1499	3661	3394	1931	33.08	23.26	24.90	88.13
FedProx [46]	1523	3701	3570	2092	31.99	19.37	19.27	87.21
MOON [45]	1516	3696	3536	1836	32.37	19.68	19.02	84.14
FedGen [96]	1506	3675	3551	1414	30.96	19.39	18.53	89.86
SCAFFOLD+DBE	1434	3549	3370	1743	63.61	45.55	45.09	96.73
FedProx+DBE	1439	3587	3490	1689	63.22	42.28	41.45	96.62
MOON+DBE	1432	3580	3461	1683	63.26	43.43	41.10	96.68
FedGen+DBE	1426	3563	3488	1098	63.26	42.54	41.87	97.16
A large number of FL methods design algorithms based on the famous FedAvg [37, 56, 69]. Although we describe DBE based on FedAvg for example, DBE can also be applied to other traditional FL methods to improve their generalization and personalization abilities. Here, we apply DBE to another four representative traditional FL methods: SCAFFOLD [38], FedProx [46], MOON [45], and FedGen [96]. They belong to four categories: update-correction-based FL, regularization-based FL, model-split-based FL, and knowledge-distillation-based FL, respectively. In Table 4, DBE promotes traditional FL methods by at most  $-22.35\%$  in MDL (bits) and  $+32.30$  in accuracy (\%), respectively. Based on the results of Table 2 and Table 4 on Tiny-ImageNet, FedAvg+DBE achieves lower MDL and higher accuracy than close methods MOON and FedGen.
5.2 Comparison with Personalized Federated Learning Methods
5.2.1 Personalization Ability on Various Datasets
Table 5: The test accuracy  $(\% ,\uparrow)$  of pFL methods in two statistically heterogeneous settings. Cifar100† represents the experiment with 100 clients and joining ratio  $\rho = 0.5$  on Cifar100.
Settings	Pathological setting	Practical setting
FMNIST	Cifar100	TINY	FMNIST	Cifar100	Cifar100†	TINY	TINY*	AG News
Per-FedAvg [22]	99.18	56.80	28.06	95.10	44.28	38.28	25.07	21.81	87.08
pFedMe [67]	99.35	58.20	27.71	97.25	47.34	31.13	26.93	33.44	87.08
Ditto [47]	99.44	67.23	39.90	97.47	52.87	39.01	32.15	35.92	91.89
FedPer [3]	99.47	63.53	39.80	97.44	49.63	41.21	33.84	38.45	91.85
FedRep [20]	99.56	67.56	40.85	97.56	52.39	41.51	37.27	39.95	92.25
FedRoD [14]	99.52	62.30	37.95	97.52	50.94	48.56	36.43	37.99	92.16
FedBABU [61]	99.41	66.85	40.72	97.46	55.02	52.07	36.82	34.50	95.86
APFL [21]	99.41	64.26	36.47	97.25	46.74	39.47	34.86	35.81	89.37
FedFomo [89]	99.46	62.49	36.55	97.21	45.39	37.59	26.33	26.84	91.20
APPLE [52]	99.30	65.80	36.22	97.06	53.22	—	35.04	39.93	84.10
FedAvg	80.41	25.98	14.20	85.85	31.89	28.81	19.46	19.45	87.12
FedAvg+DBE	99.74	73.38	42.89	97.69	64.39	63.43	43.32	42.98	96.87
To further show the superiority of the DBE-equipped traditional FL methods to existing pFL methods, we compare the representative FedAvg+DBE with ten SOTA pFL methods, as shown in Table 5. Note that APPLE is designed for cross-silo scenarios and assumes  $\rho = 1$ . For Per-FedAvg and FedBABU, we show the test accuracy after post-FL fine-tuning. FedAvg+DBE improves FedAvg at most +47.40
on Cifar100 in the pathological setting and outperforms the best SOTA pFL methods by up to +11.36 on Cifar100† including the fine-tuning-based methods that require additional post-FL effort.
5.2.2 Personalization Ability Under Various Heterogeneous Degrees
Following prior methods [45, 50], we also evaluate FedAvg+DBE with different  $\beta$  on Tiny-ImageNet using 4-layer CNN to study the influence of heterogeneity, as shown in Table 6. Most pFL methods are specifically designed for extremely heterogeneous scenarios and can achieve high accuracy at  $\beta = 0.01$ , but some of them cannot maintain the advantage compared to FedAvg in moderate scenarios. However, FedAvg+DBE can automatically adapt to all these scenarios without tuning.
Table 6: The test accuracy  $(\% ,\uparrow)$  and computation overhead  $(\downarrow)$  of pFL methods.
Items	Heterogeneity	pFL+MR	Overhead
β = 0.01	β = 0.5	β = 5	Accuracy	Improvement	Total time	Time/iteration
Per-FedAvg [22]	39.39	21.14	12.08	—	—	121 min	3.56 min
pFedMe [67]	41.45	17.48	4.03	—	—	1157 min	10.24 min
Ditto [47]	50.62	18.98	21.79	42.82	10.67	318 min	11.78 min
FedPer [3]	51.83	17.31	9.61	41.78	7.94	83 min	1.92 min
FedRep [20]	55.43	16.74	8.04	41.28	4.01	471 min	4.09 min
FedRoD [14]	49.17	23.23	16.71	42.74	6.31	87 min	1.74 min
FedBABU [61]	53.97	23.08	15.42	38.17	1.35	811 min	1.58 min
APFL [21]	49.96	23.31	16.12	39.22	4.36	156 min	2.74 min
FedFomo [89]	46.36	11.59	14.86	29.51	3.18	193 min	2.72 min
APPLE [52]	47.89	24.24	17.79	—	—	132 min	2.93 min
FedAvg	15.70	21.14	21.71	—	—	365 min	1.59 min
FedAvg+DBE	57.52	32.61	25.55	—	—	171 min	1.60 min
5.2.3 MR Improves Personalized Federated Learning Methods
Since pFL methods already create personalized models or modules in their specific ways, applying personalized PRBM to the local model might be against their philosophy. To prevent this, we only apply the MR to pFL methods. Besides, the local training schemes (e.g., meta-learning) in Per-FedAvg, pFedMe, and APPLE are different from the simple SGD in FedAvg, which requires modification of the mean calculation in MR, so we do not apply MR to them. According to Corollary 1, MR can promote the local-to-global knowledge transfer between server and client. Therefore, pFL methods can benefit more from a better global model achieving higher accuracy on Tiny-ImageNet with the 4-layer CNN, as shown in Table 6. However, their MR-equipped variants perform worse than FedAvg+DBE (Table 5, TINY) since the representation bias still exists without using PRBM.
5.2.4 Computation Overhead
We evaluate FedAvg+DBE in total time and time per iteration on Tiny-ImageNet using ResNet-18, as shown in Table 6. The evaluation task for one method monopolizes one identical machine. FedAvg, FedBABU, and FedAvg+DBE cost almost the same and have the lowest time per iteration among these methods, but FedAvg+DBE requires less total time than FedAvg and FedBABU. Note that the fine-tuning time for FedBABU is not included in Table 6. Since pFedMe and Ditto train an additional personalized model on each client, they cost plenty of time per iteration.
Due to the naturally existing statistical heterogeneity and the biased local data domains on each client, FL suffers from representation bias and representation degeneration problems. To improve the generalization and personalization abilities for FL, we propose a general framework DBE including two modules PRBM and MR, with a theoretical guarantee. Our DBE can promote the bi-directional knowledge transfer in each iteration, thus improving both generalization and personalization abilities. Besides, we conduct extensive experiments to show the general applicability of DBE to existing FL methods and the superiority of the representative FedAvg+DBE to ten SOTA pFL methods in various scenarios.
This work was partially supported by the Program of Technology Innovation of the Science and Technology Commission of Shanghai Municipality (Granted No. 21511104700 and 22DZ1100103). This work was also supported in part by the Shanghai Key Laboratory of Scalable Computing and Systems, National Key R&D Program of China (2022YFB4402102), Internet of Things special subject program, China Institute of IoT (Wuxi), Wuxi IoT Innovation Promotion Center (2022SP-T13-C), Industry-university-research Cooperation Funding Project from the Eighth Research Institute in China Aerospace Science and Technology Corporation (Shanghai) (USCAST2022-17), Intel Corporation (UFunding 12679), and the cooperation project from Ant Group ("Wasm-enabled Managed language in security restricted scenarios").
We provide more details and results about our work in the appendices. Here are the contents:
- Appendix A: The extended version of the Related Work section in the main body.
- Appendix B: Proofs of Corollary 1 and Corollary 2.
- Appendix C: More details about experimental settings.
- Appendix D: Additional experiments (e.g., a real-world application).
- Appendix E: Broader impacts of our proposed method.
- Appendix F: Limitations of our proposed method.
- Appendix G: Data distribution visualizations for different scenarios in our experiments.
As the number of users and sensors rapidly increases with massive growing services on the Internet, the privacy concerns about private data also draw increasing attention of researchers [37, 68, 71]. Then a new distributed machine learning paradigm, federated learning (FL), comes along with the privacy-preserving and collaborative learning abilities [37, 56, 79]. Although there are horizontal FL [46, 56, 79], vertical FL [53, 63, 79], federated transfer learning[15, 51], etc., we focus on the popular horizontal FL and call it FL for short in this paper.
Traditional FL methods concentrate on learning a single global model among a server and clients, but it suffers an accuracy decrease under statistically heterogeneous scenarios, which are common scenarios in practice [47, 56, 67, 86]. Then, many FL methods propose learning personalized models (or modules) for each client besides learning the global model. These FL methods are specifically called personalized FL (pFL) methods [20, 22, 69].
A.1 Traditional Federated Learning