SlowGuess/NeurIPS_2023_OCR huggingface dataset:
How to apply DP? Applying noise (e.g., Gaussian or Laplacian) to performance-sensitive features before sharing can protect features with DP guarantee [26, 16], i.e.,  $\mathbf{x}_p = \mathbf{x}_s + \mathbf{n}$ . Consequently, the server can collect protected features  $\mathbf{x}_p$  from clients to construct a global dataset and send the dataset back to clients. To give a vivid illustration, we provide a guessing game<sup>3</sup> in Figure 1, showing 4 images. The first image represents the DP-protected performance-sensitive features of one image (A, B, or C). It is hard to identify which one of the raw images (A, B, C) the first image belongs to.
Training with DP. The protected performance-sensitive features are shared, thus, the server can construct a globally shared dataset. Using the global dataset, clients can train local classifier  $f(\cdot ;\phi_k)$  parameterized by  $\phi_{k}$  with the private and shared data:
$$
\min  _ {\phi_ {k}} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim P \left(X _ {k}, Y _ {k}\right)} \ell \left(f (\mathbf {x}; \phi_ {k}), \mathbf {y}\right) + \mathbb {E} _ {\left(\mathbf {x} _ {p}, \mathbf {y}\right) \sim P \left(\mathbf {h} \left(X _ {k}\right), Y _ {k}\right)} \ell \left(f \left(\mathbf {x} _ {p}; \phi_ {k}\right), \mathbf {y}\right), \tag {10}
$$
where  $f(\cdot ;\phi_k)$  is trained for model aggregation,  $(\mathbf{x}_p,\mathbf{y})$  are protected performance-sensitive features that are collected from all clients, and  $\mathbf{h}(X_k)\coloneqq \mathbf{z}(X_k) + n$  with noise  $n$ . Pseudo-code of how to apply FedFed are listed in Appendix B.
DP guarantee. Following DP-SGD [27], we employ the idea of the  $\ell_2$ -norm clipping relating to the selection of the noise level  $\sigma$ . Specifically, we realize the constraint in Eq. (9) as follows:  $\| \mathbf{z}(\mathbf{x};\theta)\| _2\leq \rho \| \mathbf{x}\|$ , with  $0 < \rho < 1$  (i.e.,  $\| \mathbf{x}_s\| \leq \rho \| \mathbf{x}\|$ ). For aligning analyses with conventional DP guarantee, we let the random mechanism on performance-robust features  $\mathbf{x}_r$  be  $\mathbf{x}_r + n,n\sim \mathcal{N}(0,\sigma_r^2\mathbf{I})$ . Similar for performance-sensitive features, we have  $\mathbf{x}_s + n,n\sim \mathcal{N}(0,\sigma_s^2\mathbf{I})$ . Since  $\mathbf{x}_r$  is kept locally, an adversary views nothing (or random data without any entropy). This keeps equal to adding a sufficiently large noise, i.e.,  $\sigma_r = \infty$  on  $\mathbf{x}_r$  to make it random enough. Consequently, FedFed can provide a strong privacy guarantee. Moreover, for each client, we show that FedFed requires a relatively small noise level  $\sigma$  for achieving identical privacy, which is given in Theorem 3.3 (detailed analysis is left in Appendix D).
Theorem 3.3. Let  $\sigma$  be the noise scale for FedFed and  $\sigma'$  be the noise scale for sharing raw  $\mathbf{x}$ . Given identical  $\epsilon, \delta$ , we attain  $\sigma < \sigma'$  such that  $\sigma \propto \| \mathbf{x}_s \|^2$ .
Besides each client, we give the privacy analysis for the FL system paired with the proposed FedFed.
Theorem 3.4 (Composition of FedFed). For  $k$  clients with  $(\epsilon, \delta)$ -differential privacy, FedFed satisfies  $(\hat{\epsilon}_{\hat{\delta}}, 1 - (1 - \hat{\delta}) \Pi_{i} (1 - \delta_{i}))$ -differential privacy,
<sup>3</sup>We reveal the answer in Appendix F.2
Table 1: Top-1 accuracy with(without) FedFed under different heterogeneity degree, local epochs, and clients number on CIFAR-10.
centralized training ACC = 95.48% w/(w/o) FedFed
ACC↑	Gain↑	Round ↓	Speedup↑	ACC↑	Gain↑	Round ↓	Speedup↑
α = 0.1, E = 1, K = 10 (Target ACC =79%)	α = 0.05, E = 1, K = 10 (Target ACC =69%)
FedAvg	92.34(79.35)	12.99↑	39(284)	×7.3(×1.0)	90.02(69.36)	20.66↑	44(405)	×9.2(×1.0)
FedProx	92.12(83.06)	9.06↑	62(192)	×4.6(×1.5)	90.73(78.98)	11.75↑	48(203)	×8.4(×2.0)
SCAFFOLD	89.66(83.67)	5.99↑	34(288)	×8.4(×1.0)	81.04(37.87)	43.17↑	37(None)	×10.9(None)
FedNova	92.23(80.95)	11.28↑	33(349)	×8.6(×0.8)	91.21(65.08)	26.13↑	32(None)	×12.7(None)
α = 0.1, E = 5, K = 10 (Target ACC =85%)	α = 0.1, E = 1, K = 100 (Target ACC =49%)
FedAvg	93.24(83.79)	9.45 ↑	17(261)	×15.4(×1.0)	84.06(49.72)	34.34↑	163(967)	×5.9(×1.0)
FedProx	91.39(82.32)	8.97 ↑	76(None)	×3.4(None)	87.01(50.01)	37.00↑	127(831)	×7.6(×1.2)
SCAFFOLD	92.34(85.31)	7.03 ↑	15(66)	×17.0(×4.0)	79.60(52.76)	26.84↑	171(627)	×5.7(×1.5)
FedNova	92.85(86.21)	6.64 ↑	31(120)	×8.4(×2.2)	86.64(45.97)	40.67↑	199(None)	×4.9(None)
"Round": communication rounds arriving at the target accuracy. "None": not attaining the target accuracy. Results in " $(\cdot)$ " indicates training without FedFed.
Table 2: Top-1 accuracy with(without) FedFed under different heterogeneity degree, local epochs, and clients number on FMNIST.
centralized training ACC = 95.64% w/(w/o) FedFed
ACC↑	Gain↑	Round ↓	Speedup↑	ACC↑	Gain↑	Round ↓	Speedup↑
α = 0.1, E = 1, K = 10 (Target ACC =86%)	α = 0.05, E = 1, K = 10 (Target ACC =78%)
FedAvg	92.34(86.73)	5.61↑	14(121)	×8.6(×1.0)	90.69(78.34)	12.35↑	16(420)	×26.3(×1.0)
FedProx	92.09(87.73)	4.36 ↑	32(129)	×2.1(×0.9)	89.68(82.03)	7.65↑	16(44)	×26.3(9.5)
SCAFFOLD	91.62(86.31)	3.89↑	29(147)	×4.2(×0.8)	80.48(76.63)	3.85↑	139(None)	×6.2(None)
FedNova	92.39(87.03)	5.36↑	18(88)	×6.7(×1.4)	89.72(79.98)	9.74↑	16(531)	×26.3(×0.8)
α = 0.1, E = 5, K = 10 (Target ACC =87%)	α = 0.1, E = 1, K = 100 (Target ACC =90%)
FedAvg	92.26(87.43)	4.83↑	19(276)	×14.5(×1.0)	92.71(90.21)	2.5↑	243(687)	×2.8(×1.0)
FedProx	91.79(86.63)	5.16↑	34(None)	×8.1(None)	92.82(90.17)	2.65↑	284(501)	×2.4(×1.4)
SCAFFOLD	92.92(87.21)	5.71↑	8(112)	×34.5(×2.5)	90.28(84.87)	5.41↑	952(None)	×0.7(None)
FedNova	92.30(87.67)	4.63↑	8(187)	×34.5(×1.5)	91.04(85.32)	5.72↑	589(None)	×1.2(None)
where,  $\hat{\epsilon}_{\hat{\delta}} = \min \left\{\frac{k\rho\sqrt{R\log(1 / \delta)}}{\sigma_s},\frac{\rho\sqrt{R\log(1 / \delta)}}{\sigma_s}\left[\frac{e^{\rho\sqrt{R\log(1 / \delta)} / \sigma_s} - 1}{e^{\rho\sqrt{R\log(1 / \delta)} / \sigma_s} + 1} k\right. + \epsilon (2k\log (e + \right.$ $(\rho \sqrt{kR\hat{\delta}\log(1 / \delta)}) / (\sigma_s^2\hat{\delta}))^{1 / 2}],\frac{\rho\sqrt{R\log(1 / \delta)}}{\sigma_s}\cdot [\frac{e^{\rho\sqrt{R\log(1 / \delta)} / \sigma_s} - 1}{e^{\rho\sqrt{R\log(1 / \delta)} / \sigma_s} + 1} k + \sqrt{2k\log(1 / \hat{\delta})} ]\} .$
Atop composition theorem [28], Theorem 3.4 shows that FedFed is able to provide a strong privacy guarantee for FL systems.
We organize this section in the following aspects: a) the main experimental settings that we primarily followed (Sec 4.1); b) the main results and observations of applying FedFed to existing popular FL algorithms (Sec 4.2); c) the sensitivity study of hyper-parameters on effecting model performance (Sec 4.3); and d) empirical analysis of privacy with two kinds of attacks (Sec 4.4).
Federated Non-IID Datasets. Following previous works [10, 29], we conduct experiments over CIFAR-10, CIFAR100 [30], Fashion-MNIST(FMNIST) [31], and SVHN [32]. Following [5], we employ latent Dirichlet sampling (LDA) [33] to simulate Non-IID distribution. In our experiments, following [5, 34], we set  $\alpha = 0.1$  and  $\alpha = 0.05$  by LDA. Besides, we evaluate FedFed with other two widely adopted partition strategies:  $\# C = k$  [4, 5] and Subset method [9], mainly including label skew and quantity skew (Appendix F.1 shows more detail of data distribution).
Table 3: Top-1 accuracy with(without) FedFed under different heterogeneity degree, local epochs, and clients number on CIFAR-100.
centralized training ACC = 75.56% w/(w/o) FedFed
ACC↑	Gain↑	Round ↓	Speedup↑	ACC↑	Gain↑	Round ↓	Speedup↑
α = 0.1, E = 1, K = 10 (Target ACC =67%)	α = 0.05, E = 1, K = 10 (Target ACC =61%)
FedAvg	69.64(67.84)	1.8↑	283(495)	×1.7 (×1.0)	68.49(62.01)	6.48↑	137(503)	×3.7(×1.0)
FedProx	70.02(65.34)	4.68 ↑	233(None)	×2.1(None)	69.03(61.29)	7.74↑	141(485)	×3.6(1.0)
SCAFFOLD	70.14(67.23)	2.91↑	198(769)	×2.5(×0.6)	69.32(58.78)	10.54↑	81(None)	×6.2(None)
FedNova	70.48(67.98)	2.5↑	147(432)	×3.4(×1.1)	68.92(60.53)	8.39↑	87(None)	×5.8(None)
α = 0.1, E = 5, K = 10 (Target ACC =69%)	α = 0.1, E = 1, K = 100 (Target ACC =48%)
FedAvg	70.96(69.34)	1.62↑	79(276)	×3.5(×1.0)	60.58(48.21)	12.37↑	448(967)	×2.2(×1.0)
FedProx	69.66(62.32)	7.34↑	285(None)	×1.0(None)	67.69(48.78)	18.91↑	200(932)	×4.8(×1.0)
SCAFFOLD	70.76(70.23)	0.53↑	108(174)	×2.6(×1.6)	66.67(51.03)	15.64↑	181(832)	×5.3(×1.2)
FedNova	69.98(69.78)	0.2↑	89(290)	×3.1(×1.0)	67.62(48.03)	19.59↑	198(976)	×4.9(×1.0)
Table 4: Top-1 accuracy with(without) FedFed under different heterogeneity degree, local epochs, and clients number on SVHN.
centralized training ACC = 96.56% w/(w/o) FedFed
ACC↑	Gain↑	Round ↓	Speedup↑	ACC↑	Gain↑	Round ↓	Speedup↑
α = 0.1, E = 1, K = 10 (Target ACC =88%)	α = 0.05, E = 1, K = 10 (Target ACC =82%)
FedAvg	93.21(88.34)	4.87↑	105(264)	×2.5(×1.0)	93.49(82.76)	10.73↑	194(365)	×1.9(×1.0)
FedProx	91.80(86.23)	5.574↑	233(None)	×1.1(None)	93.21(79.43)	13.78↑	37(None)	×9.9(None)
SCAFFOLD	88.41(80.12)	8.29↑	357(None)	×0(None)	90.27(75.87)	14.4↑	64(None)	×5.7(None)
FedNova	92.98(89.23)	3.75↑	113(276)	×2.3(×1.0)	93.05(82.32)	10.73↑	37(731)	×9.9(×0.5)
α = 0.1, E = 5, K = 10 (Target ACC =87%)	α = 0.1, E = 1, K = 100 (Target ACC =89%)
FedAvg	93.77(87.24)	6.53↑	105(128)	×1.2(×1.0)	91.04(89.32)	1.72↑	763(623)	×0.8(×1.0)
FedProx	91.15(77.21)	13.94↑	142(None)	×0.9(None)	91.41(88.76)	2.65↑	733(645)	×0.8(×1.0)
SCAFFOLD	93.78(80.98)	12.8↑	20(None)	×6.4(None)	92.73(88.32)	4.41↑	507(687)	×1.2(×0.9)
FedNova	93.66(89.03)	4.63↑	52(177)	×2.5(×0.7)	84.05(81.87)	2.18↑	None(None)	None(None)
Models, Metrics and Baselines. We use ResNet-18 [35] both in the feature distillation and classifier in FL. We evaluate the model performance via two popular metrics in FL: a) communication rounds and b) best accuracy. Typically, the target accuracy is set to be the best accuracy of vanilla FedAvg. As a plug-in approach, we apply FedFed to prevailing existing FL algorithms, such as FedAvg [4], FedProx [6], SCAFFOLD [7], and FedNova [22] to compare the efficiency of our method. More experimental settings and details are listed in Appendix B.1.
Generative Model Selection. We consider two generative models in this paper to distill raw data, i.e., Resnet Generator, a kind of vanilla generator used in many works [36, 37], and  $\beta$ -VAE [38], an encoder-decoder structure by variational inference. We verify the effectiveness of FedFed with different generative models. According to results shown in Figure 2 (a), we observe that  $\beta$ -VAE gets better performance in FedFed. Thus, we mainly report the results of  $\beta$ -VAE in the rest.
Result Analysis. The experimental results on CIFAR-10, CIFAR-100, FMNIST, and SVHN are shown respectively in Tables 1, 2, 3, and 4. It can be seen that the proposed method can consistently and significantly improve model accuracy under various settings. Moreover, FedFed can promote the convergence rate of different algorithms<sup>4</sup>. Specifically, FedFed brings significant performance gain in CIFAR-10 under the  $K = 100$  setting shown in Table 1 and also notably speeds up the convergence rate in FMNIST under the  $\alpha = 0.05$ ,  $E = 1$ ,  $K = 10$  setting. However, attributable to the original FL methods almost reaching a performance bottleneck, FedFed achieves limited performance gain in SVHN and FMNIST. FedFed also has limited improvement in CIFAR-100. A possible reason
4More supporting figures of convergence rate are moved to appendix F.4
Figure 2: More facts of FedFed. (a) Convergence rate of different generative models (i.e.,  $\beta$ -VAE[38] and ResNet generator[37]) compared with vanilla FedAvg. (b) Test accuracy and convergence rate on different federated learning algorithms with or without FedFed under  $\alpha = 0.1$ ,  $E = 1$ ,  $K = 100$ . (c) Test accuracy on FMNIST with different noise level  $\sigma_s^2$  in Theorem 3.3, obtaining various privacy  $\epsilon$  (lower  $\epsilon$  is preferred). As the noise increased, the level of protection gradually increased.
Table 5: Experiment results of different Non-IID partition methods on CIFAR-10 with 10 clients.
Partition Method	Test Accuracy w/(w/o) FedFed
FedAvg	FedProx	SCAFFOLD	FedNova
α = 0.1	92.34(79.35)	92.12(83.06)	89.66(83.67)	92.23(80.95)
#C = 2	89.23/42.54	88.17/58.45	84.43/46.82	89.54/45.42
Subset	90.29/39.53	89.11/32.87	89.92/35.26	90.00/38.52
Table 6: Experiment results with different noise adding on CIFAR-10.
Noise Type	Test Accuracy on Different Noise with FedFed
FedAvg	FedProx	SCAFFOLD	FedNova
Gaussian Noise	92.34	92.12	89.66	92.23
Laplacian Noise	92.30	91.36	91.24	91.73
is that existing methods can achieve performance comparable to centralized training. Moreover, we conducted an additional experiment involving sharing the full data with DP protection. The results, presented in Appendix F.3, indicate that sharing the full data with DP protection leads to a degradation in the performance of the FL system. This degradation occurs because protecting the full data necessitates a relatively large noise to achieve the corresponding protection strength required to safeguard performance-sensitive features.
Surprising Observations. We find that training among 100 clients in CIFAR-10 and CIFAR100 reaches a significant improvement (e.g., at most  $40.67\%$ ). A possible reason is that the missed data knowledge can be well replenished by FedFed. Moreover, all methods paired with FedFed under various settings can achieve similar prediction accuracies, demonstrating that FedFed endows FL models robustness against data heterogeneity. Table 5 shows that two kinds of heterogeneity partition cause more performance decline than LDA ( $\alpha = 0.1$ ). Yet, FedFed attains noteworthy improvement, indicating the robustness against Non-IID partition.
DP Noise. To explore the relationship between privacy level  $\epsilon$  and prediction accuracy, we conduct experiments with different noise levels  $\sigma_s^2$ . As shown in Figure 2(c), the prediction accuracy decreases with increasing noise level (more results in Appendix F.5). To verify whether the FedFed is robust against the selection of noise, we also consider Laplacian noise in applying DP to privacy protection. The results of Laplacian noise are reported in Table 6, demonstrating the robustness of FedFed.
Hyper-parameters. We further evaluate the robustness of FedFed against hyper-parameters. During the feature distillation process, as the constraint parameter  $\rho$  in Eq. 9 decreases, the DP strength to protect performance-sensitive features decreases, and the performance of the global model decreases, owing to the less information contained in performance-sensitive features.
4.4 Privacy Verification
Besides the theoretical analysis, we provide empirical analysis to support the privacy guarantee of FedFed. We wonder whether the globally shared data can be inferred by some attacking methods. Thus, we resort to model inversion attack [39], widely used in the literature to reconstruct data. The results<sup>5</sup> in Figure 3 (a) and (b) indicate that FedFed could protect globally shared data. We also conduct another model inversion attack [40] and the results can be found in Appendix E.1.
5 More details and results can be found in Appendix E.1.
(b)
Figure 3: Attack results on FedFed. (a) shows the protected data  $\mathbf{x}_p$ . (b) reports the model inversion attacked data. (c) shows results of membership inference attacks: the green star represents the recall for FedFed, while the blue stars show the searching process varying with  $\sigma_s^2$  for sharing raw data.
Additionally, we perform membership inference attack [41] to illustrate the difference between FedFed and sharing all data features with DP protection. The results illustrated in Figure 3 (c) show that noise level  $\sigma^2 = 0.3$  over raw data  $\mathbf{x}$  can achieve similar protection as noise  $\sigma_s^2 = 0.15$  over globally shared data  $\mathbf{x}_p$ , which aligns with Theorem 3.3. More details can be found in Appendix E.2
Federated Learning (FL) models typically perform poorly when meeting severe Non-IID data [2, 7]. To tackle data heterogeneity, advanced works introduce various learning objectives to calibrate the updated direction of local training from being far away from the global model, e.g., FedProx[6], FedIR [42], SCAFFOLD [7], and MOON [43]. Designing model aggregation schemes like FedAvgM [33], FedNova [22], FedMA [44], and FedBN [45] shows the efficacy on heterogeneity mitigation. Another promising direction is the information-sharing strategy, which mainly focuses on synthesizing and sharing clients' information to mitigate heterogeneity [9, 46, 47]. To avoid exposing privacy caused by the shared data, some methods utilize the statistics of data [48], representations of data [12], logits [49, 10], embedding [50]. However, advanced attacks pose potential threats to methods with data-sharing strategies [51].
FedFed is inspired by [34], where pure noise is shared across clients to tackle data heterogeneity. In this work, we relax the privacy concern by sharing partial features in the data with DP protection. In addition, our work is technically similar to an adversarial learning approach [52]. Our method distinguishes by defining various types of features and delving into the exploration of data heterogeneity within FL. More discussion can be found in Appendix G.
In this work, we propose a novel framework called FedFed to tackle data heterogeneity in FL by employing the promising information-sharing approach. Our work extends the research line of constructing shareable information of clients by proposing to share partial features in data. This shares a new route of improving training performance and maintaining privacy. Furthermore, FedFed has served as a source of inspiration for a new direction focused on improving performance in open-set scenarios [53]. Another avenue of exploration involves deploying FedFed in other real-time FL application scenarios, such as recommendation systems [54] and healthcare system [55], to uncover its potential benefits.
**Limitation.** In reality, limited local hardware resources may limit the power of FedFed, since FedFed introduces some extra overheads like communication and storage overheads. We leave it as future works to explore a hardware-friendly version or real-world application [56, 57]. Additionally, FedFed raises potential privacy concerns that we leave as a future research exploration, such as integrating cryptography [58, 59]. We anticipate that our work will inspire further investigations to comprehensively evaluate the privacy risks associated with information-sharing strategies aimed at mitigating data heterogeneity.
We thank the reviewers for their valuable comments. Hao Peng was supported by the National Key R&D Program of China through grant 2021YFB1714800, NSFC through grants 62002007, U21B2027, 61972186, and 62266027, S&T Program of Hebei through grant 20310101D, Natural Science Foundation of Beijing Municipality through grant 4222030, and the Fundamental Research Funds for the Central Universities, Yunnan Provincial Major Science and Technology Special Plan Projects through grants 202103AA080015, 202202AD080003 and 202303AP140008, General Projects of Basic Research in Yunnan Province through grant 202301AS070047. Yonggang Zhang and Bo Han were supported by the NSFC Young Scientists Fund No. 62006202, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, CCF-Baidu Open Fund, HKBU Faculty Niche Research Areas No. RC-FNRA-IG/22-23/SCI/04, and HKBU CSD Departmental Incentive Scheme. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. Xinmei Tian was supported in part by NSFC No. 62222117.
[1] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210, 2021.
[2] Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering, 2021.
[3] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[4] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[5] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965-978. IEEE, 2022.
[6] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
[7] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[8] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261. PMLR, 2019.
[9] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
[10] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.
[11] Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint arXiv:2008.04489, 2020.
A Overview of FedFed Framework
Figure 4: Overview of FedFed
Figure 5: The Pipeline of Feature Distillation
The training process of the federated learning system with FedFed can be summarized into four phases. Firstly, every participant of FL distils the local private raw data  $\mathbf{x}$  and decouples  $\mathbf{x}$  into performance-robust features  $\mathbf{x}_r$  and performance-sensitive features  $\mathbf{x}_s$  during the feature distillation phase. Then all clients send  $\mathbf{x}_p$  to construct a global dataset,  $\mathbf{x}_s$  with DP protection and keep  $\mathbf{x}_r$  locally. In the local training phase, the selected clients sample a subset from the globally shared dataset and update the local model with local raw data  $\mathbf{x}$  and sampled information  $\mathbf{x}_p$  from server. Finally, clients upload the latest local model and aggregate a global model based on a certain aggregation algorithm, like the weighted average strategy in FedAvg. Figure 4 shows the overview of FedFed framework, while Figure 5 depicts its pipeline of feature distillation. The loss function of feature distillation in Figure 5 can be formulated as Eq (9).
B Federated Learning Algorithms with FedFed
B.1 Implementation Details
We list all relevant parameters in this paper in Table 7. We fine-tune learning rate in the set of  $\{0.0001, 0.001, 0.01, 0.1\}$  and report the best results and corresponding learning rate. Whereas, we use 0.01 as the learning rate. In our work, we deploy FedFed on SCAFFOLD and set the learning rate  $\eta_{k}$  to 0.0001 for the SVHN dataset. Furthermore, we use  $\eta_{k} = 0.001$  in SVHN while FedNove is deployed with our method.