SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\lambda \in (0,1)$  is the momentum. We can easily see that the parameters of  $\widetilde{G}(\cdot)$  vary very little compared to those of  $G(\cdot)$  over communication rounds, if  $\lambda$  is close to 1. We further utilize synthetic data from  $\widetilde{G}(\cdot)$  as additional training data for the global model outside of  $G(\cdot)$ , mitigating the huge exploratory distribution shift induced by the large update of  $G(\cdot)$  and achieving stable updates of the global model. Particularly, we compute the Kullback-Leibler divergence between logits of the ensemble model and the global model on the synthetic data points  $\boldsymbol{s} = G(\boldsymbol{h} = o(\boldsymbol{z},\boldsymbol{y}),\boldsymbol{w})$  and  $\widetilde{\boldsymbol{s}} = \widetilde{G}(\widetilde{\boldsymbol{h}} = o(\widetilde{\boldsymbol{z}},\widetilde{\boldsymbol{y}}),\widetilde{\boldsymbol{w}})$  respectively, which is formulated as follows:
$$
\mathcal {L} _ {m d} = \mathcal {L} _ {k l} + \alpha \widetilde {\mathcal {L}} _ {k l} = K L (f (\boldsymbol {s}, \boldsymbol {\theta}), \sum_ {i \in \mathcal {S} _ {t}} \tau_ {i, y} f _ {i} (\boldsymbol {s}, \boldsymbol {\theta} _ {i})) + \alpha \cdot K L (f (\widetilde {\boldsymbol {s}}, \boldsymbol {\theta}), \sum_ {i \in \mathcal {S} _ {t}} \tau_ {i, \widetilde {y}} f _ {i} (\widetilde {\boldsymbol {s}}, \boldsymbol {\theta} _ {i})), \tag {7}
$$
where  $\alpha$  is a tunable hyper-parameter for balancing different loss items.
Dynamic Weighting and Label Sampling. So far, how to determine  $\tau_{i,y}$  and  $p(y)$  is unclear. The appropriate  $\tau_{i,y}$  and  $p(y)$  are essential for effective extraction of knowledge from local models. For clarity, we propose dynamic weighting and label sampling, i.e.,  $\tau_{i,y} = n_{i,t}^{y} / n_{\mathcal{S}_{t},t}^{y}$  and  $p(y) = n_{\mathcal{S}_{t},t}^{y} / \sum_{y\in [C]}n_{\mathcal{S}_{t},t}^{y}$ , where  $n_{\mathcal{S}_t,t}^y = \sum_{j\in [\mathcal{S}_t]}n_{j,t}^y$  and  $n_{i,t}^{y}$  denotes the number of data with label  $y$  involved in training on client  $i$  at round  $t$ . Due to space limitations, see Appendix F for their detail study and experimental justification.
Datasets. In this paper, we evaluate different methods with six real-world image classification task-related datasets, namely Fashion-MNIST [69] (FMNIST in short), SVHN [70], CIFAR-10, CIFAR-100 [71], Tiny-imageNet $^3$  and Food101 [73]. We detail the six datasets in Appendix B. To simulate data heterogeneity across clients, as in previous works [34, 37, 38], we use Dirichlet process  $Dir(\omega)$  to partition the training set for each dataset, thereby allocating local training data for each client. It is worth noting that  $\omega$  is the concentration parameter and smaller  $\omega$  corresponds to stronger data heterogeneity.
Baselines. We compare DFRD to FedFTG [48] and DENSE [49], which are the most relevant methods to our work. To verify the superiority of DFRD, on the one hand, DFRD, FedFTG and DENSE are directly adopted on the server to transfer the knowledge of the local models to a randomly initialized global model. We call them collectively data-free methods. On the other hand, they are utilized as fine-tuning methods to improve the global model's performance after computing weighted average per communication round. In this case, in each communication round, the preliminary global model is obtained using FedAvg [14] in FL with homogeneous models, whereas in FL with heterogeneous models, the PT-based methods, including HeteroFL [29], Federated Dropout [30] (FedDp for short) and FedRolex [31], are employed to get the preliminary global model.
Configurations. Unless otherwise stated, all experiments are performed on a centralized network with  $N = 10$  active clients. We set  $\omega \in \{0.01, 0.1, 1.0\}$  to mimic different data heterogeneity scenarios. To simulate model-heterogeneous scenarios, we formulate exponentially distributed budgets for a given  $N$ :  $R_{i} = \left[\frac{1}{2}\right]^{\min \{\sigma, \lfloor \frac{\rho - i}{N} \rfloor\}}(i \in [N])$ , where  $\sigma$  and  $\rho$  are both positive integers. We fix  $\sigma = 4$
<sup>3</sup>http://cs231n.stanford.edu/tiny-imagenet-200.zip
As an example, DFRD+FedRelox indicates that DFRD is used as a fine-tuning method to improve the performance of FedRelox, and others are similar. See Tables 1 and 2 for details.
and consider  $\rho \in \{5,10,40\}$ . See Appendix D for more details. Unless otherwise specified, we set  $\beta_{tran}$  and  $\beta_{div}$  both to 1 in training generator, while in robust model distillation, we set  $\lambda = 0.5$  and  $\alpha = 0.5$ . And all baselines leverage the same setting as ours. Due to space limitations, see Appendix E for the full experimental setup.
Evaluation Metrics. We evaluate the performance of different FL methods by local and global test accuracy. To be specific, for local test accuracy (L.acc for short), we randomly and evenly distribute the test set to each client and harness the test set on each client to verify the performance of local models. In terms of global test accuracy (G.acc for short), we employ the global model on the server to evaluate the global performance of different FL methods via utilizing the original test set. Note that L.acc is reported in round brackets. To ensure reliability, we report the average for each experiment over 3 different random seeds.
Table 1: Top test accuracy (\%) of distinct methods across  $\omega \in \{0.01, 0.1, 1.0\}$  on different datasets.
Alg.s	FMNIST	SVHN	CIFAR-10	CIFAR-100
ω = 1.0	ω = 0.1	ω = 0.01	ω = 1.0	ω = 0.1	ω = 0.01	ω = 1.0	ω = 0.1	ω = 0.01	ω = 1.0	ω = 0.1	ω = 0.01
DENSE	13.99±4.39(74.26±2.64)	13.96±1.33(30.27±1.10)	13.83±2.88(14.82±3.55)	16.12±2.08(52.67±1.68)	14.53±3.56(25.95±0.42)	13.06±5.40(13.09±1.64)	10.47±1.21(46.89±0.97)	10.28±0.60(22.67±0.66)	10.96±1.93(13.03±1.56)	1.22±0.10(21.68±0.23)	1.11±0.03(13.61±0.30)	1.11±0.02(8.73±0.09)
FedFTG	48.10±1.61(73.96±2.51)	44.44±1.30(30.33±1.43)	26.96±9.83(14.79±3.61)	18.58±1.41(54.75±0.63)	17.42±2.03(26.39±0.38)	15.44±4.12(13.15±1.63)	28.73±1.03(47.67±0.92)	21.97±1.18(22.95±0.64)	15.18±0.72(13.06±1.54)	1.05±0.09(22.67±0.44)	1.06±0.10(14.01±0.49)	1.11±0.09(8.82±0.21)
DFRD	65.38±0.72(74.44±2.47)	52.55±1.61(30.44±1.42)	36.63±6.52(14.87±3.64)	24.13±0.96(55.48±1.56)	18.53±0.42(26.87±0.11)	18.18±0.56(12.83±1.90)	33.82±1.81(48.33±1.17)	24.58±0.59(23.21±0.70)	20.59±2.93(14.18±1.53)	3.21±0.36(23.31±0.39)	1.55±0.13(14.15±0.32)	1.36±0.06(8.93±0.13)
FedAvg	89.71±0.10(84.57±0.63)	83.24±1.41(62.48±5.86)	60.89±5.24(29.81±13.98)	70.63±1.95(54.21±2.54)	58.12±1.26(24.66±0.42)	32.22±1.16(12.57±1.81)	67.16±0.98(49.82±1.47)	56.36±2.09(21.92±0.83)	32.92±7.40(12.37±1.71)	57.31±0.09(47.01±0.99)	49.50±0.51(25.33±0.88)	39.22±1.08(11.25±0.52)
+DENSE	90.13±0.14(84.81±0.60)	84.10±1.55(62.80±5.76)	63.36±6.24(29.37±13.83)	74.48±1.48(58.02±2.65)	62.41±2.36(26.23±0.33)	32.84±12.37(12.72±1.82)	69.73±0.60(53.46±1.58)	57.48±3.13(22.54±0.76)	35.85±7.22(12.41±1.72)	58.43±0.05(48.00±0.78)	50.84±0.60(26.19±0.90)	40.24±1.04(11.87±0.58)
+FedFTG	91.15±0.23(85.88±0.70)	84.93±1.40(63.14±4.64)	64.80±5.66(29.89±14.08)	73.91±1.78(56.99±2.65)	61.12±1.69(25.82±0.28)	37.07±6.55(12.69±1.90)	68.85±0.96(51.05±1.59)	58.17±2.02(22.39±0.66)	33.93±7.69(12.39±1.70)	58.81±1.39(48.06±1.79)	50.70±1.11(26.40±0.47)	40.50±1.17(12.15±0.61)
+DFRD	91.65±0.89(86.16±0.53)	85.66±1.21(63.79±4.28)	65.23±7.17(29.58±13.45)	77.84±1.39(60.31±2.51)	65.79±2.27(27.26±0.20)	39.25±11.71(13.96±1.48)	71.54±0.66(53.59±1.19)	61.34±1.21(23.00±0.68)	40.77±7.47(13.49±1.57)	61.49±0.28(49.97±0.58)	53.95±0.67(29.10±0.87)	43.47±1.05(14.09±1.09)
4.2 Results Comparison
Impacts of varying  $\omega$ . We study the performance of different methods at different levels of data heterogeneity on FMNIST, SVHN, CIFAR-10 and CIFAR-100, as shown in Table 1. One can see that the performance of all methods degrades severely as  $\omega$  decreases, with DFRD being the only method that is robust while consistently leading other baselines with an overwhelming margin w.r.t.  $G_{\cdot}$  acc. Also, Fig. 3 (a)-(b) show that the learning efficiency of DFRD consistently beats other baselines (see Fig. 8-9 in Appendix H for complete curves). Notably, DFRD, FedFTG and DENSE as fine-tuning methods uniformly surpass FedAvg w.r.t.  $G_{\cdot}$  acc and  $L_{\cdot}$  acc. However, their global test accuracies suffer from dramatic deterioration or even substantially worse than that of FedAvg when they act as data-free methods. We conjecture that FedAvg aggregates the knowledge of local models more effectively than data-free methods. Also, when DFRD is used to fine-tune FedAvg, it can significantly enhance the global model, yet improve the performance of local models to a less extent.
Impacts of different  $\rho$ . We explore the impacts of different model heterogeneity distributions on different methods with SVHN, CIFAR-10, Tiny-ImageNet and FOOD101. A higher  $\rho$  means more clients with  $\frac{1}{16}$ -width capacity w.r.t. the global model. From Table 2, we can clearly see that the performance of all methods improves uniformly with decreasing  $\rho$ , where DFRD consistently and overwhelmingly dominates other baselines in terms of  $G_{\cdot}acc$ . Specifically, DFRD improves  $G_{\cdot}acc$  by an average of  $11.07\%$  and  $7.54\%$  on SVHN and CIFAR-10 respectively, compared to PT-based methods (including HeteroFL, FedDp and FedRolex). Meanwhile, DFRD uniformly and significantly outstrips FedFTG and DENSE w.r.t.  $G_{\cdot}acc$ . The selected learning curve shown in Fig. 3 (c) also verifies the above statement (see Fig. 10-12 in Appendix H for more results). The above empirical results show that DFRD not only is robust to varying  $\rho$ , but also has significantly intensified effects on the global model for different PT-based methods. However, the results on Tiny-ImageNet and FOOD101 indicate that PT-based methods suffer from inferior test accuracy. Although DFRD improves their test accuracy, the improvement is slight. Notably, DFRD improves negligibly over PT-based methods when all clients exemplify  $\frac{1}{16}$ -width capability. We thus argue that weak clients performing complex image classification tasks learn little useful local knowledge, resulting in the inability to provide effective information for the global model.
In this section, we carefully demonstrate the efficacy and indispensability of core modules and key parameters in our method on SVHN, CIFAR-10 and CIFAR-100. Thereafter, we resort to FedRolex+DFRD to yield all results. For SVHN (CIFAR-10, CIFAR-100), we set  $\omega = 0.1$  (0.1, 1.0)
(a) FMNIST,  $\omega = 0.1$
(b) SVHN,  $\omega = 1.0$
Figure 3: Accuracy curves selected of DFRD and baselines on FMNIST, SVHN and CIFAR-10.
(c) CIFAR-10,  $\rho = 5$
and  $\rho = 10$  (10,5). Note that we also study the performance of DFRD with different numbers of clients and stragglers, see Appendix G for details.
Impacts of different transferability constraints. It can be observed from Table 3 that our proposed transferability constraint  $\diamond$  uniformly beats  $\triangle$  and  $\nabla$  w.r.t. global test performance over three datasets. This means that  $\diamond$  can guide the generator to generate more effective synthetic data, thereby improving the test performance of the global model. Also, we conjecture that generating more synthetic data like those with the black circles (see Fig. 2) may positively impact on the performance of local models, since  $\diamond$  also slightly and consistently trumps other competitors w.r.t. L.acc.
Impacts of varying merge operators. To look into the utility of the merge operator, we consider multiple merge operators, including mul, add, cat, ncat and none. Among them, mul is  $o(z,y) = z \times \mathcal{E}(y)$ , add is  $o(z,y) = z + \mathcal{E}(y)$ , cat is  $o(z,y) = [z,\mathcal{E}(y)]$ , ncat is  $o(z,y) = [z,y]$  and none is  $o(z,y) = z$ . From Table 4, we can observe that our proposed mul significantly outperforms other competitors in terms of G.acc. This suggests that mul can better exploit the diversity constraint to make the generator generate more diverse synthetic data, thus effectively finetuning the global model. Also, the visualization of the synthetic data in Appendix I validates this statement.
Table 3: Test accuracy  $(\%)$  comparison among different transferability constraints over SVHN and CIFAR-10/100.
T. C.	SVHN	CIFAR-10	CIFAR-100
△	34.09±11.45	23.45±1.13	26.62±2.39
(15.38±2.50)	(15.91±1.24)	(12.76±0.51)
▽	33.89±11.22	24.27±1.82	27.18±1.78
(15.28±2.59)	(16.45±1.00)	(12.86±0.38)
◇	34.78±9.19	25.57±1.37	28.08±0.94
(15.99±1.53)	(16.74±0.72)	(13.03±0.16)
Table 4: Test accuracy  $(\%)$  comparison among different merger operators over SVHN and CIFAR-10/100.
M. O.	SVHN	CIFAR-10	CIFAR-100
mul	34.78±9.19(15.99±1.53)	25.57±1.37(16.74±0.72)	28.08±0.94(13.03±0.16)
add	32.83±8.86(16.04±1.25)	23.36±1.57(16.55±0.82)	23.46±2.04(12.22±0.44)
cat	32.84±10.80(15.97±1.46)	21.78±0.20(16.58±0.72)	24.14±2.31(12.27±0.52)
ncat	29.58±9.29(13.37±2.87)	20.93±0.80(16.87±0.62)	23.38±1.43(12.30±0.16)
none	27.79±10.70(13.99±2.82)	19.64±1.08(16.53±0.83)	20.95±2.32(12.81±2.32)
Necessity of each component for DFRD. We report the test performance of DFRD after divesting some modules and losses in Table 5. Here, EMA indicates the exponential moving average copy of the generator on the server. We can evidently observe that removing the EMA generator leads to a significant drop in  $G_{\cdot}acc$ , which implies that it can generate effective synthetic data for the global model. The reason why the EMA generator works is that it avoids catastrophic forgetting of the global model and ensures the stability of the global model trained in heterogeneous FL. We display synthetic data in Appendix I that further corroborates the above claims.
Meanwhile, we perform the leave-one-out test to explore the contributions of  $\mathcal{L}_{tran}$  and  $\mathcal{L}_{div}$  to DFRD separately, and further report the test results of removing them simultaneously. From Table 5, deleting either  $\mathcal{L}_{tran}$  or  $\mathcal{L}_{div}$  adversely affects the performance of DFRD. In addition, their joint absence further exacerbates the degradation of  $G_{acc}$ . This suggests that  $\mathcal{L}_{tran}$  and  $\mathcal{L}_{div}$  are vital for the training of the generator. Interestingly,  $\mathcal{L}_{div}$  benefits more to the global model than  $\mathcal{L}_{tran}$ . We speculate that the diversity of synthetic data is more desired by the global model under the
Table 5: Impact of each component in DFRD.
SVHN	CIFAR-10	CIFAR-100
baseline	34.78±9.19(15.99±1.53)	25.57±1.37(16.74±0.72)	28.08±0.94(13.03±0.16)
-EMA	26.97±12.28(14.17±3.09)	19.80±2.25(16.55±0.57)	24.57±0.93(12.23±0.07)
-Ltran	29.30±9.25(14.24±3.08)	22.97±1.91(16.33±1.16)	27.28±0.46(12.79±0.46)
-Ldiv	27.68±9.75(14.26±3.00)	22.12±1.03(16.46±1.15)	26.94±1.60(12.81±0.41)
-Ltran,-Ldiv	20.32±1.93(13.65±1.13)	21.97±2.48(16.52±1.39)	25.50±0.91(12.30±0.10)
premise of ensuring the fidelity of synthetic data by optimizing  $\mathcal{L}_{fid}$ .
Varying  $\beta_{tran}$  and  $\beta_{div}$ . We explore the impacts of  $\beta_{tran}$  and  $\beta_{div}$  on SVHN and CIFAR-10. We select  $\beta_{tran}$  and  $\beta_{div}$  from  $\{0.25, 0.50, 0.75, 1.00, 1.25, 1.50\}$ . From Fig. 4, we can see that DFRD maintains stable test performance among all selections of  $\beta_{tran}$  and  $\beta_{div}$  over SVHN. At the same time,  $G.acc$  fluctuates slightly with the
Figure 4: Test accuracy  $(\%)$  with varying  $\beta_{tran}$  and  $\beta_{div}$ .
increases of  $\beta_{tran}$  and  $\beta_{div}$  on CIFAR-10. Besides, we observe that the worst  $G.acc$  in Fig. 4 outperforms the baseline with the best  $G.acc$  in Table 2. The above results indicate that DFRD is not sensitive to choices of  $\beta_{tran}$  and  $\beta_{div}$  over a wide range.
Varying  $\alpha$  and  $\lambda$ . In order to delve into the effect of the EMA generator on DFRD in more details, we perform grid testing on the choices of control parameters  $\alpha$  and  $\lambda$  over SVHN and CIFAR-10. We set  $\alpha \in \{0.25, 0.50, 0.75, 1.00, 1.25, 1.50\}$  and  $\lambda \in \{0.1, 0.3, 0.5, 0.7, 0.9\}$ . It can be observed from Fig. 5 that high global test accuracies on
Figure 5: Test accuracy  $(\%)$  with varying  $(\alpha, \lambda)$ .
SVHN are mainly located in the region of  $\alpha < 1.25$  and  $\lambda > 0.5$ , while on CIFAR-10 they are mainly located in the region of  $\alpha > 0.25$  and  $\lambda < 0.9$ . According to the above results, we deem that the appropriate  $\alpha$  and  $\lambda$  in a specific task is essential for the utility of the EMA generator. Notably, high local test accuracies mainly sit in regions that are complementary to those of high global test accuracies, suggesting that pursuing high  $G_{*}acc$  and  $L_{*}acc$  simultaneously seems to be a dilemma. How to ensure high  $G_{*}acc$  and  $L_{*}acc$  simultaneously in the field of FL is an attractive topic that is taken as our future work.
In this paper, we propose a new FL method called DFRD, which aims to learn a robust global model in the data-heterogeneous and model-heterogeneous scenarios with the aid of DFKD. To ensure the utility, DFRD considers a conditional generator and thoroughly studies its training in terms of fidelity,
transferability and diversity. Additionally, DFRD maintains an EMA generator to augment the global model. Furthermore, we propose dynamic weighting and label sampling to accurately extract the knowledge of local models. At last, we conduct extensive experiments to verify the superiority of DFRD. Due to space constraints, we discuss in detail the limitations and broader impacts of our work in Appendixes J and K, respectively.
This work has been supported by the National Natural Science Foundation of China under Grant No.U1911203, and the National Natural Science Foundation of China under Grant No.62377012.