DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning
Authors: Kangyang Luo
Shuai Wang
Yexuan Fu
Xiang Li
Yunshi Lan
Ming Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Data Heterogeneity
Model Heterogeneity
Data-Free Distillation

Abstract: Federated Learning (FL) is a privacy-constrained decentralized machine learning paradigm in which clients enable collaborative training without compromising private data. However, how to learn a robust global model in the data-heterogeneous and model-heterogeneous FL scenarios is challenging. To address it, we resort to data-free knowledge distillation to propose a new FL method (namely DFRD).
DFRD equips a conditional generator on the server to approximate the training space of the local models uploaded by clients, and systematically investigates its training in terms of fidelity, transferability and diversity. To overcome the catastrophic forgetting of the global model caused by the distribution shifts of the generator across communication rounds, we maintain an exponential moving average copy of the generator on the server.  Additionally, we propose dynamic weighting and label sampling to accurately extract knowledge from local models. Finally, our extensive experiments on various image classification tasks illustrate that DFRD achieves significant performance gains compared to SOTA baselines.

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning
Kangyang Luo', Shuai Wang', Yexuan Fu', Xiang Li Yunshi Lan', Ming Gao 1.2
School of Data Science & Engineering!
KLATASDS-MOE in School of Statistics East China Normal University Shanghai, China
(52205901003, 51215903058, 51215903042 ecnu. edu, cn (xiangli, yslan, mgao) ecnu. edu. cn
Abstract
Federated Learning (FL) is hevacy-constrainge decentralized machine learning paradigm in which clients enable collaborative training without compromising private data. However, how to learn a robust global model in the data-heterogeneous and model-heterogeeneour FL scenarios is challenging. To address it, we resort to data-free knowledge distillation to propose new FL method (namely DFRD). DFRD equips a conditional generator the server to approximate the training space of the local models uploaded by clients, and systematically investigates its training in terms of fidelity, transferability and diversity. To overcome the catastrophic forgetting of the global model caused by the distribution shifts the generator across communication rounds, maintain exponential moving average copy of the generator on the server. Additionally, we propose dynamic weighting and label sampling accurately extract knowledge from local models. Finally, our extensive experiments on various image classification tasks illustrate that DFRD achieves significant performance gains compared SOTA baselines. Our code is here: http:///aoooyooooooooooocccscccc
Introduction
With surge data, deep learning algorithms have made significant progress both established and emerging fields 1133. However, in many real-world applications (e.g., mobile devices [4] loT (5), and autonomous driving [6], etc.), data generally dispersed across different clients (i.e., data silos). Owing to the high cost of data collection and strict privacy protection regulations, the centralized training that integrates data together is prohibited 7 8 Driven by this reality, Federated Learning (FL) (9] 10] has gained considerable attention in industry and academia
promising distributed learning paradigm that allows multiple clients to participate in the collaborative training of global model without access to their private data, thereby ensuring the basic privacy.
Despite its remarkable success, the inevitable hurdle that plagues FL research is the vast heterogeneity among real-world clients [1I]. Specifically, the distribution of data among clients may be non-lID
(identical and independently distributed), resulting in data heterogeneity 442 12 13 41. has been confirmed that the vanilla FL method FedAvg [14] suffers from client drift in this case, which leads to severe performance degradation. To ameliorate this issuc, a plcthora of modifications [[5] 21 34 (3] for FedAvg focus regularizing the objectives of the local models to align the global optimization objective. All the above methods follow the widely accepted assumption of model ho- mogeneity, where local models have share the same architecture the global model. Nevertheless,
"corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

when deploying FL systems, different clients may have distinct hardware and computing resources, and can only train the model architecture matching their own resource budgets [2] 24 resulting in model heterogeneity. In this case, to enable the Fl. systems with model homogeneity, on the one hand, clients with low resource budgets, which may critical for enhancing the FI. systems, will be discarded at the expense of training bias 43 23 On the other hand, keeping low complexity for the global model to accommodate all clients leads to performance drop due to the limited model capacity [30]. Therefore, the primary challenge of model-heterogencous FL is how to conduct model aggregation of heterogeneous architectures among clients to enhance the inclusiveness of federated systems. To solve this challenge. existing efforts fall broadly into two categories: knowledge distilla- tion (KD)-based methods [40 45 44 25 27] and partial training (PT) methods [30 28 29 311,
yet each of them has its own limitations. Concretely, KD-based methods require additional public data to align the logits outputs between the global model (student) and local models (teachers). But the desired public data is not always available in practice and the performance may decrease dramatically the apparent disparity in distributions exists between public data and clients' private data [33]. PT-baaed methods send width-based sub-models to clients, which extracted by the server from larger global model according to each client's 's resource budget, and then aggregate these trained sub-models to update the global model. PT-based methods can be considered as an extension of FedAvg to hoodel-heteoogeneous scenarios, which means they are implementation--frienn computationally efficient, but they also suffer from the same adverse effects from data heterogeneity as FedAvg or even more severely. In a word, how to learn a robust global model in FL with both data and model heterogeneity is a highly meaningful and urgent problem.
and
glóal Eodld
glbel model
Agg
Upbod
Dovntiood
Mashd Appreeations
Globe rosid (lose)
Cluaal Ogpiiie
Swzthedi Sara
Southati daza
9 n9
Exc
Ferus
Eas
Lat
Guneskar opdaate
Eyca
rebiddin
Fraedde land abvoodes (boo))
Geomatr
9ynth dats
Eroeniti look reddds (Tooe))
(Fuel)
Trining Geverator
Roboot Model Eidtiiation
Figure 1: The full pipeline for DFRD combined with a PT-based method. DFRD works on the server and contains two phases, training generator and robust model distillation, where Lraans Caser fid and LuI, Lu the loss objectives of the conditional generator and the global model, respectively.
To this end, we systematically investigate the training of a robust global model in FL with both data and model heterogeneity with the aid of kata-free knowledge distillation (DFKD) B5 36 53 59) See the related work in Appendix for more DFKD methods. Note that the strategy of integrating DFKD to FL is not unique us. Recently, FedFTG 48 leverages DFKD to fine-tune the global model in hooee--homogeneuus FL to overcome data heterogeneity, and DENSE [49] aggregates knowledge from heterogeneous local models based on DFKD to train global model for one-shot FL. They all equip the server, which possesses powerful hardware and computing resources, with a generator to approximate the training space of the local models (teachers). and train the generator and the global model (student) in an adversarial manner. However, the local models uploaded per communication round are not only architecturally heterogeneous but also trained on non-IID distributed private data in the situation of both data and model heterogeneity. In this case, the generator tends to deviate from the real data distribution. Also, its output distribution may undergo large shiftss (i.e., distribution shifts) across communication rounds, causing the global model to catastrophically forget useful knowledge learned in previous rounds and suffer from performance degrudation. To confront the mentioned issues, we propose novel Data-Fre Free Robust Distillation FL method called DFRD, which utilizes conditional generator to generate synthetic data and thoroughly studies how to effectively and accurately simulate the local models' training space in terms of fidelity, transferability and

---

To mitigate catastrophic forgetting of the global model, an exponential moving average (EMA) copy of the conditional generator is maintained on the server to store previous knowledge learned from the local models. The FMA generator, along with the current generator, provides training data for updates of the global model. Also, we propose dynamic weighting and label sampling to aggregate the logits outputs of the local models and sample labels respectively, thereby properly exploring the knowledge of the local models. We revisit FedFTG and DENSE, and argue that DFRD as fine-tuning method (similar to FedFTG) can significantly enhance the global model. So, we readily associate the PT-based methods in model-hetterggeneuus FL with the ability to rapidly provide a preliminary global model, which will be fine-tuned by DFRD. We illustrate the schematic for DERD as a fine-tuning method based on a PT-based method in Fig. T Although FedFTG and DENSE can also be applied to fine-tune the global model from the PT- methods after simple extensions, we empirically find that they do not perform as well, and the performance of the global model is even inferior to that of local models tailored to clients' resource budgets.
diversity 48 49
Our main contributions of this work are summarized as follows. First, we propose a new FL. method termed DFRD that enables a robust global model in both data and model heterogeneity settings with the help of DFKD. Second, we systematically study the training of the conditional generator w.r.t. fidelity, transferability and diversity to ensure the generation of high-quality synthetic data. Additionally, we maintain an EMA generator on the server to overcome the global model's s catastrophic forgetting caused by the distribution shifts of the generator. Third, we propose dynamic weighting and label sampling to accurately extract the knowledge of local models. At last, our extensive experiments on six real-world image classification datasets verify the superiority of DFRD.
Notations and Preliminaries
Notations. In this paper, we focus on the centralized setup that consists of a central server and N clients owning private labeled datasets {(X,, Y)) where X; { follows the data distribution D, over feature space X, i.e., z ~ Di, and Y, = fu- C [C] {1....., C} denotes the ground-truth labels of X, And C refers to the total number of labels. Remarkably, the heterogeneity for FL in our focus includes both data heterogeneity and model heterogeneity. For the former, we consider the same feature space, yet the data distribution may be different among clients, that is, label distribution skewness in clients (i.e., X; X2 and D. * DD, Vi & j.iij E [N]. For the latter, each client é holds an on-demand local model f. parameterized by 0, Due to the difference in resource budgets, the model capacity of each client may vary, i.e., |0.] + 00,1, 3i + j, i, j [N]. In PT-based methods, we define a confined width capability R. E (0. according to the resource budget of client which is the proportion of nodes extracted from each layer in the global model f. Note that f is parameterized by 6, and la| denotes the number of elements in vector a.
Jo=1
PT-baaed method is solution for model-heterogenee
FL, which strives to extract matching width-based slimmed-down sub-model from the global model as a local model according to each client's budget. As with FedAvg, it requires the server to periodically communicate with the clients. In each round, there are two phases: local training and server aggregation. In local training, each client trains the sub- sub-model received from the server utilizing the local optimizer. In server aggregation, the server collects the heterogeneous sub-models and aggregates them by straightforward selective averaging to update the global model, as follows [18-
0 [iAK] Ejes, Pj ies.
Σ p.0 (p.)).
where S, is a subset sampled from [N] and Pt is the weight of client i, which generally indicates the size of data held by client i. At round t, BiL.)) denotes the t parameter of layer of the global model and 0 (. denotes the parameter 0f updated by client i. We can clearly see that Eq. I independently calculates the average of each parameter for the global model according to how many clients update that parameter in round l. Instead, the parameter remains unchanged if no clients update it. Notably, if [0]] |0'| for any i E [N], PTT-based method becomes FedAvg. The key to PT-based method is to select Of from the global model 0' when given Rg. And existing sub-model
extraction schemes fall into three categories: static [28 29 random
[30] and rolling [31].
3

---

3 Proposed Method
In this section, we detail the proposed method DFRD. We mainly work on considering DFRD as a fine-tuning method to enhance the PT-based methods, thus enabling a robust global model in FL with both data and model heterogeneity. Fig. [ visualizes the training procedure of DFRD combined with a PT-based method, consisting of four stages on the server side: training generator, robust model distillation, sub-model extraction and model aggregation. Note that sub-model extraction and model aggregation are consistent with that in the PT-based methods, so we detail the other two stages. Moreover, we present pseudocode for DFRD Appendix C
3.1 Training Generator
At this stage, we aim to train a well-behaved generator to capture the training space of local models uploaded from active clients. Specifically, we consider a conditional generator G(-) parameterized by w. It takes as input a random noise Rd sampled from standard normal distribution ( I).
and a random label E [C'] sampled from label distribution p(y). i.c., the probability of sampling y. thus generating the synthetic data s = G( o(z, y), w). Note that o(z, y) represents the merge operator of z and To the best of our knowledge, synthetic data generated by well- generator should satisfy several key characteristics: fidelity, transferability, and diversity [48] 49].
Therefore, in this section, we construct the loss objective from the referred aspects to ensure the quality and utility of G(.).
Fidelity. To commence, we study the fidelity of the synthetic data. Specifically, we expect G(-) to simulate the training space of the local models to generate the synthetic dataset with a similar distribution to the original dataset. To put it differently, we want the synthetic data to approximate the training data with label y without access clients' training data. To achieve it, we form the fidelity loss Lidd at logits level:
Ljid= = CE Tra.f((s 0.), y),
(2)
iES,
where CE denotes the cross-entropy function, fi 0.) is the logits output of the local model from client i when s is given, Tiyy dominates the weight of logits from different clients {ili E S when given, And yed is the cross-entropy loss between the weighted average logits Eies, Ti 0,) and the label By minimizing fidd, s is enforced to be classified into label y with high probability. thus facilitating the fidelity of s.
In reality, the conditional generator G(-) easily generates synthetic data with low classification errors (i.e. fid close to 0) as the training pro- ceeds. This may cause the synthetic data to fall into a space far from the decision boundary of the ensemble model (i.e., Eies, fi 0.) ) if only fid is optimized, as shown in the synthetic data represented by red circles in Fig. 2 (a). Note that ds and dr denote the decision boundaries of the global model (student) and the ensem- ble model (teacher), respectively. An obvious observation is that the red circles are correctly classified on the same side of the two decision boundaries (i.c., ds and dr), making it difficult to transfer teacher's knowledge to student. We next explore how to augment the transfcrability of the synthetic data ameliorate this pitfall.
sytthetic
red ckia
(4)
Figure 2: The visualization of synthetic data and decision boundaries of global model (student) and ensemble model (teacher). Left panel: synthetic data (red circles) are far away from the decision boundary dr. Middle panel: synthetic data (black circles) near the decision boundaries dr. Right panel: synthetic data (yellow and purple circles) cross over the decision boundary dr.
Transferability is intended to guide G(-) in gen- erating synthetic data that moves the decision boundary of the global model towards that of the ensemble model, such as synthetic data with black circles in Fig. (b). However, during the training of ds to approach dr, we find that G(-) can easily generate two other types of synthetic data, the yellow and purple circles Fig. (c). Both of them misclassified by the ensemble model (dr), while the yellow circles are correctly classified and

---

the purple circles are misclassified by the global model (ds). For the conditional generator G(-) that takes label information as one of the inputs, yellow and purple circles can mislead the generator, thereby leading to dy approximating dr with a large deviation, as shown in Fig. 2 (c). Based on the above observation, we reckon that the synthetic data & = G(h = o(z, y), w) is useful if is classified as y by the ensemble model but classified not as y by the global model. To realize it, we maximize the logits discrepancy between the global model and the ensemble model on synthetic data with black circles by leveraging Kullback-Leibler divergence loss, which takes the form:
Liran = - KK() Σ
f(s.
iES:
where KL is Kullback-Leibler divergence function and [(s, 0) denotes the logits output of the global model on 8 with label y. Note that e y hold, otherwise e = 0. (+)
= 1 if arg max [(s 0) + y and arg max Lies, Ti 0.)
We would like to point out that the existing works 48 and 49 are in line with our research perspective on the transferability of generator, which aims to generate more synthetic data with black circles. However, they do not carefully frame learning objective for enhancing the transferability of
generator. Concretely, [48] does not consider the type of synthetic data, i.e., E = always holds, thus inducing the generation of synthetic data with yellow and purple circles. (A) [49] focuses on synthetic data satisfying arg max (ss 0) F arg max Eies. fi 0,), but enables the generation of synthetic data with purple circles yet. (v)
Diversity. Although we enable G(.) to generate synthetic data that falls around the real data by optimizing Liid and Ltran. the diversity of synthetic data is insufficient. Due to the fact that the generator may get stuck in local equilibria the training proceeds, model collapse occurs [57] 58). In this case, the generator may produce similar data points for each class with little diversity. Also, the synthetic data points may not differ significantly among classes. This causes the empirical distribution estimated by G(.) to cover only a small manifold in the real data space, and thus only partial knowledge of the ensemble model is extracted. To alleviate this issue, we introduce a diversity loss Ldio with label information to increase the diversity of synthetic data as follows:
Laio =ex
Σ J.KE(B]
|8j -8sl22 hi; h2 1bxl22/B3 BB).
(4)
-000=o(zz,)) olz Uz/k), w). Intuitively, Laiv takes l|,, - h& 2|2 as weight, and then multiplies it by the corresponding ||s; - sxl12 in each batch B, thus imposing a larger weight on the synthetic data points pair (8, and 8x) at the more distant input pair (hj and h2). Notably, we merge the random noise z with label y as the input of G(-) to overcome spurious solutions [53] Further, we propose multiplicative merge operator, i.e., (22,y) = x E(y), where & is a trainable embedding and x means vector demeent-wise product. We find that our merge operator enables synthetic data with more diversity compared to others, possibly because the label information is effectively absorbed into the stochasticity of = by multiplying them when updating E. See Section 4.3 for more details and empirical justification.
where B denotes the batch size and 85/k = G(hy/k =
Combining fidd, Liran and Laiv. the overall objective of the generator can be formalized as follows:
Cyen = fid + Stran Liran + Bass Ldive
(5) Of note, the synthetic data generated by a well-
where Stran and Baii are tunable trained generator should be visually distinct from the real data for privacy protection, while it can capture the common knowledge of the local models to ensure similarity to the real data distribution for utility. More privacy protection is discussed in Appendices | |and]]
3.2 Robust Model Distillation
Now we update the global model. Normally, the global model attempts to learn much as possible logits outputs of the ensemble model on the synthetic data generated by the generator based on
48 49 The updated global model and the ensemble model are then served to train G(.) with the goal of generating synthetic data that maximizes the mismatch
knowledge distillation [40] 46
NNote that 4. 4 and Q denote the strategies for the transferability of generator in 48 in 49 and in this
paper, respectively.
5

---

between them in terms of logits outputs (see transferabilityy discussed in the previous section). This adversarial game enables the generator to rapidly explore the training space of the local models to help knowledge transfer from them to the global model. However, it also leads to dramatic shifts in the output distribution of G(-) across communication rounds under heterogeneous FI. scenario (i.e., distribution shifts), causing the global model to catastrophically forget useful knowledge gained in previous rounds. To tackle the deficiency, propose to equip the server with a generator G(-) parameterized by w that is an exponential moving average (EMA) copy of G(-). Its parameters at the h communication round are computed by
W' = A. (1 w',
(6)
where À E (0. 1) is the momentum. We can easily see that the parameters of G(-) vary very little compared to those of G(.) over communication rounds, if is close 1. We further utilize synthetic data from G-) as additional training data for the global model outside of G(-), mitigating the huge exploratory distribution shift induced by the large update of G(.) and achieving stable updates of the global model. Particularly, we compute the Kullback-Leible divergence between logits of the ensemble model and the global model on the synthetic data points s G(h o(2, y); w) and
G(h = o(ž, w) respectively, which is formulated as follows:
Lmd = Lu + aLu = KL(f(s, 0),
Σ TTaff(ss,0) 0.)) + a KL(F(3,0) 0), Σ T. 0.)), (7) SES
sES,
where & is a tunable
for balancing different loss items.
Dynamic Weighting and Label Sampling. So far, how to determine Ti.y and p(y) is unclear. The appropriate Thy and p(y) are essential for effective extraction of knowledge from local models. For clarity, we propose dynamic weighting and label sampling, i.e., Ti,y n²,,/ " S,, and p(v) = n's..// Eyetcy " where n²... Ejels, "2, and n², denotes the number of data with label y involved in training on client at round f. Due to space limitations, see Appendix for their detail study and experimental justification.
Experiments
4.1 Experimental Settings
Datasets. In this paper, we evaluate different methods with six real-world image classification task-related datasets, namely Fashion-MNIST (69] (FMNIST in short), SVHN [70], CIFAR- 10,
CIFAR-100 (7], Tiny-imageNe and Food101 73 We detail the six datasets in Appendix B To
simulate data heterogeneity across clients, in previous works 334 37 38 use Dirichlet process
Dir(w) to partition the training set for each dataset, thereby allocating local training data for each client. is worth noting that w is the concentration parameter and smaller corresponds to stronger data heterogeneity.
Baselines. We compare DFRD to FedFTG 48 and DENSE [49], which are the most relevant methods to our work. To verify the superiority of DFRD, on the one hand, DFRD, FedFTG and DENSE are directly adopted the server to transfer the knowledge of the local models to randomly initialized global model. We call them collectively data-free methods. On the other hand, they are utilized as fine-tuning methods to improve the global model's performance after computing weighted average per communication round. In this case, in each communication round, the preliminary global model is obtained using FedAvg [14] in FL with homogeneous models, whereas in FL with heterogeneous models, the PT-based methods, including HeteroFL (29]. Federated Dropout [30] (FedDp for short) and FedRolex [11, are employed get the preliminary global model.
Configurations. Unless otherwise stated, all experiments are performed on a centralized network with N 10 active clients. We set w E f0.01, 0.1, 1.0) to mimic different data heterogeneity scenarios. To simulate
scenarios, we formulate exponentially distributed budgets for a given N: R: min J) E [N]), where a and are both positive integers. We fix a 4
http://cs233 In wwt.saaoooddedduueo..... mmagenet-200.zig As an example, DFRD+FedRelox indicates that DFRD is used as fine- tuning method to improve the performance of FedRelox, and others are similar. See Tables I and for details.

---

and consider E {5, 10, 40). See Appendix foo more details. Unless otherwise specified, we set Stran and Baio both to in training generator, while in robust model distillation, we set À = 0.5 and a = 0.5. And all baselines leverage the same setting as ours. Due to space limitations, see Appendix for the full experimental setup.
Evaluation Metries. We evaluate the performance of different FL methods by local and global test accuracy. To be specific, for local test accuracy (L.ace for short), we randomly and evenly distribute the test set to each client and harness the test set on each client to verify the performance of local models. In terms of global test accuracy (G.ace for short), we employ the global model on the server to evaluate the global performance of different FL methods via utilizing the original test set. Note that L.ace is reported in round brackets. To ensure reliability, we report the average for each experiment over 3 different random seeds.
Table 1: Top test accuracy (%) of distinct methods across E 1.0, on different datasets.
10
w-
4f
of B01
13.00
15.00am
13.63
16.3aans
15.0
30
1.33amm 123000
(1620
(15.01
134 To
(h 39
(3415
177 13300om))
14.3
29.5
18.15 4M
54.88
38.50 cm0
(8
(14.55
(8:00
Follig
66.90
(4457
124
154.2
(2485
(12.57614)
(12 02:37-10
(4500
25
DDNNE
00 11
S4 106155
63.36
52.44
0n 73
35
50.84 paae
48.24
+l
M1. an
5- 17
64.00
61 15
17.07
66, oc
56. 21
54.5
16.00 50
V1.65
8566:111
65.33
75882100
65.79112
71.5
61,30
48:77:7.
83417
(86.16 ku0
(65.7 120
120 55
00.31 1200
(27.2 Am))
(100%) L00
[23 (23A0)
(18449 mm)))
(4807
129 10
114.9A
4.2 Results Comparison
Impacts of varying w. We study the performance of different methods at different levels of data heterogeneity on FMNIST, SVHN, CIFAR- 10 and CIFAR-100, as shown in Table One can see that the performance of all methods degrades severely as w decreases, with DFRD being the only method that is robust while consistently leading other baselines with an overwhelming margin w.r.t. G.ace. Also, Fig. 3 (a)-(b) show that the learning efficiency of DFRD consistently beats other baselines (see Fig. 8 9 in Appendix for complete curves). Notably, DFRD, FedFTG and DENSE as fine-tuning methods uniformly surpass FedAvg w.r.t. G.ace and L.acc. However, their global test accuracies suffer from dramatic deterioration even substantially worse than that of FedAvg when they act as data-free methods. We conjecture that FedAvg aggregates the knowledge of local models more effectively than data-free methods. Also, when DFRD is used to fine-tune FedAvg, it can significantly enhance the global model, yet improve the performance of local models to a less extent.
Impacts of different p. We explore the impacts of different model heterogeneity distributions on different methods with SVHN, CIFAR-10, 10, Tiny-ImageNet and FOOD101. A higher p means more clients with To wwidth capacity w.r.t. the global model. From Table 2 we can clearly see that the performance of all methods improves uniformly with decreasing p. where DFRD consistently and overwhelmingly dominates other basclines in terms of G.ace. Specifically, DFRD improves G.ace by an average of 11.07% and 7.54% on SVHN and CIFAR-10 respectively, compared to PT-based methods (including HeteroFL, FedDp and FedRolex). Meanwhile, DFRD uniformly and significantly outstrips FedFTG and DENSE w.r.t. Gaace. The selected learning curve shown in Fig. B (c) also verifies the above statement (see Fig. 10 12 in Appendix HH for more results). The above empirical results show that DFRD not only is robust to varying P. but also has significantly intensified effects on the global model for different PT- based methods. However, the results on ImageNet and FOOD101 indicate that P'T-based methods suffer from inferior test accuracy. Although DFRD improves their test accuracy, the improvement is slight. Notably, DFRD improves negligibly over PT-based methods when all clients exemplify 16 wiithh capabiliiy. We thus argue that weak clients performing complex image classification tasks learn little useful local knowledge, resulting in the inability to provide effective information for the global model.
4.3 Ablation Study
In this section, we carefully demonstrate the efficacy and indispensability of core modules and key parameters in our method on SVHN CIFAR-10 and CIFAR-100. Thereafter, we resort to FedRolex+DFRD to yield all results. For SVHN (CIFAR-10, CIFAR- -OF), we set w 0.1 (0.1, 1.0)
7

---

(a) FMNIST, W = 0.1
(b) SVHN, & 1.0
(C) CIFAR-I00 P 5
Figure 3: Accuracy curves selected of DFRD and baselines on FMNIST, SVHN and CIFAR-10.
Table 2: Top test accuracy (%) of distinct methods across p E {5, 10, 40} on different datasets.
SVInN
CUARI
They Iropshi
Alp.
a
Themolt
25.7ami
17
K.77am
1.DDamn
-DEN0T
16 Reetas
1
1.3
01.5
(240
(8344x2))
(1:15.m))
13.352
14
1.70o017
221 104000
(3156
(447
C2:46 ml
[540
18.00000
(1.8-.o)
<8 (814
111.57 aual
(3155)c9))
124
(433
(2675
18..2
(928
(1854aoor)
(14.9
(1455220
115 mm
(1.4en00
(197-400)
(2/2:430)
(1 Tiwnss)
(1:0043)
DDENS
263
22
19.6411
4455ax3
12:32
6.4
18
+FoFFT
11 1:30
2055
26
3.50ax
12.17axa
6,2
1222.a5
0.
7:53
1,08 ABuuz
1.60uu
110
666661
(5511
(1 16sn))
(3:4-am
(146-430
(1480
(1635-44))
(1500
(1.66430)
(125 km)
(MA2
(2 (2.4
(1452440)
(14 10
03352200
115 22
11 30V
(16 2axw)
(7
15 3hann
2 T3an
(181
16.0a 1
( 37ari
(240
-DENSS
26-55-231
23
4344
2400ca
7
(1
25.55mm
3400
14
6.25
1.8-43
14
15
4D1B
3678
32.86
25.57a
6300-411
12.78cm
18.55
(1599..00
(16
(17.1
(1674.09)
(17.7 (L.7
3.36.000
(2.22 km)
1M02 am)
(A34
and p = 10 (10, 5). Note that we also study the performance of DFRD with different numbers of clients and stragglers, see Appendix G for details.
Impacts of different transferability con- straints. It can be observed from Table our proposed transferability constraint & uni- formly beats A and V w.r.t. global test perfor- mance over three datasets. This means that O can guide the generator generate more effec- tive synthetic data, thereby improving the test performance of the global model. Also, conjecture that generating more synthetic data like those with the black circles (see Fig. 2} may positively impact on the performance of local models, since also slightly and consis- tently trumps other competitors w.r.t. L.acc.
Table 3: Test accuracy (%) comparison among dif- ferent transferability constraints over SVHN and CIFAR-10/100.
C.C.
SVHN
CIFAR-10
CIFAR-100
4
34.09
23.45
26.62a2.9
(15.38=2250
(15.91=1244)
(12.76=051)
33.892m122
24.27
27. 18
(15.28:22.9)
(16.45=110)
(12.86 (6-06))
34.78
23757
28.08
(15.99:1.55)
(16.74=0772)
(13.03=0.16)
Impacts of varying merge operators. To look into the utility of the merge operator, we consider multiple merge operators, including madl add, cal, neal and none. Among them, mad is o(2, y)
Table Test accuracy (%) comparison among differ- ent merger operators over SVHN and CIFAR- 10/1000
Evy), add is o( o(2,y) y) z = Ely), eal is o(z.y) y) = [z, {y)), neat o(z, y) Iz, y] and none is o(z,3) 3) From Table we can observe that our pro- posed mul significantly outperforms other competitors in terms of G.ace. This suggests that mul can better exploit the diversity con- straint to make the generator generate more diverse synthetic data, thus effectively fine- tuning the global model. Also, the visual- ization of the synthetic data in Appendix ! validates this statement.
M. O.
SVHN
CIFAR-10
CIFAR-100
mul
34.78:9.99
25.57:1137
28.08:044
(15.99a1.55)
(16.74gu7z)
(13.03-416)
add
32.83-8.86
23.36
23.46+204
(16.04:1.25)
(16.55
(12.22=044)
eaf
32.84=11000
21.78
24.144231
(15.97=1.46)
(16.58
(12.27=n5:)
vecal
29.58-40.29
20.93
23.38+1.4
(13.37
(16.87=00))
(12.30=0.16)
27.79
19.64
20.95+230
(13.99#262)
(16.53
(12.81=23:)
8

---

Necessity of each component for DFRD. We report the test performance of DFRD after divest- ing some modules and losses in [ Here, EMA indicates the exponential moving average copy of the generator on the server. We can evidently observe that removing the EMA generator leads to a significant drop in G.ace, which implies that can generate effective synthetic data for the global model. The reason why the EMA generator works is that it avoids catastrophic forgetting of the global model and ensures the stability of the global model trained in beteroge- that further corroborates the above claims.
neous FL. We display synthetic data in Appendix Meanwhile, we perform the leave-one-out test to explore the contributions of Leran and Ldie to DFRD separately, and further report the test results of removing them simultaneously. From Table 5 deleting either Ltran or Lisi adversely affects the performance of DFRD. In addition, their joint absence further exacer- bates the degradation of G.ace. This suggests that Liran and Ldiv are vital for the training of the generator. Interestingly, Laiv benefits more to the global model than Ltran- We speculate that the diversity of synthetic data is more desired by the global model under the premise of ensuring the fidelity of synthetic data by optimizing Lyid.
Table 5: Impact of each component in DFRD.
SVIIN
CIFAR-10
CIFAR 100
baseline
34.78:9.19 (15.99115))
25.57
28.08:a31
(16.74
(1303 (1383
-EMA
26.97=+1225
19.80=215
24.57+009
(14, 17:330))
(16.552005)
(12.23
Lerax
29.30,975
22.97
27.28
(14.24
(16.331116)
(12,79.iva)
-as
27.68-99.15
22.12=1:6
26.94::100
(14, 26))
(16. (16.46=L15)
(12.8 =041)
-Lrana -odto
20.32.a1
21 21.972248
25.50
(13.65 mn)
(16.52
(12.30
Varying Stran and Adiv. We explore the impacts of Beaan and Adw on SVHN and CIFAR-10. We select Stran and Baie from {0.25, 0.50, 0.75, 1.00, 1.25,1.50), 1.50]. From Fig. 4 we can see that DFRD maintains stable test performance among all selections of Stran and Sase over SVHN. At the same time, G.ace fluctuates slightly with the increases of Btran and Bdie on CIFAR-10. Besides, we observe that the worst G.ace in Fig. outperforms the baseline with the best G.ace in Table The above results indicate that DFRD not sensitive to choices of Biran and Baiv over a wide range.
Are
Aer
Bar
Figure 4: Test accuracy (%) with varying Ptran and Baiv-
Varying a and A. In order delve into the effect of the EMA generator on DFRD in more details, we
Cot WWW55W CL
CILOAA--.0.CC Clace
FAP-11
perform grid testing on the choices of control parameters a and À over SVHN and CIFAR- 10. We set a E (0.25, 0.50, 0.75, 1.00, 1.25, 1.50]
1400
and X E {0.1, 0.5, 0.7, 0.9). can be observed from Fig. 5 that high global test accuracies on SVHN are mainly located in the region of a < 1.25 and > 0.5, while on CIFAR-10 they are mainly located in the region of a > 0.25 and À < 0.9. According to the above results, we deem that the appropriate a and À in a specific task is essential for the utility of the EMA generator. Notably, high local test accuracies mainly sit in regions that complementary to those of high global test accuracies, suggesting that pursuing high G. and L.ace simultancously seems to be a dilemma. How to ensure high G.ace and L.ace simultancously in the field of FL is an attractive topic that is taken as our future work.
Figure 5: Test accuracy (%) with varying (a. .A).
5 Conclusion
In this paper, we propose a new FL method called DFRD, which aims to learn robust global model in the data-hhetroggeneous and moter-hhetroggeeeuus scenarios with the aid of DFKD. To ensure the utility, DFRD considers a conditional generator and thoroughly studies its training in terms of fidelity,
9

---

transferabilityy and diversity. Additionally, DFRD maintains an EMA generator to augment the global model. Furthermore, we propose dynamic weighting and label sampling to accurately extract the knowledge of local models. At last, we conduct extensive experiments to verify the superiority of DFRD, Due to space constraints, we discuss in detail the limitations and broader impacts of our work in Appendixes and K respectively.
6
Acknowledgments
This work has been supported by the National Natural Science Foundation of China under Grant No.U1911203, and the National Natural Science Foundation of China under Grant No. N0.62377012.
References
[1] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition,, pp. 3213-3223.
[2] Krizhevsky, A., Sutskever, I, & Hinton, G. E. (2017) Imagenet classification with deep convolutional neural networks. Communications of the ACM 60(6): 84-90.
[3] Chen, Z., Li, B., Xu, Wu, S., Ding, S., & Zhang, W. (2022) Towards practical certifiable patch defense with vision transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15148-15158.
[4] Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y. C., Yang. Q. Niyato, D., & Miao, C. (2020) Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials 22(3): 2063..
[5] Nguyen, D. C., Ding, M., Pathirana, P. N., Seneviratne, A., Li, J., & Poor, H. V. (2021) Federated lcaming for internet of things: A comprehensive survey, IEEE Communications Surveys & Tutorials 23(3): 1622-1658.
[6] Li, Y., Tao, X., Zhang, X., Liu, J., Xu, J. (2021) Privacy-preserved federated learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems 23(7): 8423-8434.
[7] Voigt, P., & Von dem Bussche, A. (2017) The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Chum: Springer International Publishing 10(3152676): 10-5555.
[8] Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji. A. N., et al. (2021). Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning 14(1-2): 1-210.
[9] Konečný. J., McMahan, B., & Ramage, D. (2015) Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575.
[10] Konečný, J., McMahan, B., Yu, F. X., Richtárik, P., Suresh, A. Bacon, (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492.
[11] Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) Federated learning: Challenges, methods, and future directions. JEEE signal processing magazine 37(3): 50-60.
[12] Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., & Chandra, V. (2018) Federated learning with non-iid data. arXiv preprint / 1806600588
[13] Li, Q. Diao, Y., Chen, Q. & He, B. (2022) Federated leamiing non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference Data Engineering (ICDE), pp. 965-978.
[14] McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, A. (2017) Communication-effcieenn learning of deep networks from decentralized data. Artificial intelligence and statistics, pp. 1273-1282.
[15] Li, T.. Sahu, A. K., Zaheer. M., Sanjabi, M., Talwalkar, A., Smith, (2020). Federated optimization heterogeneous networks. Proceedings of Machine learning and systems 2: 429-450.
[16] Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., & Suresh, A. T. (2020) Scaffold: Stochastic controlled averaging for federated leaminn. In International Conference on Machine Learning, 5132- 5143.
10

---

[17] Acar. D. A. E.,. Zhao, Y.. Navarro, R. M.. Mattina, M. Whatmough. P. N., & Saligrama, V. (2021) Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04233.
[18] Kim, J., Kim, G., & Han, B. (2022) Multi-level branched regularization for federated learning. In International Conference Machine Learning, pp. 11058-11077
[19] Mendieta, M., Yang. T., Wang, P., Lee, M., Ding, Z., & Chen, C. (2022) Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397-8406.
[20] Lee, G., Jeong, M., Shin, Y.. Bae, S.. & Yun, S. Y. (2022) Preservation of the Global Knowledge by Not-True Distillation in Federated Learning. Advances in Neural Information Processing Systems.
[21] Zhang, Li, Z., Li, B., J., W., Ding. S., & Wu, C. (2022) Federated learning with label distribution skew via logits calibration, In International Conference on Machine Learning, pp. 26311 26311-26329.
[22] Ignatov, A.,
Timofte, R., Chou, W., Wang, K., Wu, M., Hartley, T., Van Gool, L. (2018) Ai benchmark: Running deep neural networks on android smartphones. Proceedings of the Luropean Conference on Computer Vision (ECCV) Workshops, pp. 0-0.
[23] Makhija, D., Han, X., Ho, N., & Ghosh, J. (2022) Architecture agnostic federated learning for neural networks. In International Conference on Machine Learning, pp. 14860-14870.
[24] Hong, J., Wang, H., Wang, Z., & Zhou, (2022) Efficient split-mix federated learning for on-demand and in-situ customization. arXiv preprint arXiv:2203.0974
[25] Afonin, A., & Karimireddy, S. (2021) Towards model agnostic federated learning using knowledge distillation. arXiv preprint 10.12200
[26] Cho, Y. J., Manoel, A., Joshi, G., Sim, R., Dimitriadis, D. (2022) Heterogeneous ensemble knowledge transfer for training large models in federated learning, arXiv preprint arXiv:220..1200:
[27] Fang, X., & Ye, M. (2022) Robust federated learning with noisy and heterogemcous clients. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10072-10081.
[28] Horvath, S., Laskaridis, S., Almeida, M.. Leontiadis, L., Venieris, S., & Lane, N. (2021) Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems 34: 1 12876-12889.
[29] Diao, E., Ding. J., & Tarokh, V. (2020) HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXtv:2010.0/244.
[30] Caldas, S., Konečny, J., McMahan, H. B., & Talwalkar, A. (2018) Expanding the reach of federated learning by reducing client resouree requirements. arXiv preprint 1rXiv:1812.072/0.
[31] Alam, S., Liu, L.. Yan, M. & Zhang, M. (2022) FedRolex: Model-Heterogen Heterogeneous Federated Learning with Rolling Sub-Model Extraction. Advances in neural information processing systems.
[32] Tang, Z., Zhang, Y., Shi, S., He, X., Han, B., & Chu, X. (2022). Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In International Conference Machine Learning, pp. 21111-21132.
[33] Tan, Y., Long. G., Liu, L., Zhou, T., Lu, Q., Jiang J., & Zhang, C. (2022) Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence 36(8); 8432-8440.
[34] Luo. K., Li, X.. Lan, Y. & Gao. M. (2023) GradMA: Gradient- Gradient-Memory Memory- Accelerated Federated Learning with Alleviated Catastrophic Forgetting, arXiv preprint 2rXiv:2302.143377
[35] Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu C., Xu C. & Tian, Q. (2019) Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3514-3522.
[36] Micaelli, P., & Storkey, A. J. (2019) Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems 32.
[37] Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang. N., & Khazaeni. Y. (2019, May) Bayesian nomparametric federated learning of neural networks. In International conference on machine learning, pp. 7252-7261.
11

---

[38] Wang, H.. Yurochkin. M., Sun, Y., Papailiopoulos, D., & Khazaeni, Y. (2020) Federated learning with matched averaging, arXiv preprini arX1v:2002.06440.
[39] Li. Q.. He, B., & Song. D. (2021) Model-comee contrascnt federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722
[40] Lin, T., Kong, L.,, Stich, S. U., & Jaggi, M. (2020) Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems 33: 2351-2363.
[41] Li, X., Jiang, M., Zhang. X., Kamp, M., & Dou, Q. (2021) Fedbn: Federated learning on non-iid features via local batch normalization, arXiv preprint 202.:::022.062.2.
[42] Fallah, A., Mokhtari, A., & Ozdaglar, A. (2020) Personalized federated learning: A meta-learning approach. In Advances in Neural Information Processing Systems.
[43] Pan, S. & Yang. Q. (2010) A survey on transfer learning, IEEE Transactions on knowledge and data engineering 22(10): 1345-1359.
[44] D., & Wang. J. (2019) Fedmd: Hcterogenous federated learning via model distillation. arXiv preprint arXiv:1910.05:
[45] He, C.,, Annavaram, M., & Avestimehr, (2020) Group knowledge transfer: Federated learning of large cnns at edge. Advances in Neural Information Processing Systems 33:14068-14000
[46] Zhang, S.. Liu, M., & Yan, (2020) The diversified ensemble neural network. Advances in Neurul Information Processing Systems 13011-6111
[47] Zhu, Z., Hong. J., & Zhou, J. (2021) Data- knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning, pp. 12878- 12889.
[48] Zhang, L.,, Shen, L., Ding, L., Tao, D., & Duan, L. Y. (2022) Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10174-10183.
[49] Zhang. J., Chen, C., Li. B.. Lyu, L., Wu, S., Ding, S., Shen, C. & Wu, C. (2022) DENSE: Data-Free One-Shor Federated Learning. In Advances in Neural Information Processing Systems.
[50] Heinbaugh, C. E., Luz- Ricca, E., & Shao, (2023) Data-Free One-Shot Federated Learning Under Very High Statistical Hetcrogeneity. In The Eleventh International Conference on Learning Representations.
[51] Fang, G., Song, J., Wang, X., Shen, C., Wang. X., & Song, M. (2021) Contrastive model inversion for data-free knowledge distillation. arXiv preprint arXiv:2105.08522
[52] Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha N. K., & Kautz, J. (2020) Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724.
[53] Do, K., Le, T. H., Nguyen, D., Nguyen, D., Harikumar, H., Tran, T., Rana S., & Venkatesh, S. (2022) Mo- mentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation. Advances in Neural Information Processing Systems 35: 10055-10067.
[54] Binici, K., Pham, N. T., Mitra, T., Leman, K. (2022) Preventing catastrophic forgetting and distribution mismatch in knowledge distillation via synthetic data. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 663-671.
[55] Binici, K., Aggarwal, S.. Pham, N. T., Leman. K., & Mitra, T. (2022) Robust and resource-efficient data-free knowledge distillation by generative pscudo replay. In Proceedings of the AAAI Conference on Artificial Intelligence 36(6):68896606.
[56] Patel, G., Mopuri, K. R., & Qiu, Q. (2023) Learning to Retain while Acquiring: Combating Distribution- Shift Adversarial Data-Free Knowledge Distillation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7786-7794.
[57] Odena, A., Olah, C., & Shlens, J. (2017) Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pp. 2642-2651.
[58] Kodali, N., Abernethy. J., Ja,,, J., & Kira, Z. (2017) On convergence and stability of gans, arXiv preprint arXiv:1705.0722
12

---

[59] Yoo, J., Cho, M., Kim, T.. & Kang, U. (2019) Knowledge extraction with observable data, Advances in Neural Information Processing Systems 60.
[60] Luo, I.., Sandler, M., Lin, Z., Zhmoginov, A., & Howard, A. (2020) Large-scale generative data-free distillation. arXiv preprint arXiv:2012.0557..
[61] Nayak, G. K., Mopuri, K. R., Shaj. V., Radhakrishnan, V. B., & Chakraborty, A. (2019) sero-shot knowledge distillation in deep networks. In International Conference Machine Learning, pp. 4743- 4751.
[62] Wang. Z. (2021) Date-free knowledge distillation with soft targeted transfer set synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence 35(11): 10245-10233 10255.
[63] Liu, R.,. Wu, F., Wu, C., Wang, Y., Lyu, L.., Chen, H., & Xie, X. (2022) No one left behind: Inclusive federated learning over heterogeneous devices. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3398-3406.
[64] Kim, M., Yu, S., Kim, S., Moon, S. M. (2023) DepthFL: Depthwise Federated Learning for Heteroge- neous Clients. In The Eleventh International Conference on Learning Representations.
[65] Singhal, K.. Sidahmed, H., Garrett, Z., Wu, S., Rush, J., Prakash, S. (2021) Federated reconstruction: Partially local federated learning. Advances in Neural Information Processing Systems 34: 1120-1 11232.
[66] Zhang, H., Bosch, J., & Olsson, H. H. (2021) Real-time end-to-end federated learning: An automotive case study. In 2021 IEEE 45th Annual Computers, Software, and Applications Conference, pp. 459-468.
[67] Li, Z., He, Y., Yu. H. Kang, J., Li, X., Xu, Z., Niyato, D. (2022) Data heterogeneity- federated learning via group client selection in industrial iot, IEEE Internet of Things Journal 9(8)): 17844-17857,
[68] Zeng, S., Li, 7., Yu, H., Zhang. Z., Luo, I.,, Li, B., & Niyato, (2023) Hfedms: Heterogeneous federated learning with memorable data semantics in industrial metaverse. IEEE Transactions on Cloud Computing.
[69] Xiao, H., Rasul, K., & Vollgraf, R. (2017) Fashion-mnish a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.0774
[70] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng. A. (2011) Reading digits in natural images with unsupervised feature learning.
[71] Krizhevsky, A., Hinton, G. (2009) Learning multiple layers of features from tiny images.
[72] Hc, K., Zhang, X., Ren, S., & Sun, (2016) Deep residual lcarning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.
[73] Bossard, L., Guillaumin, M., & Van Gool, (2014) Food-101 mood-101-minte 101-mining discriminative components with random forests. In Computer Vision-FCCCV 2014: 13th European Conference, pp. 446-461.
[74] Dwork, C. (2008) Differential privacy: survey of results. In Theory and Applications of Models of Computation, pp. 1-19.
[75] Geyer, R. C., Klein, T., Nabi, M. (2017) Differentially private federated learning: A client level perspective, arXiv preprint arXiv: 171207757.
[76] Cheng. A., Wang, P., Zhang, X. S., Cheng, (2022) Differentially Private Federated Learning with Local Regularization and Sparsification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10122-10131.
[77] Ma, J., Naas, S. A., Sigg, S., & Lyu, X. (2022) Privacy-preserving federated learning based on multi-key homomorphic encryption. International Journal of Intelligent Systems 37(9): 5880-5901.
[78] Zhang. L., Xu, J.. Vijayakumar, P., Sharma, P. K., & Ghosh, U. (2022). Homomorphic encryption-ban based privacy-rreeeving federated learning in iot-enabled healthcare system. IEEE Transactions on Network Science and Engineering.
13


Reviews:
[{"review_id": "mikXWI6vTA", "reviewer": "Reviewer_Yxnp", "summary": "This paper proposes a new method called DFRD for robust and privacy-constrained Federated Learning (FL) in the presence of data and model heterogeneity. DFRD uses a conditional generator to approximate the training space of local models and a data-free knowledge distillation technique to overcome catastrophic forgetting and accurately extract knowledge from local models. The experiments show that DFRD achieves performance gains compared to baselines.", "questions": "1. As this method already uses EMA on global model (G) to get \\tilde{G}, why does it need another step to find KL divergence on both G and \\tilde{G}?\n\n2 How the w, \\rho, and \\delta play roles in data heterogeneity?\n\n3 There is a line of work on KD for FL aggregation. Could the author justify why \" FedFTG [48] and DENSE [49], which are the most relevant methods to our work?\" comparing the other related work mentioned in the manuscript (such as [27], [40], [47])  and Wu et al. listed below:\n\nWu, C., Wu, F., Lyu, L., Huang, Y., & Xie, X. (2022). Communication-efficient federated learning via knowledge distillation. \n", "limitations": "Limitations are discussed in appendix", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is easy to follow\n\n2. The overall idea seems novel\n\n3. The authors justify the necessities of loss functions and validate their effectiveness in the experiment.", "weaknesses": "1. This work is well-engineered, comprising multiple components and hyperparameters. Although stability testing was performed on the same dataset, the optimal hyperparameters for different datasets appear to be unstable (see Fig 5). \n\n2. Although the authors propose loss functions aiming to improve fidelity and diversity, the synthetic images provided in the appendix are still far from achieving these goals. It is also questionable whether such characteristics are really necessary for downstream KD and ensembling.\n\n3. The EMA strategy to avoid forgetting is not new in continuous learning, so the novelty of this technique is limited in this paper.\n\n4. Based on the presented learning curves, the models do not seem to converge after 100 communication rounds for both the proposed and benchmark methods. This undermines the persuasiveness of the results reported in the tables.\n\n5. The overall performance is very low, e.g., less than 20% accuracy for 10-class classification. I have concerns about the practicality of the method given its complexity and unsatisfactory performance in tackling the challenging non-iid setting.\n\n6. I appreciate the authors' efforts to conduct numerous experiments, but how the settings overlap between experiments is not clear. It would be helpful if the authors could provide a table to show the connections. For example, I found it difficult to relate the values in the figures to the numbers in the tables. Ideally, the same experimental setting would result in the same number.\n\nMinor comments:\n\nThe pseudocode provided in the appendix is not well organized. There seems to be a convoluted operation between server and client.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer Yxnp for positive comments and precious feedback on our work. We respond to specific comments below.\n\nQ1: This work is well-engineered, comprising multiple components and hyperparameters. Although stability testing was performed on the same dataset, the optimal hyperparameters for different datasets appear to be unstable (see Fig 5). \n\nR1: Thanks for pointing out the details of Fig 5. We argue that there is no correlation between optimal hyperparameters for different datasets. Intuitively, different datasets correspond to different learning tasks, so it is difficult for any method (including DFRD) to ensure consistent optimal hyperparameters for varying learning tasks during practical experiments.\n\nQ2: Although the authors propose loss functions aiming to improve fidelity and diversity, the synthetic images provided in the appendix are still far from achieving these goals. It is also questionable whether such characteristics are really necessary for downstream KD and ensembling.\n\nR2: These images are necessary for downstream KD and ensembling, evidenced by Tables 1 and 2, that is, these images significantly augment the global model. Also, we present synthetic images under different diversity constraints (mul, add, cat, n-cat, none) in the Appendix (see Fig.14 - 18), where mul is proposed by us (see Fig. 14). The synthetic images exhibited in Fig. 14 clearly outperform other synthetic images (see Figs. 15-18) in terms of fidelity and diversity. \n\nQ3: The EMA strategy to avoid forgetting is not new in continuous learning, so the novelty of this technique is limited in this paper.\n\nR3: We agree that the EMA strategy is not new in continuous learning, but our innovation is that we are the first effort to apply the EMA strategy to heterogeneous FL with data-free knowledge distillation.\n\nQ4: The overall performance is very low, e.g., less than 20% accuracy for 10-class classification. I have concerns about the practicality of the method given its complexity and unsatisfactory performance in tackling the challenging non-iid setting.\n\nR4: In our work, we set up extensive heterogeneity settings, including data heterogeneity and model heterogeneity. In terms of data heterogeneity (i.e., non-iid settings), the accuracies of FedAvg and FedAvg+DFRD (+DENSE and +FedFTG) are much greater than 20% for different non-iid settings (including the challenging non-iid setting, i.e., $\\omega=0.01$) under model homogeneity, see Table 1. In Table 2, we specifically investigate challenging scenarios characterized by both data and model heterogeneity, resulting in a considerable number of accuracies falling below the 20% threshold, especially on Tiny-ImageNet and FOOD101. Notably, the accuracy of DFRD is greater than 20% on SVHN and CIFAR-10 in most cases. Therefore, we are confident in the practicality of our method.\n\nQ5: I appreciate the authors' efforts to conduct numerous experiments, but how the settings overlap between experiments is not clear. It would be helpful if the authors could provide a table to show the connections. For example, I found it difficult to relate the values in the figures to the numbers in the tables. Ideally, the same experimental setting would result in the same number.\n\nR5: Due to the space limitation of the main paper, we detail experimental settings in Appendix E. We also agree with the reviewer's comments and will update them in the revised paper.\n\nQ6: The pseudocode provided in the appendix is not well organized. There seems to be a convoluted operation between server and client.\n\nR6:  There are no convoluted operations between server and clients. For ease of understanding, we will add a description of the pseudocode in the revised paper.\n\nQ7: As this method already uses EMA on global model ($G$) to get $\\tilde{G}$, why does it need another step to find KL divergence on both $G$ and $\\tilde{G}$?\n\nR7: In our paper, $\\tilde{G}$ stores previous knowledge learned from the local models and is used as a complement to $G$ in DFRD. $\\tilde{G}$ does not contain information about $G$ in the latest communication round. See lines 19 to 22 in Algorithm 1 for detail. Therefore, an additional step is required to compute the KL divergence for both $G$ and $\\tilde{G}$.\n\nQ8: How the $\\omega$, $\\rho$, and $\\sigma$ play roles in data heterogeneity? \n\nR8: In our paper, Dirichlet process $Dir(\\omega)$ is utilized to assign training data to each client, which is frequently used in existing federated learning works [34,37, 38]. Here, $\\omega$ is the concentration parameter of $Dir(\\omega)$ and smaller $\\omega$ corresponds to stronger data heterogeneity. We provide the visualizations of the data partitions for the six datasets at varying $\\omega$ values, which can be found in Fig.6 and 7 in Appendix. In addition, $\\rho$ and $\\sigma$ are parameters related to model heterogeneity rather than data heterogeneity, see Appendix D for detail.\n\nQ9: There is a line of work on KD for FL aggregation. Could the author justify why \" FedFTG [48] and DENSE [49], which are the most relevant methods to our work?\" comparing the other related work mentioned in the manuscript (such as [27], [40], [47]) and Wu et al. listed below: Wu, C., Wu, F., Lyu, L., Huang, Y., & Xie, X. (2022). Communication-efficient federated learning via knowledge distillation.\n\nR9: To ensure fair comparisons, we neglect the comparison with methods that require to download auxiliary models or datasets, such as RHFL[27], FedDF[40] and FedGen [47] and the mentioned paper.  Also, we'll add the mentioned paper as one of our references.\n"}, {"review_id": "Xl2roXK1yN", "reviewer": "Reviewer_1PRp", "summary": "In the presented paper, the authors lay out a strategy for fine-tuning a global model, specifically within the Heterogeneous Federated Learning setting. To transfer knowledge from the individual local models (identified as 'teachers') to the global model (or 'student'), the authors construct and train a generative model that is capable of producing pseudo samples, inspired from the Data-Free Knowledge Distillation literature. \n\nFurthermore, the authors suggest preserving an exponentially moving copy of this generative model. This is specifically designed to provide pseudo-samples from prior communication rounds. The overarching goal of this method is to protect the global model from catastrophic forgetting - a substantial problem in machine learning where a model, after learning new information, forgets the old. By accessing these pseudo-samples from previous rounds, the global model can effectively reinforce its learning and maintain a more comprehensive understanding over the communication rounds.\n\nLastly, the authors introduce a system that employs dynamic weighting and label sampling. This is proposed to enhance the precision in extracting knowledge from local models. By implementing this technique, it is intended that the knowledge transfer process will be more accurate and effective. \n", "questions": "1. I am interested in gaining a deeper understanding of the process behind the observations and interpretations, specifically those pertaining to transferability, decision boundaries, and the positioning of synthetic data-points (L165-180). Could the authors elaborate on the methodology or rationale that led to these interpretations?\n\n2. Furthermore, the visualization provided in Figure 2 has caught my attention. I would appreciate it if the authors could elucidate how this figure was constructed. Does this visual representation arise from actual data obtained through a small-scale experiment, or is it a conceptual illustration reflecting your interpretations?\n\n3. Regarding equation 5, I noticed that the KL divergence loss is applied directly to the logits (pre-softmax) features, as opposed to the prediction distribution resulting from the application of softmax to the logits. Could the authors provide a comprehensive explanation for this choice? Does this approach hold any specific advantages or implications for the overall results of your study?\n", "limitations": "The authors have thoroughly listed out the limitations and broader impacts in the appendix.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. In this manuscript, the authors introduce an elegant yet straightforward technique to avert catastrophic forgetting, an issue that often arises in machine learning models, by maintaining a EMA generator. Remarkably, the method appears to maintain a consistent spatial footprint, equivalent to the dimensions of the generator models, throughout the various communication cycles. This ingenious mechanism facilitates the generation of pseudo samples from preceding rounds, thereby extending its usefulness and practicality in real-world applications.\n\n2. Moreover, the depth of empirical examination carried out in this paper is commendable. The authors have meticulously analyzed the performance of their technique, employing a thorough sensitivity analysis and ablation studies to evaluate the impact of the individual components of their methodology. This level of rigor and detail, often overlooked, attests to the quality of the research conducted and significantly boosts the credibility of their findings.\n", "weaknesses": "1. The manuscript offers a promising exploration into mitigating catastrophic forgetting in the context of Federated Heterogeneous Learning. However, as a reviewer, I find that the examination of catastrophic forgetting and its severity within this specific context could have been more in-depth. Works such as those by Binci et al. [1,2] and Patel et al. [3], which tackle catastrophic forgetting in adversarial settings, provide comprehensive empirical evidence via monitoring student learning curves. Such an approach would have deepened the current study's investigation, allowing readers to better appreciate its significance.\n\n2. The current research draws substantially from existing methodologies, including those used in FedFTG [4] and DENSE [5], particularly adopting the data-free knowledge transfer framework from Do et al. [6]. Consequently, it becomes challenging to discern the novel contribution of this paper. Greater emphasis on aspects like Dynamic Weighting and Label Sampling, currently found in the appendix, would have added more substance to the primary narrative of the manuscript. Additionally, a thorough exploration of the obstacles faced when adapting the adversarial data-free knowledge transfer framework [6] to a Heterogeneous FL setting would have enriched the study.\n\nMinor:\n1. In the related works section, the absence of adversarial DFKD methodologies, such as those in [1,2,3], which also aim to prevent catastrophic forgetting, is a notable omission. I suggest that the authors provide a clear rationale for their preference for the DFKD framework in [6] over the others ([3] could be excluded due its recentness), clarifying this would enhance the comprehensibility of the manuscript.\n\n\nReferences:\n\n[1] Kuluhan Binci, Nam Trung Pham, Tulika Mitra, and Karianto Leman. \"Preventing catastrophic forgetting and distribution mismatch in knowledge distillation via synthetic data.\" In WACV, 2022.\n\n[2] Kuluhan Binci, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. \"Robust and resource-efficient data-free knowledge distillation by generative pseudo replay.\" In AAAI, 2022.\n\n[3] Gaurav Patel, Konda Reddy Mopuri, and Qiang Qiu. \"Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation.\" In CVPR, 2023.\n\n[4] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. \"Fine-tuning global model via data-free knowledge distillation for non-iid federated learning.\" In CVPR, 2022.\n\n[5] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. \"Dense: Data-free one-shot federated learning.\" In NeurIPS, 2022.\n\n[6] Kien Do, Thai Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, and Svetha Venkatesh. \"Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation.\" In NeurIPS, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer 1PRp for positive comments and precious feedback on our work. We respond to specific comments below.\n\nQ1: The manuscript offers a promising exploration into mitigating catastrophic forgetting in the context of Federated Heterogeneous Learning. However, as a reviewer, I find that the examination of catastrophic forgetting and its severity within this specific context could have been more in-depth. Works such as those by Binci et al. [1,2] and Patel et al. [3], which tackle catastrophic forgetting in adversarial settings, provide comprehensive empirical evidence via monitoring student learning curves. Such an approach would have deepened the current study's investigation, allowing readers to better appreciate its significance.\n\nR1: We thank the reviewer's constructive suggestion. We will refer to [1], [2] and [3] as references, thus deepening the investigation that addresses the problem of catastrophic forgetting in adversarial settings. Also, compare to [1], [2] and [3], [6] is more memory efficient, adapts better to the student update, and is more stable for a continuous stream of synthetic samples. That's why we prefer [6]. We will update them in the revised paper, thereby enhancing the comprehensibility of the manuscript.\n\nQ2: I am interested in gaining a deeper understanding of the process behind the observations and interpretations, specifically those pertaining to transferability, decision boundaries, and the positioning of synthetic data-points (L165-180). Could the authors elaborate on the methodology or rationale that led to these interpretations?\n\nR2: As we proceeded with our work, we first delved into existing related works on transferability including references [4], [5], and [6], among others. Then, during the empirical study, we found that the generator produces two types of synthetic data with yellow and purple circles in Fig. 2 (c), which can mislead the generator. For example, the conditional generator $G(\\cdot)$ takes label information (e.g., $y=1$) as one of the inputs to generate synthetic data s. However, the ensemble model’s inferred output based on s results in label 2, i.e., $y=2$. This inconsistency degrades the quality of synthetic data generated by $G(\\cdot)$, thereby negatively affecting the global model's performance, as shown in Table 3. The aforementioned process and principles elucidate how we led to these interpretations.\n\nQ3: Furthermore, the visualization provided in Figure 2 has caught my attention. I would appreciate it if the authors could elucidate how this figure was constructed. Does this visual representation arise from actual data obtained through a small-scale experiment, or is it a conceptual illustration reflecting your interpretations?\n\nR3: Figure 2 is a conceptual illustration to give the reader a better understanding of our interpretations. And Figure 2 was carefully designed based on our findings and observations during our empirical study.\n\nQ4: Regarding equation 5, I noticed that the KL divergence loss is applied directly to the logits (pre-softmax) features, as opposed to the prediction distribution resulting from the application of softmax to the logits. Could the authors provide a comprehensive explanation for this choice? Does this approach hold any specific advantages or implications for the overall results of your study?\n\nR4: In our work, KL divergence loss is applied to the prediction distribution resulting from the application of softmax to the logits, as opposed to the logits (pre-softmax) features. Both softmax and log-softmax operations are considered within the KL divergence (please see our code). We will clear them further in the revised paper for ease of understanding.\n"}, {"review_id": "L0FofmZjGJ", "reviewer": "Reviewer_XBHz", "summary": "This paper studies how to learn a robust global model in the data-heterogeneous and model-heterogeneous FL, by proposing a data-free knowledge distillation method. The proposed method is evaluated on six image classification datasets and outperforms compared methods.", "questions": "- Why do DFRD and +DFRD show different trends in Table 1? For DFRD, L.acc is higher than G.acc, but for +DFRD, L.acc is lower than G.acc.\n- Is this method applicable to other model heterogeneity cases? Such as different network architecture?\n- Why replace FMNIST in table.1 with TinyImagnet in table.2?\n", "limitations": "The authors discussed the limitations with potential solutions, and the broader impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies data heterogeneity and model heterogeneity in FL, which are relevant and important topics.\n- The motivation for solving both data and model heterogeneity are well explained.\n- The proposed method outperforms compared methods.\n- The authors conduct many ablation studies.\n", "weaknesses": "- It is not clear how to calculate the term \\tau_{i,y}, which is important to adjust the synthetic data.\n- Only intuitive toy visualizations are shown. It lacks experimental visualizations to support the mentioned “Fidelity” of the proposed method.\n- The experiments only validate the model heterogeneity in terms of width.\n- The results presentation can be improved. The table is small, and some symbols are confusing.\n- The experiments are mainly conducted on datasets with small image sizes （e.g., 32*32 or 64*64), lack of validation on large datasets.\n- The global accuracy is sensitive to different values of \\beta, and the standard deviation is large.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer XBHz for positive comments and precious feedback on our work. We respond to specific comments below.\n\nQ1: It is not clear how to calculate the term $\\tau_{i,y}$, which is important to adjust the synthetic data.\n\nR1: We delve into the calculation of $\\tau_{i,y}$ in the paper, see lines 231-236 and Appendix F.\n\nQ2: Only intuitive toy visualizations are shown. It lacks experimental visualizations to support the mentioned “Fidelity” of the proposed method.\n\nR2: We visualize the synthetic data of generators with different diversity constraints over SVHN, CIFAR-10, and CIFAR-100 in Figures 14-18 in the Appendix, where Figure 14 is the visualization results of our proposed method. From Figure 14, we can observe that the synthetic data generated by the generators exhibits significant differences among the distinct classes, demonstrating the high fidelity of the proposed method. \n\nQ3: The experiments are mainly conducted on datasets with small image sizes （e.g., 3232 or 6464), lack of validation on large datasets.\n\nR3: We agree with this point. However, all our experiments are conducted on one NVIDIA Tesla A100 GPU with 80Gb of memory. We have tried datasets with image sizes of 128x128 and 256x256 in our experimental device, but running DFRD and baselines on these datasets would incur extreme time costs. So, we choose datasets with small image sizes for our experimental study.\n\nQ4: Why do DFRD and +DFRD show different trends in Table 1? For DFRD, L.acc is higher than G.acc, but for +DFRD, L.acc is lower than G.acc.\n\nR4: We explain the above phenomena in the paper, please see lines 271-281.\n\nQ5: Is this method applicable to other model heterogeneity cases? Such as different network architecture?\n\nR5: Our method can be applied to other cases of model heterogeneity, such as different network architectures. \n\nQ6: Why replace FMNIST in table.1 with TinyImagnet in table.2?\n\nR6: On the one hand, we want to study the performances of our method and baselines on datasets with more difficult classification tasks. On the other hand, Tables 1 and 2 are about data heterogeneity and model heterogeneity, respectively, so there is no need to force a fixed dataset. Another important reason is the space limitation of the main paper.\n\nOther suggestions provided by the reviewer to help improve the paper will be corrected in the revised paper.\n"}, {"review_id": "JFtGttvL1X", "reviewer": "Reviewer_i49s", "summary": "The paper considers learning a robust global model in heterogeneous federated learning (FL). It aims to support scenarios of both data-heterogeneous, where data distributions among clients are non-IID (identical and independently distributed), and model-heterogeneous, where the model architecture among clients is different. To this end, the authors propose a data-free knowledge distillation (DFKD) method, which utilizes a conditional generator to generate synthetic data, which simulates the local models’ training space with respect to fidelity, transferability, and diversity. Besides, the authors utilize an exponential moving average method to mitigate the catastrophic forgetting of the global model. Experiments on real-world image classification datasets are conducted to evaluate the performance of the proposed approach. ", "questions": "Q1. For the data heterogeneity considered in this work, what distribution the final test dataset follows? Is it the global distribution over all clients’ local data distributions?\n\nQ2. What is the purpose of training a generator on the server for the partial training-based methods? And how is it capture the training space of various clients’ local models? More elaboration would be helpful. \n\nQ3. Regarding the conditional generator proposed in this paper, the three characteristics (fidelity, transferability, and diversity) mainly follow those defined in [48] [49]. The contribution seems to only replace the generator with a conditional generator, which looks incremental. What is the new technical contribution in the proposed method?\n\nQ4. The EMA method is proposed to mitigate the catastrophic forgetting of the global model. However, since the global model already captures the knowledge of local historical models, it should not forget the knowledge if it is useful. I wonder why this problem exists. Is it related to how the clients extract the local models from the aggregated global model? Besides, how to set the averaging parameters \\lambda and \\alpha? \n\nQ5. It is mentioned in the paper that the generated synthetic data should be visually distinct from the real data for privacy protection. I wonder whether there is a metric to measure this. Also, if the objective is to capture the training space of local models, would it be possible to recover the clients’ local data distributions?\n\nQ6. The proposed DFKD method is claimed to be able to address the non-IID problem; then, it would be better to compare it with solutions such as FedProx, FedNova, etc. \n\nQ7. In Table 1 and Table 2, do the two rows of each method represent the local test accuracy and global test accuracy, or the opposite? The improvement seems to be very small compared to DENSE and FedFTG, for example, less than 1% on FMNIST. \n\nQ8. In Figure 3a, why the global accuracy of DENSE is too low? E.g., only 10% accuracy. \n\nQ9. In Tables 3-5, why the accuracies of the three datasets are too low? Are they well-trained or converged? \n", "limitations": "L1. It is unclear whether the proposed approach can handle the non-IID setting well, as one of the goals is to support data-heterogeneous. The authors may need to compare DFKD with non-IID solutions when the model architecture among clients is the same. \n\nL2. The clients may not obtain useful local models due to they hold non-IID data distributions, and the server aggregates the sub-models over the global data distribution.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "\n1. The problem of heterogeneous federated learning is interesting and important. \n\n2. A data-free knowledge distillation method with a detailed analysis of the conditional generator w.r.t. three characteristics. \n\n3. Experiments on six real-world datasets are conducted. \n", "weaknesses": "1. The conditional generator in the proposed approach is not well motivated and elaborated. \n\n2. The contribution of the proposed losses w.r.t. the three characteristics is incremental. \n\n3. The choice of hyper-parameters in the exponential moving average method is unclear. \n\nW4. Some of the experimental results are hard to comprehend. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer i49s for positive comments and precious feedback on our work. We respond to specific comments below.\n\nQ1: For the data heterogeneity considered in this work, what distribution the final test dataset follows? Is it the global distribution over all clients’ local data distributions?\n\nR1: We detailed the setup of the test dataset in the paper, see lines 263-269.\n\nQ2: What is the purpose of training a generator on the server for the partial training-based methods? And how is it capture the training space of various clients’ local models? \n\nR2: The purpose of training a generator on the server for the partial training-based methods is to adapt the FL system to model-heterogeneous scenarios, please see lines 41-53 and 105-119. In addition, we detail how the generator captures the training space of various clients’ local models in lines 128-209.\n\nQ3: Regarding the conditional generator proposed in this paper, the three characteristics (fidelity, transferability, and diversity) mainly follow those defined in [48] [49]. The contribution seems to only replace the generator with a conditional generator, which looks incremental. What is the new technical contribution in the proposed method?\n\nR3: We acknowledge that our study of generators follows the definitions in [48, 49] in terms of fidelity, transferability, and diversity, but they do not thoroughly study the training of the generator in these three characteristics. Therefore, we propose a more effective loss objective to address their shortcomings. Please see lines 128-209 and Experiments section for detail. Moreover, our goal is to guide the generator to better capture knowledge of the local models, which cannot simply be viewed as replacing the generator with a conditional generator, since a conditional generator also is used in [48].\n\nQ4: I wonder why catastrophic forgetting problem exists. Is it related to how the clients extract the local models from the aggregated global model? Besides, how to set the averaging parameters $\\lambda$ and $\\alpha$?\n\nR4:  We elaborated on why catastrophic forgetting exists, see lines 211-220. Furthermore, $\\lambda$ and $\\alpha$ are hyperparameters that require careful fine-tuning in the experiments. We empirically study their variations on SVHN and CIFAR-10, see lines 367-383.\n\nQ5: It is mentioned in the paper that the generated synthetic data should be visually distinct from the real data for privacy protection. I wonder whether there is a metric to measure this. Also, if the objective is to capture the training space of local models, would it be possible to recover the clients’ local data distributions?\n\nR5: To our knowledge, in the current research on federated learning and data-free knowledge distillation, there are no metrics for quantifying synthetic data quality. Therefore, existing works [49, 53] use visualization of the generator's output to discriminate differences visually. Also, the generator can only partially capture the local data distribution, which is impractical to recover the clients’ local data distributions [35, 36, 53, 56]. This is because the generator does not have access to real data, it can only approximate it with the help of teacher models.\n\nQ6: The proposed DFKD method is claimed to be able to address the non-IID problem; then, it would be better to compare it with solutions such as FedProx, FedNova, etc.\n\nR6: FedProx, FedNova and so on are regularization-based methods. To ensure fair comparisons, we omit the comparison with these methods. Notably, we chose FedFTG and DENSE as baselines since they are both DFKD methods and also address the non-IID problem (please see [48] and [49] for detail).\n\nQ7: In Table 1 and Table 2, do the two rows of each method represent the local test accuracy and global test accuracy, or the opposite? The improvement seems to be very small compared to DENSE and FedFTG, for example, less than 1% on FMNIST.\n\nR7: We elaborated on what each of the two rows represents in lines 263-269. Moreover, the less than 1% improvement of our method on FMNIST compared to DENSE and FedFTG is due to the simplicity of the classification task for FMNIST. Specifically, the reason why they perform well is that FedAvg can achieve good test accuracy on FMNIST, which provides excellent initialization of the global model in each round when DENSE, FedFTG and our method are used as fine-tuning methods. However, our method significantly outperforms DENSE and FedFTG on FMNIST when they act as data-free methods rather than fine-tuning methods, see Experimental Settings and Table 1 for detail.\n\nQ8: In Figure 3a, why the global accuracy of DENSE is too low? E.g., only 10% accuracy.\n\nR8: During our empirical study, we explored the reason for the low global accuracy when DENSE is used as a data-free method rather than a fine-tuning method. We argue that DENSE is very sensitive to the initialization of the global model. Good global model initialization can greatly improve the performance of DENSE. For example, FedAvg+DENSE can achieve good accuracies, see Table 1.\n\nQ9: In Tables 3-5, why the accuracies of the three datasets are too low? Are they well-trained or converged?\n\nR9: The low accuracy rates in Tables 3-5 are related to our experimental setup. We perform the ablation study in both data and model heterogeneous settings, please see lines 299-303 and the Experimental Setting section for detail. Also, they’re well-trained.\n\nQ10: It is unclear whether the proposed approach can handle the non-IID setting well, as one of the goals is to support data-heterogeneous. The authors may need to compare DFKD with non-IID solutions when the model architecture among clients is the same.\n\nR10: Our method proposes Dynamic Weighting and Label Sampling to handle the non-IID setup, see lines 231-236 and Appendix F for detail.\n"}, {"review_id": "S84ukZD1ws", "reviewer": "Reviewer_deo9", "summary": "The overall theoretical framework of the paper can be regarded as an extension of the FedFTG theory. Building upon the server-side learning of challenging samples, it incorporates the generation of data from the previous time step by the generator to prevent catastrophic forgetting in the global model. The experimental section of the paper is substantial, and it includes comparative analyses through ablation experiments to evaluate the effectiveness of various losses. Thus, the paper makes a good contribution.", "questions": "How the experiments simulate the model heterogenous? \nSuggestions : The paper lacks visual experiments on samples to enhance the credibility of the theoretical training of the generator.\nParameter omega be used both for generator and Dirichlet process.", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: In comparison to FedFTG, the originality of the paper lies in Section 3.2, where a diverse generator is utilized by leveraging the parameter updates from the previous time step to prevent catastrophic forgetting in the model.\n\nQuality: In comparison to FedFTG, the paper conducts thorough comparative experiments and ablative experiments that elucidate the roles of various losses, indicating a certain level of quality.\n\nClarity: The framework diagram, Figure 1, in the paper provides a clear explanation of the algorithm's flow.\n", "weaknesses": "3.1 The impact of knowledge transfer on synthesizing different datasets lacks theoretical support and experimental evidence, as demonstrated by the decision boundary theory presented in Figure 2. There is a lack of clear relationship between synthetic data and the decision boundaries of the student and teacher models. Since the Generator satisfies Equation 2, it is capable of generating samples (blue) on the decision boundary of the teacher model. However, it is not clearly explained under what circumstances it would generate samples beyond the decision boundaries (yellow and purple) without further elaboration.\n\n4.3 The originality of the theoretical part of the paper is relatively weak, as using parameters from the previous time step to enhance samples appears more like a heuristic approach rather than a theoretical one.\n\nThe algorithm/objective is not concrete. For Eq(5), the objective is to minimization for which parameters?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate Reviewer deo9 for positive comments and precious feedback on our work. We respond to specific comments below.\n\nQ1: The impact of knowledge transfer on synthesizing different datasets lacks theoretical support and experimental evidence, as demonstrated by the decision boundary theory presented in Figure 2. There is a lack of clear relationship between synthetic data and the decision boundaries of the student and teacher models. Since the Generator satisfies Equation 2, it is capable of generating samples (blue) on the decision boundary of the teacher model. However, it is not clearly explained under what circumstances it would generate samples beyond the decision boundaries (yellow and purple) without further elaboration.\n\nR1: First, we would like to state that Figure 2 is provided as a conceptual illustration to give the reader a better understanding of our interpretations. And Figure 2 was carefully designed based on our findings and observations. In the empirical study, if the generator only satisfies Equation 2, it mainly generates synthetic data with red circles, thus making it difficult to transfer knowledge from teacher to student. Therefore, an adversarial manner is needed to generate synthetic samples with black circles, i.e., Transferability of the generator. Please see lines 128-180 and lines 211-215 for detail. Also, we clearly explain the case of generating synthetic samples with yellow and purple circles, please see lines 181-187.\n\nQ2: The algorithm/objective is not concrete. For Eq(5), the objective is to minimization for which parameters?\n\nR2: For Eq. (5), the objective is to minimize the parameters of the generator. We illustrate this in line 205 and Fig. 1.\n\nQ3: How the experiments simulate the model heterogenous? Suggestions : The paper lacks visual experiments on samples to enhance the credibility of the theoretical training of the generator. Parameter omega be used both for generator and Dirichlet process.\n\nR3: In the paper, we provide a detailed description of how to simulate heterogeneous models, see lines 257-259 and Appendix D. Additionally, we visualize synthetic samples of the generator in Figures 14-18 of Appendix.\n"}]

Author rebuttal:
We thank all reviewers for their thoughtful, constructive, and positive review of our manuscript. We are encouraged to hear that the reviewers found the DFRD method we present to be interesting and practical (Reviewers i49s, 1PRp), and thoroughly-evaluated (Reviewers Yxnp, 1PRp, XBHz, i49s, deo9). Meanwhile, they view our methodology as novel (Reviewers Yxnp, 1PRp) and our manuscript as well-written (Reviewers Yxnp, 1PRp). In response to the feedback, we provide detailed responses to address each reviewer's concerns below.

Meta-review: Reviewers found this paper exciting and unanimously recommended accept.
Decision: Accept (poster)