Title: DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Data Heterogeneity

Model Heterogeneity

Data-Free Distillation
Abstract: Federated Learning (FL) is a privacy-constrained decentralized machine learning paradigm in which clients enable collaborative training without compromising private data. However, how to learn a robust global model in the data-heterogeneous and model-heterogeneous FL scenarios is challenging. To address it, we resort to data-free knowledge distillation to propose a new FL method (namely DFRD).
DFRD equips a conditional generator on the server to approximate the training space of the local models uploaded by clients, and systematically investigates its training in terms of fidelity, transferability and diversity. To overcome the catastrophic forgetting of the global model caused by the distribution shifts of the generator across communication rounds, we maintain an exponential moving average copy of the generator on the server.  Additionally, we propose dynamic weighting and label sampling to accurately extract knowledge from local models. Finally, our extensive experiments on various image classification tasks illustrate that DFRD achieves significant performance gains compared to SOTA baselines.

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning

Abstract

Federated Learning (FL) is a privacy-constrained decentralized machine learning paradigm in which clients enable collaborative training without compromising private data. However, how to learn a robust global model in the data-heterogeneous and model-heterogeneous FL scenarios is challenging. To address it, we resort to data-free knowledge distillation to propose a new FL method (namely DFRD). DFRD equips a conditional generator on the server to approximate the training space of the local models uploaded by clients, and systematically investigates its training in terms of fidelity , transferability and diversity . To overcome the catastrophic forgetting of the global model caused by the distribution shifts of the generator across communication rounds, we maintain an exponential moving average copy of the generator on the server. Additionally, we propose dynamic weighting and label sampling to accurately extract knowledge from local models. Finally, our extensive experiments on various image classification tasks illustrate that DFRD achieves significant performance gains compared to SOTA baselines. Our code is here: https://anonymous.4open.science/r/DFRD-0C83/.

1 Introduction

With the surge of data, deep learning algorithms have made significant progress in both established and emerging fields [1-3]. However, in many real-world applications (e.g., mobile devices [4], IoT [5], and autonomous driving [6], etc.), data is generally dispersed across different clients (i.e., data silos). Owing to the high cost of data collection and strict privacy protection regulations, the centralized training that integrates data together is prohibited [7, 8]. Driven by this reality, Federated Learning (FL) [9, 10] has gained considerable attention in industry and academia as a promising distributed learning paradigm that allows multiple clients to participate in the collaborative training of a global model without access to their private data, thereby ensuring the basic privacy.

Despite its remarkable success, the inevitable hurdle that plagues FL research is the vast heterogeneity among real-world clients [11]. Specifically, the distribution of data among clients may be non-IID (identical and independently distributed), resulting in data heterogeneity [42, 12, 13, 8, 41]. It has been confirmed that the vanilla FL method FedAvg [14] suffers from client drift in this case, which leads to severe performance degradation. To ameliorate this issue, a plethora of modifications [1521, 34, 39] for FedAvg focus on regularizing the objectives of the local models to align the global optimization objective. All of the above methods follow the widely accepted assumption of model homogeneity, where local models have to share the same architecture as the global model. Nevertheless,

when deploying FL systems, different clients may have distinct hardware and computing resources, and can only train the model architecture matching their own resource budgets [22, 24], resulting in model heterogeneity . In this case, to enable the FL systems with model homogeneity, on the one hand, clients with low resource budgets, which may be critical for enhancing the FL systems, will be discarded at the expense of training bias [43, 8, 23]. On the other hand, keeping a low complexity for the global model to accommodate all clients leads to performance drop due to the limited model capacity [30]. Therefore, the primary challenge of model-heterogeneous FL is how to conduct model aggregation of heterogeneous architectures among clients to enhance the inclusiveness of federated systems. To solve this challenge, existing efforts fall broadly into two categories: knowledge distillation (KD)-based methods [40, 45, 44, 25-27] and partial training (PT)-based methods [30, 28, 29, 31], yet each of them has its own limitations. Concretely, KD-based methods require additional public data to align the logits outputs between the global model (student) and local models (teachers). But the desired public data is not always available in practice and the performance may decrease dramatically if the apparent disparity in distributions exists between public data and clients' private data [33]. PT-based methods send width-based sub-models to clients, which are extracted by the server from a larger global model according to each client's resource budget, and then aggregate these trained sub-models to update the global model. PT-based methods can be considered as an extension of FedAvg to model-heterogeneous scenarios, which means they are implementation-friendly and computationally efficient, but they also suffer from the same adverse effects from data heterogeneity as FedAvg or even more severely. In a word, how to learn a robust global model in FL with both data and model heterogeneity is a highly meaningful and urgent problem.

𝑦

Global model

(update)

Ensemble  local sub-models

(fixed)

Generator

(fixed)

EMA generator

(fixed)

Synthetic  data

ℒ𝑘𝑙

𝑧

ǁ 𝑧

෤ 𝑦

ℎ

෨ ℎ

𝑒𝑦

𝑒 ෤ 𝑦

𝑦

Global model

(fixed)

Ensemble  local sub-models

(fixed)

Generator

(update)

ℒ𝑑𝑖𝑣

ℒ𝑓𝑖𝑑

ℒ𝑡𝑟𝑎𝑛

𝑧

ℎ

𝑒𝑦

Training Generator

Robust Model Distillation

Model Aggregation

Sub-Model Extraction

Agg.

Extract.

Upload.

Download.

local sub-models

global model

local sub-models

global model

Synthetic  data

ሚ

ℒ𝑘𝑙

class embedding

Synthetic  data

alternative

training

To this end, we systematically investigate the training of a robust global model in FL with both data and model heterogeneity with the aid of data-free knowledge distillation (DFKD) [35, 36, 53, 59]. See the related work in Appendix A for more DFKD methods. Note that the strategy of integrating DFKD to FL is not unique to us. Recently, FedFTG [48] leverages DFKD to fine-tune the global model in model-homogeneous FL to overcome data heterogeneity, and DENSE [49] aggregates knowledge from heterogeneous local models based on DFKD to train a global model for one-shot FL. They all equip the server, which possesses powerful hardware and computing resources, with a generator to approximate the training space of the local models (teachers), and train the generator and the global model (student) in an adversarial manner. However, the local models uploaded per communication round are not only architecturally heterogeneous but also trained on non-IID distributed private data in the situation of both data and model heterogeneity. In this case, the generator tends to deviate from the real data distribution. Also, its output distribution may undergo large shifts (i.e., distribution shifts) across communication rounds, causing the global model to catastrophically forget useful knowledge learned in previous rounds and suffer from performance degradation. To confront the mentioned issues, we propose a novel D ataF ree R obust D istillation FL method called DFRD, which utilizes a conditional generator to generate synthetic data and thoroughly studies how to effectively and accurately simulate the local models' training space in terms of fidelity , transferability and

diversity [48, 49]. To mitigate catastrophic forgetting of the global model, an exponential moving average (EMA) copy of the conditional generator is maintained on the server to store previous knowledge learned from the local models. The EMA generator, along with the current generator, provides training data for updates of the global model. Also, we propose dynamic weighting and label sampling to aggregate the logits outputs of the local models and sample labels respectively, thereby properly exploring the knowledge of the local models. We revisit FedFTG and DENSE, and argue that DFRD as a fine-tuning method (similar to FedFTG) can significantly enhance the global model. So, we readily associate the PT-based methods in model-heterogeneous FL with the ability to rapidly provide a preliminary global model, which will be fine-tuned by DFRD. We illustrate the schematic for DFRD as a fine-tuning method based on a PT-based method in Fig. 1. Although FedFTG and DENSE can also be applied to fine-tune the global model from the PT-based methods after simple extensions, we empirically find that they do not perform as well, and the performance of the global model is even inferior to that of local models tailored to clients' resource budgets.

Our main contributions of this work are summarized as follows. First, we propose a new FL method termed DFRD that enables a robust global model in both data and model heterogeneity settings with the help of DFKD. Second, we systematically study the training of the conditional generator w.r.t. fidelity , transferability and diversity to ensure the generation of high-quality synthetic data. Additionally, we maintain an EMA generator on the server to overcome the global model's catastrophic forgetting caused by the distribution shifts of the generator. Third, we propose dynamic weighting and label sampling to accurately extract the knowledge of local models. At last, our extensive experiments on six real-world image classification datasets verify the superiority of DFRD.

2 Notations and Preliminaries

Notations. In this paper, we focus on the centralized setup that consists of a central server and N clients owning private labeled datasets { ( X i , Y i ) } N i =1 , where X i = { x b i } n i b =1 follows the data distribution D i over feature space X i , i.e., x b i ∼ D i , and Y i = { y b i } n i b =1 ⊂ [ C ] := { 1 , · · · , C } denotes the ground-truth labels of X i . And C refers to the total number of labels. Remarkably, the heterogeneity for FL in our focus includes both data heterogeneity and model heterogeneity. For the former , we consider the same feature space, yet the data distribution may be different among clients, that is, label distribution skewness in clients (i.e., X i = X j and D i = D j , ∀ i = j, i, j ∈ [ N ] ). For the latter , each client i holds an on-demand local model f i parameterized by θ i . Due to the difference in resource budgets, the model capacity of each client may vary, i.e., | θ i | = | θ j | , ∃ i = j, i, j ∈ [ N ] . In PT-based methods, we define a confined width capability R i ∈ (0 , 1] according to the resource budget of client i , which is the proportion of nodes extracted from each layer in the global model f . Note that f is parameterized by θ , and | a | denotes the number of elements in vector a .

̸

̸

PT-based method is a solution for model-heterogeneous FL, which strives to extract a matching width-based slimmed-down sub-model from the global model as a local model according to each client's budget. As with FedAvg, it requires the server to periodically communicate with the clients. In each round, there are two phases: local training and server aggregation. In local training, each client trains the sub-model received from the server utilizing the local optimizer. In server aggregation, the server collects the heterogeneous sub-models and aggregates them by straightforward selective averaging to update the global model, as follows [28-31]:

where S t is a subset sampled from [ N ] and p i is the weight of client i , which generally indicates the size of data held by client i . At round t , θ t [ l,k ] denotes the k th parameter of layer l of the global model and θ t i, [ l,k ] denotes the parameter θ t [ l,k ] updated by client i . We can clearly see that Eq. (1) independently calculates the average of each parameter for the global model according to how many clients update that parameter in round t . Instead, the parameter remains unchanged if no clients update it. Notably, if | θ t i | = | θ t | for any i ∈ [ N ] , PT-based method becomes FedAvg. The key to PT-based method is to select θ t i from the global model θ t when given R i . And existing sub-model extraction schemes fall into three categories: static [28, 29], random [30] and rolling [31].

̸

̸

3 Proposed Method

In this section, we detail the proposed method DFRD. We mainly work on considering DFRD as a fine-tuning method to enhance the PT-based methods, thus enabling a robust global model in FL with both data and model heterogeneity. Fig. 1 visualizes the training procedure of DFRD combined with a PT-based method, consisting of four stages on the server side: training generator , robust model distillation , sub-model extraction and model aggregation . Note that sub-model extraction and model aggregation are consistent with that in the PT-based methods, so we detail the other two stages. Moreover, we present pseudocode for DFRD in Appendix C.

3.1 Training Generator

At this stage, we aim to train a well-behaved generator to capture the training space of local models uploaded from active clients. Specifically, we consider a conditional generator G ( · ) parameterized by w . It takes as input a random noise z ∈ ❘ d sampled from standard normal distribution N ( 0 , I ) , and a random label y ∈ [ C ] sampled from label distribution p ( y ) , i.e., the probability of sampling y , thus generating the synthetic data s = G ( h = o ( z , y ) , w ) . Note that o ( z , y ) represents the merge operator of z and y . To the best of our knowledge, synthetic data generated by a well-trained generator should satisfy several key characteristics: fidelity , transferability , and diversity [48, 49]. Therefore, in this section, we construct the loss objective from the referred aspects to ensure the quality and utility of G ( · ) .

Fidelity. To commence, we study the fidelity of the synthetic data. Specifically, we expect G ( · ) to simulate the training space of the local models to generate the synthetic dataset with a similar distribution to the original dataset. To put it differently, we want the synthetic data s to approximate the training data with label y without access to clients' training data. To achieve it, we form the fidelity loss L fid at logits level:

where CE denotes the cross-entropy function, f i ( s , θ i ) is the logits output of the local model from client i when s is given, τ i,y dominates the weight of logits from different clients { i | i ∈ S t } when y is given. And L fid is the cross-entropy loss between the weighted average logits ∑ i ∈S t τ i,y f i ( s , θ i ) and the label y . By minimizing L fid , s is enforced to be classified into label y with a high probability, thus facilitating the fidelity of s .

In reality, the conditional generator G ( · ) easily generates synthetic data with low classification errors (i.e. L fid close to 0 ) as the training proceeds. This may cause the synthetic data to fall into a space far from the decision boundary of the ensemble model (i.e., ∑ i ∈S t τ i,y f i ( · , θ i ) ) if only L fid is optimized, as shown in the synthetic data represented by red circles in Fig. 2 (a). Note that d S and d T denote the decision boundaries of the global model (student) and the ensemble model (teacher), respectively. An obvious observation is that the red circles are correctly classified on the same side of the two decision boundaries (i.e., d S and d T ), making it difficult to transfer teacher's knowledge to student. We next explore how to augment the transferability of the synthetic data to ameliorate this pitfall.

Transferability is intended to guide G ( · ) in generating synthetic data that moves the decision

𝑑𝑆

(a)

(c)

(b)

𝑑𝑇

𝑑𝑆

𝑑𝑇

𝑑𝑆

𝑑𝑇

synthetic data

real data

boundary of the global model towards that of the ensemble model, such as synthetic data with black circles in Fig. 2 (b). However, during the training of d S to approach d T , we find that G ( · ) can easily generate two other types of synthetic data, the yellow and purple circles in Fig. 2 (c). Both of them are misclassified by the ensemble model ( d T ), while the yellow circles are correctly classified and

the purple circles are misclassified by the global model ( d S ). For the conditional generator G ( · ) that takes label information as one of the inputs, yellow and purple circles can mislead the generator, thereby leading to d S approximating d T with a large deviation, as shown in Fig. 2 (c). Based on the above observation, we reckon that the synthetic data s = G ( h = o ( z , y ) , w ) is useful if it is classified as y by the ensemble model but classified not as y by the global model. To realize it, we maximize the logits discrepancy between the global model and the ensemble model on synthetic data with black circles by leveraging Kullback-Leibler divergence loss, which takes the form:

̸

where KL is Kullback-Leibler divergence function and f ( s , θ ) denotes the logits output of the global model on s with label y . Note that ε = 1 if arg max f ( s , θ ) = y and arg max ∑ i ∈S t τ i,y f i ( s , θ i ) = y hold, otherwise ε = 0 . ( ♢ )

̸

We would like to point out that the existing works [48] and [49] are in line with our research perspective on the transferability of generator, which aims to generate more synthetic data with black circles. However, they do not carefully frame learning objective for enhancing the transferability of generator. Concretely, [48] does not consider the type of synthetic data, i.e., ε = 1 always holds, thus inducing the generation of synthetic data with yellow and purple circles. ( △ ) [49] focuses on synthetic data satisfying arg max f ( s , θ ) = arg max ∑ i ∈S t τ i,y f i ( s , θ i ) , but enables the generation of synthetic data with purple circles yet. ( ▽ ) 2

Diversity. Although we enable G ( · ) to generate synthetic data that falls around the real data by optimizing L fid and L tran , the diversity of synthetic data is insufficient. Due to the fact that the generator may get stuck in local equilibria as the training proceeds, model collapse occurs [57, 58]. In this case, the generator may produce similar data points for each class with little diversity. Also, the synthetic data points may not differ significantly among classes. This causes the empirical distribution estimated by G ( · ) to cover only a small manifold in the real data space, and thus only partial knowledge of the ensemble model is extracted. To alleviate this issue, we introduce a diversity loss L div with label information to increase the diversity of synthetic data as follows:

where B denotes the batch size and s j/k = G ( h j/k = o ( z j/k , y j/k ) , w ) . Intuitively, L div takes ∥ h j -h k ∥ 2 as a weight, and then multiplies it by the corresponding ∥ s j -s k ∥ 2 in each batch B , thus imposing a larger weight on the synthetic data points pair ( s j and s k ) at the more distant input pair ( h j and h k ). Notably, we merge the random noise z with label y as the input of G ( · ) to overcome spurious solutions [53]. Further, we propose a multiplicative merge operator, i.e., o ( z , y ) = z ×E ( y ) , where E is a trainable embedding and × means vector element-wise product. We find that our merge operator enables synthetic data with more diversity compared to others, possibly because the label information is effectively absorbed into the stochasticity of z by multiplying them when updating E . See Section 4.3 for more details and empirical justification.

Combining L fid , L tran and L div , the overall objective of the generator can be formalized as follows:

where β tran and β div are tunable hyper-parameters. Of note, the synthetic data generated by a welltrained generator should be visually distinct from the real data for privacy protection, while it can capture the common knowledge of the local models to ensure similarity to the real data distribution for utility. More privacy protection is discussed in Appendices I and J.

3.2 Robust Model Distillation

Now we update the global model. Normally, the global model attempts to learn as much as possible logits outputs of the ensemble model on the synthetic data generated by the generator based on knowledge distillation [40, 46, 48, 49]. The updated global model and the ensemble model are then served to train G ( · ) with the goal of generating synthetic data that maximizes the mismatch

between them in terms of logits outputs (see transferability discussed in the previous section). This adversarial game enables the generator to rapidly explore the training space of the local models to help knowledge transfer from them to the global model. However, it also leads to dramatic shifts in the output distribution of G ( · ) across communication rounds under heterogeneous FL scenario (i.e., distribution shifts), causing the global model to catastrophically forget useful knowledge gained in previous rounds. To tackle the deficiency, we propose to equip the server with a generator ˜ G ( · ) parameterized by ˜ w that is an exponential moving average (EMA) copy of G ( · ) . Its parameters at the t th communication round are computed by

where λ ∈ (0 , 1) is the momentum. We can easily see that the parameters of ˜ G ( · ) vary very little compared to those of G ( · ) over communication rounds, if λ is close to 1 . We further utilize synthetic data from ˜ G ( · ) as additional training data for the global model outside of G ( · ) , mitigating the huge exploratory distribution shift induced by the large update of G ( · ) and achieving stable updates of the global model. Particularly, we compute the Kullback-Leibler divergence between logits of the ensemble model and the global model on the synthetic data points s = G ( h = o ( z , y ) , w ) and ˜ s = ˜ G ( ˜ h = o ( ˜ z , ˜ y ) , ˜ w ) respectively, which is formulated as follows:

where α is a tunable hyper-parameter for balancing different loss items.

Dynamic Weighting and Label Sampling. So far, how to determine τ i,y and p ( y ) is unclear. The appropriate τ i,y and p ( y ) are essential for effective extraction of knowledge from local models. For clarity, we propose dynamic weighting and label sampling, i.e., τ i,y = n y i,t /n y S t ,t and p ( y ) = n y S t ,t / ∑ y ∈ [ C ] n y S t ,t , where n y S t ,t = ∑ j ∈ [ S t ] n y j,t and n y i,t denotes the number of data with label y involved in training on client i at round t . Due to space limitations, see Appendix F for their detail study and experimental justification.

4 Experiments

4.1 Experimental Settings

Datasets. In this paper, we evaluate different methods with six real-world image classification task-related datasets, namely Fashion-MNIST [69] (FMNIST in short), SVHN [70], CIFAR-10, CIFAR-100 [71], Tiny-imageNet 3 and Food101 [73]. We detail the six datasets in Appendix B. To simulate data heterogeneity across clients, as in previous works [34, 37, 38], we use Dirichlet process Dir ( ω ) to partition the training set for each dataset, thereby allocating local training data for each client. It is worth noting that ω is the concentration parameter and smaller ω corresponds to stronger data heterogeneity.

Baselines. Wecompare DFRD to FedFTG [48] and DENSE [49], which are the most relevant methods to our work. To verify the superiority of DFRD, on the one hand, DFRD, FedFTG and DENSE are directly adopted on the server to transfer the knowledge of the local models to a randomly initialized global model. We call them collectively data-free methods . On the other hand, they are utilized as fine-tuning methods to improve the global model's performance after computing weighted average per communication round. In this case, in each communication round, the preliminary global model is obtained using FedAvg [14] in FL with homogeneous models, whereas in FL with heterogeneous models, the PT-based methods, including HeteroFL [29], Federated Dropout [30] (FedDp for short) and FedRolex [31], are employed to get the preliminary global model. 4

Configurations. Unless otherwise stated, all experiments are performed on a centralized network with N = 10 active clients. We set ω ∈ { 0 . 01 , 0 . 1 , 1 . 0 } to mimic different data heterogeneity scenarios. To simulate model-heterogeneous scenarios, we formulate exponentially distributed budgets for a given N : R i = [ 1 2 ] min { σ, ⌊ ρ · i N ⌋} ( i ∈ [ N ]) , where σ and ρ are both positive integers. We fix σ = 4

and consider ρ ∈ { 5 , 10 , 40 } . See Appendix D for more details. Unless otherwise specified, we set β tran and β div both to 1 in training generator , while in robust model distillation , we set λ = 0 . 5 and α = 0 . 5 . And all baselines leverage the same setting as ours. Due to space limitations, see Appendix E for the full experimental setup.

Evaluation Metrics. We evaluate the performance of different FL methods by local and global test accuracy. To be specific, for local test accuracy ( L.acc for short), we randomly and evenly distribute the test set to each client and harness the test set on each client to verify the performance of local models. In terms of global test accuracy ( G.acc for short), we employ the global model on the server to evaluate the global performance of different FL methods via utilizing the original test set. Note that L.acc is reported in round brackets . To ensure reliability, we report the average for each experiment over 3 different random seeds.

Alg.s

FMNIST

SVHN

CIFAR-10

CIFAR-100

ω

= 1

.

0

ω

= 0

.

1

ω

= 0

.

01

ω

= 1

.

0

ω

= 0

.

1

ω

= 0

.

01

ω

= 1

.

0

ω

= 0

.

1

ω

= 0

.

01

ω

= 1

.

0

ω

= 0

.

1

ω

= 0

.

01

DENSE

13.99

±

4.39

(74.26

±

2.64)

13.96

±

1.33

(30.27

±

1.10)

13.83

±

2.88

(14.82

±

3.55)

16.12

±

2.08

(52.67

±

1.68)

14.53

±

3.56

(25.95

±

0.42)

13.06

±

5.40

(13.09

±

1.64)

10.47

±

1.21

(46.89

±

0.97)

10.28

±

0.60

(22.67

±

0.66)

10.96

±

1.93

(13.03

±

1.56)

1.22

±

0.10

(21.68

±

0.23)

1.11

±

0.03

(13.61

±

0.30)

1.11

±

0.02

(8.73

±

0.09)

FedFTG

48.10

±

1.61

(73.96

±

2.51)

44.44

±

1.30

(30.33

±

1.43)

26.96

±

8.33

(14.79

±

3.61)

18.58

±

1.41

(54.75

±

0.63)

17.42

±

2.03

(26.39

±

0.38)

15.44

±

4.12

(

13.15

±

1.63

)

28.73

±

1.03

(47.67

±

0.92)

21.97

±

1.18

(22.95

±

0.64)

15.18

±

0.72

(13.06

±

1.54)

1.05

±

0.09

(22.67

±

0.44)

1.06

±

0.10

(14.01

±

0.49)

1.11

±

0.09

(8.82

±

0.21)

DFRD

65.38

±

0.72

(

74.44

±

2.47

)

52.55

±

1.61

(

30.44

±

1.42

)

36.63

±

6.52

(

14.87

±

3.64

)

24.13

±

0.96

(

55.48

±

1.56

)

18.53

±

0.42

(

26.87

±

0.11

)

18.18

±

0.56

(12.83

±

1.90)

33.82

±

1.81

(

48.33

±

1.17

)

24.58

±

0.59

(

23.21

±

0.70

)

20.59

±

2.93

(

14.18

±

1.53

)

3.21

±

0.36

(

23.31

±

0.39

)

1.55

±

0.13

(

14.15

±

0.32

)

1.36

±

0.06

(

8.93

±

0.13

)

FedAvg

89.71

±

0.10

(84.57

±

0.63)

83.24

±

1.41

(62.48

±

5.86)

60.89

±

5.24

(29.81

±

13.98)

70.63

±

1.95

(54.21

±

2.54)

58.12

±

1.26

(24.66

±

0.42)

32.22

±

1.16

(12.57

±

1.81)

67.16

±

0.98

(49.82

±

1.47)

56.36

±

2.09

(21.92

±

0.83)

32.92

±

7.40

(12.37

±

1.71)

57.31

±

0.09

(47.01

±

0.99)

49.50

±

0.51

(25.33

±

0.88)

39.22

±

1.08

(11.25

±

0.52)

+DENSE

90.13

±

0.14

(84.81

±

0.60)

84.10

±

1.55

(62.80

±

5.76)

63.36

±

6.24

(29.37

±

13.83)

74.48

±

1.48

(58.02

±

2.65)

62.41

±

2.36

(26.23

±

0.33)

32.84

±

12.37

(12.72

±

1.82)

69.73

±

0.60

(53.46

±

1.58)

57.48

±

3.13

(22.54

±

0.76)

35.85

±

7.22

(12.41

±

1.72)

58.43

±

0.05

(48.00

±

0.78)

50.84

±

0.60

(26.19

±

0.90)

40.24

±

1.04

(11.87

±

0.58)

+FedFTG

91.15

±

0.23

(85.88

±

0.70)

84.93

±

1.40

(63.14

±

4.64)

64.80

±

6.56

(

29.89

±

14.08

)

73.91

±

1.78

(56.99

±

2.65)

61.12

±

1.69

(25.82

±

0.28)

37.07

±

6.55

(12.69

±

1.80)

68.85

±

0.96

(51.05

±

1.59)

58.17

±

2.02

(22.39

±

0.66)

33.93

±

7.69

(12.39

±

1.70)

58.81

±

1.39

(48.06

±

1.79)

50.70

±

1.11

(26.40

±

0.47)

40.50

±

1.17

(12.15

±

0.61)

+DFRD

91.65

±

0.09

(

86.16

±

0.53

)

85.66

±

1.21

(

63.79

±

4.29

)

65.23

±

7.17

(29.58

±

13.45)

77.84

±

1.39

(

60.31

±

2.51

)

65.79

±

2.27

(

27.26

±

0.20

)

39.25

±

11.71

(

13.96

±

1.48

)

71.54

±

0.66

(

53.59

±

1.19

)

61.34

±

1.21

(

23.00

±

0.68

)

40.77

±

7.47

(

13.49

±

1.57

)

61.49

±

0.28

(

49.97

±

0.58

)

53.95

±

0.67

(

29.10

±

0.87

)

43.47

±

1.05

(

14.09

±

1.09

)

4.2 Results Comparison

Impacts of varying ω . We study the performance of different methods at different levels of data heterogeneity on FMNIST, SVHN, CIFAR-10 and CIFAR-100, as shown in Table 1. One can see that the performance of all methods degrades severely as ω decreases, with DFRD being the only method that is robust while consistently leading other baselines with an overwhelming margin w.r.t. G.acc . Also, Fig. 3 (a)-(b) show that the learning efficiency of DFRD consistently beats other baselines (see Fig. 8-9 in Appendix H for complete curves). Notably, DFRD, FedFTG and DENSE as fine-tuning methods uniformly surpass FedAvg w.r.t. G.acc and L.acc . However, their global test accuracies suffer from dramatic deterioration or even substantially worse than that of FedAvg when they act as data-free methods. We conjecture that FedAvg aggregates the knowledge of local models more effectively than data-free methods. Also, when DFRD is used to fine-tune FedAvg, it can significantly enhance the global model, yet improve the performance of local models to a less extent.

Impacts of different ρ . We explore the impacts of different model heterogeneity distributions on different methods with SVHN, CIFAR-10, Tiny-ImageNet and FOOD101. A higher ρ means more clients with 1 16 -width capacity w.r.t. the global model. From Table 2, we can clearly see that the performance of all methods improves uniformly with decreasing ρ , where DFRD consistently and overwhelmingly dominates other baselines in terms of G.acc . Specifically, DFRD improves G.acc by an average of 11 . 07% and 7 . 54% on SVHN and CIFAR-10 respectively, compared to PT-based methods (including HeteroFL, FedDp and FedRolex). Meanwhile, DFRD uniformly and significantly outstrips FedFTG and DENSE w.r.t. G.acc . The selected learning curve shown in Fig. 3 (c) also verifies the above statement (see Fig. 10-12 in Appendix H for more results). The above empirical results show that DFRD not only is robust to varying ρ , but also has significantly intensified effects on the global model for different PT-based methods. However, the results on Tiny-ImageNet and FOOD101 indicate that PT-based methods suffer from inferior test accuracy. Although DFRD improves their test accuracy, the improvement is slight. Notably, DFRD improves negligibly over PT-based methods when all clients exemplify 1 16 -width capability. We thus argue that weak clients performing complex image classification tasks learn little useful local knowledge, resulting in the inability to provide effective information for the global model.

4.3 Ablation Study

In this section, we carefully demonstrate the efficacy and indispensability of core modules and key parameters in our method on SVHN, CIFAR-10 and CIFAR-100. Thereafter, we resort to FedRolex+DFRD to yield all results. For SVHN (CIFAR-10, CIFAR-100), we set ω = 0 . 1 (0 . 1 , 1 . 0)

0

20

40

60

80

100

communication rounds

0.20

0.22

0.24

0.26

0.28

0.30

0.32

L

.

acc

0

20

40

60

80

100

communication rounds

0.10

0.20

0.30

0.40

0.50

G

.

acc

DENSE

FedFTG

DFRD

(a) FMNIST,

ω

= 0

.

1

0

20

40

60

80

100

communication rounds

0.10

0.20

0.30

0.40

0.50

0.60

L

.

acc

0

20

40

60

80

100

communication rounds

0.20

0.30

0.40

0.50

0.60

0.70

0.80

G

.

acc

FedAvg

FedAvg+DENSE

FedAvg+FedFTG

FedAvg+DFRD

(b) SVHN,

ω

= 1

.

0

0

50

100

150

200

250

300

communication rounds

0.11

0.12

0.13

0.14

0.15

0.16

0.17

L.acc

0

50

100

150

200

250

300

communication rounds

0.05

0.10

0.15

0.20

0.25

0.30

G.acc

FedDp

FedDp+DENSE

FedDp+FedFTG

FedDp+DFRD

(c) CIFAR-10,

ρ

= 5

and ρ = 10 (10 , 5) . Note that we also study the performance of DFRD with different numbers of clients and stragglers, see Appendix G for details.

Impacts of different transferability constraints. It can be observed from Table 3 that our proposed transferability constraint ♢ uniformly beats △ and ▽ w.r.t. global test performance over three datasets. This means that ♢ can guide the generator to generate more effective synthetic data, thereby improving the test performance of the global model. Also, we conjecture that generating more synthetic data like those with the black circles (see Fig. 2) may positively impact on the performance of local models, since ♢ also slightly and consistently trumps other competitors w.r.t. L.acc .

Impacts of varying merge operators. To look into the utility of the merge operator, we consider multiple merge operators, including mul , add , cat , ncat and none . Among them, mul is o ( z , y ) = z ×E ( y ) , add is o ( z , y ) = z + E ( y ) , cat is o ( z , y ) = [ z , E ( y )] , ncat is o ( z , y ) = [ z , y ] and none is o ( z , y ) = z . From Table 4, we can observe that our proposed mul significantly outperforms other competitors in terms of G.acc . This suggests that mul can better exploit the diversity constraint to make the generator generate more diverse synthetic data, thus effectively finetuning the global model. Also, the visualization of the synthetic data in Appendix I validates this statement.

Necessity of each component for DFRD. We report the test performance of DFRD after divesting some modules and losses in Table 5. Here, EMA indicates the exponential moving average copy of the generator on the server. We can evidently observe that removing the EMA generator leads to a significant drop in G.acc , which implies that it can generate effective synthetic data for the global model. The reason why the EMA generator works is that it avoids catastrophic forgetting of the global model and ensures the stability of the global model trained in heterogeneous FL. We display synthetic data in Appendix I that further corroborates the above claims.

Meanwhile, we perform the leave-one-out test to explore the contributions of L tran and L div to DFRD separately, and further report the test results of removing them simultaneously. From Table 5, deleting either L tran or L div adversely affects the performance of DFRD. In addition, their joint absence further exacerbates the degradation of G.acc . This suggests that L tran and L div are vital for the training of the generator. Interestingly, L div benefits more to the global model than L tran . We speculate that the diversity of synthetic data is more desired by the global model under the

premise of ensuring the fidelity of synthetic data by optimizing L fid .

Varying β tran and β div . We explore the impacts of β tran and β div on SVHN and CIFAR-10. We select β tran and β div from { 0 . 25 , 0 . 50 , 0 . 75 , 1 . 00 , 1 . 25 , 1 . 50 } . From Fig. 4, we can see that DFRD maintains stable test performance among all selections of β tran and β div over SVHN. At the same time,

G.acc fluctuates slightly with the

0.25

0.50

0.75

1.00

1.25

1.50

tran

15

20

25

30

35

40

45

accuracy

(a) SVHN

G

.

acc

L

.

acc

0.25

0.50

0.75

1.00

1.25

1.50

div

15

20

25

30

35

40

45

(b) SVHN

G

.

acc

L

.

acc

0.25

0.50

0.75

1.00

1.25

1.50

tran

16

18

20

22

24

26

(c) CIFAR-10

G

.

acc

L

.

acc

0.25

0.50

0.75

1.00

1.25

1.50

div

16

18

20

22

24

26

(d) CIFAR-10

G

.

acc

L

.

acc

increases of β tran and β div on CIFAR-10. Besides, we observe that the worst G.acc in Fig. 4 outperforms the baseline with the best G.acc in Table 2. The above results indicate that DFRD is not sensitive to choices of β tran and β div over a wide range.

Varying α and λ . In order to delve into the effect of the EMA generator on DFRD in more details, we perform grid testing on the choices of control parameters α and λ over SVHN and CIFAR-10. We set α ∈ { 0 . 25 , 0 . 50 , 0 . 75 , 1 . 00 , 1 . 25 , 1 . 50 } and λ ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } . It can be observed from Fig. 5 that high global test accuracies on

0.1

0.3

0.5

0.7

0.9

0.25

0.50

0.75

1.00

1.25

1.50

35.24

35.82

35.97

35.56

35.66

35.49

35.89

35.90

35.16

35.67

35.79

35.47

35.82

35.64

35.06

35.68

35.77

35.79

36.14

35.95

34.79

35.76

35.56

35.87

35.04

34.78

35.21

35.10

35.78

35.95

(a) SVHN,

G

.

acc

0.1

0.3

0.5

0.7

0.9

16.04

16.01

15.89

16.02

15.96

16.08

16.01

15.98

16.07

16.05

16.05

16.01

15.96

16.01

15.98

16.01

15.98

15.96

16.02

16.00

16.11

16.05

15.97

16.07

16.00

16.10

16.13

15.99

15.90

16.07

(b) SVHN,

L

.

acc

0.1

0.3

0.5

0.7

0.9

23.28

23.65

23.93

23.75

24.35

24.85

24.04

25.57

25.23

24.38

25.41

24.43

24.76

24.55

25.61

25.84

25.04

25.43

26.49

24.43

25.40

24.30

25.27

24.98

25.16

26.03

25.31

25.82

26.06

24.41

(c) CIFAR-10,

G

.

acc

0.1

0.3

0.5

0.7

0.9

16.69

16.55

16.50

16.61

16.56

16.69

16.49

16.74

16.70

16.55

16.75

16.60

16.53

16.66

16.72

16.59

16.53

16.71

16.45

16.39

16.65

16.49

16.64

16.48

16.56

16.60

16.40

16.71

16.66

16.89

(d) CIFAR-10,

L

.

acc

34.8

35.0

35.2

35.4

35.6

35.8

36.0

15.9

16.0

16.0

16.1

16.1

23.5

24.0

24.5

25.0

25.5

26.0

16.4

16.5

16.6

16.7

16.8

SVHN are mainly located in the region of α < 1 . 25 and λ > 0 . 5 , while on CIFAR-10 they are mainly located in the region of α > 0 . 25 and λ < 0 . 9 . According to the above results, we deem that the appropriate α and λ in a specific task is essential for the utility of the EMA generator. Notably, high local test accuracies mainly sit in regions that are complementary to those of high global test accuracies, suggesting that pursuing high G.acc and L.acc simultaneously seems to be a dilemma. How to ensure high G.acc and L.acc simultaneously in the field of FL is an attractive topic that is taken as our future work.

5 Conclusion

In this paper, we propose a new FL method called DFRD, which aims to learn a robust global model in the data-heterogeneous and model-heterogeneous scenarios with the aid of DFKD. To ensure the utility, DFRD considers a conditional generator and thoroughly studies its training in terms of fidelity ,

transferability and diversity . Additionally, DFRD maintains an EMA generator to augment the global model. Furthermore, we propose dynamic weighting and label sampling to accurately extract the knowledge of local models. At last, we conduct extensive experiments to verify the superiority of DFRD. Due to space constraints, we discuss in detail the limitations and broader impacts of our work in Appendixes J and K, respectively.

6 Acknowledgments

This work has been supported by the National Natural Science Foundation of China under Grant No.U1911203, and the National Natural Science Foundation of China under Grant No.62377012.

References

Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3213-3223.

Krizhevsky, A., Sutskever, I., &Hinton, G. E. (2017) Imagenet classification with deep convolutional neural networks. Communications of the ACM 60 (6): 84-90.

Chen, Z., Li, B., Xu, J., Wu, S., Ding, S., &Zhang, W. (2022) Towards practical certifiable patch defense with vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 15148-15158.

Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y. C., Yang, Q., Niyato, D., &Miao, C. (2020) Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials 22 (3): 2031-2063.

Nguyen, D. C., Ding, M., Pathirana, P. N., Seneviratne, A., Li, J., & Poor, H. V. (2021) Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials 23 (3): 1622-1658.

Li, Y., Tao, X., Zhang, X., Liu, J., &Xu, J. (2021) Privacy-preserved federated learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems 23 (7): 8423-8434.

Voigt, P., &Von dem Bussche, A. (2017) The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10 (3152676): 10-5555.

Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning 14 (1-2): 1-210.

Koneˇ cný, J., McMahan, B., & Ramage, D. (2015) Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575.

Koneˇ cný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016). Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 .

Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine 37 (3): 50-60.

Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., &Chandra, V. (2018) Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 .

Li, Q., Diao, Y., Chen, Q., &He, B. (2022) Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pp. 965-978.

McMahan, B., Moore, E., Ramage, D., Hampson, S., &yArcas, B. A. (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pp. 1273-1282.

Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., &Smith, V. (2020). Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2 : 429-450.

Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., &Suresh, A. T. (2020) Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pp. 51325143.

Acar, D. A. E., Zhao, Y., Navarro, R. M., Mattina, M., Whatmough, P. N., & Saligrama, V. (2021) Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 .

Kim, J., Kim, G., & Han, B. (2022) Multi-level branched regularization for federated learning. In International Conference on Machine Learning , pp. 11058-11073.

Mendieta, M., Yang, T., Wang, P., Lee, M., Ding, Z., &Chen, C. (2022) Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8397-8406.

Lee, G., Jeong, M., Shin, Y., Bae, S., & Yun, S. Y. (2022) Preservation of the Global Knowledge by Not-True Distillation in Federated Learning. Advances in Neural Information Processing Systems .

Zhang, J., Li, Z., Li, B., Xu, J., Wu, S., Ding, S., &Wu,C. (2022) Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning , pp. 26311-26329.

Ignatov, A., Timofte, R., Chou, W., Wang, K., Wu, M., Hartley, T., &Van Gool, L. (2018) Ai benchmark: Running deep neural networks on android smartphones. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops , pp. 0-0.

Makhija, D., Han, X., Ho, N., & Ghosh, J. (2022) Architecture agnostic federated learning for neural networks. In International Conference on Machine Learning , pp. 14860-14870.

Hong, J., Wang, H., Wang, Z., &Zhou, J. (2022) Efficient split-mix federated learning for on-demand and in-situ customization. arXiv preprint arXiv:2203.09747 .

Afonin, A., &Karimireddy, S. P. (2021) Towards model agnostic federated learning using knowledge distillation. arXiv preprint arXiv:2110.15210 .

Cho, Y. J., Manoel, A., Joshi, G., Sim, R., &Dimitriadis, D. (2022) Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703 .

Fang, X., &Ye, M. (2022) Robust federated learning with noisy and heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10072-10081.

Horvath, S., Laskaridis, S., Almeida, M., Leontiadis, I., Venieris, S., & Lane, N. (2021) Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems 34 : 12876-12889.

Diao, E., Ding, J., &Tarokh, V. (2020) HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264 .

Caldas, S., Koneˇ cny, J., McMahan, H. B., & Talwalkar, A. (2018) Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210.

Alam, S., Liu, L., Yan, M. &Zhang, M. (2022) FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction. Advances in neural information processing systems .

Tang, Z., Zhang, Y., Shi, S., He, X., Han, B., &Chu, X. (2022). Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In International Conference on Machine Learning , pp. 21111-21132.

Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., &Zhang, C. (2022) Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence 36 (8): 8432-8440.

Luo, K., Li, X., Lan, Y., &Gao, M. (2023) GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting. arXiv preprint arXiv:2302.14307 .

Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu C., Xu C. &Tian, Q. (2019) Data-free learning of student networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 3514-3522.

Micaelli, P., & Storkey, A. J. (2019) Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems 32 .

Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., & Khazaeni, Y. (2019, May) Bayesian nonparametric federated learning of neural networks. In International conference on machine learning , pp. 7252-7261.

Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., &Khazaeni, Y. (2020) Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 .

Li, Q., He, B., &Song, D. (2021) Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722.

Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020) Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems 33 : 2351-2363.

Li, X., Jiang, M., Zhang, X., Kamp, M., &Dou, Q. (2021) Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623.

Fallah, A., Mokhtari, A., & Ozdaglar, A. (2020) Personalized federated learning: A meta-learning approach. In Advances in Neural Information Processing Systems.

Pan, S. J., &Yang, Q. (2010) A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22 (10): 1345-1359.

Li, D., &Wang, J. (2019) Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581.

He, C., Annavaram, M., &Avestimehr, S. (2020) Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems 33 :14068-14080.

Zhang, S., Liu, M., & Yan, J. (2020) The diversified ensemble neural network. Advances in Neural Information Processing Systems 33 :16001-16011.

Zhu, Z., Hong, J., &Zhou, J. (2021) Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning , pp. 12878-12889.

Zhang, L., Shen, L., Ding, L., Tao, D., & Duan, L. Y. (2022) Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10174-10183.

Zhang, J., Chen, C., Li, B., Lyu, L., Wu, S., Ding, S., Shen, C. & Wu, C. (2022) DENSE: Data-Free One-Shot Federated Learning. In Advances in Neural Information Processing Systems .

Heinbaugh, C. E., Luz-Ricca, E., &Shao, H. (2023) Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity. In The Eleventh International Conference on Learning Representations.

Fang, G., Song, J., Wang, X., Shen, C., Wang, X., &Song, M. (2021) Contrastive model inversion for data-free knowledge distillation. arXiv preprint arXiv:2105.08584 .

Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha N. K., & Kautz, J. (2020) Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8715-8724.

Do, K., Le, T. H., Nguyen, D., Nguyen, D., Harikumar, H., Tran, T., Rana S., &Venkatesh, S. (2022) Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation. Advances in Neural Information Processing Systems 35 : 10055-10067.

Binici, K., Pham, N. T., Mitra, T., &Leman, K. (2022) Preventing catastrophic forgetting and distribution mismatch in knowledge distillation via synthetic data. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pp. 663-671.

Binici, K., Aggarwal, S., Pham, N. T., Leman, K., & Mitra, T. (2022) Robust and resource-efficient data-free knowledge distillation by generative pseudo replay. In Proceedings of the AAAI Conference on Artificial Intelligence 36 (6):6089-6096.

Patel, G., Mopuri, K. R., &Qiu, Q. (2023) Learning to Retain while Acquiring: Combating DistributionShift in Adversarial Data-Free Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7786-7794.

Odena, A., Olah, C., &Shlens, J. (2017) Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning , pp. 2642-2651.

Kodali, N., Abernethy, J., Hays, J., &Kira, Z. (2017) On convergence and stability of gans. arXiv preprint arXiv:1705.07215.

Yoo, J., Cho, M., Kim, T., &Kang, U. (2019) Knowledge extraction with no observable data. Advances in Neural Information Processing Systems 60 .

Luo, L., Sandler, M., Lin, Z., Zhmoginov, A., & Howard, A. (2020) Large-scale generative data-free distillation. arXiv preprint arXiv:2012.05578 .

Nayak, G. K., Mopuri, K. R., Shaj, V., Radhakrishnan, V. B., & Chakraborty, A. (2019) Zero-shot knowledge distillation in deep networks. In International Conference on Machine Learning , pp. 47434751.

Wang, Z. (2021) Data-free knowledge distillation with soft targeted transfer set synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence 35 (11): 10245-10253.

Liu, R., Wu, F., Wu, C., Wang, Y., Lyu, L., Chen, H., & Xie, X. (2022) No one left behind: Inclusive federated learning over heterogeneous devices. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pp. 3398-3406.

Kim, M., Yu, S., Kim, S., &Moon, S. M. (2023) DepthFL: Depthwise Federated Learning for Heterogeneous Clients. In The Eleventh International Conference on Learning Representations.

Singhal, K., Sidahmed, H., Garrett, Z., Wu, S., Rush, J., &Prakash, S. (2021) Federated reconstruction: Partially local federated learning. Advances in Neural Information Processing Systems 34 :11220-11232.

Zhang, H., Bosch, J., &Olsson, H. H. (2021) Real-time end-to-end federated learning: An automotive case study. In 2021 IEEE 45th Annual Computers, Software, and Applications Conference , pp. 459-468.

Li, Z., He, Y., Yu, H., Kang, J., Li, X., Xu, Z., &Niyato, D. (2022) Data heterogeneity-robust federated learning via group client selection in industrial iot. IEEE Internet of Things Journal 9 (18): 17844-17857.

Zeng, S., Li, Z., Yu, H., Zhang, Z., Luo, L., Li, B., &Niyato, D. (2023) Hfedms: Heterogeneous federated learning with memorable data semantics in industrial metaverse. IEEE Transactions on Cloud Computing.

Xiao, H., Rasul, K., & Vollgraf, R. (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 .

Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., & Ng, A. Y. (2011) Reading digits in natural images with unsupervised feature learning.

Krizhevsky, A., &Hinton, G. (2009) Learning multiple layers of features from tiny images.

He, K., Zhang, X., Ren, S., &Sun, J. (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778.

Bossard, L., Guillaumin, M., &Van Gool, L. (2014) Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference , pp. 446-461.

Dwork, C. (2008) Differential privacy: A survey of results. In Theory and Applications of Models of Computation , pp. 1-19.

Geyer, R. C., Klein, T., & Nabi, M. (2017) Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 .

Cheng, A., Wang, P., Zhang, X. S., &Cheng, J. (2022) Differentially Private Federated Learning with Local Regularization and Sparsification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10122-10131.

Ma, J., Naas, S. A., Sigg, S., &Lyu, X. (2022) Privacy-preserving federated learning based on multi-key homomorphic encryption. International Journal of Intelligent Systems 37 (9): 5880-5901.

Zhang, L., Xu, J., Vijayakumar, P., Sharma, P. K., &Ghosh, U. (2022). Homomorphic encryption-based privacy-preserving federated learning in iot-enabled healthcare system. IEEE Transactions on Network Science and Engineering .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe overall theoretical framework of the paper can be regarded as an extension of the FedFTG theory. Building upon the server-side learning of challenging samples, it incorporates the generation of data from the previous time step by the generator to prevent catastrophic forgetting in the global model. The experimental section of the paper is substantial, and it includes comparative analyses through ablation experiments to evaluate the effectiveness of various losses. Thus, the paper makes a good contribution.\n\nSTRENGTHS:\nOriginality: In comparison to FedFTG, the originality of the paper lies in Section 3.2, where a diverse generator is utilized by leveraging the parameter updates from the previous time step to prevent catastrophic forgetting in the model.\n\nQuality: In comparison to FedFTG, the paper conducts thorough comparative experiments and ablative experiments that elucidate the roles of various losses, indicating a certain level of quality.\n\nClarity: The framework diagram, Figure 1, in the paper provides a clear explanation of the algorithm's flow.\n\nWEAKNESSES:\n3.1 The impact of knowledge transfer on synthesizing different datasets lacks theoretical support and experimental evidence, as demonstrated by the decision boundary theory presented in Figure 2. There is a lack of clear relationship between synthetic data and the decision boundaries of the student and teacher models. Since the Generator satisfies Equation 2, it is capable of generating samples (blue) on the decision boundary of the teacher model. However, it is not clearly explained under what circumstances it would generate samples beyond the decision boundaries (yellow and purple) without further elaboration.\n\n4.3 The originality of the theoretical part of the paper is relatively weak, as using parameters from the previous time step to enhance samples appears more like a heuristic approach rather than a theoretical one.\n\nThe algorithm/objective is not concrete. For Eq(5), the objective is to minimization for which parameters?\n\nQUESTIONS:\nHow the experiments simulate the model heterogenous? \nSuggestions : The paper lacks visual experiments on samples to enhance the credibility of the theoretical training of the generator.\nParameter omega be used both for generator and Dirichlet process.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers learning a robust global model in heterogeneous federated learning (FL). It aims to support scenarios of both data-heterogeneous, where data distributions among clients are non-IID (identical and independently distributed), and model-heterogeneous, where the model architecture among clients is different. To this end, the authors propose a data-free knowledge distillation (DFKD) method, which utilizes a conditional generator to generate synthetic data, which simulates the local models’ training space with respect to fidelity, transferability, and diversity. Besides, the authors utilize an exponential moving average method to mitigate the catastrophic forgetting of the global model. Experiments on real-world image classification datasets are conducted to evaluate the performance of the proposed approach.\n\nSTRENGTHS:\n1. The problem of heterogeneous federated learning is interesting and important. \n\n2. A data-free knowledge distillation method with a detailed analysis of the conditional generator w.r.t. three characteristics. \n\n3. Experiments on six real-world datasets are conducted.\n\nWEAKNESSES:\n1. The conditional generator in the proposed approach is not well motivated and elaborated. \n\n2. The contribution of the proposed losses w.r.t. the three characteristics is incremental. \n\n3. The choice of hyper-parameters in the exponential moving average method is unclear. \n\nW4. Some of the experimental results are hard to comprehend.\n\nQUESTIONS:\nQ1. For the data heterogeneity considered in this work, what distribution the final test dataset follows? Is it the global distribution over all clients’ local data distributions?\n\nQ2. What is the purpose of training a generator on the server for the partial training-based methods? And how is it capture the training space of various clients’ local models? More elaboration would be helpful. \n\nQ3. Regarding the conditional generator proposed in this paper, the three characteristics (fidelity, transferability, and diversity) mainly follow those defined in [48] [49]. The contribution seems to only replace the generator with a conditional generator, which looks incremental. What is the new technical contribution in the proposed method?\n\nQ4. The EMA method is proposed to mitigate the catastrophic forgetting of the global model. However, since the global model already captures the knowledge of local historical models, it should not forget the knowledge if it is useful. I wonder why this problem exists. Is it related to how the clients extract the local models from the aggregated global model? Besides, how to set the averaging parameters \\lambda and \\alpha? \n\nQ5. It is mentioned in the paper that the generated synthetic data should be visually distinct from the real data for privacy protection. I wonder whether there is a metric to measure this. Also, if the objective is to capture the training space of local models, would it be possible to recover the clients’ local data distributions?\n\nQ6. The proposed DFKD method is claimed to be able to address the non-IID problem; then, it would be better to compare it with solutions such as FedProx, FedNova, etc. \n\nQ7. In Table 1 and Table 2, do the two rows of each method represent the local test accuracy and global test accuracy, or the opposite? The improvement seems to be very small compared to DENSE and FedFTG, for example, less than 1% on FMNIST. \n\nQ8. In Figure 3a, why the global accuracy of DENSE is too low? E.g., only 10% accuracy. \n\nQ9. In Tables 3-5, why the accuracies of the three datasets are too low? Are they well-trained or converged?\n\nLIMITATIONS:\nL1. It is unclear whether the proposed approach can handle the non-IID setting well, as one of the goals is to support data-heterogeneous. The authors may need to compare DFKD with non-IID solutions when the model architecture among clients is the same. \n\nL2. The clients may not obtain useful local models due to they hold non-IID data distributions, and the server aggregates the sub-models over the global data distribution.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies how to learn a robust global model in the data-heterogeneous and model-heterogeneous FL, by proposing a data-free knowledge distillation method. The proposed method is evaluated on six image classification datasets and outperforms compared methods.

STRENGTHS:
- This paper studies data heterogeneity and model heterogeneity in FL, which are relevant and important topics.
- The motivation for solving both data and model heterogeneity are well explained.
- The proposed method outperforms compared methods.
- The authors conduct many ablation studies.

WEAKNESSES:
- It is not clear how to calculate the term \tau_{i,y}, which is important to adjust the synthetic data.
- Only intuitive toy visualizations are shown. It lacks experimental visualizations to support the mentioned “Fidelity” of the proposed method.
- The experiments only validate the model heterogeneity in terms of width.
- The results presentation can be improved. The table is small, and some symbols are confusing.
- The experiments are mainly conducted on datasets with small image sizes （e.g., 32*32 or 64*64), lack of validation on large datasets.
- The global accuracy is sensitive to different values of \beta, and the standard deviation is large.

QUESTIONS:
- Why do DFRD and +DFRD show different trends in Table 1? For DFRD, L.acc is higher than G.acc, but for +DFRD, L.acc is lower than G.acc.
- Is this method applicable to other model heterogeneity cases? Such as different network architecture?
- Why replace FMNIST in table.1 with TinyImagnet in table.2?

LIMITATIONS:
The authors discussed the limitations with potential solutions, and the broader impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the presented paper, the authors lay out a strategy for fine-tuning a global model, specifically within the Heterogeneous Federated Learning setting. To transfer knowledge from the individual local models (identified as 'teachers') to the global model (or 'student'), the authors construct and train a generative model that is capable of producing pseudo samples, inspired from the Data-Free Knowledge Distillation literature. \n\nFurthermore, the authors suggest preserving an exponentially moving copy of this generative model. This is specifically designed to provide pseudo-samples from prior communication rounds. The overarching goal of this method is to protect the global model from catastrophic forgetting - a substantial problem in machine learning where a model, after learning new information, forgets the old. By accessing these pseudo-samples from previous rounds, the global model can effectively reinforce its learning and maintain a more comprehensive understanding over the communication rounds.\n\nLastly, the authors introduce a system that employs dynamic weighting and label sampling. This is proposed to enhance the precision in extracting knowledge from local models. By implementing this technique, it is intended that the knowledge transfer process will be more accurate and effective.\n\nSTRENGTHS:\n1. In this manuscript, the authors introduce an elegant yet straightforward technique to avert catastrophic forgetting, an issue that often arises in machine learning models, by maintaining a EMA generator. Remarkably, the method appears to maintain a consistent spatial footprint, equivalent to the dimensions of the generator models, throughout the various communication cycles. This ingenious mechanism facilitates the generation of pseudo samples from preceding rounds, thereby extending its usefulness and practicality in real-world applications.\n\n2. Moreover, the depth of empirical examination carried out in this paper is commendable. The authors have meticulously analyzed the performance of their technique, employing a thorough sensitivity analysis and ablation studies to evaluate the impact of the individual components of their methodology. This level of rigor and detail, often overlooked, attests to the quality of the research conducted and significantly boosts the credibility of their findings.\n\nWEAKNESSES:\n1. The manuscript offers a promising exploration into mitigating catastrophic forgetting in the context of Federated Heterogeneous Learning. However, as a reviewer, I find that the examination of catastrophic forgetting and its severity within this specific context could have been more in-depth. Works such as those by Binci et al. [1,2] and Patel et al. [3], which tackle catastrophic forgetting in adversarial settings, provide comprehensive empirical evidence via monitoring student learning curves. Such an approach would have deepened the current study's investigation, allowing readers to better appreciate its significance.\n\n2. The current research draws substantially from existing methodologies, including those used in FedFTG [4] and DENSE [5], particularly adopting the data-free knowledge transfer framework from Do et al. [6]. Consequently, it becomes challenging to discern the novel contribution of this paper. Greater emphasis on aspects like Dynamic Weighting and Label Sampling, currently found in the appendix, would have added more substance to the primary narrative of the manuscript. Additionally, a thorough exploration of the obstacles faced when adapting the adversarial data-free knowledge transfer framework [6] to a Heterogeneous FL setting would have enriched the study.\n\nMinor:\n1. In the related works section, the absence of adversarial DFKD methodologies, such as those in [1,2,3], which also aim to prevent catastrophic forgetting, is a notable omission. I suggest that the authors provide a clear rationale for their preference for the DFKD framework in [6] over the others ([3] could be excluded due its recentness), clarifying this would enhance the comprehensibility of the manuscript.\n\n\nReferences:\n\n[1] Kuluhan Binci, Nam Trung Pham, Tulika Mitra, and Karianto Leman. \"Preventing catastrophic forgetting and distribution mismatch in knowledge distillation via synthetic data.\" In WACV, 2022.\n\n[2] Kuluhan Binci, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. \"Robust and resource-efficient data-free knowledge distillation by generative pseudo replay.\" In AAAI, 2022.\n\n[3] Gaurav Patel, Konda Reddy Mopuri, and Qiang Qiu. \"Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation.\" In CVPR, 2023.\n\n[4] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. \"Fine-tuning global model via data-free knowledge distillation for non-iid federated learning.\" In CVPR, 2022.\n\n[5] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. \"Dense: Data-free one-shot federated learning.\" In NeurIPS, 2022.\n\n[6] Kien Do, Thai Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, and Svetha Venkatesh. \"Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation.\" In NeurIPS, 2022.\n\nQUESTIONS:\n1. I am interested in gaining a deeper understanding of the process behind the observations and interpretations, specifically those pertaining to transferability, decision boundaries, and the positioning of synthetic data-points (L165-180). Could the authors elaborate on the methodology or rationale that led to these interpretations?\n\n2. Furthermore, the visualization provided in Figure 2 has caught my attention. I would appreciate it if the authors could elucidate how this figure was constructed. Does this visual representation arise from actual data obtained through a small-scale experiment, or is it a conceptual illustration reflecting your interpretations?\n\n3. Regarding equation 5, I noticed that the KL divergence loss is applied directly to the logits (pre-softmax) features, as opposed to the prediction distribution resulting from the application of softmax to the logits. Could the authors provide a comprehensive explanation for this choice? Does this approach hold any specific advantages or implications for the overall results of your study?\n\nLIMITATIONS:\nThe authors have thoroughly listed out the limitations and broader impacts in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method called DFRD for robust and privacy-constrained Federated Learning (FL) in the presence of data and model heterogeneity. DFRD uses a conditional generator to approximate the training space of local models and a data-free knowledge distillation technique to overcome catastrophic forgetting and accurately extract knowledge from local models. The experiments show that DFRD achieves performance gains compared to baselines.\n\nSTRENGTHS:\n1. The writing is easy to follow\n\n2. The overall idea seems novel\n\n3. The authors justify the necessities of loss functions and validate their effectiveness in the experiment.\n\nWEAKNESSES:\n1. This work is well-engineered, comprising multiple components and hyperparameters. Although stability testing was performed on the same dataset, the optimal hyperparameters for different datasets appear to be unstable (see Fig 5). \n\n2. Although the authors propose loss functions aiming to improve fidelity and diversity, the synthetic images provided in the appendix are still far from achieving these goals. It is also questionable whether such characteristics are really necessary for downstream KD and ensembling.\n\n3. The EMA strategy to avoid forgetting is not new in continuous learning, so the novelty of this technique is limited in this paper.\n\n4. Based on the presented learning curves, the models do not seem to converge after 100 communication rounds for both the proposed and benchmark methods. This undermines the persuasiveness of the results reported in the tables.\n\n5. The overall performance is very low, e.g., less than 20% accuracy for 10-class classification. I have concerns about the practicality of the method given its complexity and unsatisfactory performance in tackling the challenging non-iid setting.\n\n6. I appreciate the authors' efforts to conduct numerous experiments, but how the settings overlap between experiments is not clear. It would be helpful if the authors could provide a table to show the connections. For example, I found it difficult to relate the values in the figures to the numbers in the tables. Ideally, the same experimental setting would result in the same number.\n\nMinor comments:\n\nThe pseudocode provided in the appendix is not well organized. There seems to be a convoluted operation between server and client.\n\nQUESTIONS:\n1. As this method already uses EMA on global model (G) to get \\tilde{G}, why does it need another step to find KL divergence on both G and \\tilde{G}?\n\n2 How the w, \\rho, and \\delta play roles in data heterogeneity?\n\n3 There is a line of work on KD for FL aggregation. Could the author justify why \" FedFTG [48] and DENSE [49], which are the most relevant methods to our work?\" comparing the other related work mentioned in the manuscript (such as [27], [40], [47])  and Wu et al. listed below:\n\nWu, C., Wu, F., Lyu, L., Huang, Y., & Xie, X. (2022). Communication-efficient federated learning via knowledge distillation.\n\nLIMITATIONS:\nLimitations are discussed in appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}