The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
Authors: Huancheng Chen
Chianing Wang
Haris Vikalo
Venue: ICLR 2023 poster (2023)
Keywords: Federated Learning
Representation Learning
Knowledge Distillation

Abstract: Heterogeneity of data distributed across clients limits the performance of global models trained through federated learning, especially in the settings with highly imbalanced class distributions of local datasets. In recent years, personalized federated learning (pFL) has emerged as a potential solution to the challenges presented by heterogeneous data. However, existing pFL methods typically enhance performance of local models at the expense of the global model's accuracy. We propose FedHKD (Federated Hyper-Knowledge Distillation), a novel FL algorithm in which clients rely on knowledge distillation (KD) to train local models. In particular, each client extracts and sends to the server the means of local data representations and the corresponding soft predictions -- information that we refer to as ``hyper-knowledge". The server aggregates this information and broadcasts it to the clients in support of local training. Notably, unlike other KD-based pFL methods, FedHKD does not rely on a public dataset nor it deploys a generative model at the server. We analyze convergence of FedHKD and conduct extensive experiments on visual datasets in a variety of scenarios, demonstrating that FedHKD provides significant improvement in both personalized as well as global model performance compared to state-of-the-art FL methods designed for heterogeneous data settings.

Published as a conference paper at ICLR 2023
THE BEST OF BOTH WORLDS: ACCURATE GLOBAL AND PERSONALIZED MODELS THROUGH FEDERATED LEARNING WITH DATA-FREE HYPER-KNOWLEDGE DISTILLATION
Huancheng Chen', Chianing Wang", Haris Vikalo* 'The University of Texas at Austin, TX Toyota Motor North America, CA
ABSTRACT
Heterogeneity of data distributed across clients limits the performance of global models trained through federated learning, especially in the settings with highly imbalanced class distributions of local datasets. In recent years, personalized fed- erated learning (pFL) has emerged a potential solution to the challenges pre- sented by heterogeneous data. However, existing pFL. methods typically enhance performance of local models at the expense of the global model's accuracy. We propose FedHKD (Federated Hyper-Knowledge Distillation), a novel FL algo- rithm in which clients rely on knowledge distillation (KD) train local models. In particular, each client extracts and sends to the server the means of local data representations and the corresponding soft predictions information that we refer to as it to the clients support of local training. Notably, unlike other KD-based pFI. methods, FedHKD does not rely a public dataset nor it deploys a generative model at the server. We analyze convergence of FedHKD and conduct exten- sive experiments visual datasets in variety of scenarios, demonstrating that FedHKD provides significant improvement in both personalized as well as global model performance compared to state-of -frt FL methods designed for hetero- geneous data settings.
The server aggregates this information and broadcasts
INTRODUCTION
Federated learning (FL), communicationn-eficcieen and crivacy-preserving alternative training centrally aggregated data, relies on collaboration between clients who own local data to train a global machine learning model. A central server coordinates the training without violating clients' privacy the server has no access to the clients' local data. The first ever such scheme, Federated Averaging (FedAvg) (MeMahan al. 2017), alternates between steps: (1) randomly selected client devices initialize their local models with the global model received from the server, and proceed to train on local data; (2) the server collects local model updates and aggregates them via weighted averaging to form a new global model. As analytically shown in (McMahan et al. 2017). FedAvg is guaranteed to converge when the client data is independent and identically distributed (iid).
A major problem in FL systems emerges when the clients' data is heterogeneous (Kairouz et al., 2021.. This common setting in practice since the data owned by clients participating in fed- erated learning is likely to have originated from different distributions. In such settings, the FL procedure may converge slowly and the resulting global model may perform poorly on the local data of an individual client. To address this challenge, number of FL methods aiming to enable learning on non-iid data has recently been proposed (Karimiredyy et al. 2220 [i et al. 2020 la Acar et al. 2021 Liu et al. 2021 Yoon et al. 2021 | Chen & Vikalo 2022 Unfortunately, ly, these methods struggle to train globa model that performs well when the clients' data distributions differ
significantly.
Difficulties of learning on non-iid data, as well as the heterogeneity of the clients' resources (c.g., compute, communication, memory, power), motivated a variety of personalized FI. (pFL.) techniques

---

Published as a conference paper at ICLR 2023
Arivazhagan et al. 2019 T Dinh et al. 2020 Zhang et al. 2020: Huang al. 2021 Collins al. 2021 Tan et al. (022). J. In pFL system, each client leverages information received from the server and utilizes customized objective to locally train its personalized model. Instead of focusing on global performance, a pFI. is concerned with improving the model's local performance empir- ically evaluated by running the local model data having distribution similar to the distribution of local training data. Since most personalized FL schemes remain reliant upon on gradient or model aggregation, they are highly susceptible 'stragglers' that slow down the training convergence pro- cess. FedProto Tan et al. (021) is proposed to address high communication cost and limitations of homogeneous models in federated learning. Instead of model parameters, in FedProto each client sends to the server only the class prototypes the means of the representations of the samples in each class. Aggregating the prototypes rather than model updates significantly reduces communica- tion costs and lifts the requirement of FedAvg that clients must deploy the same model architecture. However, note that even though FedProto improves local validation accuracy by utilizing aggre- gated class prototypes, it leads to barely any improvement in the global performance. Motivated by the success of Knowledge Distillation (KD) (Hinton et al. 2015) which infers soft predictions of samples as the 'knowledge' extracted from a neural network, number of FL methods that aim to improve global model's generalization ability has been proposed (Jeong et al. 2018b; Li & Wang, 2019 Lin al., 2020 Zhang al. 2021). However, most of the existing KD-based FL methods require that a public dataset is provided to all clients, limiting the feasibility of these methods in practical settings.
In this paper we propose FedHKD (Federated that relies on prototype learning and knowledge distillation to facilitate training on heterogencous data. Specifically, the clients in FedHKD compute mean representations and the corresponding mean soft predictions for the data classes in their local training sets; this information, which we refer to as slyper-knowledge. is endued by differential privacy via the Gaussian mechanism and sent for aggregation to the server. The resulting globally aggregated hyper-knowledge is used by clients in the subsequent training epoch and helps lead to better personalized and global performance. A number of experiments on classification tasks involving SVHN (Netzer et al.) (011). CIFARIO and CIFAR 100 datasets demonstrate that FedHKI) consistently outperforms -tate-of-the-art approaches in terms of both local and global accuracy.
Distillation), a novel FL framework
2 RELATED WORK
2.1 HETEROGENEOUS FEDERATED LEARNING
Majority of the existing work on federated learning across data-heterogeneous clients can be orga- nized in three categories. The first set of such methods aims reduce variance of local training by introducing regularization terms in local objective (Karimireddy et al., 2020 Li et al. 2020) a Acar al. 202T). (Mendieta al., 2022) analyze regularization-bass based FL algorithms and, motivated by the regularization technique GradAug in centralized learning ( (Yang er al. 2020), pro- pose FedAlign. Another set of techniques for FL on heterogeneous client data aims to replace the naive model update averaging strategy of FedAvg by more efficient aggregation schemes. To this end, PFNM (Yurochkin et al. 2019) applies Bayesian non-parametric method to select and merge multi-layer perceptron (MLP) layers from local models into a more expressive global model in layer-wise manner. FedMA ( ((Wang et al.] (020a)) proceeds further in this direction and extends
the same principle to CNNs and LSTMs. (Wang et al.) 2020b) analyze convergence of heteroge- neous federated learning and propose a novel normalized averaging method. Finally, the third set of methods utilize either the mixup mechanism (Zhang et al. 2017) generative models to enrich diversity of local datasets (Yoon et al.) 2021 Liu et al. 2021 Chen Vikalo 2022). However, these methods introduce additional memory/computation costs and increase the required communication resources.
2.2 PERSONALIZED FEDERATED LEARNING
Motivated by the observation that global model collaboratively trained on highly heterogeneous data may not generalize well on clients' local data, a number of personalized federated learning (pFL) techniques aiming to train customized local models have been proposed (Tan et al. 2022 They can be categorized into two groups depending on whether or not they also train a global model. The pFL techniques focused on global model personalization follow procedure similar to the plain vanilla FL clients still need to upload all or a subset of model parameters to the server to enable global model aggregation. The global model is personalized by each client via local adaptation
2

---

Published as a conference paper at ICLR 2023
steps such fine-tuning (Wang et al. 2019 Hanzely et al.] 2020;
Schneider & Vlachos
2021
creating a mixture of global and local layers (Arivahhagan et al. 2019 Mansour et al., 2020 Deng
cet al. 2020 Zec et al., 2020 Hanzely & Richtárik
Collins et al. 2021;Chen & Chao 2021
regularization (T Dinh et al. 2020 et 202 and meta learning (Jinn et 2019 Pallah et al,
(2200). However, when the resources available different clients vary, is impractical to require
that all clients train models of the same size and type. To address this, some works waive the global model by adopting multi-task learning (Smith et al. (0177) or hyper-network frameworks (Shamsian
et al., 2021 Inspired by prototype learning (Snell et al. 2017 Hoang et al.) 2020 Micciell &
Ozay
2021). , FedProto (Tan et al., (021) utilizes aggregated class prototypes received from the
server to align clients' local objectives via regularization term; since there is no transmission of
model parameters between clients and the server, this scheme requires relatively low communication resources. Although FedProto improves local test accuracy of the personalized models, it does not benefit the global performance.
2.3 FEDERATED LEARNING WITH KNOWLEDGE DISTILLATION
Knowledge Distillation (KD) (Hinton et al. 2015). a technique capable of extracting knowledge from neural network by exchanging soft predictions instead of the entire model, has been intro- duced to federated learning to aid with the issues that arise due to variations in resources (com- putation, communication and memory) available
the clients (Jeong et al., 2018a Chang et al. 2019 Itahara et al., (020). FedMD (Li & Wang, 2019), FedDF (Lin et al. 2020) and F-dKT- PFL (Zhang et al.j 2021) transmit only soft-predictions as the knowledge between the server and client models. However, these KD-based federated learning methods require that a public dataset is made available to all clients, presenting potential practical challenges. Recent studies (Zhu et al. 2021 Zhang et al. (022) explored using GANs (Goodfellow et al., 2014) to enable data-free federated knowledge distillation in the context of im- age classification tasks; however, training GANs incurs considerable additional computation and memory requirements.
clients, allowing for personalizeddhetrrrgeneee
In summary, most of the existing KD-based schemes require shared dataset help align local models; others require costly computational efforts to synthesize artificial data or deploy student model the server and update it using local gradients computed when minimizing the divergence of soft prediction on local data between clients' teacher model and the student model (Lin et al.) 2020). In our framework, we extend the concept of knowledge to 'hyper-knowledge' combining class prototypes and soft predictions on local data to improve both the local test accuracy and global generalization ability of federated learning.
METHODOLOGY
3.1 PROBLEM FORMULATION
Consider a federated learning system where m clients own local private dataset Di,..., Dm the distributions of the datasets may vary across clients, including the scenario in which a local dataset contains samples from only fraction of classes. such an FL system, the clients communicate locally trained models to the server which, in turn, sends the aggregated global model back to the clients. The plain vanilla federated learning (MeMahan et al. 2017) implements aggregation as
D M
W
where w² denotes parameters of the global model at round f; w denotes parameters of the local model of client at round t. - 1; is the number of participating clients; and M = Σm [D.]. The clients are typically assumed to share the same model architecture. Our aim is to learn a personalized model w; for each client which not only performs well on data generated from the distribution of the jth client's 's local training data, but can further be aggregated into a global model w that performs well across all data classes (i.e., enable accurate global model performance). This is especially difficult when the data is heterogenous since straightforward aggregation in such scenarios likely leads to inadequate performance of the global model.
3.2 UTILIZING SYPERRKKNWWLES
Knowledge distillation (KD) based federated learning methods that rely on a public dataset require clients to deploy local models to inference make predictions for the samples in the public
3

---

Published as a conference paper at ICLR 2023
dataset; the models' outputs are then used to form soft predictions according to
(T) T Σ, exp(2)/T)
where denotes the hhh element in the model's output for a given data sample; qi is the 1th ele- ment in the soft prediction q: and T is the so-called "temperature" parameter. The server collects soft predictions from clients (local knowledge), aggregates them into global soft predictions (global knowledge), and sends them to clients to be used in the next training round. Performing infer- ence on the public dataset introduces additional computations in each round of federated learning, while sharing and locally storing public datasets consumes communication and memory resources. It would therefore be beneficial to develop KD-based methods that do not require of public datasets; synthesizing artificial data is an option, but one that is computationally costly and thus may be impractical. To this end, extend the notion of distilled knowledge to include both the averaged representations and the corresponding averaged soft predictions, and refer to it as "hyper- knowledge"; the wyyeerknnwwedge"" is protected via the Gaussian differential privacy mechanism and shared between clients and server.
Feature Extractor and Classifier. We consider image classification as an illustrative use case. Typically, a deep network for classification tasks consists of two parts (Kang et al., 2019): (I) feature extractor translating the input raw data (i.e., an image) into latent space representation; (2) classifier mapping representations into categorical vectors. Formally,
h RR. (I:;), 2, = G=
where æ; denotes raw data of client i, Rø. (.) and Gw,, (.) are the embedding functions of feature extractor and classifier with model parameters Φs and ws, respectively; h; is the representation vector of x;; and z; is the categorical vector.
Evaluating and Using Knowedee. The mean latent representation of class j in the local dataset of client is computed as
N Q( ,T)
T
N Σ
where N is the number of samples with label j in client i's dataset; Q(,T) is the soft target function; f and z/ are the data representation and prediction of the 7th client's 1tth sample with label j. The mean latent data representation h² and soft prediction q are the hyper-knowledge of class j in client i; for convenience, we denote K. = (h,, q!). there are n classes, then the full hyper-knowledge of client i is K.; (K!,.... ..... K ). As comparison, FedProto (Tan et al.) 2021 only utilizes means of data representations and makes no use of soft predictions. Note that to avoid the situations where K?¹ = 0, which may happen when data is highly heterogeneous, FedHKD sets a threshold (tunable hyper-parameter) which is used to decided whether or client should share its tlype--nowleddee in particular, if the fraction of samples with label j in the local dataset of client i is below v, client i is not allowed to share the client sharing hyper-knowledge for class j. the server sets 1 = 0. A flow diagram illustrating the computation of hlyper-nooleedee is given in Appendix. A.3
K:/- If there is no participating
Differential Privacy Mechanism. It has previously been argued that communicating averaged data representation promotes privacy (Tan et 2021); however, hyper-knowledge exchanged between
server and clients may still be exposed to differential attacks (Dwork) 2008 Geyer et al. 2017). A
number of studies (Geyer et al., 2017 Sun et al. 202T Gong et al.) & Vikalo 2022) that utilize differential privacy to address security concerns in federated learning have been proposed. The scheme presented this paper promotes privacy by protecting the shared means of data representations through a differential privacy (DP) mechanism (Dwork et al. 2006a b) defined below.
| [221]] Ribero al. [202]] Chen
Definition 1 ((s, 8) Privacy) A randomized function F D + R provides (5, differential privacy if for all adjacent datasets d, d' E D differing on at most one element, and all S E range(FF, it holds that
TP(F(d) E E ST (Fd))) E 5
(5)
4

---

Published as a conference paper at ICLR 2023
where denotes the maximum distance between the range of F(d) and F(d') and may be thought of as the allotted privacy budget, while 8 is the probability that the maximun distance is not bounded by
Any deterministic function f D + R. can be endued with arbitrary (e, diifferrntt privacy via the Gaussian mechanism, defined next.
Theorem (Gaussian mechanism) A randomized function F from any deterministic func- lion] f D > R perturbed by Gaussian noise (0, 57 o²),
F(d)= ((d) N (0, S3
(6)
achieves (e, -idifferentt privacy for any a > / 3 / Here SJ denotes the sensitivity of
function f defined as the maximum of the absolute distance [f(d) f (d')).
We proceed by defining a deterministic function fi(d)) & (1 = 1 NO hg*((( which evalu-
No
ates the tth element of fig, where de is the subset of client i's local dataset including samples with label, j only; h. denotes the representation of the hhth sample in dy while h2.²(0 (1) is the 1hh element of hith In our proposed framework, client i transmits noisy version of its hyper-knowledge to the server,
(1) (1( + xl).
where (1) ~ (S))) o²); 0² denotes a param-prr shared by all clients. (5))² is the sensitive of function Si(.) with client i's local dataset.
Lemma If h/
1. (I) is bounded by Ç > 0 for any k, then
lfi(d) fi(d.)) 2
N
Therefore, S
25 Note that (8,)² depends NY. the number of samples in class j. and thus A discussion on the probability that differential privacy is broken can be found in the Section 4.3 Proof of Lemma i provided in Appendix A..
NO
differs across clients in the heterogeneous setting.
3.3 GL.OAL HYPER- KNOWLEDGE. AGGREGATION
After the server collects hyper-knowledge from participating clients, the global hyper-knowledge for class j at global round + 1, 1,,,,,4 KiH+1= = (Hi-+1 Qi++1), is formed as
m
m
H0+11
Σ C.h Qli+11 = -
where P: N No denotes the number of samples in class j owned by client i, and N E Ny. For clarity, we emphasize that h:r denotes the local hyper-knowledge about class of client i at global round t. Since the noise is drawn from N (0, (S)) .o).. its effect on the is alleviated during aggregation assuming sufficiently large number of -
quality of participating clients, i.e.,
E 7 (1)
P.h (1) + Iaxt
0 (1) >
p (1) + +0
(10)
with variance σ² ETh i= (S))². ?. In other words, the additive noise is "averaged out" and effectively near-eliminated after aggregating local syper-knowledge For simplicity, we assume that in the above expressions No- + 0.
3.4 LOCAL TRAINING OBJECTIVE
Following the aggregation at the server, the global hyper-knowledge is sent to the clients participat- ing in the next FL round assist in local training. In particular, given data samples (æ. y) D,, the loss function of client i is formed as
B:
1
C(D,, Ri wi) =
CELoss(Gu, (Rg,,
B
k=
(11)
1 (Q(G,,HH'),T @²112 +
Σ |R - 7/'1122
71
B,
5

---

Published as a conference paper at ICLR 2023
where B, denotes the number of samples in the dataset owned by client i, is the number of classes, CELoss(",)) denotes the cross-entropy loss function, || la denotes Euclidean norm, Q(: T)) is the soft target function with temperature T, and À and y syyper-paameett
Note that the loss function in (TT) consists of three terms: the empirical risk formed using predictions and ground-truth labels, and two regularization terms utilizing eepeee.nowledge. Essentially, the second and third terms in the loss function are proximity/distance functions. The second term is to force the local classifier to output similar soft predictions when given global data representations while the third term is to force the features extractor to output similar data representations when given local data samples. For both, we use Euclidean distance because is non-negative and convex.
3.5 FEDHKD: SUMMARY OF THE FRAMEWORK
The training starts at the server by initializing the global model 0² = ( , where Φf and we denote parameters of the global feature extractor and global classifier, respectively. At the beginning of each global epoch, the server sends the global model and global hyper-knowledge to clients se- lected for training. In turn, each client initializes its local model with the received global model, and performs updates by minimizing the objective in Eq. TT the objective consists of three terms: (I) prediction loss in a form of the cross-entropy between prediction and ground-truth, (2) classifier loss reflective of the Euclidean norm distance between the output of the classifier and the corresponding global soft predictions; and (3) feature loss given by the Euclidean norm distance between represen- tations extracted from raw data by local feature extractor and global data representations. Having completed local updates, clients complement their local -noweede by performing inference on local data, and finally send local model as well as local hyper-knowledge the server for ag- gregation. The method outlined in this section is formalized as Algorithm For convenience, we provided visualization of the FedHKD procedure in Appendix. A.4
Algorithm FedHKD
Input:
D Wf + LocalUpdate(ds w K,D,,. a² 1,
Datasets distributed across m clients, D (D1, D.,... Dm]; client participating rate μr; hyper-parameters À and y the sharing threshold v; variance o² characterizing dif- ferential privacy noise; temperature T; the number of global epochs Tp-
8:
end for
9:
Aggregate global hyper-knowledge K. by
Eq. g
10:
Aggregate global model (Φ'+1, _ w
r++1
Output:
11: end for
The global model 0Ti+1 T6 (,,1, 11)
12: return 0T. (ob (i:-t,,,ii- -1)
13:
1: Server executes:
14: LocalUpdate( K',,,,, 02 i):
2: randomly initialize (s' w/), K. = 0
15: Φf f w t- w', (I, D:
3: for /. = 1,,,,..,, T, do
16: for each local epoch do
4: S. + [my]] clients selected at random
17: (6, w OptimAlg C(x, K, X,)))
5: send the global model o' ,w', K. to clients
18: end for
in St
19: update local hyper-knowledge K.;
6: for E S do
20: return 6i, Ki
3.6 CONVERGENCE ANALYSIS
To facilitate the convergence analysis of FedHKD, we make the assumptions commonly encountered
in literature (Li et al. 2019 2020 Tan et al. 2021) The details in assumptions and proof are in
Appendix A.6
Theorem 2. Instate Assumptions 1-3 round the loss function is bounded as
A.
For an arbitrary client, after each communication
E
++11
<
C.
E
|
EL (EV2 + +02)
2
2
(12)
2
+ 2AmoLs (L2 + 1) EV + 2YTOLZE..

---

Published as a conference paper at ICLR 2023
Theorem 3. (FedHKD convergence rate) Instate Assumptions 1-3 A.6.1 hold and define regret A L L If the learning rate is set v7 for arbitrary client after
2A
T eF - (FV2 FVV
(13)
o²²) 42m/.3 (I2 1) EV 4y
global rounds (c > 0), it holds that
VC 12
(14)
TE
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
In this section, we present extensive benchmarking results comparing the performance of FedHKD and the competing FL methods designed to address the challenge learning from non-iid data, All the methods were implemented and simulated in Pytorch (Paszke et al., 2019), with models trained using Adam optimizer (Kingma Ba 2014). Details of the implementation and the selection of hyper-parameters are provided in Appendix. Below we describe the datasets, models and baselines used in the experiments.
Datasets. Three benchmark datasets used in the experiments: SVHN Netzer al., 201 CIFARIO and CIFAR 100 (Krizhevsky et al.] 2209). To generate heterogeneous partitions of local training data, follow the strategy (Yoon al. 2021 Yurochkin al. 2019 Li al. 202 lab and utilize Dirichlet distribution with varied concentration parameters / which controls the level of heterogeneity. Since our focus on understanding and addressing the impact of class heterogeneity in clients data on the performance of trained models, we set equal the size of clients' datasets. Fur- thermore, to evaluate both personalized as well as global model performance, each client is allocated local test dataset (with the same class distribution as the corresponding local training dataset) and global test dataset with uniformly distributed classes (shared by all participating clients); this allows computing both the average local test accuracy of the trained local models as well as the global test accuracy of the global model aggregated from the clients' local models.
Models. Rather than evaluate the performance of competing schemes simple CNN network as in (McMahan et al.] 2017: Li et al.) 2020 202Ia), we apply two widely used benchmarking models better suited practical settings. Specifically, deploy ShuffleNetV2 (Ma et al.. 2018) on SVHN and ResNet18 8 (He et al.] (016)) on CIFAR1Ó/100. As our results show, FedHKD generally outperforms competing methods both (very different) architectures, demonstrating remarkable consistency and robustness.
Baselines. We compare the test accuracy of FedHKD with seven state-of-the-art federated learning methods including FedAvg (McMahan et al., 2017), FedMD (Li & Wang 2019), FedProx (Li et al., 2020), Moon (Li et al. 2021a), FedProto (Tan al. 2021). FedGen (Zhu et al. (02I) and FedAlign (Meddicta et al. 2022). We emphasize that the novelty of FedHKD lies in data-free knowledge distillation that requires neither a public dataset nor a generative model; this stands in contrast to FedMD which relies on a public dataset and FedGen which deploys generative model. Like FedHKD, FedProto shares means of data representations but uses different regularization terms in the loss functions and does not make use of soft predictions. When discussing the results, we will particularly analyze and compare the performance of FedMD, FedGen and FedProto with the performance of FedHKD.
4.2 PERFORMANCE ANALYSIS
Table shows that FedHKD generally outperforms other methods across various settings and datasets. For each dataset, we ran experiments with 10, 20 and 50 clients, with local data gener- ated from a Dirichlet distribution with fixed concentration parameter 3 0.5. As previously stated, we focus the heterogeneity in class distribution of local dataset rather than the heterogeneity in the number of samples. To this end, an increasing fraction of data is partitioned and allocated to the clients in the experiments, maintaining the size of local datasets as the number of clients increases. A single client's averaged training time per global round is computed across different settings to char- acterize the required training time. To provide more informative comparison with FedProto (Tan)
7

---

Published as a conference paper at ICLR 2023
Table 1: Results on data partitions generated from Dirichlet distribution with the concentration
parameter / = 0.5. The number of clients is 10, 20 and 50; the clients utilize 10%, 20% and 50% of the datasets. The number of parameters (in millions) indicates the size of the model stored in the memory during training. A single client's averaged wall-clock time per round is measured across 8 AMD Vega20 GPUs in a parallel manner.
Dataset
Scheme
Local Acc
Global Ace
Params
Time
Puh Data
# Chienis
10
20
50
10
20
50
FedAvg
0.6766
0.7329
0.6544
0.4948
0.6364
0.5658
1.286
5.22
No
FedProx
0.6927
0.6717
0.6991
0,5191
0.6419
0.6139
2.572
5.56
No
Moon
0.6602
0.7085
0.7192
0.4883
0.5536
0.6543
3.858
12.32
No
FedAlign
0.7675
0.7920
0.7656
0.7138
0.7437
1.286
16.67
No
SVIN
FedGen
0.5788
0.5658
0.4679
0,3622
0.3421
0.3034
1.357
6.66
No
FedMD
0.8038
0.8086
0.7912
0.6812
0.7344
0.8085
1,286
10.67
Yes
FedProto
0.8071
0.8148
0.8039
0.6064
0.6259
0.7895
1.286
5.42
No
FedHKD
0.8064
0.8157
0.8072
0.6405
0.6884
0.7921
1 1,286
5.70
No
FodHKD
0.8086
0.8381
0.7891
0.6781
0.7357
0.7891
1,286
6.33
No
FedAvg
0.5950
0.6261
0,5825
0.4741
0.5516
0.1773
1 11,209
8.71
No
FedProx
0.5981
0.6295
0.6490
0.4793
0.5258
0.5348
22,418
10.25
No
Moon
0.5901
0.6482
0.5513
0.4579
0.5651
0,3514
33.627
20.52
No
FedAlign
0.5948
0.6023
0.6402
0.4976
0.5134
0.5641
11.209
36.24
No
CIFARIO
FedGen
0.5879
0.6395
0.6533
0,4800
0.5408
0.5651
11.281
10.52
No
FedMD)
0.6147
0.6666
0.6533
0,5088
0.5575
0.5714
11.209
22.51
Yes
FedProto
0.6131
0.6505
0,5012
0.5548
11,209
11.68
No
FedHKD>
0.6227
0.6515
0.6675
0,5049
0.5596
0.5074
11,209
11.26
No
FodHKD
0.6254
0.6816
0.6671
0.5213
0.5735
0.5493
11,209
12.83
No
FedAvg
0.2361
0.2625
0.2658
0.2131
0.2748
0.2907
11,215
14.17
No
FedProx
0.2332
0,2814
0.2955
0.2267
0.2708
0.2898
22,430
19.81
No
Moon
0.2353
0.2729
0.2652
33,645
36.28
No
FedAlign
0.2467
0.2617
0,2854
0.2281
0.2729
0.2933
11.215
27.61
No
CIFARIO0
FedGen
0.2393
0.2701
0.2739
0.2176
0.262
0.2739
11.333
17:45
No
FedMD
0.2681
0.3054
0.3293
0.2323
0.2669
0,2968
11.215
29.04
Yes
FedProto
0.2568
0.3188
0.3170
0.2121
0.2756
0.2805
11.215
14.88
No
FedHKD> 0.2551
0.2997
0.3016
0.2286
0.2715
0.2976
11,215
14.59
No
FedHIDD
0.2981
0.3245
0.3375
0.2369
0.2795
0.2988
11,215
15.14
No
Table 2: Results on data partitions generated with different concentration parameters (10 clients).
Local
Local
CIFARTO
SVHN
B 0.2
3 b
= 0.2
B
B 0.2
3 = 5
= 0.2
FodAvg
0.5917
0.4679
0.325
0.3483
0.6227
0.3833
0,2581
0.6238
FedProx
0,6268
0.4731
0.3845
0.5521
0.7481
0.6598
0.4325
0.7121
Moon
0.5762
0.3794
0.3229
0.4256
0.7440
0.6568
0.3764
0.7128
FedAlign
0.6434
0.4799
0.4446
0.5526
0.8161
0.7414
0.5904
0.7919
FedGen
0.6212
0.4432
0.4623
0.4432
0.7248
0.6542
0.5304
0.7251
FedMD
0.6532
0.5543
0.8415
0,8144
FedProto
0.6471
0,4802
0.3887
0.5488
0.8446
0.7363
0.5493
0.8055
*
0.6798
0.4857
0.4459
0.5494
0.8344
0.7314
0.5357
0,8044
FedHKD
0.6789
0,4976
0.4736
0,5573
0.8462
0.7420
0.6241
0.8083
al. 2021), we ran two setting of our proposed method, labeled as FedHKD and FedHKD*; (1) Fe deploys the second and third term in Eq. [T]iing À 0.05 and 7 0.05; (2) FedHKD* excludes the constraint on Feature Extractor Rs by setting À = 0.05 and 7 = 0.
Accuracy comparison. The proposed method, FedHKD, generally ranks as either the best or the second best in terms of both local and global accuracy, competing with FedMD without using public data. On SVHN, FedHKD significantly improves the local test accuracy over FedAvg (by 19.5%, 14.3% and 20.6%) as well as the global test accuracy (by 37. 17.0% 15.6% and 39.5%) ) in experiments involving 10, 20 and 50 clients, respectively. The improvement over FedAvg carry over the exper- iments on CIFARIO 10, with 5.1 %, 8.9% and 14.5% increase in local accuracy and 14.5%, 9.9% and 45.6% increase in global accuracy in the experiments involving 10, 20 and 50 clients, respectively. On CIFAR 100, the improvement of global accuracy is somewhat more modest, but the improvement in local accuracy is still remarkable, outperforming FedAvg by 26.3%, 23.6% and 26.99 26.9% the ex- periments involving 10, 20 and 50 clients, respectively. The local test accuracies of FedHKD* and FedProto are comparable, but FedHKD* outperforms FedProto in terms of global test accuracy (as expected, following the discussion in Section 3.2h. FedAlign outperforms the other two regulariza- tion methods, FedProx and Moon, both locally and globally; however, but is not competitive with the other methods in which clients' local training is assisted by additional information provided by the server. While it has been reported that FedGen performs well on simpler datasets such as MNIST (LeCun et al. 1998) and EMNIST (Cohen et al. 2017), it appears that its ML-P-based gen-
8

---

Published as a conference paper at ICLR 2023
erative model is unable to synthesize data of sufficient quality to assist in KD-based FL on SVHN and CIFARIO/100 - on the former dataset, FedGen actually leads to performance deterioration as compared to FedAvg.
Training time comparison. We compare training efficiency of different methods in terms of the averaged training time (in second) per round/client. For faimess, all the experiments were ducted on the same machine with AMD Vega20 GPUs. As shown in Table | the training time of FedHKD, FedHKD*, FedProto and FedGen is slightly higher than the training time of FedAvg. The additional computational burden of FedHKD is due to evaluating two extra regularization terms and calculating local syper-knowledger The extra computations of FedGen are primarily due to training a generative model; the MLP-based generator leads to minor additional computations but clearly limits the performance of FedGen. FedMD relies on a public dataset of the same size as the clients' local datasets, thus approximately doubling the time FedAvg needs to complete the forward and backward pass during training. Finally, the training efficiency of Moon and FedAlign is inferior to the training efficiency of other methods. Moon is inefficient it requires more than double the training time of FedAvg. FedAlign needs to pass forward the network multiple times and runs large matrix multiplications to estimate second-order information (Hessian matrix).
Effect of class heterogeneity. We compare the performance of the proposed method, FedHKD, and other techniques as the data heterogeneity is varied by tuning the parameter B. When 3 0.2, the heterogeneity is severe and the local datasets typically contain only one or two classes; when / = 5, the local datasets are nearly homogeneous. Data distributions are visualized in Appendix [A.2] As shown in Table 2 FedHKD improves both local and global accuracy in all settings, surpassing other methods except FedMD on SVHN dataset for 3 5. FedProto exhibits remarkable improvement on local accuracy with either extremely heterogeneous (8 = 0.2) or homogeneous (B 5) local
data but its global performance deteriorates when 8 0.2.
4.3 PRIVACY ANALYSIS
In experimental setting, clients share the same network architecture (either ShuffleNetV2 or ResNet18). In both network architectures, the outermost layer in the feature extractor a batch normalization (BN) layer (Toffe & Szegedy 2015). For batch of vectors B {V1,..., 1%) the input of the BN layer, the operation of the BN layer is specified by
PB
U.. B
c UB)
U/ PB
(15)
0B
Assuming b is sufficiently large, the law of large numbers implies é ~ ~O 1). Therefore. -3 < ey S with probability 99.73% (almost surely). Consider the experimental scenarios where client i contains N, = 1024 samples in its local dataset, the sharing threshold is v = 0.25, N > =N; = 256, 5 = 0.01, and = 0.5. According to Theorem to obtain 0.5-differential privacy with (5))²
confidence 1 - -5- 99% we set a
2
5 is / 6.215. According to Lemma
2
256 j. Setting o = 7 (large privacy budget), the variance of noise added to the hyper-
knowledge K?: of client i should be (5))20² 0.0269.
CONCLUSION
We presented FedHKD, novel FL algorithm that relies on knowledge distillation enable effi- cient learning of personalized and global models in data heterogeneous settings; FedHKD requires neither a public dataset nor a generative model and therefore addresses the data heterogeneity chal- lenge without a need for significantly higher resources. By introducing and utilizing the concept of khoper-knowledge""" information that consists of the means of data representations and the cor- responding means of soft predictions, FedHKD enables clients to train personalized models that perform well locally while allowing the server to aggregate a global model that performs well across all data classes. To address privacy concerns, FedHKD deploys differential privacy mechanism. We conducted extensive experiments in a variety of setting on several benchmark datasets, and pro- vided theoretical analysis of the convergence of FedHKD. The experimental results demonstrate that FedHKD outperforms ofate-of-the-an federated learning schemes in terms of both local and global accuracy while only slightly increasing the training time.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263, 2021.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary, Fed- erated learning with personalization layers. arXiv preprint arXiv:1912.008888 2019
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv: 2019.
Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2021.
Huancheng Chen and Haris Vikalo. Federated learning in non-iid settings aided by differentially private synthetic data. arXiv preprint arXiv:2206.0068. 2022.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference neural networks (IJCNN,, pp. 2921-2928 IEEE, 2017.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International Conference on Machine Learning, pp.
PMLR, 2021.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 1346/, 2020.
Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation, pp. 1-19. Springer, 2008
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Annual international conference on the theory and applications of cryptographic techniques, pp. 486-503. Springer, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006b.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568, 2020.
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv: 2017.
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15076-15086. 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neurul information processing systems, 27, 2014.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.055/6 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and opti- mal algorithms for personalized federated learning. Advances in Neural Information Processing Systems, 33:2304-223 2,,,,
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint AXXiv:5533002533 2(7), 2015.
Nghia Hoang, Thanh Lam, Bryan Kian Hsiang Low, and Patrick Jaillet. Learning task-agnostic em- bedding multiple black-box experts for multi-task model fusion. International Conference on Machine Learning, pp. 4282 4292. PMLR, 2020.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In AAAI, pp. 7865-7873, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-45646 PMLR, 2015.
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. Distillation-bar semi-supervised federated learning for communication-effiiinn collaborative training with non-iid private data. arXiv preprint arXiv:2008.061 180. 2020.
E Jeong, S Oh, H Kim, J Park, M Bennis, and SL Kim. Federated distillation and augmentation under non-iid private data. NIPS Wksp. MLPCD, 2018a.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong- Kim. Communication-effiieen on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint mrXiv:18111 1.479,, 2018b.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per- sonalization via model agnostic meta learning. arXiv preprint 19XX:::909.124400 2019.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 1911v:1910.09217. 2019.
Sai Praneeth Karimireddy, Satyen Kale. Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143, PMLR, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 142.699412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. N/A, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2324 1998.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition, pp. 10713-10722, 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 454,9-450, 2020.
11

---

Published as a conference paper at ICLR 2023
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR. 2021b.
Xiang Li, Kaixuan Huang. Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg non-iid data. arXiv preprini arXiv: (rXXiv:997..02/88 2019.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning, Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain gen- eralization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1013-1023, 2021.
Ningning Ma, Xiangyu Zhang. Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference computer vision (ECCV), pp. 111--31,22 2018.
Yishay Mansour, Mehryar Mohri, Jae Ro. and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint urXiv:2002. 10619, 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efiie learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1282. PMLR, 2017.
Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8397-8406, 2022.
Umberto Michieli and Mete Ozay. Prototype guided federated learning of visual feature representa- tions. arXiv preprint 2rXiv:2105.08982, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, N/A, 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Mónica Ribero, Jette Henderson, Sinead Williamson, and Haris Vikalo. Federating recommenda- tions using differentially private prototypes. Pattern Recognition, 129:108746, 2022.
Johannes Schneider and Michalis Vlachos. Personalization of deep learning. In Data Science- Analytics and Applications, pp. 89-96. Springer, 2021.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. In International Conference on Machine Learning, pp. 9489-9502. PMLR, 2021.
Virginia Smith, Chao-Kai Kai Chiang. Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, Ad- vances neural information processing systems, 30, 2017.
Jingwei Sun, Ang Li, Binghui Wang, Huanrui Yang. Hai Li, and Yiran Chen. Soteria: Provable de- fense against privacy leakage in federated learning from representation perspective. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93 9311-9319, 2021.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in Neural Information Processing Systems, 13:21394-2144 2020.
12

---

Published as a conference paper at ICLR 2023
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning over heterogeneous devices. arXiv preprint arXiv:2105.00245 2021.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoolos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint Siti:::002.004444 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7623 2020b.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.1023: 2019.
Taojiannan Yang, Sijie Zhu, and Chen Chen. Gradaug: A new regularization method for deep neural networks. Advances in Neural Information Processing Systems, 33:14207-14218, 2020.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. arXiv preprint arXiv:2107.0023: 2021.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Interna- tional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Edvin Listo Zec, John Martinsson, Olof Mogren, Leon René Sütfeld, and Daniel Gillblad. Federated learning using mixture of experts. arXiv preprint arXiv, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical 2017.
risk minimization. arXiv preprint
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang. Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Process- ing Systems, 14:10022-10104, 2021.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 10174-10183, 2022.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized feder- ated learning with first order model optimization. arXiv preprint arXiv:2012.08565, 2020.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning, pp. 18878-18889, PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 EXPERIMENTAL DETAILS
General setting. We implemented all the models and ran the experiments in Pytorch (Paszke al. 2019) (Ubuntu 18.04 operating system. 8 AMD Vega20 GPUs), Adam (Kingma & Ba (014) opti- mizer was used for model training in all the experiments; learning rate was initialized 0.001 and decreased every 10 iterations with a decay factor 0.5, while the hyper-parameter 7 in Adam was set to 0.5. The number of global communication rounds was set to while the number local epochs was set to 5. The size of data batch plicity set to 1. For SVHN (Netzer et al. (011) dataset, the latent dimension of data representation was set to 32; for CIFARIO/1I 100 (Krizhevsky et al. 2009), the latent dimension set to 64.
set to 64 and the participating rate of clients was for sim-
Hyper-parameters. In all experiments, the FedProx eLi al. 2020) typer-parametee Mprox was set to 0.5; the Moon (Li et al.
2021a) hyper-parameter Pmova in the proxim'Tal term was set to 1. In FedAlign (Mendieta et al.] 2022). the fractional width of the sub-network was set to 0.25, and the balancing parameter Malign was
0.45. The generative model required by FedGen (Zhu al. 2021] is the MLP-based architecture proposed in (Zhu et al., (021). The hidden dimension of the generator was set 512: the latent dimension, noise dimension, and input/output channels were adapted to the datasets. The number of epochs for training the generative model in each global round was set 5, and the ratio of the generating batch-size and the training batch-size was set to 0.5 (i.e, the generating batch-size was set to 32). Parameters O'generative and B Peeneratvee were initialized 10 with a decay factor 0.98 in each global round.
FedMD (Li Wang 2019), we set the regularization hyper-parameter Amd to 0.05; the size of the public dataset was set equal to the size of the clients' local training dataset. FedProto (Tan et al.) 202T), the regularization hyper-parament Aproto was set to 0.05. The hyper-paramest to 0.05 and 0, respectively; as for FedHKD, the two 0.05, respectively. Variance of the Gaussian noise added the generated hyper-knowledge was set to 7; threshold y that needs to be met to initiate computation of hyper-knowledge was set to 0.25.. Temperature for FedHKD and Moon algorithm
À and y in proposed method FedHKD* were set and r were set to 0.05 and
to 0.5.
A.2 DATA PARTITIONING
For convenience, we used datasets encapsulated by Torchvision To obtain the global test dataset, we directly load SVHN, CIFAR1O and CIFAR100 test set in Torchvision without any sampling. For the local training and test sets, first utilized Dirichlet distribution sample partitions local datasets from the encapsulated set (m denotes the number of clients). Then we divided the
local dataset into training and test set in 75%/25% proportion. Figures [ 2 and B visualize the
class distribution of local clients by showing the number of samples belonging to different classes
each client (colors distinguish the magnitude the darker the color, the more samples are in the corresponding class)
300
30
250
259
308
259
Cient
350
Dlant
106
10
(a) 8 = 0.2
(b) 8 0.5
(c) 3 =
Figure 1: 10% of the training set points in CIFARI0 are sampled into 10 partitions according to a Dirichlet distribution (10 clients). As the concentration parameter varies (3 = 0.2, 0.5, 5), the partitions change from heterogeneous to homogeneous.
14

---

Published as a conference paper at ICLR 2023
Class ID
80
175
150
10
125
Client ID
100
30
75
50
25
Figure 2: 50% of the training set points in CIFARIO are sampled into 10 partitions according to Dirichlet distribution (50 clients). With concentration parameter 3 0.2, the partition extremely heterogeneous.
Class ID
20
40
60
80
25
10
20
Client ID
155
30
10
-5
Figure 3: 50% of the training set points in CIFAR100 are sampled into 10 partitions according to
Dirichlet distribution (50 clients). With concentration parameter 3 5. the partition is relatively homogeneous.
15

---

Published as a conference paper at ICLR 2023
A.3 FLOW DIAGRAM ILLUSTRATING COMPUTATION OF HYPER- -EEER-KNOWLDGGE
Figure illlstrates computation of local hyper-knowledge by client. At the end of local training, each participating client obtains fine-tuned local model consisting of feature extractor R+(+) and a classifier (-). There are three steps in the process of obtaining local hyper-knowledge for class of client k: (1) Representations of data samples in class j. generated by the feature extractor, are used to compute the mean of data representations for that class; (2) A classifier generates soft predictions for the obtained data representations, thus enabling computation
the mean of soft predictions for class j: (3) After adding Gaussian noise to the mean of data representations, the noisy mean of data representations and mean of soft predictions are packaged into local hyper-knowledge for class j.
Client
BHB BESIIEE I10CON
cat
adding nose DELERNIN RIOEE!!
cat
R
dog
adding
dog
data
feature extractor
bird
ackling nose
data representations
classifier
averages
soft predictions
averages
lyper-innwlldde of client
Pypper Hipee-knowtedge of ciemm
Hyper-knowwedde of cat soccasssss
aggregation
hyper knowledgg of cat
DOORDOOM
hypar-knnueedga of dog:
byper knowledge of dog.
Server
Pyper knowlddge of bird.
10000000
hyper knw.eddg of bird
Figure 4: A flow diagram showing computation, encryption and aggregation of hyper-nnowledge.
16

---

Published as a conference paper at ICLR 2023
A.4 DETAILS OF THE FEDHKD ALGORITHM
Figure. [ illustrates the iterative training procedure of FedHKD. At the start of training, global hyper-knowledge is initialized an empty and thus in round 1 each client trains its local model without global hyper-knowledge. Following local training, each client extracts representations from local data samples via feature extractor and finds soft predictions via a classifier, computing local hyper-knowledge as shown in Figure. The server collects local updates from clients, aggregates them into global hyper-knowledge and model, and then sends the results back to the clients. From this point on, clients perform local training aided by the global knowledge, Alternating local training and aggregation lasts for T rounds where denotes the number of global epochs.
and model
Global round St Train local movel
Global round 1: Train local model
extract lenooeedges uning local und detu
Client/Edge
Glubal Trale medel
Colo/Edge
gluuel browsedgees
uddeal knowlcdgess
inowdedgge using
Server
for aggregation
Global round Tran local model lecwdeigess
Global round trais local model aided knowledgee
"anouledgee medel
local hyper knowledges local model updates
global hyper knowledges global madel
Figure 5: A flow diagram showing FedHKD steps. The blue dashed line indicates sending local hyper-knowledge and model updates from clients to the server while the green dashed line indicates broadcasting global hyper-knowledge and model from the server clients.
17

---

Published as a conference paper at ICLR 2023
A.5 PROOF OF LEMMA 1
To compute 8 client's mean of class j representation, Rt, we consider the deterministic function
(averaging in an element-wise manner) fi(d,) 12 N Σx k= (1) where d' is the subset
of the 1th client's local dataset collecting samples with label j: h.t denotes the data representation of the hthh sample in di while hz)(()) (1) is the 1th element of h.*
Lemma 1. If |ng (4) is bounded by > 0 for any k, then
- - si(d")) 2 N/¹
(16)
Proof:
Without loss of generality, specify
= the (4)
No h4 (4), N3 (4), [el = =N/,
(17)
and
e' = (M)(0 (M) N (0). (0), |e'| = N 1,
(18)
where e and e' denote adjacent sets differing in at most one element. Define = (1,..., 1 with
| = NY - 1. Then
e'ee' + h NO
(1)
I'ee
(i(d)) s(d(()
N
N
No. 1) h (1) eTee
N! (N)
(Nz -1- A (1)
1Td
(19)
+
N/ (N) -
No N -1]
(Ng - 1)
(NY - -1)
No (N -1)
N2 NNg 11)
5
2
N
N N?
18

---

Published as a conference paper at ICLR 2023
A.6 CONVERGENCE ANALYSIS OF FEDHKD
It will be helpful to recall the notation before restating the theorems and providing their proofs. Let Rø. (-) Rddr Rd, denote the feature extractor function of client i, mapping the data of dimension d, into the representation space of dimension d,. Let (+) Rd. R denote the classifier's function of client i, projecting the data representation into the categorical space of dimension n. Let Fo,=(d,,..)) (+) Gw.(C (-) Rø. (-) denote the mapping of the entire model. The local objective function of client is formed as
B.
1
L(D;, (.)
(Ro,
B.
(20)
1 (H)), 2 7 BB, Σ
n
Σ
B, Ro. (2xx) 77 [|2,
==1
where Di denotes the local dataset of client i; input a% and label y/k are drawn from D;; B; is the number of samples in batch of D; Q(; is the soft target function with temperature T; HP denotes the global mean data representation of class j; Q the corresponding global soft prediction of class y&: and À and y the hyper-pan paree--arameerss. Note that only Φ; and w; are variables in the loss function while the other terms are constant.
Let denote the current global training round. During any global round, there are E local training epochs. Assume the loss function is minimized by relying on stochastic gradient descent (SGD). To compare the loss before and after
aggregation the server, denote the local epoch by e E 1 1...., E}: e = & indicates the epoch between the end of the server's aggregation in the previous communication round and the first epoch of the local training in the next round. After E epochs of local training in communication round i, the local model of client is denoted as (Ø. E.t (..)). At the global communication round + 1, client i initializes the local model with the aggregated global model, (. : d Although client i does not begin the next training epoch, the local model changed and so is the output of the loss function, At the server, the global model is updated
g+
P.
(21)
where 0 is the local model of client i after E local training epoches at round t; pı is the averaging weight of client i, where Em Pi 1. h and q are aggregated
Hott1
h
(22)
Qdi++1
p.
(23)
A.6.1 ASSUMPTIONS
Assumption 1. (Lipschitz Continuity). The gradient of the local loss function L(.) is Li-Lischhitz continuous, the embedding functions of the local feature extractor Rg (-) is Lz-Lipschitz continuous, and the embedding functions of the local classifier G (-) composition with soft prediction function Q(- is L3- Lipschitz continuous,
|VC(0)) - VC(0')) S L1 | 0² 11 Vt,, to 0,
(24)
Vt,, to > 0, | (.) Ro (+)1 s L2 - 0 1|,
(25)
I|Q (Gutz (-) - Q (Gw's (+)) s La - V1, 0.
(26)
Incquality 24 also implies
(((')) -(('²) < (VC(0")) 8hh 8²2) +
| g/2|13
Vt, 1 >0 0.
(27)
2
19

---

Published as a conference paper at ICLR 2023
Assumption 2. (Unbiased Gradient and Bounded Variance). The stochastic gradients batch of client i's data Ço. denoted by gf V (0) 5.), is an unbiased estimator the local gradient for each client i,
S D: [ VL 1,,,2,....
(28)
with the variance bounded by o²,
E g VC (0:) so², vi (1, o> > 0.
(29)
Assumption 3. (Bounded Expectation of Gradients). The expectation of the stochastic gradient is bounded by V.
m), E |1 S V2, e (1,2..... ...,,,.,,,........ V >> 0.
(30)
A.6.2 LEMMAS
Lemma 2. Instate Assumptions 1-3. The loss function after E' local training epoches at global round ++ can be bounded
L1 Te 2
LE
ct.+1
11
(31)
2
where Te is the step-size (learning rate) at local epoch e.
Proof:
(1) cotLet1 S cattl + [e -
+ Li
10ttlet1 - 0
2
(32)
I1 r
1
C T7e V g(tt)) +
nc
g
e E { 1, 1....,. 1}.
2
where inequality (1) follows from Assumption 1. Taking expectation of both sides (the sampling batch 1.+1), obtain
E [ettet1
(2) S gatt1 De V
+
L1
n
||
c
11]
2
2
3 cat+1 7e [c
+1
2
L1
2
ne
2
[[ ()
V
(33)
(4) cat+1_ <
V
2
2
+
L1
12200
2
2
Inequality (2) follows from Assumption 2: (3) follows from V E random variable; (4) holds due Assumptions 2-3. Let us set the learning step the start of local training to 77 = 7oo- By telescoping.
E [x]] where x is
L 2
E [[ < Ld.t11
Σ
112
?e
(34)
The above inequality holds due to the fact that the learning rate 17 is non-increasing.
Lemma 2. Following the model and hyper-knowlede aggregation at the server, the loss function of any client at global round + 1 can be bounded as
E C
(.(++) < C
L E2V² + 2N (L2 + +1) EV + LyV
(35)
2
20

---

Published as a conference paper at ICLR 2023
Proof:
c2.((+1)_
-ce C(0 K(Ht) L( ,1t)
C(0 111
111 k - C k(tt)
+ L(O xx+t1) -( ,0''
(ve.t E. oF x11 0.:0)
L
g 2
0
2
+ C. k(tt
RM K²)
(36)
L 2
E,t V
p2
Ei
+
Σ
P3
0 E,
0
j=1
+ C( k L( (())
where inequality (1) follows from Assumption 1, and (2) is derived from Eq. of both side,
21
Taking expectation
(1)
L
E
-
E.
E
Σ p g
+ (EC(O,,,,Ct)) EE(O,,,
2
L1
m
0
(0..1 - 1. -
Ps
2
1
+ EC F.
K
EC(OEA F.t. FK))
(2) 1
2
E
0
+ EL 1(tH)) EL( ,K))
E-1
(37)
LB
E
Σ
Ne9
c.
EECOEE, ECCORA
2
(3)
L E
B-1
Em
9
2 (CC(BI,,X41 EEC(OE,
2
(4)
11 L
E-1
E
E
8. 9
1 +
- EC(BEA,
c=
(6)
NiLi
v 2
E2V² + EL (((+) EL(OEA,
Due to Lemma and the proof of Lemma Li et al. 2019), inequality (1) holds as 0 E0.
Ejn P. inequality (2) holds because E EX x < E 2| where X 0 E. inequality (3) due to Jensen inequality; inequality (4) follows from that fact that the learning rate Te is non-increasing; inequality (5) holds due to Assumption 3. Let consider the term F. x)+ F.. Xt);; note that the model parameters FF are unchanged and thus the first term in the loss function 20 can be neglected. The difference between the two loss functions is
21

---

Published as a conference paper at ICLR 2023
due to different global hyper-knowleede K' and K L( ,x++) - C( (0R,A,K' K(')
1
(10@ (Gwg a. (Beittt) - Qi+t1 - ( 241111)
n
Bi
((ni(())))) Huuett 11 - Ru (2xx) - Hulr ) (2)
+
7 B:
- Σ
(11@ (G.f Qii + Que -qiet1| ||2 2
- ((o -
71 j=1
(1 (..s( Hmeeet11 - -... - - -pmmeee) 11)
+ B
2 1 1
G (G. Q (C + +1///ttt -
B. B
Huall+1 - -
B:
Σ
LL3 [ He.i1l2 + - +
1
[| -H (12)
B
k=1
(38)
where (1) is due to the triangle inequality, ||a + b cllz 1 |alll + libll2 + l|e|l2 with a = Q (Gsm: Q b = Q ( (G a. (Hthtt1) Q (. and c = Q Quet1;
inequality (2) holds due to Assumption 1. Then, let us consider the following difference:
1
m
1700011 -15
h.h
Pih
m
h
h:
Pi
2
N
Pi
R Ro. n.t (x)))
R 1 (I0
N
k=1
(39)
N!
(1)
2
P
R
E. (xx) 60
R Rog.. (30x)
(2)
N
2
P4
To2
Ex
E,t-1
L2
Ps
E
1=1
Inequality (1) holds due to Jensen's inequality, while inequality (2) follows from Assumption
22

---

Published as a conference paper at ICLR 2023
For convenience (and perhaps clarity), we drop the superscript j denoting the class. Taking expec- tation of both sides,
E
7
Hll12
L2
Prl
B,t
E
(1)
1,2
Pi
R
Φ
R.t
+ IR
1,, F.t-1
1=1
(2)
L
Pi
1/0 FVV.V + E
Σ
Pj F
Φ3 F
1=1
1,2
100 EV E
Pjc Fx
2
+ Φ
1-1
Φ. Fit-1
(3)
< L2
Pt
1/0 BV +
E
E
Φ5 1
+ $
0 E
(4)
m
B.t-1
L2
Pi
TVEV +
E-1
L2
EV
E
Σ
ncgi
e-t-1
1=1
(5)
S L2
Pt (mEV + EVEV
1=1
LV,
(40)
where (1) follows from the triangle inequality; inequality (2) holds due to Assumption and the update rule of SGD; since f(x V is concave, (3) follows from Jensen's inequality; inequality
(4) holds due to the fact that E |EX xI2 < E ||X]| where X D $ 2 inequality
(5) follows by using the fact that the learning rate r)e is non-increasing.
Similarly,
E E
-0'122 I L3 Σ pil u
w. E
(41)
21j0 T.g F.V
Combining the above inequalities, we have
E c2.1+++1]] < c
L 2s (L2 + +1) EV + LyMoLEEV.
E²V²
(42)
2
A.6.3 THEOREMS
Theorem 2. Instate Assumptions 1-3. For an arbitrary client, after each communication round the loss function is bounded as
t+1
E Σ
L
EL (FV2 + +02)
E
<
C
2
2
(43)
2
+ 2AmoL3 (L2 + EV + 2YTOLZEV.
23

---

Published as a conference paper at ICLR 2023
Fine-tuning the learning rates no. À and y ensures that
MiLiB (EV2 +0²) + 2dmoLs (L2 + 1) EV + 2y
F-1
ne 2
| 1 < 0.
Tle
(44)
Corollary 1. (FedHKD convergence) Let rjo > T7e atjo for e E {l...., E -1,00 1. The loss function of an arbitrary client monotonously decreases in each communication round if
2a | (L2 + 1)V L07 Ve Ell...,E-11 1}
0770 < Tle
(45)
L1 a²
|C + (EV* +22)
where a denotes the hyper-parameter controlling learning rate decay.
Proof:
Since To < lte in each local epoch e we have a.
+ Tee (EV2 + 0²) + 22 Te La +
(no
netn
27
Te L2V
||
<0. (46)
20
a
a
1/e
2
Dividing both sides by ns.
Ie
(EV2 o²) 22 - (L2 +
27 27 L2V - -1--1oll vel11
| <0. (47)
2a
a
a
Factoring out Te on the left hand side yields
In (EV2 + +0²) +
In
|V2 ne < V 2X a
Lo (L2 + 1) 27 L2V. (48)
2a²
2
a
Dividing both sides by
(EV2 + 0²) + b /V 113) results in
2a
20²
L - 4. + 1)V 40r42V Ve E F1....,,--11
Tee
(49)
L1
a²
|V + 2
Theorem 3. (FedHKD convergence rate) Instate Assumptions 1-3 and define regret A = = - L. If the learning rate is set to 1). for an arbitrary client after
2A
(50)
cF (2n- T)) - T E (EV2 + 0²) 42n/ts (12 + 1) EV - 4y
global rounds (e 0), holds that
VL 12 Se.
(51)
TE
Proof:
According to Theorem 1,
T E-1
(1-
T
T
1+1]
+
L
(EV2 + 0²)
TF ==
TF.
TE
2
1=1
[=1
+ 2A (L2 + 1) V + 2-7L2V
A + 1241 (EV2+oo) + 2AnL3 (L2 + 1) V + 2mL2V
< TE
2
17² 1
< 1)
2
(52)
Therefore,
7" L 1 2
r² E
- S
(EV2 +02) 22//2 EV - 2MLLEV,
(53)
2
which is equivalent
24
T2 E (2n) 11) 17 1, E (EV2 + 02) - 4d (12 + EV 4y
(54)
24


Reviews:
[{"review_id": "JiPaUUy4stW", "reviewer": "Reviewer_GssC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors propose FedHKD (Federated Hyper-Knowledge Distillation), a novel FL framework\nthat relies on prototype learning and knowledge distillation to facilitate training on heterogeneous\ndata. The clients in FedHKD compute mean representations and the corresponding mean soft predictions \n(they are called \"hyper-knowledge\" in the paper) for the data classes in their local training sets. And then,\nthe hyper-knowledge. is endued by differential privacy via the Gaussian mechanism and sent for aggregation \nto the server. The resulting globally aggregated hyper-knowledge is used by clients in the subsequent training \nepoch and helps lead to better personalized and global performance.\nThe main contributions of this paper is: \n1. The authors propose a framework called FedHKD, which trains both the global model and the local models well\nin federated settings.\n2. provide a detailed privacy analysis and a convergence analysis.\n3. make extensive experiments to prove FedHKD works.\n\n", "summary_of_the_review": "This paper is good in general. It would be better if the author explain the origin of the loss function.\n\n", "strength_and_weaknesses": "Strength:\n1. The authors do a lot of experiments to prove that their method is good and compare with many existing methods.\n2. The related works are sufficient.\n3. The authors provide rigorous theoretical analysis, including privacy analysis and conversion analysis.\n\nWeaknesses:\n\n1. The proposed framework is similar to the paper Fair and Robust Federated Learning Through Personalization. \nThey all combine data replication to calculate the global model and local model. In the reviewer's opinion, \nthere is a slight lack of novelty.\n2. It will be clearer if there is a structure flow diagram to explain how the proposed scheme works.\n3. The author does not explain why the loss function is constructed as in the paper. In other words, \nwhy can't the regularizer be in other forms.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HC5dtVEWlJa", "reviewer": "Reviewer_yjPN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a federated learning technique to produce good local models and a good global model.  They do so by designing a system that trains a global model without communicating client gradients to the server.  Their system extends recent FL work, which trains local models with client mean class prototypes (FedProto), by also sending client mean (soft) predictions and training a global model.  They call this \"Hyper Knowledge.\" Thus FedHKD is a hybrid approach:  like pure prototype or knowledge distillation, clients avoid sending gradients or parameters, but clients still receive weights from a global model to update their local models.   They evaluate the approach against a number of competing FL systems across three data sets and different levels of non-iid client training data distributions.  Their approach produces models that often do as well and sometimes better than other systems. ", "summary_of_the_review": "The paper presents a useful design point in the very large space of FL systems.  However the paper doesn't make the system objective clear or provide anecdotal or empirical evidence for the target operating regime.  While it borrows from personalized FL and KD FL systems, it doesn't provide for model architecture heterogeneity.   The initial results appear promising but not necessarily dominating over competing systems in the experiments shown.    \n\n[Post Author Response]\nAfter reading through the discussion, the author's response has clarified issues (e.g., pointing out the significance of the DiffP piece), and though I remain on the fence about this work, I'm comfortable adjusting the score to a 6 (marginally above).   There is no doubt the work makes some contributions in the FL space that builds personalized and global models.   FedHKD builds mostly better local and global models when comparing to a scheme that is also data-free and admits different client network architectures.   But if one says the performance results are solid but not dominating over another data-free scheme, the claim that being data-free is a significant contribution seems to be weakened. ", "strength_and_weaknesses": "Strengths\n* Presents an interesting point in the design space between recent work in prototype and KD-based FL systems\n* Evaluated on different a handful of different models and datasets, while exploring different client data distributions\n* Provides theoretical convergence analysis of the FL system \n\nWeaknesses\n* The paper borrows from prior work (which is fine), but it doesn't clearly articulate how it's the \"best of both worlds.\" \n* There isn't a clear set of objectives for the work.  Is it to build good personalized or local models?  Is it to build the best global model in the presence of non-iid data?  What is the real-world scenario where the problem your addressing rises?  \n* The evaluation lacks clear lessons learned.  The system is competitive but not dominating.  \n* It is not clear what the benefits are of the resulting global model vs. the local models.  What about testing a version of FLHKD where the global model isn't sent to the clients?  What was the penalty in terms of bandwidth relative to FedProto? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BxfxWo8CwBC", "reviewer": "Reviewer_HZm8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an extension to the idea of federated learning which works in the situation of heterogeneous data. This is achieved without the need for public data. The approach provides security through the use of differential privacy for the exchanged data.", "summary_of_the_review": "The paper reads well, but a lack of details on what is going on makes it hard to follow in places. Moving some of the theorems to supplementary material would give more space for further explanation.", "strength_and_weaknesses": "Strengths:\n1. The paper is technically well presented.\n2. The idea appears novel and sound.\n\nWeaknesses:\n1. A number of the theorems in the main body of the paper do not appear to be used within the main body. It would seem to make more sense to put these into the supplementary material so that more space can be used for better presentation of the main ideas.\n2. A number of equations are presented without clear explanation of what they mean.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2uxRS7V9fV", "reviewer": "Reviewer_y1Yi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of global and personalized federated learning, aiming to improve both at the same time. This paper proposes a method, FedHKD, that leverages hyper-knowledge distillation to improve both global and local learning. Specifically, hyper-knowledge is defined as the class-wise mean representations and mean logit predictions. The server aggregates meta-knowledge from all clients, and sends the aggregated hyper-knowledge back for clients to learn. Compared with other works which use knowledge distillation to improve FL, FedHKD ahas the advantage of not requiring public datasets, and does not require training generative models. The authors present theoretical analysis on the convergence of FedHKD. Empirically, FedHKD outperforms state-of-the-art federated learning methods in both local and global accuracy. ", "summary_of_the_review": "My main concerns of this paper are 1) missing a related work, and 2) experimental results require more justification. I recommend a weak reject at this stage. I am open to recommending accept if the authors can clarify my doubts. ", "strength_and_weaknesses": "Strengths: \n- The proposed FedHKD seems efficient and general. FedHKD does not require local datasets, and does not require training generative models, which makes it more applicable in practice. \n- The paper is clearly organized and easy to follow. The main advantages and motivations of FedHKD are clearly stated. The method is clearly described. \n\nWeaknesses and Questions. \n- Missing an important related work. I appreciate the authors' attempts to bridge local and global federated learning. However, as the authors study class distribution heterogeneity only, I think FedROD (Chen and Chao, 2022) should be discussed and compared, as FedROD also aims to bridge local and global federated learning under class distribution heterogeneity. \n- Experiment results do not comply with existing works. For example, with ResNet18 on CIFAR-10, with Dirichlet label distribution $\\beta=0.5$ and 20 clients, FedHKD achieve global accuracy of 0.5735. However, as reported in Fed-ROD, with a simple CNN and $\\beta=0.3$ (larger distribution heterogeneity), 20 clients, Fed-ROD achieves 0.768 (even FedAvg achieves 0.686). These numbers differ significantly, and the authors should justify the difference. \n- \"Local Acc\" is not clearly described. First, the evaluated model is not clearly stated (I assume that the models evaluated are the local models before aggregation). Second, it has been studied that FedAvg+local fine-tuning (Cheng et al. 2021) is a powerful baseline in personalized FL. I think is is more appropriate and fair to evaluate local accuracy after some local fine-tuning. \n- DP on hyper-knowledge seems not necessary. As aggregating the hyper-knowledge requires only addition, is it possible to leverage secure aggregation (Bonawitz et al. 2017) instead of DP, which adds no noise? \n- Minor points: It is suggested to replace \\cite with \\citep in some places, such as Section 4.1. Also, in conclusion, there is a 'FedHDK' which should be a typo. \n\nH. Chen and W. Chao, ON BRIDGING GENERIC AND PERSONALIZED FEDERATED LEARNING FOR IMAGE CLASSIFICATION. ICLR 2022\n\nK. Bonawitz et al. Practical Secure Aggregation for Privacy-Preserving Machine Learning, CCS 2017\n\nGary Cheng et al. Federated Asymptotics: a model to compare federated learning algorithms. https://arxiv.org/abs/2108.07313\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster