pFedKT: Personalized Federated Learning via Knowledge Transfer
Authors: Liping Yi
Xiaorong Shi
Gang Wang
xiaoguang Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Personalized Federated Learning
Knowledge Transfer
Local Hypernetwork
Contrastive Learning

Abstract: Federated learning (FL) has been widely studied as a new paradigm to achieve multi-party collaborative modelling on decentralized data with privacy protection. Unfortunately, traditional FL suffers from Non-IID data distribution, where clients' private models after FL are even inferior to models trained standalone. Existing approaches to tackle this challenge fall into two directions: a) pursuing a better global model through mitigating biases of private models, and b) improving personalized private models by personalized federated learning (PFL). Still, both of them have limited accuracy improvements in private models. To this end, \textit{we design pFedKT, a novel personalized federated learning framework with knowledge transfer, towards boosting the performances of personalized private models on Non-IID data}. It involves two types of knowledge transfer: a) transferring \textit{historical private knowledge} to new private models by local hypernetworks; b) transferring \textit{the global model's knowledge} to private models through contrastive learning. After absorbing the historical private knowledge and the latest global knowledge, the personalization and generalization of private models are both enhanced. Besides, we derive pFedKT's generalization and prove its convergence theoretically. Extensive experiments verify that pFedKT presents $0.31\%-3.46\%$ accuracy improvements of private models than the state-of-the-art baseline.

Under review as a conference paper at ICLR 2023
PFEDKT: PERSONALIZED FEDERATED LEARNING VIA KNOWLEDGE TRANSFER
ABSTRACT
Federated learning (FL) has been widely studied a paradigm to achieve multi-party collaborative modelling on decentralized data with privacy protection. Unfortunately, traditional horizontal FL suffers from Non-IID data distribution, where clients' private models after FL are even inferior to models trained stan- dalone. To tackle this challenge, most existing approaches focus on personalized federated learning (PFL) to improve personalized private models but present lim- ited aceuracy improvements. To this end, we design pFedKT, novel personal- ized federated learning framework with private and global knowledge transfer, towards boosting the performances of personalized private models on Non-III) data. It involves two types of knowledge transfer: a) transferring historical pri- vate knowledge to new private models by local hypernetworks; b) transferring the global model's knowledge private models through contrastive learning, Af- ter absorbing the historical private knowledge and the latest global knowledge, the personalization and generalization of private models are both enhanced. Be- sides, we derive pFedKTT's generalization and prove its convergence theoretically. Extensive experiments verify that PFedKTT presents 1.38% 1.62% accuracy im- provements of private models compared with the state-of-the baseline.
INTRODUCTION
With frequent privacy leakage, directly collecting data and modelling it would violate privacy pro- tection regulations such as GDPR (Kairouz& & et al. (021). To implement collaborative modelling while protecting data privacy, horizontal federated learning (FL) came into being (MeMahan & all 2017]. As shown in Fig. (a), FL consists of a central server and multiple clients. In each communi- cation round, the server broadcasts the global model (abbr. GM) to selected clients; then clients train it locally on their local datasets and upload trained private models (abbr. PMs) to the server; finally, the server aggregates received private models to update the global model. The whole procedure is repeated until the global model converges. In short, FL fulfils collaborative modelling by allowing clients to only communicate model updates with the server, while data is always stored locally.
However, FL still faces several challenges such as communication efficiency, robustness to attacks, and model accuracy which we focus on in this work. The motivation for clients to participate in FL is to improve their local models' quality. However, the decentralized data held by clients often not independent and identically distributed (Non-IID) (Kairouz & et al. 2021), and the global model aggregated through typical FL. algorithm FedAvg (McMahan & IID data may perform worse than clients' solely trained models. IZhao & et all (2018) have verified this fact experimentally and argued that the global model aggregated by skewed local models trained
2017 based Non-
on Non- IID data deviates from the optima (model trained on all local data). To alleviate the accuracy degradation caused by Non-IID data, personalized FL (PFL) methods (Shamsian been widely studied to improve clients' personalized model quality.
2021 have
Existing researches implement PFL by fine-tuning Mansour & all (2020): Wang & al (019)) model mixup Arivazhagan & et all (2019); Collins & et al (2021) and etc. But they suffer from limited improvements in the accuracy of private models.
To further improve personalized private models on Non-III data, we propose novel personalized FL framework named pFedKT with two types of transferred knowledge: 1) private knowledge: we deploy a local hypernewoork for each client to transfer historical PMs' knowledge to new PMs; 2) global knowledge; we exploit contrastive learning to enable PMs to absorb the GM's knowledge. We analyzed PFedKT's generalization and proved its convergence theoretically. We also conducted extensive experiments to verify that PFccKT fulfils the state-of-the-ar PM's accuracy.
a) We devised two types of knowledge transfer to simultaneously enhance the generalization and personalization of private mod- els. b) We analyzed PFedKT's generalization and convergence in theory. e) Extensive experiments verified the superiority of TTedKT on the accuracy of personalized private models.
Contributions. Our main contributions are summarized follows:

---

Under review as a conference paper at ICLR 2023
Test Accuracy
100
10-3
Server
GM
Apgregate
21-4
0.6
111
04
18:32
Clienty
GM
28423
18:22
0.0
Rounes
(a) FedAvg
(b) GM v.s. PM
Figure 1: (a): Workflow of FedAvg in the t-th round. (b): The test accuracy of the GM and 20 PMs are recorded per 10 rounds. Since the server has no data, we evaluate the test accuracy of the GM on clients' test datasets and calculate mean test accuracy as the GM's accuracy. We evaluate the test accuracy of a local model PM after local training on its local test data as the PM's accuracy.
RELATED WORK
Recent personalized federated learning (PFL.) approaches include: a) Fine-tuning, in FL's last round, clients fine-tune the received GM on local data get PMs (Wang & et al 2019 Man- sour & et al (0220). b) Federated meta-learning, some methods apply meta-learning in FL, such as MAML-based distributed variants (Li & et all 2017; Fallah & al 2020b a). e) Federated multi-task learning. it treats each client as a learning task, e.g., MOCHA (Smith & et al 2017), FedU (Dinh et al| 2021) d) Model mixup, the PM's parameters are split into two parts, only one part is shared through the server and another is trained locally, as in FedPer (Arivazhagan & 2019), FedFu Yaao et al. 2019), FLLDA (Peterson et al., 2019), LG-FEDAVG (Liang & et all MAPPER (Mansour al. 2020 FedRep (Collins & al 2021 D. PFedGP (Achituve & al
al
2020
(Sun & et al 2021). e) Aggregation Delay, RADFed Xue et al. (2021) proposed redistri bution rounds that delay aggregation alleviate the negative impacts model performance due to Non-IID data. f) Federated clustering, the server clusters PMs with similar parameter distributions and performs aggregation within clusters, e.g., HYPCLUSTER (Mansour & all 20200) ClusterFL (Ouyang & et all 2021). I, CFL (Agrawal & et al 2021). g) Local aggregation, instead of aggrega- tion within the server's clustered groups, FedFOMO Zhang & al 2021) makes each client pull other clients' PMs and selects more beneficial ones for local aggregation to update its own PMs. h) Knowledge distillation-basett FedPHP (Li & et al.] (02TT) linearly accumulates historical PMs and new trained PMs to teach the received GM through knowledge distillation in each round of FL. FML (Shen al. 2020) makes each client's PM interact with the GM through mutual learn- ing. KT-pFL (Zhang et al. (022)) allocates public dataset to each client, and only logits computed on the public dataset are shared through the server. i) Contrastive learning- based, MOON LL let al. 202T) utilizes contrastive learning to make PMs close to the GM, towards obtaining better GM. i) Hypernetwork- based, pFedHN [Shamsian & et all 202T deploys a global hypernetwork on the server learn PMs' parameter distributions and generate personalized parameters for PMs. The latest work Fed-RoD (Chen & Chao 2022 trains private personalized headers with parameters generated by local hypernetworks. It improves both the GM and PMs, but extra communication cost incurs by communicating hypernetworss.
2021
3 PRELIMINARIES AND MOTIVATION
3.1 UTILITY OF PRIVATE MODELS
As the workflow of FedAvg shown in Fig I (a), we abbreviate the private model as PM and the global model as GM, and the detailed definition of FL is introduced in Appendix [A] It's worth noting that: in FedAvg, 1) clients no longer store PMs after uploading them to the server; 2) in the next round, clients regard the received GM as PM and then train PM on local datasets, i.e., the trained PMs only play as "temporary models" for aggregation and their utilities not sufficiently developed.
To explore the utilities of PMs, we train a CNN model on a natural Non-IID FEMINIST dataset in an FL system with 20 clients. From Fig I (b), observe that there are always some PMs performing better than GM in each round (some PMs show lighter pixels than GM), so we can further develop PMs' self-utility during FL to boost the accuracy of personalized private models.
2

---

Under review as a conference paper at ICLR 2023
Hypernetwork
CIFAR-10
CIFAR-10
1.0
3.0
NK
2.5
NK with 11N
0.
100 205
2.0
Target Model
0.5
MK without
MN with HN
32-32d3
16-34-16 16424-2
0.0
100
200
100
Rounds
Rounds
(b)
(C)
Figure 2: (a): A hypernetwork generates parameters for a target model. (b): the test accuracy of two target models (NN with/out HN) vary as communication rounds on CIFAR-10 dataset.
3.2 MOTIVATION
3.2.1
PRIVATE KNOWLEDGE ENHANCEMENT
To fully utilize PMs' self-utility, FedPHP linearly accumulates historical PMs and new trained PMs to teach the received GM through knowledge distillation in each round of FL. Since historical private models with obsolete parameters compromise convergence, linearly stacking them with manually selected weights may suffer from degraded accuracy, which has been verified in our subsequent experiments (Sec. 6.2). A hypernetwork (abbr. HN) (Ha & et al (017)) is generally a small model that generates parameters for large target models, belonging to the category of generative networks in unsupervised learning. pFedHN and Fed-RoD exploit hypernetworks to personalize PMs, but they have few weaknesses. In PFedHN, the footprint of the server's 's hypernetork is designed to be larger than that of PMs, especially, the hypernetwork's output layer's parameter capacity is equal to that of a complete PM, which burdens the storage cost and computation overhead of the server. Fed-RoD requires to communicate local hypernetwrrss (which generate parameters for personalized headers) between clients and the server. introducing extra communication cost which is the main bottleneck of FL Kairouz & al. 2021
To enhance personalization while avoiding the above problems, our pFedKT attempts to allocate a local hypernetwork for each client to learn its PM's historical private knowledge (parameter dis- tributions) and then generate parameters for its complete private model. In other words, we use the local hypernetwork accumulate historical PM's knowledge to develop PMs' self-utility. The clients' local hypernetworks used in our PFedKT are simple fully connected networks with lower parameter capacity than PMs, so it has lower computation complexity than the server's 's large hyper- network in PFedHN. Besides, the server and clients still communicate PMs in pFedKT, which has communication cost the same as FedAvg and lower than Fed-RoD.
To verify the feasibility of the above insight, we conduct preliminary experiments Specifically, we train a randomly initialized hypernetwork (abbr. HN) and a target model (abbr. NN, i.c., PM) with parameters generated by the hypernetwork in an end-to-end form the CIFAR-10 dataset. How to use hypemeeworks to generate parameters for target models and how to update hypernetwork are detailed in Appendix B and C Fig. 2 displays the structures of the target model and hypernetwork, as well as the experimental results. We observe that the final test accuracies of the solely trained target model (NN without HN) and the target model trained with the hypernetwork (NN with HN) are 01.84 and 93.84%, respectively, i.e., the latter performs better, indicating that regarding HN as meta-model to continually learn PM's historical knowledge boosts PM's s accuracy. Therefore, we can safely utilize the local hypernetwork to accumulate PM's 'ss private knowledge.
single client.
3.2.2 GLOBAL KNOWLEDGE ENHANCEMENT
To train a better GM on Non-IID data, MOON Li et al. 2021 the current round's 's PM (anchor) close to the received GM (positive) and away from the last round's PM (negative). Since it initializes the current round's s PM (anchor) with GM's parameters, the PM's personalization is impaired. Nevertheless, it still Inspires us transfer global knowledge to PMs.
utilizes contrastive learning to keep
Unlike MOON, our PFedKT regards the PM generated by a hypernetwrk which has absorbed historical private knowledge the anchor in contrastive learning. And it also keeps the generated PM close to GM for acquiring global knowledge. This discrepancy facilitates the effective fusion of the latest global knowledge and historical private knowledge, promoting PM's personalization and generalization in available classes of local datasets.
3

---

Under review as a conference paper at ICLR 2023
HN: Hypernetwork diint in (t)-th round
Server
GM
@ Aggregaae
PMj
PM Private Model generated IIN in (t) rund
GM Global model aggregated in (t-1) round
PM Private Model after truining in (I- 1)-th round
PM Private Model training in (t) round
Clienty
GM
HNN Feokeciinn
GM
HN
PM k
Contrastive losss
PM
train
PM
P
Super Find loss
Riste stame
Figure 3: Left: workflow of PFedKT. Right: KL divergence between two models is calculated on logits R (Dx); and the local hypernetwork used in PFedKT is simple fully connected (FC) network with much smaller footprint than that of PM.
4 METHODOLOGY
In this section, we first outline pFedKT's workflow, and then detail the two types of knowledge transfer: a) private knowledge transfer by local hypernetworks and b) global knowledge transfer by contrastive learning. Finally, we analyze pFedKT's computational budgets and storage costs.
4.1 OVERVIEW
Principle. Building on the above motivations, we devise novel personalized FL framework named PFedKT, which involves two types of knowledge transfer: 1) Transferring knowledge from old PM to new PM. We configure a local hypernetwork (with much smaller footprint than that of PM) for each client to learn the old PM's knowledge and transfer it to the new PM. 2) Transferring knowledge from GM to PM. We exploit contrastive learning to keep the new PM (have carried old private knowledge) close GM. In this way, during each round of local training, the private knowledge from the old PM, the global knowledge from the latest GM, and absolutely the knowl- edge from local data are simultaneously incorporated into the trained new PM, which facilitates personalization and generalization in available classes.
Workflow. Specifically, displayed in Fig. 3 the complete workflow of PFedKT framework in- cludes the following steps: in the 1-th round, a) the server first broadcasts the global model GM to selected clients. 2) The k-th client uses its local hypernetwork HN, (with the embedding of pre-divided chunk id input) to generate parameters for the target private model P PM (to be trained) in a stacked manner, where we detailed the generation procedure in Appendix B 3) Then, regard the generated private model PM the anchor, the received global model GM positive item, the last round's 's trained private model PM as negative item. The k-th client computes the distances d and dæ measured by KL (Kullback-Leibler divergence Kullback & Leibler (1951) from PM to GM and PM respectively. After that, the k-th client computes the contrastive loss with the two distances. 4) The k-tth client trains PM with contrastive loss and supervised loss from labels on local dataset Dr. After training, the trained private model is marked as PM k. 5) The k-th client updates the old private model PM --- with the latest trained private model PM g. 6) The k-th client updates the hypernetwork HN & to HN with the parameter vari- ations between generated PM and trained PM where we detailed the hypernetwork' 's updating procedure in Appendix C 7) The k-th client uploads the trained private model PM to the server. 8) The server aggregates the received private models [PM ...] through the weighted aggregation rule of FedAvg, and updates the global model to be GM+++1 The above steps are executed iteratively until PMs converge. In the end, acquire personalized PMs. A detailed algorithm description of pFedKT is given in Appendix D. Alg.
&.
4.2 PRIVATE KNOWLEDGE TRANSFER VIA LOCAL HYPERNETWORKS
Motivated by the availability of hypemneworrs validated in Sec. 3.2.] we view hypernetworks as "information carriers" " to continuously transfer old private knowledge of previous PMs to new PMs generated by hypernetoorks. In particular, deploy a local hypernetwork for each client. Utilizing hypernetworks to achieve private knowledge transfer involves two directions: a) knowledge release (forward) and b) knowledge absorption (backward).
4

---

Under review as a conference paper at ICLR 2023
Knowledge release (forward). As step ② in Fig. 3 first use the hypernetwork so to generate parameters for the new private model f.g. in which hypernetworss release old knowledge of the last round's trained private models - to the new generated model h
Then, the generated private model of will be trained local dataset Dx- and the trained private model is denoted as w which will be uploaded to the server for aggregation.
Knowledge absorption (backward). Instead of abandoning trained private models after uploading them to the server as in FedAvg, we use local hypernetwrrss to absorb the knowledge (parameter distribution) of trained private models. This procedure implemented by updating hypernetworks' parameters with parameter variations between generated new PM and trained new PM, as step ) in Fig. 3 Specifically, according to the rule of updating hypernetworks Eq. (T0) of Appendix [C we utilize the difference between the generated new PM 0% and trained new PM w to update the local hypernetwork 4 i.e.,
f- p - "JN (veg 0 A wf.)-
Pk
where N the hypernetwork's learning rate. Then the updated hypernetwork vp absorbs the knowledge of the latest trained PM w
Since the two way knowledge transfer executed in each round, hypemetworks continuously learn historical private knowledge and transfer it to the new generated PMs during the whole FL, which promotes the personalization of PMs.
4.3 GL.OBAL. KNOWLEDGE TRANSFER VIA CONTRASTIVE LEARNING
Once the new PM b is generated by the local hypernetwork, old private knowledge has been trans- ferred into the new model. To make 0f further obtain the latest GM's knowledge, we exploit con- trastive learning to bridge GM w' and PM f% Specifically, view 0 as the anchor, and keep it close to GM W (positive) since we hope 0% to learn knowledge from other clients via GM; while keeping &% away from the last round's PM w (negative), so as to avoid being trapped in local optimum and slowing down convergence due to excessively skewing previous PM's obsolete pa- rameters. We use triple! loss (Schroff & et al.) (0155) in typical contrastive learning as pFedKT's contrastive loss con Then, we calculate contrastive loss by:
Cu Re (Dx), fur -Ru (Dk), Lug- (D)):
d dKL(Cww (lb()) d dKI(L (
lcon maxfdkk dKL a, 0).
The distance d of 0f (anchor) and wt (positive), and the distance d of of (anchor) and w/=1 (negative) are measured by the KL divergence of their logits L(w-) (i.e, extracted representation Rw- (Dx)). 2 0, is the maximum margin between the amchor-positvn distance and anchor- negative distance.
0, the generated PM 0 is as far from GM wt from the last round's PM w 1 i.c., "neutral" status. If a > O, then d + a S dxL. i.e., generated PM 0% is close to the GM w and away the last round's PM w and vice versa.
After computing contrastive loss leon, we further calculate the supervised loss Suup (e.g., Cross Entropy loss) with the training model's predictions and labels. Finally, we linearly weight the two types loss to build the complete loss function f, i.e.,
4 Lcon + (1 11) koup:
(3)
where μ E (0, 1) is the weight of contrastive loss. Next, we train the generated PM 0g local data D₁ through gradient descent with the complete loss function J, and obtain trained PM wf, i.e.,
wf k NN ( Da),
where NNN is the private model's learning rate.
With contrastive learning, the trained PM of absorbs the GM's global knowledge (i.e., private knowl- edge from other clients), enhancing the generalization of PMs in available classes of local data.
5

---

Under review as a conference paper at ICLR 2023
4.4 COMPUTATIONAL BUDGET AND STORAGE COST
In this section, we analyze the computational complexity and storage overhead of pFedKT compared with state-of-the-an pFedHN. Limited to pages, the detailed analysis and comparisons with more baselines are presented in Appendix G
After careful comparisons, we summarized as follows:
Computational Complexity. pFedKT consumes comparable computational cost to pFedHN. In cross-silo FL scenarios, multiple enterprises with sufficient computational power can tolerate exe- cuting PFedKT. Besides, pFedKT inherently offloads the serial learning tasks of the hypernetwork on the server in PFedHN to parallel clients' sub-tasks, which reduces computation delay and tackles the blocking issue that possibly occurred on the server in pFedHN.
Storage Overhead. Since the local hypernetwork in pFedKT has a smaller footprint than the hy- pernetwork deployed on the server in pFedHN. Hence, pFedKT shows a lower storage cost than pFedHN from the perspective of one device (client in pFedKT or server in pFedHN).
THEORETICAL ANALYSIS AND PROOF
In this section, we analyze pFedKT"ss generalization and prove its convergence in theory.
5.1 ANALYSIS FOR GENERALIZATION
We refer to the theoretical analysis in Shamsian & et al (202T) and derive similar conclusions in Theorem 5.1 the detailed assumptions and derivations are illustrated in Appendix B
Theorem 5.1 If one client has at least M = O 2/2 (embam + HN size ylog( (ladk)) + log
samples, for hypernetwork ip there least probability that satisfies: [f(()) (p)) Sc
where f is loss function, erabaim and HN size are the input embedding dimension and size (param- eter capacity) of hypernetworks (the hypernetoork is fully connected model with chunk id as input embedding). Lw, Lug are assumed Lipschitz constants, T, E are constants defined in derivation. This Theorem reflects that PFedKT's generalization is impacted by both the hypernetwork's 's input em- bedding dimension and size, Lipschitz constants Ly and Lg which have been verified to marginally affect the hypernetwork's utility in (Shamsian & et al (202T). So, we experimentally verify how the hypernetwork's 's input embedding dimension and size affect pFedKT'ss generalization in Sec. 6.3.
5.2 PROOF FOR CONVERGENCE
Insight. Shamsian & et al (022)) explains that the mapping of hypernetworks to generated target models is essentially similar to the principle of PCA dimension reduction. That is, the hypernetwork can be viewed as the main component (core information) of the target model after reducing dimen- sion. Therefore, target models generated by hypernetworks would have similar convergence rate to pure target models, as shown in preliminaries (Fig. 2 (b) in Sec. 3).
Proof. We refer to the convergence proof in Li et al. 2020 and derive the following Theorem (detailed proof is presented in Appendix|e E
Theorem 5.2 Assuming (WT) is the average loss in the T-th round, f is the minimum loss of during T rounds; n. Y, B, C, p. are defined constants in Li al. [020); ; wa the initial model,
w" is the optimal model with minimum loss. Then we can get: E[f(wr) f T-T B+C 2L
2x
I.po, ~ (1/T)) where L.p and ao are constants defined our extra assumptions.
From Theorem 5.2 we conclude that pFedKT has the same convergence rate O(1/T) with FedAvg.
EXPERIMENTS
We implement PFedKT and all baselines with PyTorch and simulate their FL processes on NVIDIA GeForce RTX 3090 GPUs with 24G memory. We evaluate PFedKT on two image classification datasets: CIFAR-LO/100/ Krizhevsky & et al 2009) with manually Non-IID division and a large
real-world Non-IIID dataset: Stack Overflow? [
The Codes will be made public after acceptance.
https://www.csttormmt wwwp://wwwww.oooo.cccc///.. kriz/cifar.html
http:://www.teeoooffo.oo//cdddee hww.a//ggooooioiioaaaooaaaaaaaan data

---

Under review as a conference paper at ICLR 2023
6.1 SETTINGS
Datasets and Models. Referring to the Non-IID data divisions in Shamsian & et al (2021 1221y Charles & et al. (202T), we manually divide the three datasets into Non-lll> distributions. Specifically, for CIFAR- 10, we assign only 2 classes of data to each client, 50 clients totally; for CIFAR-100, we assign only 10 classes of data to each client, 50 clients totally; for Stack Overflow, we assign only posts from one author to each client, 100 clients totally. In CIFAR-10/100 different clients has different numbers and features. Specifically, if one class requires to be allo- cated into 10 clients, then we use random. fni form(low; high) function (low, high E (0, 1)) to produce 10 ratios of data counts and then generate this class's data subnets with different numbers and features to different clients. In Stack Overflow, each user's posts show naturally diverse in class and features. After dividing Non-IID data into each client, each client's local data is further divided into training set, evaluating set, and testing set with the ratio of 8:1:1, i.c., the testing set is stored locally in each client and shows a consistent distribution with local training set. We train a small CNN model and large CNN model on the CIFAR-I0 and 100 datasets in two image classification tasks, respectively, and train an LSTM model with the same structure as McMahan & et al (2017) on Stack Overflow dataset in word prediction task. We the same hypernetworks in three tasks. The structures of two CNN models and hypernetworks are shown in Appendix F.] Tab. 2
one class allocated to
Baselines. We compare PFedKT with the following algorithms: 1) Local Training, in which each client trains its model locally. 2) FedAvg, a typical FL algorithm. 3) Cluster-based PFL meth- ods: HYPERCLUSTER. 4) PFL methods with model mixup: FedRep, FedPer. LG-FEDAVG, MAPPER, PartialFed. 5) PFL methods with local aggregation: FedFOMO. 6) PFL methods with knowledge distillation: FMI., FedPHP. 7) PFL methods related to our pFedKT: MOON with con- trastive learning which regards GM as the anchor, pFedHN with the server's hypernetwork, and Fed-RoD with personalized headers generated by hypernetworks.
Metrics. We measure the trained private models' mean accuracy and denote it as PM@Ace (%)
Training Strategy. We set grid-searched optimal FL. hyperparameters for all algorithms: the client sampling rate C is 0.1; the learning rate of the local target model (7 is le - 2, using the SGD optimizer with the momentum of 0.9, weight decay of 5e - 5, and batch size of 64, local epochs of {10, 50, 100}, and the hypernetwork's learning rate (THN) is 5e 3; the total communication rounds are at most 500. Our pFedKT's s unique hy hyperparameters are reported in Appendix
Tab. 3
6.2 COMPARISONS WITH BASELINES
Limited to pages, here we only report the comparison results on the CIFAR- dataset, the results on the Stack Overflow dataset are recorded in Tab. 4 of Appendix F2.T Tab. [ records PMs'
mean accuracy of our PFedKT and all baselines, and Fig. 4 displays how the PMs' mean accuracy
varies with rounds and 50 clients' individual PMs' accuracy after convergence.
Results. As shown in Tab. D our PFedKT'ss mean PM accuracy outperforms all baselines. On the CIFAR-10 dataset, PTedKT'ss mean PM's accuracy is 98.12%, which is 1.11% improved than the second-highest PM Ace 97.01% achieved by pFedHN. On the CIFAR- dataset, pFedKT's mean PM's accuracy is 69.38% increased 1. 1.62% than the second-highess PM@ Acc 67.76% achieved by FedPHP. The (00-classification task is a bit more complex than the 1 10-classification task, so it is in- spiring that PTedKT fulfils a larger improvement of PM's accuracy on the 100 dataset. Since pFedHN's s global hypernetwork may not fully learn private models' knowledge in a more complex task while our PFedKT's local hypernetoork can well learn its own generated private model's knowl- edge and then still maintains the highest accuracy. Besides, in Fig. 4 it's obvious to see that: (1) our PFedKT converges to the highest PM's accuracy. (2) Overall, the 50 PMs' individual accuracies after convergence in PFedKT are better (lighter color) than the baselines, which demonstrates the highest personalization degree of PFedKT.
Analysis. The PM@ Acc of PFedHN and Fed-RoD are second only to our PFedKT, which benefits from that: PFedHN uses the server's large hypernetwork to learn clients' private knowledge and transfer it to each client, promoting PM's feature extraction ability: Fed-Rol utilizes local hyper- networks to learn personalized header's knowledge, improving PM's prediction ability. FedPHP marginally improves PM due to linear cumulative knowledge; MOON fails to improve PM may be due to choosing GM as the initial anchor. Profiting from transferring historical private knowledge via local hypernetworks into PMs and transferring global knowledge via contrastive learning into PMs, PM's 's personalization and generalization in available classes are both enhanced in PFedKT, so it achieves the otate-of-the-art personalized PM's accuracy.
7

---

Under review as a conference paper at ICLR 2023
Table The PMs' mean accuracy of PFedKT and compared baselines on CIFAR-IO (Non-IID: 2/10) and CIFAR-100 INon-IID: 10/100) datasets. For fair comparisons, we record the best P.@AAc (%) during 500 rounds for all algorithms. Bold: highest, italic: second-highest.
Dataset
ClrAR-I0
CIFAR-IOU
Local Training
95.11
59.78
FedAvg (McMahun & et al
2017
96.42
65.73
HYPCLUSTER (Mansour ci 2020
95.75
63.30
FedRep (Collins & al 2021
96.74
66.64
FedPer (Anvazlaaaan & cl al
2019
96.61
66.28
LG-FEDAVG (Liang cl 2020
92.60
58.51
MAPPER (Munsour
2020
95.68
61.91
PartialFed (Sun & cl al
20
95.48
61.62
FodFOMO ZZiaa & 2021
94.19
53,4
FML (hhen
83.18
47.82
HedPHPP 11.
2021
96.60
67.76
MOON el.
89.21
48.64
pRedHN (Shamiiian ét 2021
97.01
67.41
Fod-RoD (Chen Chao 2042)
96.61
67.50
pFedKT (Ours)
98.12
69.38
CIFAR-10 (Non-ID: 2/10
CIFAR-100 (Non DD: 10/100)
MOOK
04
0.85
reenco
0.5
pheckt
NOOX
00-00000000
Ted
0.70
0.75
MOON
0.4
Fed RAD
pFodHk
plcdKT
Taxl Accurecy of P% CoAA-100dtenii
Deti
0.65
Fed
p1edh11
0.60
pladKT
0.2
200
400
503
200
300
400
62
Communicat Rounds
Communication Rounds
08-28
Figure 4: CIFAR-10/100 datasets, left-two: the smoothed PMs' mean accuracy varies with communication rounds; right-two: 50 clients' individual PM's accuracy after convergence.
Non-IID and Client Participating Rate. In addition, we also verified that pFedKT presents su- periority in highly Non-IID degrees and shows robustness to diverse client participation rates. The detailed experimental settings, results and analysis are reported in Appendix F.2.2
6.3 CASE STUDY
In this section, we study the affects of several cases on PFedKT, which include: the hypernetwork's input embedding dimension and size, the weight of contrastive loss, the margin of triplet loss, and diverse loss functions. All case studies are executed within 500 rounds on CIFAR- 10 (Non-IID: 2/10) and CIFAR-100 100 (Non-IID) 10/100) datasets. For stable comparisons, record the average of P Acc (%) within the last 30 rounds.
6.3.1 INPUT EMBEDDING DIMENSION AND SIZE OF HYPERNETWORK
A. Input embedding dimension. Sec. 5. analyzed that the hypemetwork's 's input embedding di- mension may affect PFedKT's generalization,
experimentally explore its actual influence. Like Shamsian & et all 2021) computing the embedding dimension by [1 + N/3] (N the to- 50), we select {1, 3, 4, 10). i.e., embbim E (51, 26, 17, 13, 6). Fig. 7 of Appendix F3 displays that the accuracy of PM varies with different embedding dimen- sions, and the detailed values are recorded in Tab. of Appendix F3 From Fig. we can see that: on the CIFAR- 10 dataset, the embedding dimension has random effects on pFedk I: on the CIFAR- 100 dataset, the PM's accuracy rises as the embedding dimension increases. However, the larger input embedding dimension has higher computation complexity, which slows down local training. In Appendix F3 Tab. [.] when embedding dimensions are and on CIFAR-10/100 datasets respectively, KT obtains the best PM.
tal number of clients,
B. Size. Sec. 5. also mentioned that PFedKT's generalization may be affected by the hypernet-
work's size, so vary the number of hypernetworks' hidden layers E {1, 2, 3, 4, 5J. and results are reported in Fig. and Tab. 7 of Appendix H From Fig. 7 we find that the accuracy of PM drops as hypernetwork' size increases, which is probably due to that larger hypernetworks require more parameters to be trained, degrading their generalization. In Tab. 7 pFedKT performs the best PM when the number of the hypernetwork's hidden layers is 1, on CIFAR-10, 100 datasets.
8

---

Under review as a conference paper at ICLR 2023
6.3.2 HYPERPARAMETERS IN CONTRASTIVE LEARNING Next, we explore how the following key parameters of contrastive learning affect pFedKT: (1) μ., which controls the weight of contrastive loss; (2) a, the margin of triplet loss; and (3) different combinations of loss functions and distance measurements.
A. Weight of contrastive loss. We vary μ E {0, 0.0001, 0.001, 0.01, 0.1, 0.9), and the results are reported in Fig. and Tab. X of Appendix + Tab. g shows that pFedKTT with f 0.001, 0.0001 obtain the best PM on CIFAR-10/100, Fig. 7 presents that when fa > 0.1, PM's accuracy degrades obviously as ye rises, which may because that: larger weight of contrastive loss and smaller weight of supervised loss lead to PM's insufficient training, since PM accesses less supervised information from labels. Hence, the weight of contrastive loss should be set smaller than that of supervised loss.
B. Margin of triplet loss. We vary a E 0.1, 1, 5, 10, 20, 30, 40, 50), and the results are reported in Fig. 7 and Tab. 9 of Appendix F.3 Tab. 9 shows that pFedKT with a 0.1, 5 achieve optimal PM on CIFAR- 10, 100 datasets, respectively. Fig. 7 displays that: when a 2 30, PM's accuracy drops obviously since larger a leads initialized PM to be overly biased to the immature GM, directly compromising PM's personalization. Therefore, setting an appropriate a is necessary to balance the private knowledge and global knowledge transferred to PM.
C. Loss functions. We also explore how diverse combinations of distance measurements and loss functions affect pFedKT, and the detailed experimental settings, results and analysis are given in Appendix F3 From the results in Appendix F.3 Tab. TO, we conclude that the combination of triplet loss and KL divergence we designed in p PFedKT performs better model accuracy than others.
6.4 ABLATION STUDY
PFedKT involves two important parts: (1) transferring private knowledge via hypernetwork (HN) and (2) transferring global knowledge via contrastive learning. To verify the effectiveness of each part, we conduct ablation experiments. We explore the following four cases: (A) Without part- (1,2), then PFedK'T degenerates into FedAvg. (B) Only executing part-(1), i.e., only repeating the local training steps: generating PM by HN + training PM- + updating HN's parameters. C) Only executing part-(2), viewing the PM initialized with GM as the anchor, GM the positive item, and the last round's s PM as the negative item, then this case degrades to MOON, but we still use triplet loss with KL divergence to compute contrastive loss. D) Executing both the two parts, i.e., PFedKT.
As shown in Appendix F.3. Tab. I1 case-(A) (i.e., FedAvg) has the lowest PM accuracy. Com- pared case-(B) with case-(A), the PM's accuracy obviously improved, indicating that using hy- pernetworks to transfer private knowledge is reasonable. Compared case-(C) with case-(A), there are marginal improvements in PM's accuracy, reflecting that MOON takes limited improvements on model accuracy. Compared case-(D) with case-(B), pFedKT with contrastive loss takes slight accuracy improvement than without it. Fig. g in Appendix showed that pFedKT with contrastive loss has more stable convergence than w/o contrastive loss CIFAR-10, 10, and it converges faster to the highest accuracy within 300 rounds while PFedKT w/o contrastive loss requires 500 rounds for convergence on CIFAR- The above results indicate that the global knowledge transferred by contrastive loss enhances PM's generalization in available classes with lower computation cost. Case-(D) (i.e.. PT)dKT) achieved the highest PM's accuracy, demonstrating that the two-type knowl- edge transfers are necessary to enhance PM's personalization.
Summary. Overall, PFedKT fulfils the state-of-the-art personalized PM's performances. Besides, both the hypernetwork's input embedding dimension and size influence pFedKT's generalization. And it's necessary to select the proper weight of contrastive loss, the margin of triplet loss, and the combination of the contrastive loss function and distance measurement. Finally, ablation experi- ments verified the feasibility and effectiveness of both two types of knowledge transfer in PFedKT.
7 CONCLUDING REMARKS
In this paper, we proposed a novel personalized FL framework named PFedKT to boost personal- ized PMs. It consists of two types of knowledge transfer: a) transferring historical private knowledge to PMs by local hypemnetorrs, and b) transferring global knowledge to PMs through contrastive learning. The two-type knowledge transfer enables PMs to acquire both historical private knowledge and the latest global knowledge, which promotes PM's personalization and generalization in avail- able classes simultaneously. Besides, we theoretically and experimentally verified that TFedKT's generalization is related to hypernetworks' input embedding dimension and size and also proved its convergence. Extensive experiments demonstrated that PFedKT achieves the state-of-the the-hh--tt per- sonalized PM's accuracy. In practice, PFedKT can be broadly applied to cross-silo FL scenarios.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Idan Achituve and et al. Personalized federated learning with gaussian processes. abs/2106.15482 2021.
CoRR,
Shaashwat Agrawal and et Genetic CFL: optimization of hyper-parameters in clustered federated learning. CoRR, abs/2107.07233, 2021.
Manoj Ghuhan Arivazhagan and al. Federated learning with personalization layers. CoRR,
abs/1912.008118 2019.
Jonathan Baxter. A model of inductive bias learning. J. Artif. Intell. Res., 12:149-198, 2000.
Zachary Charles and et al. On large-cohort training for federated learning, In Proc. NeurIPS, 2021.
Hong- Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning. In Proc. ICLR. OpenReview.net, 2022.
Shuo Chen, Gang Niu, Chen Gong, Jun Li, Jian Yang, and Masashi Sugiyama. Large-margin con- trastive learning with distance polarization regularizer. In Proc. ICML, volume 139, pp. 1673- 1683. PMLR. 2021.
Liam Collins and al. Exploiting shared representations for personalized federated learning. In Proc. ICML, Virtual Event, volume 139, pp. 2089-2099. PMLR, 2021.
Canh T. Dinh and et al. Fedu: A unified framework for federated multi-task learning with laplacian regularization. CoRR, abs/2102.071453 148, 2021.
A. Fallah and et al. Personalized federated learning with theoretical guarantees: model-agnostic meta-learning approach. In Proc. NeurlPS, virtual, 2020a.
A. Fallah and et al. On the convergence theory of gradient-based model-agnostic meta-learning algorithms. In Proc. AISTATS, Online. PMLR, 2020b.
David Ha and et al. Hypemetworks. In Proc. ICLR 2017, Toulon, France. OpenReview.net, 2017.
Peter Kairouz and et al. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1-210, 2021.
Alex Krizhevsky and et al. Learning multiple layers of features from tiny images. 2009.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86, 1951.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proc. CVPR, pp. 10713-10722, 2021.
Tian Li and et al. Federated optimization in heterogeneous networks. In Proc. MLSys, Austin, TX, USA. mlsys.org. 2020.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In Proc. ICLR, 2020. URL htw.:tppse//ppeneeeeeeeeeee//e/e
id=HJxNAnVEDS
Xin-Chun Li and et al. Fedphp: Federated personalization with inherited private models. In Proc. ECML-PKDD, volume 12975, pp. 587-602. Springer, 2021.
Zhenguo Li and al. Meta-sgd: Learning learn quickly for few shot learning. CORR,
abs/1707.09835, 2017.
Paul Pu Liang and et al. Think locally, act globally: Federated learning with local and global representations. CoRR, abs/2001.0155 005523, 2020.
Yishay Mansour and et al. Three approaches for personalization with applications to federated learning. CoRR, abs/2002. 10619, 2020.
10

---

Under review as a conference paper at ICLR 2023
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. CoRR, 1ess20002.10619, 2020.
Brendan McMahan and et al. Communicationn-fficiennn learriinnnetttnnnngggggggnn of deep networks from decentralized data. In Proc. AISTATS, Fort Lauderdale, FL, USA, volume 54, pp. 1273-1282. PMLR, 2017.
X. Ouyang and et al. Clusterfi: a similarity-aware federated learning system for human activity recognition. In Proc. MobiSys, USA, pp. 54-66. ACM, 2021.
Yijiang Pang, Boyang Liu, and Jiayu Zhou. smoothing. CoRR, 0tss/207..051277 2022.
RUSH: robust contrastive learning via randomized
Daniel W. Peterson, Pallika Kanani, and Virendra J. Marathe. Private federated learning with domain adaptation. CoRR, abs/1912.06773, 2019.
Qi Qian and et al. Softtriple loss: Deep metric learning without triplet sampling. In Proc. ICCV, pp- 6449 6459.6647.1000 IEEE, 2019.
Florian Schroff and et al. Facenet: A unified embedding for face recognition and clustering, In Proc. CVPR, pp. 815-823, 2015,
Aviv Shamsian and et al. Personalized federated learning using hypernetworks. In Proc. ICML, Virtual Event, volume 139, pp. 9489-99999 9502. PMLR, 2021.
Tao Shen, Jie Zhang. Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Fei Wu, and Chao Wu. Federated mutual learning. CoRR, Abs/2006.16765 2020.
Virginia Smith and et al. Federated multi-task learning. In Proc. NeurlPS, Long Beach, CA, USA, pp. 4424 4434, 2017.
Benyuan Sun and et al. Partialfed: Cross-domain personalized federated learning via partial initial- ization. In Proc. NeurlPS, 2021.
Kangkang Wang and et al. Federated evaluation of on-device personalization. CoRR, abs/1910.10252, 2019.
Ye Xue, Diego Klabjan, and Yuan Luo. Aggregation delayed federated learning. CoRR,
abs/2108.0744 07.433, 2021.
Xin Yao, Tianchi Huang, Chenglei Wu, Riao Zhang, and Lifeng Sun. Towards faster and better federated learning: A feature fusion approach. In Proc. ICIP, pp. 175-179. IEEE, 2019.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. In Proc. NeurlPS, pp. 10092-10104, 2021.
Michael Zhang and et al. Personalized federated learning with first order model optimization. In Proc. ICLR, 2021.
Yue Zhao and et al. Federated learning with non-iid data. CoRR, wtt.//800.00000 2018.

---

Under review as a conference paper at ICLR 2023
A DEFINITION OF FL
FedAvg [MMMahn & et al 2017 is typical federated learning algorithm. Assuming that there are N clients in total, as shown in Fig. the server samples fraction C' of clients St (|S* N C = K) to participate in FL and broadcasts the global model to them. The k-th client initializes its local model W/ with the received global model and then trains on local datasets Dx, the training objective is:
Σ M(ww); min Fa(Wa) Tikk tEDe
(5)
where n D&l: f(() = l( wiww),, i.c., the loss of i-th instance (æ4, y:) on the local model wik The local epoch is E, batch size is B, so local training executes D0 iterations. Then, clients upload trained local models to the server, and the server aggregates received local models to update the global model:
Wk:
(6)
where n is the total number of instances owned by all clients. All the steps iteratively execute until the global model converges or the maximum round reaches.
B HOW TO USE HYPERNETWORKS TO GENERATE PARAMETERS FOR TARGET MODELS?
Taking the hypernetwork and CNN (target model) for CIFAR-10 in Tab. 2 as examples, here we introduce how (2.6 + 09) of CNN's parameters are more than the parameters amounts (400) of the hypernetwork's output layer, so we first divide CNN's parameters into multiple chunks with ordered ids, each chunk has no more than 400 ( hypernetwork's 'ss output dimension) parameters. Then we sequentially input the embedding of chunk id into the hypernetwork to get the corresponding chunk's 's target parameters. Finally, we concatenate the parameters from all chunks and reshape them with the CNN's parameter shape. To sum up, we call the hyperparameters multiple times to generate parameters for the target model in stacked form. Since each client's 's local hypernetwork is unrelated to others, we applied the above parameter generation way for all clients in our pFedKT.
the hypernetwork to generate parameters for target models. Since the volume
C How TO UPDATE HYPERNETWORK?
Assuming that the hypernetwork is yo, and it generates parameters for the target model w = h((; yp) (u is the hypernetwork's 's input embedding). The hypernetwork & and the generated target model are trained in an end-to-end manner. Specifically, the generated target model w first executes gradient descent, then the hypemetwork &p is updated also through gradient descent. We assume that the loss function of the generated target model w is f(w) = ((h(w; sp)), so we refer to Shamsian & 2021 and also utilize chain rule derive the following equation:
V V = Vwl(w) Vew = (Vow)" Vwl(w).
For V. (()), we can use its first-order gradient approximation Shamsian & et al 2021 Zhang & et all 2021 to represent
Vul(w) == Aw w
(8)
where @ and are the target models after/before training. So replace Vol(w) of Eq. with Eq. and get:
pel(w) (V pF) Aw
After computing the gradients of the hypernetwork y, its parameters are updated through gradient descent, i.e.,
9 p THN (H (Vow)" Aw,, where MHN is the learning rate of the hypemetwork sp.
(10)
12

---

Under review as a conference paper at ICLR 2023
Algorithm 1:
PFedKT
Input: N, number of clients; K, number of selected clients: total number of rounds; NJN learning rate private target networks: vjN,, learning rate of private hypernetworks; E, local epuchs, batch size; weight of contrastive loss, a, margin in triplet loss
Randomly initialize the global model w private model wi = private hypernetwork i/:: for each round r=0, R-I R-R/Radd do
S randomly select K clients from N clients
Clients execute:
for each client & S" do
Receive the latest global model w' form the server
Utilize hypernetork 4ik to generate initial private model 0
Train 0% on local dataset Dz to get trained private model w by contrastive learning: < split local dataset Dj: into batches of size B
wh +
for each local epoch from 1 to E do for each batch b E B do
Compute logits: Lut Ru)( Lut RRy L --
Compute distances: d'kL = dxi( fuz), dxl = dxl ( fug))
Compute contrastive loss: loon max dxc, + ax 03 triplet loss Compute supervised loss: loup = Cross Errossnntropy of w (b), label)
Complete complete loss function: f j loon (1 11) lanp
Gradient descent: wū <- wi V
end end
Store trained model w locally
Use trained private model w and initial private model 03 to update hypernetorkk: w + o N 022) 10"''(0 (6% wh) // according to Eq.
Upload private trained model w to the server end
Server executes:
Receive private models [.. from clients and aggregate them by: w >i=1 "K w/ // mx. number of kethh client's samples; n, total number all clients' samples
Send the updated global model w to clients selected in the next round
end
Return personalized private models [w] [t w
w B-11
D PFEDKT ALGORITHM
Here, we illustrate the detailed algorithm of pFedKT in Alg.
E DETAILED THEORETICAL ANALYSIS AND PROOF
In this section, we detailed the assumptions and derivations for Theorem 5.1 and Theorem 5.2 respectively.
E. DETAILED DERIVATIONS FOR THEOREM 5.1
We assume that the k-th client's local dataset Dj (x({a) 33 ) Dr Pk. Since assume that the private model Wg nonlinear, the following derivations are adaptive to non-convex situa- tions. Then the empirical loss and expected loss of the k-th client's private model wj: be denoted as:
(k) (k)
Del
Dal
Empirical loss (w)) =
E(x
Vi
(K),
wx):
(11)
D
Expected loss f(wx) Ey, (((x(() (K)) wxx))
(12)
13

---

Under review as a conference paper at ICLR 2023
Since utilize local hypernetworks to generate parameters for private models, we h(u; (x)) we can replace W/ with h(;; (p)) in the above two losses, i.e.,
So
DD Σ i=1
Empirical loss (+x) |
E(X; Ui ; (();
(k) (6),
(13)
Expected loss f(pk)) = R, [((x((), yd h(v; spx))
(14)
Since each client holds its own hypernetwork which is unrelated to other's hypemeeworks,, i.c., the hypernetwork's 's input embedding is independent of clients, so the variant is only (P,:
We refer to the assumptions about the parameters yp of hypernetworks Shamsian in which the hypernetwork's 's parameters are bounded in a spherical space with radius r. Besides, we also assume the following two Lipschitz conditions:
2021),
Assumption E.1 The supervised loss luu of the private model w Lipschitz smooth, iee, lxup satisfies:
|1 (2r,y:1)))))))))))))))))())))( - eau y; w2)l S L ||w] - well-
(15)
As analyzed in Qian et al. 2019): Chen et al. 2021 Pang et al. (2022), we can also assume that the triplet loss Loon of PFedKT is Lipschitz smooth. So we can update the above assumption
Assumption E.2 The complete loss of the private model w satisfies:
[|(( w1) E(x,, yı (2)22 Lwll - w2ll 11l
(16)
Assumption E.3 The mapping from the hypernetwork yp the target private model is Lipschitz smooth, i.e.
[h(w;4p1) - hi: (2) Luller - p2ll-
(17)
Among the above two Lipschitz conditions, L Lip are Lipschitz constants.
Based on the above assumptions and the derived threshold of local data volume M O( log ((e,,))) in Shamsian & et al (2021); Baxter (2000). C(c, is the covering number of HI, H2 is parameterized by the hypernetwork yo. And the distance between two hypernetworks
(P1 i2 dataset with distribution P can be computed by:
d(pi, p22) = (M(e))))=((((((0))))))))))) - -M)r
S I (L11((((( - ((((1)
(18)
s Lopllll p2ll-
Then we choose e-covering parameter space, in which we can always find at least one neighbor with T Top distance to (pi, we can get:
lo,,C(e, H))) O((embain + HN log( r rhw Tug )),
(19)
where embdim and HN size are the input embedding and parameter capacity of the hypernetwork.
Similar to Theorem 1 in Shamsian & et al (2021), we conclude that PFedKT's generalization affected by the hypernetwork's 's input embedding and size, and also the above Lipschitz constants, illustrated in Theorem 5.1
E.2 DETAILED DERIVATIONS THEOREM 5.2
Based on above Lipschitz conditions, we further make the following assumptions:
Assumption E.4 The gradients and parameters of models are bounded, i.e..
B(llgw))) [| < a [B[ 0z² B(llg 112 < 03² [[ a
(20)
14

---

Under review as a conference paper at ICLR 2023
where op is the hypernetwork and w is the target model with parameters generated by the hypernet-
work s. 0,2,3,4 are constants.
Li al 2020) have proved that FedAvg can converge to O(//T) on Non-IID dataset when partial clients participate in FL, i.c.,
2x
B + 2L =*
[[f(wT)] -
(21)
7 7
where [[[(w)) is the average loss in the T-th round; f is the minimum loss of w during T rounds' optimization; K, y, B, C, f4, L are constants assumed derived in [L et al. (2020); wp the initialized model, w" is the optima with f*. Since the above conclusion has been proved in Li et al. 22200, here we no further detail it.
In our pFedKT, each client's s private model is generated by its private hypernetwork ye, so we have:
We = h(v; fo), w h((;)()))
(22)
Hence, Eq. 21 can be replaced as:
2x B + +T
B[f(wr)] f
+ 22) - h(u; h();;""113>>
(23)
μ
From Assumption E.3 the above equation can be further derived as:
2k B+C
Bif(wr)] - f"
+L Lollyp 2²112))
(24)
+ T
According to the Lipschitz condition in Assumption E.4 we can further get:
2x
B C + 2L L O(1/T),
E[f(wr)]
(25)
+T
where Li and 0 are the constants defined in Assumption E.3 and Assumption E.4 Therefore, our PFedKT has the same convergence rate ( with FedAvg.
F EXPERIMENTAL DETAILS
F.1 MODEL STRUCTURES AND HYPERPARAMETERS
We describe the structures of CNN models used on CIFAR- 10/100 datasets and hypemeeworks used for all tasks in Tab. 2 And we also report the detailed pFedKT's hyperparameters used in three tasks Tab. 3
Table 2: Structures of CNN models on CIFAR- 10/100 datasets and hypernetworks used for CIFAR- 10 and Stack Overflow datasets. Note: the hypernetwork used on CIFAR- 100 dataset has 2 hidden layers, i.e., its structure: fel (emb. dim, 100) + fc2 (100, 100) - fc3 (100, 100), + fc4 (100, 400).
CNN (CIFAR-IO) 10)
CNN (CIFAR-
Hypernetwork
layer name
input size
output size
filter
input size
output size
fiker
input size
output size
filter
convl 1
3x32x32
16x28x28
3x3x16
3x32x32
16x28228
3xxx16
max pooll
16x28x28
16x14x14
16x28x28
16x14x14
conv2
16x14x14
32x10x10
5x5x32
16x14x14
32x10x10
5x5x32
max pool2
32x10x10
32x5x5
2x2
32x10x10
32x5x3
2x2
800
1920
800
1920
Emb dim
fc2
1920
80
1920
80
100
100
fe3
80
10
80
100
100
400
F.2
MORE EXPERIMENTAL RESULTS OF COMPARISONS WITH BASELINES
Here, also detailedly report the experimental results Stack Overflow dataset and the com- parison results of our TT and four related advanced baselines on CIFAR- 10/1000 dataset with different Non-IID degrees and diverse client participation rates.
15

---

Under review as a conference paper at ICLR 2023
Table 3: The settings of PFedKT's hyperparameters in three tasks, emb dim the hypernetwrr''s input embedding dimension; n hidden is the hypernetwork's number of hidden layers; is the weight of contrastive loss; is the margin in triplet loss. Note that all the hyperparameters are approximate optimal due to our coarse-grained searching.
Dataset
emb.dim
n.didden μ
contrastive loss
distance
CIFAR-IO (Non-IID)
CIFAR- (Non-IID)
2
0.001
triplet loss
KT. divergence
Stack Overflow
0.1
Table 4: The experimental results of our PFedKT and baselines on Stack Overflow dataset (a large natural real-world Non-IID dataset).
Aoc Local Training
FedAvg
FaUPHP
MOON
pFedHN
Fod-RoD
pFeaKT (ours)
AMe
15.71=2.30
23.5240.05
23.7911.41 23.6440003
19:10:0001
24.98.0.155
25.03=0.01
F.2.1 EXPERIMENTAL RESULTS ON STACK OVERELOW DATASET
We report the experimental results on large natural real-world Non-IID dataset Stack Overflow in
Tab. We can observe that our pFedKT presents the best PM's accuracy, again verifying its utility.
F.2.2 COMPARISONS WITH BASELINES ON DATASETS WITH DIFFERENT NON-IID DEGREES
To explore how the Non-IID degree affects the performances of our pFedKT and four related ad- vanced baselines, we allocate {1, 2, ..., 10} classes of data into one client for the CIFAR- 10 dataset and divide { (10, 20, .... 100} classes of data into one client for CIFAR- 10R00000 dataset, and the results are reported in Fig. and Tab.
observe that PM's accuracy drops the IID degree rises (the number of classes owned by one client rises). Since clients hold more classes of data, their PMs have lower preferences each class (i.e., compromised personalization), so PMs' accuracy degrades. This result is also consistent with the argument in (Shen et al. (2020): GM targets to improve its gener- alization but PM aims improve its personalization, to GM. Our PFedKT presents superiority on severely Non-IID data, but performs marginally worse than baselines high IID data, such as CIFAR-I0 10 with class (8. 9, 10} and CIFAR- 100 with class (70, 80, 90, 100). Nevertheless, decentralized data held by devices participating in FL are often highly Non-IID (Kairouz et al. 2027). hence our pFedKT's utility is still practical.
Results. From Fig.
Non-IID is beneficial to PM but harmful
Analysis. pFedHN uses server's hypernetwork to learn local models' parameter distributions instead of simple weighted aggregation in FedAvg, when local datasets are more IID, the server's hypernetwork learn local knowledge more evenly and hence showing higher accuracy as III degrees increase. FedPHP allows each client's 's linearly accumulated old local model to teach the received global model in knowledge distillation paradigm, it actually trains the global model, so it adapts more to IID data. Our PFedKT performs worse than baselines in IID data because it emphasizes the local training of personalized private models and increasing IID degrees may compromise personal- ization.
CIFAR-10 (Non-IID) PM@Acc
CIFAR-100 (Non-IID) PM@Acc
100
FeaPtiP
FeaPHP
PM@AAc (%)
MOON
PM@ (%)
MOON
pFedHN
pFedHN
80
Fad-ROD
40
Red-ROD
pFedKT
pFecKT
20
60
10
20
40
60
80
100
class
class
Figure 5: Test accuracy of PM on CIFAR-10/100 datasets varies with different Non-IID degrees.
16

---

Under review as a conference paper at ICLR 2023
Table Numerical test accuracy of PM on CIFAR-10/100 datasets with different Non-IID degrees.
PM@ Ace, CIFAR-I0 (Non-IID)
class
6
10
FedPHP
100.00
52.46
75.02
72.99
69.09
68.01
66.40
66.99
65.30
63.60
MOON
100.00
51.78
71.22
72.46
65.06
63.29
64.36
64.84
63.27
DFedHN
100.00
90.03
81.93
76.82
72.17
68.20
67.23
65.10
64.65
Fed-RoD
100.00
88.41
81.63
74.71
69.45
66.12
64.95
64.61
63.78
84
PFedKT
100.00
90.48
84.44
75.94
71.51
66.64
67.21
64.76
63.29
60.47
PM@ Ace, CIFAR-I00 (Non-IID)
class
30
50
60
80
100
FedPHP
6.47
32.43
28.60
26.81
24.90
23.86
23.13
23.31
22.74
21.68
MOON
5.78
28.07
26.90
26.42
23.57
21.67
22.71
22.17
20.57
19.43
pFedHN
58.20
41.40
34.22
30.81
26.36
25.08
24.42
21.68
22.79
19.70
Fed-RoD
58.25
45.37
37.01
31.78
27.50
25.31
24.26
22.96
23.37
20.41
PFedKT
61.24
44.79
33.00
32.02
25.35
24.57
20.74
19.80
18.16
15.51
F.2.3 COMPARISONS WITH BASELINES UNDER DIFFERENT CLIENT PARTICIPATION RATES
To evaluate the effects of our TFedKT and baselines under different client participation rates (i.e., fraction C)2 we conduct experiments on CIFAR 10/100 datasets with 50 clients. We vary fraction C' (0.1,022, ..... }} and report the results in Fig. 6 We can observe that: a) our PFedKTT presents the highest PM accuracy any fraction setting: b) FeeKKT's PM accuracy less affected by fraction. In addition, also test our PFedKT and offte-of-the-art Fed-RoD under the FL settings with 500 clients and lower frac 0.01 client participation rate. The PM accuracy of Fed-RoD and our PFedKT 79.7%% 79.98% All the above results verify that TFedKT is robust to client participation rates.
CIFAR-10 dataset and 30.85% 31.63% CIFAR- dataset.
CIFAR-10 (Non-IID) PM@Acc
CIFAR-100 (Non-lID) PM@AcC
60
PM@AA (%)
PM@Acc (%)
80
Fedlvg FedROD pFecKT
40
FedROD
60
20
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
frac
frac
Figure 6: Test accuracy of PM varies with different client participation rates (fraction C)) on CIFAR- 10/100 datasets.
F.3 DETAILED EXPERIMENTAL RESULTS IN CASE STUDY
Here we report the detailed experimental results of five cases on CIFAR-10 (Non-IID: 2/10) and CIFAR-100 (Non-IID: 10/100) datasets in Fig. and Tab. 6
CIFAR-10 1D (Non-IID: 2/20)
CIFAR-100 (Non-IID: 10/1001
CIFAR-10 Mon-llID: 2/10)
CIFAR 100 (Non-IID: 10/100)
82
%
PMgiAc
90.0
PMgvAc
40
60
70
80
emb. dim
emb dim
TL h.hidden
n_hidden
CIFAR-10 (Non-IID: 2/10)
CIFAR-100
10/100)
CIFAR-10
2/10)
CIFAR-100 (Non-IID: 10/100)
54
80
PAnggAce (5))
PWigAcc (N)
PMBAAce
60
85.0
50.0
86
20
0.
50
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
40
40
slpha
alpha
Figure 7: PMs' mean test accuracy varies with the hypernetwork's 's input embedding dimension and
number of hidden layers (size), the weight μ of contrastive loss, and the margin in triplet loss.
17

---

Under review as a conference paper at ICLR 2023
Table 6: The test accuracy of PM varies with the hypernetwork' input embedding dimension.
Table The test accuracy of PM varies with the hypernetwork's 's number of hidden layers (size).
Dataset
CIFAR-IO (Non-lID)
CIFAR-TOO (Non-IID)
Dataset
Con-IIO CIFAR-IOR (Non-IID)
embb dam
PM@AAc
PM Acc
hidden
PM@Acc
PM@ Ace
6
89.30
58.46
1
86.16
52.21
13
90.34
59.99
85.78
53.76
17
90.24
3
83.13
48.77
26
89.90
62.07
75.56
17.26
62.83
5
57.79
12.15
Table 8: The test accuracy of PM varies with the weight μ of contrastive loss.
Table 9: The test accuracy of PM varies with the margin a triplet loss.
Dataset
CIFAR- 10 (Nom-
CIFAR-100 100 (Non-IID)
Dataset
CIFAR-10 (Non-11))
CIFAR-100 (Non-1ID)
1
PM Ace
PMo
alpha
PM Acc
Ace
$2.39
48.92
0
86.16
52.21
0.0001
83.47
52.74
0.1
87.54
51.38
86.16
85.63
53,20
0.01
84.43
50.50
5
87.27
$3.73
53.30
11.41
10
87.02
52.19
0.3
47.32
8.67
20
86. 74
$3.65
0.5
35.23
5,35
30
86.74
53.65
40
85.04
$2.06
0.9
19.07
1.24
50
84.56
47.91
C. Loss functions in contrastive learning
There are various ways to measure the distances between two models' logits vectors, such as KL divergence, 1- (cosine similarity), 1.2 norm, and MSE (the square of 1.2 norm). There are also diverse contrastive loss functions, such as triplet loss (Schroff & et al. 2015), loss in MOON (Li et al. 202T), and etc. In fact, we pursue PM be close to GM via contrastive learning, so naive way is to add the euclidean distance (L2 norm) between the GM and PM as the regularization to the supervised loss, like FedProx (Li & al 2020). We evaluated several combinations of the above distance measurements and loss functions, and the results are reported in Tab. TO
We can observe that the combination of triplet loss and KL divergence that designed PFedKT achieves the best PM on the CIFAR-10 10 dataset. pFedKT with loss used in MOON (Li et al. (02T) gets the highest PM's accuracy on the CIFAR-100 dataset but lower PM's accuracy on CIFAR- TO dataset than pFedKT with triplet loss and KI. divergence. Other combinations also show worse model performances than our designed loss for PFedKT. It's worth noting that pFedKT with con- trastive loss has higher accuracy than with L2 regularization, which verifies using contrastive loss to keep new PM (generated by HN) away from old PM necessary and does prevent being into local optima.
Table 10: PM's test accuracy varies with combinations of distance measurements and loss functions.
Dataset
CIFAR-10
CII-AR-O00
Loss Function
Distance Measurement
PM@ Ace
PM@ Ace
KL divergence
86.16
52.21
1- (cosine similarity)
84.38
53.48
Triplet Loss
1.2 norm
85.94
52.41
MSE
59.18
12.96
Moon loss
cosine similarity
84,66
53,81
KL divergence
84,84
52.96
cosine similarity
84.39
$3.22
1.2 Regularization
1.2 norm
85.99
51.34
MSE
83.55
49.17
Table 11: Results of ablation experiments. "HN- - denotes private knowledge transfer by hy- pernetworks, "CL" represents global knowledge transfer through contrastive learning.
Datasct
CIFAR-10
CIFAR-100 PM@ Acc
Case
HN
PM
CL
PM@Acc
A
K
51.64:1.022
4.5911039
B
/
90.10=0.51
61.23:10.67
C
X
51.78.10.07
16.40.10.03
D
90.34++0.12
61.66+00008
18

---

Under review as a conference paper at ICLR 2023
CIFAR-10 (Non-IID: 2/10)
CIFAR-100 (Non-IID: 10/100)
0.90
0.6
Test Accuracy
0.85
Test Accuracy
0.5
0.80
0.75
0.3
0.2
Case B
0.65
Case
0
100
200
300
400
500
100
200
300
400
500
Communication Rounds
Communication Rounds
Figure 8: PMs' mean test accuracy of case (HN->PM w/o CL) and case D (HN-PM w/ CI.) vary with communication rounds on CIFAR-10/100 datasets.
G PFEDKT'S COMPUTATIONAL COMPLEXITY, STORAGE COST AND MODEL PERFORMANCES
In this section, compare pFedKT, state-of-the-art PFedHN and other baselines in computational complexity, storage overhead, and model performances.
G.1 COMPUTATIONAL COMPLEXITY
Taking the CNN (CIFAR-10) 10) and the hypernetwork in Tab. 2 as examples, we denote CNN (CIFAR- 10) as NN (i.e., target private model) and the hypernetwork as HN(small). In pFedHN (Shamsian et all 202T), one large hypernetwork deployed on the server, which we denotes as HNllagge. For fair comparisons, we set the same number of hidden layers of HN(small) and HN(large), i.e., 100) - fc2(100, 100) - fc3(100, NN Param). We compute the tree models' parameter capacity and computational overhead for one-time forward operation, and the results are recorded Tab. 12
the structure of HN(large) is: fc
Table 12: The three models' parameter capacity and computational overhead (FLOPs) for one-time forward operation.
Metric Layers
Farameter Capacity
Computational Overhead (FLOIS)
NN
HN (smal))
HN (large)
NN
HN (small)
HN (large)
(575"541)*16
conv2
(5-5-16+))332
(16-55555H(((55555))))) 32
fel
(800+1)+1900
(13-1)-101
(13+1)* 100
2*M0P 1920
2-13"100
2913-100
fo2
(1920s1)*80
(180-17-100
(100+1))100
241920'80
2-100" 100
2=100"100
(80+1)100
(100-11-400
(100+)"* 1706455
2-100"400
Istal (number)
1706558
31900
172363758
7823000
102600
341314200
Total (MB/GB)
6.5096 MB
0.1980 MB
657,5156 MB
0.0291 GB
0.0004 GB
1.2715
Computational Complexity
(Non-IID: 2/10)
(Non-ID:: 10/100)
Hiilmmal) ploneantiioogg
2.8
2.0
Acourscy
0.9
0.8
Taxt Accuracy
1.6
0.7
aare HN)
0.4
FFo argg HN)
Fediiiilaal HNI
pfu ael Hill
2.4
0.2
1.2
0.6
200
300
400
500
100
200
300
400
500
HRL_uupue
Round
Round
(a)
(b)
(c)
Figure 9: (a): the ratio of computational complexity (FLOPs) between HN (small) and HN (large) varies with the output dimension of HN (small); (b-c): PM's accuracy of pFedKT with HN (small) and HN (large) CIFAR-10/10 10/100 datasets varies with rounds.
From Tab. 12 it requires to call HN (small) Parrasossoutpput of HN (small) times to generate the whole parameters for one NN in a stacking form, so generating NN by HN (small) requires (NN Paras/output of HN(smalI)*HN (small) FLOPPs, i.e., (1706458/400) 102600 1.6306 GB
19

---

Under review as a conference paper at ICLR 2023
FLOPs. Using HN (large) to generate one NN consumes 1.2715 GB FLOPs, shown in Tab. 12 Hence the ratio of the former and the latter is about 1.28x, and it tends to be 1x as the output dimension of HN (small) increases, as shown in Fig. 9 (a). In short. pFedKT consumes comparable computational cost to PFedHN.
Besides, PFedHN updates the server's HN (large) once if it receives one private model. Using one private model to update HN (large) and then using the updated HN (large) to generate parameters for the private model consume 1.2715#2 GB FLOPs. When multiple private models reach the server simultaneously, computational blocking may occur due to the high computational complexity HN (large). Whereas, our PFedKT deploys one HN (small) on each client. From the perspective of computational complexity, our pFedKT inherently offloads the training tasks with the server's ss large hypernetwork in pFedHN to clients' sub-tasks, which tackles the above blocking issue.
G.2 STORAGE OVERHEAD
PFedHN's server requires about 657 MB to the HN (large), while our PFedKT's N clients consume N 0.1980 MB storage cost. When the number N of clients participating in FL is about 3321, our PFedKT has comparable storage cost pFedHN. But in the cross-silo FL scenario, there are often few companies or institutions joining in FL (Kairouz & et al. 202T), so our pFedKT has obvious strength than pFedHN in terms of storage cost.
G.3 MODEL PERFORMANCE
We have compared the model performances of pFedHN and our PFedKTT in Sec. 6.2 here do not repeat it. As illustrated above, we require to call HN (large) once or HN (small) multiple times to generate parameters for one NN. Here, we also test our pFedKT with private HN (small) and private HN (large) on CIFAR- 10/100 datasets, and the results are shown in Tab. 13 and Fig. 9 (b)- It can be seen that TFedKT with HN (large) shows similar model performances with FedAvg, which is consistent with our conclusion of the case study on HN's size in Sec. ??: larger HN is harder to train, hence showing worse model accuracy. This evaluation also verifies the strength of our calling HN (small) multiple times model performances.
Table 13: The results of pFedKT with HN (large) and HN (small) on CIFAR-10/100 datasets.
Dataset
CIFAR-10
CIFAR-100
Method
PM@ Acc
PM@Acc
FedAvg
51.64
4.59
pFedHN
90.03
58.2
PFedKT (small HN)
90.34
61.66
PFecKT (large HN)
51.67
5.04
G.4 COMPARED WITH MORE BASELINES
We also compare our PFedKT with the typical FedAvg and related MOON, Fed-ROD in computa- tional overhead and storage efficiency.
TTedKT V.S. FedAvg. Compared with FedAvg, PFedKT introduces extra local computation of local hypernetwork and contrastive loss, and additionally stores a local hypernetwork and a previous local model. But, show in Tab. [FeKKTTs accuracy is improved 38. 7% in CIFAR- 10 and 57 .07% C00 Therefore, in cross-silo FL scenario, increased cost compared with obvious accuracy improvement is acceptable for participating enterprises and institutions.
FT v.s. MOON. Compared with MOON, PFedKT introduces the computation of local hyper- network and additionally store local hypernetwork for each client. But PFedKT improves 38.56% and 56.41% accuracy than MOON in CIFAR- 10 and CIFAR- respectively.
FT v.s. Fed-ROD. Compared with FedAvg, Fed-ROD's 's extra computations involve that using a local hypernetwork to generate parameters for a personalized header and locally train personalized branch, while it also requires to store a local hypernetwork additionally. Compared with Fed-ROD, our KTedKT introduces the following computations: using a local hypernetwork to generate pa- rameters for complete local target model and calculate contrastive loss. Since PFedKT's local
20

---

Under review as a conference paper at ICLR 2023
hypernetwork generates parameters for whole local model and Fed-ROD's local hypernetwork generates parameters for the personalized header of local model, pFedKTT incurs relatively higher computation cost. In addition, pFedKT requires to store local hypemeeworr and previous local model, but Fed-ROD only stores a local hypernetwork, so pFedK'T has a slightly higher storage overhead. However, similar the above analysis, pFedK'T improves 0.31% and 3.46% accuracy improvements than Fed-ROD in CIFAR-IOL datasets, which is acceptable.
21


Reviews:
[{"review_id": "MIeC7vLhT3R", "reviewer": "Reviewer_fgAG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle the problem of non-iid data in FL. They propose the use of hypernetworks and the target network to cope with this. Clients use a contrastive loss to mitigate non-iid. \nThe key idea is for clients to utilize prior local weights and new global weights obtained from the server. They also provide a theoretical convergence analysis. \nThe computational experiments show that the proposed algorithm outperforms the benchmarks. ", "summary_of_the_review": "The use of prior local models is definitely a nice idea however I think the authors don't do much with it. The application of the contrastive loss is not new and thus not a contribution. \n\nA have issues with theoretical results. \nTheorem 4.1 require a large number of samples which pretty much goes against non-iid. Clearly a large number of samples does not imply idd but it does exclude many non-iid situations. Theorem 4.1 is loosely stated but it seems to show convergence to global optimal in absence of convexity. This clearly cannot hold. \nThe same remark holds for Theorem 4.2. \n\nThe experiments are limited. The data is non-iid only in terms of class distribution but not in terms of features. This is clearly quite limited. \n\nTable 1: Improvements over local training are 'too good to be true.' I guess it questions if local training has been optimized. For CIFAR-10 the improvement definitely doesn't impress (90.34 vs 90.03). \n\nThere is one definitely relevant reference that is missing: Y. Xue, D. Klabjan, and Y. Luo. Aggregation Delayed Federated Learning.", "strength_and_weaknesses": "Strengths: \nThe use of prior local models is novel. \n\nWeaknesses: \nThe theoretical analysis is unclear and the connections with non-iid is unclear. \nThe computational experiments do not show extensive improvements and are flaky. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HlJM1vOoF76", "reviewer": "Reviewer_Luyb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to improve the performance of personalized federated learning, and, for which, the authors propose two knowledge transfer schemes. In particular, the historical knowledge learned in the local clients is transferred from the hypernetwork, which stores the knowledge of previous local models, to the current local model. Also, the global knowledge, obtained by the aggregation of local models, is transferred to the local clients based on the contrastive learning loss, where the similarity between the updated local model and the aggregated global model is maximized while the similarity between the updated local model and the previous local model is minimized. The authors validate the performance of their model, named as pFedKT, on two image (i.e., CIFAR-10/100) and one language (i.e., Stack Overflow) datasets, on which the pFedKT outperforms relevant personalized federated learning baselines.", "summary_of_the_review": "The main idea of the proposed global knowledge transfer scheme based on the contrastive loss is not convincing (See Weaknesses above), and there are some improvement points, such as efficiency analyses in Section 3.4 which are not sufficient, and model analyses which are perhaps conducted without the model convergence. Therefore, I cannot recommend the acceptance.", "strength_and_weaknesses": "### Strengths\n* The proposed historical knowledge transfer scheme with the hypernetwork brings the performance improvement for the personalized federated learning tasks.\n* The authors make effort to theoretically analyze the generalization and convergence bounds of the proposed pFedKT, while they are mostly inspired by the previous work [1] though.\n* The authors perform extensive analyses on the proposed pFedKT, by varying the hyperparameters, and by ablating the knowledge transfer mechanisms. \n\n### Weaknesses\n* The proposed global knowledge transfer scheme is not convincing enough in terms of both the motivation- and experiment-sides. \n* * At first, the authors argue that, by maximizing the similarity between the locally updated model and the globally aggregated model while minimizing the similarity between the locally updated model and the previously updated local model, the proposed pFedKT improves the generalization performance. However, this design choice is not convincing, since the authors already transfer the historical knowledge in the previously updated local model with the hypernetworks, meanwhile, the historical knowledge is negatively considered (i.e., historical information is avoided) in the contrastive learning loss. Therefore, two objectives are conflicts in the federated learning.\n* * In the experiment-side, the proposed global knowledge transfer scheme also does not bring the meaningful performance improvement, i.e., not much helpful for the personalized federated learning. For example, in Figure 7, the proposed global knowledge transfer scheme based on the contrastive loss does not bring the performance improvement. Similarly, in Table 11, the results w/ and w/o contrastive losses are very similar. \n* * Furthermore, in the experimental-side, it is unclear whether the proposed global knowledge transfer scheme can provide better generalization ability empirically. While the authors provide the theoretical result for the generalization bound, I suggest authors to include additional empirical results, if possible, which makes it more convincing.\n* In Section 3.4, the authors only compare the computational and storage efficiencies of the proposed pFedKT against the complex pFedHN model. It is meaningful to discuss the efficiencies of the most basic FedAvg model, as well as the other contrastive- and hypernetwork-based federated learning models, such as MOON and Fed-ROD. \n* Also, in Section 3.4, the authors argue that the proposed pFedKT has obvious strengths against the pFedHN model, since the pFedHN baseline has the larger hypernetwork in the server-side, while the proposed pFedKT has smaller hypernetworks in the client-side. However, this is not convincing, since if we have 1,000 clients, we have 1,000 individual hypernetworks distributed to 1,000 clients, and, in the global view, the size of 1,000 individual hypernetworks would be larger than the size of one hypernetwork in the server.\n* The analysis results in Section 5.3 may be problematic. The authors argue the proposed pFedKT can converge during 100 rounds, therefore, conduct analyses with 100 rounds; however, pFedKT does not converge until 100 rounds, as shown in Figure 4.\n* Since knowledge distillation-based federated learning methods share similar sprits to the proposed knowledge transfer-based federated learning model: the knowledge distillation allows the local/global models to transfer their knowledge effectively, I suggest authors to compare such knowledge distillation-based federated learning baselines: FedPHP, FML and KT-pFL, discussed in the related work section.\n\n---\n\n[1] Personalized federated learning using hypernetworks, ICML 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6KBYeOsGcO", "reviewer": "Reviewer_5c7g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes pFedKT to address the Non-IID data issue in federated learning. pFedKT is based on personalized FL with knowledge transfer. During local training, it utilizes a hypernetwork to generate a local model. The local model is updated with a contrastive loss as regularization, which limits the distance between the local model and global model to incorporate the knowledge of global model into local model. The hypernetwork is then updated with the updates in local model such that the hypernetwork can absorb the knowledge of the existing local models to generate the local model for the next round. Experiments show that pFedKT achieves better accuracy than the other baselines.", "summary_of_the_review": "The paper proposes a new PFL algorithm. While the presentation is good, I think the experiments need further improvement and the theoretical analysis needs more justification.", "strength_and_weaknesses": "Strength: The paper works on an important problem of FL. The idea is clear and the baselines in the experiments are comprehensive.\n\nWeaknesses:\n1. The improvement of pFedKT is low compared with other personalized FL approaches such as pFedHN. From Table 5, the accuracy of pFedKT is even lower than pFedHN in many cases. \n\n2. Each client has a fixed number of classes in the experiments, which is not practical in most applications. More real-world federated datasets can be used in experiments.\n\n3. The local iterations per round is fixed to 100 in the experiments. Experiments with different numbers of local iterations are necessary since the baselines may not work well with a large number of local iterations. \n\n4. The paper assumes that the contrastive loss is Lipschitz smooth and refers to [1] to support the assumption. I cannot see a clear relation between the assumption and the reference. The paper needs to provide more explanation about it.\n\n[1]  Softtriple loss: Deep metric learning without triplet sampling. \n\n5. The paper compares many baselines in the experiments. However, some of them are not personalized FL approaches (e.g., FedProx, SCAFFOLD). The authors may consider removing them to save space.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject