Spectral Co-Distillation for Personalized Federated Learning
Authors: Zihan Chen
Howard Hao Yang
Tony Quek
Kai Fong Ernest Chong
Venue: NeurIPS 2023 poster (2023)
Keywords: Personalized federated learning
spectral bias
co-distillation
communication efficiency
TL;DR: We propose a spectral co-distillation framework that is well-suited for personalized federated learning.

Abstract: Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose $\textit{spectral distillation}$, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.

Spectral Co-Distillation for Personalized Federated Learning
Zihan Chen', Howard H. Yang", Tony Q.S. Quek', and Kai Fong Ernest Chong'*
SSingapore University of Technology and Design (SUTD) ZZejjinng University/Univerrityy of Illinois Urbana-Chanpaign Institute, Zhejiang University zihan_ bahhan_hen@sutd.. edu.sg
Abstract
Personalized federated learning (PFL) has been widely investigated address the challenge of data heterogeneity, especially when a single generic model is inadequate satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such similarity is primarily based on either partitioning the model architecture into generic versus personalized components, modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose spectral distillation, novel distillation method based model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait- free local training protocol. Through extensive experiments multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, well our wait-free training protocol.
Introduction
With the rapid rise in mainstream popularity of artificial intelligence (Al) models such as ChatGPT [I] and LoRA (2], there has been an increasing shift towards the development of personalized AI
assistants [3. Hence, in future where personalized AI services become mainstream, training AI models on personal data while preserving data privacy would become increasingly important 4 and maintaining the quality for such models would require collaborative training across multiple distributed
models. Personalized federated learning (PFL. emerges as promising privacy-preservee learning paradigm that is well-equipped to meet such requirements [3] As an extension of federated learning (FL), PFL aims to train a customized machine learning model for each client or each group of clients with similar preferences [5). When faced with inconsistencies in the objective functions of different clients, conventional FL fails to generalize well with just a single model, while in contrast PFL promises to generalize well across all clients, even in the presence of data heterogeneity (e.g.. label distribution skew and label quantity skew) 14].
To tackle the challenges of personalization, numerous works have focused on designing new PFL systems, or enhancing the performance of personalized models from different aspects, such robustness, fairness, and model convergence [1) Under federated settings. personalization is achieved through capturing the (dis- siiiiaiiiyy of the local versus globally shared model representations. In
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

practical FL/PFL applications of collaboratively training deep neural networks (DNNs), only the model parameters (e.g, model weights or gradients) are exchanged between the clients and the server 8 19 Existing DNN-based PFI. methods capture this (dis-)similaate either by decoupling the model architecture into groups of layers/channels(20/ (5), or by designing local optimization methods with regularization based directly on model weights [17 [5]. Unfortunately, the motivations for such approaches are based on empirical observations, without an overarching theory to explain model (dis-) (dis-)similarrty in relation to training dynamics.
In deep learning theory, the training dynamics of DNNs have been studied from the lens of Fourier analysis [26]_ A crucial insight from this analysis is that there is an implicit self- refulaaaaaiiittooon effect arising from the training process itself. Given a target function f to learn, the model tends learn the lower frequencies of the Fourier spectrum of f first before learning the respective higher frequencies. Such a bias in this training process is called spectral bias 27 28 Informally, spectral bias describes the commonly encountered phenomenon that DNNs first learn low-level features before learning
high-level features.
Motivated by this insight, we can distinguish different levels of features in a model representation by looking at its Fourier spectrum. Intuitively, diverse personalized models would still share the same low-level features, and a global generic model would contain the same low-level features. Hence, despite any inconsistencies in the objective functions of different clients, there would be no conflict in learning low-level features for both the generic and personalized models. Consequently, with the expected similarity in the lower frequency components of the Fourier spectra of both the generic and personalized models, we can distill the knowledge of the lower Fourier coefficients to boost the performance of the generic model. Dually, the entire Fourier spectrum of the generic model, which includes the "averaged" high-level features across all clients, would benefit the training of the personalized models. By combining both perspectives, we shall propose a co-distillation framework for PFL that captures (dis-)similarity in models via spectral information.
Typically, when designing PFL systems, a compute-and protocol is implicitly assumed for local training [1S] 233. This means that the locally updated generic models would be sent by the clients to the server after all local computation tasks have been completed. Such a protocol would yield a period of idle waiting where clients have to wait for the next aggregated model to be broadcasted. By circunventing this compute-anee wait protocol, we shall utilize the local idle time for training to reduce the total PFL runtime.
Overall, our contributions can be summarized follows:
We propose a spectral co-distillation framework for PFL. In particular, this is the first ever use of spectral distillation in PFL to capture the (dis-)similarity of the generic and personalized models. Also, this is the first ever bi-directional knowledge distillation directly
between the generic and personalized models.
We propose a wait-free local training protocol for our spectral co-distillation framework, to reduce the total PFL
where we utilize the idle time during global communication runtime.
Through extensive experiments on multiple datasets with heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation framework with the wait-free communication protocol design for PFL, with respect model generalizability and the total PFL runtime.
2
Related work
PFL. In PFL, prior efforts have focused on training multiple personalized models via leveraging the similarity and relationships between the global generic model and the local personalized models, such as via model interpolation/mixturr 29) model decoupling (22] and personalized optimization with customized regularizers [5]. In DNN- based FL applications, decoupling- based approaches divide the model into private part (kept at the local side) and shared part (exchanged between the server and clients) B3
25 23 In particular, FedPer [22] and FedRep (24] share the shallow layers and train personalized deep layers, while contrast, LG-Fee (20 and CD2- maintain personalized shallow layers and channels [2]]. respectively. Moreover, Fed-RoD proposes framework to achieve state-of-the--an (SOTA) performance for generic and personalized models simultaneously, based on
the "two-loss, two-predictor" design (2). APFL [5] and L2GD 29 consider using mixture of local

---

Broadcast
Update
Broadcast
A communication round
GM
GM
Frozen
Loss
Loa
PM
PM
Spectral Diveggence
Cran-Crtropy Lom
Generic model update
Personalized model update
Spectral co-distillation framework with wait-frree local training for PFL, in which the generic model (GM) training and the personalized model (PM) training are carried out via spectral distillation in two different stages.
Figure 1:
and global models to achieve personalization, in which the mixture weight controls the personalization level. Personalized local training methods have been recently explored, which include local fine- tuning in FedBABU 30)] bi-level optimization in Ditto [15] feature alignment in FedPAC [31],
and personalized model sparsification in FedMask [16 32 and PerFedMask (33] More broadly,
meta-learning 34 35). gaussian processes [36], and hyper-network-
approaches 337 have been
investigated PFL. Specifically, there another type PFL that aims train personalized models
at the level of clusters of clients with similar preferences [38-40].
Knowledge Distillation (KD) in FL. KD has been widely explored in knowledge transfer scenarios, which usually is used to transfer knowledge from the pre-trained teacher model to the student model via minimizing the distance from the latent logit outputs of the models [41] 42) KD-based FL frameworks have been developed with diverse setups, such as FedMD weeeedge-tranffr--beee
On PFL frameworks are investigated in 46 with different
43 and FedDF 44
the other hand, model structures at the local clients, which could address the system heterogeneity and improve communication efficiency. However, such methods rely on the assumption of having access public labeled/unlabelled dataset, which may not be realistic assumption in FL applications (3). Moreover, co-distillation methods have been investigated communication-effiiieeee decentralized scenarios to improve generalizability [45]
3 Proposed framework
The main goal of this work is to train generic global model and multiple personalized models simultaneously. As summarized in Sec. our proposed framework consists of three major compo- nents: spectral distillation-base model training, and the wait-free sequential we first provide the preliminary and problem formulation for PFL and model spectrum in Sec. [ Next, we present our proposed spectral distillation approach for PFL in Sec. 3.22 co-distillation- based generic model training in Sec. [.3 and the wait-free local training protocol in Sec. 3.4 accordingly. Moreover, the summarized algorithm is given in Sec. 3.5
personalized model training, spectral co-distillation-base generic communication protocol. In this section,
3.1 Preliminaries
Problem formulation for FL and PFL. Consider FL system consisting of server and N clients, which client has a loss function f. Id + used for training on its local private dataset D. = [(2], where n, = |Dal denotes the size of the local dataset of client i. In conventional FL, the objective of all the participating clients in this system to find global model e Ø that

---

solves the following minimization problem 19:
N Σ S(l)
F(1o) = n4
minimize WERd
where E n, the total number of training samples across the N clients. In typical communication round t, a subset St of clients is selected to conduct local training, starting from the latest global model weights wg. Let w denote the weights of client i's model after local training. At the end of communication round i, the server would collect local models from the selected clients to update the global model via Federated Averaging (FedAvg), i.e. w/
Eies, plac, in which pf = n2/ Ekes, R2 represents the ratio of the local data samples in client over the total number of data samples in the selected subset S, of clients for communication round t.
There are two general types of PFI.: a) training N personalized models for all N clients; and b) training 1 generic model and N personalized models simultaneously. In this work, we investigate the latter one, which we term as "PFL+*" This means each client has a local personalized model "pp for its private dataset D, and all clients jointly participate in the training of the generic model wiG After local training at client i, the updated generic model is denoted by wG,i. Thus, PFL can be formulated using regularized loss function with regularization term R2 (Wp,i, WG..). For example, R2("p.á Wa,i) could represent the similarity/divergenne between the global and local models' features, such as model weights, feature centroids, and prototypes. In method, Rp(Wp. Wo.s) represents cross- model distillation during the training client i's personalized model. Therefore, the objective of personalized model training in PFL+ can be formally formulated bi-level optimization problem 17
(P1): minimize "H,ERd subject to wa,i < updated generic model from wG.
[na((pp)) == Se(Wp.)) + Ap wa.s) for each client
(2)
where the regularization coefficient Ap is used to control the level of personalization. For client i, when referring to specific communication round t, we shall denote the untrained personalized model and updated generic model by w",7 and w respectively.
3.2 Personalized local model training
Motivated by both theoretical and empirical insights of the spectral bias inherent in the training dynamics of DNNs, we explore the use of the Fourier spectrum of the generic model for knowledge distillation to enhance the training of personalized local models. particular, we propose distillation regularization term representing the divergence between the full model spectra of the generic and personalized models.
First, we introduce some notation. Given vectors = (p1, (pp,....Pp).... (qI,..., 9a) in Rdd, define the divergence function (plllg) E p: log p: log 9- (By convention, 0 Ooo = 0.) Note that when and are stochastic vectors representing parameter vectors of multinomial distributions P and Q, then D (ppllg) is identically the Kullback-Leibler (KL) divergence from P to Q. Next, let DFT Cd - Cd denote discrete Fourier transform, let e Cd ..,) +> II..,l....., =ll),, and define the function : Rd >> Rdd by s 2 o DFT. For an input vector of the weights of DNN model, the output vector after applying shall be called the spectrum vector of that model (28). Thus, in communication round i, the spectrum vectors of the personalized model up, of client and updated generic model w/,, are written as s(w))) and s(wrr)) )), respectively. We shall represent the divergence of the personalized and generic models by (s(w p.s ([l (lss(w) the divergence of their spectrum vectors.
+ IRd be the map given by
Concretely, we define (n("p, wa.)) == (s((w(-)) (lll=="))llsssss (1s((() and let fi be the cross-entropy loss LCe for all i. Then the personalized objective function fp. of client in communication round t (ef. 22 is given by:
CP (u 1-1 Iw6,)) = LCE(u,, (P.) ApD(s( ())(((()))))s(((b ()s((wé,)).
(4)

---

Broadcast
Model Update
Broadcast
GM training
PM training
Idle/waiting
A communication round
(a)
Broadcast
GM Update
Broadcast
GM training
PM training
A communication round
(b)
Figure 2: A comparison of the (a) conventional compute-and-we and-wait protocol with the (b) proposed wait-free training protocol.
For simplicity, we use common innerinvaiiant Ap for all clients throughout training. Since we are distilling the knowledge of the spectrum vector s(wg,) in (4), we term our approach spectral distillation.
3.3 Generic model training
Given PFL+ training framework, it is natural connect the roles of generic and personalized models to the roles of the teacher and student models in distillation, where the training of one model guided by the knowledge distilled by the other. Co-distillation extends this idea. Intuitively, the role of each model alternates between teacher and student for knowledge distillation during training. In PFL+, since we concurrently training both the generic and personalized models, either of them could be used for knowledge distillation. The key challenge for applying co-distillation to PFL+ is that it is not obvious what knowledge should be distilled from the personalized models enhance the training performance of the generic model.
In the theory of deep learning, it is well-known that when training DNN, there is learning bias towards the lower frequencies of its Fourier spectrum [27 28). In fact, the lower-frequency components of this spectrum are robust to random weight perturbations. Hence, with diverse personalized models, we would still expect the lower-frequency components of the spectra of all models (both generic and personalized) to be similar. Consequently, could use such lower- frequency components for knowledge distillation to enhance generic model training.
Motivated by this, we propose truncated spectrum-based distillation loss as the regularizer for generic model training. Given 0 < < 1, let f, RRd + [R[rd] be the projection map onto the first [Td] entries, and define $ == 2, o s. Then the loss function for generic model training, which depends on the truncated spectrum vectors ((wa,) and (w ). is given by:
(wb,, = L (D,) X (D (f(wE,)))
where the regularization term Rg (w w6,7 = (3(wg,)) || ((((²))) depends the hyperparam- eter T, and Ag the coefficient of this regularization term. Analogous to (P1), the objective of generic model training in PFL+ could be formulated as the following bi-level optimization problem:
(P2): minimize WgERd
[(wa)=
= Σ (Je(wa) Ag AqRa(wa, Wp,))
(6)
iml
subject to Wp.i + output of (P1) for client i, for i = I.
Overall, by combining the two spectral distillation approaches together, we get training framework for PFL+, which shall call spectral co-distillation.
5

---

Algorithm Spectral Co-Distillation with Wait-free Training for PFL+
Inputs: N, T, Ops MJ. w (u = Eg, Ep
Outputs: Generic model Wig. personalized models (
1: for = to T do
2: for each client = 1 to N in parallel do // Generic model training and update
3: w + GMU " 1)
4: Upload weights w to server I/ Personalized model training (Task 1: Line 5)
5: "pp + PMMPDATE(WW
p.k: 1=1 G
6: do sequentially (Task 2: Lines 7-8)
Perform Tasks & 2 in parallel
7: Uplink communication of generic model to server
8: Generic model aggregation to obtain wy at server
9: return w (up 1
function GMÜPDATE("N p.k.
Require: WG 1--1 p are the latest generic model and personalized model.
1: Wh f u
2: for j to EG do
3: Wi < w Na V - (w
// Using truncated low frequency spectrum information
4: return W/
function PMUPDATE(W p.k v
Require: wp.k waee are the latest personalized model and updated generic model of client k.
1: w0 + w
1-1
p.l
2: for = 1 Ep do
3: Wy Wi-1 7pp V // Using full model spectrum information
4: return W/
3.4 Wait-free Local Training Protocol
In the context of federated computing, the total runtime, which includes both local computation and communication time throughout the entire training process, direct indicator of communication efficiency. However, current PFL frameworks adopt a compute-and-wal protocol for local training. This means that each round, the client performs both generic and personalized model updates only after all local computation tasks have been completed, and resumes local training upon receiving the latest global model broadcasted from the server. consequence, there is idle waiting time between model update and model broadcast; see Fig. 2(a).
To improve the communication efficiency of PFL training with respect to the total runtime, we propose a wait-free local training protocol, as depicted in Fig. ([b). In our protocol, the client updates the generic model according to the conventional generic FL training and trains the personalized model during the global communication time period. Unlike existing PFL frameworks, local clients would send the updated generic model to the server before the start of the personalized model training. Thus, our protocol eliminates idle waiting time, thereby dramatically reducing total runtime. Furtherfore, it could be easily incorporated into existing PFL frameworks, such as Ditto [15), to further improve the efficiency; see Tab.
Discussion on the proposed protocol and related work. Our proposed wait-free local training protocol is specially designed for the PFL+ scenario, where each client trains two models locally. For simplicity, we use this protocol in our experiments. under the assumption of synchronized PFL+. For comparison in the asynchronized PFL+ setting [47]. see Appendix. Related work that reduce the total training runtime, such as delayed gradient averaging 488 and wait-free decentralized FL training [49], are designed for conventional FL and does not deal with the PFL+ scenario. Furthermore, we also provide discussion on how our wait-free local training protocol could be adapted to the partial client participation scheme in FL in the Appendix.

---

0.1
Methods
GM
PM
GM
PM
GM
PM
FedAvg
85.35
(80.33 L 0.38)
80.76 0.13
(74.51 1 0.48)
73.51 1
(72.68 0.39)
FedProx
85.61 # 0.08
(86.28 = 0
80.54 = 0.14
(76.8
71.96
(73.77 0.30)
FedDyn
86.03 0.13
(85.33 0.19)
80.88 0.18
73.62 0.14
(74.25 + 0.58)
FedGen
86.17 # 0.32
(85.24 0.47)
79.86 = 0.34
(7.52
71.36 # 0.28
(71.42 + 0.63)
FedAvgM
85.44 0.05
(82.85 = 0.28)
81.04 0.09
(75.71 = 0.33)
72.87
(72.96 0.14)
pledMe
85.58 #: 0.23
88.17 = 0.17
79.33 = 0.14
84.66 0.11
72.11
81.18 + 0.15
Ditto
85.34 +: 0.10
87.55 0.09
80.70 - 0.13
83.39
73.45 0.18
80.08 0.20
FedRep
(85.61 +: 0.19)
87.32 0.11
(80.33 +0.23)
84.10
(73,50
79.74 0.31
FedRoD
86.02 1: 0.12
91.67 0.16
81.31 1: 0.15
85.91 $5.110.11
74.64 1: 0.07
81.37 0.17
FedBABU
(85.67 1: 0.24)
91.34 = 0.19
(79.57 1 0.23)
83.22 1: 0.33
(73.88 10.19)
80.58 1 0.22
Ours
86.37 1 0.15
92.25 0.21
81.27 1 0.18
86.59 1 0.17
75.52 0.11
82.69 0.16
Table I: Average (3 trials) and standard deviation of the best test accuracies for generic/peeronnalzzed
models of various methods on CIFAR-10 10 with different non-lID settings. See also Remark
=
Q = 0.1
Methods
GM
PM
GM
PM
FedAvg
48.37 0.22
(52.64 = 0.48)
38.61 + 0.27
(39.27 + 0.42)
FedProx
47.33 0.15
(53.85 0.33)
39.55 0.18
(41.33 + 0.38)
FedDyn
49.24 0.27
(57.20 0.35)
40.43
(40.92 + 0.26)
FedAvgM
18.55
55.60
(40.85 + 0.19)
PFedMe
47.29 0.27
61.52 0.25
38.22 0.23
45.88 + 0.32
Ditto
48.37 + 0.25
60.47 = 0.27
39.61 + 0.19
43.12 + 0.28
FedRep
(46.32 + 0.23)
58.76 1276:00 0.36
(40.11 0.35)
45.22 + 0.19
FedRoD
50.07 + 0.16
62.51 0.15
40.58 - 0.22
45.99 + 0.14
FedBABU
(48.52 + 0.30)
60.33 0.28
(37.35 + 0.29)
44.72 + 0.28
Ours
51.39 1 0.22
63.15 0.16
40.67 0.14
46.82 + 0.23
Table 2: Average (3 trials) and standard deviation the best test accuracies for generic/personalized models of various methods CIFAR-100 with different non-llD settings. See also Remark 4.1
3.5 Algorithm Summary
Our proposed spectral co-distillation framework combined with our wait-free local training protocol, given in Algorithm [ an overview, we begin every communication round with the server broadcasting the global generic model w to each client for local computation. Each client would send back the updated generic model w after EG local computation steps for global model aggregation, then immediately start the personalized model training and continue until the global generic model w is received, which marks the start of the next communication round + 1.
Remark convergence analysis. Note that the global loss function includes a weighted sum of the local loss functions and a regularizer. The regularizer is given in the form of the divergence function D, which equivalent to KI. divergence; ef. Sec. 3.2 As demonstrated in (50) the KI. divergence usually exhibits convexity in terms of the model parameters. Consequently, since the model training
undergoes (stochastic) gradient descent, is possible establish convergence rate for the training of the global model (under the commonly employed assumption of smoothness of the local loss functions).
4 Experiments
4.1 Experiment setup
Datasets, DNN models, federated settings, and evaluation metrics. We evaluated our proposed PFL+ framework with N clients CIFAR-10/100 [5]], and iNaturalit-22017, using model archi-
7

---

Methods
FedProx
FydDyn
Ditto
FedRep
FedRoD
FedBABU
Ours
GM
39.46+0.39
39.35=0.27
39.33=0.33
39.81=0.41
40.16-0.35
39.23+0.53
41.75=0.37 45.87
PM
41.58:10.27
0..9
41.88 +0.41
42.07:10.24
44.54 0.29
42.36:10.44
Table 3: Average (3 trials) and standard deviation the best test accuracies for generic/personalized
models of various methods on iNaturalist-2017 with non-IID setting a 0.1. See also Remark 4.1
tectures ResNet-18/34 [52 and ResNet-50, respectively. For the experiments on CIFAR- 10 (resp. CIFAR-100) 100), we used N 100 (resp. N 50). For experiments on iNaturalist-2011 [53, we used N = 20. For dataset partition, we used the symmetric Dirichlet distribution emulate world heterogeneous data distributions [1, where the heterogeneity is controlled by the concentration parameter (A smaller indicates higher degree of data heterogeneity.) For evaluation, we used two performance metrics:
Generic model evaluation: global test accuracy (same metric in conventional FL).
Personalized model evaluation: weighted average of local test accuracies.
For every client, the PM is evaluated on a local test set, whose underlying distribution is the same as implemented with full client participation
that for the local training set. All the experiments scheme. Further experiment details, results on partial client participation, and the computation overhead discussion are provided in the Appendix.
Remark 4.1. For generic FL methods, personalized model (PM) accuracies obtained by evaluating the generic model (GM) on local test sets. For PFL. methods without GM training, GM accuracies are obtained by evaluating the averaged PM the global test set.
Baselines. We compared our proposed method with the following SOTA PFI. methods: pFedMe 17 Ditto [15). FedRoD [36), FedRep [24], and FedBABU (30. Moreover, to have a fair performance evaluation of the generic models, we also include methods designed for conventional FL as baselines:
FedAvg 19) FedProx [IO, FedDyn [T,, FedGen (54, and FedAvgM [9].
4.2 Performance comparison with state-of-the-art methods
We evaluated the generalizability of our proposed spectral co-distillation framework, as well as the communication cost performance of our wait-free training protocol for PFL+.
Generalizability over heterogeneous settings. We compared the best test accuracies with mul- tiple baselines over the different levels of data heterogeneity, using the same system configura- tion. Tab. [ and Tab. 2 give the main results summary, our proposed framework achieves the best test accuracies across diverse heteroge- neous data settings, outperforming all PFL and conventional FL baselines on both PM and GM test accuracies concurrently. We also investigated the performance on the real-world dataset iNaturalist2017 in Tab. 3 where our proposed method also achieves the best GM/PM ac- curacies. We attribute such consistent outperformance to the bi-directional co-distillation design. This demonstrates that: a) the spectral information of the generic model useful for knowledge dis- tillation during personalized model training; and b) using truncated spectral information of the person- alized models could boost the per- formance of the generic model via careful spectrum truncation. (See Appendix for sensitivity analysis of the truncation ratio т and other hyper-parameters:
CIFAR-10 and CIFAR-100, respectively. In
3 epochs
epochs
Methods
40%
80%
40%
80%
Speedup
Ours (w/ WF)
1.82 x
1.56
2.21x
1.85x
Ditto w/ WF
1.97
1.38
2.87 x
1.93x
FedRoD w/ WF
1.75x
1.54
2.42x
2.19x
Table 4: Communication cost comparison of various methods for personalized model accuracies on CIFAR-10 reach tar- get accuracy (40%/80%) with non-IIID setting 0.1. The speedup factors are with respect to the performance of the corresponding methods without WE.
Communication cost compari- son. To demonstrate the superior- ity of the wait-free training protocol (WF), we evaluated the communica-

---

tion cost performance of SOTA methods with/without the protocol on non-lII CIFAR- (a = 0.1). in terms of the total runtime Suxal for PM to reach the target test accuracy (40%/80%) A smaller Stotal indicates higher communication efficiency. For PFL methods that train generic and personalized models using the compute-and-wey
local training protocol, we evaluated Ditto and FedRoD. We conduct experiments with different numbers of epochs for local PM training (3 or 5 epochs). As shown in Tab. 4 our proposed wait-free training protocol could significantly improve the efficiency of convergence time and has the potential to boost the time efficiency of PFL+ methods.
0.1
Method
GM
PM
GM
PM
Ours
86.37 + 0.15
92.25 + 0.21
75.52 + 0.11
82.69 + 0.16
Ours w/o SCD-GM
85.35 + 0.11
91.86 + 0.17
73.51 + 0.17
81.03 + 0.20
Ours w/o SCD-PM
82.74 0.39
79.65 = 0.83
68.96 + 0.47
70.51 = 1.21
Ours w/o Both
85.35 + 0.11
79.65 + 0.83
73.51 + 0.17
70.51 + 1.21
Table 5: Ablation study results non-IlI) CIFAR-11 10 (average and standard deviation of 3 trials). SCD-GM (resp. SCD-PM) represents the spectral distillation approaches adopted during the training of generic (resp. personalized) model.
4.3 Ablation results
Ablation study. In our proposed spectral co-distillation framework, we introduce the bi-directional spectrum knowledge distillation to bridge the training of generic and personalized models with the target for training good generic and personalized models simultaneously. To achieve the target, truncated and full model spectrum information are adopted in different training stages. Here, we conduct an ablation study to evaluate the effectiveness of these two components (see Tab, [o the effects of each component), in which we apply the distillation approaches in the two training stages separately. In the setup where both SCD-PM and SCD-GM removed (Case I). the GM training is identical to FedAvg. In the case of removing only SCD-PM while keeping SCD-GM (Case II), each PM would be trained locally without any knowledge distilled from the G.M. This is akin the client training its model by itself, separately from the server. Naturally, the PM performance would be drastically lower. As SCD-GM is kept in Case II, where the GM the student and the PM is the teacher, since the PM's performance is drastically lower, we would expect drop in the GM's performance. Informally, the model would be worse off with the distillation of bad knowledge, than without distillation.
As demonstrated in Tab. 5 both the distillation methods can boost the accuracy performance of generic and personalized models, whereas the bi-directional distillation can bridge the training performance of the generic and personalized models. Specifically, we can observe that, the SCD-PM module effectively transfers the knowledge from the generic model to the personalized model and avoids over-fitting during local training.
Generalizability on new join- ing clients. In real-world PFL system, dynamic client participa- tion should be regarded as an im- portant factor to consider during algorithm design, in which there would be continually new clients joining the system during train- ing. The PFL system needs to rapidly train good personalized model that could generalize well on the new client's local data. To evaluate the generalizability of the system, we simulate a dy- namic participation system with 80 in-training clients and new clients on CIFAR- 10 (partitioned
80
FedAug
Ditto
FedRoD
Test accuracy
Ours
70-
65-
60
55
5
10
Number of local epochs
Figure 3: Performance comparison for generalizability on new clients of various methods.

---

by the Dirichlet distribution with
0.1), and deal with new clients with the global model-based fine-tuning approach. Fig. gives the results of the average test accuracies of the new clients. Among all evaluated methods, our method has the best average test accuracies, illustrating the fast adaptive capability of our method.
Conclusion
In this work, propose spectral co-distillation framework for PFL to learn better generie and personalized models simultaneously. As far as we know, this is the first work in PFL that represents the (dis-)similarity of models via their Fourier spectra. Even without co-distillation, there have been no other works that explore spectral distillation in PFL (or even in FL). The advantage of this new approach is clear from our experiments: We achieved outperformance in both generic and personalized model training. Our framework also incorporates a simple yet effective wait-free local training protocol to reduce the overall local training time.
Limitations. Our proposed spectral co-distillation framework, as currently formulated, does not deal with stragglers and adversarial attacks. Their influence performance would require further investigation. Also, our protocol assumes a synchronized network connection, which may not be practical for scenarios with large system/network heterogeneity. Moreover, it would be good to consider a more realistic local training protocol design that takes into account the issues of network/system heterogeneity; we leave the extension as future work.
Acknowledgements
This work is supported by the National Research Foundation, Singapore under its AI Singapore Program (AISG Award No: LISG-RP-2019-011 and by Ministry of Education, Singapore, under its Tier 2 Research Fund (MOE-T2EP20221--05 111002221-016)) 1-0016), and under its SUTD Kickstarter Initiative (SKI 2021_03 (__0), in part by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Future Communications Research & Development Programme, in part by the National Natural Science Foundation of China under Grant 62271513 in part by the Zhejiang Lab Open Research Project (No. K2022PD0AB05), and in part by the Zhejiang-Singppor Innovation and AI Joint Research Lab.
References
[1] E. A. Van Dis, J. Bollen, W. Zuidema, R. van Rooij, and C, L. Bockting, "Chatgpt: five priorities for research," Nature, vol. 614, no. 7947, pp. 224-226 2023.
[2] E. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Y. Li, S. Wang, L.. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv preprint arXiv:2106.09685 2021.
[3] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, "Towards personalized federated learning," IEEE Trunsuctions on Neurul Networks and Learning Systems, 2022.
[4] J. P. Albrecht, "How the gdpr will change the world," Eur. Data Prot. L. Rev, vol. 2, p. 287, 2016.
[5] Y. Deng, M. M. Kamani, and M. Mahdavi, "Adaptive personalized federated learning," arXiv preprint arXiv:2003. /3461, 2020.
[6] J. Wang, Z. Charles, Z. Xu, G. Joshi, H. B. McMahan, M. Al-Shedivat, G. Andrew, S. Aves- timehr, K. Daly, D. Data, et al., "A field guide to federated optimization," arXiv preprint urXiv:2107009111 2021.
[7] J. Xu, 7. Chen, T. Q Quek, and K. F. E. Chong, "Fedcorr: Multi-stage federated learning for label noise correction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10184-10193, 2022.
[8] Q. Yang, Liu, T. Chen, and Y. Tong, "Federated machine learning: Concept and applications," " ACM Transactions Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1-19, 2019.
10

---

[9] T.-M. H. Hsu, H. Qi, and M. Brown, "Measuring the effects of non-identical data distribution for federated visual classification," in International Workshop on Federated Learning for Data Privacy in Conjunction with NeurIPS, 2019.
[10] Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," Proceedings of Machine Learning and Systems, vol. pp. 429-450, 2020.
[11] D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama, "Federated learning based dynamic regularization," in International Conference on Learning Represen- tations, 2020.
[12] Z. Chen, K. E. Chong, and T. Q- Quek, "Dynamic attention- -baee communication-efficiiii federated learning," in International Workshop Federated and Transfer Learning for Data Sparsity and Confidentiality Conjunction with IJCAI (FTL-IJCAI2021)) 2021.
[13] H. Tang, C. Yuan, Z. Li, and Tang, "Learning attention -uidded pyramidal features for few-shot fine-grained recognition," Pattern Recognit., vol. 130, 108792, 2022.
[14] Z. Chen, S. Liu, H. Wang, H. H. Yang, T. Q. Quek, and Z. Liu, "Towards federated long-tailed learning," " in International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 (FL-IJCAI 22), 2022.
[15] T. Li. S. Hu, A. Beirami, and V. Smith, "Ditto: Fair and robust federated learning through personalization," in International Conference Machine Learning, pp. 6357-6368, PMLR, 2021.
[16] A. Li, Sun, X. Zeng, M. Zhang, H. Li, and Chen, "Fedmask: Joint computation and communication-efficii personalized federated learning via heterogeneous masking," in Pro- ceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 42-55, 2021.
[17] C. T Dinh, N. Tran, and J. Nguyen, "Personalized federated learning with moreau envelopes," Advances in Neural Information Processing Systems, vol. 33, pp. 21394-21405, 2020.
[18] D. Chen, D. Gao, W. Kuang, Y. Li, and B. Ding. "pfl-bench: A comprehensive benchmark for personalized federated learning," Advances in Neural Information Processing Systems, vol. 35, pp. 9344-9360, 2022.
[19] B. MeMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, "Communication-effici" learning of deep networks from decentralized data," in Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017.
[20] P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, "Think locally, act globally: Federated learning with local and global representations." in International Workshop Federated Learning for Data Privacy in Conjunction with NeurIPS. 2019.
[21] Y. Shen, Y. Zhou, and L. Yu, "Cd2-pfed: Cyclic distillation distillation-guided channel decoupling for model personalization in federated learning," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10041-10050, 2022.
[22] M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary, "Federated learning with personalization layers," arXiv preprint arXiv: 1rXiv:1912.008188 2019.
[23] H.-Y. Chen and W.-L. Chao, "On bridging generic and personalized federated learning for
image classification." in International Conference on Learning Representationss 2022.
[24] L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, "Exploiting shared representations for personalized federated learning," in International Conference on Machine Learning, pp. 2089- 2099, PMLR, 2021.
[25] K. Pillutla, K. Malik, A.-R. Mohamed, M. Rabbat, M. Sanjabi, and L.. Xiao, "Federated learning with partial model personalization," in International Conference on Machine Learning, pp. 177558 PMLR, 2022.

---

[26] J. Y. Zhang, and T. Luo, "Overview frequency principle/spectral bias in deep learning." arXiv preprint arXiv:2201.073955 2022.
[27] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville, "On the spectral bias of neural networks," in International Conference on Machine Learning. pp. op.5301-35 5301-5310, PMLR, 2019.
[28] S. Fridovich-Keil. R. Gontijo Lopes, and R. Roclofs, "Spectral bias in practice: The role of function frequency in generalization," Advances in Neural Information Processing Systems, vol. 35, pp. 7368-7382. 2022.
[29] F. Hanzely and P. Richtárik, "Federated learning of a mixture of global and local models," arXiv preprint arXiv:2002.055 16, 2020.
[30] J. Oh, S. Kim, and S.-Y. Yun, "FedBABU: Toward enhanced representation for federated image classification," in International Conference on Learning Representations. 2022.
[31] J. Xu, X. Tong, and S.-L. Huang, "Personalized federated learning with feature alignment and classifier collaboration." in The Eleventh International Conference on Learning Representations, 2023.
[32] A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, "Lotteryfl: Empower edge intelli- gence with personalized and communication-efficin federated learning," in 2021 IEEE/ACM Symposium on Edge Computing (SEC), pp. 68-79, IEFE, 2021.
[33] M. Setayesh, X. Li, and V. W. Wong. "Perfedmask: Personalized federated learning with opti- mized masking vectors," in The Eleventh International Conference on Learning Representations, 2023.
[34] Y. Jiang, Konečný, K. Rush, and Kannan, "Improving federated learning personalization via model agnostic meta learning," arXiv preprint arXiv: 12488, 2019.
[35] A. Fallah, A. Mokhtari, and A. Ozdaglar, "Personalized federated learning with theoretical guar- antees: A model-agnostic meta-learning approach," Advances in Neural Information Processing Systems, vol. 33, pp. 3557-3568, 2020.
[36] I. Achituve, A. Shamsian, A. Navon, G. Chechik, and E. Fetaya, "Personalized federated learning with gaussian processes," " Advances Neural Information Processing Systems, vol. 34, pp. -992-8406, 2021.
[37] A. Shamsian, A. Navon, E. Fetaya, and G. Chechik, "Personalized federated learning using hypernetworks," in International Conference on Machine Learning, pp. 9489-9502, PMLR, 2021.
[38] F. Sattler, K.-R. Müller, and W. Samek, "Clustered federated learning: Model-agnostic dis- tributed multitask optimization under privacy constraints," IEEE transactions on neural networks and learning systems, vol. 32, no. 8, pp. 3710-3722, 2020.
[39] M. Duan, D. Liu, X. Ji, R. Liu, Liang, X. Chen, and Y. Tan, "Fedgroup: Efficient federated learning via decomposed similarity based clustering," in 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking pp. 228-237, IEEE, 2021.
[40] A. Ghosh, J. Chung, D. Yin, and K. Ramchandran, "An efficient framework for clustered federated learning," A Advances in Neural Information Processing Systems, vol. 33, pp. 19586- 19597, 2020.
[41] W.-C Chen, C.-C Chang, and C.-R. Lee, "Knowledge distillation with feature maps for image classification," in Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pp. 200-215, Springer, 2019.
12

---

[42] B. Qian, Y. Wang, H. Yin, R. Hong, and M. Wang. "Switchable online knowledge distillation." " in Computer Vision-ECC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pp. 1494466, Springer, 2022.
[43] D. Li and J. Wang, "Fedmd: Heterogenous federated learning via model distillation," arXiv preprint arXiv:1910.03330 2019.
[44] T. Lin, L. Kong, S. U. Stich, and M. Jaggi, "Ensemble distillation for robust model fusion in federated learning," Advances in Neural Information Processing Systems, vol. 33, pp. 2351- 2363, 2020.
[45] Y. J. Cho, J. Wang, T. Chirvolu, and G. Joshi, "Communication--ffcciai and model- heterogeneous personalized federated learning via clustered knowledge transfer," IEEE Journal of Selected Topics in Signal Processing, 2023.
[46] C. He, M. Annavaram, and S. Avestimehr, "Group knowledge transfer: Federated learning of large cnns at the edge," Advances in Neural Information Processing Systems, vol. 33, pp. 14068- 14080, 2020.
[47] J. Nguyen, K. Malik. H. Zhan, A. Yousefpour, M. Rabbat, M. Malek, and D. Huba, "Federated learning with buffered asynchronous aggregation," in International Conference on Artificial Intelligence and Statistics, pp. 3581-3607, PMLR, 2022.
[48] L. Zhu, H. Lin, Y. Lu, Y. Lin, and S. Han, "Delayed gradient averaging: Tolerate the commu- nication latency for federated learning," Advances in Neural Information Processing Systems, vol. 34, pp. 29995-30007, 2021.
[49] M. Bornstein, Rabbani, Z. Wang. A. Bedi, and F. Huang, "SWIFT: Rapid decentralized federated learning via wait-free model communication," in Workshop Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurlPS 2022), 2022.
[50] A. Akbari, M. Awais, M. Bashar, and Kittler, "How does loss function affect generaliza- tion performance of deep learning? application to human age estimation," in International Conference on Machine Learning. pp. 141-151, PMLR, 2021.
[51] A. Krizhevsky, G. Hinton, et al., "Learning multiple layers of features from tiny images," " 2009.
[52] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[53] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and Belongie, "The inaturalist species classification and detection dataset," Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3769-8778 2018.
[54] Z. Zhu, J. Hong, and J. Zhou, "Data-free knowledge distillation for heterogencous federated learning," in International conference on machine learning, pp. 12878 12889, PMLR, 2021.
[55] J. Luo, M. Mendieta, C. Chen, and S. Wu, "Pgfed: Personalize each client's global objective for federated learning," arXiv preprint arXiv:2212.01448. 2022.
[56] E. O. Brigham, The fast Fourier transform and its applications. Prentice-Hall, Inc., 1988.
[57] C. Xie, S. Koyejo, and Gupta, "Asynchronous federated optimization," arXiv preprint arXiv: arXiv:1903.033: 1903.03934, 2019.
[58] J. Nguyen, K. Malik. H. Zhan, A. Yousefpour, M. Rabbat, M. Malek, and D. Huba, "Federated learning with buffered asynchronous aggregation," in International Conference on Artificial Intelligence and Statistics, pp. 3581-3607, PMLR, 2022.
13

---

A Appendix
In the appendix, we provide further details organized into the following parts:
Sec. Bec.BBing introduces the implementation details for our proposed method with respect to the
federated system settings, evaluation metrics, and the data partitions used in our experiments. In particular, we provide the implementation details for the experiments in "Generalizability
new joining clients".
Sec. C
gives further sensitivity analysis on the key parameters т, Àg. and Ap-
Sec. gives discussion on the computational overhead.
Sec. F
gives discussion on the differences and relations to asynchronized (personalized)
federated learning (FL).
Sec. [F] a discussion on how our wait-free local training protocol could be adapted for
partial client participation.
B Implementation details
Federated system settings. All experiments were implemented using Pytorch. For all methods, we used an SGD local optimizer with momentum 0.5 and with no weight decay. We train all methods over a total number of T 500 global communication rounds. Batch size for 1IFAR-10/100 [51] and NNtturalist-2017 [33] are 10 and 128, respectively. As our aim is to train a personalized model for each client, we conducted the training using a full participation scheme, i.e., in every communication round, all clients would be selected to participate in training. For our proposed method, we used learning rate of 0.01 (resp. 0.003) for both ya and 1/p when training on CIFAR-10/100 (resp. iNaturalist-2011 2017).
Details on the data partition. We evaluated the performance of all methods with non-IID data partition. We simulated a non-lID data distribution via the symmetric Dirichlet distribution, in which we used the concentration parameter to control data heterogeneity across the FL system [9] M. For the entire dataset D with M classes, we sample a stochastic vector d, with length N (total number of clients) for each 1 S < M via the Dirichlet distribution with common parameter a. We shall treat d, as the vector representing the portion of the data samples in class j to be allocated to the clients. For example, if the second entry of d equals 0.05, then client 2 would be randomly allocated with 5% of the data samples from D in class 1. This allocation has no overlaps; two different clients have common data samples. In particular, smaller a indicates a higher degree of data heterogeneity, whereas a larger a will tend to generate identical-lik distributions. (As a + +oc, the data allocations become uniform.)
Performance metries. The detailed calculation methods for the two metries used in the evaluation are given as follows:
We used the global test accuracy (same metric used in conventional FL) to evaluate the collaboratively trained global generic model, where the accuracy refers to the global test set (e.g., for CIFAR-IO, the test set has 10,000 images).
We used the weighted average of local test accuracies for personalized model evaluations, where each local test accuracy is obtained with respect to a local test set, and where the weights of the weighted average are the ratios of local dataset sizes R1 22 n ...... nx Let acc; denote the test accuracy for the personalized model at client i.Then the weighted average of 7 local test accuracies is computed as Σi= ay acc; Note that for each client, the distribution
of the local test set for each client is the same as the distribution of the local training set. Specifically, we perform sampling for the local test sets according to the distribution of training data for each respective client, where sampling is performed without replacement on the test set of the corresponding benchmark dataset, which obtained via the Dirichlet distribution.
Details for the experiments on new joining clients. To evaluate the generalizability of our proposed federated co co-distillation framework when there are new joining clients, we simulated PFL system with 80 existing clients and 20 new clients on the CIFAR-10 dataset with non-IID data partition. In particular, we first partitioned the dataset into 100 local training sets via the Dirichlet distribution. We then performed training for the pre-determined 80 clients over 300 communication rounds for all methods, after which conducted local fine-tuning [55] 23 on the new clients independently with
14

---

certain number of local epochs based on the well-trained global generic model. (We tried 5 and 10 local epochs in our experiments.) The reported performance is the weighted average of the local test accuracies over the 20 new clients, in which the weights are the ratios of the dataset sizes relative to these 20 clients.
C Sensitivity analysis
Recall from Sec. 3.3 of the main paper that is the truncation ratio used to compute the truncated spectrum vectors. In our experiment results (as reported the main paper), we used т = 0.4. To conduct sensitivity analysis with multiple
demonstrate the effect of this key hyperparameter, values of 0 < т S on two non-IID CIFAR-10 settings; see Tab. 6 spectral distillation for generic model training (which is equivalent to the setup "Ours w/o SCD-GM" in Tab. 5 of the ablation studies in the main paper ), while 7 = indicates there is no spectrum truncation. In summary, personalized models (PM), which shows the effectiveness of the truncated spectrum in generic model training and implicitly demonstrates the characteristies of the low frequency of model Fourier spectrum.
represents there is no
0.4 achieves the best performance for both generic model (GM) and
Method
a
0.1
GM
PM
GM
PM
T 0
85.35 + 0.11
91.86 + 0.17
73.51 + 0.17
81.03 + 0.20
0.1
85.18 + 0.26
+ +
72.66 + 0.11
80.51 + 0.25
0.3
85.42 0.22
72.59 0.38
82.60 + 0.19
T 0.4
86.37 + 0.15
92.25
75.52 0.11
82.69 + 0.16
T 0.5
86.33 0.08
92.04 0.29
74.91 0.31
82.50 + 0.36
T = 0.7
86.08 0.43
91.79 0.44
72.85 0.28
81.44 + 0.43
T 0.9
86.25 + 0.39
92.07 + 0.28
72.57 + 0.31
82.09 + 0.37
T 1.0
85.47 + 0.26
90.33 + 0.41
73.55 0.17
81.36 + 0.22
Table 6: Sensitivity analysis for hyper-parameter (i.c., the truncation ratio for spectrum truncation, used generic model training) on non-IID CIFAR-10 (average and standard deviation of 3 trials).
We also provide experiment results for the sensitivity analysis of the coefficients A and Ap on CIFAR-10 10 with data setting a = 1 in Tab. and Tab. g
Setup
0.01
0.05
0.1
0.3
GM
84.76
86.37
86.20
83.55
PM
90.33
92.25
91.35
89.64
Table 7: Sensitivity analysis for hyper-parameter Ag on non-llI CIFAR-10.
Sctup
0.01
0.05
0.1
0.3
GM
86.37
86.33
85.61
85.72
PM
92.25
91.89
89.47
89.33
Table Sensitivity analysis for hyper-parameter Ap on non-IID CIFAR-10.
D Computational overhead of the proposed method
In our proposed method, the additional computational overhead is dominated by the computation of spectral vectors, when compared to other model similarityldiffernee-eess PFL methods 17]. In our experiments, we use the famous fast Fourier transform (FFT) algorithm to compute the discrete Fourier transform of an input (complex) vector. Informally, the FFT algorithm is efficient and
15

---

widely used algorithm that converts an input signal from the original domain into representation in the frequency domain.
In our framework, when given an input weight vector "f. E Rd (i.e. there are a total of model weights), its associated spectral vector s(w is computed using the FFT algorithm, with correspond- ing computational complexity (dd log d); see, e.g., [66]
E Discussion on the differrences/relations to asynchronized FL
Our main goal for proposing the wait-free PFL.+ training protocol is to reduce the total running time overhead and thereby increase the training time efficiency. It should be noted that wait-free local training protocol is synchronous, meaning that all the clients perform local computing, pushing generic model gradients, well as generic model updates, our proposed protocol allows local computing during global communication.
synchronized manner. Similar (48]
In similar context of reducing the total running time overhead, some asynchronized FL (AFL) methods have been proposed. We shall clarify that our wait-free local training protocol is vastly different from existing AFL methods, from the aspects of both the motivation and the communication computation design. Specifically, AFL. seeks to tackle the challenge of slow client in a heterogeneous FL system by reducing idle time, while in contrast, computation design, therehy eliminating idle time. In particular, the usual idle waiting time has been replaced by training time for the personalized model in our proposed method. Since method fully utilizes the idle waiting time at the local side with personalized model training, we did not consider incorporating the conventional AFL set-up into our framework.
method adopts joint communication-
To better demonstrate the outperformance our wait-free (WF) local training protocol, we evaluated the performance of two tate-of-the--o art AFL methods (FedAsync [57] and FedBuff [58] combined with the compute-and-aa wait PFL+ system using Ditto [15] (the PFL system that trains generic and personalized models simultaneously); see Tab. The speedup factors are computed with respect to the performance of "Vanilla Ditto", where a larger speedup factor signifies better efficiency. The results demonstrate the outperformance of our proposed WF protocol in reducing the total running time overhead.
3 epochs
5 epochs
Methods
40%
80%
40%
80%
Speedup
Vanilla Ditto
1x
1x
1x
1x
Ditto w/ Our WF
1.97
1.38 x
2.87x
1.93x
Ditto w/ FedAsync
1.36
1.15 x
1.29x
1.22x
Ditto w/ FedBuff
1.40
1.26 x
1.23x
1.40x
Table Communication cost comparison of various methods for personalized model accuracies on CIFAR-10 10 to reach target accuracy (40%/80%) with non-lll) setting a = 0.1. The speedup factors are with respect to the performance of the Vanilla Ditto with the compute-and protocol.
Discussion on the applicability to the partial client participation scheme
In this work, we consider full client participation scheme. However, in general, we may want the option that an edge device/client could be not selected to participate in training for numerous consecutive communication rounds, especially in the case of an FL system with a large number of clients. In this subsection, we discuss how our wait-free local training protocol should be extended/modiffed.
For an unselected client, the client should not be continuously training its personalized model using the "stale" GM model weights, which could result in model overfitting on limited data, and model drift/divergence.
To address this issue, there are two straightforward solutions:
16

---

The first solution is to adopt broadcasting scheme with regular latest GM broadcasts by the server. Such a scheme would ensure that every client receives the latest GM weights. thereby alleviating the negative impact of stale GM weights for unselected clients, and avoiding potential client drift. In real-world network connections, the bandwidth of the downlink connection from the server to the clients is typically larger than the corresponding bandwidth the uplink connection. Hence, such design would be practical.
The second solution is to set upper limit on the number of local training steps. The basic underlying idea is to terminate the local PM training after a certain number local training steps. This could be an alternative option address the issue of limited communication for cross-device FL scenarios. should be noted that such design would still reduce the overall total training time, since the PM training is still performed within the idle waiting time in conventional FI..
We give further experiment results (see below) for partial participation with two typical participation rates (c=0.1 and 0.2) on non-IID CIFAR-10 10 with a in Tab. TO
c00.1
0=0.2
Methods
GM
PM
GM
PM
FedAvg
83.16
78.25
83.84
79.03
FedProx
82.97
78.23
83.75
79.44
FedRoD
84.22
89.31
84.90
89.68
Ours
84.35
89.56
85.11
90.37
Table 10: Sensitivity analysis for hyper-parameter No on non-IID CIFAR-10. 10.
These results show that our method is applicable for partial participation, and even outperforms our baselines.
17


Reviews:
[{"review_id": "O0YPbDBCO7", "reviewer": "Reviewer_8564", "summary": "1. This paper proposes a new personal FL methodology: FedSCD. The authors facilitates the generic and personal models to capture global/local distributions through spectral distillation. A co-distillation framework is introduced to help generic and personal models to learn from each other. From the Fourier analysis, since the DNN tends to learn the low frequency features before the high frequency features, the feature level in model representation can be obtained from Fourier spectrum. According to the author, the global model can learn from some low-level features that are commonly shared among diverse local models, and the averaged high level features can benefit the personalized model. \n2. Instead of regularizing the local model not to diverge too much from the global model, the spectrum distillation minimizes the loss from the KLDivergence between the spectrum vector of generic and personal model. \n3. The paper also designs a wait-free local training protocol that aims to improve the training efficacy by paralleling the process of server aggregation and local personalized model training.", "questions": "In Table 5, the method without both SCD-PM and SCD-GM share same GM acc with method without SCD-GM and same PM without SCD-PM. But intuitively, since the training scheme is different, the acc should be different. Can you explain this?\nPlease also refer to the Weakness section.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea that utilizing Fourier spectrum information to seek for common global model architecture is novel and properly aligns with the personalization Federated Learning scheme\n2. The distillation process, compared to other knowledge distillation methods, does not require public data access on server side, which is more applicable to real-world setting.", "weaknesses": "[Methodology Setup and Justification]\n1. Co-spectrum distillation design.\n\nThe design of personal model spectrum distillation is somehow counter-intuitive. In line 56-58, the author says “Dually, the entire Fourier spectrum of the generic model, which includes the \"averaged\" high-level features across all clients, would benefit the training of the personalized models.” If the low frequency feature is shared among all clients and should be learned by the generic model, the personalized models should have different high frequency features since each clients have distinct data distribution/scenario that they need to adapt on. Then how can the averaged high-leave features benefit client personalization? \n\nHowever, empirically, the spectrum distillation on personal model greatly improves the performance, should such improvement be credited from the fact that during the spectrum distillation, the commonly shared low frequency features in personal model generalized better? So the improvement is actually from better trained low frequency features instead of averaged high-level features? I would like to see author provides more theoretical adjustments on this. \n\n2. Value of Wait-free protocol.\n\nThe wait-free protocol can be easily implemented under personal FL framework and effectively reduce the communication cost, but it is off-track from the main topic of the paper: co-spectrum distillation. Also, the effectiveness of the proposed wait-free protocol is mainly based on the assumption of all clients share similar training speed. If the fastest client and slowest client has large gap in training speed, the fastest client still need wait for the slowest client. The problem of heterogeneous hardware/computation power is also under explore, and many works have made contribution on this topic[1]. If reducing the communication cost is also a main focus of the paper, the wait-free protocol needs better design to become a new methodology.\n\n3. Missing Fourier Spectrum Analysis in Neural Networks in Related Work.\n\nSpectrum analysis of neural networks is apparently a key component in the proposed methodology, but not everyone is familiar with this topic. It will be nice to include a short description of Fourier spectrum in neural networks.\n\n---\n\n[Experiment Setting]\nOverall, the experiment needs better simulation to mimic real settings. The flaws of experiment are below:\n1. client participation rate.\n\nFor all experiment, all clients participate in the aggregation in each round, which is not real in either cross-device or cross-silo scenarios. It is necessary to also show the performance under different client participation rates. \n\n2. Different number of clients in different datasets.\n\nIn CIFAR10/CIFAR100/iNaturalist-2017, the client number are set to be different(100/50/20) without explanation. More justification on what kind of setting(i.e. cross-silo or cross device) are these experiments trying to mimic on is needed. \n\n3. Other KD related baselines.\n\nSince FedSCD borrows the idea of knowledge distillation, the author should also consider to include baselines also incorporate KD such as  FedKD[2] and FedMD[3]\n\n4. The different non-iid setting on different dataset.\n\nIn Table 1, experiment with CIFAR-10, the author shows the result of Dirichlet distribution with alpha = 0.1, 0.5, 1. However, in Table 2, experiment with CIFAR-100, only alpha = 0.1 and 1 are shown. In Table 3, experiment with iNaturalist-2017, only alpha = 0.1 is shown. I would like to see the authors to also make the heterogeneous settings in CIFAR-100 and iNaturalist-2017 as diverse as the setting in CIFAR-10.\n\n---\n\n\n[Theoretical Analysis]\n1. Convergence Analysis of spectrum distillation in FL.\n\nSpectrum analysis itself has a strong theoretical cornerstone, and at the same time, the methodology of this paper is a significant alteration of traditional FL practice. I would look forward to seeing the authors do some analysis or discussion of the theoretical aspects of the effectiveness of FedSCD, such as the rate of convergence.\n\n2. More intuitive explanation of the accuracy drop in table 5.\n\nAfter removing the PM-SCD in ablation study, although the GM-SCD is applied, the overall performance drops compared to naive setting where both PM-SCD and GM-SCD are removed. It will be nice to provide some intuitive explanation of such accuracy drop. \n\n---\nReference:\n1. SWIFT: RAPID DECENTRALIZED FEDERATED LEARNING VIA WAIT-FREE MODEL COMMUNICATION\n\n2. FedKD: Communication Efficient Federated Learning via Knowledge Distillation\n\n3. FedMD: Heterogenous federated learning via model distillation\n\n\n========================Score Modification=======================\n\nThe author's supplementary experiments and subsequent responses after the rebuttal phase have addressed a number of my concerns. Consequently, I am inclined to elevate my score to a 5. Nonetheless, I still would like to urge the author the author to fully implement the pledged amendments and conduct comprehensive additional experiments for the next versions. Also, I recommend giving further attention to the other issues that I raised yet remain unaddressed such as the theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and suggestions.\n> Why spectral co-distillation is not counter-intuitive\n\nIn existing PFL works, a common starting point is that the generic model (GM) and personalized models (PMs) share the same low-level features, while the high-level features are assumed to be more client-specific. In such works, the low/high-level features are associated to different parts of the model. It is natural to only use low-level features for GM training, since after all the low-level features are shared, while high-level features correspond to personalization. In our work, we adopt the same starting point but instead relate low/high-level features with low/high-frequency components of the spectrum, which is well-justified by existing theory on spectral bias. Thus our work is consistent with the general PFL intuition.\n\nAlthough high-frequency components correspond to personalization, the averaged high-frequency components would still play the role of regularization. This is crucial for minimizing the risk of overfitting PMs with limited data. \n\nWe also performed an ablation study to understand the effect of partial (low-frequency only, high-frequency only) versus full spectrum for SCD-PM, on CIFAR-10 with $\\alpha=1$.  \n\n| Setup | Low only | High only | Full  |\n| ----- | -------- | :-------- | ----- |\n| Acc   | 90.23    | 91.05     | 92.25 |\n\nThese results show the efficacy of our SCD-PM design.\n\n> Wait-free protocol\n\nAs pointed out, the wait-free protocol is not our main focus. However, we still choose to include it, since it is a natural and easy extension that uses the two distillation stages in our co-distillation framework. As highlighted by Reviewer rUNY as a strength, our protocol is a general method that can be adopted by other PFL methods. We thank the reviewer for highlighting SWIFT. In Sec. 3.4 of our paper, we have cited SWIFT.\n\n>Describing Fourier spectrum for unfamiliar readers\n\nWe thank the reviewer for the suggestion. We plan to build on lines 42-48 and give further elaboration in our revision.\n\n> Partial client participation\n\nTo address the reviewer's concern, we give further results (see below) for partial participation with two typical participation rates ($c$=0.1 and 0.2) on CIFAR-10 with $\\alpha=1$. \n\nMethods   $c$=0.1   $c$=0.2  \nSetup       GM   PM   GM   PM  \nFedAvg    83.16   78.25   83.84   79.03   \nFedProx   82.97   78.23   83.75   79.44  \nFedRoD   84.22    89.31   84.90   89.68  \nOurs        84.35   89.56   85.11    90.37\n\nThese results clearly show that our method is applicable for partial participation, and even outperforms our baselines. We shall give an analysis in our revision.\n\n> Clarification on experiment setup\n\nWe chose different numbers of clients $N$ on different datasets to show that our method generalizes well on diverse system scales, including both cross-device ($N=100$) and cross-silo ($N=20$) FL settings.\n\n\n> Other KD-related baselines\n\nWe agree that incorporating KD-related baselines would be good. However, for both suggested methods (FedKD, FedMD), we believe they are not suitable. \n\nFedKD is evaluated only on NLP tasks, and their code needs major modifications before it can be applied to the CV tasks for our experiments. FedMD requires public data, thus it is not a fair choice for other baselines without this requirement.\nTo address the concern, we instead choose FedGen [1] as an additional KD-related baseline.\n\nA comparison of CIFAR-10 is given below. We shall give an analysis in our revision.\n|        | $\\alpha=1$ |       | $0.5$ |       |\n| :----- | ---------- | ----- | ------------ | ----- |\n|        | GM         | PM    | GM           | PM    |\n| FedGen | 86.24      | 85.79 | 80.86        | 79.14 |\n| Ours   | 86.37      | 92.25 | 81.27        | 86.59 |\n\n[1] Data-Free Knowledge Distillation for Heterogeneous Federated Learning\n\n> Diverse non-IID settings\n\nTo enhance the demonstration of our outperformance in diverse non-IID settings, we performed further experiments on FEMNIST. Overall, this means our method outperforms the baselines over **4** datasets, 2 of which involve real-world settings. Due to limited time, we are unable to perform cost-intensive experiments on iNaturalist as requested. For results on FEMNIST, see table in response to Reviewer rUNY.\n\n> Theoretical analysis\n\nWe agree that further theoretical analysis would benefit our work. We are encouraged that the reviewer thinks our approach is a significant alteration of FL practice. Since our experiments show significant improvement over baselines adopting the traditional PFL paradigm, we believe our work would attract further attention, not just for the development of new PFL methods, but also for further theoretical interest in the role of spectral information in PFL.\n\n> Clarification for ablation study\n\nIn the setup where both SCD-PM and SCD-GM are removed (Case i), the GM training is identical to FedAvg. In the case of removing only SCD-PM  while keeping SCD-GM (Case ii), each PM would be trained locally without any knowledge distilled from the GM. This is akin to the client training its model by itself, separately from the server. Naturally, the PM performance would be drastically lower. \n\nAs SCD-GM is kept in Case ii, where the GM is the student and the PM is the teacher, since the PM's performance is drastically lower, we would expect a drop in the GM's performance. Informally, the model would be worse off with the distillation of bad knowledge, than without distillation. \n\nWe shall give a detailed clarification in our revision.\n\n> Clarification for ablation study II\n\nIt is not true that the training schemes are different.\n\nWithout SCD-PM, each PM will be trained only by itself, separately from the server. Thus, with or without SCD-GM, the PM's training is exactly the same. So we expect the same accuracy. Analogously, without SCD-GM, the GM's training is identical to FedAvg. Thus, with or without SCD-PM, we expect the same accuracy for GM."}, {"review_id": "87fxN8k6J9", "reviewer": "Reviewer_FUrR", "summary": "This paper introduces a spectrum based co-distillation for personalized federated learning; which involves the local model training with regularization/distillation with respect to global parameters's spectral representation and the global model training with regularization/distillation with respect to local parameters's spectral representation. ", "questions": "1. Can you highlight both the winner and the second in Table 1? When $\\alpha=0.5$, the winner should be FedRod? \n\n2. Why other methods should be affected by $\\alpha$ which is a parameter in your model?\n\n3. What are the results if you don't employ spectral based distillation? ", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Personalized FL is an important and timely topic\n2. The approach using spectral transformation for distillation is novel and interesting \n3. The experimental approaches show the effectiveness of the proposed approach ", "weaknesses": "1. The technical contribution is relatively low \n2. The experimental results used only two datasets, not very convincing. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments.\n\n> The technical contribution is relatively low\n\nWe have the following novel technical contributions, which have also been pointed by all the other reviewers.\n\n1. We are the first to characterize model (dis-)similarity of personalized models (PMs) versus the generic model (GM) in PFL via fourier spectra of models. This provides a new perspective for modelling the relationship between the PMs and GM.\n\n2. We propose a co-distillation framework to bridge the training of the GM and PMs in a two-way manner.\n\n3. We proposed a wait-free training protocol to reduce the total running time in PFL, which could also be easily incorporated into other PFL frameworks.\n\n\n> The experimental results used only two datasets, not very convincing.\n\nFor our experiments, we evaluated the performance of our method over **three** different datasets (**not two**), CIFAR-10 in Table 1, CIFAR-100 in Table 2, and iNaturalist-2017 in Table 3, and with different heterogeneous data settings to demonstrate the efficacy and outperformance. Moreover, in this rebuttal, we have provided further experiment results on an additional 4th dataset FEMNIST. See response to Reviewer rUNY.\n\n\n\n> Can you highlight both the winner and the second in Table 1? When $\\alpha=0.5$, the winner should be FedRod?\n\nWe thank the reviewer for this comment. We will highlight the top 2 accuracies for Table 1 in our revision. \nWe apologize for the typo for the highest GM accuracy under $\\alpha=0.5$ in Table 1; this should be FedRoD.\n\n\n\n> Why other methods should be affected by $\\alpha$ which is a parameter in your model?\n\n\nThe parameter $\\alpha$, which is used throughout all experiments, is the concentration parameter for the symmetric Dirichlet distribution, used to control data heterogeneity across the clients in the FL system. Full description and details can be found in lines 224-227 of the main paper, and also in lines 20-30 of the supplementary material. Note that different $\\alpha$ values would correspond to different levels of data heterogeneity. There is extensive empirical evidence in the FL literature that a higher level of data heterogeneity would result in a higher degree of performance degradation. Therefore, all the methods would be affected by $\\alpha$.\n\n\n\n> What are the results if you don't employ spectral based distillation?\n\nPlease see **Table 5** of our paper. In this table, we provide an ablation study that includes the case of not using spectral distillation. Specifically, we show that, without spectral distillation at the GM and/or without spectral distillation at the PMs, there is a drop in performance.\nHence, spectral distillation (both for GM and PMs) is effective and crucial for the outperformance of our method over the baselines."}, {"review_id": "wOpnGtBBmL", "reviewer": "Reviewer_rUNY", "summary": "This paper proposes a method for PFL, which aims to address the challenge of data heterogeneity. The main idea is to use spectral distillation to capture the relations between the generic global and personalized local models based on the similarity of spectral information. The paper introduces a co-distillation framework that establishes a bi-directional bridge between generic and personalized model training and a wait-free local training protocol that utilizes the local idle time in conventional PFL. The paper evaluates the proposed method on two datasets with heterogeneous settings and shows that it outperforms other existing PFL methods.", "questions": "Please refer to the Weaknesses section, and address the issues raised there. I will re-evaluate my assessment of this paper if these Weaknesses are resolved.", "limitations": "The author discusses the limitation of this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper presents an interesting idea of using spectral distillation for PFL to share low-level features, which differs from existing methods relying on model weight similarity. The paper provides some intuition on why spectral distillation can better capture the generic versus personalized model representations. Furthermore, the paper proposes a wait-free local training protocol that can improve the communication efficiency of PFL, which is a general technique that can be adopted by other PFL methods.", "weaknesses": "1. The clarity of this paper should be improved. Please be careful with using uncommon abbreviations, or provide explanations where appropriate. GM and PM should be clarified in the Figure 1 caption; in Line 232, SM should be spelled out as Supplementary Material; In Table 4, what do 3/5 epochs mean?  I guess it is the local training steps.\n\n2. Experiment settings need to be improved. The evaluation of PM and GM is not comparable, as they are completely different sets. It would be better to provide additional results of PM and GM on the same test set, such as the global test set. Moreover, the datasets used in the experiments are obtained by splitting with Dirichlet distribution, which are not real FL datasets. Please consider following the experimental setup in pFL-Bench [1], using datasets like FEMNIST [2], and reporting more fairness-related metrics, such as standard deviation.\n\n3. The reproducibility of the experiments is questionable. I could not find how the local test set was generated in CIFAR-10, such as the train/test ratio, in the main text or the supplementary material.\n\n[1] pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning, Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Datasets and Benchmarks Track\n\n[2] LEAF: A Benchmark for Federated Settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive feedback. Below we address the highlighted weakness point-by-point.\n\n> The clarity of this paper should be improved. Please be careful with using uncommon abbreviations, or provide explanations where appropriate. GM and PM should be clarified in the Figure 1 caption; in Line 232, SM should be spelled out as Supplementary Material; In Table 4, what do 3/5 epochs mean? I guess it is the local training steps.\n\nWe thank the reviewer for the suggestion to improve clarity. We will update the caption of Figure 1 with an explanation of the abbreviations (PM and GM). Our proposed new caption is given as follows:\n\n\"Spectral co-distillation framework with wait-free local training for PFL, in which the generic model (GM) training and the personalized model (PM) training are carried out via spectral distillation in two different stages.''\n\nWe have checked that there are only two instances of the abbreviation SM in our paper.\nHence, we shall replace both instances with the full wording \"supplementary material''.\n\nRegarding \"3 or 5 epochs'' in Table 4, they both refer to the number of epochs for local PM training. We apologize for the missing explanation.\n\n\n\n> Experiment settings need to be improved. The evaluation of PM and GM is not comparable, as they are completely different sets. It would be better to provide additional results of PM and GM on the same test set, such as the global test set. Moreover, the datasets used in the experiments are obtained by splitting with Dirichlet distribution, which are not real FL datasets. Please consider following the experimental setup in pFL-Bench [1], using datasets like FEMNIST [2], and reporting more fairness-related metrics, such as standard deviation.\n\nWe thank the reviewer for valuable comments on how our experiment setup could be improved. We first clarify that in the PFL+ scenario (see line 130 in main paper), which is the scope of our paper, the GM and PMs do not just have different training sets, but they also have different evaluation setups.  \nFor the results in Tables 1--3, we compare the results of different methods in a single column, rather than directly comparing them across GM and PMs.\nIn particular, the evaluation of each PM is based on \"personalized\" data distribution. This means that testing all PMs on the same global test set would change the scope of our PFL+ work.\nMoreover, for the test set design, we adopted a setup that is identical to most PFL+ works [1] , in which each PM is evaluated on a local test set, whose underlying distribution is the same as that for the local training set.\n\nWe thank the reviewer very much for pointing out the relevance of pFL-Bench [2] and FEMNIST in LEAF [3]. In our revision, we will cite both papers and provide further discussion on them. As requested, we have conducted a comparison on this pFL-Bench setup. (The full comprehensive comparison will be given in our revised paper and supplementary material.)  \nFor the performance of PMs, we evaluated both the test accuracy and standard deviation across clients during averaging, where this standard deviation serves as a fairness-related metric. For these experiments, we used 200 clients and the same data setup as given in the pFL-Bench paper.\n\n| Methods | $Acc$     | $\\sigma$ |\n| ------- | --------- | -------- |\n| FedAvg  | 82.35     | 6.23     |\n| FedProx | 81.87     | 5.92     |\n| Ditto   | 85.26     | 5.04     |\n| FedRoD  | 86.03     | 7.36     |\n| Ours    | **87.51** | 6.77     |\n\n[1] FedMask: Joint Computation and Communication-Efficient Personalized Federated Learning via Heterogeneous Masking\n\n[2] pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning\n\n[3] LEAF: A Benchmark for Federated Settings\n\n\n\n> The reproducibility of the experiments is questionable. I could not find how the local test set was generated in CIFAR-10, such as the train/test ratio, in the main text or the supplementary material.\n\nNote that in our supplementary material, we have provided a description of the local test set sampling (see lines 40-44). \nHowever, we agree that the details on the generation of local test sets could be made clearer.\nOverall, the local test set for each client is sampled from the test set of the corresponding benchmark dataset, such as the 10000 test images in CIFAR-10. Thus, we plan to revise the last sentence (lines 42-44) in the supplementary material to the following:\n\n\"Specifically, we perform sampling for the local test sets according to the distribution of training data for each respective client, where sampling is performed without replacement on the test set of the corresponding benchmark dataset, which is obtained via the Dirichlet distribution.''\n\nWe hope our proposed revised wording would improve clarity."}, {"review_id": "ZOUyy6tUQz", "reviewer": "Reviewer_UBhD", "summary": "This paper introduces a novel spectral co-distillation framework for PFL, leveraging a Fourier spectrum perspective to measure model similarities. With the assumption that local models share low-frequency features and the global model's Fourier spectrum includes averaged high-level features from all clients, the authors propose a bi-directional knowledge distillation approach to improve performance on both sides. Additionally, a wait-free local training protocol is introduced to reduce runtime. Experimental results on multiple datasets with diverse heterogeneous data settings validate the effectiveness of the framework.", "questions": "- The calculation process of the model spectrum is unclear based on the existing descriptions (line 151-155 in the main file, line 73-74 in the appendix). Clarifications are needed, similar to prior work [22][23]. It is better to provide detailed explanations of how the overall model parameters are organized into a single vector and how the spectral vector is calculated. Offering a specific example, such as AlexNet, would greatly enhance the understanding of these processes.\n- The explanation for the results of the ablation study in Table 5 is inaccurate. Upon comparing the \"w/o SCD-PM\" line with the \"w/o Both\" line, it can be observed that solely adopting SCD-GM is detrimental to performance (GM's performance decreases for both $\\alpha=1$ and $\\alpha=0.1$, while PM's performance remains steady). However, when comparing the \"Ours\" line with the \"w/o SCD-GM\" line, the performance improves upon the further adoption of SCD-GM in addition to SCD-PM. This indicates that SCD-PM holds a higher priority and plays a more fundamental role.", "limitations": "A section specifically addressing the limitations and broad impact is missing.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Novel Aspect of Measuring Model Similarity**: The novel frequency perspective for measuring model similarity is intriguing and holds promise for producing a more principled solution for PFL. This aspect deserves encouragement and further exploration.\n\n**Interesting Motivation**: The consideration of shared low-frequency components among personalized models and the generic model provides a unique perspective on describing their relationship. ", "weaknesses": "**Lack of Theoretical Analysis**: The authors highlight in lines 39-41 that existing works lack theoretical support and heavily rely on empirical observations. However, the core motivation behind spectral distillation (line 177) is also intuitive, and no theoretical analysis or guarantees are provided in the paper. This limits the contribution of the proposed method.\n\n**Insufficient Experiments**: \n- The paper would benefit from empirical analysis of model spectra to justify or validate the core motivation. Specifically, it is important to investigate if the robustness of low-frequency components against uniform random perturbation, as demonstrated in Ref[23], remains applicable in the Non-IIDness setting of FL. This would support the assumption of low-frequency component sharing among local models. Providing experimental evidence would solidify and amplify the significance of this assumption.\n- Empirical analysis for key factors $\\lambda_p$ and $\\lambda_G$ is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Below we address the reviewer's concerns point-by-point.\n\n> [Lack of Theoretical Analysis] The authors highlight in lines 39-41 that existing works lack theoretical support and heavily rely on empirical observations. However, the core motivation behind spectral distillation (line 177) is also intuitive, and no theoretical analysis or guarantees are provided in the paper. This limits the contribution of the proposed method\n\nWe thank the reviewer for the comment. For the lack of theoretical analysis of our framework, please see our response to Reviewer jcpB, which addresses what we shall include in our revised paper.\n\n> [Insufficient Experiments] The paper would benefit from empirical analysis of model spectra to justify or validate the core motivation. Specifically, it is important to investigate if the robustness of low-frequency components against uniform random perturbation, as demonstrated in Ref[23], remains applicable in the Non-IIDness setting of FL. This would support the assumption of low-frequency component sharing among local models. Providing experimental evidence would solidify and amplify the significance of this assumption.\n\nWe thank the reviewer for this insightful suggestion.\nFirst, we agree that exploring the robustness of low-frequency components to random perturbations would amplify the significance of our proposed spectral co-distillation framework.\n\nAs suggested, we have conducted additional experiments with the label noise provided in [1] on CIFAR-10 with data setting $\\alpha=1$, where the label noise ratio is denoted by $\\beta$. The experiment setup is the same as described in Sec. 3.1 of [1]. \n\n\n\n| $\\beta$ | 0     | 0.1   | 0.2   | 0.3   |\n| ------- | ----- | ----- | ----- | ----- |\n| GM      | 86.37 | 85.43 | 82.78 | 80.51 |\n| PM      | 92.25 | 89.36 | 85.38 | 82.11 |\n\nWe should also note that a detailed analysis of the robustness of spectral-based methods for FL/PFL is not the main focus of this paper, but we would certainly be keen to explore this as future work.\n\n[1] Spectral Bias in Practice: the Role of Function Frequency in Generalization\n\n> Empirical analysis for key factors $\\lambda_G$ and $\\lambda_p$ is missing.\n\nWe thank the reviewer for pointing this out. We have provided additional experiment results for the sensitivity analysis of the coefficients $\\lambda_\\text{G}$ and $\\lambda_\\text{p}$ on CIFAR-10 with data setting $\\alpha=1$. A comprehensive sensitivity analysis will be provided in  our revised supplementary material.\n\nSensitivity analysis of $\\lambda_\\text{G}$ \n\n| $\\lambda_\\text{G}$ | 0.01  | 0.05  | 0.1   | 0.3   |\n| ------------------ | ----- | ----- | ----- | ----- |\n| GM                 | 84.76 | 86.37 | 86.20 | 83.55 |\n| PM                 | 90.33 | 92.25 | 91.35 | 89.64 |\n\nSensitivity analysis of $\\lambda_\\text{p}$ \n\n| $\\lambda_\\text{p}$ | 0.01  | 0.05  | 0.1   | 0.3   |\n| ------------------ | ----- | ----- | ----- | ----- |\n| GM                 | 86.37 | 86.33 | 85.61 | 85.72 |\n| PM                 | 92.25 | 91.89 | 89.47 | 89.33 |\n\n\n\n> The calculation process of the model spectrum is unclear based on the existing descriptions (line 151-155 in the main file, line 73-74 in the appendix). Clarifications are needed, similar to prior work [22][23]. It is better to provide detailed explanations of how the overall model parameters are organized into a single vector and how the spectral vector is calculated. Offering a specific example, such as AlexNet, would greatly enhance the understanding of these processes.\n\nWe thank the reviewer for this suggestion. The calculation of the spectral vector directly follows the code provided by the spectral bias paper [1]. We will provide more details in our revised paper.\n\n[1] On the Spectral Bias of Neural Networks\n\n> The explanation for the results of the ablation study in Table 5 is inaccurate. Upon comparing the \"w/o SCD-PM\" line with the \"w/o Both\" line, it can be observed that solely adopting SCD-GM is detrimental to performance (GM's performance decreases for both $\\alpha=1$ and $\\alpha=0.1$, while PM's performance remains steady). However, when comparing the \"Ours\" line with the \"w/o SCD-GM\" line, the performance improves upon the further adoption of SCD-GM in addition to SCD-PM. This indicates that SCD-PM holds a higher priority and plays a more fundamental role\n\nPlease note that our summarized explanation for the results of the ablation study in Table 5 is correct.\nReviewer 8564 raised a similar question, and a detailed explanation can be found in our response to the last question from Reviewer 8564."}, {"review_id": "nLcmufidZ7", "reviewer": "Reviewer_6pBd", "summary": "This paper proposes a novel approach to address the challenge of data heterogeneity in personalized federated learning (PFL) by introducing a spectral co-distillation framework based on model spectrum information. The proposed method combines a wait-free local training protocol with bi-directional spectrum knowledge distillation to bridge the training of generic and personalized models. The authors evaluate their approach on CIFAR-10/100 and iNaturalist-2017 datasets using ResNet-18/34 and ResNet-50 architectures, and show that it outperforms existing PFL and conventional federated learning baselines on both personalized and generic model test accuracies. The proposed wait-free training protocol also significantly improves the efficiency of convergence time and has the potential to boost the time efficiency of PFL+ methods. The authors conclude that their approach demonstrates the usefulness of spectral information for knowledge distillation during personalized model training and the potential of using truncated spectral information of personalized models to boost the performance of the generic model.", "questions": "see above", "limitations": "see above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed spectral co-distillation framework is a novel approach that addresses the challenge of data heterogeneity in personalized federated learning.\nThe wait-free local training protocol significantly improves the efficiency of convergence time and has the potential to boost the time efficiency of PFL+ methods.\nThe proposed approach outperforms existing PFL and conventional federated learning baselines on both personalized and generic model test accuracies across multiple datasets and architectures.\nThe authors provide a sensitivity analysis of the truncation ratio τ and demonstrate the usefulness of spectral information for knowledge distillation during personalized model training.\n", "weaknesses": "The proposed approach may require additional computational resources to compute the spectral information of the models, which could increase the overall training time and cost.\nThe authors do not provide a comprehensive analysis of the privacy and security implications of the proposed approach, which could be a concern in real-world applications.\nThe authors only evaluate their approach on a limited set of datasets and architectures, and it is unclear how well it would generalize to other domains and settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive feedback. Below we address the highlighted weakness point-by-point.\n\n> The proposed approach may require additional computational resources to compute the spectral information of the models, which could increase the overall training time and cost. The authors do not provide a comprehensive analysis of the privacy and security implications of the proposed approach, which could be a concern in real-world applications. The authors only evaluate their approach on a limited set of datasets and architectures, and it is unclear how well it would generalize to other domains and settings.\n\n\n\n**1. Computational resource requirement**\nWe agree that our proposed spectral distillation process would require additional computational resources; \nthis was discussed in Section 3 of the supplementary material. Nonetheless, our proposed wait-free training protocol could  mitigate the impact of additional running time via an effective utilization of the idle time for PM training.\n\n\n\n**2. Privacy and security implications**\nSpectrum vectors and their truncated versions are computed solely on the model weights.\nIn FL, since model weights are already transmitted between the server and the clients, \nthe computation of such spectrum vectors maintains the same level of privacy and security already presented in FL.\n\n\n\n**3. Experiments**\nTo recap, we have provided experimental results on three datasets, and on different architectures with diverse non-IID data settings, including an evaluation on iNaturalist-2017, a large real-world image dataset. Such experiment setups are typically chosen in existing FL works, so we adopt the same setup for a fairer comparison with the baselines. As suggested, we have conducted additional experiments on a new task over an FL-oriented benchmark data setup, FEMNIST in pFL-Bench [1]. The results are given as follows.\n\nWe compare both the accuracy and its standard deviation across clients during averaging as a fariness-related metric for PM performance. We used 200 clients and the same data setup as given in the pFL-Bench paper.\n\n| Methods | $Acc$ | $\\sigma$ |\n| ------- | ----- | -------- |\n| FedAvg  | 82.35 | 6.23     |\n| FedProx | 81.87 | 5.92     |\n| Ditto   | 85.26 | 5.04     |\n| FedRoD  | 86.03 | 7.36     |\n| Ours    | 87.51 | 6.77     |\n\n\nWe hope our outperformance in this additional real-world setup would address the reviewer's concerns.\nA comprehensive comparison will be given in our revised supplementary material.\n\n[1] pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning"}, {"review_id": "IEhzVMlqKk", "reviewer": "Reviewer_jcpB", "summary": "The authors proposed a personalized federated learning (PFL) method, where the (dis-)similarity of models are represented through their Fourier spectra. The authors also proposed a bi-directional knowledge distillation between the generic and personalized models and a wait-free local training protocol that can utilize the idle time during global communication so as to reduce the total runtime. Extensive experiments and ablation studies are conducted to demonstrate the performance. ", "questions": "1. What are some limitations of this wait-free local training protocol? What kind of methods can adaptively include this protocol into their framework?\n2. In the case that a FL system has a large number of edge devices/clients, the participation rate may be low. If I’m understanding correctly, the personalized models keep on training until the updated global weights are received. If an edge device doesn’t participate in training for a long time, wouldn’t this protocol incur a lot of local computation? Also, if the personalized models keep on training without the most updated version of the global weights, is it possible that the personalized models will drift away from the rest of the system?", "limitations": "The authors have adequately addressed the limitations. Their work does not seem to have any potential negative societal impact. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The authors are the first to characterize (dis-)similarity of models via Fourier spectra in PFL, and the proposed bi-directional knowledge distillation is novel. Further, the proposed simple wait-free local training protocol seems to have the potential of speeding up many existing PFL methods. \n2. The paper is well organized and clearly written. \n3. The experiment results seem strong. ", "weaknesses": "The use of Fourier spectra is motivated by previous studies on deep learning theories. As it stands, the method seems like a mathematically motivated heuristic. It would be excellent if certain convergence guarantees or error bounds can be proved, even if just for the most vanilla case. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very encouraged by the reviewer's positive feedback. Below, we address the comments point-by-point.\n\n> The use of Fourier spectra is motivated by previous studies on deep learning theories. As it stands, the method seems like a mathematically motivated heuristic. It would be excellent if certain convergence guarantees or error bounds can be proved, even if just for the most vanilla case.\n\nWe thank the reviewer for the thoughtful comment. \nWe agree that the theoretical analysis would benefit the significance of the proposed method. However, given the word limit and limited time for rebuttal, we could not include a complete convergence analysis in this response. \n\nThat said, we can stretch the approaches for an analysis of the convergence rate under the proposed framework, as follows: First, note that the global loss function includes a weighted sum of the local loss functions and a regularizer, whereas the latter is given in the form of a KL divergence. As demonstrated in [1], the KL divergence usually exhibits convexity in terms of the model parameters. Consequently, noting that the model training undergoes (stochastic) gradient descent, we can establish a convergence rate for the global part (under the commonly employed assumption of smoothness of the local loss functions). The convergence rate of the local training follows similar approaches. \n\nIn particular, the intricate operations involved in our proposed framework will be reflected in the multiplicative factors in the convergence rate. We shall highlight this and provide the analysis in our revised supplementary material.\n\n[1] How Does Loss Function Affect Generalization Performance of Deep Learning? Application to Human Age Estimation\n\n\n\n> What are some limitations of this wait-free local training protocol? What kind of methods can adaptively include this protocol into their framework?\n\n1) Limitations of our wait-free training protocol are described in lines 311-312 of the main paper, and also in lines 78-80 of the supplementary material.\n   Essentially, our protocol assumes a synchronized network connection, which may not be practical for scenarios with large system/network heterogeneity.\n   We shall leave the extension to the case of asynchronous PFL as our future work.\n\n2) This protocol is designed for PFL+ (see line 130 of our main paper), in which a generic model (GM) and $N$ oersonalized models (PMs) are trained. Our protocol could easily be incorporated into any methods for PFL+ where the training of the GM and PMs can be decoupled into a sequential process, such as Ditto [1] and FedRoD [2]. In particular, in Table 2 of the supplementary material, \n   we have shown how our wait-free local training protocol, when applied to Ditto, gives a significant reduction in the total running time overhead.\n\n[1] Ditto: Fair and Robust Federated Learning Through Personalization\n\n[2] On Bridging Generic and Personalized Federated Learning for Image Classification\n\n> In the case that a FL system has a large number of edge devices/clients, the participation rate may be low. If I’m understanding correctly, the personalized models keep on training until the updated global weights are received. If an edge device doesn’t participate in training for a long time, wouldn’t this protocol incur a lot of local computation? Also, if the personalized models keep on training without the most updated version of the global weights, is it possible that the personalized models will drift away from the rest of the system?\n\nWe thank the reviewer for the thoughtful comments, which cover the key considerations for how our wait-free local training protocol should be extended/modified. \n\nFor partial client participation over an FL system with a large number of clients, it is possible that an edge device/client would not be selected to participate in training for numerous consecutive communication rounds. \nFor such an unselected client, the client should not be continuously training its personalized model using the ``stale'' GM model weights, which as the reviewer has aptly pointed out, could result in model overfitting on limited data, and model drift/divergence.\n\nTo address this issue, there are two straightforward solutions:\n\ni) The first solution is to adopt a broadcasting scheme with regular latest GM broadcasts by the server. Such a scheme would ensure that every client receives the latest GM weights, thereby alleviating the negative impact of stale GM weights for unselected clients, and avoiding potential client drift.\nIn real-world network connections, the bandwidth of the downlink connection from the server to the clients is typically larger than the corresponding bandwidth of the uplink connection. Hence, such design would be practical.\n\nii) The second solution is to set an upper limit on the number of local training steps. The basic underlying idea is to terminate the local PM training after a certain number of local training steps. This could be an alternative option to address the issue of limited communication for cross-device FL scenarios. It should be noted that such design would still reduce the overall total training time, since the PM training is still performed within the idle waiting time in conventional FL.\n\nWe will include the above elaboration in our revision, both in our main paper and in the supplementary material."}]

Author rebuttal:

Meta-review: In this submission, the authors propose a co-distillation method to better deal the relations between global model and personalized models. Compared with existing pFL methods, this is a new perspective to examine such relations. The presented evaluations also demonstrate its effectiveness and great potentials. To this end, I recommend accepting this submission.

For the final decision, I lower the weight of reviewer FUrR as his comments are short and less meaningful. But all the other reviewers are pretty helpful. Hope the authors find the discussion with reviewers useful and make this submission a better one.
Decision: Accept (poster)