Learning Shareable Bases for Personalized Federated Image Classification
Authors: Hong-You Chen
Jike zhong
Mingda Zhang
Xuhui Jia
Hang Qi
Boqing Gong
Wei-Lun Chao
Li Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated learning
Computer vision

Abstract: Personalized federated learning (PFL) aims to leverage the collective wisdom of clients' data while constructing customized models that are tailored to individual client's data distributions. The existing work of PFL mostly aims to personalize for participating clients. In this paper, we focus on a less studied but practically important scenario---generating a personalized model for a new client efficiently. Different from most previous approaches that learn a whole or partial network for each client, we explicitly model the clients' overall meta distribution and embed each client into a low dimension space. We propose FedBasis, a novel PFL algorithm that learns a set of few, shareable basis models, upon which each client only needs to learn the coefficients for combining them into a personalized network. FedBasis is parameter-efficient, robust, and more accurate compared to other competitive PFL baselines, especially in a low data regime, without increasing the inference cost. To demonstrate its applicability, we further present a PFL evaluation protocol for image classification, featuring larger data discrepancies across clients in both the image and label spaces as well as more faithful training and test splits.

Under review as a conference paper at ICLR 2023
LEARNING SHAREABLE BASES FOR PERSONALIZED FEDERATED IMAGE CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Personalized federated learning (PFL) aims to leverage the collective wisdom of clients' data while constructing customized models that are tailored individual client's data distributions. The existing work of PFI. mostly aims to personalize for participating clients. In this paper, we focus on a less studied but practically important scenario generating a personalized model for new client efficiently. Different from most previous approaches that learn a whole or partial network for each client, we explicitly model the clients' overall meta distribution and embed each client into a low dimension space. We propose FEDBASIS, a novel PFL algorithm that learns set of few, shareable basis models, upon which each client only needs to learn the coefficients for combining them into a personalized network. FEDBASIS is parameter-efffcientt, robust, and more accurate compared to other competitive PFL baselines, especially in a low data regime, without increasing the inference cost. To demonstrate its applicability, we further present PFL evaluation protocol for image classification, featuring larger data discrepancies across clients in both the image and label spaces as well as more faithful training and test splits.
INTRODUCTION
Recent years have witnessed gradual shift in computer vision and machine learning from simply building a stronger model (e.g., image classifier) to taking more users' aspects into account. For instance, more attention has been paid to data privacy and ownership in collecting data for model training (Jordan & Mitchell, 2015; Papernot et al., 2016) Building models that are tailored to users' data, preferences, and characteristics have been shown to greatly improve user experience (Rudovic et al., 2018). 8). Personalized federated learning (PFL.) is a relatively new machine learning paradigm that can potentially fulfill the demands of both worlds (Kulkarni et al., 2020). On the one hand, follows the setup of federated learning (FL): training models with decentralized data held by users (i.e.. clients) (Kairouz et al., 2019). On the other hand, it aims to construct customized models for individual clients that would perform well for their respective data distributions.
While appealing, existing work of PFL has mainly focused on how to train the personalized models, e.g., via federated multi-task learning (Li et al., 2020a; Smith et al., 2017), model interpolation (Man- sour et al., 2020), fine-tuning (Chen & Chao, 2022; Yu et al., 2020), etc. Specifically, existing algorithms mostly require saving for each client whole or partial model (c.g., a ConvNet classifier or feature extractor). This implies linear parameter complexity with respect to the number of clients, which is parameter-inefffcieett and unfavorable for personalized cloud service the overall system needs linear space of storage, not to mention the efforts for profiling, versioning, and provenance, for every client. Less attention has been paid how to deploy and maintain the personalized system.
A practical challenge of previous work is how to fulfill new clients' queries, who did not involve in the training phase. Beyond training personalized models for the participated clients only, we focus on preparing to serve new clients with fast, data-efficient personalization. A promising solution is Model Agnostic Meta-Learning (MAML) (Finn et al., 2017) that aims to learn a good initialization such that it can be adapted to new task fast, e.g., in a few SGD steps. The model-based idea has been inserted into PFL as well, by learning a model ready to be fine-tuned on each client's 's local data (Fallah al., 2020). However, it still lears the parameters of whole or partial model for each client. Several recent studies (Pillutla et al., 2022: Wu et al., 2022; Fallah et al., 2020) show that when individual clients' data scarce, fine-tuning may suffer from overfitting and being sensitive

---

Under review as a conference paper at ICLR 2023
Ours
0i
Basis composition
A
Basis1
Basis2
Basis3
D i
Conventional
X
0.8
0.15
0.05
Personalized model 0
Combinations
ar
0:
Model space
Learning on client i's data
Figure 1: conventional PFL, each client learns high-dimensional model. the overall parameters scale with number elients. In our FEDBASIS, we Iearn a few sharable basis models of the same network architecture. After the basis models are trained, a new client only needs to learn short combination vector as the coefficients combine them in the parameter space into personalized network thus more data efficient and robust.
to hyperparameters such as learning rates and the number of steps, eventually hurting some clients' test performance. though the average personalized performance could be improved.
To address such a dilemma, we propose to improve the robustness of a personalization system by reducing the overall parameter complexity. Specifically, number of personalized parameters from the number of clients. We hypothesize that the clients' local distributions are not disjoint and could share some latent variables (e.g., domains, superclasses, etc). Learning a separate personalized model for each client could be redundant and unfavorable for the generalization of new data/clients. Specifically, we are interested in learning meta-mot mottmmddel that can generate a personalized model for every client such that the overall parameter complexity is bounded by the size of such a meta-model while providing flexibility adapt the whole network.
aim to decouple the required total
We propose novel model architecture and learning algorithm for PFL. Our idea is to learn few, shareable basis models of the same architecture, which can be combined layer by layer form personalized model with learnable combination coefficients, inspired by (Changpinyo et al., 2016; Evgeniou & Pontil, 2007). The inference memory footprint and computation cost of the combined personalized model do not scale with #basis. An illustration is in Figure 1. It can be treated as Principal Component Analysis (PCA) on the collections of nigh-dimensional neural networks, essentially learning sharable bases across clients.
Learning the basis models in a federated setting, however, is nontrivial. As will be discussed in section 4, naively training them via the FEDAVG procedure (McMahan et al,, 2017) i.e., iterating between local model training for multiple epochs and global aggregation - would simply result in non-specialized bases that unable to construct personalized models. We, therefore, present an improved coordinate descent style federated algorithm to overcome this problem. We name this architecture and algorithm FEDBASIS. FEDBASIS enjoys several desired properties. It maintains built-in overall parameter efficiency but also maintains high personalized classification accuracy. After the basis models are trained, new client only needs learn very few parameters, i.c., the coefficients for combining them, to accommodate the distribution discrepancy, which is more robust learning rates and the training size. Last but least, FEDBASIS a stateless algorithm and does not increase inference-timee cost, suitably for cross-device deployment.
To demonstrate the applicability and generalizability of FEDBASIS, we further present PFLBED, set of benchmark datasets for cross cross-domain PFL. We point out some existing PFI. evaluations that either pose huge distribution mismatches between training and testing (thus mislcading) (Caldas et al., 2018; Li et al., 2020a) or focus the cases that only either labels the input domains are non-III across clients (thus less comprehensive) (Chen & Chao, 2022; Sun et al., 2021). PFLBED carefully designed resolve both problems. Concretely, according to domains by leveraging either domain annotated datasets (Li et al., 2017: Venkateswara al., 2017) natural attributes like users, PFLBED is able to capture more diverse and realistic PFL scenarios to reflect real-world challenges.
split the datasets into personalized portions
2 RELATED WORK
Many approaches have been developed to improve different dimensions of PFL. We focus on a less studied route by learning meta-model to summarize all the client models. Our FEDBASIS
2

---

Under review as a conference paper at ICLR 2023
architectures were inspired by networks proposed improve the accuracy of single neural network model in centralized learning (Yang et al., 2019; Chen et al., 2020; Zhang et al., 2021c). Our novelty is in extending such concept to PFL, identifying difficulties in optimization, and resolving them accordingly. In the following, we summarize existing PFI. approaches.
Multi-task learning (MTL). Many previous works formulate personalization over a group of clients as multi-task (MTL) (Zhang & Yang. 2017; Ruder, 2017: Evgeniou & Pontil, 2004; 2007; Jacob al., 2009; Zhang & Yeung, 2010) leveraging the clients' task relatedness to improve model generalizability. These methods typically focus on regularizer designs while each client learns for its own model. For instance, (Smith et al., 2017: Zhang et al., 2021a) encouraged related clients to learn similar models; (Li et al., 2020a; Dinh ct al.. 2020; Deng et al., 2020; Hanzely et al., 2020; Hanzely & Richtárik, 2020; Corinzia & Buhmann, 2019; Li & Wang, 2019) regularized local models with a learnable global model, prior, or set of data logits.
Mixture of models. Assuming the data distribution of each client is a mixture of underlying distributions, another approach is based on mixture models: (Peterson et al., 2019; Agarwal et al., 2020; Zec et al., 2020; Marfoq et al., 2021) (separately) learned global and personalized models and performed a mixture of them for prediction. However, the computation cost in inference scales linearly to the number of the models in the mixture since this approach aggregates the experts on outputs but not on the model weights like ours (which is arguably more challenging).
Personalized layers. Which layers/components in a network should be personalized (to tailor local distributions) be shared (to collaborate across clients' data) is crucial question that attract many research (Shen et al., 2022; Liang et al., 2020; Li et al., 2021b; Bui et al., 2019; Arivazhagan et al., 2019). Some works (Ma al., 2022; Sun et al., 2021) propose learning-based methods for such decisions. Our goal is to summarize all personalized parameters thus orthogonal to this direction. In this paper, we consider the whole network adaptable but combining these techniques select a partial network to further improve will be our future work.
General representations. Another line of research focuses on learning universal feature extractor. To generate the personalized model, each client only needs to learn an output head (Collins et al., 2021; Chen & Chao, 2022), a Gaussian process tree classifier (Achituve et al., 2021), or a k-NN classifier (Marfoq et al., 2022). Such an approach is simple and remarkably strong on single-domain class-non- PFL but likely sub-optimal when the input domains are also non-lID (different styles, locations, data collection, etc). We agree on the concept of learning powerful, general representation but go beyond single-domain features and maintain multiple basis models shared by all clients to serve the tross-domain scenarios.
Meta-learning, Meta-learning is the most relevant to ours that also learns a meta-model that can be personalized rapidly (Khodak et al., 2019; Chen et al., 2018; Fallah al., 2020; Jiang et al., 2019). It requires to split/reuse the training data as a meta- validatio set, which might not be favorable if the training size is small. Other algorithms model the relationships between clients (Zhang et al., 2021b; Huang et al., 2021) for initializing, or regularize personalized models. However, they still need a linear parameter complexity for the final personalized models. We are inspired by (Evgeniou & Pontil, 2007) and formulate each client model as a linear combination of a few basis models. This makes our work clearly different from existing PFL works as we bypass the linear parameter complexity.
The closest work to ours is (Shamsian et al., 2021) that summarizes local models into a HyperNet- work (Ha et al., 2017). Our work is inspired by the concept of reducing model complexity, but more effective and scalable implementation. We provide more detailed comparison in subsection 4.3.
BACKGROUND
We first provide a short background. In generic federated learning (GFL). the goal remains the same - to train a "global" model hgg say a classifier parameterized by 0. However, the training data are now collected and separately stored by M clients: each client E [M] keeps private set
D,, = ((x, (:n) where x is the input (c.g., images) and y E {1..... C} = [C] is the truth label. Given the loss function (e.g., cross-entropy). Let D U,. Dm denote the pseudo aggregated
3

---

Under review as a conference paper at ICLR 2023
data from all clients and Lm denote the empirical risk of client m, the problem is
M Dm D
min (( 0
=
(m)
(m)))
Σ
ho
(I)
Dmml
m=1
(EL.).JED
Since the data are decentralized, Equation cannot be solved directly. The standard federated averaging (F (FEDAYG) (McMahan et al., 2017) algorithm decomposes the optimization into multi- round process of iterations between local training at the clients and global aggregation at the server.
Let /00)0 denote the global/local model after round t, the two steps can be formulated as
M
1 m D
Local: ( arg ming 2m (0), initialized by g(t-1); D;-;;
Global: ((t)
(2)
In contrast to GFL, personalized federated learning (PFL) aims to learn for each client m a customized model Om perform well client m's data. While there is no agreed objective function, many existing works (Smith et al., 2017: Li et al.. 2020a; Dinh et al., 2020; Hanzely et al., 2020; Hanzely & Richtárik, 2020; & Wang, 2019) solve an optimization problem similar
M Σ (0m) R(2, (.,))))..... m=1
min (9.0 .Ar M
where R is regularizer; & is introduced relate clients to encourage learning similar models for overcoming their limited data. Unlike Equation 1, Equation seeks to minimize each client's empirical risk (plus regularizer) by the personalized model 6.. but not single global model 0
Our assumption. The clients' local data share similarity (e.g.. domains, styles, classes, etc) - a common assumption made in multi-task learning (Evgeniou & Pontil, 2007) it is likely that can use a much smaller set of models (vI,..., Ux ). K < M, |u| |0). to construct high-quality personalized models and meanwhile largely reduce the number of parameters.
FEDBASIS: PERSONALIZED FEDERATED LEARNING WITH BASES
4.1 MOTIVATION AND FORMULATION
Reducing overall parameter complexity. While both solving Equation 3 and fine-tuning the FEDAVG's global model can lead to personalized models, they require learning and saving the parameters of a whole (or partial) model for each of the M clients - i.e., linear parameter complexity O(M x ()). This particularly redundant when a huge number of clients are involved and their distributions are similar. Besides, model parameters learned specifically for each client would be vulnerable to overfitting, even with regularization.
To resolve these issues in PFL, we propose a novel way to bypass the linear parameter complexity, inspired by (Changpinyo et al., 2016; Evgeniou & Pontil, 2007). We represent each personalized model 0m by
0m (a V) Σ am [k] x Uk.
(4)
where == ( is a set of K basis models shareable among clients, and am E A K- is K -ddimensionnal vector on the (K 1)-simplex that records the personalized convex combination coefficients. That is, each personalized model is convex combination of set of basis models.
With this representation, the total parameters to save for all clients become
O(K x |0| + K x M) O(K x 01).
(5)
Here, O(K x M) corresponds to all the combination coefficients A {a)..... am,, which is negligible since for most of the modern neural network models, 0 > M.
4

---

Under review as a conference paper at ICLR 2023
Objective function. Building upon the model representation in Equation 4 and the optimization in Equation 3, we define our FEDBASIS PFL problem as
min {=(am V= (ov M Σ (0m),
where On..
Σ am [k] --am x Uk--
(6)
We note that both the basis models and the combination coefficient vectors are to be learned. We drop the regularization term in Equation 3 as the convex combination itself is form of regulariza- tion (Evgeniou & Pontil, 2007), where we implement a by a softmax function in our experiments.
Training. We discuss how to optimize Equation 6 in subsection 4.2.
Personalization for new clients. To generate the personalized model for a client, the client receives V and finds its combination coefficients Oxn by SGD with its local data. Since |amm is merely K' parameter per client, it can be robustly learned with fewer data. A single personalized model Om is then constructed by convexly combining the parameters of basis models in V layer-by-layer, according to O.m- The inference time on each image thus remains a constant. This is sharply different from the mixture of experts (Reisser et al., 2021), which combines the predictions of expert models, not their parameters- the inference cost of prediction on each image is #expert times more.
4.2 FEDERATED LEARNING ALGORITHM
Similarly to Equation 1, Equation cannot be solved directly since the clients' data are decentralized. A baseline training algorithm is to learn V with FEDAVG directly
1 arg min (a, V), initialized by { x K, D(1-1)),
Local:
(t) D aam
K
M
Global:
D00
Σ
M
m
m==1
We use (a,, V) concise notation for L. (0= [ [k]
vy). Note that in local training,
client m only updates her own coefficients (t) not others'; all basis models in ii(t) can potentially be updated. The embedding (!) is initialized every round locally and we do not keep it stateful.
Bases collapse. Unfortunately, such naive training can hardly achieve better performance than using a single basis. To un- derstand, we investigate the federated training dynamics using a preliminary experiment on the PACS image classification dataset (Li et al., 2017) (ResNet18, K = 4 bases, M = 40, lo- cal epochs = 5). Specifically, we check a) the average pairwise cosine similarity between the basis model parameters; b) the en- tropy of the learned combination vectors. High entropy implies an almost uniform combination vector. In Figure 2, we found that both the pairwise similarity and the entropy increase along with local training iterations and along with training rounds. In other words, the bases gradually collapse to similar parameters, and the combination vectors of all clients nearly collapse to uniform combinations. Consequently, each basis model does not learn specialized knowledge; the whole bases V basically degrade to single global model.
Within local training
1.0
Combinations
cosine similritt
0.4-
0.2
Along training rounds
125
1.00
0.55
0.50
0.5
10
Figure Cosine similarity between bases and the entropy of clients' combi- nation vectors on PACS dataset. FED- BASIS by baseline training collapses to non- specialized bases and uniform com- binations: (upper) within the first round of local training for one client; (lower) along training rounds, using basis mod- els aggregated at the server and combi- nations entropy averaged over clients.
By taking deeper look at Figure 2, we found that the collapse problem happens primarily in local training. To explain it, let us analyze the gradients derived at local training (cf. Equation 7)
Vv, Cm (a, V)- = ai X (0).
Valk Fm V) Uy VeL. (0).
(8)
Interestingly, while with different magnitudes, we found that V,, Cy (a, V) pushes every local basis model Us E (5(0) to-
wards the same direction (since az [k) ' 0). As local basis models gets similar towards (n)
5

---

Under review as a conference paper at ICLR 2023
their inner products with VeLn will become larger (i.e., positive) and similar, which would in turn push ax [k] to be larger via a similar strength. By forcing a to be on the (K -1)--simples (we do so by reparameterizing via softmax function), will inevitably become uniform. In other words, the more SGD iterations we perform within each round of local training, the more similar the local basis models and the more uniform the combination coefficients will be. We note that this phenomenon does not appear if we aggregate every iteration, which is infeasible in FL setting due to limited communication rounds.
We propose the following treatments to prevent the collapse problem.
Coordinate descent for the combination coefficients and bases. Within each round, propose first update (for multiple SGD steps) while freezing V, and then update V (for multiple SGD steps) while freezing a. We note that the beginning of each round of local training, 2k- VeLm (0) not necessarily positive. Updating with frozen V thus could potentially enlarge the difference among elements in ar: forcing the personalized model attend a subset of bases. After we start to update V. we freeze to prevent the collapse problem.
Sharpening combination coefficients Since a[k] 2 O, updating vy locally with Vo (or, V) would inevitably increase the cosine similarity between basis models. The exception is when some af 0, which results gradients. We therefore propose to artificially and temporally enforce this while calculating Vv, (a, V). We implement by learning 2b RK and reparameterizing
via a softmax function sharpened with a temperature 2 2 ox(k) Ex exp (+1)
(*)k)/T)
Improved training algorithm. Putting these treatments together, we present an improved training algorithm for FEDBASIS based on Equation 7. See the supplementary for the pseudo codes.
Local:
initialize {ar, V} by { K
x K, D(t-1)),
[Step 1]
(1)
= arg min fam (ox, V),
[Step 2]
(0f
SHARPEN (a (1)
[Step 3]
(i(t)
arg arg miny f. (t) V),
[Step 4]
M
Global:
((t)
Σ
(0) m
(9)
M
===1
4.3 THEORETICAL MOTIVATION
The benefits of such formulation that decouples the overall parameter complexity from the number of clients have been theoretically studied in the Theorem authors learn a linear hypernetwork of size Q to reconstruct every local model's parameters given the corresponding embedding. For brevity, the minor detailed assumptions are listed in section 4.5 in Shamsian al. (2021). Let K be the embedding size, M be the number of clients, and L be the sum of the Lipschitz constants of the hypernetwor and the embeddings AA There exist
Shamsian al. (2021), where the
Q N N O 2 log Me²
K
L
log )
(10)
such that if the number of samples per client |Pm] N, we have with probability at least 1-
for all 0m (am, V) that the generalization gap between the true and empirical risk (am V)
f. V)1 < The second term implies that summarizing many clients with a hypernetwork can notably improve generalization. The first term depends on K(< Q)
Our formulation of Equation 4 is indeed linear hypernetwork over (0m.) thus follows Equation 10. Our FEDBASIS has several advantages compared to the fully-connected network implementation of the hypernetwork in Shamsian et al. (2021). In their experiments, to handle 10 to 100 clients, the hypernetwork size is notoriously large, Q = 100j0]. making it hard to scale to deeper modern networks. While for our FEDBASIS formulation, Q K with a small K (4 to 8 in experi- ments), suggesting we can achieve a better bound. Morcover, reconstruction of the parameters is disconnected from the test loss; directly learn both the embedding and the bases in local training.

---

Under review as a conference paper at ICLR 2023
PADS
Clemr-P
Clent-P2
Client-P1 Photo domain
Training
Domains
Cllrt-A1
Cllet-Ad
100
Personalized Test
12ient-11
100
10e
100
190
we
108
199
Photo domain
Shared Test
C1
Class non-lID clients
Figure Our proposed construction federated personalized datasets PFLBED.
4.4 PRACTICAL EXTENSION
Layer-wise combinations. So far, we apply the same coefficient Om [k] to combine the whole ok into 6m (cf. Equation 4). Such a formula can be slightly relaxed to decouple coefficients for different layers. For instance, in our experiments on ResNets, we use one vector for each of the 4 blocks and the classifier (instead of one vector for the whole network) for combinations.
The major basis and warm-start for the bases. One concern that an individual basis does not learn the general knowledge since each basis is likely updated by partial clients but not trained on all data. We show this can be resolved easily by introducing two tricks. We found them generally make the learning smoother thus adopting them by default.
First, we maintain a major basis that is always included in the combinations. For instance, Equation becomes Om (amm, V) 1/v + 1 Ex Om [k] Uk. where v' is the major basis similar to the global model FEDAVG and other bases can personalize on top of it.
Second, we see FEDBASIS as way to summarize many clients' personalized/local models for new clients. is suitable to serve as post-processing tool for conventional FL algorithm. We propose a simple way here. Practically, K basis model with the centroids. It warm-starts FEDBASIS since each basis already learns general knowledge and is somehow specialized. In our experiments, we first run FEDAVG for few rounds and collect its global/local models (Chen & Chao, 2022) to warm-start the major/non-major bases, respectively. FEDBASIS can essentially become an extension on generic FL method like FEDAVG.
can collect {0m cluster them into K clusters, and initialize the
5 PFLBED: bases FOR BUILDING PERSONALIZBI BENCHMARKS
There have been many efforts on building datasets (Caldas et al., 2018; Hsu et al., 2020; Reddi et al., 2021) for generic FL. For PFL, how should a dataset be constructed into a reliable evaluation protocol for PFL algorithm development? As side contributions, we propose the following aspects:
Cross-domain with non-lID P., (x, y). A realistic personalized dataset should have the joint distribution P.,. (2. y) differ from client to client, not just Pm (z) (e.g.. domains) or Pm (y) (i.e., class labels). Both the training data sizes and the class distributions should be skewed among clients.
Sufficient test samples and matched training/test splits. The test set should be large enough for reliable evaluations. This is challenging when there are many clients, each with a small data size. For example, the popular 62-class hand-wrind written character FEMNIST dataset (Caldas et al., 2018) only has 226 images for each writer on average; many classes only have < 1 image. It is unfaithful to split each client into train/test sets due to mismatches Pm, (y). Indeed, we found a large discrepancy M Σm (ptrain()) ptast (y) = 0.77 even with 50% 50% split.
To achieve these desired properties, propose transform cross-domain dataset D into clients' sets ((Dtrasn / with the following procedures, illustrated in Figure
1. Separate D based on its domain annotations.
For each domain, first split a class-balanced test/validation set which will be shared with all clients from this domain. Take the rest as the training set.
3. For each domain, create heterogeneous partition (Hsu et al., 2019) for M' clients. An M'. dimensional vector 9e is drawn from Dirichlet distribution for class and assign the training set of class e to client m' proportionally to [m²].. Each client's 's images are from a single domain.
4. Record the class distributions P.,. (y) of each client's training set.
7

---

Under review as a conference paper at ICLR 2023
5. For each client in each domain, assign the whole test set of this domain as Disst Compute
M Σm Σ, Pm (b.)))( as the wlient-wise average personalized accuracy.
1 2. (v:)
Evaluation on new clients with few-shot samples. As mentioned before as our focus, we consider personalizing for a new client rapidly. We thus split the clients into a participated group and an unparticipated group. After the model is trained, it is personalized with the new client's training data (which is supposed to be small size in cross-device setup), and follows the same testing protocol.
Examples. In this paper, we consider two image object recognition datasets widely used in domain adaptation tasks. Both provides 4 handcrafted domain annotations including PACS (Li et al., 2017) and Office-Home (Venkateswara et al., 2017). For both datasets, following the pro- posed procedures, we first split the samples of each domain into 60%/20%/5%//15% for partici- bated/unparticipatt//vdddd vaeiiiiaatee/aliiaatiooeeess sets. The participated/uppatiiiinnn
sets are further split into 20/10 clients per domain by class non-lID sampling from Dirichlet(0.3) (Hsu et al., 2019).
Natural domains. We further include a more realistic Google Landmark (GLD-v2) (Weyand et al., 2020) classification dataset is split into 233 clients based on photographers in the GLD-User23k version (Hsu et al., 2020), thus Pm. (z, y) is naturally non-IID, We provide visualizations in the suppl.
6 EXPERIMENTS
Setups. We use a ResNet-18 (He et al., 2016). Following (Hsu et al., 2020), it is trained from an ImageNet pre-trained model with standard ImageNet-style data pre-processing and 224 resolutions. We use the SGD optimizer; momentum = 0.9, weight decay le 4. and local learning rate = 0.01. For the two handcrafted/GLD datasets, we run for 100/200 rounds with 16/ /64 batch sizes with 5 local epoch for each participated client (sample 100% 10%) in each round. Statistics of the datasets are summarized in Table 1. We evaluate on new clients. Different training sizes Madal//Mddrrttelaaagge used for personalization are considered for each client. See more details in suppl.
Baselines. Besides FEDAVG (McMahan et al., 2017), we compare to state-of-the- art methods including KNN-PER (Mar-
Table I: Summary of datasets and setups.
Dataset
Size
#CCass
Domain
Split
# Part Clients
foq et al., 2022) and FEDREP (Collins et al., 2021) that are based on general fea- tures. FEDBN (Li et al., 2021b) keeps lo- cal batchnorm parameters in training and we learn the batchnorm parameters for each new client. PFEDHN (Shamsian et al., 2021) based on hypernetworks and based on MAML are the relevant approaches to ours. As pointed out by Yu et al. (2020); Wang et al. (2019); Chen & Chao (2022); Cheng et al. (2021). fine-tuning (FT) serves as a very strong bascline. We also consider linear probe (LP) that trains linear classifier only for all applicable methods.
PACS Office Home
15.5K
9K
65
7
Handeratted REL
Hamdcrafted BFLBED
80/40
80/40
GLD23K
23K
203
Natural
User
117/116
(Fallah et al., 2020)
FEDBASIS. We train FEDBASIS with 5 local epochs for both a and V as coordinate descent described in subsection 4.2. We warm-start with FEDAVG for 30% of the total rounds and finish the rest, as described in subsection 4.4. We set the temperature 7 = 0.1 for sharpening the combinations, and the number of bases is 4/4/8 / besides the major basis for PACS/Office How.fieee-oooGL..D.... respectively. Since the clients are class non-IID, we learn the combinations and the whole classifier for personalization.
Main studies: new clients with low data. Table 2 summarize the results for personalization on class non-llI new clients of the three cross-domain datasets. We observe fine-tuning perform generally stronger than linear probe, but less robust (larger gaps between the last and best epoch). Since the data have large domain discrepancies, the ideal features are likely domain-specific. We found with fine-tuning, FEDAVG is competitive against recent PFL methods like FEDREP and KNN-PER. Interestingly, we observe nearest-neighhor KNN-PER seem to be less effective in such low-data regimes, consistent with Marfoq et al. (2022)'. We found PFEDHN is hard to achieve better performance than FVG+ as the hypernetoor for a ResNet requires large number of parameters thus less generalized (supporting subsection 4.3). The most effective baseline in such few-shot scenario is PER PER-FFDDAG++FM acknowledging that modeling few-shot personalization from meta view promising direction. Our FEDBASIS conceptually also maintains a meta model over clients by learning to combine basis models, outperforming the baselines especially on harder
We were able to reproduce the results in the original paper and did observe better performance than FEDAVG when each client has more samples (e.g., thousands).
8

---

Under review as a conference paper at ICLR 2023
Table 2: Averaged personalized test accuracy (%) on class non-lII new clients sampled from Dirichlet(O.3).
Each method is learned on each client's training data of different sizes for 20 epochs with learning rate selected from f0.005, 0.01, 0.05). We report both the Last epoch and the Best by validation.
Method/Daiinel
PACS
Office Home
GLD22k
Training Size
S
S
Epoch
Last Beat
Last
Best
|
Best]
|
Last
Best
[
Last
Best
|
FEDREP+LP
87.4 87.4
0.0
92.5
92.4
0.1
75.6
75.6
0.0
76.0
76.1
0.1
15.7
77.6
19
78.8
78.8
0.0
80.1
80.8
0.7
PEDREP+FT
89.8
89.8
0.0
92.4
92.5
0.1
74.2
76.1
1.9
75.2
76.4
1.2
79.2
79.9
0.7
81.5
82.5
10
81.5
83.5
2.0
FEDBN+LP
86.2 88.2
2.0
92.4
92.4
76.9
77.0
78.1
78.1
0.0
74.1
74.5
0.4
76.6
76.6
76.4
76.5
FEDBN+FF
90.8
92.1
1.3
93.0
93.1
79.2
70.5
77.8
81.8
PFEDHN
85.4
85.5
74.1
74.3
74.5
75.6
77.2
FFEDHN+LP
90.4
90.4
0.0
90.6
90.6
75.1 75.1
17.4 77.4
0.0
77.0
77.6
0.6
78.5
78.5
0.0
79.1
79.5
FFEDHN+FT
90.5
91.2
0.7
77.6
77.6
78.6
81.6
PER-FEDAVO+FT
95.4
95.6
96.2
96.3
84.4
86.1
86.2
78.5
85.3
6.8
79.9
KNN PER
71.6
71.6
50.4
54.5
54.0
57.4
69.2
KNN-PER+FT
72.7
72.7
0.0
79.4
79.7
$2.4
54.2
54.4
0.2
54.2 54.5
0.3
$7.1 57.8
69.5
70.2
FEDAYG
88.1
88
73.1
73.1
45.4
45
45,4
PRDAVO+LP
88.2 90.1
1.9
90.5
90.5
0.0
76.6
76.6
77.0
77.0
0.0
80.8
81
0.7
80.9
81
0.9
83.3
83.5
0.0
FEDAVG+FT
86.1
91.9
81.5
84.5
FEDBASIS
99.2
95.2|
0.0 96.2
96.2
0.0
87.4
87.5
0.1
87,5 87.7
0.2
87.4
87.4
0.0
87.6
87.6
0.0
89.0
89.1
0.1
Office-Homc
PACS
GLD23k
Book3
Brsis Figure 4: Visualization of the learned combinations {amm) }. Note that basis 0 is the major basis and each ResNet block shares a combination vector (e.g.. (4 + major) bases x4 blocks for PACS).
BDsi
datasets Office-Home and GLD. Notably, we learn much fewer parameters per client but still remain competitive to fine-tuning the whole model and more robust to the choice of epochs.
Visualization. To understand what FEDBASIS learns, we visu- alize the learned combinations in Figure 4. The clients are cross- domains and class non-IID. Interestingly, we see the clients group according domains (see Office-Home Block3&44.
Table 3: Ablation studies of FED- BASIS: coordinate descent, major basis, and temperature T.
CD MB т |Office GLD
Ablations. We provide ablation study (training size M) in Ta- ble 3, verifying our designs in subsection 4.2 and subsection 4.4.
X
d
0.1
83.5
85.8
0.1
87.2
83.3
Robustness of personalization. FEDBAST ASTS can personalize the features with only a few parameters of the combinations. Comparing fine-tuning in Table 4 (Office- observe it is much less sensitive to learning rates and training epochs. We note that selecting the best epoch may not be always feasible in practice since the clients may not have enough data for validation, thus we believe is important to consider such robustness, especially in sew-shot personalization.
1.0
87.1 85.5
/ 0.1
87.5
87.6
(M)), we
Table 4: FEDBASIS is more robust to personalization learning rates.
Method Learning rates
Last/Best Acc. 0.005
0.05
FEDAVG+FT
78.2/78.5 65. 65.4//5.6
1/86.2 FVGG
6/.5/8 87.5/87.7 87.6/87.6
FEDBASIS
7 CONCLUSION
We study personalized federated learning (PFL) for new clients. We aim to bypass the parameter complexity in maintaining personalized models and overcome their vulnerability to hyperparamters when personalized with few training data. We propose novel PFL architecture and algorithm FEDBASIS, which constructs each personalized model by a few, shareable basis models. Our training algorithm is designed systematically and mathematically soundly to overcome the difficulty of optimization. We also present a carefully designed evaluation PFLBED. Our empirical studies demonstrate the effectiveness of FEDBASIS, opening up new direction for further PFL research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems, 34:8392- 8406, 2021. 3
Alekh Agarwal, John Langford, and Chen-Yu Wei. Federated residual learning. arXiv preprint PrXiv:2003.12880 2020. 3
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint
2019. 3, 21
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. CoRR. abs/2004. 10340, 2020. 16
Duc Bui, Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, and Kang G Shin. Federated user representation learning. arXiv preprint arXiv:
2019.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint 2018. 2, 7
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. CVPR, 2016. 2,
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and X. He. Federated meta-learning with fast convergence and efficient communication. arXiv: Learning, 2018. 3
Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022. 1, 2, 3, 7,
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR), June 2020. 3, 14
Gary Cheng, Karan Chadha, and John Duchi. Fine-tuning is fine in federated learning. arXiv preprint
1xXiv:2008.07333 2021. 8
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In ICML, 2021. 21
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213-3223, 2016. 16
Luca Corinzia and Joachim M Buhmann. Variational federated multi-task learning. arXiv preprint arXiv:1906.06220 2019. 3
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint 1441,,, 2020. 3, 21
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau
envelopes. NeurlPS, 2020. 3, 4
An Evgeniou and Massimiliano Pontil. Multi-task feature learning. NeurIPS, 2007. 2, 3, 4,
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In KDD, 2004. 3
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. In NeurLPS, 2020. 3, 8, 19, 21
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1135. PMLR, 2017. 1
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. In ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2023
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.055/6, 2020. 3, 4,4, 21
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. NeurIPS, 2020. 3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint Stti::9990063333 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real- world data distribution. In European Conference Computer Vision, pp. 76-92. Springer, 2020. 7, 8, 16
Yutao Huang. Lingyang Chu, Z. Zhou, Lanjun Wang. Liu, Jian Pei, and Yanxin Zhang. Personalized cross-silo federated learning non-iid data. In AAAI, 2021. 3
Laurent Jacob, Francis Bach, and Jean-Philip Shan-Philippe Vert. Clustered multi-task learning: A convex formulation. In NeurlPS, 2009. 3
Yihan Jiang, Jakub Konecný, Keith Rush, and S. Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. ArXiv,
2019. 3
Michael I Jordan and Tom M Mitchell. Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255-266 2015.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019. 1
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICMI., 2020. 21
M. Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive gradient-base -ased meta-learning methods. In NeurIPS, 2019. 3
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning. pp. 5637-5664. PMLR, 2021. 16
V. Kulkarni, Milind Kulkarni, and A. Pant. Survey of personalization techniques for federated learning. 2020 Fourth World Conference Smart Trends in Systems, Security and Sustainability ( (WorldS4), pp. 794-77477 794, 2020.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-55500 2017. 5,
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.0m
2019. 3.4
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through. arXiv preprint arXiv:2012.04221, 2020a. 1, 1, 4
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.
Federated optimization in heterogeneous networks. In MLSys, 2020b. 21
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6366 6368. PMLR, 2021a. 21
11

---

Under review as a conference paper at ICLR 2023
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fed( Federated learning on non- ) features via local batch normalization. In ICLR, 2021b. 3, 8
Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis-Philiipe Morency. Think
locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 2020. 3, 21
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personal- ized federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10101, 2022. 3
Yishay Mansour, Mehryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 10619, 2020. 1
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34, 2021.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, pp- 15070-15092. PMLR, 2022. 3,
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, al. Communication-effcienntt learning of deep networks from decentralized data. In AISTATS, 2017. 2, 4, 8, 21
Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael Wellman. Towards the science of security and privacy in machine learning. arXiv preprint / 1.03814, 2016.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019. 16
Daniel Peterson, Pallika Kanani, and Virendra Marathe. Private federated learning with domain adaptation. arXiv preprint 1rtiv::912..67770 2019.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In International Conference Machine Learning, pp. 17716-17758. PMLR, 2022.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. In ICLR, 2021. 7
Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling. Federated mixture of experts. arXiv preprint arXiv:2107.06724, 2021.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 170600/700 2017.
Ognjen Rudovic, Jaeryoung Lee, Miles Dai, Björn Schuller, and Rosalind W Picard. Personalized machine learning for robot perception of affect and engagement in autism therapy. Science Robotics, 3(19):ea006760, 2018.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. In ICML, 2021. 3, 6, 8
Yiqing Shen, Yuyin Zhou, and Lequan Yu. Cd2-pfed: Cyclic distillation- guidedd channel decoupling for model personalization in federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10041-10050, 2022. 3
Virginia Smith, Chao-Kai Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In NeurlPS, 2017. 1,3,4,2 21
Benyuan Sun, Hongxing Huo, Yi Yang, and Bo Bai. Partialfed: Cross-domain personalized federated learning via partial initialization. Advances in Neural Information Processing Systems, 34:23309- 23320, 2021. 3
12

---

Under review as a conference paper at ICLR 2023
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017. 2,
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Franccoise Beaufays, and Ram- age. Federated evaluation of on-device personalization. ArXiv, 19111900.102222 2019.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2575-2584, 2020. 8
Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and Virginia Smith. Mot- ley: Benchmarking heterogeneity and personalization in federated learning. arXiv preprint arXiv:2206.09263 2022.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019. 3, 14
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation.
arXiv preprint arXhv:2002.04756 2020. 1,
Edvin Listo Zec, Olof Mogren, John Martinsson, Leon René Sütfeld, and Daniel Gillblad. Federated learning using mixture of experts. arXiv preprint arXiv:2010.02056, 2020. 3
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. Advances Neural Information Processing Systems, 34, 2021a. 3
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In ICLR, 2021b.
Mingda Zhang, Chun-Te Chu, Andrey Zhmoginov, Andrew Howard, Brendan Jou, Yukun Zhu, Li Zhang, Rebecca Hwa, and Adriana Kovashka. Basisnet: Two-stage model synthesis for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3081-3090, 2021c. 14
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv: 14, 2017.
Yu Zhang and Dit- Yan Yeung. A convex formulation for learning task relationships in multi-task learning. 2010.
13

---

II end Under review as a conference paper at ICLR 2023
Supplementary Materials
We provide details omitted in the main paper.
Appendix A: pseudo codes and more discussion of FEDBASIS (cf. section 4 of the main paper).
Appendix B: additional experimental details, results and analyses (ef. section and section of the main paper).
Appendix C: additional discussion on the datasets (cf. section 5 of the main paper).
Appendix D: additional results during the rebuttal.
A FEDBASIS ALGORITHM
Algorithm I: FEDBASIS- federated training
Server input :initial global basis parameter V;
Client m's input :initial local loss Lm, temperature
for < 1 to I rounds do
Communicate for each client m [M] in parallel do Initialize {amm. V/ by
clients m E [M];
K, V):
C' = arg min am L. (CEm, V) om < SHARPEN(QE i
V arg miny Lm (ar V):
Communicate Vi, to the server;
end
Construct 1 == 37 +M V%:
Server output :D:
Algorithm 2: FEDBASIS- generate personalized model
Client wn's input :initial global basis parameter V. local loss L.:: Initialize Omm by x K;
a arg min.... L. (amm; V)
Construct O.. (om.. V) Σn Ou [k] Us:
Client m's output :0
We provide summary in algorithm I for training our FEDBASIS (ef. subsection 4.2 in the main paper) and algorithm shows how use for generating personalized model. Similar to the FEDAVG algorithm, our FEDBASH ASIS also executes a multi-round training procedure between the local training at the clients and aggregation at the server.
The goal of FEDBASIS is collaboratively train K basis models V = (v } which can be used to combine into personalized models based on each client's combination coefficient E R (or more specifically, A (K- see Equation 4) within limited T' rounds of communications. The parameters are linearly combined layer by layer. Such specialized layers (Yang al., 2019; Chen al., 2020; Zhang et al., 2021c) improve the performance with little extra inference cost. Our contribution is to extend such concepts to personalization in FL setting, identify optimization issues, and resolve them.
To effectively learn the bases for personalization, in subsection 4.2, we introduce several important techniques in the local training to avoid bases collapse and encourage each basis to learn specialized knowledge. In each round of local training at client m, it first initializes the bases V using the one broadcast by the server. Next, we train Om and V with coordinate descent. We update am (for multiple SGD steps) while freezing V (line 5 in algorithm 1). To force the personalized model to attend to a subset of bases, we sharpen Osm by injecting a temperature into the Softmax function (line 6 in algorithm 1). Then, update V (for multiple SGD steps) while freezing On. Finally, the updated bases are sent back to the server for a basis-wise average with other clients' updates.
The FEDBASIS formulation enjoys several desired properties.
14

---

Under review as a conference paper at ICLR 2023
PACS
90
Acc (%)
80
20
Top-k bases
Figure 5: Reducing 40 fine-tuned personalized models into top-k bases by PCA.
The total parameter size of models does not scale with the number of clients. FEDBASIS ultimately outputs the bases with combination coefficients am for each client m. Each client only has amml K personalized parameters, which is negligible compared to the model.
For local training, the forwarding combined model only needs be generated per mini-batch but not per instance, making it scalable to batch sizes. The size of communications is K times more but K is typically small.
FEDBASIS does not increase clients' computation cost in inference. After training, the basis models are combined into a single personalized model. This is sharply different from the mixture of experts in that input needs to go through every expert and ensembles the predictions, where the cost is linear to the number of experts.
B MORE DETAILS AND RESULTS
B. SPLIT NEW CLIENTS FOR EVALUATION
In our experiments, demonstrate the data efficiency of each of the methods, we consider different training sizes (Small/Moderate/Ilar Large) for personalization each client. Concretely, for Office-Home and PACS, we use 50%/100% of each client's training set as the S/M setting for personalization, respectively. We note that, in PFLBED, we already split a relatively small set (20% of the overall data) and further split it into several new clients. On the other hand, for the GLD-v2 dataset, the clients are already split by User IDs, we thus randomly split 10%/20%/40% of each new client's data as the training set and take the rest as the test/validation sets (we split 20% for validation).
B.2 ANOTHER BASELINE: PRINCIPAL COMPONENT ANALYSIS (PCA)
Our FEDBASIS architecture is represent personalized models by set of few basis models. In the main paper, due to the page limit, we mainly present methods that directly learn the basis models. Here, we present another baseline, building upon reverse way of thinking: How can we summarize many personalized models into combinations of few basis models given the federated constraint that no data are available at the server? A straightforward way to achieve such model compression is to perform Principal Component Analysis (PCA) on the collection of all the personalized model parameters. That is, we can try to represent each personalized model with a few eigenvectors (as {VI..... }x with the top-k eigenvalues found by PCA.
Different from our more challenging experiments in the main paper, we consider an ideal case of personalization in centralized setting, train global model with mini-batches SGD on PACS datasets. We first train a global model and fine-tune it on each client's 's full dataset to obtain 40 personalized models (0m), ). Then, we perform PCA their vectorized parameters.
As shown in Figure 5, we observe the averaged personalized performance drops drastically as the number of eigenvectors decreases. For instance, reducing into 4 bases leads to slumps in the accuracy
15

---

Under review as a conference paper at ICLR 2023
of 18.1% for PACS. It demonstrates the challenge of this problem. We hypothesize that the poor performance is likely because (1) personalized models produced by fine-tuning do not simply lie on dimeeniimenion space and/or (2) such PCA linear method cannot guarantee to maintain the accuracy since the reconstruction of parameters is not tied to the real loss such as Equation 3, as we can also observe some fluctuations accuracy along with the changes of top-k.
Alternately, we instead investigate using k-means clustering on the personalized models {0m) param- eters to cluster them into models. We again see a significant accuracy drop of 21.4% for PACS.
models and use each client's assigned centroid as the personalized
Therefore, we are motivated to solve our proposed objective Equation 6 that aims to directly learn the bases such that all personalized models can be their linear combinations while minimizing the local empirical risks.
C MORE DISCUSSIONS ON THE DATASETS
C.I DISCUSSIONS ON PFLBED
In section 5, we provide several aspects including cross-domain and class non-IlI Pm (Z., y), sufficient test samples, matched training/test splits, and distributional robustness evaluated with the class- balanced accuracy. We propose standardized process called PFLBED to construct a faithful personalized dataset for PFL algorithm development. As examples, we propose to transform some existing datasets including PACS and Office-Home that are widely used in rench-marking domain adaption tasks, into PFL datasets. These datasets are suitable for experimental use in research since they are created with clear domain differences such as image styles like Photo Art. We also propose to use the existing naturally partitioned dataset GL.D-v2, a dataset consisting of landmark photographs taken from various locations around the world by different photographers where each partition contains 1 photographer's photos. For this reason, we can view the style difference amongst the photographers as the domain gap thus treating each partition (a.k.a client) as an independent domain. Because the number of samples contained in each partition also varies, we naturally believe this dataset is faithful personalized dataset for evaluating PFL algorithms along with PFLBED.
Here we discuss future work of PFLBED to extend to more datasets. We identify some promising datasets. DomainNet (Peng et al., 2019) contains domains of 345 different objects. The WILDs benchmark (Koh et al., 2021) collects several datasets across different applications and each domain is defined by attributes such as users, locations, different cameras (Beery et al., 2020), etc. (Hsu et al., 2020) presents two realistic datasets of species classification and landmark recognition split by locations or users for generic FL but not for PFL. Cityscape (Cordts et al., 2016) is a popular self- self-driving dataset that contains driving scene data from many cities in Germany. In this paper, highlight the importance of PFL dataset construction for faithful evaluation and focus on some more experimental datasets. We hope our efforts can inspire future work propose more datasets suitable for PFL research.
C.2 VISUALIZATIONS OF PFLBED DATASET CLIENT DISTRIBUTION
Here we show example client distributions of our proposed datasets for PFLBED. For PACS and Office-Home datasets, we follow the procedures outlined in section 5 where each client is sampled from Dirichlet(O.3) within each domain. For each dataset, we first record the occurrences Nmd of each label d D within each client m M as ClxD for D total labels and M total clients. We then concatenate all M clients' label counts as CDIxD We visualize the distribution using C DDM that the size of each point is proportional to label count Nmd for total of M x D points and each column CDx1 can be viewed as a single client's label distribution. As we can see, clients show both label space P., and domain space (m:( heterogeneity. It's worth noting that although Figure 6 does not directly show domain differences through color differences, each client can be directly treated an independent domain for the reason described in subsection C.I.
16

---

Under review as a conference paper at ICLR 2023
Google Landmark
190
180
Class labels
170
180
190
200
210
220
230
Client IDs
Figure 6: Clients distribution of GLD23K dataset. Each number the horizontal axis represents particular client for total of M 233 clients. Each number on the vertical axis represents particular class label for total of 203 classes. The maximum number of samples per class is 100.
PACS
Class labels
4
2
10
+5
10
Client IDs
Figure 7: Clients distribution of PACS dataset across different domains. Each number on the horizontal axis represents a particular client for total of M 80 clients. Each number on the vertical axis represents a particular class label for total of D = 7 classes. The maximum number of samples per class 256.
17

---

Under review as a conference paper at ICLR 2023
OfficeHome
55
40
Class labels
6P
<>
+
2
15
10
3
Client IDs
Figure 8: Clients distribution of Office-Home dataset across different domains. Each number on the horizontal axis represents a particular client for total of M = 80 clients. Each number on the vertical axis represents particular class label for total of 65 classes. The maximum number of samples per class is 49,
18

---

Under review as a conference paper at ICLR 2023
C.3 ADDITIONAL VISUALIZATIONS OF GLD23K CLIENT DISTRIBUTION
Google Landmark
800
700
600
# of samples
500
400
300
200
100
client #
Figure Clients label distribution of GLD23K dataset. Each number the horizontal axis represents particular client for total of M 233 clients. Vertical axis represents the number of samples each client has.
D ADDITIONAL RESULTS DURING REBUTTAL
D.1 MORE RESULTS ABOUT THE ROBUSTNESS FEDBASIS
In both Table and Table 4 in the main paper, we demonstrate the robustness of the FEDBASIS on the choices of learning rates and stopping epochs when it is fine-tuned for new clients, compared to other baselines. Note that, in the current Table 2. for each method and each dataset, we highlight the difference ( (A) between stopping the fine-tuning by the last epoch or by the best epoch selected by validation.
As requested by Reviewer 2JAA, we plotted out the dynamics of training on new clients, featuring more sets of different learning rates along the tine-tuning epochs for both our FEDBASIS and the most competitive baseline SER-FEDAVG+FT Office-Home with the small training size setting. As shown in Figure 10 and Figure 12, it is quite clear that FEDBASIS much more robust to various learning rates and does not require early stopping. We attribute it to the clear advantage that FEDBASIS only needs to personalize much fewer parameters when adapting to a new client, thus enjoying the robustness. We further note for BER--FEDAG++T.. although with proper tuning it can achieve decent performance (still lower than ours), this requires validation set for each client thus likely not practical in the real world.
Table 2. We focus on the more challenging datasets
We focus the FO version of FER-FEDAAG+FT (Fallah et al., 2020) due to its better accuracy and training efficiency on the datasets in our experiments. In Table 5, we provide a comparison on FEDG-EEA with the two variants FO and HF introduced in (Fallah et al., 2020) and confirmed FO is better the performance.
19

---

Under review as a conference paper at ICLR 2023
80
FedBasis Ir=0.1
FedBasis Ir=0,05
70
Test Acc
FedBasis Ir=0.01
FedBasis lr-0.005
60
Per-FedAvg Ir=0.1
Per-FedAvg Ir=0.05
50
Per-FedAvg Ir=0.01
Per-FedAvg Ir= 00000
40
0
10
20
Epochs
Figure 10: Fine-tuning training curves of Office-Home (small) dataset (cf. Table 2 the main paper) of PER-FEDAVG+FT and FEDBASIS with various fine-tuning learning rates.
90
85
80
Test Acc
FedBasis
FedBasis w/ L2
75
Per-FedAvg
Per-FedAvg w/ L2
70
65
60
5
10
15
Epochs
Figure 11: Fine-tuning training curves of Office-Home (small) dataset (cf. Table 2 in the main paper) of FER-FEDAVG+FT and FEDBASIS with fine-tuning learning rate 0.005 with f-2 regularization.
90
85
Test Acc
80
FedBasis
75
Per-FedAvg
70
65
60
40
60
80
100
Rounds
Figure 12: Federated training curves of Office-Home (small) dataset (cf. Table 2 in the main paper) of FER-FEDAVG+FT and FEDBASIS along the rounds. In each evaluated round (note that FEDBASIS warm-starts from 30 rounds of FRDAVG, described in subsection 4.4), the adaption procedure to evaluate on the new clients to report the averaged personalized accuracy as the same cf. Table 2.
20

---

Under review as a conference paper at ICLR 2023
Table 5: More results of PER-FEDAVG+FT with FO and HF variants in Fallah et al. (2020). The
results on based on Office-Home (Moderate training size) and cf. Table the main paper.
Method
Last/Best Acc.
Learning rates
0.001
0.005
0.01
0.05
0.1
FEDAVG+FT
78.1/78.1
78.2/78.5
70.5/76.6
65.4/75.6
38.1/73.1
PER-FEDAVG+ET (FO)
86. 1/86.1 86.
65.3/85.4
60.5/83.4
40.1/ 10.1/82.6
PER-FEDAVG+FT (HF)
85.3/85.3
85.5/85.5
63.4/83.6
43.7/81.6 34.8/80.7
FEDBASIS
87.6/87.6 87.5/87.7
87 87.6/87.7 87.6/87.6 87.5/87.6
Table 6: Average test accuracies on various partitions of CIFARIO and CIFARI00 with participation
rate 0.1, following the same experiment setting Table 1 in Collins et al. (2021), where the results of the baselines are copied from there as well. Our results use the authors' codes and their pipeline (Collins al., 2021) by plugging in our algorithm. We report the mean and variances the accuracy over 3 runs of different random seeds.
CIFARIO
CIFAR100
(# clients, classes per client)
(100,
(100, 5)
Local Only
70.68
75.29
FedAvg (McMahan et al., 2017)
51.78
23.94
FedAvg+FT
73.68
79.34
FedProx (Li al., 2020h)
50.99
20.17
FedProx+FT
72.75
78.52
SCAFFOLD (Karimireddy et al., 2020)
47.33
20.32
SCAFFOLD+FT
68.23
78.88
Fed-MTL (Smith et al., 2017)
58.31
71.47
PerFedAvg (Fallah et al., 2020)
67.20
72.05
LG-Fed (Liang et al., 2020)
63.02
72.44
L2GD (Hanzely & Richtárik, 2020)
59.98
72.13
APFL (Deng et al., 2020)
72.29
78.20
Ditto (I.i et al., 2021a)
70.34
78.91
FedPer (Arivazhagan et al., 2019)
73.84
76.00
FedRep (Collins al., 2021)
75.68
79.15
FedBasis (Ours, # basis= 3)
75.5 + 0.71
80.8 + 0.54
Table 7: Variance (a personalized test accuracy (%) over 3 differrent runs ef. Table
Method/Dataset
PACS
Office Home
GLD23k
Training Size
M
M
S
M
Epoch
Last Best
Last Best Last
Best
Last Best Last Best Last Best
Last Best
FEDREP+LP
0.22 0.16
0.21
0.22
0.31 0.26 0.33 0.25
0.55 0.56 0.47 0.52
0.29
FEDREP+FT
0.36 0.29 0.41
0.38
0.66
0.56
0.71
0.39
0.78
0.89
0.88
0.75
0.74
0.71
FEDBN+LP
0.15 0.16
0.31
0.15
0.12
0.23
0.28
0.19
0.36
0.41
0.29
0.21
0.51
0.39
FEDBN+FT
0.33 0.45
0.41
0.42
0.67
0.59
0.55
0.62
0.68
0.66
0.56
0.48
0.65
0.62
PFEDHN
0.78 0.64
0.56
0.57
0.46
0.51
0.48
0.55
0.41
0.28
0.55
0.56
0.39
0.28
PEDHHN+LP
0.36 0.44 0.29
0.36
0.27
0.31
0.28
0.25
0.87
0.86
0.82
0.75
0.78 0.58
FFE
0.85
0.97 0.56 0.77
0.77
0.75
0.64
0.70
1.01
0.89
0.88
0.91 1.15
PER- FER-FEDAVG+FT
0.51 0.46 0.37 0.41
0.70
0.61
0.63
0.66
0.51
0.25
0.48
0.45
0.34 0.38
KNN-PER
0.34
0.19
0.38
0,52
0.57
1,56
1.28
0.85
0.95 0.95
KNN-PER+FT
1.75 1.41
1.25
1.39
0.57
0.60
0.78
0.69
0.27
0.56
0.48
0.71
0.55 0.61
FEDAVG
0.23 0.25
0.29
0.24
0.38
0,50
0.54
0.39
0.55
0.56
0.54 0.27
FEDAVG+LP
0.15
0.21
0.20
0.17
0.29
0.22
0.31
0.44
0.63
0.65
0.48
0.59
0.64 0.59
FEDAVG+FT
0.52 0.39
0.44
0.51
0.57
0.46
0.60
0.58
0.68
0.71
0.59
0.58
0.61 0.48
FEDBASIS
0.52
0.56
0.45
0.50
0.38
0.39
0.42
0.45
0.55
0.66
0.47
0.68 0.29 0.45
21

---

Under review as a conference paper at ICLR 2023
Client training
Client test set
E.g., Split by 80/20 %
Client local data
Conventional PFL split
10
Our proposed PFLBed
100
100
10
C
P
(y)
Shared Test set of the same domain
Figure 13: Illustration of the difference between traditional PFL split and proposed PFLBED in cf. section 5 for a client. For the conventional way, given that each client may have limited data per class, after training/test split, the distribution is longer matched, leading to unfaithful evaluation. On the contrary, Our proposed way use shared test set from the same domain and re-weight the examples evaluation by classes (e.g., weighted accuracy).
22


Reviews:
[{"review_id": "MJY-Qh7Tb2", "reviewer": "Reviewer_5bfz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for personalized federated learning. The key idea is to represent neural network weights as a linear combination of multiple bases, where the bases are shared across clients and only coefficients are personalized. Doing so makes it easier to personalize the model for a new client who did not participate in the training, by just training the coefficient parts. A coordinate decent approach is introduce to train the both of personalized coefficients and shared bases in an alternate fashion. Experimental results on a new benchmark dataset shows the effectiveness of the proposed method over a variety of existing methods.", "summary_of_the_review": "Overall the paper is well written and presenting some technical contributions for personalized FL. Nevertheless, I am currently neutral about this submission due to its limited experimental evaluation. If the proposed method is confirmed to (1) statistically outperform baseline methods (with standard deviations or confidence intervals regarding multiple clients and multiple random seeds) and (2) work effectively also for other datasets/data splitting scheme used in prior work (e.g., those used in pFedHN), I would be willing to upgrade my score. Code submission for a complete reproduction of this work is also highly welcome.", "strength_and_weaknesses": "\n## Strong points\n- Overall, the paper is well written, structured, and easy to read.\n- The proposed work is well motivated. I agree that learning to personalize a learned model for a new client who was absent in the training is an important and interesting topic of personalized FL.\n- The proposed method is designed reasonably and technically solid. While learning to linearly combine multiple kernels is a known idea (Yang et al., 2019; Chen et al., 2020; Zhang et al., 2021c), adopting it to personalized FL is new as far as I have checked.\n- A new benchmark dataset (more precisely, data splitting protocol) is introduced for personalized FL.\n\n## Weak points\n- While I believe that the proposed method seems to have some technical merits, the current experimental results only demonstrate that \"the proposed method worked better than existing methods on the proposed datasets\", which is a bit less convincing to me. Overall, it is not very clear what motivates to develop a new benchmark dataset, though I appreciate the introduction of new dataset itself. In the paper, it was argued in the introduction that the existing PFL evaluations are (1) misleading due to the mismatches between training/test distributions and (2) less comprehensive because the non-iidness arises only in either labels or input domains, not both. While I agree with (2), I would think that train/test mismatches in (1) could happen in practice as well studied in domain adaptation and generalization. The paper could have been stronger if the proposed method is compared with baselines also on other datasets (i.e., data splitting scheme) used in prior works. \n- It remains not obvious if the proposed method really outperforms baselines from the current results. In Table 2, the performance gains from the most strong baseline (Per-FedAvg + FT) are less than 3 percent points in most cases, on average. How about the variance of test accuracy across clients and multiple runs with different random seeds? Such additional information is necessary to fully demonstrate that the performance improvements by the proposed method are statistically significant. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KjuyDsPGHwW", "reviewer": "Reviewer_2JAA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "First, this paper propose a personalized federated learning (PFL) method, FEDBASIS, based on coordinate descent trying to learn a set of bases for the overall system and personalized coefficients in federated learning setting. Second, a benchmark for PFL, PFLBED, is proposed discussing the procedures to build non-i.i.d. of joint distributions and to maintain reliable evaluation simultaneously.", "summary_of_the_review": "The ideas of FedBasis are not novel but the motivations of benchmarks are of interest. Some experiments are not solid enough to support their claims.", "strength_and_weaknesses": "Strengths:\n[+] An architecture and algorithm, FedBasis, is proposed to decouple ‘the required total number of personalized parameters from the number of clients’.\n[+] Novel benchmark for PFL: the motivation is worth considering.\n[+] Remarkable experiments’ results.\n\nWeaknesses:\n[-] It seems like that the straightforward ideas of FedBasis are not novel and fail to decouple the personalized parameters from the number of clients in practice due to the requirement of the set of basis and calculation of the gradient based on them during training (of the set of basis even during inference, ). Can you provide more explanation about decoupling rather than duplicating parameters with more copies?\n[-] The procedures of benchmark are not comparing to existing benchmark (e.g. LEAF) experimentally on discrepancy as you claim in Section 5. Can we employ them on any classic ML dataset?\n[-] What about test on LEAF benchmark? And the results are relevant to benchmarks or not?\n[-] FedBasis is claimed to be not sensitive to learning rate and training epoch, but Per-FedAvg is. How about to well-tune two versions of Per-FedAvg, FO and HF.\n[-] The poor experiments about robustness of personalization lead to low persuasiveness.Why not consider comparing methods to reduce structural risk and to use regularization to deal with it, such as pFedMe, since the clients may not have enough data for validation in practice?\n[-] The accuracy, loss, etc. of these training processes are not shown but only tables of acc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4wYqnlugnM", "reviewer": "Reviewer_ES1X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers a new personalized federated learning scenario where this federated training needs to learn some intrinsic model property and generalize this knowledge into new client. This considered scenario is interesting. To overcome it, this work attempts to construct the shareable model bases which each client utilizes to learn combination coefficient. In addition, this work evaluates the effectiveness of the proposed method on datasets with domain-shift.", "summary_of_the_review": "This work considers a new personalized federated learning scenario where this federated training needs to learn some intrinsic model property and generalize this knowledge into new client. This considered scenario is interesting. To overcome it, this work attempts to construct the shareable model bases which each client utilizes to learn combination coefficient. In addition, this work evaluates the effectiveness of the proposed method on datasets with domain-shift.\n\nHowever, I also have several concerns on this work:\n\n1. How to learn and select the representative model bases is not clear? \n\n2. When the new clients come from new domain or other new distributions, I wonder if the proposed method also can solve it. In addition, the idea of generalizing model into new clients (domains) is very similar with domain generalization (DG). Thus, it would be better to discuss the association and difference with some DG works.\n\n3. This work only conducts experimental comparisons with some baselines on new data setup, which will be not convincing. Thus, it would be better to add some experiments as the traditional PFL works (e.g. FedRep) on Cifar-10/100 or Fe-MNIST.\n", "strength_and_weaknesses": "Strength: \n\n1. The considered new PFL scenario is interesting and practical.\n\nWeaknesses: \n\n1. Some details of the proposed method are not clear.\n\n2. The experimental settings are not convincing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject