Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching
Authors: Wonyong Jeong
Sung Ju Hwang
Venue: NeurIPS 2022 Accept (2022)
Keywords: federated learning

Abstract: In real-world federated learning scenarios, participants could have their own personalized labels incompatible with those from other clients, due to using different label permutations or tackling completely different tasks or domains. However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use a synchronized set of labels, and (2) they train on the same tasks from the same domain. In this work, to tackle these challenges, we introduce Factorized-FL, which allows to effectively tackle label- and task-heterogeneous federated learning settings by factorizing the model parameters into a pair of rank-1 vectors, where one captures the common knowledge across different labels and tasks and the other captures knowledge specific to the task for each local model. Moreover, based on the distance in the client-specific vector space, Factorized-FL performs a selective aggregation scheme to utilize only the knowledge from the relevant participants for each client. We extensively validate our method on both label- and domain-heterogeneous settings, on which it outperforms the state-of-the-art personalized federated learning methods. The code is available at https://github.com/wyjeong/Factorized-FL.

Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching
Wonyong Jeong Graduate School of Al KAIST, Seoul, South Korea wyjeongčkaist. ac.
Sung Ju Hwang Graduate School of AI KAIST, Seoul, South Korea s jjhwang82kkaist. ac kr
Abstract
In real-world federated learning scenarios, participants could have their own person- alized labels incompatible with those from other clients, due to using different label permutations tackling completely different tasks domains. However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use synchronized set of labels, and (2) they train on the same tasks from the same domain. In this work, to tackle these challenges, tackle label- and husk-heterogeneous federated learning settings by factorizing the model parameters into a pair of rank-1 vectors, where captures the common knowledge across different labels and tasks and the other captures knowledge specific to the task for each local model. Moreover, based the distance in the client-specific vector space, Factorized-PP performs a selective aggregation scheme utilize only the knowledge from the relevant participants for each client. We extensively validate our method on both label- and cemainnheterogeneuus set- tings, on which outperforms the state-of-ther afrt personalized federated learning methods. The code is available at https:/ggtttbbboo///yooo//////// wow/wwjeong/Factoriziieee
introduce Factorized-FF which allows to effectively
Introduction
Personalized Federated Learning (PFL) aims to utilize the aggregated knowledge from other clients while learning a client-specific model that is specialized for its own task and data distribution, rather than learning universal, global model for tackling all local tasks [1, 16, 5, 28). While various personalized federated learning approaches have shown success in alleviating the data heterogeneity problem, yet, they also limited as they follow the common assumptions of the standard federated learning setting, that (1) all participants use a synchronized set of labels, and (2) clients tackle the same task from the same domain.
In many real-world scenarios, the first assumption may not hold since the labels for the same task could be differently annotated depending on the user environment, i.e. language or nationality (Figure 2 Left). In other words, when working with the same set semantic classes, the labels across multiple clients could be completely different. For example, the same "Car" class may be given the label "Vehicle" or "Auto".
The second assumption severely limits the pool of devices or institutions that can participate in the collaborative learning process. However, clients working on different tasks and domains may have similar classes, or a common underlying knowledge, that may be helpful for the local models being trained other clients (Figure Right). Thus, it would be helpful we can allow the local models with heterogeneous tasks and domains to communicate the common knowledge across them.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

DFFA-100
Comparison of Gradient Updates
Agnostic Personalized PL.
Clert upáioo
1. Disitennce from
MNIST Partion
MNIST T Partition Performance (36)
Heterogerenus
Perrazed
fcualty leto
Partiton
MNITT
WRIST Port iPeruttell
VUUST Port (Syncrorized)
OPAK, 10 ihoters Domain
ma CIFAk LG IHeters Domaini
HI POBST 2. (525rrorni ml WAIST Pt (Pomuui)
0
MNIST Parthlon
10
Local Training Steps
Local Training Epoch
Communication Round
(a) Illustration of the Parameter Space (b) Distance of Gradient Updates (c) Performance Degeneration
Figure I: Challenges of Label & Domain Heterogeneous Scenarios (a) illustrates label and domain hetero- geneity in parameter space, (b) shows normalized L2 distance of the gradient updates from that of model trained on MNIST Partition I. (c) shows performance degradation on MNIST partition caused by the label and domain heterogeneity while performing federated learning.
Agnostic Personalized Federated Learning
However, federated learning under label and domain het- erogeneity is a nontrivial problem, as most methods suf- fer from severe performance degeneration in such set- tings (Table 1). We analyze this phenomenon in Figure 1. Specifically, we train equally-initialiin different datasets and observe how different the gradient updates become as training goes on (we measure nor- malized 122 distance between them). Learning on two MNIST partitions will yield the smallest difference be- tween gradients (Figure 1 (a) and (b) Gray). Interestingly, simply permuting the labels of the partition 2 makes the gradient updates to largely diverge from the original gradients, compared to those of the model trained with synchronized labels (Figure I (a) and (b) Red). Moreover, learning on the completely different dataset (CIFAR-10) 10) makes model gradients diverge more severely compared to learning on the dataset with permuted labels (Figure 1 (a) and (b) Blue).
models on four
Label Heteroganoous Scanario
Domain Heteroganeous Scanario
Label and Domain Heterogeneous FL Scenarios. Left labels are not synchro- nized across all clients. Right the local clients learn on different tasks and/or domains.
Figure 2:
The heterogeneity in the labels and domains tackled by each participant leads to severe performance degeneration when performing federated learning (Figure (c)). We measure performance on MNIST partition 1 while aggregating model trained on the different dataset for every 3 epochs. We observe averaged models suffer from crucial performance degeneration in both FL scenarios. Note that the classifier layers not shared when performing federated learning. However, such label- and domain- heterogeneous FL settings have been relatively overlooked. Although few previous works [1, 16, 25] briefly introduced such extreme settings as motivations, they did not formally define the problems nor tackle such scenarios, and rather targeted standard non-lID scenarios.
We name this challenging problem as the Agnostic Personalized Federated Learning (APFL) problem, where participants with personalized labels or from multiple domains can collaboratively learn while benefiting each other. An APFL. problem has two critical challenges: (1) Label Heterogeneity for the discrepancy of the labels, due to the lack of synchronized labeling scheme across the clients; and (2) Domain Heterogeneity for the discrepancy in the task and domains tackled by each participant. To tackle these challenges, we propose a novel FL framework, dubbed Factor factorized-- -ze which factorizes model parameters into basis vectors and aggregate them in the factorized parameter space. This allows to factorize the model aggregation to take place in a semantic parameter basis space which is more robust to the use of different labels and data. Also, the factorization results in the separation of the client-generi and client-specific knowledge, and thus prevents the aggregation of incompatible knowledge across clients. Moreover, to further alleviate the model from collapsing into degenerate solution, we measure the task similarity across the clients using the factorized parameters, to allow selective aggregation of the knowledge among the relevant models that work similar tasks or domains. We extensively validate our method on both label- and mamainnheterogeneuus settings, and show that it significantly outperforms the current state-of-the-art methods with lower communication cost due to using factorized parameters. Our contributions can be summarized as follows:
We introduce the Agnostic Personalized Federated Learning (APFI.) problem and study its two critical challenges, Label and Domain Heterogeneity.
We propose Factorized-FL. for tackling the APFL problem, which factorizes model parameters to reduce parameter dimensionality for alleviating knowledge collapse, and utilize them to measure task- similarity for matching relevant clients.
We validate our method in both label- and domain- heterogeneous scenarios and show our method outperforms the current state-of of- the-art personalized federated learning methods.
2

---

2 Related Work
Federated learning Many federated learning algorithms have been proposed since the introduction of FedAvg [18], but, we specifically focus works that aim tackle the heterogeneity problems, e.g. Non-IIDness, Some studies focus on regularization methods [19, 15], correcting disparity between server and clients [26, 9], or contrastive learning [13]. Recently, many existing works also tackle architecture-level heterogeneity [24, 29, 3].
Personalized federated learning aims to improve the individual local models instead of the universal global model, via the mixture methods [17, 2. 7], meta-lcarning [5]. partial network aggregation [I, 16], or hyper-networks [25]. Recent approaches aim to enhance personalization performance by avoiding to aggregate irrelevant knowledge that are not helpful. Zhang et al. [28] download the models from other clients to aggregate only the beneficial ones for the local task at each client. Sattler et al. [23], Duan et al. [4] measure client-wiss -iiee similarity by using the gradient updates. Our method also measures client similarity but in more efficient way, by utilizing 1D vectors from factorization.
Re-parameterization for federated learning Jeong et al. [8], Yoon et al. [27] decompose model parameters (using an additional set of parameters) to train them with different objectives, which do not reduce the dimensionality of model parameters, Some approaches factorize high dimensional model parameters into low rank matrices. Konečný et al. [10] introduce structured update which model directly learns factorized parameter space, and Nam et al. [20] propose to use the Hadamard product of low rank matrices to enhance communication efficiency. Unlike prior works, utilize rank-1 I vectors factorization to tackle heterogeneity in domains and tasks, by allowing the 1D vectors to capture task- besk-genrral and the client-specifie knowledge. Also, minimize the information loss from factorization by utilizing sparse bias matrices.
3 Problem Definition
We begin with the formal definition of the conventional federated learning scenario, and then introduce our novel Agnostic Personalized Federated Learning (APFL) problem.
3.1 Preliminaries
Our goal is to solve multi-class classification task under FL setting. Let fg be global model (neural network) at the global server and F { be set of K local neural networks, where K is the number of local clients. D = {x,, ya i= be given dataset, where N is the number of instances, E x D is the ish examples in a size of width W, height H, and depth D, with a corresponding target label y E fl....,CC C} for the C-way multi-class classification problem. The given dataset
D is then disjointly split into K sub-partitions D = {xx,i, Ne ) s.t. D = U Dr. which are distributed the corresponding local model fx- Let R be the total number of the communication rounds and r denote the index of the Tth communication round. At the initial round r=1, the global model f initializes the global weights 0% and broadcasts g( to an arbitrary subset of local models that are available for training round r, such that Fi() c F, |](r)]= = KO), and K(() < K, where k(()) is the number of available local models at round Then the active local models Ss F(r) perform local training to minimize loss L(o(t)) on the corresponding sub-partition Ps. and update their local weights of (+++1 <- o(²) L/V where of*) is the set of weights for the local model fü at round and (()) the loss function. When the local training is done, the global model F
collects and aggregates the learned weights ffr+2) +
N Di=1 a(?) and then broadcasts newly updated weights to the local models available at the next round r + 1. These learning procedures are repeated until the final round R.
No k()))
On the other hand, Personalized Federated Learning aims to adapt the individual local models fi.k to their local data distribution Di.k, to obtain specialized solution for each task at the local client, while utilizing the knowledge from other clients. Thus merging the local knowledge for personalized FL
is not necessarily done in the form of g No
N Σk'r o and the specific ways utilize the
knowledge from others depends on the specific algorithm, i.e. g(r+1) t o6 KK(r) wi((k)) -
where w(-) is weighing function [28].
3

---

Conventional Factorization
Hadamard Product of Low Rank Matrices
Rank 1 Vector
Highly Sparse Blas (Ours)
B¹ 'any)
Bi
B (o0p)
W (mcki))
A2 Lawneng
(nx1)
(enive)
W2
(noc))
Low Matrix
Global Weights
Local Weights
Global Vector
(Filiar Baxa))
(Fiter Coufiisiot)
Local Vector
Local Blas [Sparss Matrid
A
B
Wi
W, O W2
Figure 3: Illustration of Parameter Factorization Methods: Left shows conventional matrix factorization with two low rank matrices with rank Middle represents the method utilizing Hadamard product of low rank matrix for federated learning [20]. Right illustrates our factorization method for agnostic personalized federated learning, which utilizes rank vectors and highly sparse bias.
3.2 Agnostic Personalized Federated Learning
Agnostic Personalized Federated Learning (APFL) is a scenario where any local participants from diverse domains with their own personalized labeling schemes can collaboratively learn, benefiting each other. There exist two critical challenges that need to be tackled to achieve this objective: (1) Label Heterogeneity and (2) Domain Heterogeneity.
This scenario assumes that the labeling schemes nized across all clients, as described in Section and Figure Left. Most underlying setting for this scenario is the same as the conventional single-domain setting with synchronized labels that is described in Section 3. except that labels are arbitrarily permuted amongst clients. The local data Dj for the local model f& is now defined as Dr = {xx, 9P ()x,,)) N₂, i=1. where (xx(.) is a mapping function for the local model f& which maps given class ykei with a randomly permuted label (y...)) Let the jth layer out of L layers in the neural networks of local model fx be &g and the last layer t be the classifier layer. Since each client has differently permuted labels, the personalized classifiers 4. are no longer compatible to each other. While we can merge the layers below the classifier in this setting, training with heterogeneous labels could still lead to large disparity in the local gradients even in the initial communication round, as described in Figure
Label Heterogeneity
perfectly synchro-
Domain Heterogeneity This scenario presumes that local clients learn on their own dataset D, that completely different from the datasets that are used at other clients, described in Section and Figure 2 Right. In this setting, K disjoint datasets Di.K are assigned to the K local clients fi.. where Dr = i : is the dataset assigned to the local model ft- The number of target classes may differ across clients, such that yk {1, f1..... Ck}. We assume complete disjointness across clients, such that there is no instance- and class-wise overlap across the datasets: Ø = n D& and
K
Ø 1 Ca. where Cs is set of classes for client k. Similarly to the label-heterogennouus scenario, the personalized classifiers ff. are no longer compatible to each other due to the heterogeneity in the data and the labels. Hence, the aggregation done for the layers before the classifier, but they
will be also incompatible as the learned model weights will be largely different across domains.
4 Factorized Federated Learning
4.1 Factorization of Model Parameters
Wang et al. [26] discussed that the conventional knowledge aggregation, that is often performed in a coordinate-wise manner, may have severe detrimental effects on the averaged model. This because the deep neural networks have extremely high-dimensional parameters and thus meaningful element-wir wise neural matching is not guaranteed when aggregating the weights across different models trained under diverse settings.
One naive solution to this problem is to factorize model parameters into lower dimensional space, iee. low rank matrices, as shown in Figure 3 (Left). Conventional approaches, such as SVD, Tucker, or Canonical Polyadic decomposition, however, factorize model parameters after training [12. 22] is done. Thus, the dimensionality at the time of knowledge aggregation will remain the same as the unfactorized model. Konečný et [10], Nam et al. [20] pre-decompose model parameters to low rank matrices for FL scenarios. While Konečný et al. [10] use naive low rank matrices, Nam et al. [20] use two sets of low rank matrices to improve expressiveness and utilize them as global and local weights (Figure (Middle)). Unlike prior works, our approach utilizes rank- I vectors to

---

perform aggregation in the lowest subspace possible for compatibility, while effectively yet efficiently enhancing expressiveness with sparse bias matrices, as shown Figure (Right) and Figure 5 (Left). Another crucial difference of our method from the previous factorization methods is that our rank-1 vectors have distinet roles. Our factorization will separate the common knowledge from the task- or domain-specific knowledge, since could be thought as the bases (the common knowledge across clients) and v could be thought as the coefficients (client-specific information).
In Figure 4 (a) and (b), which show the experimental results with the factorized model, we observe that trained on two datasets becomes closer to each other (MNIST Partition 1) while (personalized fil- ter coefficient) remain largely different as FL goes on. With this observation, we further aggregate u while allowing v to be different across clients, to allow per- sonalized FI.. Further, we use the client specific for similarity matching, to identify relevant local models from other clients. In the following paragraphs, we describe our factorization method in detail, for both fully-connected and convolutional layers.
Cemmaaysee dirve to
Dieance from
Viiitar Couflkond)
Offier Cooff.
her Baeri
(Tiber Bare)
LOCA Tranring Spach
Local Training Epoch
(a) Hetero. Labels (b) Hetero, Domains Figure 4: Analysis of and v: The normalized L distance of the gradient updates of u and while learning on (a) MNIST Partition and (b) CIFAR- 10 compared to learning on MNIST Partition 1.
Factorization of Fully- CollneCende Layers We assume that each local model fx has a set of local weights 0g across all layers; that is, 0g (W The dimensionality of the dense weight W for each fully connected layer W k E R where and O indicate respective input and output dimensions. We can reduce the I x O complexity by factorizing the high order matrix into the outer product of two vectors as follows:
W
x where ul RR V E R
However, such extreme factorization of the weight matrices may result in the loss of expressiveness in the parameter space. Thus, we additionally introduce highly sparse bias matrix μ to further capture the information not captured by the outer product of the two vectors as follows:
w u x @ ph where ij E R
ER. ud E /
We initialize with zeros so that it can gradually capture the additional expressiveness that are not captured by u and during training. We can control its sparsity by the apper-parameter for the sparsity regularizer described in 4.3.
Factorization of Convolutional Layers The difference between the fully-connected and con- volutional layers is that the convolutional layers have multiple kernels (or filters) such that W E FxFxxxxx where F is a size of filters (we assume the filter size is equally paired for the simplicity). To induce to capture base filter knowledge and to learn filter coefficient, it is es- sential to design u E IR F.F and v E RTO but not in arbitrary ways, such as u E T.F and v E RO-F or E Oo and IR.FFF We observe that performance is degenerated when the parameters are ambiguously factorized (Figure 7 (h)). Our proposed factorization method for convolutional layers follows:
W = (u} V @ pb),, where u' E FR v R..
(3)
M ERF-XIXO ,T()) RE-FXIO FxPxIXO
(()) is the weight reshaping function. Note that we reparameterize our model at initialization time. Then we reconstruct and train full weights of each layer With, while optimizing u I:I. I.I. and μl I:I. respectively, during training phase.
v
4.2 Similarity Matching
Since we assume label- and Nomain-heterogeneon FL scenarios, aggregating the parameter bases across all clients may not be optimal, since some of them could be highly irrelevant. Yoon et al. [27] and Zhang et al. [28] also demonstrate that avoiding aggregation of irrelevant models from other clients improves local model performance. Yoon et al. [27] achieve this goal by taking the weighted combination of task-specific weights from other clients, and Zhang et al. [28] suggest downloading the models from other clients and evaluating their performance on a local validation set, at each client. However, since they require additional communication and computing cost at the local clients, we provide more efficient yet effective approach to match models that are beneficial to each other.
5

---

Filter Base
Filtrr Conficients
Rected Barnels
Factorized Parameter Space
12345 1.0
Client2 Client
PESCEGEGE
1-
0.25
Filter
1,0
Sparse Bon Maetre
Filter
0.65
Client K
Similarity Matching
Xv
Reshaping
(a) Factorization for Base and Specific Knowledge
(b) Similarity Matching on Factorized Space
Figure 5: Illustration of the parameter factorization & similarity matching algorithm (a) We multiply two factorized vectors u and to obtain kemel matrix. Then we add the sparse bias matrix j to complement non-lincarity and reshape the matrix into the original kemee shape. (b) We match relevant clients utilizing the factorized vector that captures client-specific knowledge. Then we aggregate based the similarity.
Efficient similarity matching Our method utilizes factorized vector Vi for measuring similarity across different models, at the central server. Since v are devised to learn personalized coefficients, assume that clients trained on similar tasks domains will have similar Specifically, ---1 of the layer before the classifier layer for similarity matching. The similarity matching function as
Vg 22.), is defined as the cosine similarity between target client fü and the other clients }
{fr
follows:
M(y Fopein (0sl0;
2 Tlick
(4)
The similarity scores for those with the cosine similarity scores lower than the given threshold т, are set to zero. Our method is significantly more efficient than the similarity matching approaches which use full gradient updates for clustering clients [23, 4], as we only use single vector v from each client that are obtained without additional cost.
2-11
Personalized weighted averaging We allow each local model to individually perform weighted aggregation of the model weights from other clients, utilizing their similarity scores:
exple.co o.) EK explc- O.)
Σ
u s... f1,2...... 13.
(5)
where is hyperparameter for scaling the similarity score d;- We always set dk, the similarity score for itself, as 1.0. We name this model that aggregates only basis vector ug from each client as Factorized-FL. a. This will result in encoding common knowledge across local models in ug which is especially suitable for domain-heterogeneonn scenarios. However, for hebl--hetroogeeouus scenarios, we empirically observed that aggregating all parameters results in better performance, as the factorization helps averaging them in more compatible space. We thus introduce an additional variant, namely Factorized-FL. B, which aggregates only the parameter bases across all clients, but also coefficient and bias terms as well. Yet, note that Factorized-FL. 3 require significantly larger communication overhead over Factorized-FL. a.
4.3 Learning Objective
Now we describe our final learning objective. Instead of utilizing the single term 0g for local weights of neural network fk. now let Uk. Vk. and M& be sets of ug, Vk, and μ4 of all layers in fk. s.t. Ui = H- VE (vi LL and M = = He then our local objective function is,
Σ C(B:UE, VR, + Aepuriity ||Mall1.
min
24. V.
BEDa
where the standard cross-entropy loss performed on minibatch E Dr. We add the L sparsity inducing regularization term to make the bias parameters highly sparse, controlling its effect with hyperparameter Aspaniity As for the full training procedure, please see the pseudo-code of the algorithm in the supplementary file (Section A).

---

Table I: Performance comparison of label and domain heterogeneous scenario. Top (label heteregeneous scenario): train 20 clients each dataset (CIFAR-10 & SVHN for 250 training iterations ( (F=5, R=50). Bottom (domain & label heterogeneous scenario); We train clients for 500 training iterations Rb=5,)=100)
on 20 sub-datasets from 5 heterogeneous domains (4 clients per domain). Labels are also permuted for all
partitions. We measure averaged performance over three trials with different seeds. (Table I Bottom will be moved to Appendix and new set of experiments for the pure domain heterogeneous setting will be added.)
Datased
Method
Accuracy
Stanfard ITD
Cost [Gb)
Accuracy
Permited LED
Cost (Gb)
Accuracy (5)
Standard Non-TID
Cost (Gh)
Accuracy [%]
Permuted Non-IID
Cost [Gbb
Stand-Alene
64.31
63.99
17.79
46.06
Focting (18)
70.28 (5:
20.39
1.20
20.39
tt:
FedPros [34]
70154 0:33
20.39
20139
47.56 daz)
CIFAR-10
Per Foddng [5]
Clustered-Rt (23)
69.4 (2: (0)
70.84 (t L00)
20.39
20.30
65.39 58 0:50
20.33
20.39
$3.35 1277
51.93 1550
20.39
20.39
49 (2 133
47.50 1800
20.39
PaESROMD (28)
70.39 c1 0.79
122.33
64.22 26 c2 wet)
122.53
50.69 (1: 166
122.35
46.73 (1 1000)
122.55
predPara (20)
67.96 (s: 1.25
7.4
65. 12 127)
7.4
55 55.88 1.300
7.4
50.23 (::0
7.4
Factorized EL o (Ours)
66.97 (t: 1.30)
0.32
67.91 189)
0.32
50.34 1.39
0.32
50.24 (s: 1.0)
0.32
Factorized-FL FL / (Ours)
76.26 1.5)
18.25
70.59 2020
18.25
65.30 1.30
18:25
56.61 1.30)
18.25
Stand Alone
$4.15 0.33)
84.32 (c: 6331
62, 30 ct: bo8))
62.11 (t 0.19)
Fecking [18)
R8.55 0.13
20.30
87 83 cL. (.24)
20.33
76. 76.03% (1. D000
20.39
69.73 (1 (.13)
20.39
FesProx [14]
$9.04 (::33.3
20.39
87.31 cz 020
20.39
76.61 ctt 6000
20.39
69. 10.40 (2 0.73)
20.39
SVHN
Cleatered-FF EL [23)
Pwr-Peckog kog [5]
s9,42 0.37)
18.4661.-99)
20.39
30.99
87 79 (2 (M)
87.35 13 (220
20.30
20.39
74,27
74 90
20.39
20.39
08..4 d.4)
68.67 (2:075)
30.59
20.39
Foaroso (28)
$8.34
122.33
84.03
122.33
72.12
122.35
61.45 0.80)
122.33
pPadPare (30)
70.26
Factorized o (Ouns)
36 16.56 (2:0
022)
66, 0/7
66.12 (2:0
Factorised-FL / (Qurs)
91.04 ca 8.73)
18:25
to (4b)
18:25
81,07 (SS
18:25
74.63 4.44
18.25
Method
Houschold
Fruit& Food
Tree&Flower
Transport
Animals Accuracy [9t
AVERAGE
Accuracy [%]
Accuracy [%]
Accuracy [%]
Accuracy [98]
Accurccy [%)
[Gb]
Alond-Alone
59.38 0.70
63.74 (a 1.70
61.20 (= 160
63.22 2.10
S8,40 1.200
35 1,900
FodAvg [18]
55,08 2.49)
2.5)
57.76 1.7)
57.96 1.7)
(#:
56.42 L.55)
FedProx [14]
56.77 (:: 2,59
61.33 (t. 1.280
58.14 c= a50
55.79 c2 (482)
51,43 2.17)
56.71 (: 1.52)
40.78
Cluatered-FL [23]
59.44 2:31)
66.93 ca use
60.03
1.35)
62.17 2.55)
55.01 ca (2:3:11)
59:20 2,100
40.78
Per-Fedavg [5]
1.50)
61.62
1.50
64.36 1.77
(=
62.92 1.63)
40.78
FadPOMa [28)
59.70 cE: 1.7k)
64.32 (t: 1.40)
63.87
(a 2:19)
62.57 (297)
57.75 cl 2:281
62.07 (:: 1.80)
244.56
pFadPara [20)
60.35 3,30
63.56
61.98
2000
60.16 6:60
66.12 2.0)
61.11 761)
15.98
Pactorized-FL FL (Ours)
64,06 (t: 0.16)
68.55 (= 0.16
61.39 (=: 2:23)
66.93 (z 1.89)
61.33 (s: 3,50)
64.49 (t: 1.97)
0.64
Factorized-FL (Oun))
65.004 (a: 8.7)
65,53 (a 2:79
68.14 (= 164)
65.04 c2 185)
64.45 ct 3.47)
63.93 2.25)
36.5
Standard IID (0F
Permuted tD (CIFAR-10)
Standard NenilD (CIFAR-10)
Permuted NonilD (CIFAR-10)
Hatero- Domain (C1F
Communication Round
co Round
Communicasion Round
Communication Round
Communiccaton Round
Standard ID (SVHN)
Permuzed ID ISVHN)
Standard NoniD (SVHN)
Permuzed Nonil (SVHN)
Hatero- Domain (CIFAR-100)
Is faerro
iss Aoemooy
iss bai
Accanayy
Communicatton Round
Communication Round
Communication Round
Communication Round
Transiassion Cost (Go)
(a) IID
(b) Permuted IID
(c) NonIID
(d) Permuted NonlID
(c) Hetero-Doman
Figure 6: Test Accuracy Curves & Communication Costs We plot the averaged test accuracy curves on both
CIFAR-10 and SVHN for (a) IID, (B) Permuted IID, (c) Non-III, and (d) Permuted Non-IID partitions. For (e),
we plot test accuracy curves (top) and communication costs (hottom) for domain & label heterogencous dataset.
All plots for communication costs are represented in the supplementary matcrial.
5 Experiment
5.1 Experimental Setup
Baselines We first consider well-known baseline FL methods, such as (1) FedAvg [18] and (2) FedProx [14]. We evaluate our factorization technique with (3) pFedPara [20] which also uses kernel factorization technique for personalized FL scenarios. Our similarity matching approach is compared to (4) Clustered-Fi [23] and (5) FedFOMO [28], which measuring client- similarity or helpfulness. (6) Per-FedAvg [5] is also used for evaluation as it shows great performance on heterogeneous federated learning scenarios. We also show local training model, (7) Stand-Alone for the lower bound performance. Detailed information are described in Appendix (Section C).
Datasets (1) Label Heterogeneous Scenario: we use CIFAR- 10 [11] and SVHN [21] datasets and create four different partitions for each dataset, which are conventional IID and non-lID as well
7

---

& / Anolysis Matching
Vorn Naien kg
Client
Communication Bound
(a) r=1
(b) 2=2 (c) 50 (d) r=75 (e) 1000
(l) Freq. of Client Matching
(g) Ablation Study
Rink
Threshold
Backhere
Apands
Paraas
Nun
Aue.
196)
Bookrooe
Nun
PC
Heal
BX
Regular
Kenel
Rasseioaiion
FxO F
Pictensation (Oun)
F FXX
Acc 196)
Acc: [%6]
RooNes-
(1005))
50%
81-89%%
d0.e%
62,34%
ResNie
3
10
63.27
64.03
64.15
0.75
0.85
0.5
62.83
61.06
63.59
Fasteizzl
RodNe-9
Se-4
70-4
3e-
1o-3
2,253 ki (885))
2,515 MY (971))
1.744 (8995)
1,006 M1 (8065)
03. 13.
67.67%
03.605%
Re
53,2555
50.01%
58 365
68.05%
61 1113
56.72%
68.82%
68.42%
67.93%
66.23%
64.39%
(h) 7
(i)
() Aspursity
(k) Analysis of Factorization Methods
Figure 7: Analysis on Factorized-FL. algorithms (a) and (e): While - show strong similarity across all clients (upper row), v show exclusive similarity hased on domains (bottom row). Darker colors indicate higher scores, (f): the frequency of client matching after 100 rounds. (g): component analysis. (h), (i), and (j): analysis on 7. T, and A. (k) the effect of how to factorize model parameters
as permuted IID and permuted non-IID, whose labels are permuted and incompatible to each other. We split the datasets into 20 partitions and then simply permuted the labels on the same partition for label permuted settings. (2) Domain Heterogeneous Scenario: we use CIFAR-100 datasets [1 and create five sub-datasets grouped by 10 similar classes, such as Household Objects, Foods, Flower,, Transport, and Animals. We assign clients for each sub-dataset, i.e. Client 1-4 to Household Objects, Client 5-8 to Fruits&Fodds, Client 9-12 to Trees&Flowents Client 13-16 to Transport, and Client 17-20 to Animals. We then permute the labels for all partitions to simulate further realistic scenarios. Further descriptions are elaborated in Appendix (Section B).
5.2 Experimental Result
Label-heterogeneous FL As shown in Table (Top), for the standard IID and Non-IID settings, all FL methods obtain higher performance than the local training baseline SStand-Alonee which con- firms that the locally learned knowledge is beneficial to others, when the data and label distributions homogeneous across clients. However, when the labels (Permuted IID/Non-IIDD, all previous FI. methods suffer from significant performance degeneration, even lower than that of the local training baseline. Again, note that we do not share the classifier layers to ensure fairness across all algorithms in these permuted settings. We conjecture that this is caused by the label permutation leading the local model to evolve a permuted set of features that are not coordinate- compatible to others when aggregated. Contrarily, our method Factorized shows consistent performance regardless of whether the labels are permuted or not. Factorized-FL. 3 even largely outperforms all baselines with significantly superior performance. Test accuracy curves over the communication round and transmission cost are Appendix (Section D).
not synchronized across all clients
a
Nomain-heterogeneouu FL Table 1 (Bottom) shows the experimental results for the domain and label heterogeneous scenarios. We observe that the conventional FL baselines, i.e. FedAvg, FedProx, fail to obtain better performance over purely local training baseline (Stand-Alone) due to the naive aggregation of extremely heterogeneous knowledge, which causes detrimental knowledge collapse. FedFOMO and Clustered-FL shows slightly higher performance (1-2%p) over Stand Alone model, as they can avoid irrelevant clients when aggregating local knowledge. The other personalized FL methods, i.c. Per-FedAvg and pFedPara. also show 1-2%p higher performance over Stand Ane model as they are specialized for personalized FL scenarios. However, on average, our method largely outperforms all baseline models even with the smallest communication costs (Factorized-FL o), as shown in Figure 6 (e). In the figures, we plot the convergence rate of our Factorized-FP over communication round (Top) and transmission cost (Bottom), compared to the baseline models. Both models consistently obtain superior performance compared to the other models. Particularly, unlike pFedPara which uses low rank matrices for knowledge sharing, Factorized-FL. a only communicates with factorized vectors, such as 2/ for base knowledge sharing and vL- for similarity matching and this allows our method to largely reduce the communication costs while showing the best performance as shown in Figure (e) Bottom.
8

---

Effect of kernel factorization In Figure 7 (g), we perform an ablation study of our factorization method in the nomain-heterogenne tion methods, we compare Factorized matching, against FedAvg. As shown, Factorized-FedAvg achieves higher performance over the original FedAvg model. As the only difference between the two is whether kernel is factorized or not, this clearly demonstrates that our factorization method alone improves the model performance by alleviating knowledge collapse. We further analyze the effect of the sparse bias matrix M. When we remove M from the Factorized -FedAvg model, Sactorized-FedMee observe large performance drop. This shows that that the bias term is essential in compensating for the loss of expressiveness from the rather extreme factorization. With only Li and V. we use 90% less model parameters (0.2 (M) compared to the regular kemel model (2.574M). Also, in Figure 7 (k), we observe that model factorized by u E R. and V E R R.O perform better than the model factorized by E R and TRO-F. This is because in the former case, the factorization will separate the base knowledge from the task-specific knowledge, as descrihed in Figure 5 (Left), but in the latter case the factorization does not have such natural interpretation. We also show the scalability and reliabillity of our factorization method with different architectures, consisting of convolutional, fully-connected layers, batch normalization, and skip connections as shown in Figure 7 (k).
scenario. To clearly see the effectiveness of our factoriza-
variant of Factorized-FL. 33 without similarity
w/o μ in the figure, we
Effect of similarity matching To verify the effectiveness of our similarity matching method using the personalized factorized vector v, we visualize the inter-elient similarity of u/,"" and v from the second last layer of 20 clients in a domain heterogeneous setting. As shown in Figure 7 from (a) round 1 to (e) round 100, we observe u f--1 (upper row) are indeed highly correlated with other clients (the darker color indicate higher similarity scores) while V (bottom row) are relatively uncorrelated to each other, as expected our assumption that Uf capture base knowledge across all clients and 5 capture personalized knowledge. Further, we also observe that v obtains higher similarity to that of the parameters trained on the same domains (but with permuted labels), ie. Client 1-4, Client 5-8, Client 9-12, Client 13-16, and Client 17-20, showing that they are effective in capturing task- and domain- similarities across models. We further visualize the frequency of the elient matching across clients in Figure (f), after 100 communication rounds. With only a single vector parameter v/= our method finds which clients's 's knowledge will be helpful to which clients. For further analysis on our similarity matching, we compare it against Random and Worst Matching baselines under the multi domain scenario. The random matching baseline randomly selects three arbitrary models to be aggregated at each round, while the worst matching baseline selects three most dissimilar models. As shown in Table 7 (g), both baselines significantly suffer from the performance degeneration compared to our similarity matching algorithm, which shows its effectivencss.
fn.
Analysis on 7 and Asparsity Figure 7 (h) and (i) show the analysis of the hyperparameters, y and 7, on the domain- -etrrogneetoggeeuuus dataset. As shown in Figure 7 (h), the performance slightly decreases with larger y, since higher rank weight matrices may result in larger coordinate-wiss incompatibility of the parameters, which could lead to knowledge collapse when performing aggregation. Also, the lower 7s show the lower performance, as less similar knowledge can be aggregated more frequently. In Figure 7 (i). we train Client 1 on CIFAR-10 10 IID Partition for 20 epochs using both factorized and regular models. As shown, our factorized model outperforms regular models (2.574M) even with 30% less parameters (1.784M). We further analyze the sparsity in Section D.2 of Appendix.
Conclusion
We introduced a realistic federated learning scenario where the labeling schemes are not synchronized across all participants (label heterogeneity) and the tasks and the domains tackled by each local model are different from those of others (domain heterogeneity). We then proposed novel method to tackle this problem, whose local model weights are factorized into the product of two vectors plus a sparse bias term. We then aggregate only the first vectors, for them capture the common knowledge across clients, while allowing the other vectors and the sparse bias term to be client-specific, accounting for label and domain heterogeneity. Further, we use the -lient-specific vectors to measure the similarity scores across local models, which are then used for weighted averaging, for personalized federated learning of each local model. Our method not only avoids knowledge collapse from aggregating incompatible parameters across heterogeneous models, but also significantly reduces the communication costs. We validate our method on both label and domain heterogeneous settings, on which it largely outperforms relevant basclines.

---

Limitation and Potential Societal Impacts
Limitations Federated Learning (FL) is devised to protect the privacy of sensitive personal data by strictly avoiding sharing them with other users, institutions, and companies. The common limitation in FL is that, even though almost all existing methods regard sharing locally learned parameters of gradients distribution, which violates the preservation of privacy. Although our work alleviates this probability by significantly reducing the dimensionality of parameters and partially sharing them, it may be still sub- sub-optimal in an aspect of security, which is an open research domain for federated learning.
safe, they still may contain some possibility to reproduce the local data
Potential Societal Impact Federated learning is a collaborative learning framework where multiple clients actively participate in the learning procedure anywhere and anytime. Nowadays, numerous terminal devices, i.e. smartphones, smartwatches, Io'I' devices, etc, are utilized as participants of the collaborative learning scheme. However, as the number of participants in federated learning increases, the number of communications between them increases accordingly. Such situations leading to the exhaustive communication costs may worsen the global environmental crisis, i.e. global warming. Even though our method can alleviate such risks, since ours significantly reduces the communication costs by partially sharing the factorized vectors, it essential to develop further optimal solutions for compressing the knowledge in both efficient and effective ways for federated learning that the massive number of participants collaborate.
Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IIT)) grant funded by the Korea goVermmen(MMITT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST) It was also results of a study on the "HPC Support" Project, supported by the 'Ministry of Science and ICT' and NIPA.
References
[1] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers, 2019.
[2] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, 2020.
[3] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2021.
[4] Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Fedgroup: Efficient clustered federated learning via decomposed data-driven measure, 2021.
[5] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach, 2020.
[6] Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors. Proceed- ings of the 34th International Conference on Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017. URL https://procesdingg
mlr press/v70/finnt html.
[7] Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models, 2021.
[8] Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learn- ing Representations, 2021. URL https:///pennrevirr.ct www.eeeeeetttteteeeeeeoooaeereee /ttp://mmmdd
10

---

under [9] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. 2021.
[10] Jakub Konečný, H Brendan McMahan, Felix X
Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprini
2016.
[11] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[12] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv: 1412.655,, 2014.
[13] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning, 2021.
[14] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv: 2018.
[15] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020.
[16] Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy P. Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philipee Morency. Think locally, act globally: Federated learning with local and global representations, 2020.
[17] Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning, 2020.
[18] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efinn effficientt learning of deep networks from decentralized data. In AISTATS, 2017.
[19] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning, 2019.
[20] Hyeon- Woo Nam, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for cetpruuiiiaaiiofffffiieeee federated learning. International Conference on Learning Reprexentutions, 2022. URL https: /wwp//eeeeeeeeeeeeeeeeeeee review.
[21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011. URL hatt:://flddd stanford. eetu/housemmberr// htwpe/emmmmmereree/e/eeeeeeeeeee http:/ippppp1//1 how.oetubessssebbrr...oooo pdf.
[22] Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network, 2020.
[23] Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multi-task optimization under privacy constraints, 2019.
[24] Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyur Kim. Federated knowledge distillation, 2020.
[25] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. arXiv preprint arXiv:2103. 04628,, 2021.
[26] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papaitiopoulos, and Yasaman Khaz- aeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020. URL https:://pperreiieee
[27] Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Funho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer, 2021.

---

[28] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In Internationnal Conference on Learning Representations, 2021. URI. https: htttps://ppenrritiii
[29] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou, Data-free knowledge distillation for heteroge- neous federated learning, 2021.
12


Reviews:
[{"review_id": "y2a72xISaVu", "reviewer": "Reviewer_dsaH", "summary": "This paper considers two new heterogeneous FL settings, (a) hetero-label FL where the clients have different labeling orders, i.e., the same label space but the ground-truth ids are different permutations across clients, and (b) hetero-domain FL where the clients have disjoint datasets in both instance and label class. To deal with these two challenging scenarios, the authors propose to factorize the model parameters into the product of two vectors plus a sparse bias term, in which the first vector is personalized for a specific client and the second one is shared with all participants. Experiments on CIFAR-10/100 and SVHN datasets with various partition manners show promising results with the proposed method. The studied problem and proposed method are interesting, and the paper has a clear organization. However, I have several concerns about the practicability of the hetero-label FL setting, the writing quality, and the soundness of the paper.\n", "questions": "See questions mentioned in the above weaknesses. ", "limitations": "IMO, there is no potential negative social impact.\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed hetero-domain FL setting is novel and practical, which has good potential to extend the application and research scope of FL.\n2. The paper has a clear organization and the proposed factorization method is easy to follow.\n3. The authors propose to capture client-general and client-specific knowledge via a global-shared base vector and local-personalized coefficient vectors, respectively, which is intuitive and seems to be simple yet effective.\n\nWeaknesses:\n1. The practicability of the hetero-label FL setting is questionable. I agree that clients may have different label spaces as different labeling schemes can be adopted. However, the label permutation seems a bit strange and tailored for the simulation, as the clients in realistic applications can easily reorder the labels to make their label orders consistent with negligible computation and communication costs (at the beginning of the FL courses and communicating only once).\nFurther, if reordering information is forbidden to be exchanged, the users can also leverage privacy-preserving entity resolution techniques to align the label IDs, which are well-studied in vertical FL literature such as [1].\n\n2. The proposed factorization method is simple, while the authors have not given more in-depth analysis and discussions on such topics as (1) The convergence of the proposed method since the converged communication round is also important in FL; (2) The expansibility of the proposed method in terms of the model structures. The authors focus on the fully-connected and convolutional layers. Can the proposed method be applied to other popular and practical NN models containing BN layers, residual structures, attention structures, etc., (3) The space-time complexity analysis with the proposed factorization method. The authors claimed that the proposed method improved FL efficiency multiple times. However, a more formal time-space complexity analysis is lacking in terms of FL training, inference, and communication processes.\n\n3. The experimental datasets should not be applied with the label permutation when examining the performance in the domain-heterogeneous FL setting (see line 265 and paragraph 280). The permutation mixed the effect of label heterogeneity, making the analysis in paragraph 280 of domain heterogeneity confounded. Besides, the number of experimental clients is a bit small (20 clients usually correspond to cross-silo FL). How does the proposed method perform on larger client scales and cross-device FL scenarios?\n\n4. The writing has a lot of room for improvement in terms of preciseness, typos, and grammatical mistakes. For example,\n- line 23, \"the same set of labels that are in the same order\" The elements in a set are *un-ordered*.\n- line 43, the numbering of the figures and the order in which the figures appear do not match: Figure 1 appeared after Figure 2\n- line 44, the term \"instance-wise manner\" is un-clear\n- line 36 ~ 50. the example is a bit hard to read and it is unclear whether the authors make the classifier layers personalized to each client in the example. Without the classifier layer personalization, the gradients' divergence is obvious and comparison is trivial. \n- line 90, downloads -> download\n- line 95, use -> using\n- line 97, introduces -> introduce\n- line 115, initialize -> initializes\n- line 127, utilized -> utilize\n- line 129, wher -> where\n- line 139 and line 149. there are two notations for the client datasets, P_k and D_k. The authors used D_k in equation (6) and how about P_k.\n- line 151, the last equation only expresses that there is no instance-wise overlap, while not mentioning the class-wise overlap.\n- line 169, uses -> use\n- line 177, shows -> show\n- equation 1 and equation 2 lack periods\n- line 211, another term \"task-heterogeneous FL\", compared to the term \"label-hetero FL\", it is better to unify them\n- line 213, demonstrated -> demonstrate\n- line 214, achieves -> achieve\n- line 250, Models -> Baselines,  This section lacks an introduction to the adopted models\n- line 257, missing citations for CIFAR-10 and SVHN\n- line 259, which labels -> whose labels\n- line 331, \"As shown, \" incomplete sentence\n- in appendix Figure 4, there is a reference error: \"Table ??\"\n\nReference\n\n[1] Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KEFC2hd0sf", "reviewer": "Reviewer_7Csy", "summary": "This paper introduces a new Agnostic Personalized Federated Learning (APFL) problem and study the personalized federated learning method with parameter factorization and similarity matching.  Specifically, the authors focus on label heterogenous and domain heterogeneous problems.  All in all, the proposed method shows good performance compared to other personalized federated learning baselines.", "questions": "Some of my concerns/suggestions for improvements: \n\n1. Could you please elaborate more on why rank-1 vectors with sparse matrix? What if we consider other values of rank gamma != 1 with local bias (Figure 3 left)? I did not see (or may miss) the ablation study of different rank values, and perhaps discussions of trade-offs for rank > 1. I would love to see more analyses rather than only Figure 7 (h). (Same for Figure 7 (i))\n2. In line 46-50, could you please elaborate more on why permuting the labels and learning on different datasets make gradients more diverge? \n3. In Figure 2 (c), if we run for more epochs, what are the performance? Will it still look ‘consistent’ as in the current version? \n4. Why Factorized-FL alpha and Factorized-FL beta performs ‘inconsistently’ compared to each other in Table 1 page 7 w.r.t both Label-heterogeneous and Domain-heterogeneous?\n5. [Minor] Figure 7 (a)-(e) are hard to see.\n\n", "limitations": "The authors did have clear discussions about limitations and potential negative societal impact in Section A in the Appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Some strengths:\n\n1. The paper tackles an interesting problem with personalized federated learning. The paper has very extensive experiments with great analyses. The contributions are good.\n2. According to Table 1 on page 7, Factorized-FL (both versions) shows good results across datasets and scenarios (domain and label heterogeneous) \n3. It’s also good that the authors provided extensive experiments on the Appendix as well as source code for reproducibility.\n\n\nSome of my concerns/suggestions for improvements: \n\n1. Could you please elaborate more on why rank-1 vectors with sparse matrix? What if we consider other values of rank gamma != 1 with local bias (Figure 3 left)? I did not see (or may miss) the ablation study of different rank values, and perhaps discussions of trade-offs for rank > 1. I would love to see more analyses rather than only Figure 7 (h). (Same for Figure 7 (i))\n2. In line 46-50, could you please elaborate more on why permuting the labels and learning on different datasets make gradients more diverge? \n3. In Figure 2 (c), if we run for more epochs, what are the performance? Will it still look ‘consistent’ as in the current version? \n4. Why Factorized-FL alpha and Factorized-FL beta performs ‘inconsistently’ compared to each other in Table 1 page 7 w.r.t both Label-heterogeneous and Domain-heterogeneous?\n5. [Minor] Figure 7 (a)-(e) are hard to see.\n\n\nOverall, the paper is good in my opinion. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JhplJs591ai", "reviewer": "Reviewer_BK7j", "summary": "This paper introduces a personalized federated learning method based on parameter factorization and similarity matching. The authors factorize the model parameters into a pair of vectors to condense common and personalized knowledge, which are used for guiding the model update aggregation. Extensive experiments show that the proposed method can better handle both label and domain heterogeneity of data in federated learning than many existing methods.", "questions": "N/A", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper studies a setting with stronger data heterogeneity.\n2. The proposed method is interesting.\n3. The experimental evaluation is solid.\n4. This paper is well written.\n\nWeakness:\nI only have one concern on the experimental setting. The authors group the 100 classes to simulate the domain heterogeneous scenario, but this setting is quite similar to the standard extreme label heterogeneous scenario where each client only keeps one class of samples. Thus, the boundary between label and domain heterogeneous scenarios are somewhat vague. I think a simplified scenario where different clients keep different domains of data but their label space is similar (a good example is review sentiment analysis), is also interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JWgkqFL3JCd", "reviewer": "Reviewer_fR1b", "summary": "In this paper, the authors study a new and interesting problem, i.e., they consider the heterogeneity of both the labels and tasks in personalized federated learning, which is very different from the assumptions in previous works, i.e., a same set of labels and a same task in different clients. The studied problem, termed as the Agnostic Personalized Federated Learning (APFL) problem, is well motivated, and the associated challenges are well identified.\n\nThen the authors then propose a novel solution, i.e., Factorized-FL, for the studied problem. Specifically, Factorized-FL factorizes model parameters to reduce the parameter in order to alleviate the knowledge collapse, and utilizes them to measure the task-level similarity for matching relevant clients. Empirical studies on public datasets also show the effectiveness and efficiency of Factorized-FL.\n", "questions": "1 Is there some simple approach to address the label heterogeneity challenge, e.g., label alignment before training?\n\n2 In federated learning, privacy protection is an important issue. In particular, in the similarity matching step, the calculation of similarity requires V of other clients, which may disclose the privacy of the clients. However, the authors do not discuss about the privacy issue. Is there some theoretical guarantee about the privacy protection?\n", "limitations": "The authors are suggested to include more discussions or theoretical results on privacy protection.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nIt is an interesting and novel work though the technique is relatively simple.\n\nQuality: \nThe main idea is simple, i.e., factorization of the parameter into a sharing part and a non-sharing part.\n\nClarity: \nThe paper is well presented with good illustrations and explanations.\n\nSignificance:\nIt is a nice work which addresses an important problem, i.e., heterogeneity in labels and tasks, and achieves significant improvement over the SOTA methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept