Agnostic Personalized Federated Learning with Kernel Factorization
Authors: Wonyong Jeong
Sung Ju Hwang
Venue: ICLR 2022 Submitted (2022)
Keywords: Federated Learning
Personalized Federated Learning

Abstract: Considering the futuristic scenarios of federated learning at a worldwide scale, it is highly probable that local participants can have their own personalized labels, which might not be compatible with each other even for the same class, and can be also possibly from a variety of multiple domains. Nevertheless, they should be benefited from others while selectively taking helpful knowledge. Toward such extreme scenarios of federated learning, however, most existing approaches are limited in that they often assume: (1) labeling schemes are all synchronized amongst clients; (2) the local data is from the same single dataset (domain). In this sense, we introduce an intensively realistic problem of federated learning, namely Agnostic Personalized Federated Learning (APFL), where any clients, regardless of what they have learned with their personalized labels, can collaboratively learn while benefiting each other. We then study two essential challenges of the agnostic personalized federated learning, which are (1) Label Heterogeneity where local clients learn from the same single domain but labeling schemes are not synchronized with each other and (2) Domain Heterogeneity where the clients learn from the different datasets which can be semantically similar or dissimilar for each other. To tackle these problems, we propose our novel method, namely Similarity Matching and Kernel Factorization (SimFed). Our method measures semantic similarity/dissimilarity between locally learned knowledge and matches/aggregates the relevant ones that are beneficial to each other. Furthermore, we factorize our model parameters into two basis vectors and the sparse masks to effectively capture permutation-robust representations and reduce information loss when aggregating the heterogeneous knowledge. We exhaustively validate our method on both single- and multi-domain datasets, showing that our method outperforms the current state-of-the-art federated learning methods.

Under review as a conference paper at ICLR 2022
AGNOSTIC PERSONALIZED FEDERATED LEARNING WITH KERNEL FACTORIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Considering the futuristic scenarios of federated learning at worldwide scale, it is highly probable that local participants can have their own personalized labels, which might not be compatible with others (even for the same class), and can be also possibly from a variety of multiple domains. Nevertheless, they should be benefited from others while selectively taking helpful knowledge. Toward such extreme scenarios of federated learning, however, most existing approaches are limited in that they often assume: (1) labeling schemes are all synchronized amongst clients; (2) the local data is from the same single dataset (domain). In this sense, we introduce loosely constrained federated learning, namely Agnostic Personalized Federated Learning (APFL), where any clients, regardless of what they have learned with their personalized labels, can collaboratively learn while benefiting each other. We then study two essential challenges of the agnostic personalized federated learning, which are (1) Label Heterogeneity where local clients learn from the same single domain but labeling schemes are not synchronized with others and (2) Domain Heterogeneity where the clients learn from the different datasets which can be semantically similar or dissimilar to each other. To tackle these problems, we propose our novel method, namely Similarity Matching and Kernel Factorization (SimFed). Our method measures task-level similarity based on locally learned knowledge and matches the relevant ones for personalized knowledge reflection. Furthermore, we factorize our model parameters into two basis vectors and the highly sparse masks to significantly reduce the dimensionlaity of parameter space for alleviating knowledge collapse and information loss when reflecting the heterogeneous knowledge. We extensively validate our method on both single- and multi-domain datasets, showing that our method outperforms the current state-of-the-art federated learning methods.
INTRODUCTION
Federated Learning (FL) aims to learn the universal global model while merging locally-lcarned knowledge from multiple clients rather than collecting the local data directly to the central server (McMahan et al., 2017; Li et al., 2020b; Wang et al., 2020; Karimireddy et al., 2021; Li al., 2021). Although such decentralized features of federated learning have various advantages, such as preservation of the data privacy, reduction of computing and data maintenance costs, etc., it at the same time shows significant vulnerability in terms of the local heterogeneity. When the local data is not independent and identically distributed (Non-IID), the locally-leamed knowledge might not be mutually helpful depending on the level of distributional shifts, yielding performance degradation of the universal model, which is considered as the critical challenge of federated learning.
One of the successful solutions to such problem Personalized Federated Learning (PFL.), whose main goal is to adapt individual clients to their own local distribution better, rather than learning the single universal model to generalize over all local heterogeneous distributions, which might be sub-optimal (Arivazhagan et al., 2019; Liang et al., 2020; Fallah et al., 2020; Zhang et al., 2021). For example, FedFOMO (Zhang et al., 2021), which is the current state-of-the art approach, downloads few set of random other clients' knowledge and then selectively takes only beneficial knowledge that are performing well on the local obejective, which is different form the conventional federated learning methods that often aggregate all local knowledge. Since such personalized federated learning

---

Under review as a conference paper at ICLR 2022
Conventional Federated Learning
Agnostic Personalized Federated Learning
Server
Server
Server
Label Gbn
Client 1
Client 2
Client K
Client 1
Client 2
Client K
Client 1
Client 2
Client 3
Client
Local Data
X
G
Simple Single Domain Scenario Single Domain Scenario w/ Label Heterogeneity Multi Domain Scenario (Domain Heterogencity)
Figure 1: Overview of the Conventional and Agnostic Personalized Federated Learning: Left illustrates the conventional federated learning. Middle represents label heterogeneity where labels are not synchronized with others. Right shows the multi-domaii scenario where each local client learns K different datasets.
approaches are client-oriented, they have shown stronger robustness compared to the the traditional federated learning approaches in handling the local heterogeneity at each local device.
Toward further realistic scenarios of federated learning at world-wide scale, however, most existing methods are limited in that they often suppose that (1) target labels are the exactly same across all clients and (2) local data is from a certain single dataset (domain). For the former supposition, all clients share the synchronized labeling schemes in the single domain scenarios (see Figure 1 Left), while is more natural that labels are slightly different depending on the user environment (see Figure Middle). For example, diverse labels, such as "Cat", "Kitty", or "Neko" (in Japanese), can be used for the single "Cats" class. Furthermore, both index and number of classes can vary per client, such as { [I:Cats, 2:Dogs. .... 10:Horse) for some clients, [1:Horses, Cats, .... 7: Dogs) for the other clients. We find out that such Label Heterogeneity is highly non-trivial problem in that the most current approaches are significantly degenerated (see Table 1).
For the latter assumption, majority of existing methods mainly tackle distributional shifts, c.g. Non- IID, sampled by certain distribution, e.g. Dirichlet dist., from the same single dataset. However, considering multiple domains is highly essential (see Figure/ Right), since sharing knowledge even between the heterogeneous domains can be beneficial to each other depending on the semantic similarity or dissimilarity amongst the local domains. For instance, if we have to learn COVID-19 19 very quickly but with scarce data, utilizing the similar knowledge, e.g. Pneumonia, can be more effective than learning from scratch or irrelevant knowledge, e.g. vehicles. We also find that discriminating which domain knowledge is beneficial to which domain is also very difficult, as the existing methods fail to handle such Domain Heterogeneity (see Table 2).
To this end, we introduce an extremely realistic problem of federated learning, namely Agnostic Personalized Federated Learning (APFL), where any clients, regardless of what they have learned with their personalized labels, can collaboratively learn while benefiting each other. We then study two critical challenges of it: (1) Label Heterogeneity which labeling schemes are not synchronized for each client; (2) Domain Heterogeneity which each client learns non-identical datasets. To tackle the problems, propose novel method, namely Similarity Matching and Kernel Factorization (SimFed). Our method measures task-level similarity based on locally learned knowledge and matches the relevant ones for personalized knowledge reflection. Furthermore, we factorize our model parameters into two basis vectors and the highly sparse masks to significantly reduce the dimensionlaity of parameter space for alleviating knowledge collapse and information loss when reflecting the heterogeneous knowledge. We validate our method on both single domain datastes (Permuted IID/Non-IID) and multi-domain datasets (CIFAR- 100 20 superclasses) and show our method outperforms the current state-of-the approaches on all experiments. In sum, the main contributions of our work are as follow:
We introduce realistic problem of Agnostic Personalized Federated Learning (APFL). and discuss two possible critical challenges of it, namely Label- and Homain-Heterogeneeli
We propose a novel method to tackle the problems, namely Similarity Matching and Kernel Factorization (SimFed), which measures the task-level similarity and factorizes network param- eters to significantly reduce dimensionality for alleviating knowledg collapse when reflecting the heterogeneous knowledge.
We extensively validate our method in both label- and domain- domain-heterogeneoue scenarios and show that our method outperforms the current -tate-of-the-art baselines.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Federated Learning A variety of algorithms have been proposed for federated learning after firstly introduced by McMahan al. (2017), but we specifically focus on studies tackling the heterogeneity problems, e.g. Non-IID, FedProx (Li et al., 2020b) intorduces proximal regularization to enforce locally-learee knowledge to be close to the global knowledge. FedMA (Wang et al., 2020) learns the layer-wise permutation when averaging weights, while SCAFFOLD (Karimireddy et al., 2021) utilizes control variates to adjust for the client-drift in the local updates. MOON (Li et al., 2021) performs model-level contrastive learning between global and local models. In this paper, as we are pursuing the task-level heterogeneity, we do not cover the architecture-level heterogeneity (Seo et al., 2020; Zhu et al., 2021; Diao et al., 2021; Shamsian et al., 2021).
Personalized Federated Learning aims to improve the individual local clients instead of learning the universal global model via clustering-basee based approaches (Sattler et al., 2019; Ghosh et al., 2021), the mixture methods (Mansour et al.. 2020; Deng et al., 2020; Hanzely & Richtárik, 2021). or meta- learning approach (Fallah et al., 2020). Except for them, FedPer (Arivazhagan et al., 2019) divides local models into base and personalized layers and shares only base layers, while LG-FedAvg (Liang al., 2020) instead shares the prediction layers with the others and keeps the base layers locally. FedFOMO (Zhang et al., 2021), the current state of the art method, downloads and selectively aggregate other clients' knowledge.
Learning from Multiple Domains Beyond learning from the single domain, several recent studies have been proposed achieve the universal global model well gell--eneralized over multiple domains via the mixture of the client target distributions (Mohri et al., 2019). a distributed multi-task tear-nn framework (Smith et al., 2018; Li al., 2020a), domain adaptation for unlabeled data (Peng et al., 2020; Yao et al., 2021). All these prior works, which aim to learn the global model from the multiple domains, however, are orthogonal to our direction, which is to adapt the individual client to their own local domain well that is mainly based personalized federated learning approaches.
Other Remarkable Methods Similarly to our kernel factorization method Jeong et al. (2021); Yoon et al. (2021) decompose model parameters and train them with different objective. The main difference between ours and the prior works is that ours is designed to reduce the dimensionality of parameters into the low rank vectors for alleviating information loss often caused by the conventional coordinate wise aggregation (Wang et al., 2020). Our similarity matching method is similar to (Duan al., 2021; Sattler al., 2019) which use client-level similarity. While the methods calculate the raw gradient updates for discriminating relevant clients, our method does not utilize gradient updates directly calculate similarity between clients. Rather, ours uses efficiently and effectively encoded model representations to measure client-level relevance.
3 PROBLEM DEFINITION
We begin with the formal definition of the conventional federated learning and describe Agnostic Per- sonalized Federated Learning (APFL). We then define two essential scenarios for Label Heterogeneity and Domain Heterogeneity, respectivel...........
3.1 PRELIMINARIES
Our main task is to solve image classification problem using deep neural networks. Let Jg be global model (neural network) at the global server and F = K_ be set of K local neural networks. D = (x,, Ji be a given dataset, where K is the number of local models, N is the number of instances, X; E x is ith examples in a size of width W, height H, and depth D, with corresponding target class Yi (1,...,C) C} for the C-way multi-class classification problem. The given dataset is then disjointly split into K sub- sub-partitions Ps = {x NK,,,.yK,,1. ) No, s.t. D = Uk Pb. which are distributed to the corresponding local model ft- Let R be the total number of the communication rounds and be the current round index. At the first round r=1, the global model fa initialize the global weights o()) and broadcasts ((!) to an arbitrary subset of local models that are available for training at round r, such that f((r) F, |/(() K(r), and
KG) < K, where K is the number of available local models round r. Then the active local
3

---

Under review as a conference paper at ICLR 2022
models S& F(r)) perform local training to minimize loss ((0 on the corresponding sub-partition & ), where o is local weights for local model f& at round r and [() is the loss function to minimize locally. When the local training is done,
Pk and update their local weight o - x v/V((0
the global model collects and aggregates the learned weights 0%2 g1r+1) + No. broadcasts newly updated weights to the local models available at the next round r These learning procedures are repeated until the final round R, and this entire framework refers to as Federated Learning. On the other hand, Personalized Federated Learning aims to adapt the individual local models fI.K to their local data distribution PI.K better rather than achieving the globally universal model fg over all local distribution, and thus merging local knowledge is not necessarily a form of g(r+1)
k(t)) e and then
N -i=1
No
k(r) 0% depending on the algorithms.
3.2 AGNOSTIC PERSONALIZED FEDERATED LEARNING
Agnostic Personalized Federated Learning is personalized federated learning where any local partici- pants from diverse domains with their own personalized labeling schemes can collaboratively learn while benefiting each other. To achieve such objective, following two critical challenges should be tackled: (1) Label Heterogeneity and (2) Domain Heterogeneity.
Scenario for Label Heterogeneity This scenario assumes that the labeling schemes are not perfectly synchronized across all clients, as described in Section and Figure/ Middle. Most underlying setting for this scenario is the same the conventional single-domained setting described in Section 3.1, except that labels are arbitrarily permuted amongst clients. The local data Pk for the local model fir is now defined as Pk = {xk,is PP& (yk,)) ( ] where (+)(.) is a mapping function for local model f& that maps a given class yk,, with randomly permuted label PR. (yk..). Let the jth layer out of I. layers in the neural networks of local model f& be EL and the last layer &f be a classifier layer. Now that each client has differently permuted labels, the personalized classifiers Ht are no longer compatible each other, and thus they are enforced to be remained locally and not undesirably merged for the fairness of this label-hhterogeneeuus scenario.
Scenario for Domain Heterogeneity This scenario presumes that local clients learn on their own entire datasets D, rather than partitions Px from the single dataset, as described in Section I and Figure Right. In this setting, K disjoint datasets Di.K are assigned to the K local clients fi:K, where D (XE,s, &k. N is the entire dataset assigned to the local model fi. where the number of target classes are possibly different, such that yk. E {1, Ck} Between the datasets,
there is no instance-wise and class-wise overlapping, such that Ø N Da. Similarly to the hetrroohttoo scenario described above, each client has disjoint target classes, and thus the personalized classifiers f no longer compatible each other. Hence, they enforced to be remained locally and not shared each other for the fairness of this domain hemaain-httrrogeeouu scenario.
4 SIMFED: SIMILARITY MATCHING AND KERNEL FACTORIZATION
We now elaborate on in-depth description of our novel algorithm for agnostic personalized federated learning, namely Similarity Matching and Kernel Factorization (SimFed) in the following subsections. For better understanding, our overall framework is illustrated in Figure 2
4.1 SIMILARITY MATCHING
We first define beneficiality of clients as the degree of performance improvement when merging their knowledge. If merging knowledge between certain clients improves individual model's performance (or global model performance), then those participants can be seen as beneficial to each other. If not, participants be seen not helpful each other, which we should avoid merging them.
Zhang et al. (2021) demonstrated that avoiding irrelevant clients improves local performance. They suggested to download few other clients' weights and validate them locally for discriminating helpful models. This is an intuitive approach, but we want a more efficient and effective way to find helpful models. How measure lask-level similarity in an efficient and effective way? Our assumption
4

---

Under review as a conference paper at ICLR 2022
Projection to Latent Space
Kernel Factorization for Dimensionality Reduction
REXEXINO
RE
Citentz
Q0.65
R
Unbiased Orieera Input
0.25
1x0
0.1
Similarity Matching & Knowledge Reflection
Figure 2: Illustration of Similarity Matching and Kernel Factorization: We match relevant clients based on the model embeddings obtained from the criteria input. Then we reflect the relevant knowledge based the similarity. Our keree weights are factorized reduce the dimensionality parameters
that we can identify those helpful models if we can recognize task-level similarity, as Yoon et al. (2021) revealed that clients learned on similar tasks improves local performance.
For this, our simple-yet-efferte idea to view the individual local models as respective set encoders that can effectively transform their local data distribution to their own personalized embedding vectors. If we equally feed the same inputs to all encoders, then they will interpret the criteria inputs depending on what they have learned in their own local environments and produce their own personalized transformations, which we can efficiently utilize them as representations of local knowledge measure the relevance. Formally, the neural networks fa can be represented as:
fx(()) (5) E& (z)x)),
(1)
where f& is a combination of an encoder Es (z and a predictor Ps (8(z), where z E Rd is d- length transformed latent vector of input data x and ÿ E RC is a logit for the final prediction. We generate the unbiased criteria input XN E R H DD from the Gaussian normal distribution, such that vi ~ (0, 1) where v, is its element, and assume that K local models fi:K, we equally feed the single criteria input XA to K local encoders so that we can generate a set of K multiple personalized transformations ZN as follows:
located at the server. Since we have
ZN ZN, = Es (ZXN))
(2)
Now our remaining task is to compute the relevancy between the transformations ZN which each local knowledge is efficiently encoded. We use the cosine similarity to measure the distances between target model fa with the rest of models in F, as follows:
22k, ZN. θ) {(a,, 0,)lo,
Z | 2 T. 2, O)
7 7
(3)
where θ is set of K local weights maintained at the server and ?(-) is a similarity function that returns a set of pairs of the similarity score o; and the corresponding local weights 0 for all i E (1, .2, KJ. while satisfying the similarity scores greater or equal to the given threshold
4.2 KERNEL FACTORIZATION
Wang et al. (2020) discussed that the conventional knowledge aggregation, which is performed in a foordinate-wist manner, may have severe detrimental effects on the averaged model. This is because the current deep neural network have often extremely high dimensional parameter space and the meaningful element- wise neural matching is not guaranteed when aggregating multiple weights, e.g.
g(++1)+ No
Such naive aggregation could be less serious when each local updates share common features, for example, in cases of learning from the same data distribution across all clients. However, when each local model learns heterogeneous domains, such simple mixture of knowledge can be highly problematic. It even could lead to forgett which forgets what they have previously learned.
We assume that such problem can be alleviated if the dimensionality of parameter space is significantly reduced as it minimizes information loss and detrimental knowledge collapse. order to do that, novel idea is to factorize the high order model parameters into lower rank components, rank 1 vectors. Formally, local model fy has local weights 6, that is a set weights of its layers, such that 0r (w: H The type of each layer w then split into fully-connected convolutional layers.
5

---

Under review as a conference paper at ICLR 2022
Factorization of Fully-Connected Layers The dimensionality of the dense weight wi. for the fully-connected layers drastically increased by multiliccation of input and output dimensions, such that wi E R /xx0, where I and O indicate respective input and output dimensions. We can reduce the x O complexity into rank complexity via factorizing the high order matrix as follows:
w = u x v, where u E Rxx1 . R
However, the multiplication of the two vectors interact only in linear manner, it might be hard to capture non-linen con-lineartty using just two of them. As such, we add the light-weighted sparse mask μ to the weights additionally capture non-linearity, as follows:
W = u X v 2 Je where E R R VE VERIXG R1x
E
(5)
Factorization of Convolutional Layers The difference between fully-connected layers and convolutional layers is that the convolutional layers have multiple kernels (or filters) such that w E RFXFIIX, where F is a size of filters. Similarly to the factorization of fully-connected layer, reduce high order parameters into rank vectors and the higly sparse masks to support non-linearity for the convolutional layers, as follows:
k क (u'
pA), where u' E RR- v ERRXIOO RIXO ,1' E RRIFFXOFF
(6)
x(-) R. + R
Note that we factorize our parameters before training and thus the weights of each layer w act like the regular weights during training, while learning the basis vectors and the masks separately.
4.3 SIMFED ALGORITHM
Putting similarity matching and kernel factorization altogether, we now describe our objective functions for our factorized parameters and personalized knowledge reflection.
Learning Objective Instead of utilizing the single term 0₁ for local weights of neural network fk.
now let Çk. Lk. and Φe be sets of ug, Vk. and jıg of all layers in fk. s.t. ÇA- = {uj) & = {vf H=)
and Φx = (1 1 then our local objective function can be defined as follows:
min Σ SkIEE
L(B; Ek. Φx)) + X b1/1,,111,
BED
where is the standard cross-entropy loss performed on all minibatch E Dr- We add the L sparsity inducing regularization term to make the mask parameters highly sparse yet to efficiently capture the non-linearity, whose intensity can be controlled by the hyper-parameter λ₁.
Personalized Knowledge Reflection We avoid direct averaging the knowledge amongst different clients. We instead individually reflect only dif- ference between target local model and the other clients to preserve local reliability more effectively. inspired by Zhang et al. (2021). First, given local model fk. select the other beneficial knowledge that are relevant to fü via (2(-) described in Eq. 3, returning set of J pairs of (aj, Sj. Ej Φz). Sec- ond, we separately update each factorized parame- ters Cj. E and Φ, while minimizing the collapse of information as follows:
Algorithm SimFed Algorithm
Function RunServer()
initialize F, Xv.7
each round 2, R
jr(r) +- select K clients from M for each client / F in parallel do
if then
(01.1. Su.. (1.)) Qik Zv.
S 5
explarg
esplors)
g(r) +- gl))
coplorin
Σ/ (5) g
d(r) +- (n)
eaplo.a,)
Sun coptacayy
end if
cie+1) (1+1)
(r+1)
zx +- RunClienti
e 10 xx)
Ca c + expla-or O.))
Σ(G
(C$
(k),
end for
Explao o,)
end for
1=1
Function RunClient(Ce. Ex. Xv)
(8)
each local epoch from do for minibatch B D. do Ocxcmo +- 0cxcao v/V ((300)
where a scalar value for scaling the reflection ra- tio of similarity score oj. This personalized knowl- edge reflection is also equally applied to E, and dj. respectively (please Algorithm 1).
end for
Rvck +- (Xw: Ocxegs) return Ee. Ry.k

---

Under review as a conference paper at ICLR 2022
5
EXPERIMENT
5.1 EXPERIMENTAL. SETUP
Baseline Models We choose our baseline models from three categories, which are the conventional federated learning, personalized federated learning, and the local training model. For the conven- tional federated learning algorithms, we choose conventionally well-known algorithms, such as FedAvg (McMahan al., 2017) and FedProx (Li al., 2018). Particularly, for Agnostic-FL (Mohri et al.. 2019), it tackles mutl-domained scenario that is similar to our goals and thus we compare the model with our method. For personalized federated learning methods, we select FedPer (Ari- vazhagan et al., 2019), Per- Fer-Fe (Fallah et al., 2020), Clustered-FI. (Sattler et al., 2019), and FedFOMO (Zhang et al., 2021). In particular, Clustered-FL clusters clients based on client-level similarity which approach is related to ours. Moreover, FedFOMO can discriminate which clients are helpful or not, which shares a similar approach to our methods. For the local training model, Stand Alone, ironically, it could be the most powerful baseline in such extremely heterogeneous settings, e.g. label- or domain-levee heterogeneity, where knowledge transfer becomes non-tre -riviaiial challenges.
Training Details We use ResNet-9 architecture for all base algorithms and our method and train on 32 x 32 sized images. We leverage Stochastic Gradient Descent (SGD) as our optimizer and we ensure that training configurations are equally set for all algorithms, such as 5e-2 for learning rate, le-4 for weight decay, 128 for batch size, etc, unless otherwise we state clearly in our paper for fairness. For our algorithm, we use 0.9 for T and 30 for a. A is set to 1e-4. For all our experiment, please note that the classifier layers are not aggregated in the label- and domain heetiinnhterogeneous settings for fairness.
Datasets (1) Single-domain dataset: we use three datasets for three respective experiments, which are MNIST, CIFAR-10, and CIFAR-100, for the single-domain scenario. For the first step, we split each dataset into train, validation, and test sets, such that 55, 990/7, 000/7, 000 for MNIST, 47, 985/6, 000/6, 000 for CIFAR-IO and 100. We then split the train set into K local partitions P1:20 (K=20) in identical manner (all instances in each class are evenly distributed to all clients) or a non-identical way (instances in each class are sampled from Dirichlet distribution with a=0.5). We further permute the labels for each class per local partition Pa for Permuted IID and Permuted Non-IID setting to simulate the aubel-Heterogeneitty (2) Multi-domain dataset: we use CIFAR-100 datasets (60, 000) and split it into 20 datasets D1:20 grouped by 20 superclasses which are also introduced as "coarse" labels, such as mammals, household furniture, vehicles, etc. In each domain Dr (3, 000), non-overlapping subclasses are assigned to each client. Then we split each dataset into train (2, 400), validation (150), and test (450) sets. Please see further details for the dataset configuration in Section 3.2.
5.2 EXPERIMENTAL RESULT
Label Heterogeneity As shown in Table 1, for the conventional IID and Non-IID settings, all federated learning methods show higher performance than the pure local training baseline (Stand- Alone), meaning that the locally learned knowledge is helpful each other and sharing them certainly improves each local performance. Interestingly, however, when the labels are not synchronized for each clients (the permuted IID and Non-IID settings), we observed that all existing federated lcarning methods are degenerated and show even lower than the Stand-Alone models for both IID and Non-IID partitions. Again, note that we do not share the classifier layers for fairness for all algorithms in this permuted settings. The permutation of labels makes your model heterogeneously learn even if the given data is from the same class. Our method, however, shows strongest performance particularly for the permuted settings and actually improves over Stand-Alon models failed. This clearly shows that our method can effectively reflect the other knowledge while minimizing the information collapse via kernel factorization as well as similarity matching.
models which all other existing
Domain Heterogeneity Table 2 shows the experimental results for the multi-domail scenario with non-overlapping (disjoint) 20 domains from CIFAR-100 dataset. In this extremely heterogencous setting, similarly to the permuted labels scenarios, we observe that the prior federated learning algorithms cannot properly share local knowledge amongst clients and fail to achieve the better
7

---

Under review as a conference paper at ICLR 2022
Table I: Performance Comparison of Single-domain Scenario We train 20 clients on each dataset for 100 rounds respectively. We measure local accuracy at the final round and report averaged scores over three trials with different seeds. Note that the classifier layers are not aggregated in the permuted settings for fairness.
Single-domain Federated Learning
MNIST
CIFAR-10
CIFAR-100
Method
Conventnl. IID
Permuted IID
Conventnl. IID Permuted IID
Conventnl. HID
Permuted IID
Stand-Alone
96.91 (1 0.13)
97.26 (1 0.06)
59.62 (1 0413
62.01 (1202
25.12 (1 0:24)
22.92 (1 0.12
FedAvg (MeMahan et al., 2017)
97.87 (4: 0.12)
95.26 (4: 0.25)
65.52 (#: 0.55)
45.99 cà: 0.44)
45.36 (#: 0.17)
11.48 (4 0.11)
FedPProx (Li et 2018)
97.91 (#
95.41 0.19)
65.06 (t: 0.37)
45.07 (33)
44,94 0.23)
11.58 (4 (17)
Agnostic-Fi Fl. (1oorrr et al., 2019)
97.87 cz: 0:10)
95.13 0:1
64.69 (#: 0.92)
46.37 (2:071
44.63 (t: 0.313
(1
Clustered-FL (Sattlrr al., 2019)
97.57 (#: 0.08)
(+ 0.39)
65.39 (=: 0.53)
(331)
42.83
0.29)
17.94 0.19)
FedPer (Arivarhagan et al., 2019)
98,11 (t 0.11)
94.81 (27)
68.70 (t: 0.29)
45.12 (+: 0.30
49.12 (t: 0.38)
11.26 (1)
Per-FedAvg (Fallah et al., 2020)
97.91 ct: 0.14)
94,02 (:10..1)
66.12
(1: 0.37)
47.23
(1: 0.38)
50.58 0.13)
12.08 0.22)
FedFOMO (Zhang et ol., 2021)
97.53 (1 0.08)
94.97 (1 0,59
67.08
(1: 0.28)
51.12
(.:0.233
48.13
0.49)
11.08 0.51)
SinFeed (Ours)
98.28 (t: n.12)
97.83 (st 0.36)
71.01 (#: 0.89)
64.21 (+: 0.29)
52.01 (+h 0.18)
24.11 0.23)
Method
IID (C)
IID
Non IID (C)
Non: IID (P)
Non (C)
Non (P)
Stand-Alone
81.52 (#: 0.43)
81.73 (# 0.45)
39.77 (=: 0.11)
44.46 (# 0.23)
18.53 ( xxas)
17.36 0.35)
FedAvg (McMahan et al., 2017)
91.89 (+ 0.47)
72.34 (#: 0.19
44.37 (#: 0.31)
31.43 (# 015)
26.32 (t 0.34)
8.87 (#: 0.36)
FedProx (Li et al., 2018)
89.84 (1: 0.32)
66.32 (1: 0.22
42.63 (1: 0.25)
31,91 (1: 0.84
26,46 0.35)
5.81 (1: 0.83)
Agnostic-FL (Mohri et al., 2019)
91.51 (1 0.48)
70:97 0.31
43.33 (1: 0.2)
32.28 (.19)
25,14 0.31)
8:21 (1: 0,73)
Clustered-FL. (Sattle et al., 2019)
90.59 (#: 0.29)
77.78 (#: 0.57)
49.68 (#: Uax)
39.33 (+: 0.34)
25.73 (# llax)
11.61 (# (+10
FedPer (Arivazhagan al., 2019)
90.32 (t:
70.45 (#: 0.19)
50.43
29.45
33.58 0.ss)
5.68 (#: 0.17)
Por-FedAvg (Fallah et al., 2020)
90.66 (z: 0:38)
73.37
(2: 0.43)
45.91
(#: 0.73)
31.23 (1.0
27.39 (1: 0.56)
6.49 (2: 0.30)
FedFOMO (Zhang et al., 2021)
91.74 (t 0.33)
68.94 (+ 0.29)
50.62 (+: 0.9)
30.15 (+: 0.35)
28.29 (+: 0.23)
8,93 (#: 0.19)
SimFed (Ours)
92.13 (1: 0.27) 84,94 (::
52.13 (1: 0.23)
45.37
34.11 0.70
18.38 6.11)
Table 2: Performance Comparison of Multi-domain Scenario We train 20 clients on 20 heterogencous datasets simultancously for 100 rounds. Wc mcasure local accuracy at the final round and report averaged scores over three trials. Note that in the multi-domained settings the classifier layer is not merged for faimesss
Multi-domain Feslerated Lourning
CIFAR-100, Domainss, 5 independent sub-classes for each domsin
Method
Aqu. Manmmals
Fish
Flowars
Food
Fruits & Vege.
Elee Devizes
Furniture
Stand-Alose
50.15 (50
62 (t: 1.20
64.07
68:44
73.56 3:300
04 2:50)
65.48 (s: 1,35)
FodAvg (McMahan at., 2017)
48.07 (t: 1.76
60.37 (E: 1.85)
53.63 (E: 2.50)
45.48 (1: 1.50
58 58.00 ::: 1.55)
48.15 (k: 2.16
51.63 (2: 1.500
FedProx (Li et 2018)
15.63 (t: 1.00
59.18 (t: 1.30)
42,96 (2:4
16.07 (#3)))
58.67
(# 2:45
$1.78
(t: 2.02)
51.93 (e: 1.79)
Agnostic-FL. FL. (Molri et al., 2019)
48.52 (s:22tt13)
39.78
1.790
52.59 (20)
46.81 (.37
54.44 3.54
49.48
1.74)
51.48 (#: 3.10)
Clustered (Sattler et al. 2019)
47.78
62.81
3.63)
(2:4.14
(2: 9.47)
6.00)
53.33
4.60)
53.19 7.64)
PedPer (Arvazhaaga et al., 2019)
48.11 (1 1:2)
60.67 (1.)49)
57.67 (17077
41:11
(11 8.6
60.22
(.. 2.44)
49.23
(1, 0.56
48.22 (.) 5.96)
Per Ped/Aeg (Fallah al.. 2020)
46.78 (::2
60.00 (1: 1.1)
48.33 (b: 1.77)
49.78 cb 0.000
53.00
(1: 5.44)
47.44
(2: 2,33)
50.89 (2: 561)
FedROMO (Zhang al.,
48.07
3.57
50.67
55. 35.26:22
7.45)
45.56
1.45)
48.
Similarity Matching (Durs)
(1:15)
64.67 (1: 1899
64.52 (1: 1.00)
70.67 (1) 0.65
67
:1: 5.5)
67.78
(1: 0.60)
66.96 (1: 6.4%)
4 Factesization (Ours)
$6.59 1.33
69.11 (1 275)
66.30 (1. 2.33)
70.37 (1. 1.310
73.26 (1. 1.71)
72.59
1.37)
65.93 (1: 1.323
Insects
Camivores
Objects
Outdr: Scenes
Med. Mammals
Invertebrates
Stand-Aloee
66,07 100)
62.15 (1: 4.60
76.07 c1: 7.00)
81 85 (st: 3.45)
61 [a: 5.00
66.81 (1: 5.67)
61.93 1.40)
FodAvg (McMahan et al., 2017)
55:18 (72)
$1.41 (t: 1.39)
59.19 de 2.47)
70.59 c+ 5.00
$2.44 (#: 3,62)
$2.81 (th 1.57)
56.22 (#: 1.75)
FodProx (Li al., 2018)
(4:25)
49.04 2:70
61.04
74 89
3.15
51 41
(s 3.29)
55.04 0.50
52.59 1.83)
Agnostic-FL (Mohri
2019)
54.00 (+: 030
49.70 (4: 4.70)
6.95)
1.54)
50.74 3,63)
54,37 3.99)
49.85 2.313
Clustered-FF. (Sanllr et al., 2019)
63.56 (1. 50m
$5.41 (1 4.00)
66.59 ca 11.83
72.07 c1 5.00
$3.70 cal 5.13
55,70 (1. 6.09
53.41 (á: 3.55)
PedPer (Arivuzhagan et al., 2019)
56.33 (t: 367)
51.89 (t: 4.50)
62.44 dt 1.50)
72.67
(t: 2:22)
51.78 (:t: 4.00)
54.33
(s) 3.22)
51.44 (5: 4.33)
Per-PedAug (Fallah al., 20200)
$7.89
55.22
53.78
73. 44
53. $3.56:22 0.89
$2.44
48
FedFOMO (Zhang et al., 2021)
56,37 (1: 4.0)
53.85 (1: 0.33)
3.00
72.00 (.): 2.00)
(1: 1.84)
54.37
1880
53.41 (3: 5.333
Similarity Masching (Ours)
64.81 (1. 396)
58.67 (1: 4.9
76.44 ck: 320
84.44 (1. 0.26)
58. 30 ::: 9.29)
66.52 (2: 0.201
55.04 cá: 5210
Factorization (Chirs
69.11
63.56
2677
83
(::
76.15
61.41 7.50)
Peopole
Reptiles
Small Mammals
Trees
Vehicles
Vehicles
AVERAGE
Stand-Aloze
38.74 (1 0.66)
55.78 (1 2.7)
50.22 (1: 2.00
61.19 (1) 2:400
71.48 (1 1.55
79,04 (1: 1.711
64.19 0.79)
FodAvg (McMahan et al., 2017)
36.07 (
50000 (1: 19)
46.37
54.96 (t: 1.45
52.22 (s 2.10
63.85 16
53.35 (2: 9.4)
FodProx (Li 2018)
34,52 140)
19/18 (t: 1.54)
1.59)
63.33 0.73)
$2.39 0.4)
Agnosti-FF. (Mohri (Mauli et al., 2019)
34.89 0.3
60.3011 (t: 200)
45.85 cb: 3.27
55.41 (t: 1.17)
49.93 ( 1.21)
65:41 (s: 4.00)
52.64 (=1)
Clustered FL (Sattlr et al., 2019)
37.93 (t: 1o0
53.33 (t: 1990
47.70 ct: L.6)
56.74 ct 0.46)
51.70
ca: 3.523
66.52 (s) 3.94)
55.76 (s: 2.37)
FedPer (Anvarhagan al., 2019)
34.33
46
46.33
53.78
50.33
64
53.09 1.56)
Per-FedAwg (Fallah al.., 2020)
38.67 0.22)
52.22 (1: 0.22)
0.44)
(::: 4.89)
::: 180
63.56 5.10)
52.92 (a: 1.5%)
PddPOMO (Zhang et al., 2021)
36.67 (1. 1.79)
$2.52 (1 1766)
40.15 ck. 1.71)
53.41 (d. n.m0
52.22 (dl: 1.500
26 (1: 6.00
53.41 (3: 6.57)
Similarity Masching (Ours)
36.07 1.70)
58.44 1.40
50.67
59.11 (t:
68. (d: 5.70)
76.07 2500
63.31 0.55)
Factorization (Ours)
41.26 (1. 2.80)
61.33 (1 2.4)
$2.30 (1. 4.88)
62.00 (1). 3.30
69.19 (1: 12,00)
83.41 1440)
67.35 (1: 6.6)
performance over pure local training baseline (Stand-Alone), showing that the domain heterogeneity is critical challenges of federated learning. Even though it seems that federated learning does not work in this domain heterogeneous settings, our method, however, shows the strong possibility. SimFed (Similarity Matching) achieves much higher scores over the existing federated learning algorithms, meaning that it properly utilizes other clients' knowledge. However, it still shows slightly lower
8

---

Under review as a conference paper at ICLR 2022
Clients, Rounds
77
20
50
20
Grent
KSNP Viscotization for Domain Emboeiing Spaces
(a) Bar plot for the Per-Client Performance
(b) Latent Space Visualization
Figure 3: Per-client Performance and Embeddings Space for Local Knowledge: (a) we show bar plot for the per-elient performance comparison. (b) we visualize 2,000 latent embeddings locally learned knowledge (100 Criteria Inputs for 20 Clients). The labeled number is correspondent client id (and also dataset in Table 2)
(a) Inter-elient Similarity (r=1) (b) Inter-client Similarity (r=50) (c) Inter-elient Similarity (r= (=0)
Figure 4: Inter-Client Similarity: We visualize inter-client similarity at (a) the beginning with r=1, (b) middle
with r=50, and (c) the end with r=100 of 100 training rounds total.
performance compared to the Stand-Alone model. Even though we choose the beneficial knowledge, the reflection of the knowledge should properly handled, which an another critical challenges. With kernel factorization, SimFed finally improves (4%p higher in average) over Stand-Alone model, meaning that ours can discriminate which knowledge from others are beneficial or harmful and successfully reflects the knowledge to adapt to the given local domain while effectively alleviating risks caused by domain heterogeneity. In Figure Left, we visualize per-client performance. All clients of our method shows way higher performance over all federated learning methods. In few datasets (Fruits Vege., Invertebrates, and Vehicles 1), ours shows quite marginal performance compared the Stand-Alone method. Intuitively, we conjecture that those datasets are not really correlated to other local domains in the feature level aspects. Even this case, however, if some new clients from domains similar those datasets newly participate in the collaborative learning, then we believe that the performance can be improved as ours can be benefited from the relevant clients.
Alleviation of Information Loss Zhang et al. (2021) discussed aggregating with unhelpful clients leads to significant performance degeneration. In this sense, we match only relevant clients amongst all clients based on our similarity matching algorithm, ensuring that irrelevant knowledge aggregation can be effectively prevented. This reduces information loss and knowledge collapse, we can see Similarity Matching outperforms most federated learning algorithms Table 2. However, the conventional knowledge aggregation is performed in extremely high dimensional parameter space in a coordinate-wise manner, which may have severe detrimental effects on the averaged model, as studied by Wang al. (2020). In this sense, we significantly reduce the dimensionality of model parameters into low rank vectors (with highly sparse masks) by our factorization method, which empirical results further improves performance achieved by similarity matching (+ Factorization in Table 2). The results demonstrate information loss is further largely reduced.
Effectiveness of Similarity Matching Our robustness on label and domain heterogeneity firstly comes from correctly discriminating which knowledge is helpful not. To see the effectiveness, we compare performance between our similarity matching algorithms and Random Matching & Worst Matching baselines on multi-domain scenario with the exactly same configurations as described above. Random matching baseline randomly pick two arbitrary models to aggregate, while worst matching chooses two the most dissimilar models for knowledge reflection at each round. As shown in Table 3, both random and worst matching models significantly suffer from
Table Similarity Matching
Method
Acc.(%)
Worst Matching
48.83%
Random Matching
52.12%
SimFed (Ours)
67.35%

---

Under review as a conference paper at ICLR 2022
the performance degeneration compared to ours (around 15%p for Random Matching and 20%p for Worst Matching), demonstrating that we properly choose the beneficial knowledge for each round.
In Figure 4 visualize inter-client similarity during training under multi-domain scenario. As shown in the figure, local models seem to be strongly correlated with cach other at the first step. However, as learning progressed, we can observe that clients indeed are related to just few of them, which makes more sense in this multi-domain scenario, demonstrates why aggregating all other clients leads significant performance degeneration. Also, we see that our algorithm can identify the relation between clients more precisely as learning processed. In Figure 3 Right, we visualize our latent embeddings of locally learned knowledge. We utilize 100 Gaussian criteria inputs and transform them by 20 local clients (after 100 rounds training in multi-domain scenario), yielding 2, 000 latent embeddings of local knowledge, As shown in in the figure, we can observe that the latent embeddings are grouped clearly and does not overlap on the other domains. This shows our encoding method successfully encodes the same things same and the different things differently. This reliability and consistency can help discriminating task-level similarity, which eventually improves local performance.
Effectiveness of Kernel Factorization Even though we success-
Table 4: Kernel Factorization
fully match the relevant clients, merging the knowledge should be
carefully handled as we observed in multi-domain scenario (Table 2). Minimizing knowledge collapse and information loss is critical in such heterogeneous scenarios. Our kernel factorization method further improves performance achieved by only similarity matching technique as shown in Table 2. This is because we significantly reduce the di- mensionality of model parameter space into low rank vector space. This alleviates the information loss when merging the heterogeneous knowledge. For further analysis, we compare two baseline models, which are ours with no factorization (w/o Factorization) and ours with no masks (w/o Masks σ), in multi-domain scenario with the same training configurations. When we eliminate the factorization technique (which equivalent to the Similarity Matching only), the model shows about 4%p lower performance compared to our full model, demonstrating factorization has beneficial effect on improving performance multi-domain scenario. When we remove the sparse masks phi, we observe significant performance drop, demonstrating it successfully captures the non- non-linearity supporting the low rank vectors.
Method
Acc-(%)
SimFed (Ours)
67.35%
w/o Factorization
63. 63.31%
w/o Masks &
22. 22.26%
In another analysis, conversely add our factorization tech- nique to the existing federated learning methods, e.g. FedAvg + Kernel Factorization, and see how the factorization affects the pure federated learning method. We conduct experiments on CIFAR-IO in permuted iid and non-idd settings for 100 rounds. As shown, the combined model shows 3 - -4%p higher accuracy over pure federated learning algorithms. As the only difference between pure and combined models is the dimensionality of parameter space. which we demonstrate our factorization method alleviate the knowledge collapse and information loss caused by the coordinate- wise aggregation in high dimensional parameter space. Ours still outperforms the baseline model, meaning that our similarity matching algorithms further improves performance.
Table 5: More Factorization Analysis
Mothod IID (P)
NonlID (P)
FedAvg 45.99%
31.43%
Factorization
49.72%
34.75%
SimFed (Ours)
64.21
45.37%
6 CONCLUSION
In this work, we discussed the realistic problem of federated learning at worldwide scenarios, where (1) the labeling schemes are possibly not synchronized across all participants and (2) local clients can be from any multiple domains. which we named such challenges as Label Heterogeneity and Domain Heterogeneity. To tackle these problems, we proposed novel method, Similarity Matching and Kernel Factorization (SimFed). Our method measures lasel similarity based on locally learned knowledge and matches the relevant ones for personalized knowledge reflection. Furthermore, we factorize our model parameters into two basis vectors and the highly sparse masks to significantly reduce the dimensionlaity of parameter space for alleviating knowledge collapse and information loss when reflecting the heterogeneous knowledge. We validate our method on both single domain datastes (Permuted IID/Non-IID and multi-domani datasets (CIFAR-100 20 superclasses) and show our method outperforms the current state-of-the-ee approaches on all experiments.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers, 2019.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, 2020.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients, 2021.
Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. Fedgroup: Efficient clustered federated learning via decomposed data-driven measure, 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: meta- learning approach, 2020.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning, 2021.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models, 2021.
Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervise learning with inter-client consistency disjoint learning. In International Conference on Learning Representations, 2021. URL https: : / opwnreview..e.eeeeeepepe net/ forum meee/forummdddcec6FFF 6CC
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, 2021.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. arXiv preprint arXiv:/8/2.00 18/2.06/27 2018.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Federated multi-task learning for competing constraints. CoRR, abs/2012.04221,, 2020a. URL URtptt::///aciiv..o /bs// 2012. 04221.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks, 2020b.
Paul Liang, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations, 2020.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications federated learning. 2020.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effiiiee learning of deep networks from decentralized data. In AISTATS, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning, 2019.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. In International Conference on Learning Representations, 2020. URL https://openreviee. net/ met/foos forum? feet/forumiddd-teer3333a -d=
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multi-task optimization under privacy constraints, 2019.
Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. Federated knowledge distillation, 2020.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemeeworks. arXiv preprint arXiv:2 2021.
11

---

Under review as a conference paper at ICLR 2022
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar, Federated multi-task learning, 2018.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- erated learning with matched averaging. In International Conference on Learning Representations, 2020. URL https::////o /ttp:///eeeeeeeeee/mm///e net /tt./oormmmiiii//////// Bugliddl-kkuuusssueo
Chun-Han Yao, Boqing Gong, Yin Cui, Hang Qi, Yukun Zhu, and Ming- Hasun Yang. Federated multi-target domain adaptation, 2021.
Jachong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer, 2021.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021. URL https: / /oppnrevieweweeeeeooreeeeereeeee htte://ooommmi-hhhgu-oo0e0 9cw.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning, 2021.
12


Reviews:
[{"review_id": "c0wY33DLAtT", "reviewer": "Reviewer_o4KB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n(1)\tThe two scenarios in federated learning studied in this paper including label heterogeneity and domain heterogeneity are interesting and practical. \n(2)\tThe proposed method views the individual local models as respective encoders that can transform their local data distribution to embedding vectors shared among clients, instead of adding any secondary models, which is simple but effective.\nCons:\n(1)\tIn Section 4.2, authors assume that if the dimensionality of parameter space to compute is significantly reduced as well as knowledge is separately captured, then the loss of essential information of model aggregation in domain heterogeneity scenario can be alleviated, which is not well justified.\n(2)\tThere is no reason why the basis vector parameters u can capture the input-related knowledge and v can learn the output-related knowledge.\n(3)\tThere is no detailed justification or no reference to key formulas in the paper. The Eq. (8) for updating factorized parameters is mistaken.\n(4)\tSome experimental results in Table 1, Table 3 and Table 4 are missing, and the setting of similarity threshold is not presented, which harms the reproducibility of the proposed method. The variant method “w/o Factor” in Table 4 is confusing.\n(5)\tGiven the heterogeneity of different local datasets, the projection these datasets into a latent intermediate space for all clients can not well ensured. The authors did not give sufficient rational analysis for this assumption. This operation may destroy the privacy leak of local data.\nThe studied problem in this paper is interesting and practical, but the technical idea and technical contents are not well justified, and the experimental results are not well organized and analyzed.\n\n====\nComments after rebuttal:\nFirst of all, the authors make good efforts during the rebuttal period and clarify my concerns on privacy issue. There are still several issues not well addressed.\n1. I just checked the author's response and the comments from other reviewer. From my own experience in personalized federated learning, existing solutions indeed can handle different label spaces or non-i.i.d. data across clients. FedBN (ICLR2021) can handle the feature heterogeneity. The authors do not well position their own work and the main technical idea is not well analyzed yet. I admire the addtional results and more ablation study.\n2. The authors still do not well revise Eq. (8), the subscript j for the numerator is not well referred from the context, why not merge with the summation also not discusssed.\n3. The kernel factorization is not well analyzed yet, mostly are simply supported by improved results.\n4. Overall, the main technical idea is not well analyzed and manuscript is not so clearly presented yet. I would like to keep my scorings.", "summary_of_the_paper": "This paper studies two challenges of personalized federated learning: (1) Label Heterogeneity where label schemes are not synchronized in local clients and (2) Domain Heterogeneity where the datasets owned by the clients can be semantically dissimilar. The authors propose a method called Similarity Matching and Kernel Factorization which measures semantic similarity/dissimilarity between locally learned knowledge and aggregates the relevant ones. Furthermore, the method factorizes the model parameters into two basis vectors and sparse masks to capture representations of the heterogeneous knowledge.", "summary_of_the_review": "This paper studies the novel and interesting scenarios in federated learning, i.e. label heterogeneity and domain heterogeneity, and the experimental results show that the proposed method outperforms other federated learning methods on single- and multi-domain datasets. The main logical structure of the paper is relatively clear. However, there are also some shortcomings that need to be improved, e.g. the proposed assumption is not well justified and the experimental setting should be detailed. Overall, this manuscript is not mature enough.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vuw53hZ9nGu", "reviewer": "Reviewer_85V7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have the following major concerns.\n\n- The comparison baselines for Label Heterogeneity do not seem informative. By design, many of these baseline methods would implicitly require the labels to be correctly aligned. A more reasonable way is to compare the proposed framework to those methods that take label mismatch into account. For example, you may simply integrate permutation-robust techniques into FedAvg to deal with label heterogeneity cases. I would like to see more comparisons based on such kinds of baselines.\n\n- It is unclear why we should use the cosine similarity. Each local model may have different preferences of scaling resulting given different domains of the data source. Also, what exactly/mathematically does the keyword ’semantic’ mean?", "summary_of_the_paper": "This paper investigates the FL under the label and the domain heterogeneity.\nSpecifically, it considered two settings where (i) the labels may not be precisely aligned (ii) the data domain may not be the same. To tackle these two challenges, it proposed the SIMFED framework which first calculates the similarity between the embedding of different local models and then conducts a kernel factorization to solve the problem of heterogeneous data domains.", "summary_of_the_review": "Overall, I found the paper is a heuristic work on personalized federated learning with little justification and little motivation. The scope of its application is unclear. The empirical results do not seem to be sufficient to support the claimed contributions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "IW6nJ26bQnv", "reviewer": "Reviewer_3krS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The research problem that benefits clients by sharing some common knowledge with a well-designed mechanism is an interesting topic\n+ To decompose parameters to lower level rank for local models, to my best knowledge, is a novel idea to explore.\n+ The experiment results are sufficient enough to support their claims\n\nWeakness:\n- There are some research works discussing how to use general knowledge across clients by dividing parameters into different parts, like continual learning in FL, where each client has several tasks to conduct [1]. Also, the basic idea to group users based on metric similarity has been explored in FL as well [2,3];\n- I am a little concerned about the privacy guarantees and extra computational requirements on the server-side. The extracted knowledge from the clients is leveraged by reflecting the difference between the target local model and the other clients. One of the benefits of using FL is protecting users’ privacy. Though clients are not sharing data in this work, we need to pay attention to how to leverage extracted knowledge. Also, the kernel factorization of FC layers and convolutional layers happens on the server-side. I am not sure how much extra computation it would take and whether it would cause a delay in the iterative FL update process.\n- More discussion about the effectiveness of similarity matching and kernel factorization is expected. Furthermore, in most FL work, convergence guarantee is also another point that is expected to discuss besides the experiments.\n\n[1] Yoon, Jaehong, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. \"Federated continual learning with weighted inter-client transfer.\" In International Conference on Machine Learning, pp. 12073-12086. PMLR, 2021.\n\n[2] Duan, Moming, et al. \"FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering.\"\n\n[3] Jeong, Wonyong, et al. \"Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning.\" arXiv preprint arXiv:2006.12097 (2020).\n", "summary_of_the_paper": "Authors propose an algorithm for clients to learn from each other even under the setting of label heterogeneity and domain heterogeneity. The basic idea is they leverage metrics to measure the similarity between the representative knowledge and corporate clients accordingly, which can maintain the generally shared information and reduce information loss at the same time. They conduct appropriate experiments to test the algorithm and provide a comparison with current the-state-of-the-art fl methods. The main contribution is to propose an approach to support clients to benefit each other across different types of labels and domains. They leverage the kernel factorization approach on FC and convolutional layers and update them separately on the server-side. They provide extensive experiment results to support their claims.", "summary_of_the_review": "The authors do provide a fine solution to solving a real problem of generalizing knowledge across clients by kernel factorization. Extensive experiments are provided as well. However, if we consider each part of the techniques, like model parameter kernel factorization, grouping users based on similarity, I may expect more novel contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HuSsl8CQP31", "reviewer": "Reviewer_9bEM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nWell motivated challenges to tackle: label heterogeneity and domain heterogeneity. Then the authors propose similarity matching and kernel factorization to address the issues and show compelling results in experiments. \n\nWeakness: \n1. In table 2, it shows that Similarity Matching + Factorization did not outperform Stand-Alone for the \"Fruits & Vege\" dataset. It would benefit if authors could clarify why, and share some intuitive explanation why or in what scenarios that the proposed algorithms underperform comparing to the \"stand alone\" approach. \n2. The paper would benefit with some explanation why each baseline models & architecture were chosen here for comparison. \n3. It was not clear why SimFed reduces information loss when reflecting the heterogeneous knowledge. \n4. Kernel Factorization outperform SimFam in most datasets, excepts some. It would benefit if the paper could explain why certain dataset behaves that. \n\nMinor typo: \n1. In page 2, it says \"We exhaustively validate our method on both singl[e] domain datastes (Permuted IID/Non-IID) and multi domain datasets (CIFAR-100 20 superclasses) and show our method outperforms the current state-of-the-art approaches on all experiments.\" It should be \"single\" rather than \"singl\". \n", "summary_of_the_paper": "The main contribution of the work are as follow: \n1. introduce the problem of Agnostic Personalized Federated Learning (APFL), and discuss its two possible issues (Label- and Domain-Heterogeneity)\n2. propose the method called Similarity Matching and Kernel Factorization (SimFed) to tackle the problem. \n3. Validate the method in both label- and domain-heterogeneous scenarios, and show its superiority.  ", "summary_of_the_review": "marginally above the acceptance threshold, given that the weakness shared above. Overall it was well motivated problem, and the authors proposed approaches and show compelling results. As pointed out in the above for its weakness, the paper would be strong by providing these insights and intuitions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8l7AihhmEI", "reviewer": "Reviewer_EkKK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem studied in this paper as well as presented experimental results are interesting. However, the paper has some weaknesses:\n\n1- It seems that the paper assumes that models, are neural networks (either with dense layer of convolutional layers). However, this is not mentioned in sections 1 and 3. Furthermore, this paper studies only classification problems. However, authors do not make this clear in the paper especially in section 3. Therefore, the problem setting and assumptions are not clearly defined in the paper. The authors are suggested to revise section 3 to address this issue.\n\n2- The paper needs more rounds of proof-reading and there are some grammatical errors particularly in section 4.1 which makes understating this paper difficult. I think the paper is not well-written and the presentation of the paper in this sense should be improved. For example, in section 4.1, the sentences \"Our underlying assumption is that we can discriminate which local models are helpful or harmful each other if we can differentiate which local data distributions are semantically relevant or irrelevant.\" and \"we can measure similarity between the local domains in the embedding space, which eventually enables us to determine which models are beneficial each other accordingly.\" are not clear that what they want to express. In fact, the authors do not make it clear that what it means by stating the models benefit each other. I think the authors should specify that in what sense the models can be harmful or helpful to each other.\n\n3- There are some personalized federated learning algorithms as well as agnostic federated learning algorithm. I am curious that what happens if we can simply combine these algorithms together to obtain some agnostic personalized federated learning algorithms. It would be great if the authors can compare the performance of their algorithm with some algorithms which are trivial combination of agnostic federated learning and personalized federated learning. This can better shows the effectiveness of the proposed algorithm dealing with label heterogeneity.", "summary_of_the_paper": "The paper proposes a personalized federated learning algorithm to deal with label heterogeneity and domain heterogeneity. Specifically, the paper focuses on the cases where clients performs classification tasks while the label set of data samples are vary among clients. Furthermore, the data samples may be distributed among clients from different domains. ", "summary_of_the_review": "Overall, I think the paper studies an interesting problem. However, there some issues in the paper that need to be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies two aspects of personalized federated learning: (1) Clients having their own labeling scheme. (2) Domain heterogeneity across clients. They propose a way to collaborate across clients by similarity matching. The key novelty is to measure similarity of client pairs, based on on how much their representation layer agrees (measured with cosine similarity). A second novelty is a low-rank factorization of model weights. Empirical evaluations show wins on MNIST, CIFAR10, 100.  

Reviewers had various grave concerns. On the method side, they were concerned that thee is not enough theoretical insight and analysis of the proposed approach, esp. the kernel factorization and its effect.  On the empirical side, they were concerned that comparisons were not made with most recent baselines. There was a large number of PFL approaches published in 2021. e.g. FedBN.  Among these, its worth noting pFedHN (ICML2021) which actually discussed the case of heterogeneous (permuted) labels (their Sec 3.3).

In a discussion, reviewers appreciated the responses by the authors, the additional experiments and ablation studies. Unfortunately however, they found that the paper is not ready for publication in ICLR.
Decision: Reject