Precision Collaboration for Federated Learning
Authors: Sen Cui
Abudukelimu Wuerkaixi
Jian Liang
Weishen Pan
Jianwei Zhang
Changshui Zhang
Fei Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: federated learning
personalized federated learning

Abstract: Inherent heterogeneity of local data distributions, which causes inefficient model learning and significant degradation of model performance, has been a key challenge in Federated Learning (FL). So far, plenty of efforts have focused on addressing data heterogeneity by relying on a hypothetical clustering structure or a consistent information sharing mechanism. However, because of the diversity of the real-world local data, these assumptions may be largely violated. In this work, we argue that information sharing is mostly fragmented in the federated network in reality. More specifically, the distribution overlaps are not consistent but scattered in local clients. In this work, we propose the concept ``Precision Collaboration'' which refers to learning from the informative overlaps precisely while avoiding the potential negative transfer induced by others. In particular, we propose to infer the local data manifolds and estimate the exact local data density simultaneously. The learned manifold aims to precisely identify the overlaps from other clients, and the estimated likelihood allows to generate samples from the manifold in an optimal sampling density. Experiments show that our proposed PCFL significantly overcomes baselines on benchmarks and a real-world clinical scenario.

Under review as a conference paper at ICLR 2023
PRECISION COLLABORATION FOR FEDERATED LEARN- ING
Anonymous authors
Paper under double-bling review
ABSTRACT
Inherent heterogeneity of local data distributions, which causes inefficient model learning and significant degradation of model performance, has been key chal- lenge in Federated Learning (FL). So far, plenty of efforts have focused on ad- dressing data heterogeneity by relying on hypothetical clustering structure a consistent information sharing mechanism. However, because of the diversity of the real-world local data, these assumptions may be largely violated. In this work, we argue that information sharing is mostly fragmented in the federated network in reality. More specifically, the distribution overlaps are not consistent but scattered among local clients. We propose the concept "Precision Collaboration" which refers to learning from the informative overlaps precisely while avoiding the potential negative transfer induced by others. In particular, we propose to infer the local data manifolds and estimate the exact local data density simultaneously. The learned manifold aims to precisely identify the overlaps from other clients, and the estimated likelihood allows to generate samples from the manifold in an optimal sampling density. Experiments show that proposed PCFL significantly overcomes baselines on benchmarks and a real-world clinical scenario.
INTRODUCTION
Federated learning (FL) has drawn considerable interest from variety of disciplines in recent years. FL enables collaborative model learning without the need to access the raw data across different clients, which facilitates real-world scenarios where privacy preservation is crucial, such as finance (Yang et al., 2019), healthcare (Xu et al., 2021) and criminal justice (Berk, 2012). While it is common that the data samples in local clients are non-i.id.. existing research reveals that data heterogeneity could lead to non-guaranteed convergence, inconsistent performance and catastrophic forgetting across different clients (Qu et al., 2022). Despite the promise of FL, an increasing concern how to effectively handle data heterogeneity before FL is applied in real-world data scenarios.
In view of this challenge, an important direction is personalization. A variety of efforts have been made to explore this direction. For example, Ghosh et al. (2020) proposed to cluster the clients according to their sample distributions and build a customized model for each cluster. However, their hypothesis excludes the possibility of knowledge transfer across clusters. Li al. (2021b) enhanced personalized model learning by introducing a global regularization term, which assumed that the shared knowledge was consistent across all clients.
Considering the diversity of local data, in this paper, we study more flexible and general scenario where the distribution overlaps could be fragmented as shown in Figure 1 (a). Since the informative and ambiguous data shards exist simultaneously in another client, collaborating with all data could do harm to the model learning. An interesting and challenging problem is how to selectively collaborate with the favorable part of other clients in privacy- preservvng way.
In this paper, we put forward the concept "Precision Collaboration" for fragmented information sharing. To begin with, we argue that data heterogeneity comes from inconsistent local data manifolds. In particular, the data manifolds of different local clients could share different overlaps. Maximizing the benefit of collaboration requires precise utilization of these overlaps. Moreover, local data are usually gathered from the manifold based on a particular density. If we want to generate data from the manifold, a precise distribution density approximation for each client could facilitate model learning.

---

Under review as a conference paper at ICLR 2023
n
(b) global and local manifold Figure I: Overview of our proposed PCFL. (a) Fragmented distribution overlaps exist among clients; (b) learn the global data manifold and determine the local manifold for each client; (c) the data from other clients lie on the local manifold M are identified informative overlaps: (d) learn a precise local density for synthetic data generation.
(a) data in local clients
(c) identify overlapped data (d) sample from M
To realize our proposed precision collaboration, we develop novel framework named PCFL shown in Figure 1. We assert that the key to precisely collaborative model learning is identifying and utilizing the distribution overlaps scattered in other clients. These overlaps between clients indeed correspond to specific data manifold region. We propose to infer the local data manifold to identify the overlaps. While it is hard to learn the local manifold from the insufficient data local clients directly, we firstly infer the underlying manifold M of the data from all clients, so that the data from all overlapped distributions are utilized for the manifold inference. Then the local manifold M C MM9 of the i-th client could be determined by local data D' as shown in Figure (b).
From Figure (c), the local data manifold M is used to identify the beneficial overlaps from other clients. In particular, if subset of the data from D lies on M' this subset is the overlaps between the i-th and j-th clients. To further boost the local model training, we suggest sampling from M with an optimal sampling probability estimated from local data as shown in Figure 1 (d), which effectively mitigates the potential distribution discrepancy. We highlight our key contributions as follows:
While existing research studies FL under certain assumptions about the information sharing, we investigate a more general learning scenario where the data sharing a common distribution is fragmented among local clients;
We achieve work PCFL. Our framework identifies the meaningful overlaps and excludes ambiguous information from other clients, which avoids potential negative transfer:
precise collaboration for the federated network by proposing frame-
PCFL could be used to improve other SOTA algorithms in plug-and -nd---ady way. Empirical experiments corroborate that PCFL significantly outperforms all baselines on a series of benchmark data sets and a real-world clinical data set.
RELATED WORK
2.1 FEDERATED LEARNING AND DATA HETEROGENEITY
Recent years have witnessed growing attention to federated learning (McMahan et al., 2017), of which several challenges have been concerning topics including communication efficiency (Konečný et al., 2016), privacy (Agarwal et al., 2018) and data heterogeneity (Karimireddy et al., 2020). While data heterogeneity could cause the lack of convergence and the potential of catastrophic forgetting (Qu et al., 2022), there are researchers aiming to tackle the heterogeneity by learning a global model. For example, Li et al. (2020) propose proximal term to restrict the local updates to be closer to the initial model. Mohri et al. (2019) seek a fair model performance distribution by maximizing the model performance on any arbitrary target distribution. Li et al. (2021a) develop MOON that corrects local training by maximizing the agreements of representation between local and global models. Instead of pursuing a balanced performance distribution, we interested in achieving the best performance for each client by precisely learning the shared informative overlaps from others.
2.2 PERSONALIZED FEDERATED LEARNING
In addition to reaching global consensus, personalized model learning also attracts widespread concemn in FL community, which may boost the flexibility of learned models when adapting to local distributions (Cui et al., 2022; Li et al., 2021b). Plentiful research have proposed techniques for
2

---

Under review as a conference paper at ICLR 2023
shared
n
D
D*
D
D
D
D
D
shared
D
D
D2
D=
D
shared
(b) common partial sharing Figure 2: Illustrations of the three assumptions on data heterogeneity. (a) clustered sharing: 1. all information are shared within the clusters; 2. the sharing not consistent across all clients; (b) common partial sharing; 1. partial information within clients are shared; 2. the sharing are consistent across all clients; (c) fragmented sharing: 1. partial information within clients may be shared with others, 2. the sharing are not consistent.
(a) clustered sharing
(c) fragmented sharing
trade-off between local and global models. For example, Fallah et al. (2020) proposed to train local models that can quickly adapt to local data starting from an initial shared model in a meta-learning way. Some works train personalized models by interpolating between global and local models (Deng et al., 2020; Dinh et al., 2020). Li et al. (2021b) achieve such a trade-off through regularizing local models close to the global model. There other works suggesting a partially shared model structure for efficient information transferring (Liang et al., 2020; Collins et al., 2021). Nonetheless, we are concerned that global model is hard to model the various shared information between clients. The fragmented knowledge requires precise identification when collaboratively learning from others.
To mitigate the potential overfitting when learning from limited local data, some works also attempt generative methods to improve the model performance (Du & Wu, 2020; Zhu et al., 2021). In particular, Zhu al. (2021) regulate local training with the distilled knowledge from all clients. Du & Wu (2020) lead into GAN for generating similar data for local clients. However, generating data in an arbitrary density could result in distribution discrepancy. An optimal sampling density may present more benefits for local learning tasks.
3
NOTATIONS AND PROBLEM DEFINITION
3.1 NOTATIONS
Suppose there are N clients in federated network, each client owns a private dataset D with n data samples. The dataset D* {x*, k yk) consists of the input space X and output space y*, We use {x. y denote data point, and 2 E .M denotes the data manifold. The input space and the output space are shared across all clients. In the following, we also use D to denote the i-th client without causing further confusion.
The goal of each client is to learn a best model to predict the label by collaborating with others. For example, McMahan et (2017) propose FedAvg, which learns global model f for all clients by minimizing the empirical risk over the samples from all clients, i.e.,
N
(I)
===1" k=li=l
where F is the hypothesis space and denotes the loss objective of all clients. From Eq. FedAvg assumes that the data from different clients associate with a common data manifold M and sampling density (2) i.e.,
V D' D' D',..,DN-IJ s.t., ZE D C C,, ((()).
3.2 ASSUMPTIONS ON DATA HETEROGENBITY
However, i.i.d. assumption in (2) is largely violated as the local data distributions may be significantly distinctive. In this event, learning a consensus by averaging the local gradients could cause severe performance degradation on certain clients (Li et al., 2019b; Cui al., 2021). There research studying federated learning with non-i.i.d, data and the assumptions on data heterogeneity mainly from two perspectives.
Clustered sharing. As shown Figure 2 (a), the clients partitioned in each cluster own a common data manifolds (M³) and sampling density (p) (z)), i.e.,
3

---

Under review as a conference paper at ICLR 2023
ViE NO. (0.,,..,,K (K < N), s.t., 2 D' M and 2 (z).. (3) From (3), clustered sharing requires that all message is shared within the clusters, and there is no knowledge transferring across clusters.
Common partial sharing. From Figure 2 (b). common distribution overlap is shared across all clients. Mcanwhile, each client owns specific knowledge that cannot be leveraged by others.
Formally, each client associates with specific data manifold M and the overlapped region of the manifold is shared across all clients, i.e.,
i [0, 1,..,,,. . 1), 8.1,, 2 ZE D' C CM and z ~ p2(2),
Vijjk E (0. -1 ( s.t., M nM' # and n c MMM. Compared with the previous assumptions above, study more general scenario fragmented sharing, where the shared distribution overlaps are scattered among the clients. Besides, these overlaps are inconsistent across all clients shown in Figure 2 (c).
Fragmented sharing. The local data 2 E D' are sampled from the local manifold M in a particular density p' (2), and there exist overlaps among data manifolds, i.e.,
2 E M' C Rd
2~ (1)
(5a)
3 j&j& E f0, 1.... N - 1}, s.t., M MM #
(5b)
where d in (5a is the dimension z. Eq.(5b) implies the shared overlaps may not be consistent across all clients, e.g., n M 0.
4
METHODOLOGY
4.1 PRELIMINARY: NORMALIZING FLOW
Normalizing flow. The generative method NF achieves exact likelihood estimation through an invertible transformation from a known distribution to a complex target distribution. Given a target dataset D {20, ..., Zn-1) } 2 Rd and base variable
Ra with a known density pe(e).
classic NF methods learn diffcomorphism ʃ 6 g(z;) which maps pz to the density pe:
(2) = Pe (9(=)) |dct Jg (9(2))
(6)
where det Jy (9(2)) E Raxd denotes the Jacobian matrix evaluated at g(2). Since g is bijective, it trackable and (6) could be effectively computed. By fitting the dataset D, the approximated distribution p' (z) is optimized through pushforward operation. To enhance the scalability of g, one could compose several diffeomorphisms, g = 9n-1 .... Ii o go for larger model capacity.
4.2 AN OVERVIEW OF PCFL
M unshared region generate data with the estimated density pa (=(2)
N-1 J- (M nM) shared region learn from the overlapped data
M'
Figure 3: Two meanings of Precision Collaboration. (1) for the shared region of the local manifold
M', PCFL precisely learns from the overlapped data of other clients; (2) for the unshared region of M PCFL generates synthetic data with the exact density p (z).
I Learning optimal personalized model f for the i-th client expects sufficient utilization of the shared overlaps with other clients. However, due to privacy concerns, one cannot identify these overlaps with direct access to the raw data. We suggest leveraging the overlaps via the learned data manifold to prevent privacy leakage. As shown in Figure 3, in general, our proposed precisely collaborative learning scheme contains:
for the shared overlaps in other clients, we aim to precisely learn from the shared overlaps identified by the local data manifold M
for the remaining unshared region of M we expect to advance models with the generated synthetic data from M in an optimal sampling density.
4

---

Under review as a conference paper at ICLR 2023
4.3 PRECISION COLLABORATION I: LEARNING FROM THE SHARED OVERLAPS
From Figure (a), different clients could share different distribution overlaps, and the distribution overlaps associate with the overlapped region of the local data manifolds. While the data manifold of local clients is mostly agnostic and hardly be inferred by limited local data, we propose to learn the global data manifold with the data on all clients. this way, all data are utilized and contribute the manifold inference.
hu))
g(2)
Split
u
ul pur (u')
2 ~ P2(z)
e
(u)
9 (e)
U (0 vector)
(=(2)
Par(u))
Figure 4: Illustrations of the manifold learning via a NF method. For a complex distribution Pa (z), we learn tractable injective chart g h, which models (=(=) simple distribution p (u').
Learn the global manifold. The data = E are usually supported on an unknown diver-dimensional manifold M. In realization, we propose to NF method to learn the global manifold M
Given the data z, a bijective transformation go is used to obtain the latent representation e E E,
go(2), where z 90 (e),
which avoids the risk of information loss during encoding.
While a classical NF requires the latent variable e E to have the same dimension with the data space Z, following the design in (Brehmer & Cranmer, 2020), we separate the latent space E U x V as shown in Figure 4, where U IRd denotes the coordinates the manifold. V = 0d-d' denotes the remaining coordinates, which are the directions orthogonal to the manifold.
To model the density Pu (u), transform the variable to the variable u' with the given density par (w') using a bijective model ha:
(8) where Split(e) denotes deleting the d - d' dimensional 0 vector from e, and Pad(u) denotes the inverse operation. Please note that in the rest of this paper we will use gā to denote Split o go and 9g denotes 9p" o Pad.
u' = hu), where Split(e),
After the training of the model with the parameters 0 and Φ, we learn diffeomorphism from the data = ~ (z(z) to a lower dimension space u' ~ pu (u') with the encoder h o gg. This means that we transform the original data manifold MS to the projected data manifold U". Note that the decoder is the inverse of the encoder. The data is reconstructed given the latent variable u' E U', i.e.
9 h (u').
Following the work in (Brehmer & Cranmer, 2020), we train go and h by two-stage optimization framework. In particular, we first train go to obtain the projection onto the manifold by minimizing the reconstruction error. Then, optimize he to approximate the density by maximizing the likelihood (Brehmer & Cranmer, 2020). More implementation details could be found Appendix due to the page limit.
Determine the local manifold. A local data manifold M' should contain the local data D'. Con- sidering that the original global manifold M and the latent space U' (U' Rd is topologically equivalent, we propose to approximate the local manifold with the projected representation:
M g -- o h (U"),, where U = [he o gō (x))))
where th denotes the set of the samples transformed to U from Dí, and T is called the projected local data manifold, which is computed as the convex hull of U
Note that th may have clustered structure. In the realization, The union of the convex hull of all clusters the projected local data manifold, and the original local data manifold (M') could be obtained by M = g -1 o hr (U").
could also firstly cluster the U
Identify the data overlaps from other clients. Since we cannot determine the data overlaps directly because of privacy concerns, propose identify the overlaps using the learned local manifolds.
5

---

Under review as a conference paper at ICLR 2023
Note that the data overlaps correspond to the overlaps of the data manifolds. For example, suppose D is a subset of D', if D lies on M', D is the data overlap between and D From Eq,(9),
M is reconstructed by Do 1-1 o hhg with U Therefore, D could be identified as follows:
D 2
2 lhg gā E U k 1, l,....,r?? ne CM'.
(10)
From (q0( the data overlap Did is the subset in which each sample is transformed in the projected local data manifold U
By learning from the overlaps identified from other clients, we have the following objective,
N
1 E(((s))))) a. N Σ
min J:EF n ==1
(11)
k=0,
where a > O is the regularization parameter, which controls the trade-off between the risk on the i-th client and other clients.
4.4 PRECISION COLLABORATION II: LEARNING WITH AN OPTIMAL SAMPLING DENSITY
In Sec.4.3, learn personalized models from the data overlaps between clients. However, the model performance on the unshared data cannot be improved by collaborating with others. The specific region M has overlap with others, which is formulated
M 9o h (U?', where U U -
U o- (U' nU*),
N-1
(12)
We propose advance the model by generating data sampled from the local manifold M.
While an arbitrary sampling density could generate data D deviated from the local distribution d(D", D*) c, this could induce bias to the learned model. An optimal utilization of the synthetic data expects a sampling density close to p¹ (z). Therefore, we propose to sample from M with the exact estimation of (2).
Exact likelihood estimation. Note that we learn the manifold by applying a normalizing flow framework, which achieves the exact likelihood estimation simultaneously.
Since we learn the global data manifold, the global data density p=(=) is transformed to (u'). For the local data density pl (=), we have the following proposition.
Proposition 1. (proof in Appendix) For any data point E .M the local density (z()) satisfies
p=())) = c. Pu (he o ge(z)) |det Jng (he go()))
|det [J (g0 (x)) Jog (gō (ob()))
(13)
where e is a proportionality constant, and Jn, and Jnž are the Jacobian matrix of h& and go-
respectively.
From the Proposition 1, to sample = E EMC in the density (2), we could firstly sample u' ~ pw and choose u' T defined in Eq.(1 Then we transform the sampled u' to the data space by 9ø
ho (u'). The final objective is follows
Σ ((f(x))) Rer ni
min
1
N
k=
(14)
where the sampled (I, y) in the third term satisfies (x, y) E M
4.5 MORE DISCUSSION ON PCFL
From the objective formulated in (1.),,), PCFL advances a generative framework for efficiently collaborative learning. In addition the properties summarized as follows, more discussions could be found in Appendix,
Scalability: Since PCFL achieves a smarter utilization of data from other clients, it is
pluggable for other FL. algorithms. Experiments in Sec. 5 also verify that PCFL could benefit other SOTA baselines;
Similarity metric: PCFL identifies the overlaps among federated networks, which inspires a novel metric for measuring the similarity between clients. More information about it could be found in Appendix.

---

Under review as a conference paper at ICLR 2023
0.1
ground truth
(c) FedAvg
.0.4
the projected data manifold
local training
(d) PCFL
Figure 5: Illustrations of the synthetic experiments. (a) the learning tasks of the six clients; (b), (c) and (d) are the performance of the models learned by local training, FedAvg and PCFL; (e) the learned projected global data manifolds. The points denote the samples from different clients. The colored lines denote the identified local manifolds.
EXPERIMENTS
We intuitively show the motivation our method by conducting experiments on synthetic data. We compare method with various baselines wide range of benchmark datasets, including image and tabular datasets. More importantly, the practicability of our method is validated real-world clinical federated scenario eICU dataset (Pollard al., 2018). The source codes made publically available at https::/ g thub. com coom/pffle pecf//cffl
5.1 SYNTHETIC EXPERIMENTS
Synthetic data. Suppose there are 96 clients: D {1,2..., .6). The data points {x, y) is generated from two objectives y = sin(x) + or y = -sin(x) ) + shown in Figure 5 (a), where (0, 0.1) denotes label noise.
Fragmented data overlaps. To generate heterogeneous and overlapped local data, we sample from the overlapped ranges. In particular, separate the input space X into four intervals [0, $1. [ T). [sr, 3d and 3x 2T), and each client randomly chooses two different intervals to sample data. To create conflicting learning tasks, the label of the selected 48 clients is calculated by = sin(x) + and the label of the remaining 48 clients is calculated by = sin((r) + e.
In this setting, learning global model for all clients could hurt the model per- formance there are two conflicting learning tasks as shown Figure 5 (c). The best way of collaborative learning for each client is identifying the data overlaps which are sampled from the identical objective with the same inter- vals. For example, D consists of the data sampled from [0, 3 and [ T]. while D consists of the data sampled from 12 π| and [x, shown in Fig- ure (a). Learning an optimal model for D needs to precisely identify the data overlap sampled from [] in D From Figure (c), PCFL efficiently obtains local data manifolds and identifies the data overlaps between clients. Therefore, PCFL learns better model by precision collaboration which maximizes the benefits and avoids potential negative transfer from other clients shown in Figure (d).
Table 1: CIFARIO
100
Accuracies CIEARIO
Methods
ACC (8)
FedRep
local
68.9-1.1
FedAvg
PCFL+buscline
FodAvg
72.3
90
5
FedProx
71.5
Accuray
Fed-MILL
68.4 68.422.2
80
PerFedAvg
6.
LG-FedAvg
69.2
TP.3
FedPer
82.24.
70
46.9
FedRep
82.2-1.a
APFL
64.5
L2GD
10.0g.
60
FedAvg
FedRep
Ditto
82.1
PCFL
87.3
Figure 6: CIFARIO
Datasets. We adopt three benchmark image datasets: CIFAR10 (Krizhevsky al., 2009), FEM- NIST (Caldas et al., 2018), CclebA (Liu et al., 2015), and a tabular dataset Adult (Kohavi et al., 1996). We create the federated environment with data heterogeneity for CIFAR10 by randomly

---

Under review as a conference paper at ICLR 2023
allocating several classes each client following the work (McMahan et al., 2017). We use K to denote the number of clients and to denote the number of classes in each client. For CIAFRIO. K = 150, S 5 means there are 150 clients and each client contains 5 classes of images. For FEMNIST which has 10 classes of handwritten letters, we consider the setting of K 200, S = 5. The number of samples in each client is determined according a log-normal distribution (Li et al., 2019a). The task on CelebA is to classify whether the celebrity in the image is smiling (Li et al., 2021b). There are 545 clients and 21 samples per client in average. For the tabular dataset Adult, the task is to predict whether an individual's income is beyond 50K/year based on some census features, including age, race, workclass, etc. Following the setting in (Mohri al., 2019), all individuals are split into two clients. One is PhD client and the other is non-PhD client.
5.2
BENCHMARK EXPERIMENTS
Table 2: FEMNIST
Accuracies FEMNIST
Baselines. We compare our method
Methods
ACC (%)
100
local
FodRep
with various baseline, including global and personalized methods. Global baselines include: 1) Fe-
local
99.9.
FedAvg
FodProx
FedAvg
84.9.1.7
85.42.
90
94.1
Accuray
bed-MILL
60.7
dAvg (McMahan et al., 2017); 2)
PerFedAvg
26.8
LG-FedAvg
35.2.114
FedProx (Li et al., 2020). Person-
FedPer
82.9
alized baselines include: 1) Fed-
FedRep
82.8
MTL (Smith al., 2017); 2) PerFe-
APFL
62.3+1.4
60
50.9
L2GD
14.5
dAvg (Fallah et al., 2020); 3) LG-
85.4
local
FedAvg*
FedRep
FedAvg (Liang al., 2020); Fed- Per (Arivazhagan et al., 2019); 5) Fe-
PCPT.
91.924
Figure 7: FEMNIST
dRep (Collins al., 2021); 6) APFL (Deng al., 2020); 7) L2GD (Hanzely & Richtárik, 2020); 8) Ditto (Li et al., 2021b).
Experimental Results. The accuracy of all methods on CIFARIO dataset are shown in Table 1'. PCFL outperforms all baselines on this classification task. Since each client has insufficient data samples (n 333), FedAvg (72.3%) learning from all data has a better performance compared with local (68.9). From Table 1, FedRep (82.2%) surpasses other baselines by learning global feature extractor. As a pluggable method, PCFL could be used to enhance the performance of other art methods. From Figure 6, PCFL improves the performance of FedRep by 5.1%, which indicates that PCFL effectively identifies the informative knowledge from others.
Similar phenomena could also be found in the experimental results FEMNIST. FedAvg achieves better performance compared with other baselines because of the relatively slighter heterogeneity. PCFL also outperforms all baselines both learning tasks. Moreover, PCFL successfully boosts the models learning on the three baselines as shown in Figure 7. More experimental results could be found in Appendix.
Table 3: Adult
Accuracies on Adult
Methods
average
non-PhD
PhD
98
local
Ditho
local
8
8344.1
70.2
90
FedAvg
PCPl--baseline
FedAvg
83,3
8344_
72.9..2
FedProx
8
83.5
71.8
Fed-MITL
83.3
83.42.2
69.12.4
800
PerFedAvg
83.4+.2
83.6-
70.2=2.0
APFL
83.0+
83.2-.1
69.14.3
L2GD
70.4+.0
70.4..0
71.8
Ditto
8
83,5
75.7
Twvel
Felligg
Diloo
Sccal
FalAvg
Diset
PCEL
83.5+
83,6
77.3
Figure 8: Adult
We also evaluate the availability of our proposed method on a tabular data Adult, and the results on the two clients are shown in Table Since the classifier on Adult is one-layer MLP. LG-FedAvg. FedPer and FedRep degrade into local training. Compared with PhD client which has 413 training samples, non-PhD has more than 30000 training samples. From Table all methods achieve similar performance on non-PhD client. Because of the severe distribution discrepancy, naive averaging may not lead to optimal accuracy on non-PhD client. From Figure 8, by leveraging the favorable data
We show the best performance of PCFL in all tables. For all figures in experiments, show the results of Local, FedAvg and the hest baseline.
8

---

Under review as a conference paper at ICLR 2023
in non-PhD client and the learned manifold of PhD client, PCFL substantially improves the model performance of local training (? 7.1%).
5.3 REAL DATA EXPERIMENTS
To further verify the practicability of PCFL, we conduct experiments on a real-world clinical dataset eICU (Pollard al., 2018). eICU contains the patients ICUs with hospital information. Naturally, hospitals located in different areas are local clients as in (Cui et al., 2022), where the patient data are kept confidential. We preprocess the data following the work (Sheikhalishahi et al., 2019) and each data spans 1-hour window. The task is to predict in in-hospital mortality of each instance using the 48 18-hour monitoring data.
Table 4: eICU
AUC on eCCU
In the experiments, we randomly select hospitals in the federated network. We use a Bi-LSTM to implement this bi- nary classification. We use AUC as the metric due to the severe label imbalance (more than 90% samples have negative labels).
Methods
AUC(%)
90
local
73.7+1.4
85
FedAvz
PCFI--bavline
FedAvg
73.2..5
Fed-MIL
FedProx
77.221.6
78.2+-2
30
7/4
714
LG-FodAvg
PerFedAAg
73.8
74.5..2
75
7p1
FedPer
74.3
FedRep
74.1
From Table 4, PCFL still maintains the best model utility. FedAvg achieves a comparable performance compared with local. While different hospitals own different populations which could result in data heterogeneity, Ditto learns a robust personalized model and achieves better performance (78.3%). The results shown in Figure prove that PCFL could also benefit the baselines in real-world scenarios.
APEL
68.3g.8
L2GD
72.0
local
FodAvg
Ditt
Ditto
78.3
80.0g.6
Figure eICU
5.4 ABLATION STUDIES
While PCFL formulated in (q.(14) consists three terms: 1). loss of local training; 2). loss on the identified overlapped data from other clients; 3). loss on sampled data from the manifold, to analysis the effect of each component, we conduct ablation studies several datasets. More ablation studies and implementation details could found Appendix.
When 0, only the local data and sampled data from local manifolds are used for model learning. When 3 = 0, only the local data and the identified overlapped data from other clients are utilized for model Icarning. Experiments in Table 5 demonstrate that 1). both the identified distributional overlaps (3 = 0) and the data sampled with a learned distribution density (a = 0) facilitate the
model learning: 2). the identified overlaps (B 0) could achieve more performance gain than the generated data (a = 0).
Table 5: Ablation studies PCFL formulated Eq.(14).
Dataset
local
0
PCFL
CIFARTO
68.9+1.1
76.7
79.4..7
84.1
FEMNIST
59.9
75.5
82.7+.8
89.7
CelebA
69.3
82.642.
80.8
85.8
eICU
73.7+.4
76.6+.3
77.4+.5
78.6
CONCLUSION
In this paper, we propose a precise collaboration framework PCFL for more general FL learning scenario, where the fragmented and shared knowledge is distributed among other clients. Experiments on benchmark datasets and real-world clinical dataset verify the superiority of our method because of optimal and precise utilization of the shared information. Our framework determines the overlaps between clients, which suggests several attractive topics, such as identifying malicious clients or noisy data in the federated network. Moreover, PCFL encourages novel similarity metric stated in Sec. C.I. This metric could be used to provide incentives or impose charges on each client, to promote the practicality of FL in real-world applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Naman Agarwal, Ananda Theertha Suresh, X Yu Felix, Sanjiy Kumar, and Brendan McMahan. epsgd: Communication-efficcann and petterentialll-privatte distributed sgd. In NeurIPS, 2018.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv:1912.000 2019.
Richard Berk. Criminal forecasts of risk: A machine learning approach. Springer Science & Business Media, 2012.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33:442 453, 2020.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint ar arXiv:1812.01090 2018.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. 2021.
Sen Cui, Weishen Pan, Jian Liang, Changshui Zhang, and Fei Wang. Addressing algorithmic disparity and performance inconsistency in federated learning. Advances in Neural Information Processing Systems, 34:26091-26102 2021.
Sen Cui, Jian Liang, Weishen Pan, Kun Chen, Changshui Zhang, and Fei Wang. Collaboration equilibrium in federated learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 241-2511 2022.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv::203.. 13461, 2020.
Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. arXiv preprint arXiv:2006.08845 2020.
Wei Du and Xintao Wu. Advpl: Adversarial personalized learning. In 2020 JEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 90-98. IEEE, 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with
theoretical guarantees: A monel-agnostic meta-leaming approach. Advances in Neural Information Processing Systems, $3:3557-3566 2020.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586- 19597, 2020.
David Hallac, Jure Leskovec, and Stephen Boyd. Network lasso: Clustering and optimization in large graphs. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 387-396, 2015.
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.055// 2020.
Alexander Jung. Networked exponential families for big data over networks. IEEE Access, 202897-202909, 2020.
Alexander Jung and Nguyen Tran. Localized linear regression in networked data. IEEE Signal Processing Letters, 1697)1100- 2019.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Ron Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tre -tree hybrid. Kdd, volume 96, pp. 202-207, 1996.
10

---

Under review as a conference paper at ICLR 2023
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05592,, 2016.
Steven G Krantz and Harold Parks. Geometric integration theory. Springer Science Business Media, 2008.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10713-10722) 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smithy. Feddane: A federated newton-type method. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pp. 1227-1231. IEEE, 2019a.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. International Conference Learning Representations, 2019b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 450, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357-6368. PMLR, 2021b.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philipee Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint vxXiv:2001.01522 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015,
Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera Arcas Communication-efiee effficienn learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, pp. 4625-4625. PMLR, 2019.
Tom Pollard, Alistair EW Johnson, Jesse Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1-13 13, 2018.
Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 10061-10071, 2022.
Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benchmarking machine learning models on eicu critical care dataset. arXiv preprint arXiv:1910.00964, 2019.
Virginia Smith, Chao-Ka Kai Chiang. Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv: 1705. 10467, 2017.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint 1910 10252, 2019.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions Information Forensics and Security, 15:3454-3469 2020.
11

---

Under review as a conference paper at ICLR 2023
Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5(1):1-19, 2021.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in Neural Information Processing Systems, 32, 2019.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. arXiv preprint arXiv:2105.10056 2021.
A THEORETICAL PROOFS
A.1 PROOF OF PROPOSITION
Suppose there is smooth and injective mapping 96 Rd + Rd with d' S d, U E R is the latent variable and has Z == In (U). From differential geometry (Krantz & Parks, 2008), have
(2( = Pu (g)(2)) det J/3 (2)( Jog (oā(z))
(15)
Suppose there is a smooth and bijective mapping ha : Id. - Rd', U' E Rd is the latent variable which has U = hz(U"). We have
Pu(u) Pw' (u)( det Jh. (ho (ho(w))
(16)
According to the chain rule, combining (15) and Eq.(1 we have
1/2
p=(z) = Pw (hs o g% |det Jhe (hs o (b))) det J (0b(z)) Jog (oō(
(17)
Since we learn a global manifold MS with the data from all clients, the density of the data from all clients pg (z) is approximated in (17)) From the definition of M in (12), if there is data point 2 E M z will cannot be sampled from any other manifolds M (j F i) but M',
Vz E M,, s.t., # + +)
(18)
Therefore, we have
E Milk M²)= (2 E M M2 (
p E (MM pl(2 E Mi)
pl(2, E My).
vl Mi
(19)
Combining with Eq.(1 for E M we have
-1/2
(=( Par (he (x)) det Jn, (hs 0o(2)) and Proposition holds.
det J (96(2)) Ja; (o)
(20)
B PIPELINE OF OUR FRAMEWORK PCFL
The pipeline of the global manifold learning M is elaborated in Algorithm 1. We leam global manifold model in the federated learning setting. There are two phases of training. Firstly, only the parameters of go are updated as in Line 5-7. Then the parameters of h are updated as in Line 9-10. The learned manifold model 97 ha is utilized in our framework PCFL, whose pipeline is elaborated in Algorithm 2. To begin with, the local manifolds of clients are extracted based (q) and the distribution overlaps are calculated based on (10) Since only the borders of convex hulls are exchanged, there is no leakage of sensitive information. The data from the overlapped distribution of other clients are used to train the models. They are utilized by transmitting the average gradients through the server as in Line 7-14.
12

---

Under review as a conference paper at ICLR 2023
Algorithm Learn the global manifold in the federated learning framework
Input: epoch Tm. batch size Bms initial manifold model M with the parameters and
Φ.
1: for wort= =0,.... mm do
2: randomly select a subset of clients St
3: for client D S₁ in parallel do
4: draw mini-batch x²
mm D
5: if < / then
6:
calculate the loss: Bm Li=i
Bin
24 Og (2') ||:
7:
then calculate the gradients of loss with respect parameters 8:
8:
else
9:
calculate the loss: B.n Eenn (logpu (he go (=')) log det Jn (hs o 90 (=')))
10:
then calculate the gradients of loss with respect parameters d;
11:
end if
end for
13: Server aggregates the gradients of selected clients and update the parameters 0 and ø.
end for
15:
Output: the learned manifold model and hg-
Algorithm 2 Federated learning framework PCFL
Input: epoch T, batch size B, initial models (N.....NN-. hyperparameters and
8;
1: all the clients determine the local manifold M and U based on Eq,(9), and send T to the Server.
2: the Server calculates the overlaps of U between clients, calculates U based on Eq.(1 and sends them each client:
3: for t =0 0....,
do
4: randomly select a subset of clients S₁,
5: the selected clients send their local models to the Server;
6: for client D' E S, in parallel do
7: draw mini-batch (e',v")= Dr:
8: calculate the loss (()t,)")) (E(f(r")) + 8 (x v), and update the
model f³ using the gradients loss; for 1: ..., N. draw mini-batch (I* y²) Dirk
9:
do
10:
calculate the loss a EEzt wtt)emmee ((")))))))))))))))))))))())))( )*), and update the model f using the
gradients of loss; end for
13:
the Server aggregates the parameters of from other clients and send the average to the
i-th client;
14:
then the i-th client D' updates the model f with the received parameters and local gradients. end for
15:
16:
end for
17:
Output: the learned personalized models { PN-1).
C MORE DISCUSSIONS ABOUT PCFL
C.1 A NEW METRIC CLIENT SIMILARITY
Our framework PCFL inspires novel metric for measuring the similarity between local clients. For example, suppose the i-th client and j-th client has the identical local manifold M MM the similarity between clients is close 1. On the contrary, if the two local manifolds are disjoint M n M 6, the measured similarity should be O. In particular, we propose to measure the similarity as the Intersection of Union (IoU) of the projected local manifold,
S(D', D') loU(U",UU))
(21)
13

---

Under review as a conference paper at ICLR 2023
A communication-effitee client-level collaboration. Our proposed metric allows efficient collabo- rator identification which reduces the communication and computation overhead. For example, we require D' to collaborate with certain clients who have higher client similarity:
N-1
a²
min FEF Ei=D,s L n
E
Ei(f(x)) (xf)
(22)
where é 2 is pre-defined threshold. Note that the objective in (2)) is different from clustered FL. methods. Clustered FI. methods learn a common model for each cluster while (2. learns a personalized model for each client. Experimental results shown in Sec. E.2 verify that this method achieves a comparable performance while reducing communication and computation overhead.
Previous work has explored the problem of identifying similar datasets in graph network for downstream learning tasks (Hallac et al., 2015). In particular, Jung (2020) formulate the learning from distributed local datasets as a convex optimization problem, and proposes to cluster the local datasets according to the learned parameters. Jung & Tran (2019) extend network lasso methods in regression tasks under a clustering assumption. These cluster-based methods could be applied in federated learning with a proper design for prrvacy-preserving.. In our experiments, we use network lasso to cluster the local datasets under the federated setting. In Table 6, we show the comparison of PCFL and the clustered methods. Our method outperforms all cluster-based methods, which demonstrates that precision identification of overlaps in other clients facilitates model learning, Moreover, an interesting direction is the application of our proposed similarity metric in the graph network. For example, the manifold learning of local datasets in the graph network may also be used for similarity measurement.
Table 6: More experimental results on eICU
Methods
AUC (%)
local
73.7
FedAvg
73.2+.5
FedProx
78.2+.2
Fed-MTL
77.2+1.6
PerFedAvg
73.8
LG-FedAvg
74.5+.2
FedPer
74.3+.7
FedRep
74.1
APFL
68.3
L2GD
72.0+.6
Ditto
78.3
Clustered FL
74.7+.3
Network Lasso
76.3
ours
80.0
C.2 COMPUTATION COMPLEXITY AND OPTIMIZATION EFFICIENCY
From Algorithm and Algorithm PCFL is realized by a two-staged optimization framework. For the training of the normalizing flow in the first stage, PCFL L learns a global model for all clients, which has the computation complexity as FedAvg. For the identification of the manifold overlaps, it has O(1) time complexity the server only computes it once. For the training of local models the second stage, PCFL learns a personalized model for each client, which has the computation complexity as other personalized methods. From the above all, PCFL achieves similar computation complexity as basclines.
A classical NF method requires fixed dimensionality of the latent space, which is the same the dimension the data. In this case, learning such NF model could bring huge computation overhead when the data is dign-dime In the first phase of our framework, leam low-dimensions manifold in a NF method, which significantly reduces the computation overhead.
Our method learns from local data, data overlaps of other clients, and sampled data in the manifold. By precision collaboration, we avoid learning from all data. We make comparisons of run-tin on-time consumption with the baselines. The experiments are conducted on the same device NVIDIA GeForce
14

---

Under review as a conference paper at ICLR 2023
RTX 2080 Ti. The results eICU dataset are displayed in Table 7. As pluggable method, the time consumption of PCFL is comparable to the corresponding baselines. Fed-MTL involves computing the correlation of the parameters among all client models, which could result in more computation overhead.
Table Run-time consumption comparisons
Methods
Run-time consumption
local
33 min
FedAvg
57 min 41
FedProx
56 min 35
Fed-MTL
101 min
PerFedAvg
79 min 47
LG-FedAvg
55 min 27
FedPer
57 min 43
FedRep
40 min 37
APFL
92 min 13
L,2DD
63 min 4
Ditto
71 min 34
PCFL
74 min 27
C.3 PRIVACY PRESERVING
PCFL maintains data confidentiality as baselines because there is no shared data between local clients. PCFL achieves
as baselines because our framework learns models by commu- nicating model parameters only. Federated learning may need further exploration to maintain data privacy. Some researchers claim there information leakage when sharing models or gradients (Zhu et al., 2019). To alleviate this issue, there are research proposing to apply other techniques to FL. methods, such as differential privacy (Wei et al., 2020), secure multi-party computation, etc. PCFL is also compatible with these techniques.
D ABLATION STUDIES
D.1 MORE ABLATION STUDIES ABOUT PCFL
PCFL is pluggable for other FL algorithms. We test local, FedAvg, FedRep and Ditto which are implemented with/without our method as in Table 8 and Table 9. In the datset Adult, all individuals are split into two clients, one of which is PhD client and the other non-PhD client. The non-PhD client contains 32148 training samples while the PhD client contains 413 samples. Therefore the non-PhD client of Adult can not benefit much from federated learning methods. other datasets, our method boosts the baselines by large margins.
Table Experiment results of PCFL implemented on CIFARIO. FEMNIST, and CelebA (%)
FedAvg
PCFL (FedAvg)
FedRep
(FedRep)
CIFARIO
68.9
84.1 15.2)
72.3
79.5 7.2)
82.2
87.3.1.2 5.1)
FEMNIST
59.9
89.7 (t 20.8)
85.44.
90.241.2 (f 4.8)
82881.4
91.9+.4 (1 9.11
ColebA
69.3
85.8 (t 16.5)
85.2
89.5 (1 4.3)
68.1
71.2 (f 3.1)
Table 9: Experiment results of PCFL on eICU and Adult (%)
Datasct Adult non-PhD PhD
local
PCPL (local)
FodAvg
PCFL (FodAvg)
Ditto
PCFL (Ditto)
83.44.1
83.64.1 (f .2)
8
83.6 (1 .2)
835
83.6.. .1)
70.24.4
77.3 (f
729+ 2
76.8 (1 3.9)
75.7
76.8
eCCU
73.741.4
78.62.4 (1 4.9)
73.2
80.0 (f 6.8)
78.3
79.64.4 (f) 1.3)
E EXPERIMENTS AND IMPLEMENTATION DETAILS
E. MORE EXPERIMENTS
The experimental results on CelebA are shown in Table 11 and Figure 10. Moreover, we conduct experiments FEMNIST more heterogeneous settings with more clients. We partition the dataset
15

---

Under review as a conference paper at ICLR 2023
into 400 clients with the Dirichlet distribution aoo (0.1) and (0.5) following the work in (Wang et al., 2019). We compare our method with the baselines, and the results are shown in Table 10. With more clients, each client has fewer training samples. Local method shows poor perfor- mance (63.6% 6% in anr (0.5)). Global methods (FedAvg and FedProx) achieve better performance under less heterogeneous setting ( (Dirano(0.55 degrades. Under two settings (Dir (0.1) and (ii400(0.55) PCFL outperforms all baselines by identifying the informative overlaps for each client.
while the performance of personalized methods
Table 10: More experimental results on FEMNIST (Acc %)
methods
D0.1
Diraoo
local
71.8+.8
63.6+.4
FedAvg
69. 1..5
80.6+1.7
FedProx
67.84.3
79.9+.2
Fed-MTL
81.84.9
60.1+.4
PerFedAvg
12.4
46.7+.8
LG- G-FeAAAg
86.8..8
49.5+.5
FedPer
91.1+.3
76.8+.2
FedRep
91.8
74.61.3
APFL
79.9
60.91.5
L2GDD
77.6
39.9
Ditto
91.7+.7
82.9+
PCFL
96.1+.
88.34.6
E.2 EVALUATION ABOUT THE PROPOSED NOVEL METRIC
Table 11: CelebA
Accuracies on CelehA
In Sec. C.I, we propose novel metric for measuring the distance between local clients, which could be used for communication- efficient client-level collaboration. We conduct experiments on eICU dataset, in which select the most similar 7 clients for each client learn personalized model. The experimental results shown in Table 12.
Methods
ACC (96)
Incal FedAvg"
FedRep
local
69.3 +1.1
PCFL-baseline
FodAvg
85.2+21
90
FedProx
81.2+1.2
Accurncy
Fed-MIL
68.2+.4
80
PerFedAvg
68.64.8
LG-FedAwg
68.411.2
70
44.5
FedPer
68.2
FedRep
68.1+.0
60
APFL
71.44.6
L2GD
67.912.0
local
FedAvg
FedRop
Ditto
84.5+.6
Figure 10: CelebA
PCFL.
89.5
From Table 12, our method for identifying the collaborators achieves a comparable performance compared with baselines and reduces computation and communication overhead by collaborating with a subset of local clients.
To explore the effect of on the performance of the learned models, we E by controlling the number of clients to collaborate for each client. There are 14 clients in eICU dataset. We test the number of the collaborator (C) to be 1, 3, ... etc. The results are shown in Table ??. When C the learned model achieves the highest AUC (78.0). When C >>> 7, the performance tends to remain unchanged.
E.3 IMPLEMENTATION DETAILS
Our method is implemented with Pytorch and all experiments are run times to calculate the average results with stds. We use a four-layer MLP for the synthetic experiment, three-layer MLP for FEMNIST, two-layer CNNs for CIFAR 10 and CelebA, and one-layer MLP for Adult. Following the work (Collins et al., 2021), for all the methods we sample 10% of the clients in every global epoch. We train the models for 200 global epochs on FEMNIST, CIFAR10 10 and CelebA, 50 on Adult. And we train 15 local epochs for FEMNIST, CIFARIO and Adult in every global epoch, 25 for CelebA. All models trained with stochastic gradient descent. We use grid search to find the optimal hyperparameters a and 9 in the validation set of each dataset. We set a = 0.5, 3 = 0.5 for CIFAR10, 10, CelebA, Adult; and
1, 3 0.5 for FEMNIST and elCU. Besides, we test different
16

---

Under review as a conference paper at ICLR 2023
Table 12: Experimental results eICU
Methods
AUC (%)
local
73.7
FedAvg
73.2+.5
FedProx
78.2+.2
Fed-MTL
77.2+1..
PerFedAvg
73.8+.3
LG-FedAvg
74.5
FedPer
74.31.7
FedRep
71.1
APFL
68.3+.8
L2GD
72.0+.6
Ditto
78.3+.1
ours
78.0
Table 13: Experimental results on eICU with adaptive
C
AUC (%)
1
69.0..4
3
75.4+.2
5
75.5+.7
7
78.0+.1
9
77.1+.9
11
76.8+.2
13
77.01.3
manifold dimensions d' for each benchmark dataset. We keep as small as possible while ensuring reconstruction quality the validation set. We d' 256 for CIFAR10 and CelebA, d' = 12 for FEMNIST, d' 32 for Adult and eICU. For synthetic experiment, the data dimension = 3 and manifold dimension d' since element of data identically equals to O. The source codes made publically available at https: : / thub. com coom/pcfl.//mm http//cfl/pcclll../
E.4 DATASETS
In our experiments, CIFAR10, FEMNIST, CelebA and Adult are all public dataset. For the synthetic experiment, the data point = (xc, 0, y has three elements. We add zero element data so that the manifold dimension smaller than the data dimension, which simulates the situation in real-world datasets. We create the federated environment with data heterogeneity for CIFARIO and FEMNSIT by randomly allocating several classes to each client following the work (McMahan et al., 2017). For the dataset eICU, we follow the procedure the website https: : e ciccu-rdd mi t edu and got the approval for the dataset. We follow the data preprocessing as in Sheikhalishahi et al. (2019) and randomly select 14 hospitals introduced in the main text.
E.5 COMPUTING RESOURCES
Part of the experiments is conducted on a local server with Ubuntu 16.04 system. It has two physical
CPU chips which are Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz with 32 logical kernels. The other experiments are conducted on remote server. It has 8 GPUs which are GeForce RTX 2080 Ti.
17


Reviews:
[{"review_id": "ifJpuF995F", "reviewer": "Reviewer_kbbP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The papers approaches federated learning from heterogeneous data from a manifold learning perspective. Each local datasets is obtained by sampling from a local manifold. The intersections or overlaps of these local manifolds is used to measure the similarity between local datasets. ", "summary_of_the_review": "see above. ", "strength_and_weaknesses": "Strength: \nThe manifold learning perspective to FL from heterogeneous data is novel to the best of my knowledge. Authors do a good job in motivating and explaining this perspective on a high level. \n\nWeaknesses: \n* The novelty and relevance of the proposed approach needs more motivation and explanation. \n* More theoretical analysis of the proposed method (generalization bounds, computational complexity) is required. \n* The numerical experiments should include comparison with clustered FL and graph-based FL methods such as network Lasso. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ld9RRzHEvQ", "reviewer": "Reviewer_XA4j", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of the heterogeneity of local data distributions in federated learning. The authors argued that the distribution overlaps are not consistent but scattered in local clients. They proposed to infer the local data manifolds to learn from the informative overlaps and estimate the data density to generate samples from the local manifold. Experimental results verified the effectiveness of the proposed method.", "summary_of_the_review": "The paper raises relatively novel research questions. However, the proposed method has some shortcomings in motivations and privacy protection. In addition, the authors seem to have exposed personal information.\n===============\nI checked the author's rebuttals, some of them are useful and clarifying my concerns. I am choose to improve my scorings.", "strength_and_weaknesses": "Strengths:\n1)\tThis paper studies a more practical and general scenario where the data distribution overlaps across different clients could be fragmented, that is, the informative and ambiguous data shards exist simultaneously in other clients.\n2)\tThe authors proposed to utilize data manifolds to identify the meaningful overlaps and exclude ambiguous information from the clients.\nWeaknesses:\n1)\tIn section 4.3, the authors used the normalizing flow to infer the local data manifolds. However, there is no motivation to separate the latent space into two parts. What is the point of doing so?\n2)\tThe authors utilized two bijective transformation including g_{\\theta} and h_{\\Phi} to transform the original data manifold to the lower projected data manifold. However, the authors did not explain how to learn the parameters of the two transformation in the method and experiment section.\n3)\tIn Equation (16), the authors required client i to collaborate with certain clients who have a higher client similarity. But it is confusing to minimize the loss of the model on other clients’ datasets.\n4)\tIn Section 4.4, the final objective in Equation (14) uses both the data of client i and part of the data of other clients to train the local model. However, this private data should be kept locally on each client. So how did the authors train models with the data at the same time? \n5)\tThe authors should experiment with the sensitivity of the parameters of the proposed method, such as the threshold \\epsilon, which has a very large impact on efficiency.\n6)\tDoes the code link in the paper expose the author information? (PhD student of Tsinghua University in China, Department of Automation).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7iHPc8UBX2S", "reviewer": "Reviewer_psQN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use normalizing flow (NF), an invertible generative model, to improve performance of personalized FL under a fragmented sharing setting, where local data distributions partially overlap. The overall goal is to identify clients with shared data and utilize information from those clients to boost performance. Specifically, a global data manifold is learned by the generative model. The learned data manifold is then used to construct local data manifolds, which are split into shared and unique components. For the shared component, the algorithm collects invertible representations of the shared data from the other clients and invert them back to the image space using NF. For the unique component, the algorithm samples additional generated data from the learned local data manifold. ", "summary_of_the_review": "The paper presents a novel method for personalized FL. However, the reviewer has severe concerns over privacy issues raised in the method. Specifically, invertible hidden representations of local data are passed from the clients to the server and shared among clients. If the author could comment on this issue, I will be happy to raise my score. ", "strength_and_weaknesses": "Pros:\n\n1, The paper cleverly uses the properties of normalizing flow, e.g., exact likelihood and invertibility, to find data overlaps and sample additional training data. \n\n2, The proposed algorithm demonstrates performance  on multiple benchmarks. \n\n\nCons:\n\n**1, Privacy concern on sharing invertible representations.** Even though the paper claims to preserve privacy by not sharing clients’ data directly, invertible hidden representations of local data are shared from the clients to the server and among clients. The performance improvement might have come from the high-fidelity reconstruction of the original image from the hidden representations using the NL model.  This could defeat the purpose of privacy preservation, especially when a powerful invertible generative model is involved [1]. \n\n**2, Contribution of each of the components not clear.** Two types of generative data are used on each client: inverted data from other clients and sampled data from the learned local manifold. Only the first one is directly related to collaboration among clients. It is not clear how much improvement is from collaboration and how much is from the generated data. It is also not clear how many generated data from the local manifold is used, which could be an important factor to the improvement. An ablation study would resolve this concern.  \n\n\nMinor: \n\n**3, Clarity on the use of the similarity metric.** It is not clear how eq.16 is used in the final algorithm and how the hyperparameter $\\epsilon$ is set. \n\n[1] Hitaj, Briland, Giuseppe Ateniese, and Fernando Perez-Cruz. \"Deep models under the GAN: information leakage from collaborative deep learning.\" Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. 2017.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1HlYa9lXHhd", "reviewer": "Reviewer_te42", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper tackles data heterogeneity of local data distributions in personalized federated learning. Unlike existing works assuming consistent information sharing, this paper introduces fragmented information sharing where the distribution overlaps are not consistent in local clients, which is a more realistic assumption.\n- This paper proposes two losses for effective local training: prevents overfitting to local biased data by 1) identifying meaningful overlaps between clients, and 2) generating data from the local manifold with an optimal sampling density.\n- Extensive experiments demonstrate the proposed method outperforms existing methods on multiple image classification benchmarks and a real-world clinical data set.", "summary_of_the_review": "At this point, this paper, in my opinion, is worthy of acceptance.", "strength_and_weaknesses": "### Strong points\n\n- This paper tackles a challenging but more realistic federated learning scenario where the shared information may not be consistent across all clients.\n- By adopting Normalizing flow based on the bijective model, communicating information about data distributions as a form of data manifold is interesting. Identifying overlaps between the local manifolds and sampling pseudo data with exact likelihood estimation are straightforward approaches to prevent each client from the biased local minima due to the skewed local data.\n- The toy experiment demonstrates the effectiveness of the methods intuitively.\n- The performance gain is non-trivial on multiple benchmarks.\n\n### Weak points\n\n- The performance gain may be attributed to scaling up the loss by introducing additional cross-entropy loss with the subset of local data. It is necessary to compare the results of the component analysis with each optimal learning rate.\n- The paper should evaluate the proposed method on more heterogeneous settings: a large number of clients (lower data points for each client), lower class overlaps, and heterogeneous data split sampled by Dirichlet distribution.\n\n### Questions\n\n- How does the server calculate the overlaps of $U^{\\prime i}$ between clients, calculate $U^{\\prime i}_s$ while the server does not have any raw data of clients?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject