Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper tackles data heterogeneity of local data distributions in personalized federated learning. Unlike existing works assuming consistent information sharing, this paper introduces fragmented information sharing where the distribution overlaps are not consistent in local clients, which is a more realistic assumption.
- This paper proposes two losses for effective local training: prevents overfitting to local biased data by 1) identifying meaningful overlaps between clients, and 2) generating data from the local manifold with an optimal sampling density.
- Extensive experiments demonstrate the proposed method outperforms existing methods on multiple image classification benchmarks and a real-world clinical data set.
strength_and_weaknesses: ### Strong points

- This paper tackles a challenging but more realistic federated learning scenario where the shared information may not be consistent across all clients.
- By adopting Normalizing flow based on the bijective model, communicating information about data distributions as a form of data manifold is interesting. Identifying overlaps between the local manifolds and sampling pseudo data with exact likelihood estimation are straightforward approaches to prevent each client from the biased local minima due to the skewed local data.
- The toy experiment demonstrates the effectiveness of the methods intuitively.
- The performance gain is non-trivial on multiple benchmarks.

### Weak points

- The performance gain may be attributed to scaling up the loss by introducing additional cross-entropy loss with the subset of local data. It is necessary to compare the results of the component analysis with each optimal learning rate.
- The paper should evaluate the proposed method on more heterogeneous settings: a large number of clients (lower data points for each client), lower class overlaps, and heterogeneous data split sampled by Dirichlet distribution.

### Questions

- How does the server calculate the overlaps of $U^{\prime i}$ between clients, calculate $U^{\prime i}_s$ while the server does not have any raw data of clients?
clarity,_quality,_novelty_and_reproducibility: - Clarity and quality: Writing is clear and easy to follow
- Novelty: The proposed problem setting is realistic and the proposed methods are novel
- Reproducibility: The authors submitted the code to reproduce the result.
summary_of_the_review: At this point, this paper, in my opinion, is worthy of acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use normalizing flow (NF), an invertible generative model, to improve performance of personalized FL under a fragmented sharing setting, where local data distributions partially overlap. The overall goal is to identify clients with shared data and utilize information from those clients to boost performance. Specifically, a global data manifold is learned by the generative model. The learned data manifold is then used to construct local data manifolds, which are split into shared and unique components. For the shared component, the algorithm collects invertible representations of the shared data from the other clients and invert them back to the image space using NF. For the unique component, the algorithm samples additional generated data from the learned local data manifold. 
strength_and_weaknesses: Pros:

1, The paper cleverly uses the properties of normalizing flow, e.g., exact likelihood and invertibility, to find data overlaps and sample additional training data. 

2, The proposed algorithm demonstrates performance  on multiple benchmarks. 


Cons:

**1, Privacy concern on sharing invertible representations.** Even though the paper claims to preserve privacy by not sharing clients’ data directly, invertible hidden representations of local data are shared from the clients to the server and among clients. The performance improvement might have come from the high-fidelity reconstruction of the original image from the hidden representations using the NL model.  This could defeat the purpose of privacy preservation, especially when a powerful invertible generative model is involved [1]. 

**2, Contribution of each of the components not clear.** Two types of generative data are used on each client: inverted data from other clients and sampled data from the learned local manifold. Only the first one is directly related to collaboration among clients. It is not clear how much improvement is from collaboration and how much is from the generated data. It is also not clear how many generated data from the local manifold is used, which could be an important factor to the improvement. An ablation study would resolve this concern.  


Minor: 

**3, Clarity on the use of the similarity metric.** It is not clear how eq.16 is used in the final algorithm and how the hyperparameter $\epsilon$ is set. 

[1] Hitaj, Briland, Giuseppe Ateniese, and Fernando Perez-Cruz. "Deep models under the GAN: information leakage from collaborative deep learning." Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. 2017.
clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear. However, the role of certain equation and values of hyperparamters are not immediately obvious from the main paper. 
summary_of_the_review: The paper presents a novel method for personalized FL. However, the reviewer has severe concerns over privacy issues raised in the method. Specifically, invertible hidden representations of local data are passed from the clients to the server and shared among clients. If the author could comment on this issue, I will be happy to raise my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The papers approaches federated learning from heterogeneous data from a manifold learning perspective. Each local datasets is obtained by sampling from a local manifold. The intersections or overlaps of these local manifolds is used to measure the similarity between local datasets. 
strength_and_weaknesses: Strength: 
The manifold learning perspective to FL from heterogeneous data is novel to the best of my knowledge. Authors do a good job in motivating and explaining this perspective on a high level. 

Weaknesses: 
* The novelty and relevance of the proposed approach needs more motivation and explanation. 
* More theoretical analysis of the proposed method (generalization bounds, computational complexity) is required. 
* The numerical experiments should include comparison with clustered FL and graph-based FL methods such as network Lasso. 



clarity,_quality,_novelty_and_reproducibility: Clarity and Reproducability: 

* The main contributions listed in Section 1 are too vague. Pls point out the theoretical 
result (Prop., Them.) or methodological result (Algorithm) that you consider the main contribution. 
In my opinion, the main contribution of the paper is the application of basic manifold learning 
techniques to obtain a measure for the similarity between two local dataset. If this is the case, 
then authors should explain what the pro and con of this new measure is compared e..g, to KL divergence 
of Wasserstein distance. 

* The numerical experiments needs to be explained in more detail. At least Section 5.1. and 5.2. 
lack many details on how the proposed PCFL method is applied. What are the hypothesis spaces used 
to fit the local datasets? What is the dimension d' of the latent space used for the manifold learning step? 
Also, how did you precisely compute the accuracies listed in Tables 1 -3 ? are these computed on a val. or 
test set ? How did you tune the hyper-parameters of the baseline methods ? 


Novelty: 

* The novelty of the approach via manifold learning should be motivated better. What is the specific 
challenge in using manifold learning as a proxy for distribution learning required to find out which 
local datasets to pool ? What is the fundamental new idea that sets your approach apart from existing 
work on clustered FL such as Ghosh et al. (2020)? I do not understand what you mean by "However, their
hypothesis excludes the possibility of knowledge transfer across clusters." 

* Authors should also compare their approach to graph-based methods for federated learning. 
These methods use a similarity graph, representing pair-wise similarities of distributions 
underlying local datasets, to construct regularisation terms. A popular choice for this 
regularization term are measures of total variation as used by network Lasso 

Network Lasso: Clustering and Optimization in Large Graphs
D. Hallac, J. Leskovec, and S. Boyd, Proceedings SIGKDD, pages 387-396, 2015.

These total variation based methods for FL provide a smooth trade-off between learning tailored local 
models (that might overfit if local data is too small) and a common global model (that might under-fit 
for heterogeneous local datasets). This trade-off has been studied in a recent line of work: 

A. Jung, "Networked Exponential Families for Big Data Over Networks," in IEEE Access, vol. 8, pp. 202897-202909, 2020, doi: 10.1109/ACCESS.2020.3033817.

A. Jung and N. Tran, "Localized Linear Regression in Networked Data," in IEEE Signal Processing Letters, vol. 26, no. 7, pp. 1090-1094, July 2019, doi: 10.1109/LSP.2019.2918933.

The two latter works study the relation between local models (their loss function), similarity 
structure between local datasets and the resulting cluster structure. This analysis might be combined 
with your similarity measure to obtain theoretical performance bounds for your proposed method. Moreover, 
it would be interesting to compare these similarity graph based methods with your approach. It seems 
that you can use your similarity measure to obtain the edges (and their weights) of the similarity/empirical 
graph used by these methods. 

Quality:  

* pls explain the notation used for the expectations in (14)

* as to the first contribution: what do you mean precisely by "..shared knowledge is fragmented among local clients;" ?

* as to the second contribution: how do you show that your method achieves a more precise collaboration ? 

* as to the third contribution: "Because of the adaptability of our proposed framework.." pls explain in more detail 
how your method offers the claimed adaptability. 


* what is a “federated network” ?

* what precisely is an "overlap"  ? is it a subset/region of the sample space?

* "While it is hard to learn the local manifold from the insufficient data in local clients directly," 
why and in what precise sense is the local client data insufficient ? 

* i do not understand the meaning of Eq. (2) 

* "While the data manifold of local clients is mostly agnostic..." what is an agnostic data manifold ? 

* "We suggest leveraging the overlaps via the learned data manifold to prevent privacy leakage." Do you study the privacy leakage of your method in more detail ? 

* it is unclear how PCFL is combined with "baseline" methods to obtain the results depicted in Figure 7, 8 

* "However, i.i.d. assumption in Eq.(2) is largely violated as the local data distributions may be
significantly distinctive." pls justify this claim with citations or describing a specific application domain where 
i.i.d. assumption is violated. 

* “We intuitively show the motivation of our method by conducting experiments on synthetic data. “ 
Im not sure if numerical experiments are effective for building intuition about some method. 

* "..needs to precisely identify the data overlap sampled from.." this is unclear to me. 
summary_of_the_review: see above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of the heterogeneity of local data distributions in federated learning. The authors argued that the distribution overlaps are not consistent but scattered in local clients. They proposed to infer the local data manifolds to learn from the informative overlaps and estimate the data density to generate samples from the local manifold. Experimental results verified the effectiveness of the proposed method.
strength_and_weaknesses: Strengths:
1)	This paper studies a more practical and general scenario where the data distribution overlaps across different clients could be fragmented, that is, the informative and ambiguous data shards exist simultaneously in other clients.
2)	The authors proposed to utilize data manifolds to identify the meaningful overlaps and exclude ambiguous information from the clients.
Weaknesses:
1)	In section 4.3, the authors used the normalizing flow to infer the local data manifolds. However, there is no motivation to separate the latent space into two parts. What is the point of doing so?
2)	The authors utilized two bijective transformation including g_{\theta} and h_{\Phi} to transform the original data manifold to the lower projected data manifold. However, the authors did not explain how to learn the parameters of the two transformation in the method and experiment section.
3)	In Equation (16), the authors required client i to collaborate with certain clients who have a higher client similarity. But it is confusing to minimize the loss of the model on other clients’ datasets.
4)	In Section 4.4, the final objective in Equation (14) uses both the data of client i and part of the data of other clients to train the local model. However, this private data should be kept locally on each client. So how did the authors train models with the data at the same time? 
5)	The authors should experiment with the sensitivity of the parameters of the proposed method, such as the threshold \epsilon, which has a very large impact on efficiency.
6)	Does the code link in the paper expose the author information? (PhD student of Tsinghua University in China, Department of Automation).

clarity,_quality,_novelty_and_reproducibility: The paper is presented relatively clearly, but the motivation of some parts of the method is unclear. The problem studied in the paper is novel and practical. The method section is missing some descriptions on how to train the model parameters, which harms the reproducibility.
summary_of_the_review: The paper raises relatively novel research questions. However, the proposed method has some shortcomings in motivations and privacy protection. In addition, the authors seem to have exposed personal information.
===============
I checked the author's rebuttals, some of them are useful and clarifying my concerns. I am choose to improve my scorings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission proposes a method for personalized federated learning based on manifold estimation.  The claim is that it will better allow clients to make use of overlap between distributions on different clients while achieving privacy preservation by a data abstraction.  A majority of reviewers felt that this submission was below the acceptance threshold.  Main concerns included a lack of theoretical guarantees, both from a statistical generalization perspective and from a privacy perspective.  During the rebuttal process, the reviewers were asked about the main contributions, and the assertion of improved privacy was again raised.  However, privacy should mean privacy by design with provable guarantees, rather than intuitive assertions.  An additional issue was that answers regarding key properties of manifolds were not clear in the discussion, with the authors frequently using the word "agnostic" to describe a manifold in the rebuttal process (while it is used once in the paper).  It is not clear what "mostly agnostic" means, and the responses often lacked formal rigor as a result.  On the balance, the algorithm and empirical results are interesting, but there is a mismatch between the claims of the paper and the theoretical (and to a lesser extent empirical) evidence for them.  Additional theory and experiments backing privacy claims should be given, or these claims should be removed.
justification_for_why_not_higher_score: There is a mismatch between one of the main claims in the paper, and the theoretical and empirical evidence for it.

As a separate note, the last reviewer raised a concern of anonymity violation, which has not yet been assessed.
Decision: Reject