Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a federated learning algorithm called FedProf, by using a training rule that updates the model based on divergences between data representation. 
main_review: Strenth:

I personally like the style of this work, which attempts to formulate and present results on FL in a more theoretical way.

Weakness:

Overall the presentation on theoretical results can be improved, and the connection of some statements to the main results is not clear.

1. For example in Proposition 1 and 2, all non-linear components in the neural network seem to be bypassed and the statements are simply equivalent ways of stating CLT. If it is the case, this needs to be clearly mentioned before those propositions. Besides, it is also not clear to me that how this over-simplified model is related to the rest of the works.

2. The main Theorem needs to be better stated and explained. For example, the most relevant question to the readers is how the construction of profiles affects the performance of the algorithm.   

3. To better understand the main Theorem, it would help if the stated learning bound can be compared to some benchmarks, e.g., the performances of using fixed \lambda.

4. Minor: the last equations in all Proposition proofs are stated in terms of convergence in distribution, which may not hold.

 
summary_of_the_review: Overall, the results in this paper can be better presented, and the connections between certain statements are not clear. Therefore, I recommend a rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the federated learning scenarios where local data is biased, noisy, or even irreverent, which could significantly degenerate the global performance, and the authors propose FedProf, which utilizes data representation profiling and matching to mitigate the impact of low-quality clients during training.

main_review: This paper is well-written and easy to understand. The motivation is clear and intuitive. The proposed method is technically sound and the results are strong.

I have a few comments, particularly for the evaluation part.


- More Datasets: it seems to work well on the reported datasets. I wonder if the tendency is still consistent with more benchmark datasets. I would like to suggest common datasets such as CIFAR-10 and CIFAR-100. I also would like to see the results when the number of classes is increased.

- Standard Deviation: Can you provide statistical information in Table 2 and 3? I would like to see the variance of the proposed model. Few performances seem marginal compared to the base models. 


- The Current Baseline Models: I found that the baseline models used in the paper are slightly outdated. I suggest [1], the state-of-the-art method tackling similar scenarios, via performing model-level contrastive learning.


- Overview: Please consider adding the concept illustration for FedProf, which can be helpful for common readers to capture high-level ideas. 

[1] Model-Contrastive Federated Learning, CVPR'21




summary_of_the_review: Overall, I enjoyed reading the paper, and I'll raise my score if my concerns are properly addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes FedProf, a new client sampling scheme that speeds up the convergence of FedAvg type algorithms. The author proves convergence of FedProf under a set of simplifying conditions and they demonstrate the utility of FedProf via empirical studies.
main_review: Strengths: The proposed method is intuitive and simple to implement. It also comes with theoretical guarantees (though the assumptions are highly idealized). 

Weaknesses and some questions:
- If I understand correctly (pls correct me if I'm wrong), Proposition 1 and 2 are consequences of Lyapunov CLT, thus one needs each coordinate of the representation vector to be independent. It is not clear to me if the independence assumption would hold in practice.
- Following my previous point, I am curious about what will happen if we replace the product Gaussian (Eq. 2) by a Gaussian with some non-identity covariance matrix.
- Is the choice of KL divergence (Eq. 4) essential for the performance? What will happen if we choose other metrics?
- How sensitive is the performance with respect to the choice of $\alpha$?
summary_of_the_review: I appreciate the simplicity of the idea and the performance boost it brings about. From what I understand, it can be applied to any FL algorithms that involve client sampling. Therefore, I think this paper deserves 6: marginally above the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a user selection algorithm for federated learning (FL). The key motivation is to select high quality clients for update and thus to reduce the impact of low quality data on FL training. A hidden hypothesis is that high quality data has similar representations while noisy and low-quality data has different distributions of representations. Based on this hypothesis, the key idea is to select users based on their representation layer distribution difference from the global model, the higher the difference, the lower the chance of the client being selected.  Furthermore, the authors observed and proved that representation layers follow a Gaussian distribution, which makes the representation difference learning more efficient. The authors evaluated the proposed algorithm in comparison to existing algorithms using a small-scale sensor dataset and a large-scale EMNIST dataset. The proposed algorithm performs well in the evaluation.
main_review: The key idea of the paper is to select “high-quality” data under the hypothesis noisy and low-quality data has different distributions of representations. This hypothesis needs further justification. First, FL is known for dealing with non-iid heterogeneous datasets. Consider a scenario where clients have high-quality but heterogeneous datasets. How general does this hypothesis hold? In some sense, the idea here is opposite to another idea that has been used for client selection where clients with high loss are more likely to be selected. 

Also, I wonder if the algorithm could get stuck in a local suboptimal condition. Specifically, let’s say there are two clusters of clients with different focuses. By chance, initial training selected users from one cluster, then the global model will further select users from the same cluster. Could this happen?

The current evaluation is limited to two simple and homogeneous tasks with added noise, which provides an ideal situation where the proposed algorithm would shine. While these evaluations illustrate the effectiveness of the proposed idea under ideal conditions, more thorough evaluations under general heterogeneous conditions are needed to better understand the pros and cons of the proposed algorithm. 

Also, most of the comparison algorithms were proposed in 2019 except one that focuses on communications. Given the fast development of FL, I wonder if there are more recent algorithms that should be compared with. 

The authors made an observation that representation layers follow a Gaussian distribution. I understand that this makes it more efficient to calculate distributional difference. However, in principle, this can be applied to cases where Gaussian distribution does not hold, right? Can you elaborate more on this, both in terms of using Gaussian distribution to approximate a non-Gaussian one and using the true non-Gaussian distribution?  

In the proofs of Prop. 1 and Prop. 2, central limit theorem is applied and thus requires the Lyapunov’s condition (Def. 1). Could you comment on how restrictive this condition is? For example, in your experiments, does this assumption hold?

One clarification question, in the experiments, how did you select the representation layer? 



summary_of_the_review: In summary, the authors propose to more favorably select FL clients whose representation distribution is more similar to that of the global model. While the current evaluation results are encouraging, data heterogeneity in FL and model robustness need to be carefully considered.
review_title: null

Meta-review: This paper proposes a federated learning method called FedProf that adaptively selects different subsets of the clients' data in training the global model. There were several concerns brought up in the reviews and discussion. The multivariate Gaussian (with identity covariance) assumption on the neural network representation is limited. The paper also claimed to provide privacy preservation, but there is no formal statement of the actual privacy guarantees. (The fact that it's running federated learning does not guarantee privacy protection.) The presentation could use improvement. The reviewers had issues trying to understand the main theorem. Overall, there is not sufficient support for acceptance.
Decision: Reject