Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method to deal with long-tailed recognition in a federated setting. Different from previous works, they pay much attention to privacy, which is the core idea of federated learning. Since the global label distribution is inaccessible in this setting, they instead use the model updates of the clients as surrogates. In the experiments, they show the proposed method achieves comparable results, compared to methods with global priors. 
strength_and_weaknesses: Strength:

1. Overall, the paper is clear and easy to follow. 

2. They especially consider the importance of privacy in federated learning, while alleviating the problem of the long tail. 

3. The analysis of the proposed method is comprehensive and the results of the experiment are also good.

Weaknesses:

1. The setting of this paper is not clear. In "dataset and setup" in Section 4, a brief operation of how to simulate FL is introduced. "the global dataset is partitioned into 100 local datasets with the Dirichlet distribution". For each local dataset, do they draw samples from each class by following Dirichlet distribution?

2. The combination of federated learning and long-tailed datasets is good. However, in this paper, only a primitive federated method is considered. Most of the baselines focus on different reweighting methods in long-tailed recognition. I am curious about the results of directly applying the latest federate methods into long-tailed datasets, including the works discussed in the related works. 

3. Can you show some visualization examples of the data distribution of local clients?
clarity,_quality,_novelty_and_reproducibility: Most parts of the paper are clear, however, the presentation of Local Proxy Information and global Proxy Information seems not very clear. Can you give an illustrated figure for more intuitive understanding? 

My biggest concern still lies in the setting of the paper. Is the long-tailed setting in the paper reasonable? do we have other options? Happy to discuss more about this. 
summary_of_the_review: The paper considers a more strict long-tailed recognition setting by retaining the constraint of privacy in federated learning. This paper is well presented and the results are also good. However, some points are worth further discussing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the issue of heterogeneous, imbalanced label ratio among clients of a federated learning system. That is, the setting is that the global label distribution has a long tail; and each local dataset has imbalanced label ratio.

Previously, this can be addressed by re-balancing strategies which can be applied to either each client separately or to the server directly. Local re-balancing appear to perform worse than global re-balancing but on the flip side, global re-balancing requires local clients to share their label distribution. This does not preserve the privacy of data.

To combine the best of both worlds, the paper proposes a concept of local proxy information (LPI) that (1) makes use of some aggregated form of local parameter gradient to (roughly speaking) summarize statistics of the local label distribution; and (2) can be combined via weighted averaging in a fashion almost similar to the direct combination of local label distributions albeit with the exclusion of those with negative local proxy. 

The combined proxy is referred to as the global proxy information (GPI), which is core to the proposed label-distribution ensemble learning. First, cosine distances between LPI and GPI are used to partition clients into groups. Then, in each round, an expert component is created for each group: alpha x S clients will be selected for the intended group; while (1 - alpha) x S client will be selected for the rest. 

Once created, selected clients will download its corresponding expert model, bake in local data using the balanced softmax loss parameterized by the GPI and send the updated expert back. Next, to complete the communication round, the server aggregates updates for each expert; the ensemble is the average of those experts.

The proposed algorithm is tested on long-tailed versions of CIFAR-10, CIFAR-100 and ImageNet. The comparison baselines include FedAvg and many other class-prior based re-balance algorithms. 
strength_and_weaknesses: Strengths:

+ The problem is well-motivated; the presentation of the proposed algorithm is well-structured.
+ The algorithm is tested on multiple large-scale benchmark datasets for image classification (CIFAR-100, ImageNet)
+ The proposed algorithm appears novel & reportedly performs better than the baselines

Weaknesses:

- Lacking coverage of other types of FL algorithms that deal with data heterogeneities, such as personalized federated learning
- Despite the claim, the theoretical analysis in Section 3 does not shed insight on how global rebalancing is better than local rebalancing -- I will elaborate more on the next section
- This is also questionable whether one can estimate local label distribution from LPI - will elaborate in the next section
- It is also not clear which of the re-balance baseline corresponds to the analysis of Section 3; and more importantly, how these baselines are transplanted into the context of FL., i.e. are they applied locally or globally  -- do they all require sharing local label distribution etc.
clarity,_quality,_novelty_and_reproducibility: Novelty & Quality:

The proposed algorithm is novel to me. It is quite interesting (although there is no theoretical analysis) how the clients can be clustered based on their (unknown) label distribution using the LPI & GPI. 

However, I have the following reservations regarding the contribution claims (some of which were mentioned above in Weaknesses):

- Section 3 shows how a certain global rebalancing technique will induce the same result regardless of whether the setting is federated or centralized; it also shows there is a difference between the ensuing loss if the same rebalancing technique is instead applied at local sites. I agree with all those results but I do not follow how these explain for the improved performance of global rebalancing over local rebalancing: being different does not explain why one is better than the other. In my opinion, the delta term in Theorem 1 needs more intuitive interpretation. Otherwise, the current one reads local re-balance is worse than global re-balance because they are different, which (in my opinion) is not a valid explanation.

- Regarding the privacy claim: if clients can be clustered in terms of their label distribution similarity using the LPI then it is actually not safe to say the LPI cannot be used to infer the local label distribution -- privacy might not be preserved if the server has access to a small (balanced) validation set that could be used to probe the performance of each client cluster, which will reveal what are their label distributions; in addition, the definition of LPI also suggests that once the training converged, the Z and s terms in Eq. (5) will become more or less fixed and so, the server might be able to solve a linear system to assess the local label distribution -- it might not be exact depending on whether the system is under-constrained but it will still reveal extra information about those label distributions so the claim of privacy preservation needs to be revisited.

- For the same problem, personalized federated learning could be applied as a viable solution technique. The authors should discuss this; and provide empirical comparison to advocate for the practical significance of the proposed solution.

Clarity: 

- The writing is pretty good and has conveyed the key idea effectively. However, some algorithmic details still remain vague. For instance, how do the rebalance techniques used in baselines get incorporated into the FL pipeline & which one is the strategy analyzed in Section 3?

- It is also strange local clients with negative LPI were excluded from the aggregation that produces GPI -- can the authors elaborate more?

- How are the hyper-parameters such as M and alpha selected?

Reproducibility:

The algorithm is well-described so I believe its reproducibility. Is the code released somewhere? Reproducibility is strengthened if the code is also released.
summary_of_the_review: The paper presents a new idea for handling the long-tailed and imbalanced label distributions in FL. However, key concerns arise regarding (1) the lack of comparison with other types of FL algorithms that could potentially handle the same issue; (2) ambiguities over what the theoretical analysis implies; as well as potential flawed claim on privacy preservation of LPI & GPI. If these issues are addressed properly, the paper would be acceptable but for now, my preliminary assessment is that it is marginally below bar. But, I do look forward to further discussion with the authors to (hopefully) recalibrate my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies Federated Long-tailed Data, which is a highly challenging task. To address this task, this paper proposes to use a privacy-preserving proxy to guide model training and develops a label-distribution-agnostic ensemble learning framework. Both theoretical analysis and empirical verification are provided.
strength_and_weaknesses: Strengths:
1. The studied task is practical and highly challenging.
2. Exploring privacy-preserving proxy to guide model training is new to me.
3. The empirical results seem promising.

Weaknesses:
1. Introduction is good-writing, but the method section is hard to follow. 
2. Please carefully check the proof of Theorem 2. It seems not correct from Eq.12 to Eq.13.
3. please clarify why local label distributions cannot be derived from the LIP values if it is relevant to the residual.
4. It is unclear to me why the estimated GPI based on the beginning communication round can keep balanced prior for guiding model training.
5. For global proxy information, how can you know n^k, i.e., the class-level sample number in each client?
6. Please ablate the expert number M.
clarity,_quality,_novelty_and_reproducibility: Clarity: the writing of the method is hard to follow.
Quality: the quality is overall good, but there are a few issues that should be resolved.
Novelty: novelty is fair.
Reproducibility: the code is expected to release by the authors.
summary_of_the_review: I like this paper overall, but there are a few issues that need to be addressed before acceptance.

**********Post rebuttal***************
Thanks a lot for the response. Most of my concerns have been clarified. Even so, I still have some concerns about the biased prediction at the early stage. However, it is just a minor concern, so I keep my original score of 6 but increase my confidence to 5.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a Label-Distribution-Agnostic Ensemble (LDAE) for Federated learning with heterogeneous or long tailed distribution data. The proposal involves using multiple experts and their control their grouping based on a novel global objective with privacy constraints. Experiments showed significant gains. 

The paper claims about privacy and the settings are not well formed and requires additional scrutiny. Specially, the capability to do group users based on similar data distributions and other assumptions made need more thorough evaluations. Given that privacy preserving is one of the main objectives, this problem becomes critical as pointed out by one of the reviewers after significant discussions. 
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject