Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper focuses on a federated learning setting where there is a form of potentially high bias on the client-side (I use the work bias in the basic sense (i.e., favored towards to) instead of the pure statistical sense). These biases are exemplified by the datasets used by the paper, which come in the form of unbalanced labeling on client devices, and differences in the specification of client devices (i.e., high-end phones vs low-end phones). For these biases, the paper proposes to mix the datasets between the server (or datacenter) and the clients. Specifically, the paper uses two datasets:

* CelebA: the dataset is partitioned into people who are "smiling" and who are "not smiling." The claim is that the majority of pictures on people's phone are smiling. Here, the server contains the non-smiling pictures and the clients contain smiling pictures and the task is to predict smiling or not smiling.

* StackOverflow and Wikipedia: the StackOverflow dataset is used for clients, and the Wikipedia dataset is used for the server.

The paper also proposes a mixed loss function, which is the sum of the loss on server data and the loss on client data.


strength_and_weaknesses: In Section 2 Algorithms, it would be more helpful to explain the notation used and the various components in the two algorithms in this section rather than relegating details to the appendix. Additionally, it is not clear to me what the components ClientOptimizer, ServerOptimizer, CentralOptimizer, and MergerOptimizer in this context are, what each performs (they appear to be blackbox functions), how they relate to equation (1), and how they connect to Section 5.

The rationale on the StackOverflow and Wikipedia (and the allocation of 25% server, 75% client) dataset as it pertains to client devices is not thoroughly explained in the experimental section.

Section 4.3 as it pertains to "mixing different loss" and to the focus of the paper is not clear to me.
clarity,_quality,_novelty_and_reproducibility: Clarity and quality require improvement. The novelty is low.
summary_of_the_review: The proposed idea and presentation require more work.

Update: Thank you to the authors for their reply. I will maintain my review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To mitigate distribution shift and improve FL performance in inference scenarios, the paper proposes the Mixed FL, which adds centralized data to the server and co-trains a powerful and robust global model between clients and the server. And the 1/2-way gradient transfer could work to alleviate the influence caused by non-iid data. Thus, the model could perform better facing unfamiliar datasets with different distributions compared with clients' local datasets. Many experiments have been conducted to validate their algorithms in different domain tasks. Related theoretical explanations are provided for the convergence. 

strength_and_weaknesses: Strengths:
1. The general algorithm of Mixed FL is well constructed and seems valid.
2. The relevant experiments are relatively sufficient and able to reflect the effectiveness of proposed algorithm.

Weakness:
1.In 2-WAY GRADIENT TRANSFER, the client's gradient information will be passed to the server. However, issues related to data privacy during gradient tranmission do not appear to be explored in the paper.
2. Some technique behind the algorithm may not be that novel, such as computation offloading and gradient augmentation.

clarity,_quality,_novelty_and_reproducibility: The paper is readable overall. However, the logic and presentation are not coherent in some places. For example, in Introdution section, the author mentions "Mixed loss f might be a more useful training objective for many reasons, including..." and then introduces two aspects including alleviating distribution skew and reducing communication and computation. From their statement, I cannot see that reducing communication and computation is directly related to this mixed loss. Although placing part of the computation of the embedding table on the server does reduce client-side communication and computation, the presentation and organization of this part may need to be reconsidered to highlight its relevance with the mixed loss. And the related work section is not very organized.

summary_of_the_review: Overall, this work constructs a well-established algorithm for mixed FL. But some of its ideas are not very novel. This algorithm is more like integrating some existing technologies, such as compute offloading and gradient transferring. However, the experimental part is fully carried out, which can reflect the effectiveness of the algorithm. Moreover, the related theoretical proof is also relatively solid, although there are still some limitations. Again, further revisions are still required in the organization and presentation of some paragraphs.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a joint (mixed) FL strategy that jointly trains using a combination of centralized and federated models without transferring any data across server or client and vice versa. 

strength_and_weaknesses: # Pros:

- The paper is well-written, easy to understand and follow along, however, I must admit that I did not pay an in depth attention to some of the theoretical analysis presented in section 5.

- Literature is covered adequately and the proposed paper is presented amongst the body of knowledge while it also contributes to extend the existing knowledge in the domain.

# Cons:

- Where are the references to the works in literature that showed the techniques such as using some portion of data at server to train the server model after aggregation, next another work that showed the linear combination of server and client models. Of course they fall under the category of “sequential central-then-FL” category as opposed to the joint FL in this paper.

- On page 2, “... large embedding table” is mentioned, does not make sense as to what it refers to at this point, it only makes sense after reading the contributions as one of the bullets talks about recommender systems. 

- Also, the footnote 1 is actually the real difference from the so called sequential FL coined here vs the joint FL proposed, make this clear from the abstract itself rather than undermining it in the footnote.

- The notations in Algorithm 1 and 2 are never described but left for the reader to interpret. 

- After looking at the results in Fig 1 and Fig 2, what is the point of having three variations especially all the three proposed algorithms more or less converge to the same accuracies in most of the cases.

- How movie recommendation embeddings are regularized is still unclear, please provide a formula on how this is done especially given the FL settings, even if the regularizations are there in literature. Maybe some reference to appendix also should make the concepts clear and how they play out in mixed FL.

- The way the paper presentation flows is a bit odd from the regular style of paper writing or formatting. For example, section 2 (algorithms) comes first and then section 3 (related work), then section 5 (convergence) comes after experiments, why not swap section 2 and 3 then place 5 as 4 and then experiments and so on.

- The empirical explanations in section 5.3 does not sound all that convincing, especially the trends of the Language modeling and movie recommendations are similar compared to the smile classification. The general common sense points to the fact that the celebA is custom partitioned and that probably is something to do with the observations in the paper. Did you rule that aspect out (or similar simple perspectives), before trying to prove the observations with such a complex empirical analysis?

- The FL settings such as the number of clients, the generation of non-IID distributions, optimizers user, etc are not explained.

- Also, the results show curves over a certain number of rounds that compare w.r.t the oracle. Obviously oracle has a single number, why not compare the end of training results with the oracle as well as the other state-of-the-art FL algorithms.

clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: Quality of the results are adequate however, the clarity can be greatly improved by changing the presentation

Novelty: The proposed approach is novel and simple.

Reproducibility: The presentation and the details provided are reasonable to reproduce however having a bit more clarity will improve the reproducibility. 

summary_of_the_review: Overall, the proposed approach, mixedFL sounds decent, however, the presentation can be improved further and also, the comparison with the state-of-the-art FL algorithms might help position the paper better as to how the proposed modifications benefit w.r.t the more adanced FL optimization algorithms.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose algorithms for federated learning (decentralized data) alongside the presence of a centralized dataset (termed "mixed federated learning"). The paper presents 3 algorithms based on gradient exchange between the clients and the server, benchmarks the corresponding performance on 3 tasks (simple classification, language modeling, movie recommendation) and theoretically analyzes the convergence properties. The benefits of the proposed methods are in terms of improved predictive power (due to the centralized data complementing the decentralized) and in terms of computation and communication efficiency (due to offloading intensive computations to the server side).
strength_and_weaknesses: ### Strengths

- The problem of distribution shift across federated learning clients is very important. The idea of tackling it by combining decentralized with centralized training is both practical and promising.

- The authors perform an empirical evaluation comparison based on 3 tasks with applications on federated learning.

- The authors discuss the convergence of the proposed solutions.

### Weaknesses

- The paper needs improvement in terms of clarity (see detailed review below).

- The convergence analysis assumes unbiased gradients (and thus iid distribution which is an invalid assumption on the federated learning setup). The authors also acknowledge this fact on their conclusion.

- The proposed methods require additional hyperparameters (ηc, ηm).

- The evaluation doesn't employ any baseline for the task of employing the centralized data (e.g., based on transfer learning).

- The use case of bias mitigation (section 4.2) is interesting; however there is a questionable (in terms of practicality) assumption on the existence of a non-private representative dataset for a subset of the clients.

- The paper does not assess the performance of the proposed methods in the presence of noise necessary to ensure differential privacy guarantees (given that federated learning on its own is not sufficient to provide strong privacy guarantees).



clarity,_quality,_novelty_and_reproducibility: ### Clarity 

- The paper is overall well-written (in terms of language) and easy to understand. The description of the 3 algorithms is particularly easy 
 to follow given the color separation of the differences between these 3 algorithms and the baseline (federated averaging).

- The definitions of “centraloptimizer” and “mergeoptimizer” are missing. This is crucial for understanding the main functionality of the proposed algorithms. The authors provide an example for the mergeoptimizer (averaging) but do not discuss alternatives or how they would impact the performance.

- It is not clear how gradients are initialized for 2-way. There doesn't seem to be any related information in the supplementary material as well.

- The paper presents 3 methods for mixed FL. However, one of them (2-way) appears to be superior both in terms of convergence and empirical performance. It would be helpful if the authors highlight what is the merit of presenting and evaluating all 3 methods in the main paper (e.g., illustrate properties of the best performing method). 

### Quality

- The paper is based in multiple parts on related work that is unavailable publicly (introduction, related work section, section 4.1, section 4.3). These references are redundant and authors should only cite publicly available work; otherwise any claims can be part of the paper under review.
  - The main motivation for the problem of performance degradation (communication + computation) is based on claims from such unavailable work ("Reducing Client Computation and Communication"). Therefore, there is a gap in understanding the importance of use-cases where mixed FL can improve the communication and computation performance.

- Figures 1,2 (smile) lack the FL baseline (i.e., train only on decentralized data).

- The evaluated approach for the bias mitigation (based on stack overflow + wikipedia) does not seem to reflect the motivating setup (high VS low end users).

- The predictive performance evaluation lacks baselines that also employ the centralized dataset/training (e.g., transfer learning). 

### Novelty

- The idea of assisting federated learning given the presence of a centralized (non-private) dataset has been studied before in various works as the authors also discuss in their related work section.

- The proposed methods are novel but naturally build up on ways of combining gradients from different data distributions.

### Reproducibility

- The authors provide the code necessary to reproduce the experiments. The code is organized and documented.

- The authors provide details about the experimental setup (e.g., hyperparameters) in their supplementary material.

summary_of_the_review: The paper is overall promising but requires a substantial amount of improvements mainly in terms of clarity and evaluation.
The novelty is limited but that is not an issue if the approach is clearly described and shown to be improving the state of the art. The paper would be substantially better in the presence of a convergence analysis that does not assume iid data (e.g., based on bounded dissimilarity assumptions).

The overall recommendation is a weak reject given the weaknesses of the work that are only partly addressable (given the necessary effort) with high confidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies an interesting setting for federated learning where there exists a shift of distribution from the client side to the server side. This setting is practically important and the proposed solution of a joint mixed strategy that joints trains  a more powerful global model between clients and server by adding centralized data to the server.   The authors provide  organized & documented code and the experiments appear reproducible. The main criticisms from reviewers include (1) limited novelty (2) more clarity on the writing is needed (3) the convergence analysis used overly simplified iid assumption which is not standard for FL, as typical FL analysis uses the noniid bounded difference assumption. Overall, this paper is promising and the authors are encouraged to invest some non-trivial efforts to significantly improve the paper. 

justification_for_why_not_higher_score: (1) limited novelty 
(2) more clarity on the writing is needed 
(3) the convergence analysis used overly simplified iid assumption which is not standard for FL
Decision: Reject