Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Federated Robust Decoupling (FED-ROD) to excel on both generic FL and personalized FL. With extensive results, it illustrates that strong personalized models emerge from the local training step of generic FL algorithms, and show that class-balanced objectives are effective for improving the generic FL performance when clients have different class distributions. Lastly, the paper demonstrates that FED-ROD enables zero-shot adaptation and much effective fine-tuning for new clients.
main_review: The paper proposes Federated Robust Decoupling (FED-ROD) to excel on both generic FL and personalized FL. With extensive results, it illustrates that strong personalized models emerge from the local training step of generic FL algorithms, and show that class-balanced objectives are effective for improving the generic FL performance when clients have different class distributions. Lastly, the paper demonstrates that FED-ROD enables zero-shot adaptation and much effective fine-tuning for new clients.

weakness: 
1. The paper overall is a bit hard to follow. It would help to possibly re-structure tables, figures with the main body.  
2. In the experiment part, it would be stronger to expand and provide summary and insights from tables and figures in the body. For example in "Ablation Analysis",  
2. conclusion is weak and would requires some work to summarize the contribution. 
summary_of_the_review: Technical content of the paper is overall pretty strong and idea seems pretty significant and novel. However, overall the paper is hard to follow. It would benefit and stronger with better organization (as pointed out in the weakness in the above).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on a challenging problem in federated learning setting where data across different local workers are imbalanced, and most of the current work in the community are optimizing either generic performance or personalized performance. This paper proposes a new framework, Fed-RoD, which combines both global objective and local objective, to achieve that the trained model is suitable for both generic and personalized task. The main contributions are:
1. Focuses on an interesting problem which aims to increase both local and global performance in imbalanced federated learning setting;
2. Designed a two-loss, two-predictor framework to achieve the goal;
3. Based on the extensive experiments, the performance achieves state-of-the-art on both sides.
main_review: Strengths:
1. The problem is clearly stated and the algorithm is well-organised, also easy to understand and follow;
2. The problem itself is interesting, non-iid and imbalance is a key challenge in federated learning and real-world setting. Also, increase both local and global performances should be of interest to the community;
3. The experiments are well designed and comprehensive;
4. The experimental results provided in the paper do achieve a state-of-the-art on both generic and personalized setting;
5. Related works are properly cited.
 
Weaknesses:
1. Since no code is provided, it is useful to include more training details, for example, training hyperparameters for readers to better utilize the proposed framework;
2. Also for the experiments, how many repeat trials did you run for a experiment? It is important to also include the error bar in the results;
3. Lack of theoretical guarantees on the proposed framework.
summary_of_the_review: This paper proposes a new framework aiming to increase generic and personalized performance simultaneously in the imbalanced federated learning setting, the proposed algorithm is simple yet efficient according to the experimental results. Theoretical analysis are somehow insufficient, and some minor fix or explanations are needed to better understand the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: FL trains models both on the server and clients. Such model training becomes problematic when the data distributions on each of the clients diverge from each other. The paper answers the question of prioritizing the generic model at the server or the personalized model on each client. The paper proposes to decouple a model’s dual duties with two prediction tasks. 
main_review: ### Summary:

FL trains models both on the server and clients. Such model training becomes problematic when the data distributions on each of the clients diverge from each other. The paper answers the question of prioritizing the generic model at the server or the personalized model on each client. The paper proposes to decouple a model’s dual duties with two prediction tasks. 

### Pros:

- The paper is well-written, easy to understand and follow along.

- The paper is well motivated and driven by a real-problem in FL. 

- Considering model training on each of the clients as a class-imbalance problem and applying minimizations on the objectives considering the two losses in a multi-task setup is also interesting.

### Cons:
Following are the concerns that can be addressed for change in the final scores.

1. “the literature does not evaluate G-FL algorithms in the P-FL setup by their local models” What does this mean? The evaluations on global test sets or the local test sets? In either case, this sentence does not really make a lot of sense.

2. “taking average over model weights -- indeed acts like a regularizer” is also not a surprising finding, this indeed is general common sense to think that averaging is kind of a regularizer.

3. In fact look at the following paper about bagging (which of course is not an FL algorithm but a simple general ensemble technique of averaging multiple models) that suggests to have regularization effects due to model averaging.
    - Poggio, T., Rifkin, R., Mukherjee, S. and Rakhlin, A., 2002. Bagging regularizes. MASSACHUSETTS INST OF TECH CAMBRIDGE ARTIFICIAL INTELLIGENCE LAB.
    - Skurichina, M. and Duin, R.P., 1998. Bagging for linear classifiers. Pattern Recognition, 31(7), pp.909-930.

4. If one performs the P-FL evaluations on local test sets, at the beginning of each round with the newly communicated global model that might perform better on local test sets as opposed to weight for the local model to be fine-tuned and perform testing, not really surprising, even without regularization term because of the implicit reg effect due to averaging.

5. In P-FL, the local test sets used in the experiment are static because of the standard benchmarks used. However, in real-time, these test sets will change and there will be temporal effects that need to be considered. In such cases, building test sets in itself becomes a challenging problem. The whole paper starts with the motivation of closing the gap between G-FL vs P-FL, the gap arises due to the data shifts, for which the corresponding test sets need to be prepared. This issue is not addressed at all.

6. y_P = h^G+h^P, which is uniform weight for the P-FL part, how about you add weights and make them learable, something like this y_P = (1-\alpha)h^G + \alpha h^P and start with bigger \alpha as this function optimizes for the P-FL anyway, why not try something along these lines and see how that improves P-FL and effects the G-FL. Any of the findings in this direction are also useful since you already proposed this and have the setup.


### Post Rebuttal
Satisfied with the authors response, increasing the score to a clear accept.

summary_of_the_review: The idea of treating the non-IIDness on clients in FL as class imbalance is interesting, however, the paper can benefit from addressing the above raised concerns.
review_title: null

Meta-review: This manuscript proposes and analyses can approach to address the centralized and personalized tasks in federated learning jointly. Existing work has tackled this issue by developing separate tasks. Instead, this manuscript proposes a shared architecture that aims to optimize centralized and personalized models. One observation motivating this work is that local models trained during federated learning effectively optimize local task performance. The resulting approach results well when label shifts primarily drive the client variability. Here, the centralized components are trained to optimize a balanced risk, while the local components are trained to optimize the standard empirical risk. 

Reviewers agree that the manuscript is well-written and appropriately addresses the timely issue posed. The main concerns are the clarity of the technical contributions and technical statements during the review. The authors respond to these concerns and have satisfied the reviewers. After discussion, most reviewers are generally strongly positive about the strength of the manuscript contributions.
Decision: Accept (Spotlight)