Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the effect of using pre-trained neural network weights on federated optimization. Several combinations of server and client optimizers are considered in combination with 2 different models for vision and language tasks. It was shown that pre-trained initialization reduces the negative effects of data and systems heterogeneity on training convergence and changes the ranking of optimization methods.
strength_and_weaknesses: ## Strengths

**S1.** Thorough and extensive experimental evaluation of different optimization aspects which may be useful to the field of federated learning.

**S2.** Several drawn conclusions are novel and can be interesting for the community.

## Weaknesses

**W1.** The concept of using pre-trained weights (in federated deep learning) is not novel on its own. Some of the ideas were explored in previous works.

**W2.** Some important experimental details are missing. Namely, it is not clear how hyper-parameter tuning was performed. What clients/data were used for tuning step sizes, etc (the ranges also seem very limited)? It would be very helpful to show how pre-trained initialization affects the performance of the methods for different combinations of parameters (e.g., like it was done with heatmaps in the Adaptive FedOpt paper).  What kind of random initialization was used in the experiments? Prior literature showed that various strategies can lead to different results in the centralized case.

Please correct me, if I missed these details.

**W3.** The thesis 
> Starting from a pre-trained model signiﬁcantly reduces the difference between having non IID vs IID data at clients.

is not well-supported (or inaccurately stated), in my view. It is not fully clear what is meant by IID data splits for naturally partitioned federated datasets in the paper. Was the data somehow reallocated between the same amount of clients or simply centralized training was performed? The latter case would require different wording in my view as it studies not the difference between non-IID and IID but rather federated and centralized cases. To properly show the effect of using the pre-trained model for a level of non IIDness one ideally needs to vary the level of heterogeneity for the same problem.

**W4.** Some of the Section 5 conclusions are questionable and doubtful. For example, the Gradient Diversity (GD) of client updates is lower for the pre-trained model, while the original paper by Yin et al., 2017 argues that better convergence is achieved for higher GD.  As far as I understand the plots show results of single runs which is not very reliable. Cosine similarity behavior looks quite random and similar for different initialization, apart from maybe the start. The connection to theory is not really convincing.

I would also like to ask how exactly the Hessian matrix was computed.
clarity,_quality,_novelty_and_reproducibility: 1. The paper writing is clear and of good quality. The contributions are quite original but the comparison to prior works needs to be more detailed for the final publication. For instance, why the previous papers did not find that pre-training does not help to tackle heterogeneity issue?

2. I would like to ask why these particular deep learning models were chosen. In previous works simpler CNNs were used for FEMNIST and LSTM-based models for Stackoverflow. The author's choice makes it a bit harder to compare the results to prior works. Although, this is not critical.

3. It would be helpful to briefly and explicitly mention the client updates formulas in the main text or Appendix.

**Minor**

4. Links (references, figures, appendix) are not clickable, and the Sections outline navigation is not working properly for PDF readers.

5. Typo: on page 7 $\delta_i$ is first used instead of $\Delta_i$
summary_of_the_review: Overall, the paper represents a piece of good-quality research. I think that there are enough insights for publication. There is also a potential for an increase in my score if some of the questions and concerns are addressed by the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study federated learning starting from pre-trained models in both the image and text setting. They study a wide range of FL algorithms. Some key observations are that (a) the gap of iid to non-iid closes (b) ranking of FL algorithms is different under pre-training (c) effect of heterogneity are not severed. An investigation of how pre-training helps under heterogeneity is performed. 

strength_and_weaknesses: Strengths:
-The evaluations are extensive covering a wide array of tasks, datasets, and federated optimization settings
-The observations are potentially important: methods addressing heterogeneity may not be relevant in cases where pre-trained models can be used

No substantial weakness are noted 

Some Comments: 
- The effect of the pre-trained model is not heavily studied. It would be interesting to know if better pre-trained models lead to better results. 
- I would like to see some discussion on how the learning rates selected are different (or same) in the pre-training/random setting
- It would be interesting to see the effect on more distant combinations of pre-training and target data. For example imagenet->medical/satellite image data
- In Chen et al it was shown that FedAVG can benefit even from pre-training on synthetic data. It would be interesting to see if such approaches (and the change in ranking of algos) hold for the algorithms evaluated here.



clarity,_quality,_novelty_and_reproducibility: The article is well written, the observations are clearly stated and some of the limitations are highlighted. Code to reproduce the experiments is also provided
summary_of_the_review: The paper has a number of important findings for guiding FL research and practice. The evaluations are thoroughly done across a wide range of datasets and tasks. Insights into the observed effects are also provided. 

Post-Rebuttal: I have read the rebuttal and response to other authors. My review and score stand.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the impact of model initialization(random or pre-trained) on federated optimization methods. The experiments show that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity. The experiments also show that the pre-trained initialization reduces the training time.
strength_and_weaknesses: Strength:

1)The paper is well written, and the experiments are sufficient.

2)The paper finds that starting federated learning from a pre-trained initialization reduces the effect of both data and system heterogeneity.

3)The experiments show that the pre-trained initialization leads to communication savings and reduces the overall training time in federated learning.

4)The findings in this paper suggest directions for future related work.

Weaknesses:

1)In Figure 1 and Figure 5, what is the representation of the curve radian?

2)The α in Figure 2 is not explained in the main paper.

3)Does the model pre-trained on public data introduce security issues to users? How do we make trade-offs between security and performance?

clarity,_quality,_novelty_and_reproducibility: The paper is well organized but the presentation has minor details that could be improved.

The paper appears to be technically sound and the experimental results convincingly support the main claims.

The paper contributes some new ideas.

The author has submitted the code and the experimental details are sufficient. The contribution is clearly presented and the experimental results are thoroughly executed. I would tend to accept this paper.

summary_of_the_review: The paper is significant to model initialization(random or pre-trained) on federated optimization methods. The contribution is clearly presented and the experimental results are thoroughly executed. I would tend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the impact of initializing from a pre-trained model (which is trained using some other/available data set) on the overall performance of the model (after being trained using the actual data). The paper provides a number of interesting messages through extensive numerical studies: 1) Pre-trainig helps to achieve a better performance. 2) Pretraining helps to alleviate the  effect of both data and system heterogeneity. 

All the reviewers found the results very useful and recommended accepting the paper. 

I would like to add one point (given the last two sentences of the abstract): If we look at data heterogeneity from the lens of representation theory, then the results of the paper would make a lot of sense. There is a line of work on FL that argues that in many FL datasets and applications, even though the users data are heterogenous, they often have a shared representation. And this shared representation can be exploited to learn better models. In the case of pertaining using another (often rich) dataset, a good representation will already be learned through the pertained model. In other words, the pertained model will provide a good representation of the data that can be further adapted  ti the data of the users in the main FL training procedure. I recommend that the authors look at the literature of defeated representation learning and establish a connection to their observations in the related works. A few papers that come to my mind are the following: 

1. Collins et al: Exploiting Shared Representations for Personalized Federated Learning 
2. Pillutla et al: Federated learning with partial model personalization
3. Collins et al: FedAvg with Fine Tuning: Local Updates Lead to Representation Learning  
justification_for_why_not_higher_score: The paper has very interesting empirical findings. It could have been an oral if the authors were able to demonstrate the reasons behind their observations. 
justification_for_why_not_lower_score: It's a good paper with high scores.
Decision: Accept: notable-top-25%