Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Federated learning has two development targets of a single global model and multiple personal models under data hetegeneity. This paper focuses on multiple personal model development and proposes FedBABU, Federated Averaging with Body Aggregation and Body Update. In FedBABU they split the whole network into the body part with representation function and the head part with classification function. To make better representation part  is important for personalization and they train the local models and a global mode with the fixed head until personalization. In personalization they train each local model which consists of shared global body part and local specific head part using local data. This paper shows FedBABU outperformers other FedAvg-variant personalization algorithms in many viewpoints. Such as the final performance, robustness against hetegeneity, and faster learning speed.

main_review: This paper describes enough details and evaluations to understand their proposed methods and its superior performances compared with other methods. The idea itself is simple but there are enough supporting materials in the paper. Supplementary materials also help and  support their proposals. Focusing on the final personalization part and how the whole learning sequence should be is well described.
This paper reveals many advantages against other competitive methods. Among them its robustness with higher performance against data hetegeneity is impressive for practical federated learning. This paper also reveals that  head-part only training is enough for personalization and this leads to faster learning speed. This is also important beneficial performance point in federated learning.
As for weakness the random head part is important in FedBABU and supplemental materials argues this issue but more performance evaluations dependent on head part initialization would support their proposal.
summary_of_the_review: The problem definition is clear and there are enough materials and descriptions to support their proposals. This paper has supplementary materials and those also support their proposals.
In federated learning data  hetegeneity is a most important issue. As for their performance advantages in comparison with other competitive FedAvg-variant methods, FedBABU is robust against data etegeneity and outperforms others in personalization as well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides an interesting suggestion in the field of federated learning. In particular, it proposes to enhance federated learning performance by breaking down the network into the body and head and proposes to only train the body during the federated learning phase. Specifically, the authors show that a fixed random classifier can have comparable performance to the learned classifier, and thus propose to share a fixed random classifier across all clients. The authors show the proposed method is efficient especially when the level heterogeneity of the data is more intense.
main_review: Pros:
1. The paper takes practical issues of federated learning: what creates degradation problems in training heterogeneous data in federated learning. For me, the problem itself is real and practical.  
2. The proposed training scheme is novel and very easy to implement. The authors provide the reason why the model head and the body should be trained separately by comparing the performance (Table 2). 
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the FEDBABU training algorithm in federated learning. 

Cons: 
1. Although the proposed method provides a study(Table 2), I still suggest the authors provide more investigation of the problem of training the head in federated learning. 
How does the randomness of the shared head affect the performance?
Qualitative comparison between training the body only and training the whole network. Maybe plotting the distribution before the head.
How about sharing the fixed random two(or three?) last layers instead of one (head)

pg 5. ‘It is thought that a small s makes local tasks easier because the label distribution owned by each client is limited and the number of samples per class increases.’ <- Could you elaborate this portion more with evidence?


summary_of_the_review: Overall, I vote for accepting. I like the logical flow that 1) capturing the root of degradation lies in the head, 2) prove the feasibility of using a shared random head, 3) compare the performance between the proposed scheme and the baseline. I think the authors find/tackle the problem in a clear manner. My remaining concern is if it were really the last layer(head) only that creates a degradation problem in a federated learning setting. If so, why? Hopefully, the authors can address my concern in the rebuttal period.

=====POST-REBUTTAL COMMENTS======== 
Thanks for the authors' response. The author`s responses address well my concern, and I recommend this paper be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper highlights the importance of the shared parameters in personalized federated learning (PFL). The general idea advocated in the paper is that during training, only the feature extractor should be learned and aggregated on the server, while the classifier head should remain fixed. Personalization is achieved after training by fine-tuning the entire local model for each client. Through extensive experiments, the authors support their claim and show improved performance compared to several baseline methods.
main_review: Strengths:
- A simple method that retains the FedAvg learning scheme and achieves good results.
- The analyses in the paper are exhaustive and indeed support the author's claim for the most part. 
- The paper is written clearly and easy to understand.
- The results seem to be reproducible and code was provided.

Weaknesses:
- Although this is the first paper, that I am aware of, that stresses the importance of learning the feature extractor layers only, I do not consider this idea as an entirely novel contribution of this paper. First, as the authors state, similar ideas are presented in the few-shot learning literature, which is somewhat (yet not entirely) similar in spirit to federated learning setup (hence the success of some FSL methods in FL, such as Per-FedAvg and Reptile). Furthermore, in [1] the authors suggest exactly that learning procedure only with Gaussian processes. I think that this paper should have discussed [1] and compared to it as it also showed large improvements over PFL methods.
- Unless I missed something, I believe that the comparison to baseline methods is not entirely fair. First, it seems that the total number of updates was fixed to 320. Yet, it may be that some baselines require more updates to converge to the optimal solution. 
I believe that a better approach to compare with baseline methods would be to set a maximal number of communication rounds and use validation-based early stopping. Then this method could be evaluated against baselines in terms of performance and convergence speed. Second, from the Appendix, it seems that hyper-parameters of baseline methods were not adjusted. This can severely hurt their performance.
- Following the last bullet, did you use a validation split? If not, how did you set the hyper-parameters for your method and baseline methods? Some hyper-parameters, such as the learning rate, may have a detrimental effect on the performance [2], and a good configuration may vary between baseline methods.

General comments:
- Did you train the feature extractor with the nearest template method or in the standard way?
- The current order of the rows in Table 2 makes it a bit hard to compare between F and B rows. I would switch the ordering to the following: p=0,0.05(F),0.05(B),0.1(F),0.1(B).


[1] Achituve, I., Shamsian, A., Navon, A., Chechik, G., & Fetaya, E. (2021). Personalized Federated Learning with Gaussian Processes. arXiv preprint arXiv:2106.15482.  
[2] Hsu, T. M. H., Qi, H., & Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.
summary_of_the_review: Overall it's a nice paper, the results and the analyses are good. Nevertheless, there are some issues. First, it ignores [1] which uses a similar idea. Second, I am not entirely convinced that the empirical evolution of the baseline methods is fair. I am willing to reevaluate the paper based on the author's response and adjustments to the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new personalization algorithm for FL wherein only the "body" of a model is updated during federated training (not the terminal linear layer), and then a model is fine-tuned (or "personalized") on client data. The authors suggest that this resolves a tension in the existing literature/methods, where better global performance typically comes at the expense of worse local performance. Results are supported with experiments on CIFAR100 with comparisons to several baseline personalized FL methods, and FedAvg.
main_review: Overall, I feel that the paper is promising, but empirically underpowered to support the claims it is making. The overall concept for FedBABU aligns with broader empirical trends in the ML community related to transfer learning, but this does not eliminate the authors' burden to provide either strong empirical results, or theoretical motivation (or both) for their work. As-is, the claims made in the paper are too strong given that all experiments are only on one dataset which is synthetically partitioned into clients. I think that it could also use some revisions for clarity, both in writing and presentation of results. I provide other detailed feedback below.
summary_of_the_review: 
## Major Comments

* The paper makes some very strong claims which I don't find to be fully supported by the experiments. For example, "we argue that training the head using shared data negatively impacts the personalization". This is a strong, broad claim, but it is made on the basis of experiments on only a single, artifically-federated dataset (CIFAR100). In the absence of (i) stronger empirical evidence, or (ii) some theoretical motivation for this claim (either novel theory or via connections to existing theory), it is difficult to fully get on board with these claims, which form the foundation of the proposed algorithm. If there is empirical evidence from other works supporting the authors' conclusions here, please also cite it. (Another claim needed more support is that training the "head" during federated rounds degrades performance.)

* The claim that "randomly fixed classifiers are acceptable..." is not clearly connected to the authors' statements about orthogonality, which are relegated to the Appendix. Even on reading Appendix B, it's not clear to me how this discussion of initialization proves the authors' claim, or whether this also might imply that personalization of the head might also be unnecessary under certain conditions. Perhaps you could draw a more explicit connection, or link this to work on randomly-featured models, since I sense a connection.

* It's not clear to me why the authors opt to fine-tune the entire model, in order to accommodate FedAVG (mentioned at end of Section 5.2.2), when the results on FedBABU suggest that fine-tuning only the head is sufficient. This seems to require extra compute, and increase the risk of overfitting, while providing no benefit to the algorithm of interest, FedBABU. Please clarify. (I will note that, in T5 (Raffel et al 2020), the authors show that fine-tuning the entire model is preferable to fine-tuning only the later layers, if I remember correctly.)

* I feel that a key analysis missing from the paper is related to the degree of overfitting (or not) that happens during fine-tuning. It is unsurprising that fine-tuning a small amount improves performance, on the clients' data; it is important to know, however, if that results in a much poorer performance on out-of-distribution data (i.e. another client, or a more globally-representative dataset); I was surprised not to see such an analysis in the paper, particularly because this seems of both theoretical importance (the authors imply a kind of tradeoff between global and local model accuracy) and practical importance (it is likely that real-world client data will drift). I would suggest some empirical evidence to this end, or at the very least some discussion of it.

* The finding, in Sec 5.2.3, that FedAVG outperforms *all* preexisting personalization algorithms, except FedBABU, is to me very significant. If true, I think that this should be a "main contribution" of the paper and should be mentioned much earlier, since it suggests that this is the only current personalization algorithm that actually achieves its intended aim.

## Minor Comments

* The tables are very hard to read; I would recommend replacing them with bar graphs (with error bars) and optionally including the tables in the supplement.

* The switching nomenclature between body/extractor and head/classifier was quite confusing to me as a reader, particularly since "classifier" often refers to the *entire* model and I am inclined to read it that way. Please choose one terminology and use it consistently.

* Fig. 1 could be improved a great deal. It needs an illustration of "head" vs. "body", as this is a (the?) major difference from FedAvg that is not currently aptured. Along those lines, it would also be useful to note what is transmitted to the server in the rightmost column, since again, this differs between FedBABU and FedAvg. Also, note that the figure has room to be wider; perhaps you could add a fourth column characterizing the personalization step of FedBABU.

* It seems there is no explicit connection made to the "pretrain -> fine-tune" paradigm that is increasingly prevalent in the broader machine learning community (now also referred to as the use of "foundation models"), although some works in this area are cited (e.g. BERT). An explicit connection would be useful here, and would help motivate the use of a similar approach in FL via FedBABU. Please clarify how, if at all, you believe this method to be different from this transfer learning paradigm.

* On p.5, the claim that global training pays attention to "unnecessary and confusing information for personalization" seems like an incorrect characterization. It may be "irrelevant" to a specific client at a given point in time, but we might reasonably believe that such data actually improves the generalization capabilities of the model (including if that clients' distribution changes in the future) -- does that make it "unnecessary"?


* The authors reference multiple times using a learning rate of zero to not update the head layer (which implies that a no-op is performed, for every paramter, with learning rate zero)...why are these updated being performed at all, instead of only updating \theta_{ext}?

* The conclusion references "regularization" as reducing models' ability to personalize; I am not sure which part of the analysis/experiments this refers to. Please clarify.

## Typos etc.

* All of the "i.e."s in the abstract should be removed.

* P.1: "popular networks have one linear layer...and ResNet" -- this sentence is confusing; please revise.

* P.2 The phrase "representation learning based on the same fixed criteria" is not clear; a similar phrase is used in 5.2. Please clarify what a "criteria on learning representations" (5.2) is and what the "intuition" you are suggesting here may be.

P.3 "When the bottom layers are matched with the body" and "When the top layers are matched with the head" are unclear.

* P4 "decayed 0.1 times" --> decayed by a factor of 0.1

* Why abbreviate (B), (F) in Table 1? I would suggest "Body Updates" vs. "Head + Body Updates" or similar.

## Update after revisions

I thank the authors for their response. Having read the author response, reviewed the updated manuscript, and considered the other reviews, I am keeping my score the same. While the authors addressed several more minor points regarding the paper, I don't believe they addressed my main issue with the paper, which is that the empirical evidence is fairly weak and limited to only a single dataset; I think that either more empirical evidence, or some theoretical support for the proposed method, is needed before being able to argue for acceptance.
review_title: null

Meta-review: The paper makes some novel and interesting observation pertaining the relationship between data heterogeneity and personalization. Reviewers like the paper and ideas in general but raised several concerns. The rebuttal rectified several confusions and provided more clarification which convinced the reviewers that the paper is above bar for publication.
Decision: Accept (Poster)