Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper explores a simple but novel modification to FL approaches to better deal with settings involving non-homogeneous clients. They find that by allowing clients to mix local and global parameters for initializing local training, the added flexibility can notably improve the performance of the respective local end-models. They perform both an exploration of natural fixed mixing strategies and also propose an adaptive mixing strategy that allows for further improvements on average. 

main_review: **Originality:** As mentioned in Sec 4, specific instances of separating global/local parameters in FL have been touched upon in previous works. However, this paper certainly seems to usefully generalize such approaches, performing a more thorough empirical investigation into the space of mixing strategies and also introducing a new adaptive strategy. 

More on the presentational side though, I do think the manuscript could be more up-front about this relationship with these works (i.e., [2, 1, 14] specifically). To the authors credit, they do mention these citations propose special cases of PartialFed-Fix towards the very end of the paper. However, given this close relationship, I do feel this warrants mention much earlier on, such as in the intro and the relevant subsections of Sec 3. 

**Quality:** The results seem solid as a whole. Just had a couple comments here

(1) Given the closeness of the some of the numbers, including information about # of trials and error bars in the main (and also adding them for UODB) would be helpful.

(2) Perhaps not necessary but for completeness, would be interesting to see how the current best PartialFed results compare to SOTA centralized MT/Multi-domain learning results.

**Significance:** The problem seems well-motivated, especially judging by the demonstrated failure of more standard FL methods to be effective on the given datasets (performing worse than fully local learning).  Also, the results seem to indicate that PartialFed is indeed a valuable general strategy, leading to improvements on many of the tasks/clients involved. 

**Clarity:** There seems to be some room for improved presentation/clarity. Some things I noticed in particular are:

*  The notation in Eq. (4) seems a bit confusing in that the training loss is minimized over both $A_t^c$ and $\mathcal{W}_t^c$. The latter doesn't really appear in the loss expression?  Also, may be helpful to clarify the $u_t^c$ are the soft samples. 

* That final global models are all assumed to be fine-tuned locally could be made more clear, especially in Sec 2. This is mentioned in Sec 3 but in Sec 2, it is only implicitly mentioned in the Table 1 caption.

* Some abbreviations/notation were used without being formally defined such as fc, BN, PFL

* Not central to the paper’s main claims but “Clients can enjoy themselves from a highest privacy protocol” in line 92 does not seem strictly true, given that these approaches may still be liable to downstream inference attacks, i.e. the field has looked at things like differentially private FL to add even more protection.

* Could be nice to move Table 4 closer to the actual results table for UODB for easier cross-referencing

limitations_and_societal_impact: Yes, I do think the authors have discussed the limitations that come to mind. In terms of societal impact, it does not seem to introduce any new negative outcomes relative to traditional FL.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a novel idea PartialFed, which loads a subset of the global model’s parameters rather than loading the entire model. Specifically, they validate the algorithm with manually decided loading strategies inspired by various expert priors, named PartialFed-Fix. Then they develop PartialFed-Adaptive, which automatically selects a personalized loading strategy for each client.  
main_review: Pros:
-- This paper gives a comprehensive introduction of existing partial loading strategies. The experimental section also provides a comprehensive comparison for these methods with the proposed PartialFed-Adaptive algorithm.
-- The PartialFed-Adaptive algorithm shows certain levels of novelty to learn the initialization strategy through a data-driven approach that adaptively chooses which parameters to be loaded globally given the specific client.

Cons:
 --The PartialFed-Adaptive Algorithm is not clearly explained. e.g., what does $u$ in equation (5) stand for, and how $u$ is used in the optimization process?
-- The algorithm PartialFed-fix is not innovative enough, because there are already some existing works that split networks into global and private layers.
-- According to the experimental results, the layer-level PartialFed-Adaptive works best and outperforms stage-level and block-level PartialFed-Adaptive, but also increases the burden for training algorithms. Thus, layer-level PartialFed-Adaptive must be much more computationally demanding. I suggest that the authors can compare the training time of  PartialFed-Adaptive with different granularity to help achieve a balance between the performance and computational resources. 

===
Updates after rebuttal.

The authors provided a clearer explanation of the details of the proposed algorithm in the response, and also added some experimental results to compare the computation resources of algorithms of different loading levels which address my concerns. I raised my score but  I still reserve my opinion that compared with existing works that split networks into global and private layers, the innovation of this paper is still limited. 
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed an adaptive initialization algorithm for FedAvg, motivated by FedPer in a personalized federated learning scenario. The key contribution lies in the observation of the impact on training accuracy of different initialization strategies and the adaptive initialization algorithm. The empirical result of the algorithm shown significant improvement compared with FedAvg in training with heterogeneous data.
main_review: Originality: Good. The observation of different initialization strategies during local training is impressive, and the way of modeling the impact of the initialization patterns as a part of the loss function was also novel to the FL scenario. 

Quality: Good. The empirical results demonstrate the accuracy improvement with different initialization patterns is clear. The method used for updating the adaptive initialization pattern is promising. It has enough comparison between the proposed algorithm with the existing algorithms.

Clarity: Good. The explanation of the dataset and the key observations are easy to understand. The comparisons in the tables are also clear. However, there is some confusion in the algorithm description that needs to be clarified.

Significance: Good. The observation is important in the sense that it provides insight to understanding the impact of each block in NNs and provides a way to do personalized federated learning.

Comment to the paper:

1. In Algorithm 2, step (6) seems to indicate that for E local epochs, the parameter $w$ has not been updated. The algorithm only does $b$ SGD step on $W_b^c$ instead of doing $Eb$ steps. Is it my misunderstanding or the description of the algorithm is incorrect?

2. In other papers, such as FedPD, FedSplit, SCAFFOLD, and FedDyn which optimize local problems to $\epsilon$ accuracy and adaptively regulate the difference between the local and the global model, initialization seems not to be affected by the initialization. Is there any relation between PartialFed-Adaptive and the dynamic regularization methods?

limitations_and_societal_impact: There is no potential negative societal impact of this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new method called ‘PartialFed’ to solve the performance degrading caused by data heterogeneity in the federated learning scenario. The main difference between this work and previous work is that the proposed method partially loads global parameters given by FedAvg, while the previous methods load all of the global parameters for each starting point of the local epoch. 

For the sampling strategy for the parameters that load the global parameter, two methods are developed in this paper: “PartialFed-Fixed” and “PartialFed-Adaptive”. For the “PartialFed-Fixed”, the parameters are sampled manually based on the commonly accepted assumptions in deep network e.g. low-level knowledge learned from bottom layers. With the "ParitalFed-Adaptive", the sampling strategy is trained data-driven. Gumbel Softmax sampling is used to enable back-propagation through the sampling process. 

To validate the proposed methods, cross-domain federated classification and detection are examined. Two partial loading strategies outperform a set of state-of-art methods. Specifically, the “PartialFed-Adaptive” further reduces performance losses due to the extreme distribution heterogeneity and automatically shows the tendency that reflects the assumptions of deep network used for the design of “PartialFed-Fixed”.

main_review: The problem of non-IID data distribution in federated learning is becoming an active field recently. More precisely, this work can be integrated into the personalized layer approach, in which only parameters of some layers can be transmitted to a server and aggregated by FedAvg. Specifically, FedBN, FedPer, and LG-FEDAVG are a form of the “PartialFed-Fixed” sampling strategy. However, this paper provides more ablation studies, so they show that there are better variations on the personalized layering method. More generally, this paper proposes a method that adaptively learns the sampling strategy from data which is clearly different from previous contributions.

That being said, there are several weaknesses.

1) The important task of the work is to find a suitable sampling strategy for personalized parameters, but experiments are carried out with only one network structure for each experiment.
2) In section 3.3, Table 6 shows the experiment result on the cross-domain detection. In this result, the performance of PartialFed-Adaptive is not significantly better than the PartialFed-Fixed method. If we look at each domain respectively, PartialFed-Fixed models are better than ParitalFed-Adaptive models for eight out of 11 domains. Also, for two over three domains with large distributional or task deviations, “w/o fb (skip)” method in PartialFed-Fixed is better than “block-level” PartialFed-Adaptive model. This result can be interpreted in such a way that the proposed adaptive method may not be the optimal choice.
3) I could not see any theoretical or experimental analysis of the convergence of the proposed method, while related work including FedBN offers both.

Some minor comments are as follows.

However, it would be better if the following points were reinforced.
1) There is no description about the metric and its unit written in Table 1,2,3,5 and 6.
2) In line 102 and Equation (3), the notation f_c is used without a description. You can guess it as the fully connected layer before you see Figure 2, but it would be better to define it or to replace it with a word.
3) In line 146 and following lines, the notation k is used without a description. 
In addition, following points are typos.
1) In caption of Table 6, “se&BN denotes..” may be “w/o se&BN denotes for..”.
2) In line 172, Equ.4 should be Equ.5.


limitations_and_societal_impact: It is not clear that the partial averaging idea works for more complicated network architecture.
review_title: null

Meta-review: Thank you for your submission. The reviewers agree that the paper provides an interesting result. The authors should follow the reviewers' suggestions to improve their presentation in the paper.
Decision: Accept (Poster)