Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes SPIDER, an algorithmic framework that aims to search personalized neural architecture for federated learning. SPIDER alternatively optimizes a global model (called supernet) and a local model that is connected to global model and has its own architecture. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods with much fewer times of hyperparameter tuning.
strength_and_weaknesses: Strength:
- This paper proposes a novel method for federated neural architecture search 
- Empirical results against several existing baseline methods show some improvements of the proposed method

Weakness: 
- There are some unclear parts on the proposed method: 
  - I am a bit confused on the definition of $a_k$, i.e. the architectural parameter for each client. Is this 0/1 variables whose elements are associated with operations in supernet? The authors may need to clarify that and revise some notations. 
  - Based on the above concern, how is $a_k$ updated in Algorithm 2? Specifically, in step 7, the authors evaluate the validation accuracy of $a_k^t$ with one operation removed. Nevertheless, note that only one operation is kept in each iteration, then how can we evaluate the validation accuracy of $a_k^t$ with more operation removed? This is very confusing and require some clarifications. 
- Some more discussions are needed for the proposed method:
  - I am a bit uncertain on the differences between SPIDER and the following method: perform supernet training, then distill useful subnetworks from the trained supernet based on local data, while SPIDER distills subnetworks during training. The authors may add some discussions on this variant. 
  - Moreover, can algorithm 2 uses other NAS methods (e.g., [1])? Some discussions may also be needed to make the proposed method more complete. 
- Some recent baseline methods [2, 3] are missing in current version, which makes it hard to evaluate the actual performance of the proposed method. The authors may need to add some comparisons to them
- Ablation studies are also largely missing in current version. It is unclear how each part of the proposed method contributes to its empirical performances. 
- Minor issue:
  - It may be better to move some parts of the supplementary material to main text, e.g., architecture personalization gain, which seems critical for the whole paper
  - The authors may check the citation format. Currently citations are mixed with the main text and creates some difficulties in reading
  - Typos in abstract: line 6 “clientsin -> clients in”?

[1] Single Path One-Shot Neural Architecture Search with Uniform Sampling. ECCV 2020

[2] Personalized Federated Learning through Local Memorization. ICML 2022

[3] Federated Learning with Partial Model Personalization. ICML 2022

clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is mostly clear, though several parts require some clarifications. There are also some typos and formatting problems. 

Quality: the paper can still be further improved. Some more discussions are needed to make the proposed method complete, and the experiments also need some improvements to fully evaluate the proposed method. 

Novelty: the proposed method is novel and substantially different from existing methods in my opinion. 

Reproducibility: this paper provides some information on its hyper-parameters, but does not provide its code, which may still create some difficulties repeating its experiments. 

summary_of_the_review: While this paper proposes a novel method and empirical results show some improvements, the proposed method needs more clarifications and discussions. For the empirical results, some recent methods are missing in comparison, and ablation studies are also largely missing. Thus I choose reject as my initial score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The submission proposed NAS algorithm for personalized Federated Learning. Specifically, it alternately optimizing one homogeneous global model and one local model that is the subnetwork of the global one. Basically, the idea is not novel and the solution is straightforward, limiting its contribution to the community. Moreover, the notations used are not well-presented, making the draft hard to follow. I thus do not champion the acceptance.
strength_and_weaknesses: 
Strong points:
S1: the results seem promising.

S2: the literature review is somehow thorough.

Weakness:

W1: the idea is not novel and the solution is straightforward. The submission claims in the related work section “Where all these models search for a unified global model, a key distinction of our work with these works is that we aim to search for a personalized model for each client.” However, this is not true. Many existing work such as the following two are also providing NAS for different clients in FL.

1). Personalized Neural Architecture Search for Federated Learning, Minh Hang et al. Neurips Workshop 2021.
2). Personalized Federated Learning via Heterogeneous Modular Networks. Tianchun Wang, et al. ICDM 2022, 2022.

Actually, these two papers are using more elegant solution for PFL. The proposed solution adopts the mixture method of ideas of NAS and MAML. It needs to maintain two networks (one global and one local) that causes additional communication/storage burden in FL. The second term (regularization) term in formula (5) is very straightforward. 


W2: the notations used are not well-presented, making the draft hard to follow. The notations used are not formal. Typically, matrix, vectors, and scalars are using different fonts for easy reading. However, all the notations are using the same fonts. 

W3: There are too many typos and grammar errors indicating the low-quality of the draft. Even in the abstract, there are clear typos, e.g., “all the clientsin FL”.
More examples, in the intro “we demonstrate accuracy gain of 10%, 6%, 4% over”==>”we demonstrate an accuracy gain of 10%, 6%, 4% over “
In the related work “ meeting clients efficiency budgets”==> meeting clients’ efficiency budgets

W4: The experiments are far from complete. The authors only compared to very few PFL baselines. Many more related works are not compared. For example, the most relevant “Personalized Neural Architecture Search for Federated Learning, Minh Hang et al. Neurips Workshop 2021.” Was not compared.

W5: There is no theoretical analysis if the algorithm will converge or not.

clarity,_quality,_novelty_and_reproducibility: the notations used are not well-presented, making the draft hard to follow. The notations used are not formal. Typically, matrix, vectors, and scalars are using different fonts for easy reading. However, all the notations are using the same fonts. 

There are too many typos and grammar errors indicating the low quality of the draft. Even in the abstract, there are clear typos, e.g., “all the clientsin FL”.
More examples, in the intro “we demonstrate accuracy gain of 10%, 6%, 4% over”==>”we demonstrate an accuracy gain of 10%, 6%, 4% over “
In the related work “ meeting clients efficiency budgets”==> meeting clients’ efficiency budgets

There are no code release thus hard to validate the reproducibility.
summary_of_the_review: The submission proposed NAS algorithm for personalized Federated Learning. Specifically, it alternately optimizes one homogeneous global model and one local model which is the subnetwork of the global one. Basically, the idea is not novel and the solution is straightforward, limiting its contribution to the community. Moreover, the notations used are not well-presented, making the draft hard to follow. The experiments are also very short. Many baselines are not compared. I thus do not champion acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper formulates a personalized neural architecture search framework for the federated learning, aiming at searching for the personalized neural architectures for each client under the federated learning scenarios. The topic and the proposed idea of this paper are novel, but the experimental analysis and the technical depth of this paper are not enough.
strength_and_weaknesses: S1. The topic studied in this paper, i.e., realizing both weight and architectural personalization in the federated learing (FL) scenarios, is important and novel in the field of FL. 
S2. This paper is well-written, the problem definition and algorithm description are clear and easy to follow.
S3. In the experiments, the authors compare the proposed method with the personalized FL methods with a fixed model architecture, demonstrating the significance of realizing architectural personalization in the field of FL.

W1. A kind of solution for the problem studied in this paper is to combine the existing personalized federated training method with the existing gradient-based neural architecture search (NAS). That is using the personalized federated training method to replace the non-personalized single-machine model training method applied in the existing gradient-based NAS method. In this way, both the architectures and the weights can be optimized in a personalized manner under the FL framework, realizing both weight and architectural personalization in the FL scenarios. 
Authors should compare the above solution (using different NAS algorithms and federated training methods) with the proposed method in the experimental part, to demonstrate the superiority and the importance of the proposed method.
W2. What’s the difference between the solution mentioned in W1 with the proposed method in this paper? The solution mentioned in W1 is straightforward (it is just a simple combination of the existing works), and more appropriate solution should be designed to deal with the novel federated NAS problem. Does the novel federated NAS problem bring new challenges, requiring novel NAS solutions or federated training methods? Does the proposed method solve these challenges? Authors can discuss these contents in depth.
W3. The experimental analysis is not enough. The performance curves, i.e., the federated performance of the proposed method and other federated NAS methods at different time steps, are not given and not analyzed in the experimental parts. These curves can reflect the search effect and the search efficiency of different algorithms under the FL framework, which is important for evaluating a federated NAS algorithm.
W4. Why perturbation-based NAS is selected to realizing the NAS operations in the proposed method? Why not select the other gradient-based methods? The selection basis should be clarified in the next version. In addition, authors should add experiments to examine the differences of selecting perturbation-based NAS instead of other NAS method.
clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality of this paper is good.
summary_of_the_review: The topic is novel and important, the paper is well-written and the experiments demonstrate the significance of the proposed method. But some important baselines are missing in the experiments, the novelty of the proposed method should be discussed in depth, the experimental analysis are not enough, and the design of the proposed method are not clear. Suggest to further modify this paper for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors tackle personalized neural architecture search for federated learning, where data heterogeneity is alleviated by the architectural personalization. The authors propose a method named SPIDER that uses a supernet from which local models can be sampled while sharing the weights. The authors validate their method on several benchmarks against existing methods.

strength_and_weaknesses: **Strength**
- The paper is easy to read.
- They tackle the practical federated learning with architectural personalization.
- The results are impressive.

**Weaknesses**
- The proposed algorithm trains two models at each client, both global supernet and local model, which may increase the computation and memory consumption of local resources. Although NAS is devised to optimize consumption of given resources, the proposed framework seems to be inefficient and even not practical in real-world scenarios. 
- In the same context, the overall communication and computation costs should be discussed and compared with the baseline models. But, I can only find the comparison of the searched model’s performance and its size (parameters).
- It is better to compare with more similar existing baseline models that uses supernet for sampling the local models, i.e. HeteroFL [Diao et al 21] or FedorAS [Dudziak et al 22], in terms of both efficiency (communication costs, etc) and performance.
- More experiments should be conducted for detailed analysis, i.e. ablation study.

Diao et al, HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients, ICLR 2021.

Dudziak et al, FedorAS: Federated Architecture Search under system heterogeneity

clarity,_quality,_novelty_and_reproducibility: There are some rooms for improving the quality, clarity, and originality of the proposed method. For example, detailed comparison with some existing methods that are utilizing the supernet for Federated NAS (i.e. HeteroFL, FedorAS, etc) and more analysis for the proposed method (i.e. ablation study)
summary_of_the_review: I enjoyed reading the paper, but several improvements seem to be required, as mentioned above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a NAS algorithm called SPIDER that targets at finding personalized architectures for different client under federated learning (FL) setting. The challenges here are data heterogeneity and data invisibility. SPIDER alternately optimizes one homogeneous global supernet and one local model that is the subnetwork of the supernet. Specifically, a progressive search strategy is utilized for local clients. Experiments on several benchmark datasets show the promising performance of the proposed SPIDER.

Strengths 
- The topic is important and novel.

Weaknesses
- Experimental analysis and the technical depth of this paper are not enough
- Presentation is hard to follow. Notations used are not well-presented
justification_for_why_not_higher_score: 1. While the problem can be interesting and important, there is little technical novelty in this paper. More discussion with related works should be included to draw a distinct boundary between existing works and the proposed SPIDER.
2. Presentation should be improved.
Decision: Reject