Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper propose the Federated Neural Architecture Search (FedNAS) method to search for both global model and personalized model architectures collaboratively among edge devices and show its performance in a variety of federated learning  settings. It also investigate the role of NAS to address the challenge of data-heterogeneity in federated learning and show superiority in results. 
main_review: Strengths: 
1. The paper propose the Federated Neural Architecture Search (FedNAS) method to search for both global model and personalized model architectures collaboratively among edge devices, and show its performance in a variety of federated learning  settings. 
2. It also investigate the role of NAS to address the challenge of data-heterogeneity in federated learning and show superiority in results as well as its efficacy. 


Weakness: 
1. It was not clear the purpose of Figure 4 (1) given that the graph shows uniform local training dataset size. 
2. It seems that the paper is missing some strong baseline in NAS spaces. In related works, authors lists our some work in NAS, such as Kairouz et al,, 2019, Zhu & Jin 2020, etc.. The paper would be stronger to also benchmark against related methods in NAS. 
summary_of_the_review: Overall technical seems significant and somewhat new, but as mentioned above in the weakness, the paper is missing some comparison against stronger baselines, especially for previous methods in NAS.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors employ an existing neural architecture search method in the federated learning setting.  Specifically, the authors propose FeNAS and extend an existing NAS method MiLeNAS into federated learning to address the data heterogeneity problem and conduct personalization.  The experiments show that the proposed method is able to achieve improvement compared to some other federated learning methods. 
main_review: Strengths:
1. The authors address the data heterogeneity problems in federated learning via NAS methods. The personalized architecture shows good improvements. 


Weakness:
1. Originality: This work is a simple extension of MiLeNAS on federated learning. The NAS on clients for personalization seems different from other NAS works. But I did not see any challenge highlighted by authors and they directly adopt NAS methods for common scenarios. 
2. Quality: The aggregation on the central server follows the similar way of FedAvg. The most difference with other existing works is to introduce NAS into federated learning. However, during employing NAS into federated learning, only trivial changes are made. When architecture changes, the global aggregation still remains the similar way in traditional federated learning and leaves some unexplored spaces. Even though MiLeNAS is a recent effective method, other NAS methods should be introduced into federated learning to justify the authors' choice.  
3. Soundness: One of the most important parts of federated learning especially in new settings is convergence guarantee. The architectures on clients keep changing and personalization is needed, resulting in more heterogeneous model weights and architectures. How to guarantee the convergence of the proposed framework? I will suggest authors provide some theoretical proof to support this method for convergence. 
summary_of_the_review: The authors extend an existing NAS method MiLeNAS on federated personalization. However, the novelty of the work is somewhat limited.  It is lacking clear support evidence and insights to support why MiLeNAS was chosen. To support the idea of NAS for federated learning, more other NAS methods are preferred to provide more insights and depict the challenges of this problem.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for automatic model design in the context of federated learning and introduces an auto-ml system under the framework of federated learning.


main_review: Strength:
1. The paper is generally clearly written.
2. The codes are open-sourced for reproducibility.

Weakness:
1. The motivation is not clear. The abstract says "We propose Federated Neural Architecture Search (FedNAS) for automating the model design process in FL." and the introduction says "We aim to address data heterogeneity in FL via ... NAS." For me, it is not clear whether the paper aims to automate model design in FL or solve the non-IID problem.
2. NAS method and FL are naively combined, which undermines the novelty of this paper.
The comparison experiments conducted might be not fair enough. Model searched by NAS are generally smaller than handcrafted ones while enjoying higher accuracy. For example, in [1], a 4.6M ENAS model outperforms a 25.6M DenseNet model. In comparison, I think a DARTS model should be used as a base for previous FL methods.

[1] H.Pham et.al, Efficient Neural Architecture Search via Parameter Sharing. ICML’18.
summary_of_the_review: None
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines gradient-based NAS (DARTS-like algorithm: MileNAS) with Federated Learning (FL) setup, to improve both global and personalization performance with learned neural architecture. Since both NAS and FL learning are based on gradient, the extension to FL setup becomes intuitive and effective. Empirically FedNAS shows improved performance comparing to existing FL methods. 
main_review: The global model search with FedNAS is better is not surprising, since there is no difference between training on FL and centralized settings, on only gradients. What becomes really interesting, is the personalization performance. Comparing to existing personalization methods which fix network structure and personalize weights only, FedNAS optimizes in both neural architecture and weights, which results in better personalized performance (both by average and by individual), and smaller model. 

Contributions:

The paper's work is a natural extension of MileNAS. Since MileNAS, like DARTS, is gradient-based algorithm, then make it available to Federated Learning is natural. In this sense, the innovation is not extraordinary. 

There are some minors to be improved:

(1) Experiments are not sufficient. Although for CIFAR10 and GLD-23k, the results are pretty extensive, more experiments on other FL datasets (EMNIST-62, CIFAR100, and even Shakespeare), could make the paper stronger. 

(2) Moreover, FedNAS search architecture only applies for Computer Vision (specifically, image classification problems). How to extend NAS to NLP tasks or other paradigms, could be an interesting next step.

(3) Interpretation of the learned architecture. The performance gain originates from personalize both architecture and weights. The author confirms that the personalized performance does improve, but the interpretation is missing. It could be interesting, to see what if some clients learns less/more complex models.

summary_of_the_review: Overall, the proposed method to use gradient-based NAS for FL to improve personalization, shows good empirical performance. The complaint of mine, is that the innovation is on applying existing method (MileNAS, or DARTS-like gradient-based NAS), directly onto FL settings. There are very few modification on original method, other than the settings are different.

In all, I give this paper score 6, and I am willing to discuss with other reviewers and the authors in later rebuttal session.
review_title: null

Meta-review: This paper proposes a personalized federated learning framework based on neural architecture search, in which the local clients perform NAS to search for a better architecture for the private local data. Specifically, the authors extend MiLeNAS, which is an existing NAS algorithm, to be run in the federated learning setting, and use FedAvg for model aggregation. The proposed FedNAS framework is validated against personalized federated learning methods with predefined architectures, such as perFedAvg, Ditto, and local fine-tuning, and is shown to largely outperform them on non-IID settings with label skew and LDA distribution. FedNAS’s collaborative search for the optimal architecture also yields a better performing global model than FedAvg.

The paper received borderline ratings. Three out of four reviewers are learning negative, while one is leaning negative. The below is the summary of pros and cons of the paper mentioned by the reviewers:

Pros
- The idea of using NAS for personalized federated learning seems novel and interesting. 
- The proposed FedNAS framework is shown to be effective in tackling the data heterogeneity problem, which is a fundamental problem with federated learning.
- The authors have released the code for reproducibility.

Cons
- The technical contribution of the work seems limited, since the proposed FedNAS straightforwardly combines an existing NAS method (MiLeNAS) with federated averaging, and there is no challenge mentioned for this new problem of federated NAS. 
- The choice of a specific NAS method (MiLeNAS) is not well justified, and other NAS methods should be also considered.
- The motivation is unclear: It is not clear whether the authors aim to perform collaborative automotive design or solve personalized federated learning.
- There is no convergence analysis.

While some of the concerns have been addressed away in the authors’ responses during the rebuttal period, the reviewers did not change their ratings, and the final consensus was to reject the paper. 

I agree with the authors that combining federated learning with NAS, and applying it for personalized federated learning is a novel idea that intuitively makes sense. However, I agree with the reviewers that the current method is a straightforward combination of an existing NAS method and an existing FL algorithm, the authors should identify new challenges posed by the combination of the two methods, and identify them. 

Further, performing NAS on edge devices may be possible, but not the best solution, since it could result in large computational overhead. While the authors mention that MiLeNAS is computational suitable in such settings, there should be a proper investigation of the accuracy-efficiency tradeoff, showing how well FedNAS performs against others with the same computational budget (or training time / energy consumption).  

Overall, this is a paper that proposes a novel and interesting idea that seems to work, but the paper does not sufficiently examine challenges posed by the new problem. I suggest the authors identify the new challenges and examine the efficiency issue mentioned, and further develop their method, if necessary.
Decision: Reject