Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents FedorAS to discover and train architectures which are suitable for devices of various scales. FedorAS leverages a server-resident supernet enabling weight sharing for knowledge exchange between clients, without sharing common model architectures. This work also provides a new aggregation method called OPA for weighing updates from multiple “single-path one-shot” client updates in a frequency-aware manner. The method is demonstrated on different settings and shows a good performance while maintaining resource efficiency.
strength_and_weaknesses: The paper is well written, and the method is well demonstrated on various datasets compared with different baselines. But there are still some issues, which are discussed in the next part. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 
(+) The paper elaborates the proposed method in a detailed way, clearly explaining the background, logic of the method and the experiments. The necessary materials are also included in Appendix.
(-) The results are not consistent: In section 4.2, the author claims 48.7% better than CIFAR-10(\alpha=0.1). But in the Tab.2, there is only 26.69 (81.53-54.84) improvement. It is the same for Tab.3. In the content, it says +11.6%, but there is only 9.4 (90.6-81.2). Which is the correct one?
(-) The setting is unclear to me: After supernet training stage, does the system use the same or different model architecture for all clients? 
(a) If the model architecture is the same for all clients, how does the proposed method handle the system heterogeneity? That is, how to deal with the clients with different computational resources?
(b) If the model architecture is the different for all clients, is aggregation with OPA still working?

Quality: 
(+) This paper provides a detailed analysis of the method, and shows the importance of Supernet initialization and advantage of OPA over FedAvg.

Novelty: The design is based on the idea of supernet, some of the methods are heuristic and may need further analysis or explanation.
(-) (minor) In section 3.1 (1) Subspace sampling, it says “setting the limit to half the size of a typical network … worked well in our experiments …”. Does it apply to other experiments (e.g., different datasets)? Any explanation for this selection?

Reproducibility: 
The paper does not provide any code or algorithm block, but it clearly discusses datasets, model search space, and the package it is based on. So probably the results presented in the paper can be reproduced.

summary_of_the_review: The paper is overall well-written, providing a novel approach FedorAS to leverage a supernet for knowledge exchange between clients. The method shows better results by demonstrating on different datasets compared with multiple baselines. However, there are still some fundamental issues that need to be fixed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposed a neural architecture search (NAS) algorithm, FedorAS, under cross-device federated learning setting. The proposed algorithm follows settings of one-shot NAS utilizing a super-net search space with weight sharing. It enables knowledge exchange between server and clients via sub-networks. And a new model weight aggregation methods is introduced to compensate sampling frequencies of sub-networks at clients. Moreover, the experimental results validate the proposed algorithm with datasets in computer vision, audio data processing, and natural language processing. The comprehensive experiments cover various FL setting including IID/non-IID data distribution, cross-device FL/cross-silo FL.
strength_and_weaknesses: <Strength>

•	The paper is well-organized and well-written.

•	The experimental results supports the claims made in the manuscripts.

•	The experiments and ablation studies are comprehensive.

•	The submitted content is related to the application of neural architecture search in federated learning, which is highly relevant to the ICLR audience.

<Weakness>

•	The novelties of the proposed algorithm are limited. The idea of FedorAS is a direct combination of federated learning and single-path one-shot NAS.

•	Discussion or ablation studies are not sufficient (see below).

<Comments>

1.	What is training and searching efficiency of the proposed algorithm?

2.	How to determine that when to stop the searching process?

3.	Does the training recipe (e.g., batch size) matter for the NAS outcome? Do different training recipes or different random seeds result in different Kendall rank correlation coefficients?

4.	It is the necessary to show the gap between federated NAS (FedorAS) and the centralized NAS?

5.	Is it possible to search for different architectures with FedorAS when clients have different computing devices or budgets?

6.	How does the number of tiers affect the performance of the proposed algorithm?

7.	The necessary ablation study would be evaluating ranking correlations using the proposed super-net based NAS algorithm.

8.	Typo: Page 6, Section 4, “… we we draw …” => “… we draw …”.

clarity,_quality,_novelty_and_reproducibility: The manuscript is of moderate quality, with good clarity and limited novelties.
summary_of_the_review: The manuscript presented an interesting study of the proposed federated NAS algorithm. Its experiments are comprehensive. But its novelties could be limited as it is a straightforward combination of federated learning and single-path one-shot NAS.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method called "FedorAS" which incorporates ideas from Neural Architecture Search into the domain of Federated Learning. In the scenario where the devices participating in a Federated Learning session has varying capabilities, and heterogeneous data distribution across devices, the goal of FedorAS is to provide each device with an efficiency-tailored architecture and parameters.  

In FedorAS, the participating devices are clustered into several tiers, according to their hardware capability. Using a supernet architecture, from which a subspace is sampled for each device. The client, after receiving the subspace, trains the models within the subspace by randomly sampling the subpaths. The server then aggregates the updated client weights using a weighted averaging technique which normalizes for the number of examples passed through each subpath. After training, the server performs the NSGA-II algorithm to find the best architecture for each device tier. After that, the obtained models are fine-tuned with FedAvg among devices with the same tier or above.

The authors report that FedorAS achieves better performance than other cross-device baselines, on multiple domains such as speech, vision, and text.
strength_and_weaknesses: **Strengths:**
* FedorAS achieves superior performance over other baselines in cross-silo settings.

**Weaknesses:**
* An ablation study on the effect of the fine-tuning process is lacking. How much does the fine-tuning process affect the final performance?

* Experiments on a more extreme division of the device tiers are lacking. For example, in the CIFAR-10 experiments, it is assumed that the lowest tier devices are capable of 120.9MFLOPs, whereas the highest tier devices are assumed to be capable of 716.0MFLOPs. That is only about 6x difference. However, the FLOPs difference between real embedded devices and high-performance devices is more extreme, for example, the FLOPs difference between the M1 chip and the Raspberry Pi Zero is about 500x, given the average power for each device. [1]

[1] https://web.eece.maine.edu/~vweaver/group/green_machines.html
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and clearly written. The method proposed in the paper is novel, but the authors do not provide any code for it, which significantly undermines the reproducibility of the paper.
summary_of_the_review: FedorAS is a novel method for performing NAS in a federated learning setting, which outperforms similar existing methods. It is well-written and easy to understand, but some of the experiments are considered somewhat unrealistic, and analyses are lacking. The authors do not provide any code for reproducibility.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for federated neural architecture search, namely FedorAS, which leverages supernet for sharing knowledge across clients with device and data heterogeneity. They also introduce OPA (operation aggregation) that weighs the client updates in a frequency-aware manner. They demonstrate the effectiveness of their method against multiple existing methods.
strength_and_weaknesses: **Strength**
- The paper is well-organized and easy to read.
- They tackle the practical problem of federated learning where participants have device heterogeneity, i.e. computation and memory budgets.
- They demonstrate the effectiveness of the proposed method in various modalities, i.e. vision, speech, text.
- The results are impressive.

**Weaknesses**
- It seems better to discuss the overall costs (i.e. supernet train and model search time (ms) or # rounds, etc) and compare them to the baseline models. Since the proposed framework has three steps; (1) supernet training, (2) model search, and (3) local fine-tuning, each step may take the individual computation and communication costs. In the paper, however, I can only find the analysis for the searched models obtained from the step 3 only. More detailed discussion for overall costs is required, for example:
    - How many rounds do the stage (1) and stage (3) need? 
    - How long does the (2) model search process take? 
    - How much overall computation and memory costs are required to get the final results for your model and baseline models?
- Is the number of “tiers” the hyper-parameters? Can we have more tiers? Does varying the number of tiers affect the performance? or the overall costs for each step? 
- HeteroFL [Diao et al 21], one of the essential works tackling the same problem, is missing. The method also leverages the supernet for sampling local models considering the complexity levels, which is also similar to your tier clustering. It would be very helpful for you to discuss pros and cons and compare it with the proposed model.

Diao et al, HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients, ICLR 2021

clarity,_quality,_novelty_and_reproducibility: Overall quality and clarity of the paper are good. There are some rooms for improving the originality of the proposed method. For example, discussion of the clear difference from some existing methods that utilizes the supernet for Federated NAS (i.e. HeteroFL). The code is not provided, thus it is hard to tell the reproducibility of this work.

summary_of_the_review: I enjoyed reading the paper, but several improvement seems to be required, as mentioned above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a federated NAS framework targeting cross-device scenarios, which focuses on the device heterogeneity and communication efficiency. Specifically, the proposed FedorAS framework trains the supernet via federated learning across multiple devices, where each device performs budget-aware path sampling and trains the subnetwork on the local data, which are aggregated by considering the number of updates to each operation per client. The proposed FedorAS framework is validated on multiple benchmarks with both IID and non-IID partitioning, and is shown to largely outperform the compared baselines (e.g. FedNAS, FjORD). 

The paper initially received split reviews, with two leaning toward acceptance and two others leaning toward rejection. The reviewers found the paper well-written, the method clear, the experimental validation extensive, and the performance of the proposed method impressive. However, the reviewers were also concerned about the following factors:
- Limited novelty of FedorAS over existing federated NAS frameworks, such as FedNAS, or heterogenous FL frameworks, such as HeteroFL, since neither the idea of learning NAS frameworks in an FL framework, or federated learning of a supernet, is novel. The main novelty thus reduces down to the proposal of an efficient and device-aware federated NAS method.
- There is hidden cost in the supernet training as well as the finetuning, that is not reported in the submitted version of the paper. Also, the reduction in the communication cost by sending 25% or 50% of the supernet may not be that large if the original network is large. 
- The "tiers" and the number of tiers are manually defined and the experimental setting does not consider the scenario where there exists large difference in the computing power among the devices.

During the discussion phase, the authors provided detailed responses to the reviewers with additional experimental results. However, the paper still fell into the borderline case even after the rebuttal, and thus the reviewers had in-depth discussion over zoom. From the zoom meeting, the reviewers agreed that the paper presents promising results and the method thus may have some practical impact. However, they considered the consideration of cross-device setting as incremental over existing fedNAS frameworks, and most importantly, they were concerned with the hidden cost and insufficient reduction in the communication cost. These are critical weaknesses since they may go against the main argument that the proposed method is more efficient than the baselines, in terms of the overall training and communication cost. Thus, the main strength of the model now comes from its good performance, but the reviewers agreed that there is little analysis of how the proposed method could obtain such impressive performance.

In sum, while the paper proposes a potentially promising framework for cross-device federated learning problems, its main argument that the proposed method is more efficient is not properly validated, and the effectiveness of the proposed method is not fully analyzed. Thus, the paper could be accepted if there is a room, but will clearly benefit from another round of revision, if that is not the case.
justification_for_why_not_higher_score: The proposed method is not fully validated for its efficiency, which is claimed to be its main strength.
Decision: Reject