FedNAS: Federated Deep Learning via Neural Architecture Search
Authors: Chaoyang He
Erum Mushtaq
Jie Ding
Salman Avestimehr
Venue: ICLR 2022 Submitted (2022)

Abstract: Federated Learning (FL) is an effective learning framework used when data cannotbe centralized due to privacy, communication costs, and regulatory restrictions.While there have been many algorithmic advances in FL, significantly less effort hasbeen made on model development, and most works in FL employ predefined modelarchitectures discovered in the centralized environment. However, these predefinedarchitectures may not be the optimal choice for the FL setting since the user datadistribution at FL users is often non-identical and independent distribution (non-IID). This well-known challenge in FL has often been studied at the optimizationlayer.  Instead, we advocate for a different (and complementary) approach.  Wepropose Federated Neural Architecture Search (FedNAS) for automating the modeldesign process in FL. More specifically, FedNAS enables scattered workers tosearch for a better architecture in a collaborative fashion to achieve higher accuracy. Beyond automating and improving FL model design, FedNAS also provides anew paradigm for personalized FL via customizing not only the model weightsbut also the neural architecture of each user. As such, we also compare FedNASwith representative personalized FL methods, including perFedAvg (based on meta-learning), Ditto (bi-level optimization), and local fine-tuning. Our experiments ona non-IID dataset show that the architecture searched by FedNAS can outperformthe manually predefined architecture as well as existing personalized FL methods.To facilitate further research and real-world deployment, we also build a realisticdistributed training system for FedNAS, which will be publicly available andmaintained regularly. 

Under review as a conference paper at ICLR 2022
FEDNAS: FEDERATED DEEP LEARNING VIA NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) is an effective learning framework used when data cannot be centralized due to privacy, communication costs, and regulatory restrictions. While there have been many algorithmic advances in FL, significantly less effort has been made on model development, and most works in FL employ predefined model architectures discovered the centralized environment. However, these predefined architectures may not be the optimal choice for the FL setting since the user data distribution at FL users is often non-identical and independent distribution (non- IID). This well-known challenge in FL has often been studied at the optimization layer. Instead, we advocate for a different (and complementary) approach. We propose Federated Neural Architecture Search (FedNAS) for automating the model design process in FL. More specifically, FedNAS enables scattered workers to search for a better architecture in a collaborative fashion achieve higher accuracy Beyond automating and improving FL model design, FedNAS also provides a new paradigm for personalized FL via customizing not only the model weights but also the neural architecture of each user. As such, we also compare FedNAS with representative personalized FI. methods, including perFedAvg (based on meta- learning), Ditto (bi-level optimization), and local fine-tuning. Our experiments on a non-llD dataset show that the architecture searched by FedNAS can outperform the manually predefined architecture as well as existing personalized FL methods. To facilitate further research and real-world deployment, we also build a realistic distributed training system for FedNAS, which will be publicly available and maintained regularly.
INTRODUCTION
Federated Learning (FL) is promising approach for decentralized machine learning, which aims to avoid data sharing and lower the communication cost (McMahan et al., 2016). As such, it has gained a lot of attention in various domains of machine learning such as computer vision, natural language processing, and data mining. Despite its widespread popularity, data heterogeneity. Since users' data are not identically or independently distributed (non-IID) in nature, globally learned model may not perform optimally all user devices. When interweaving with data heterogeneity, data invisibility is another issue that has rarely been studied. For this reason, find better model architecture with higher accuracy, developers must design or choose multiple architectures, then tune hyperparameters remotely to fit the scattered data. This process is extremely expensive because attempting many rounds of training on edge devices results in remarkably higher communication cost and on-device computational burden than the data center environment.
of the key challenges of FL is
To mitigate the challenge of data heterogeneity, researchers have proposed methods train a global model, including F FedP rox Li et al. (2018), FedNova Wang et al. (2020), and FedOPT T Reddi et all (2020). Additionally, personalized frameworks such as Ditto Li et (2020), pFedMe Dinh et al. (2020), and PerFedAvg Fallah et al. (2020) have been recently developed to optimize personalized models to adapt to individual user's data. These prior works have made remarkable progress in designing optimization schemes for pre-defined model architectures operated at pure optimization. However, these algorithms all require lots of effort to tune hyperparameters: this attributed to their strong prior assumptions, which may not always match the unknown data distribution. For example, practitioners must tune the regularization parameter in Ditto Li et al. (2020) and PFedMe Dinh et al. (2020) to find proper correlation between the aggregated global model and local model. Moreover,

---

Under review as a conference paper at ICLR 2022
Step
Step 3
network weights
Step 2
coay
Step 4
pooling
0: architecture parameters.
Figure 1: Illustration of Federated Neural Architecture Search (step I: search locally; step 2: send the gradients of a and w to the server; step 3: merge gradients to get global a and w: step 4: synchronize the updated a and w to each client.)
their design is only in optimization level and does not consider the efficacy of model selection and neural architecture design, leading to a suboptimal solution when using a pre-defined model.
We aim to address data heterogeneity in FL via different and complementary approach that is based on model personalization through neural architecture search (NAS). NAS has recently gained much momentum adapt heterogeneity in neural architecture design Smithson et al. (2016); Chen et al. (2020), latency Wu et al. (2019); Tan et al. (2019); Cai et al. (2019), memory footprint Cai et al. (2018); Marchisio et al. (2020), energy consumption Hsu et al. (2018); Yang et al. (2020) for edge devices. NAS methods are often categorized into three types: gradient-based methods Liu et al. (2018), evolutionary methods Liu al. (2020), and reinforcement learning (RL)-based methods Jaafra et al. (2019). Among these, gradient- based methods are the most efficient as they can finish searching in only a few hours, compared to thousands of GPU days with other methods.
In this work, to search for a personalized neural architecture for mitigating the data heterogeneity, we adopt an improved variant of the gradient- based method, MiLeNAS He et al. (2020c), which is computationally tractable and particularly suitable for constrrec edge devices. Particularly, we propose a new method named Federated NAS (FedNAS) to search model architectures among edge devices collaboratively. As shown in Figure 1, FedNAS works in the following way. We first utilize the MiLeNAS He et al. (2020c) as a local searcher on each client's 's local data, which can be distributed easily and efficiently in search time (Step 1). Formally, it formulates NAS mixed-level problem: w = w 7/u Le a), a = a - Ja aia (w, a)) + AV av (w, a)), where represents the network weight and represents the neural architecture. Ler(w, a.) and Leal (to, a:) denote the loss with respect to training data and validation data, respectively. After the local search, each client then transmits weights and architecture then applies a weighted aggregation to obtain the server-side a and w (Step 3) and sends the updated parameters back to each client for the next round of searching (Step 4). During the searching process, we can personalize the and w parameters by alternative local adaptation. Such personalization method can either obtain higher accuracy for various data distributions, or automate the training process with lightweight papre-parrmeter searching efforts.
the server (Step 2). The server
We evaluate FedNAS comprehensively in curated non-L..D. datasets, including CIFAR-10 and GLD-23K. Our datasets cover both global model training and personalized model training. We also consider different training scenarios: cross-silo FL and cross-device FL., which has a different total number of clients and number of clients per round. We demonstrate that the personalized model architectures learned by the individual clients perform better than the fine-tuned FedAvg and other representative personalized FL methods such Ditto Li et al. (2020) and perFedArer Fallah et al. (2020) with default syper-parrmeten in most settings.
In summary, main contributions in this paper are three-fold.
1. We propose the FedNAS method to search for both global model and personalized model architectures collaboratively among edge devices and show its satisfying performance in a variety of FL settings.
2

---

Under review as a conference paper at ICLR 2022
2. We investigate the role of NAS to address the challenge of data- hettreeetrrog in FL and demonstrate via experimental results that can adapt to users' data better than existing local adaptation and personalization schemes.
3. We experimentally show that FedNAS can achieve state-of-the-art performance for both cross-silo and cross-device settings.
PROPOSED METHOD
2.1 PROBLEM DEFINITION
In the federated learning setting. there are K nodes in the network. Each node has a dataset D& := { {x y:) |-= which is non-IID, When collaboratively training deep neural network (DNN) model with nodes, the objective function defined as:
min f(w, fized
def
min
Elxi, 3i: w. RED
Σ
(1)
N
N
R=
fized
where w represents the network weight, a determines the neural architecture, and f the loss function of the DNN model. To minimize the objective function above, previous works choose fixed model architecture then design variant optimization techniques to train the model w.
We propose to optimize the federated learning problem from completely different angle, optimizing w and a simultaneously. Formally, we can reformulate the objective function as:
min f(w, O+) W
min
N
1
Σ E( Vsi a))
N
iED
In other words, for the non-IID dataset scattered across many workers, our goal is search for an optimal architecture and related model parameters to fit the dataset more effectively thus achieve better model performance. this work, we consider searching for CNN architecture to improve the performance of the image classification task.
2.2 SEARCH SPACE
Normal Cell N
Reduction Cell
Normal Call x
Reduction Cell
Normal Cell N
softmax
conv
30
S5
S6
S2
35
S6
S1
normal cell
a. reduction cell
Figure 2: Search Space
the search space definition, the search algorithm, and the performance estimation strategy Hutter et al. (2019). Our search space follows the mixed-operation search space defined in DARTS Liu et al. (2018) and MiLeNAS He et al. (2020c), where we search in two shared convolutional cells and then build it as an entire model architecture (as shown Figure 2). Inside the cell, to relax the categorical candidate operations between two nodes (e.g., convolution, max pooling, skip connection, zero) to continuous search space, mixed operation using softmax over all possible operations proposed:
Normally, NAS includes three consecutive components:
exp(a (4.5)
W= exp(a (6J) (x(2)
(3)
where the weight pş of the mixed operation ((+)) (Ir) for pair of nodes (i.j) is parameterized by vector inj Thus, all architecture operation options inside a network (model) can be parameterized More details introduced in Appendix A.1.1.

---

Under review as a conference paper at ICLR 2022
2.3 LOCAL SEARCH
Following the above-mentioned search space, each worker searches locally by utilizing the mixed- level optimization technique MiLeNAS He et al. (2020c):
Tw V (W,
(4)
a
a Tex VaLur a) + AV AVaaval(W,))
where (ir(w, ar) and (w, (x) denote the loss with respect to the local training data and validation data with and a, respectively.
2.4 FEDNAS: FEDERATED NEURAL ARCHITECTURE SEARCH
Algorithm FedNAS Algorithm.
Initialization: initialize wo and exo: clients are selected indexed by k; E is the number local epochs; 7' is the number of rounds.
2: Server executes:
3: for each round t 1,2, 1'.
each client in parallel
1
0++1 +- Clientl LeaalSearch(k,, Wt, o4)
W1-1
w
7:
ar+1
0++1
9: CienttooclSearchi/ w. a.:: // Run on client k
10:
for e in epoch do
11:
for minibatch in training and validation data do
Update = The Lur Update a (V.Cu(w, a) + AV (w,
13:
14:
15:
return w, server
We propose FedNAS, a distributed neural architecture search algorithm that aims optimizing the objective function in Equation 2 under the FL setting. We introduce FedNAS corresponding to four steps in Figure/ 1: 1) The local searching process: each worker optimizes a and simultaneously using Eq. 4 for several epochs; 2) All clients send their and w to the server; 3) The central server aggregates these gradients as follows:
K
N w
W++1
N
t+1
(5)
N k
0.+11
N
a+++
4) The server sends back the updated a and to clients, and each client updates its local and accordingly, before running the next round of searching. This process is summarized in Algorithm 1. After searching, additional evaluation stage is conducted by using traditional federated optimization method such as FedAvg McMahan et al. (2016).
2.5 PERSONALIZED FEDNAS: ALTERNATIVE LOCAL ADAPTATION
Local Adaptation. To personalize local models, we fine-tune the received global model locally. Such local fine-tuning follows Equation 4, meaning that each client alternatively optimizes its local architecture a and model weights w. We found from experiments that such fine-tuning can make local model more robust against local data heterogeneity, compared with fine-tuning and local adaptation based on predefined model and ofate-of-the-ar personalized optimization methods (e.g., Ditto Li et al. (2020) and perFedAvg Fallah al. (2020)).
Robust to Varying Data Heterogeneity and Training Scenarios. Additional to the benefit of data heterogeneity with personalization, essential feature of FedNAS is that it does require many
4

---

Under review as a conference paper at ICLR 2022
rounds of hyperparameter searching adapt to diverse data distributions. Most of the time, the default hyper-paramen already perform very well. This property of FedNAS attributed to three aspects described below.
Intuitively, the personalized architecture and weights have additive effect in adapting data heterogeneity, compared with solely personalizing the model weight, especially when the architecture search space is huge.
Most of the personalized methods are built based optimization framework with strong prior assumptions, which may not always match the unknown data distribution. For example, Ditto Li et al. (2020) and pFedMe Dinh et al. (2020) utilizes bi-level optimization and correlate the relationship of aggregated global model and local model by a regularization- based method. Practitioners have tune the À value make it work manually. Although perFedAvg Fallah et al. (2020) brings the idea of meta- meeta-lerrnnng to adapt to data heterogeneity, it is difficult for practitioners decide the boundary of its meta-train phase and meta-test phase when the data distribution is unknown.
Different training scenarios also bring additional randomness and uncertainty. For example, in the cross-device setting, the total client number and the client number per round differs from the cross-silo setting significantly, which further increases the difficulty of model selection and hyper-parameter tuning. FedNAS may be more resilient to this uncertainty in practice.
To verify the advantage of FedNAS, we run experiments to search for both personalized and global models on cross-silo and cross-device settings (see Section 3.1 and 3.2).
2.6 AUTOFL SYSTEM DESIGN
ServerManager
ClientManager
(Aggregation, Synchronization, Global Statistics
(Local Search, Synchronization)
ComManager
Aggregator
Trainer
Send Thread
Receive Thread
Deep Neural Networks (MiLeNAS, DenseNet, etc)
Abstract Communication Layer
On-Client Learning Framework
MPI (Message Passing Interface)
PyTorch
Communication Protocol Component
On-Device Deep Learning Component
Figure 3: Abstract System Architecture of AutoFI.
We design an AutoFL system using FedNAS based on F FedML He et al. (2020b), an open-source research library for federated learning. The system architecture is shown in Figure 3. This design sep- arates the communication and the model training into two core components shared by the server and clients. The first is the communication protocol component responsible for low-level communication among the server and clients. The second is the on-device deep learning component, which is built based on the popular deep learning framework Py Torch. These two components are encapsulated as ComManager, Trainer, and Aggregator, providing high-level APIs for the above layers. With the help of these APIs, in ClientManager, the client can train or search for better architectures and then send its results the server-side. In contrast, in ServerManager, the server aggregate and synchronize the model architecture and the model parameters with the client-side. More details of the system design can be found in the Appendix.
5

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS AND RESULTS
In this section, we will introduce the experimental results of FedNAS to train global model as well as personalized models. All our experiments are based on non-IID data distribution among users. In experiments, explore two types of non-lID data distributions, label skewed and latent Dirichlet allocation (LDA), which are well explored literature in FL settings Yurochkin et al. (2019), He et al. (2020a), Arivazhagan et al. (2019).
Implementation and Deployment. We set up our experiment distributed computing network equipped with GPUs. We perform experiments for two settings, FedNAS for global model search and FedNAS for personalized models search. For investigating the former setting, we set up our experiment cross-silo setting for simplicity and use 17 nodes in total, one representing the server-side and the other 16 nodes representing clients, which can be organizations (e.g,, hospitals and clinies). For personalized model search, we use larger set of nodes, 21, in total, one representing the server-side and the other 20 nodes representing clients. We pick four clients random for each round of FedNAS. For all these experiments, each node is a physical server that has an NVIDIA RTX 2080Ti GPU card inside. We deployed the FedNAS system described in Appendix 2.6 on each node. Our code implementation is based on Py'Torch 1. 1.4.0, MPI4Py 3.0.3, and Python 3.7.4.
the real world
Task and Dataset. Our training task is image classification on the CIFARIO dataset, which consists of 60000 32x32 color images in 10 classes, with 6000 images per class. For global model searching via FedNAS, we generate non-IID (non identical and independent distribution) local data by splitting the 50000 training images into K clients in unbalanced manner: sampling Pa Dir and allocating a Pc,k proportion of the training samples of class c to local client k. The 10000 test images used for global test after the aggregation of each round. The actual data distribution used for this experiment is given in Table 3.
Min 30C0, Plak 3030
labe d dttribution
.000
25000
1ea1 Ping 14044800000
2000
1000
0110
0.75
2000
0359
300
0823
20.0
0003
(a) Image Allocation per Client
(b) Label Allocation per Client
Figure 4: CIFAR10: Label Skew Partition
For personalized model experiments, we generate non-llD data by label skewed partition. In this partition scheme, we assign images only five classes to each client and keep the number of images per client the same, namely 3000, as shown in Figure 4. For each client, we further split these 3000 images into the training and testing datasets by using 75% data, i.e., 2250 images, for training, and personalization it requires each client to
the other 25% testing data. We perform this split have their own local test dataset. We also explore latent Dirichlet distribution (LDA) based non-III) data distribution for the personalized model setup, and details of this distribution for this experiment can be found in the appendix 1.1.2. Since the model performance is sensitive to the data distribution, we fix the non-IID dataset in all experiments for fair comparison.
4py/

---

Under review as a conference paper at ICLR 2022
3.1 PERSONALIZED MODELS SEARCH VIA FEDNAS
To demonstrate the efficacy of FedNAS to design better local models, we compare FedNAS with local adaption (via FedAvg), Ditto and perFedAvg. Besides FedNAS, every other method runs on manually designed architecture, ResNet18 Targ et al. (2016), which has more model parameters, 1 11M, than the 8-layer DARTs cell structure of FedNAS which has only 4M model parameters He et al. (2020c). To evaluate the performance, we use average validation accuracy of all clients as performance metric.
3.1.1 RESULTS ON NON-II,,,, (LABEL SKEW PARTITION AND LDA DISTRIBUTION)
Table illustrates the performance comparison of FedÑAS with local adaptat inn, Ditto and perFedAvg. For fair comparison, we fine-tune syper-parameters of each method, such fine-tune learning rate (Ir) hyperparameter over the set {0.1, 0.3, 0.01, 0.03, 0.001, 0.003) of each method. Batch size has been fixed 32 for all these comparisons. For Ditto, in addition to Ir, tune À over the set of (2. 1, 0.1, 0.01, 0.001 ). For perFedAvg. we tune the glohal Ir over {0.1, 1, 0.3, 0.01, 0.03, 0.001, 0.003) by keeping the local Ir {1, 3, 5, 7, 10) times higher than the global Ir.
Table I draws the comparison between different methods for the average validation accuracy of all the clients metric for Ida and label skew distribution. Interestingly, FedNAS outperforms all other methods for both label skew and Ida distribution which highlights its power to adapt to user's 's data well and perform better locally as well. Overall, for label skew distribution, it achieves average validation accuracy of 91.3% which is 5% higher than the local adaptation's 's validation accuracy and 2% higher than Di titto. We also observe that D to outperforms the local adaptaion in terms of validation accuracy but have higher standard deviation than local adaptation.
Table 1: Average local validation Accuracy Comparison of FedNAS with other personalization techniques)
Method
Parameter size
Accuracy (Label Skew)
Accuracy (Ida Distribution)
FedNAS
4M
0.913#0.025
0.907+0.024
Local Adaptation
11M
0.86420.028
0.861#0.0357
Ditto
I IIM
0.894+0.035
0.88 #0.032
perFedAvg
1 I1
0.888+0.036
0.894=0.032
uod
VUP ramee UsS
1500
(a) Local Validation Accuracy
(b) Average Validation Accuracy Distribution
(c) Average Validation Accuracy
Improvement
Figure 5: Visualization of validation accuracy of each client and accuracy improvement distribution for personalized model search.
For detailed comparison, we compare the validation accuracies of all the clients for the best round of label skew distribution. The best round for each method is selected the round that provides the highest average validation accuracy. We visualize the validation accuracy of each client 5(a), average validation accuracy distribution 5(b) and average validation improvement distribution 5(c). For accuracy improvement distribution, we subtract the validation accuracy of F FedNAS from the respective method for each client and plot the histogram. The improvement histogram shows that for
7

---

Under review as a conference paper at ICLR 2022
one of the clients the improvement can be high 15% compare perFedAvg. As compare to Ditto, we get 2.5% improvement for even clients. On the other hand, there are only two clients, client number 10 and 20, for which local adaptation performs slightly (2.5%) better. Although there are some clients for which FedNAS do not perform well as compared to these methods, is important note that the standard deviation of FedNAS more prominent in figure (b) is lowest and accuracy histograms are concentrated towards the right side, whereas for other methods, these bars fall as below 82% accuracy.
3.2 GLOBAL MODEL SEARCH VIA FEDNAS
To investigate the performance of FedNAS to design global model, we search a global model via FedNAS and compare the well-known FL. algorithm FedAvg, which runs on DenseNet Huang et al. (2017), manually designed architecture that extends ResNet He et al. (2016) but with higher performance and fewer model parameters. We run both of these experiments on the same non-IID dataset.
3.2.1
RESULTS ON NON-LL.D, (LDA PARTITION)
0.8
0.8
0.8124
0.7778
0.6
0.6
0.4
0.2
Round
Round
20
60
100
10
30
40
(a) FedAvg
(b) FedNas
Figure 6: Test Accuracy on Non-IID Dataset (multiple runs) FedAvg on DenseNet vs. FedNAS
Figure 6 demonstrates the performance of FedNAS vs. FedAvg. use a specific non-IID data distribution given in appendix 1.2 and keep it fixed for both experiments. For fair comparison, results are obtained by fine-tuning byperparameters of each method, and each method is run three times. Details of hyperparameter tunning be found in the appendix A.1.4.
Figure 6(a) shows the global test accuracy during the training process of FedAvg, whereas, Figure 6(b) reports the global test accuracy during the searching process of FedNAS. Global test accuracy has been calculated using the 10000 test images CIFAR 10 dataset. First of all, we demonstrate the compatibility of NAS for the data-heterogenneous FL setting. addition convergence of FedNAs, we show that FedNAS can achieve higher accuracy than FedAvg during the searching process (81.24% in Figure 6(b); 77.78% in Figure 6(a)). This 4% performance benefit further confirms the efficacy of F FedNAS. We also evaluate the searched architecture under this data distribution. We found that each run of FedNAS can obtain a higher test accuracy than each run of FedAvg. On average, the architecture searched by FedNAS obtains a test accuracy 4% higher than FedAvg.
Hyperparameters and visualization of the searched architecture can be found Appendix A.1.4 and 1.1.5, respectively.
Remark. We also ran experiments on other distributions of non-Ill datasets, in which FedNAS is also demonstrated to beat FedAvg, confirming that FedNAS searches for better architectures with a higher model performance.
3.3 EVALUATION OF THE SYSTEM EFFICIENCY
In order to comprehensively reflect distributed search overhead, developed the single- process and distributed version of FedÑAS and FedAvg. The single-procee version simulates the algorithm by performing client-by-client search single GPU card. As shown in Table
8

---

Under review as a conference paper at ICLR 2022
Table 2: Efficiency Comparison (16 RTX2080Ti GPUs clients, and I RTX2080Ti as server)
Method
Search Time
Parameter Size
Hyperparameter
FedAvg (single)
> days
rounds 100, local epochs=20, batch size=64
FedAvg (distributed)
12 hours
20.01M
FedNAS (single)
33 hours
rounds = 50, local epochs=5, hatch size=644
FedNAS (distributed)
5 hours
1.93M
compared with FedAvg and manually designed DenseNet, FedNAS can find better architecture with fewer parameters in less time. FedAvg spends more time because requires more local epochs to converge.
4 RELATED WORKS
Recently, Neural Architecture Search (NAS) Hutteret al. (2019) has attracted widespread attention due to its advantages over manually designed models. There are three major NAS methods: evolutionary algorithms, reinforcement learning-based methods, and gradient-based methods He et al. (2020c). While the Federated Learning (FL) domain McMahan et al. (2016); He et al. (2019), using pre- designed model architectures and optimizing by FedAvg McMahan et al. (2016) is the main method to improve model performance. To our knowledge, NAS rarely studied in FL setting to study the aspect of personal model search for real-time setting. Although Kairouz et al. (2019) first proposed the concept of automating FL. via NAS, the concrete method and details are never given. There are few works done in the direction of the use of NAS search for global model, however, no personalization exploration is provided.
For global model search, Zhu & Jin (2020) is a NAS based FL work that exploits evolutionary NAS to design master model but they utilize double-client sampling to make their method edge resource friendly. Contrary to this, we exploit gradient- based NAS method, Mi leNAS, which is comparatively faster and more resource friendly than the evolutionary and reinforcement-biee based methods. The other work in this direction Singh et al. (2020) that explores the concept of differential privacy by using DARTs as NAS solver to search for a global model. However, our proposed work uses MileNAS solver which has an extensive analysis of its performance efficiency over DARTS given in the original MileNAS work He et al. (2020c). Another work Garg et al. (2020) uses DSNAS which is another gradient based NAS algorithm to search for a global model. DSNAS works on sampling a child network from supernetwork even in search phase whereas MileNAS solver searches over the complete supernetwork, and therefore, has the potential to provide more freedom to clients to search for better and personalized architecture. Another work Xu et al. (2020) proposes a very different idea than conventional neural architecture search where they begin with a pretrained manually designed model and keep pruning the model until it satisfies the efficiency budget. Although they named their work as federated neural architecture search but model search is performed on the server side alone, none of the clients participate in searching a better model (finding architecture parameters). Clients only participate in training the pruned model's parameters communicated to clients by server. Furthermore, to the best our knowledge, we are the first work that investigate the performance of locally searched architectures in federated NAS.
5 CONCLUSION
In this paper, we propose FedNAS, a unified neural architecture search based federated learning framework to design a global model collaboratively. First, we study the compliance of gradient based neural architecture search algorithm, MileNAS, with the FedAvg algorithm. We analyze its performance for both cross silo and cross-device settings, and show its convergence for both setups. We also investigate the proposed framework, FedNAS, from the perspective of personalization and its role to overcome the challenge of hata-heterogeeeitty in FL. To test heteroggetro we explore FedNAS for both label-skewed and Ida-based non-IlI data distributions, and show via experimental results its superiority over other personalization methods, such as local fine-tunning, Ditto and pe perFedAvg.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Feder- ated learning with personalization layers. arXiv preprint arXiv:1912.00811. 2019.
Han Cai, Ligeng Zhu, and Song Han, Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332. 2018.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize for efficient deployment. arXiv preprint urXiv:/908. 1909,9979, 2019.
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou, Adabert: Task-adaptive bert compression with differentiable neural architecture search. arXiv preprint arXiv:2001.04246, 2020.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. arXiv preprint arXiv:2006.08844, 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.07948, 2020.
Anubhav Garg, Amit Kumar Saha, and Debo Dutta. Direct federated neural architecture search.
arXiv preprint arXhv:2010.06223. 2020.
Chaoyang He, Conghui Tan, Hanlin Tang, Shuang Qiu, and Ji Liu. Central server free federated learning over single-sided trust social networks. arXiv preprint arXiv: 04956, 2019.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.1355 13518, 2020a.
Chaoyang He, Songze Li, Jinhyun So. Mi Zhang, Hongyi Wang. Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang. Murali Annavaram, and Salman Avestimehr. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13512 2020b.
Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020c.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Chi-Hung Hsu, Shu-Huan Chang, Jhao-Hong Liang, Hsin-Ping Chou, Chun-Hao Liu, Shih-Chieh Chang. Jia-Yu Yu Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv: 10332, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1700-47088 2017.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automatic Machine Learning: Methods, Systems, Challenges. Springer, 2019.
Yesmina Jaafra, Jean Lue Laurent, Aline Deruyver, and Mohamed Saber Naceur. Reinforcement learning for neural architecture search: A review. Image and Vision Computing, 89:57-66, 2019.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning- arXiv preprint arXiv: (0xiv::9/220497770 2019.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, arXiv preprint arXiv: /8/2.06//22.2 2018
10

---

Under review as a conference paper at ICLR 2022
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. arXiv: 2012.04221, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv: 180.:09086000 2018.
Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, and Gary Yen. A survey on evolutionary neural architecture search. arXiv preprint arXiv:2008. 10937, 2020.
Alberto Marchisio, Andrea Massa, Vojtech Mrazek, Beatrice Bussolino, Maurizio Martina, and Muhammad Shafique. Nascaps: A framework for neural architecture search to optimize the accu- racy and hardware efficiency of convolutional capsule networks. In 2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD), pp- 1-9. IEEE, 2020.
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication--fficcii learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629 2016.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003. 00295, 2020.
Ishika Singh, Haoyi Zhou, Kunlin Yang, Meng Ding, Bill Lin, and Pengtao Xie. Differentially-prii federated neural architecture search. arXiv preprint arXiv:2006. 2020.
Sean C Smithson, Guang Yang, Warren J Gross, and Brett H Meyer. Neural networks designing neural networks: multi-objective hyper-parameter optimization. In Proceedings of the 35th International Conference on Computer-Aide -ided Design, pp. 1-8, 2016.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Sasha Targ, Diogo Almeida, and Kevin Lyman. Resnet in resnet: Generalizing residual architectures. arXiv preprint arXiv: 1603.000000 2016.
Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint orXiv:2007.07444 2020.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware- efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 00734-10742. 2019.
Mengwei Xu, Yuxin Zhao, Kaigui Bian, Gang Huang, Qiaozhu Mei, and Xuanzhe Liu, Federated neural architecture search. arXiv preprint arXiv:2002.06332, 2020.
Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration of neural architectures and heterogeneous asic accel- erator designs targeting multiple tasks. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2020.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Hangyu Zhu and Yaochu Jin. Real-time federated evolutionary neural architecture search. arXiv preprint arXiv:2003.00 02793, 2020.
11


Reviews:
[{"review_id": "tP5l5Cq8IYU", "reviewer": "Reviewer_sG5T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The paper propose the Federated Neural Architecture Search (FedNAS) method to search for both global model and personalized model architectures collaboratively among edge devices, and show its performance in a variety of federated learning  settings. \n2. It also investigate the role of NAS to address the challenge of data-heterogeneity in federated learning and show superiority in results as well as its efficacy. \n\n\nWeakness: \n1. It was not clear the purpose of Figure 4 (1) given that the graph shows uniform local training dataset size. \n2. It seems that the paper is missing some strong baseline in NAS spaces. In related works, authors lists our some work in NAS, such as Kairouz et al,, 2019, Zhu & Jin 2020, etc.. The paper would be stronger to also benchmark against related methods in NAS. ", "summary_of_the_paper": "The paper propose the Federated Neural Architecture Search (FedNAS) method to search for both global model and personalized model architectures collaboratively among edge devices and show its performance in a variety of federated learning  settings. It also investigate the role of NAS to address the challenge of data-heterogeneity in federated learning and show superiority in results. ", "summary_of_the_review": "Overall technical seems significant and somewhat new, but as mentioned above in the weakness, the paper is missing some comparison against stronger baselines, especially for previous methods in NAS. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q0NpsArO-p", "reviewer": "Reviewer_y4xh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The authors address the data heterogeneity problems in federated learning via NAS methods. The personalized architecture shows good improvements. \n\n\nWeakness:\n1. Originality: This work is a simple extension of MiLeNAS on federated learning. The NAS on clients for personalization seems different from other NAS works. But I did not see any challenge highlighted by authors and they directly adopt NAS methods for common scenarios. \n2. Quality: The aggregation on the central server follows the similar way of FedAvg. The most difference with other existing works is to introduce NAS into federated learning. However, during employing NAS into federated learning, only trivial changes are made. When architecture changes, the global aggregation still remains the similar way in traditional federated learning and leaves some unexplored spaces. Even though MiLeNAS is a recent effective method, other NAS methods should be introduced into federated learning to justify the authors' choice.  \n3. Soundness: One of the most important parts of federated learning especially in new settings is convergence guarantee. The architectures on clients keep changing and personalization is needed, resulting in more heterogeneous model weights and architectures. How to guarantee the convergence of the proposed framework? I will suggest authors provide some theoretical proof to support this method for convergence. ", "summary_of_the_paper": "The authors employ an existing neural architecture search method in the federated learning setting.  Specifically, the authors propose FeNAS and extend an existing NAS method MiLeNAS into federated learning to address the data heterogeneity problem and conduct personalization.  The experiments show that the proposed method is able to achieve improvement compared to some other federated learning methods. ", "summary_of_the_review": "The authors extend an existing NAS method MiLeNAS on federated personalization. However, the novelty of the work is somewhat limited.  It is lacking clear support evidence and insights to support why MiLeNAS was chosen. To support the idea of NAS for federated learning, more other NAS methods are preferred to provide more insights and depict the challenges of this problem. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b2d5dfse5ND", "reviewer": "Reviewer_xdiq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The global model search with FedNAS is better is not surprising, since there is no difference between training on FL and centralized settings, on only gradients. What becomes really interesting, is the personalization performance. Comparing to existing personalization methods which fix network structure and personalize weights only, FedNAS optimizes in both neural architecture and weights, which results in better personalized performance (both by average and by individual), and smaller model. \n\nContributions:\n\nThe paper's work is a natural extension of MileNAS. Since MileNAS, like DARTS, is gradient-based algorithm, then make it available to Federated Learning is natural. In this sense, the innovation is not extraordinary. \n\nThere are some minors to be improved:\n\n(1) Experiments are not sufficient. Although for CIFAR10 and GLD-23k, the results are pretty extensive, more experiments on other FL datasets (EMNIST-62, CIFAR100, and even Shakespeare), could make the paper stronger. \n\n(2) Moreover, FedNAS search architecture only applies for Computer Vision (specifically, image classification problems). How to extend NAS to NLP tasks or other paradigms, could be an interesting next step.\n\n(3) Interpretation of the learned architecture. The performance gain originates from personalize both architecture and weights. The author confirms that the personalized performance does improve, but the interpretation is missing. It could be interesting, to see what if some clients learns less/more complex models.\n", "summary_of_the_paper": "This paper combines gradient-based NAS (DARTS-like algorithm: MileNAS) with Federated Learning (FL) setup, to improve both global and personalization performance with learned neural architecture. Since both NAS and FL learning are based on gradient, the extension to FL setup becomes intuitive and effective. Empirically FedNAS shows improved performance comparing to existing FL methods. ", "summary_of_the_review": "Overall, the proposed method to use gradient-based NAS for FL to improve personalization, shows good empirical performance. The complaint of mine, is that the innovation is on applying existing method (MileNAS, or DARTS-like gradient-based NAS), directly onto FL settings. There are very few modification on original method, other than the settings are different.\n\nIn all, I give this paper score 6, and I am willing to discuss with other reviewers and the authors in later rebuttal session.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RASmV9ybQp7", "reviewer": "Reviewer_3t92", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper is generally clearly written.\n2. The codes are open-sourced for reproducibility.\n\nWeakness:\n1. The motivation is not clear. The abstract says \"We propose Federated Neural Architecture Search (FedNAS) for automating the model design process in FL.\" and the introduction says \"We aim to address data heterogeneity in FL via ... NAS.\" For me, it is not clear whether the paper aims to automate model design in FL or solve the non-IID problem.\n2. NAS method and FL are naively combined, which undermines the novelty of this paper.\nThe comparison experiments conducted might be not fair enough. Model searched by NAS are generally smaller than handcrafted ones while enjoying higher accuracy. For example, in [1], a 4.6M ENAS model outperforms a 25.6M DenseNet model. In comparison, I think a DARTS model should be used as a base for previous FL methods.\n\n[1] H.Pham et.al, Efficient Neural Architecture Search via Parameter Sharing. ICML’18.", "summary_of_the_paper": "This paper proposes a method for automatic model design in the context of federated learning and introduces an auto-ml system under the framework of federated learning.\n\n", "summary_of_the_review": "None", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a personalized federated learning framework based on neural architecture search, in which the local clients perform NAS to search for a better architecture for the private local data. Specifically, the authors extend MiLeNAS, which is an existing NAS algorithm, to be run in the federated learning setting, and use FedAvg for model aggregation. The proposed FedNAS framework is validated against personalized federated learning methods with predefined architectures, such as perFedAvg, Ditto, and local fine-tuning, and is shown to largely outperform them on non-IID settings with label skew and LDA distribution. FedNAS’s collaborative search for the optimal architecture also yields a better performing global model than FedAvg.

The paper received borderline ratings. Three out of four reviewers are learning negative, while one is leaning negative. The below is the summary of pros and cons of the paper mentioned by the reviewers:

Pros
- The idea of using NAS for personalized federated learning seems novel and interesting. 
- The proposed FedNAS framework is shown to be effective in tackling the data heterogeneity problem, which is a fundamental problem with federated learning.
- The authors have released the code for reproducibility.

Cons
- The technical contribution of the work seems limited, since the proposed FedNAS straightforwardly combines an existing NAS method (MiLeNAS) with federated averaging, and there is no challenge mentioned for this new problem of federated NAS. 
- The choice of a specific NAS method (MiLeNAS) is not well justified, and other NAS methods should be also considered.
- The motivation is unclear: It is not clear whether the authors aim to perform collaborative automotive design or solve personalized federated learning.
- There is no convergence analysis.

While some of the concerns have been addressed away in the authors’ responses during the rebuttal period, the reviewers did not change their ratings, and the final consensus was to reject the paper. 

I agree with the authors that combining federated learning with NAS, and applying it for personalized federated learning is a novel idea that intuitively makes sense. However, I agree with the reviewers that the current method is a straightforward combination of an existing NAS method and an existing FL algorithm, the authors should identify new challenges posed by the combination of the two methods, and identify them. 

Further, performing NAS on edge devices may be possible, but not the best solution, since it could result in large computational overhead. While the authors mention that MiLeNAS is computational suitable in such settings, there should be a proper investigation of the accuracy-efficiency tradeoff, showing how well FedNAS performs against others with the same computational budget (or training time / energy consumption).  

Overall, this is a paper that proposes a novel and interesting idea that seems to work, but the paper does not sufficiently examine challenges posed by the new problem. I suggest the authors identify the new challenges and examine the efficiency issue mentioned, and further develop their method, if necessary.
Decision: Reject