SPIDER: Searching Personalized Neural Architecture for Federated Learning
Authors: Erum Mushtaq
Chaoyang He
Jie Ding
Salman Avestimehr
Venue: Submitted to ICLR 2023 (2023)
Keywords: Personalized Neural Architecture Search
Data Heterogeneity
Personalized Federated Learning

Abstract: Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server. Recent advancements in FL use predefined architecture-based learning for all the clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clientsin FL. Motivated by this challenge, we introduce SPIDER, an algorithmic framework that aims to Search Personalized neural architecture for feDERated learning. SPIDER is designed based on two unique features: (1) alternately optimizing one architecture-homogeneous global model (Supernet) in a generic FL manner and one architecture-heterogeneous local model that is connected to the global model by weight-sharing-based regularization (2) achieving architecture-heterogeneous local model by an operation-level perturbation based neural architecture search method. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods with much fewer times of hyperparameter tuning.

Under review as a conference paper at ICLR 2023
SPIDER: SEARCHING PERSONALIZED NEURAL ARCHITECTURE FOR FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server. Recent advancements in FL. use predefined architecture-based learning for all clients. However, given that clients' data are invisible to the server and data distribu- tions non-identical across clients, predefined architecture discovered in centralized setting may not be an optimal solution for all the clients in FL. Mo- tivated by this challenge, introduce SPIDER, algorithmic framework that aims to Search PersonalIzed neural architecture for feDERRted learning. SPI- DER is designed based two unique features: (I) alternately optimizing architecture-honogeeeuus global model (Supernet) in a generic FL manner and one aichitecturr--eteengeeeooo local model that is connected to the global model by weight-sharing- babseii regularization (2) achieving sehitecture-heterogenne local model by an operation-lewe perturbation based neural architecture search method. Experimental results demonstrate that SPIDER outperforms other state- of-the-art art personalization methods with much fewer times of hyperparameter tun- ing.
INTRODUCTION
Federated Learning (FL) is promising decentralized machine learning framework that facilitates data privacy and low communication costs. has been extensively explored various machine learning domains such as computer vision, natural language processing, and data mining. Despite many benefits of Fl., one major challenge involved in Fl. data heterogeneity, meaning that the data distributions across clients are not identically independently (non-I.LD) distributed. The non-I.L.D distributions result the varying performance of a globally learned model across dif- ferent clients. In addition to data heterogeneity, data invisibility is another challenge in FL. Since clients' private data remain invisible to the server, from the server's 's perspective, it is unclear how to select pre-defined architecture from pool of all available candidates. In practice, it may require extensive experiments and hyper-parameter tuning over different architectures, a procedure that can be prohibitively expensive.
To address the data-heterogeenity challenge, variants of the standard FedAvg have been proposed train global model, including the FedProx Li et al. (018), FedoPT Reddi al. (020)). and FedNova Wang et al. (2020). In addition to training of a global model, frameworks that focus on training personalized models have also gained a lot of popularity. The Ditto Li et al. 20216),
PerFedAvg Fallah et al. (2000)), and pFedMe Dinh et al. (020) are some of the recent works that have shown promising results obtain improved performance across clients. However, all these works exploit pre-defined architectures and operate at the optimization layer. Consequently, in addition to their inherent typer-paramett tuning, these personalization frameworks often encounter the Bata-invisibility challenge that one has to select suitable model architecture involving lot of arper-parameter tuning.
In this work, we adopt different and complementary technique to address the data heterogeneity challenge for FL. We introduce SPIDER, an algorithmic framework that aims to Search PersonalIzed neural architecture for feDeRdted learning. Recall that in centralized setting, the neural architec- ture search (NAS) aims to search for optimal architecture to address system design challenges such lower latency Wu et al. 2019), lesser memory cost Li ei al. (2021a), and smaller energy consump-

---

Under review as a conference paper at ICLR 2023
tion Yang al. (020). For architecture search, there three well-known methods explored in literature, gradient- based Liu et al. (2118), evolutionary search Liu et al. (222)). and reinforcement learning Jaatra et al. (2019). Out of these, gradient-based methods are generally considered more
efficient because of their ability to yield higher performance in comparatively lesser time Santra
et al. (2021).
To achieve personalization at the architecture level in FL., we propose a unified framework, SPIDER. This framework essentially deploys two models, local and global models, on each client. Initially, both models use the DARTS search space-based Supernet Liu et al. (2018), an over-parameterized architecture. In the proposed framework, the global model shared with the server for the FL up- dates and, therefore, stays the same in the architecture design. On the other hand, the local model stays completely private and performs personalized architecture search, therefore, gets updated. To search for the personalized child model, SPIDER deploys cal model. The SPIDER -Searcher built upon well-known gradient- baade NAS method, named perturbation-based NAS Wang et al. (202I). The main objective of the SPIDER framework is to allow each client to search and optimize their local models while benefiting from the global model. To achieve this goal, we propose trains local and global models in an alternate fashion. However, the main challenge here is the op- timization of evolving local model architecture while benefiting from fixed global architecture. To address this challenge, SPIDER Trainer performs weight-sharing-bon -aaee regularization that reg- ularizes the common connections between the global model's Supernet and the local model's 's's child model. This aids clients in searching and training heterogeneous architectures tailored for their local data distributions. In nutshell, this approach not only yields architecture personalization in FL but also facilitates model privacy (in the sense that the derived child local model is not shared with the server at all).
each client's lo-
alternating bi-level optimization- based SPIDER Trainer that
To evaluate the performance of the proposed algorithm, we consider a cross-silo FL setting and use Dirichlet distribution to create non-L.I.D data distribution across clients. For evaluation, we report test accuracy at each client on the 20% of training data kept as test data for each client. We show that the architecture personalization yields better results than state-of-the- personalization algorithms based solely the optimization layer, such as Ditto Li et al. (002b), perFedAvg Fallah et al. 2020a)), and local adaptation Cheng et al. (2021).
To summarize, the following the key contributions of our work.
We propose and formulate a personalized neural architecture search framework for FL named SPIDER, from perspective complementary to the state-of-the-art to address data heterogeneity challenges in FL..
SPIDER is designed based on two unique features: (1) maintaining two models at each client, one to communicate with the server and the other to perform a local progressive search, and (2) operating local search and training each client by an alternating bilevel optimization and weight chasing-based regularization along the FL updates.
We extensive experiments to demonstrate the benefit of SPIDER compared with state-of-the- art personalized FL approaches such as Ditto Li et al. (2021)), perFedAvg Fallah et al. (2020a) and Local Adaptation Cheng al. (202I) on three datasets, CIFARIO, CIFAR and CINICIO. With the ResNet18 model, on the CIFARIO dataset with heterogeneous distribution, we demonstrate an increase of the average local accuracy by 2.8% 1.7% and 5.5%, over Ditto, PerFedAvg, and Local Adaption, respectively. Further, on CIFAR 100, we demonstrate an accuracy gain of 10%, 6%, 4% over Ditto, Local Adaptation, and perFedAvg, respectively. Likewise, on CINIC-10, we demonstrate an accuracy gain of 20%, 23%, and 24% over Ditto, Local Adaptation, and perFedAvg, respectively.
RELATED WORKS
Heterogeneous Neural Architecture for FL Heterogeneous neural architecture is one way to personalize the model in FL. For personalization, the primal-dual framework Smith et al. (2017)) clustering Sattler et al. (2020). fine-tuning with transfer learning Yu et al. (022b)) meta- learnig Fallah et al. (2020a), regularization-based method Hanzely & Richtárik (202)g Li et al. (021b) are among the popular methods explored in the FL literature. Although these techniques achieve improved personalized performance, all of them use a pre-defined architecture for each client. Het-
2

---

Under review as a conference paper at ICLR 2023
eroFL Diao al. (2020) is recent work that accomplishes the aggregation of heterogeneous mod- els by assigning sub sub-parts of the global model based on their computation budget and aggregating the parameters common between different clients. Similarly, work Luo et al. performs limited channel-wise search to assign sub-models meeting clients' efficiency budgets and performs the par- tial aggregation of weights at the server. Another work [in al. (2020)) accomplishes this task by forming clusters of clients of the same model and allowing for hetcrogeneous models across clus- ters. Model aggregation is based on cluster-wise aggregation followed by a knowledge distillation from the aggregated models into the global model. Given data invisibility in FL, deciding which ar- chitecture would work for which client is challenging task and requires exploration. Another work Dudziak et al. (2022) personalizes architectures to compute- power-based clusters instead of individ- ual clients. As such, our proposed method aims to achieve personalized architecture automatically and focuses on the objective of tailoring architecture to individual clients' data distribution.
Neural Architecture Search for FL Neural Architecture Search (NAS) has gained momentum in recent literature to search for a global model in a federated setting. FedNAS He et al. (2020b) explores the compatibility of MileNAS solver with Fed averaging algorithm to search for global model. Direct Federated NAS Hu et al. (2020) is another work in this direction that explores the compatibility of a one-shot NAS method, DSNAS Hu et al. (2020), with Fed averaging algorithm with the same application, in search of a global model. 7huu & Jin (202T) uses evolutionary NAS
to design a master (global) model. [Singh et al. (2020) explores the concept of differential privacy using DARTs solver Liu et al. (018)) to explore the trade-off between the accuracy and privacy of global model. Xu et al. (2020) starts with a pre-trained handcraft model and continues pruning the
model until it satisfies the efficiency budget. Another work Hoang & Kingsford (202T) divides the model architecture into global and personal components, and searches the personal component for personalization on identical and independent (IID) vision tasks. Where all these models search for unified global model or personalized components of a shared model, a key distinction of our work with these works is that we aim to search for complete personalized model for each client.
PRELIMINARIES, MOTIVATION, AND DESIGN GOALS
In this section, introduce the state-of-the methods for personalized federated learning, discuss the motivation for personalizing model architectures, and summarize our design goals.
Personalized Federated Learning A natural formulation of FL is to assume that among distinct clients, each client & has its own distribution P,, draws data samples from P,, and aims to solve supervised learning task (e.g., image classification) by optimizing global model with other clients collaboratively. At a high-level abstraction, the optimization objective is then defined as:
min G 10*
(1)
where (ww measures the performance of the model global w on the private dataset at client k (local objective), and G is the global model aggregation function that aggregates each client's local objectives. For example, for FedAvg, G(.) would be weighted aggregation of the local objectives, F& where Σi= Pk = 1.
However, distributions across individual clients are typically heterogeneous (i.e., non-L.LD.), there is a growing line of research that advocates reformulating FL as a personalization framework, dubbed as personalized FL (PFL). In PFL, the objective is redirected find personalized model V2 for device i that performs well on the local data distribution:
min (Fi(v)) Fv())),
To solve this challenging problem, various PFL methods are proposed, including FedAvg with local adaptation (Local-FL) Cheng et al. (202I): Yu et al. (2020a); Wang et al. (2019), MAML-based
PFL (MAML-FL. Fallah et al. (020b); Jiang et al. (2019), clustered FL (CFL) Ghosh et al. 2020);
Sattler et al. 2021) personalized layer- based FL (PL-FL) Liang et al. (2020) Hoang & Kingsford
2021 Pillutla et al. (022); Hoang & Kingstord 2021 federated multitask learning (FMTL)
Smith et al. 2017b) and knowledge distillation (KD)
et al. (2020a) He et al. (202aa).
3

---

Under review as a conference paper at ICLR 2023
SPIDER Trainer:
SPIDER Searcher:
Archtectura-Hettroo Heterogeneous Bi-level Optimization
each client search its own neural architacture progressively
Weight shaning
Loca mocel
(sparee
replace on the fly
weight an aring basid Requlartzatin
Fe
Phine Fixing architecture for warming
Phase Progresstve Searching Parsenelized Architacture
Phase 3: Flxing Architeccure
SPIDER Wesco (swacc)) a Difieent Web (reuzal ands (actur)) for Each Ciann
Time
Figure 1: Illustration of SPIDER frumework. SPIDER is weaving (searching) different web (neural architec- ture) for each client.
Motivation for Neural Architecture Personalization Distinct from these existing works on PFL, propose new approach to instead personalize model architecture for each client. We are moti- vated by one critical potential benefit, that is, the searched architecture for each client is expected to fit its own distinet distribution, which has the potential to provide substantial improvement over the existing PFL baselines that only personalize model weights. In addition, a personalized architecture search allows the clients even keep their local model architectures private in other clients neither know the architecture nor the weights of that architecture. This further enhances the privacy guarantees of FL and helpful in business cases that each client hopes to also protect its model architecture.
the server and
Design Goals Our goal to enable personalized neural architecture search all clients in FL.. In this context, the limitation of existing personalized FI. methods is obvious: Local-FI. and MAML- FL need every client to have the same architecture to perform local adaptations: In CFL, the clus- tering step requires all clients to share homogeneous model architecture; PL-FL only obtain heterogeneous architectures for a small portion of personalized layers, but does not provide an architecture-agnosie method to determine the boundary of personalized layers in automated method which cannot perform regularization when
mechanism; FMTL regularization-base architectures heterogeneous across clients; KD has an unrealistic assumption that the server has enough public dataset as the auxiliary data for knowledge distillation.
To circumvent these limitations, goal is to design architectur--personaiieee FL framework with the following requirements:
R1: allowing heterogeneous architectures for all clients, which can capture fine-grained data heterogeneity;
R2: searching and personalizing the entire architecture space, to avoid the heuristic search for the boundary of personalized layers;
R3: requiring auxiliary data at the client- server-side (unlike knowledge distillation- based PFL);
We now introduce SPIDER which meets the above requirements unified framework.
METHODOLOGY: SPIDER
4.1 OVERVIEW
The overall framework of SPIDER is illustrated in Figure ! Essentially, each client maintains two models in this framework: one architecture- hww.reetureeooooogeeeooues global model for collaborative train- ing with other clients, and one cefftecctur-heeerogeeeeoee same super architecture space as the global model. At a high-level, SPIDER is formulated as an rrchitecture-perssnnliiiee bi-level optimization problem (Section 4.2) and proposes the solver as
local model that initially shares the
4

---

Under review as a conference paper at ICLR 2023
the orchestration of SPIDER Trainer (Section 4.3) and -PiDERRSearrcer (Section 4.4 SPIDER
Trainer is an architecture persooaalzede training framework that can collaboratively train heteroge- neous neural architectures across clients. To allow federated training the expected heterogeneous local architectures, it enables regularization between an arbitrary personalized architecture and the global model via weight sharing. With this support, SPIDERR-eaacher is designed dynamically adjust the architecture of each client's 's local model on the way. To search a personalized architecture for the local data distribution of each client, SPIDER-Searcher is built on novel neural architecture search (NAS) method that searches optimal local Subnet progressively using operation-Iewe pertur- bation on the accuracy value as the criterion. Overall, each client's local model goes through three phases (also shown in Figure D pre-training to warm up the initial local model, progressive neural architecture search, and final training of the searched architecture rrchitecuree-personaiizze model.
SPIDER can meet the design goals R1-R3 introduced in Section 3 because 1) each client performs independent architecture personalization with its own private data (R1), 2) the search space is not restricted to portion of the model (R2), and 3) no auxiliary data is used to assist the search and train process (R3).
4.2 SPIDER FORMULATION: ARCHITECTURE AACHITEECTOR--PRRSOAALZZZ PERSONALIZED BI-LEVEL OPTIMIZATION
SPIDER aims personalize (weave) different neural architecture (web) for each client. To gener- ate heterogeneous architectures across clients, we use two models, a local model (A²) and a global model (Supernet A) each client, and formulate SPIDER as an optimization problem for each client k E (1.,......)
bi-level
min Ae CA
Au; w"
(3)
s.t. w E arg arg G (F, Fx (w,
(4)
where Fg is the local objective of the client k; w. and Ug are all learnable parameters; w denotes the parameter of the global model architecture A, og is the weight parameter of the local personalized
architecture Ag of the client k. Here, AA is a child neural architecture of a Supernet AA such that A S A. The Supernet A can be considered having a mask M, each entry representing a learnable parameter Osj, i E {1, .., e} and j E {1, ..., of where e is the total number of edges and o is the total number of operations at each edge. Hence, the size of mask M equal to e For the Supernet mask M, all azj entries are 1. Likewise, we maintain mask Mj for local models Ag. Our goal is to optimize Cii parameters of the mask M₁ such that we learn child architecture Ae = M2 @ A. Note that in Eq. we aim to learn global model AA in a federated learning setting, which formulates our in the inner optimization given in Eq. the objective each client is
outer optimization.Hower to optimize its local model's 's architecture Ag and its associated parameters 22 while benefiting from the global model w". As a tractable, yet general case study, SPIDER reuses the DARTS architecture space as Supernet A that contains set of edges (1...., e). and each edge has multiple operations (1.....)) o). Based on this definition, A& maintains the operation-level granularity: A&& edge set space is a subset of A's edge set space, and the operation set in A&'s each edge may also be a subset space. We provide further details of the search space in Appendix A.3
The difficulty of jointly optimizing the architecture A& and related weight parameters U% The key difference of our formulation from existing bi-level optimization for FL (e.g.. Li et al. (021b)) that in our case, A& is also learnable (Eq. We assume each client can have an evolving architecture Ag, i.e., (g has to optimize the architecture As and its related weight parameters vR jointly, while using complete supernet-based global model weights, w. SPIDER addresses this challenge by the orchestration of SPIDER- - and
4.3 SPIDER TRAINER: FEDERATED TRAINING ON HETEROGENEOUS ARCHITECTTRES
In this section, we describe SPIDER trainer, an architecture architecture-persnaalieee training framework that can collaboratively train heterogeneous neural architectures across clients.
To clearly show how SPIDER handles the optimization difficulty of Eq.3). we first downgrade the objective to the case that all clients use predefined (fixed) heterogeneous architectures (derived from
5

---

Under review as a conference paper at ICLR 2023
the Supernet A). More specially, reduce the aforementioned optimization framework in Eq. and Eq. to the following:
min hu(u,, As; , (U( + A Us w Wshore
(5)
s.t. w" E arrgmin G (F) F (w)
(6)
where local model's weights Uy regularized towards the global model w sharer where w Whoree are the weight parameters of the operation set space of A overlapping (sharing) with Aç. Also, the regularization typer-parameter Note that, now, only vy needs be optimized in Eq while Ag is fixed during the optimization.
We then solve E and B alternately. We summarize this optimization procedure SPIDER- Trainer with a detailed pseudo code illustrated in Algorithm D In this algorithm, we can note that the global model (line #12) and the local model (line #15) are updated alternately. The strength of this algorithm lies in its elaborate design, which provides the following key benefits:
Algorithm SPIDER Trainer
I: Initialization: initialize a clients with the k-th client has a global model w, using Supernet A, and local model ey using subnet A, with mask Mr (set As = AA at the begining); is the number of local epochs; r is the number of rounds; t, number of rounds to start search; is the recovery periods in the units of rounds.
2: Server executes:
3: for each round / = 0, 1,........ r- do for each client k in parallel do ClientLocalSearchh(. w/, N₂ w
4:
5: W
6:
w ΣE
N
function Cealaat.e w', fy: II Run on client for each epoch in do
9:
10:
for minibatch in training and validation data do
11:
At+1 M ProgressiveNAS((A My f 7, Update Global model: w²+1 = w - The V Lx (w") for cach Oiyy M do
12:
13:
14:
wi Wiy @ auj overlapping operation set space between A and As o va nh (V.L) (wf) Mep w wakere)
++1
15:
Update Local Model:
return server
(1) Enabling regularization between an arbitrary personalized architecture and the global model Most importantly, SPIDER-Trainer connects each personalized model with the global model by enabling the regularization between two different architectures: an arbitrary personal- ized architecture for the local model A& of client & and the global model with Supernet A. This is done by weight sharing. w Shore is essentially used to regularize subnet (AA)) model parameters towards the global model shared/common parameters whare" as shown in Eq.
(2) Avoiding heterogeneous aggregation SUDER-Trrinee avoids the aggregation of heteroge- neous model architectures at the server side. As such, no sophisticated and unstable aggregation methods required (e.g. knowledge distillation Lin et al. (2020a), etc.), and it is flexible to use other aggregation methods beyond FedAvg (e.g., Karimireddy ef al. (2020); Reddi et al. (202T) to update the global model.
(3) Enabling architecture privacy In this algorithm, only the global model is transmitted between the client and the server. This enables architecture privacy because each client's architecture is hidden from the server and other clients.
(4) Potential robustness to adversarial attacks The weight sharing-based regularization not only yields the benefit of personalization in FL but also makes the FL framework more robust to ad- versarial attacks. Its robustness advantage comes from its ability keep the local model private and regularize towards the global model based on its regularization parameter, as shown before by mnchitectre--hmmgenncee bi-level optimization Li et al. (022bb).

---

Under review as a conference paper at ICLR 2023
4.4 -SEDRR-EARRHER: PERSONALIZING ARCHITECTURE
Although SPIDER trainer is able to collaboratively train heterogeneous architectures, manual design of the architecture for each client impractical or suboptimal. As such, we further add a neural architecture search (NAS) component, SPIDER-Scarche in Algorithm (line #11) to adapt Aę to its local data distribution in a progressive manner. We now present the details of the SPIDER- Searcher.
dynamically changes the architecture of As during the entire federated training process. This is feasible because the weight sharing- based regularization can handle an arbitrary personalized architecture (introduced in Section 4.3) Due this characteristic, OPIDER-Searcher can search A₁ in a progressive manner (shown in Figure 1): Phase 1: At the beginning, A₁. is set equal to Supernet A. The intention of SPIDER- warm up the training the initial A& so it does not change A for a few rounds; Phase 2: After warming up. SPIDER-Searcher performs edge-by-edge search gradually. In each edge search, only the operation with the highest impact on the accuracy is kept. It also uses few rounds of training as recovery time before proceeding to the next round of search. This process continues until all edges finish searching; Phase 3: After all edges finish searching, sPIDER-Searcher does not change Ag. This serves final training of the searched architecture- personalized model. This three-phase procedure is summarized as Algorithm - Now, we proceed to elaborate how calculate the impact of operation the Supernet.
Progressive Neural Architecture Search Essentially,
Searcher in this phase
Algorithm SPIDER-Searches
1: Search Space: in the architecture A C A, & is the super set of all edges (1,...,e, Eg is the remaining subset of edges that have not been searched, and each edge has multiple operations {1,....
2: function ProgressivNASSCA,
M t)
1 2 4, and % 7 == 0 LEN(E.) then
4: i= RANDOM (E.) random selection
5: // searching without training
6: for all operation j on edge do
evaluate validation accuracy of A when operation axy set zero (remove)(AACC...
8:
in the ith row of Mt, keep only one operation as 1 corresponding the lowest value of (ACCrag))
9:
remove from E,, update A Al. @ M
10:
else
11:
return AC, and Mt directlly
12:
return updated Att and Ma after selection
Operation-level perturbation- selection In phase 2, we specify selecting the operation with the highest impact using operation-level perturbation. More specially, instead of optimiz- ing the mixed operation architecture parameters using another bi-level optimization as DARTS search) to pick optimal operation according to magnitude of a parameters mragnitude-baseld selection), we assign a constant value to a and use the impact of an operation on the local validation accuracy (perturbation) as a criterion to search on the edge. This simpli- fied method is much more efficient given that it only requires evaluation-based search rather than training- muineng-based search (optimizing aty). In addition, it avoids inserting another bi-level optimiza- tion for NAS inside bi-level optimization for FL. making the framework stable and easy to tune. Finally, this method avoids suboptimal architecture Wang et al. (202T) lead by magnitude- based selection in differentiable NAS.
(a.k.a. gradient-
EXPERIMENTS
This section presents the experimental results of the proposed method, SPIDER. All our experiments based on non-IlD data distribution among FL clients. We have used latent Dirichlet Distribution (LDA), which a common data distribution in FL to generate non-III IID data across clients He et al. 2020c). Yurochkin al. (0019)
7

---

Under review as a conference paper at ICLR 2023
5.1 EXPERIMENTAL SETUP
Task and Dataset We perform an image classification task on three well-known datasets, CI- FARIO, CIFAR100 and CINIC10. CIFARIO dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class, and CIFAR 100 dataset consists of 60,000 images in 100 classes, with 600 images per class. CIFARIO0 has more classes and comparatively fewer data per class, therefore. it is considered more challenging than CIFARIO. In addition, CINICIO consists of 270,000 32x32 color images in 10 classes, with 90,000 images per train, test, and validation sub- set. CINIC10 dataset is a larger dataset and includes images from ImageNet as well. We generate non-IID data across 8 clients by exploiting LDA distribution with parameter (a = 0.2) for the train- ing data of CIFARIO, CIFARI00, and CINICIO datasets. The LDA distribution of CIFARIO and CIFARION 100 has been shown in Appendix A.1
For personalized architecture experiments with SPIDER, split the total training data samples present at cach client into training (50%), validation (30%), and testing sets (20%). For other per- sonalization schemes used for comparison, we do not need validation data. Therefore, we split the data samples of each client with training (80%) and test (20%) for a fair comparison. In addition, fix the non-IID data distribution in all experiments. We provide Hyperparameter search details in Appendix A.2
Table I: Average local validation Accuracy Comparison of SPIDER with other personalization techniques on CIFARIO, CIFARI00 and CINICIO Datasets
CIFARIO
CIFARTOO
CINICIO
Method
Average
Parameter
Average
Parameter
Average
Parameter
Accuracy
Size
Accuracy
Accuracy
Local Adaptation ResNet18
0.87+0.02
11M
0.61:0.03
1I
0.61= 0.05
IIM
Ditto ResNet18
0.9020.03
IIM
0.57=0.03
IIM
0.64 0.05
IIM
pert'edAvg- - ResNet18
0.91
I1M
0.6420.03
11M
0.00
IIM
Local Adaptation DARTS
0.85 20.03
4.7M
0.63 20.04
4.7M
0.73 #0.04
4.7M
Ditto DARTS
0.7720.05
4.7M
0.4520:04
4.7M
0.6020.03
4.7M
SPIDER
0.9320.02
2.8M
0.6810.03
3.1M
0.8420.07
3.3M
Implementation and Deployment We implement the proposed method for distributed comput- ing with nine nodes, each equipped with GPUs. We this as cross-silo FL setting with one node representing the server and eight nodes representing the clients. These client nodes can represent real-world organizations such as hospitals and clinics that aim to collaboratively search for person- alized architectures for local benefits such as higher accuracy in preeeccires FL manner. Since we are working on a cross-silo setting, neural architecture search cost may not be a concern since devices in cross-silo rich in resources.
5.2 RESULTS
Here, we report the comparison of our proposed method SPIDER with the other state-of-the-art personalized methods: Ditto, perFedAvg, and local adaptation. Since these schemes use a pre- defined architecture, we use the Reset1 8 model because of its comparable model size. Since we are exploiting DARTS based search space in this work, we also use DARTS model Liu al. (2018) searched in a centralized setting CIFAR10 dataset as our base model for local adaptation and Ditto personalization schemes.
5.2.1 AVERAGE TEST ACCURACY
As illustrated in Table D our method, SPIDER, achieves the objective R1 by outperforming the state-of-the-arr personalization methods; Ditto, local adaptation and perFedAvg on three image classification datasets CIFARIO, CIFAR100 and CINIC10. For CIFAR100 and CINIC10 dataset, Local adaptation with the DARTS based model performed the second best and outperformed Ditto, perFedAvg and Local adaption on ResNet18. For CIFARIO, perFedAvg with Resnet18 performed the second best. Overall, we obtain 2%, 4% and 11% higher accuracy as compared to the best performing scheme from the representative state-of-the-an personalization algorithms; Ditto, local the CIFAR10. CIFAR100 and CINIC-10 datasets, respectively. For CINIC 10 dataset, we observe substantial performance gain (1 (11%) with respect to the sota personal- ization techniques. We attribute this gain to the potential of SPIDER to adapt invisible data better
adaptation, and perFedAvg,
8

---

Under review as a conference paper at ICLR 2023
by tailoring architectures for each client according to their data distribution. In addition, Ditto with DARTS model did not perform well for all three datasets and may need a bigger parammer search set than the one considered (given in Appendix A.2) to converge better.
For personalization, an important metric. Therefore, also report the standard deviation for each method in Table [ We note that the standard deviation across clients is almost similar for CIFARIO and CIFAR1ON datasets, For CINIC10, the standard deviation of SPIDER is slightly higher but this is achieved at 11 % higher accuracy. Overall, we achieve almost the same standard deviation as other baselines but with the benefit of providing higher average test accuracy.
addition to average accuracy, the standard deviation (std) is also considered
5.2.2 ARCHITECTURE HETEROGENEITY AND PERSONALIZATION GAIN
An important feature of SPIDER is that it helps each client search for its own architecture tailored for its own specific data distribution. Due to data heterogeneity across clients, we observe architectures be heterogeneous across clients shown Appendix B.4 hence achieve the objective R2, the search and training of heterogeneous architectures across clients. The objective R3, no dependence on auxiliary data, is obtained by the algorithmic framework of SPIDER it does rely on any auxiliary data and exploits only the client's 's dataset for searching and training. A byproduct of using DARTS search space is that the average parameter size the models obtained with SPIDER is quite smaller. In order to further investigate whether the architecture searched by one client is best suited for its own data distribution, we perform the following experiment.
For any given client i, we apply its final architecture with its learned weights to another client j's data and finetune i's architecture on j's data. We denote the accuracy we obtain on data j via architecture i as APtj. We calculate the architecture i personalization gain or drop (if negative) as follows,
EC-1 gi Ei (APx APjx)
We calculate it across all silos. The quantity gi signifies the personalization gain of architecture across other silos architectures on its own dataset. Next, we calculate the personalization gain of SPI-
DER scheme as the mean of all gi, i.e. Linn g. where is the total number of clients in the network. We conduct this experiment on CIFARIO, CIFARIO0, CINICIO datasets, where we finetune the ar- chitecture learned via SPIDER on other clients" data for 20 epochs and report the best accuracy. We obtain 1.96%, 3.1 1111%, 3.71% %1% personalization gain on CIFAR1O. CIFAR100 and CINICIO datasets, respectively. We observe that for the majority of the clients, its searched architecture outperforms the other architectures. This highlights the importance of architecture personalization which can be more powerful than just weight personalization (as shown by our empirical results). We report the AP. and AP, values for all i and j in Table 2
and Appendix section B.4
Table 2: CINICIO: Personalization Gain Analysis
(n) Acconacy values of Architecnure on
dotn (where and
represent eitent lDs.
(b) Accuracy Goin/Dmp Motrix (AP,, y
APij) and the rendhnt 9, tecter
1) 1=0
1=2
1=3
1=5 5
i=6
i=
i=4
i=6
1=7
87.17
1-1
36.54
90.55
87.42
87.79
87.59
87.94
S7.21
86.25
1=1
4.01
3.13
2.76
196
2.61
3.34
4.3
3.50
1=
72
73.27
18.23
73.9
75.21
74
74,73
73.11
1=2
3.44
436
0
4.74
112
361
33
4.42
4.24
3-3
54.27
84.22
35
54.68
3-3
0.
3.72
1=4
69:46
71,40
70.34
71.69
75.29
70.58
71.52
67.92
1-4
5.83
3.89
4.95
36
0.
4.71
3.77
7.37
4.87
1=3
73.00
71.95
71.15
73.16
M.00
77.53
74.18
72.03
1=3
3.63
3,58
3,08
1.37
331
3,35
3,5
1.15
1-6
83.69
$4.29
53.78
82.92
4.19
5.08
1-7
89.5
88.68
89.81
88.37
89.5
89.00
88.44
91:25
1-7
1.75
2.57
1.44
2.88
1.75
2.06
2,81
0.
2.18
CONCLUSION
We proposed SPIDER, an algorithmic framework that can search personalized neural architecture for FL. SPIDER specializes weight-sharing- based global regularization to perform progressive neural architecture search. Experimental results demonstrate that SPIDER outperforms other state- of-the-art personalization methods by searching and training a personalized architecture for each client.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gary Cheng, Karan N. Chadha, and John C. Duchi. Fine-tuning in federated learning: simple but tough-to-beat baseline. 2021.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient
federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau
envelopes. arXiv preprint arXiv:2006. 08848, 2020.
Lukasz. Dudziak, Stefanos Laskaridis, and Javier Fernandez-Marques. Fedoras: Federated architec- ture search under system heterogeneity. arXiv preprint arXiv:2206.11239) 2022.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.07948, 2020a.
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. Personalized federated learning: A meta- learning approach. ArXiv, abs/2002.07948. 2020b.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. ArXiv, abs/2006.0408s, 2020.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.055// 2020.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. arXiv: Learning, 2020a.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Fednas: Federated deep learning via neural architecture search. arXiv e-prints, pp. arXiv-2004, 2020b.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Pra- neeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007. 13518, 2020c.
Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11993-12002, 2020d.
Minh Hoang and Carl Kingsford. Personalized neural architecture search for federated learning. 2021.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Dsnas: Direct neural architecture search without parameter retraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12084-12092, 2020.
Yesmina Jaafra, Jean Lue Laurent, Aline Deruyver, and Mohamed Saber Naceur. Reinforcement learning for neural architecture search: A review. Image and Vision Computing. 89:57-66, 2019.
Yihan Jiang, Jakub Konecný, Keith Rush, and Sreeram Kannan. Improving federated learning per-
sonalization via model agnostic meta learning. ArXiv, 190119999122888 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICMI, 2020.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, and Yingyan Lin. Hw-nas-bench: Hardware-aware neural architecture search benchmark. arXiv preprint 2rXiv:2103 103.//0408/ 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv: 18/2.0//// 2018
10

---

Under review as a conference paper at ICLR 2023
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR. 2021b.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. ArXiv, abs/2001.0152., 2020.
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion federated learning. ArXiv, abs/2006.0722 2020a.
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242, 2020b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint 1rXiv: 09055 2018.
Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE Transactions Neural Networks and Learning Systems, 2021.
Mi Luo, Fei Chen, Zhenguo Li, and Jiashi Feng. Architecture personalization resource- constrained federated learning.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. International Conference on Machine Learning, pp. 17716-17758. PMLR, 2022.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
Sashank J. Reddi, Zachary B. Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecný, Sanjiv Kumar, and H. Brendan McMahan. Adaptive federated optimization. ArXiv, abs/2003.00025, 2021.
Santanu Santra, Jun-Wei Hsieh, and Chi-Fang Lin. Gradient descent effects differential neural architecture search: A survey. IEEE Access, 9:89602-89618, 2021.
Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on ral networks and learning systems, 2020.
Felix Sattler, Klaus-Rober Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neu- ral Networks and Learning Systems, 32:37 10-3722, 2021.
Ishika Singh, Haoyi Zhou, Kunlin Yang, Meng Ding, Bill Lin, and Pengtao Xie. Differentially- private federated neural architecture search. arXiv preprint arXiv:2006. 10559, 2020.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. arXiv preprint arXiv: 10467, 2017a
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S. Talwalkar. Federated multi-task learning. In NIPS, 2017b.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Franccoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. ArXiv, abss/191.. 10252, 2019.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking architecture selection in differentiable nas. arXiv preprint ar8iv:2 04392, 2021.
11

---

Under review as a conference paper at ICLR 2023
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awand efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 10734-10742, 2019.
Mengwei Xu, Yuxin Zhao, Kaigui Bian, Gang Huang, Qiaozhu Mei, and Xuanzhe Liu. Federated neural architecture search. arXiv preprint arXiv:2002.06352, 2020.
Lei Yang, Zheyu Yan, Meng Li, Hyoukjun Kwon, Liangzhen Lai, Tushar Krishna, Vikas Chandra, Weiwen Jiang, and Yiyu Shi. Co-exploration of neural architectures and heterogeneous asic accel- erator designs targeting multiple tasks. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2020.
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adapta- tion. ArXiv, abs/2002.04758, 2020a.
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adapta- tion. arXiv preprint 2Xi::2002.04758, 2020b.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Interna- tional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Hangyu Thu and Yaochu Jin. Real-time federated evolutionary neural architecture search. IEEE Transactions Evolutionary Computation, 2021.
12


Reviews:
[{"review_id": "kjH79HmrXqk", "reviewer": "Reviewer_AAmq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission proposed NAS algorithm for personalized Federated Learning. Specifically, it alternately optimizing one homogeneous global model and one local model that is the subnetwork of the global one. Basically, the idea is not novel and the solution is straightforward, limiting its contribution to the community. Moreover, the notations used are not well-presented, making the draft hard to follow. I thus do not champion the acceptance.", "summary_of_the_review": "The submission proposed NAS algorithm for personalized Federated Learning. Specifically, it alternately optimizes one homogeneous global model and one local model which is the subnetwork of the global one. Basically, the idea is not novel and the solution is straightforward, limiting its contribution to the community. Moreover, the notations used are not well-presented, making the draft hard to follow. The experiments are also very short. Many baselines are not compared. I thus do not champion acceptance.", "strength_and_weaknesses": "\nStrong points:\nS1: the results seem promising.\n\nS2: the literature review is somehow thorough.\n\nWeakness:\n\nW1: the idea is not novel and the solution is straightforward. The submission claims in the related work section “Where all these models search for a unified global model, a key distinction of our work with these works is that we aim to search for a personalized model for each client.” However, this is not true. Many existing work such as the following two are also providing NAS for different clients in FL.\n\n1). Personalized Neural Architecture Search for Federated Learning, Minh Hang et al. Neurips Workshop 2021.\n2). Personalized Federated Learning via Heterogeneous Modular Networks. Tianchun Wang, et al. ICDM 2022, 2022.\n\nActually, these two papers are using more elegant solution for PFL. The proposed solution adopts the mixture method of ideas of NAS and MAML. It needs to maintain two networks (one global and one local) that causes additional communication/storage burden in FL. The second term (regularization) term in formula (5) is very straightforward. \n\n\nW2: the notations used are not well-presented, making the draft hard to follow. The notations used are not formal. Typically, matrix, vectors, and scalars are using different fonts for easy reading. However, all the notations are using the same fonts. \n\nW3: There are too many typos and grammar errors indicating the low-quality of the draft. Even in the abstract, there are clear typos, e.g., “all the clientsin FL”.\nMore examples, in the intro “we demonstrate accuracy gain of 10%, 6%, 4% over”==>”we demonstrate an accuracy gain of 10%, 6%, 4% over “\nIn the related work “ meeting clients efficiency budgets”==> meeting clients’ efficiency budgets\n\nW4: The experiments are far from complete. The authors only compared to very few PFL baselines. Many more related works are not compared. For example, the most relevant “Personalized Neural Architecture Search for Federated Learning, Minh Hang et al. Neurips Workshop 2021.” Was not compared.\n\nW5: There is no theoretical analysis if the algorithm will converge or not.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "diGtHwjXVYE", "reviewer": "Reviewer_p6Ea", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SPIDER, an algorithmic framework that aims to search personalized neural architecture for federated learning. SPIDER alternatively optimizes a global model (called supernet) and a local model that is connected to global model and has its own architecture. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods with much fewer times of hyperparameter tuning.", "summary_of_the_review": "While this paper proposes a novel method and empirical results show some improvements, the proposed method needs more clarifications and discussions. For the empirical results, some recent methods are missing in comparison, and ablation studies are also largely missing. Thus I choose reject as my initial score. ", "strength_and_weaknesses": "Strength:\n- This paper proposes a novel method for federated neural architecture search \n- Empirical results against several existing baseline methods show some improvements of the proposed method\n\nWeakness: \n- There are some unclear parts on the proposed method: \n  - I am a bit confused on the definition of $a_k$, i.e. the architectural parameter for each client. Is this 0/1 variables whose elements are associated with operations in supernet? The authors may need to clarify that and revise some notations. \n  - Based on the above concern, how is $a_k$ updated in Algorithm 2? Specifically, in step 7, the authors evaluate the validation accuracy of $a_k^t$ with one operation removed. Nevertheless, note that only one operation is kept in each iteration, then how can we evaluate the validation accuracy of $a_k^t$ with more operation removed? This is very confusing and require some clarifications. \n- Some more discussions are needed for the proposed method:\n  - I am a bit uncertain on the differences between SPIDER and the following method: perform supernet training, then distill useful subnetworks from the trained supernet based on local data, while SPIDER distills subnetworks during training. The authors may add some discussions on this variant. \n  - Moreover, can algorithm 2 uses other NAS methods (e.g., [1])? Some discussions may also be needed to make the proposed method more complete. \n- Some recent baseline methods [2, 3] are missing in current version, which makes it hard to evaluate the actual performance of the proposed method. The authors may need to add some comparisons to them\n- Ablation studies are also largely missing in current version. It is unclear how each part of the proposed method contributes to its empirical performances. \n- Minor issue:\n  - It may be better to move some parts of the supplementary material to main text, e.g., architecture personalization gain, which seems critical for the whole paper\n  - The authors may check the citation format. Currently citations are mixed with the main text and creates some difficulties in reading\n  - Typos in abstract: line 6 “clientsin -> clients in”?\n\n[1] Single Path One-Shot Neural Architecture Search with Uniform Sampling. ECCV 2020\n\n[2] Personalized Federated Learning through Local Memorization. ICML 2022\n\n[3] Federated Learning with Partial Model Personalization. ICML 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ThD4-aDAR6", "reviewer": "Reviewer_pg98", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates a personalized neural architecture search framework for the federated learning, aiming at searching for the personalized neural architectures for each client under the federated learning scenarios. The topic and the proposed idea of this paper are novel, but the experimental analysis and the technical depth of this paper are not enough.", "summary_of_the_review": "The topic is novel and important, the paper is well-written and the experiments demonstrate the significance of the proposed method. But some important baselines are missing in the experiments, the novelty of the proposed method should be discussed in depth, the experimental analysis are not enough, and the design of the proposed method are not clear. Suggest to further modify this paper for publication.", "strength_and_weaknesses": "S1. The topic studied in this paper, i.e., realizing both weight and architectural personalization in the federated learing (FL) scenarios, is important and novel in the field of FL. \nS2. This paper is well-written, the problem definition and algorithm description are clear and easy to follow.\nS3. In the experiments, the authors compare the proposed method with the personalized FL methods with a fixed model architecture, demonstrating the significance of realizing architectural personalization in the field of FL.\n\nW1. A kind of solution for the problem studied in this paper is to combine the existing personalized federated training method with the existing gradient-based neural architecture search (NAS). That is using the personalized federated training method to replace the non-personalized single-machine model training method applied in the existing gradient-based NAS method. In this way, both the architectures and the weights can be optimized in a personalized manner under the FL framework, realizing both weight and architectural personalization in the FL scenarios. \nAuthors should compare the above solution (using different NAS algorithms and federated training methods) with the proposed method in the experimental part, to demonstrate the superiority and the importance of the proposed method.\nW2. What’s the difference between the solution mentioned in W1 with the proposed method in this paper? The solution mentioned in W1 is straightforward (it is just a simple combination of the existing works), and more appropriate solution should be designed to deal with the novel federated NAS problem. Does the novel federated NAS problem bring new challenges, requiring novel NAS solutions or federated training methods? Does the proposed method solve these challenges? Authors can discuss these contents in depth.\nW3. The experimental analysis is not enough. The performance curves, i.e., the federated performance of the proposed method and other federated NAS methods at different time steps, are not given and not analyzed in the experimental parts. These curves can reflect the search effect and the search efficiency of different algorithms under the FL framework, which is important for evaluating a federated NAS algorithm.\nW4. Why perturbation-based NAS is selected to realizing the NAS operations in the proposed method? Why not select the other gradient-based methods? The selection basis should be clarified in the next version. In addition, authors should add experiments to examine the differences of selecting perturbation-based NAS instead of other NAS method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0FIVQTuqDG", "reviewer": "Reviewer_fLM4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle personalized neural architecture search for federated learning, where data heterogeneity is alleviated by the architectural personalization. The authors propose a method named SPIDER that uses a supernet from which local models can be sampled while sharing the weights. The authors validate their method on several benchmarks against existing methods.\n", "summary_of_the_review": "I enjoyed reading the paper, but several improvements seem to be required, as mentioned above.", "strength_and_weaknesses": "**Strength**\n- The paper is easy to read.\n- They tackle the practical federated learning with architectural personalization.\n- The results are impressive.\n\n**Weaknesses**\n- The proposed algorithm trains two models at each client, both global supernet and local model, which may increase the computation and memory consumption of local resources. Although NAS is devised to optimize consumption of given resources, the proposed framework seems to be inefficient and even not practical in real-world scenarios. \n- In the same context, the overall communication and computation costs should be discussed and compared with the baseline models. But, I can only find the comparison of the searched model’s performance and its size (parameters).\n- It is better to compare with more similar existing baseline models that uses supernet for sampling the local models, i.e. HeteroFL [Diao et al 21] or FedorAS [Dudziak et al 22], in terms of both efficiency (communication costs, etc) and performance.\n- More experiments should be conducted for detailed analysis, i.e. ablation study.\n\nDiao et al, HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients, ICLR 2021.\n\nDudziak et al, FedorAS: Federated Architecture Search under system heterogeneity\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject