Distributed Architecture Search Over Heterogeneous Distributions
Authors: Erum Mushtaq
Chaoyang He
Jie Ding
Salman Avestimehr
Venue: Accepted by TMLR ()

Abstract: Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server. Recent advancements in FL use predefined architecture-based learning for all clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, we introduce SPIDER, an algorithmic framework that aims to Search PersonalIzed neural architecture for feDERated learning. SPIDER is designed based on two unique features: (1) alternately optimizing one architecture-homogeneous global model in a generic FL manner and architecture-heterogeneous local models that are connected to the global model by weight-sharing-based regularization, (2) achieving architecture-heterogeneous local models by a perturbation-based neural architecture search method. Experimental results demonstrate superior prediction performance compared with other state-of-the-art personalization methods.

Published in Transactions on Machine Learning Research (11/2023)
Distributed Architecture Search Over Heterogeneous Distributions
Erum Mushtaq
emushtag@uss.eddu
Deportment of Electrical and Computer Engineering University of Southern California
Chaoyang He
ch@ffedml.c
PelML, Inc
Jie Ding
dingjouum.. edu
School Statistics
University of Minnesota
Salman Avestimchr
avestine Crssc. edu
Department of Electrical and Computer Engineering University of Southern California
Reviewed OpenReview: https: // openreview. net/ ptt/forum? id= id=5g75NGDRk1
https://openrevveew
Abstract
Federated learning (FL) an efficient learning framework that assists distributed machine learning when data cannot be shared with centralized server. Recent advancements in FL predefined architecture-
learning for all clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, predefined architecture discovered in centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, we introduce SPIDER, an algorithmic frame- work that aims Search PersonalIzed neural architecture for feDERated learning. SPI- DER designed based on two unique foatures: (1) altcrnately optimizing one architccture- homogeneous global model in generic FL manner and herfiteceuue-eeeerogeeee models that connected to the global model by weight-sharing based regularization, (2) achieving architecture-hetrrggneee chitecture search method. Experimental results demonstrate superior prediction perfor- mance compared with other state-of-the-art personalization methods. Code is available at http://ttttth.oo..o//o http:://ghhhhhooo///////uuuuuuhh /Erum//uuuuaaaa/SSIDRR.....
local
local models by a perturbation-based neural ar-
1 Introduction
Federated Learning (FL.) is promising distributed machine learning framework that facilitates data privacy and low communication costs. It has been extensively explored in various machine learning domains such as computer vision, natural language processing, and data mining. Despite many benefits FL, one major challenge involved in FL data heterogeneity, meaning that the data distributions across clients are not identically or independently distributed (non-I.I.D). The non-I.I.D distributions result the varying perfor- mance of a globally learned model across different clients. In addition data heterogeneity, data invisibilityy is another challenge in FL. Since clients' private data remain invisible to the server, from the server's per- spective, it is unclear how to select pre-defined architecture from pool of all available candidates (Ding] jet al. 2018). In practice, it may require extensive experiments and hyper-parameter tuning over different architectures, procedure that can be prohibitively expensive.
*This work was done while Chaoyang He PhD student USC.

---

Published in Transactions on Machine Learning Research (11/2023)
The current literature on this subject addresses the data-heterogeneity challenge by exploring the variants of the standard Avd to train global model, including the FedProx (Li et al. 2018), FedDPT (Reddi ltt al. 2021 and FedNova (Wang et al. 2020). In addition to training a global model, frameworks that focus on training personalized models have also gained a lot of popularity. The PerFedAvg (Fallah et al. 2020,, pFedMe (Dinh et al., 2020), Ditto (Li et al. 2021 and SelfFL (Chen et al. 2022) are some of the recent works that have shown promising results to obtain improved performance across clients. However, all these works exploit pre-defined architectures and adapt the optimization algorithm to accommodate data heterogeneity. Consequently, in addition to their inherent paprm-prrreeter tuning, these personalization frameworks often encounter the invisibility of training data challenge (Majeed et al., 2022) that one has to select a suitable model architecture involving a lot of hyper-parameted tuning,
There is another line of work that has recently emerged that uses neural architecture search (NAS) in feder- ated learning. In FI., NAS has been explored for a global architecture search (He et al. 2020b Yuan ct al. 2022), ; partial model personalization (Hoang & Kingsford 2021), cluster-based architecture
search (Wang et al. 2022) and resource- personalization (Dudziak et. al., 2022 Isik et al. 2023)
However, we argue that global architecture search may not capture the data heterogeneity in FL with non-L.LDD data. Additionally, partial model personalization method scarches some components of a predefined model, and therefore, needs search in order to identify an optimal boundary between person- alized and shared (global) components for each client. This search can become prohibitively expensive if not conducted in an automated fashion. The cluster-based architecture search methods search the same architecture per cluster, often requiring clients to share their data distribution information form clusters, which may not be feasible for clients such as hospitals due to privacy concerns.
Alternatively, we address the data heterogeneity challenge by facilitating each client to search the entire architecture space and personalize architecture to each local client's data distribution in a non-cluster- based setting. Oftentimes, the search for an optimal architecture is regarded as a paraemerrreeter tuning problem for a given task (or a data distribution) (Kim et al. 2021). Since data is non-I.I.D due to different data-generative models at different silos, the selection of the optimal hyper-parameters/ architectures can be different across different silos. Therefore, in such a setting, we propose to enable each silo to search for a personalized architecture and capture its own data distribution.
To search personalized architectures, we introduce SPIDER, algorithmic framework that aims to Search PersonalIzed neural architecture for each individual client in feDERated learning. Particularly, we focus on the cross-silo setting where clients have ample computational resources but data heterogeneity is a main concern Huang et al. (2022). Under this setting, SPIDER deploys two models, a local and a global model, on each client. Initially, both models use the DARTS search-space--asee Supernet (Liu et al.] 2019), an over-parameterized architecture search space. The global model is shared with the server for the FL updates and, therefore, stays the same in the architecture design. On the other hand, the local model performs personalized neural architecture search and gets updated. The searched local/ /child models can be heteroge- ncous across silos. To scarch and optimize the heterogencous local models while benefiting from the global model, we exploit SPIDER Trainer which 1) trains the local and global models in an alternate fashion and 2) regularizes the common connections between the global model's Supernet and the local model's child model. This regularization essentially distills knowledge from the global model to the local model. The proposed approach not only aids in uncovering different personalized architectures across various silos but also ensures that the derived local model remains local, with no sharing to the server and other clients.
To evaluate the performance of the proposed algorithm, consider cross-silo FL setting and use the Dirichlet Distribution to create a non-I.I.D data distribution across clients. For evaluation, we report the test accuracy at each client using the 20% of training data kept as test data for that client. Furthermore, we compare our work to the state-of-the-ar predefined architecture- based personalization FL schemes. We demonstrate that the architecture personalization yields better results than state-of-the-art personalization algorithms based solely on the optimization layer, such as Ditto (Li et al. 2021), perFedAvg (Fallah et al,, 2020), local adaptation (Cheng et al. 2021 and KNN-Per (Marfoq et al. 2022). Additionally, we also compare our work to a cluster-based hypernetwork search method, FedMN (Wang et al. 2022).
To summarize, the key contributions of our work are as follows:
2

---

Published in Transactions on Machine Learning Research (11/2023)
We propose and formulate personalized neural architecture search framework called SPIDER for personalized federated learning (FL.). Architecture personalization allows each silo to capture the complexities of its local data distribution at the level of architecture design and potentially leads to improved personalization.
Personalized Architecture Search:
Model Training: SPIDER formulates architecture personalization as a bilevel optimization. It solves the bilevel optimization objective by incorporating two distinct features: 1) exploiting an alternate training- based SPIDER Trainer that trains one hetaatttcturr-hmmoeeeeeeee global model, also called Supernet, in generic FL manner and one setiiteecuur-heterggeeeeee to the global model by weight-sharing-baan bastengg-b-eddd-oo regularization, 2) scarching architecture- model by performing neural ure search locally. The proposed approach facilitates knowledge sharing from the global model to the local model, enabling effective learning from other silos in the presence of heterogencous personalized architecture settings.
Cechitecture-Heetroggnnnee
local model that is connected
local
Performance Evaluation: We conducted extensive experiments to demonstrate the benefits of SPID ER compared to ofate-of-the-art personalized federated learning approaches, including Ditto (Li et al. 2021
perFedAvg (Fallah et al.) 2020). Local Adaptation (Cheng et al.) 2021), KNN-Per (Marfoq et al. 2022 and FedMN (Wang et al., 2022). We observed significant improvements in average local accuracy com- pared to the aforementioned approaches on three datasets: CIFAR1O, CIEAR100, and CINIC10 using the ResNet18 model. Among the state-of-the-ar methods, KNN-Per obtains the highest accuracy for all three datasets. SPIDER outperforms this PFL baseline by an accuracy margin of 3.29%, 5.31%, and 1.59% on CIFARIO, CIFAR100, and CINIC10 datasets, respectively. To assess the gain achieved through architecture personalization, we measure the accuracy gap between the current client's model and models attained from other clients and fine-tuned on the current client's data. On average, observed accuracy drop with the other clients' fine-tuned model compared to its own searched architecture. We refer to this accuracy drop as the "personalization gain" achieved by architecture personalization. Averaged over all clients, we observed personalization gains of 4. 4.70%, 7.84%, and 4.19% on CIFAR1O, CIFAR100, and CINIC10 datasets, respectively.
2 Related works
Heterogeneous Architecture for FL Heterogeneous neural architecture has primarily been explored personalize models for system/hardwwe heterogeneity in cross-device FL scenarios. For instance, the work by Lin et al. (2020) accomplishes the task of heterogeneous model aggregation by forming clusters of clients assigned a predefined model and allowing for heterogeneous models across clusters. Model aggregation is based on cluster-wise aggregation followed by a knowledge distillation from the aggregated models into the global model. Another work, HeteroFL (Diao et al. 2021), aggregates heterogeneons models by assigning static sub-parts of the global model based on computat ion budgets and aggregating the common parameters across different clients. Similarly, in the work {Luo et al. 2021), a limited channel-wise search is performed to assign sub-models that meet clients' efficiency budgets, followed by partial weight aggregation at the server. These random or static partial heterogeneous model aggregation schemes susceptible to client drifts, which arise from inconsistencies between local model architectures and the global model architecture. FedRolex (Alam et al.) 2022) is a recent approach that addresses client drifts by allowing the training of sub- models extracted via a rolling window from a large pre-defined global model. However, it cannot maintain static heterogeneous local models across silos. Additionally, all these works primarily focus on addressing system heterogeneity challenges FI. by considering the computing capabilities of training devices. The sub-models are extracted randomly, statically, or based on a rolling window. In FL, where data visibility is limited, determining which architecture would be suitable for each client based on their data distribution is a challenging task that requires exploration. Our proposed method focuses on tailoring architecture to individual clients' data distributions by scarching for mode--hetcrogencous architectures across silos. By prioritizing personalized architecture design, we aim to address the challenge of data distribution variations and enable more effective model customization in FL settings.
Neural Architecture Search for FL Neural Architecture Search (NAS) has gained momentum in recent literature for searching global model in a federated setting. FedNAS (He et al. 2020b) explores the MileNAS
3

---

Published in Transactions on Machine Learning Research (11/2023)
solver (He et al. 2020d with the Federated averaging algorithm (McMahan et al.
2017 to search for a global model. Direct Federated NAS (Garg et al. 2020) investigates the compatibility of DSNAS (Hu et al. 2020
with Federated learning for searching a global model. Work (Zhu & Jin 2021) utilizes evolutionary NAS
to design a global model. Work (Singh et al. 2020) explores differential privacy using the DARTs solver (Liu et al.; 2019) to examine the trade-olf between accuracy and privacy in a global model. Work (Xul et al., 2020) start with a pre-trained handcrafted model and continue to prune the model until meets the efficiency budget. FedPNAS (Hoang & Kingsford 2021) divide the model architecture into global and personal components and search for the personal component for personalization on identical and independent (ITD) vision tasks. However, this approach remains unexplored in the non-I.I.D data distribution settings of FL. FedPM (Isik et al. 2023) addresses the communication efficiency challenge by learning a stochastic binary mask of a dense net work with fixed weights. FedorAS (Dudziak et. al.) 2022) addresses the system heterogeneity challenge in a cross-device setting by searching architectures in cluster cluster/tier- /tier-based settings to meet an efficioncy budget. FedMN (Wang et al.] (222)) proposes a cluster-based module personalization approach. Cluster-based approaches often require information exchange regarding data distributions to form clusters, which may not be feasible due to privacy concerns. This paper aims to search for an entire personalized neural architecture for each client that remains unknown on the server.
3 Preliminaries, Motivation, and Design Goals
In this section, we introduce the state-of-the-att methods for personalized federated learning, discuss the motivation for personalizing model architectures, and summarize our design goals.
Personalized Federated Learning A natural formulation of FL is to assume that among c distinct clients, cach client & has its own distribution Pk. draws data samples from Pk. and aims to solve a supervised learning task (e.g,, image classification) by optimizing global model w with other clients collaboratively. At high-level abstraction, the optimization objective is then defined as:
min G (Fi A), ((wA)))) w
(1)
where (w, A) is client k's local objective function that optimizes the weight parameters of the global model A and G is the global model aggregation function that aggregates each client's 's local objectives. For example, for FedAvg [McMahan et al. 2017]. G(.) would be weighted aggregation of the local objectives, Σi Pr (w, A), where Σi= Pk = 1. However, as distributions across individual clients are typically heterogeneous (i.e., non-I.I.D.), there is a growing line of research that advocates reformulating FL as a personalization framework, dubbed as personalized FL (PFL). In PFL, the objective is redirected to train a personalized model 2y for device in that performs well on the local data distribution while also learning from other silos, such as via aggregation function G.
To solve this challenging problem, various PFL methods are proposed, including FedAvg with local adaptation (Local-FL.) (Cheng et al.] 2021),
MAML-based PFL (MAMI-FL) (Fallah et al.] 2020 Jiang et al. 2019)
clustered FL (CFL) (Ghosh et al.
2020 Sattler et al. 2021 Wang el al. 2022 Dudziak et al. 2022) Liang
et al. 2019 Pillutla et al.] 2022),
federated multitask learning (FMTT.) Smith et al. 2017) and knowledge
distillation (KD) (Lin et al.) 2020
He al. 2020a).. Additionally, Neural Architecture Search (NAS) has gained momentum in recent literature to scarch for personalized model in federated setting. FEDPNAS
(Hoang & Kingsford 2021 personalized layer-based FI. (PI-FT.), divides the model architecture into global
and personal components and searches the personal component's 's architecture for personalization. Further,
FedMN (Wang et al.) 2022 follow clustered FL (CFL) approach to address the data heterogeneity challenge
in FL.
Motivation for Neural Architecture Personalization Distinct from these existing works on PFL, we propose a new approach to instead personalize the entire model architecture for each client. We are motivated by one critical potential benefit, that is, the searchod architecture for each client is expected to fit its own distinct distribution, which has the potential to provide a substantial improvement over the existing PFT. baselines that only personalize model weights. In addition, a personalized architecture search allows
4

---

Published in Transactions on Machine Learning Research (11/2023)
SPIDER Trainer:
SPIDER Searcher:
Architecture- Hew.rrocueeeettttooogeeenuuuuu
each client search its own neural architecture on the fly progressively
Bi-level Optimization Framework
replace on the fly
Regularzation
Client
Global model (Supemee)
Local modal (Chik)
replace
fly
weight oh serinng-beseee besedd Regularrization
Server
Client
Phase 1:
Phase 2:
Phoas Fbxing
Fixing for warming up
Progressive Searching Personalized Architeclure
Architecture
SPIDER Weave (search) Different Web (neura architecture) for Each Client
Time
Figure 1: Illustrationn of SPIDER framework. SPIDER consists of two components: Troiner (global and local model parameter optimization) and Searcher (local architecture search). Searcher weaves (searches) different web (newral architecture) for each client in three phoses: Phase (Local Architecture Pre-traints Fre-training), Phase 2 (Local Architecture Search), Phase 3 (Local Archilecture Training). To illustrate the local archilecture search, we represerd a simplified cell with nodes where each node represenis latent feature map and each edge is associaled with 3 operations (blue, orange ad yellow). We provide details about the cells of DART RTS architecture search space in Section 4-2 and Appendiz B. and show the heterogeneous client architecture cells obtained through the search process in Figure
the clients to even keep their local model architectures private in sense the server and other clients neither knows the architecture nor the weights of that architecture.
Design Goals Our goal clients in FL. In this context, the limitation of existing personalized FL methods is obvious: Local-FL and MAML-FL need every client to have the the clustering step requires all clients in one cluster to share homogencous model architecture; PL-FL can only obtain heterogeneous architectures for small portion of personalized layers, but it does not provide an ceattectur--annostee method to determine the boundary of personalized layers in an automated mechanism; FMTL a rogularization-based method which cannot perform regularization when architectures are heterogeneous across clients; KD has an unrealistic assumption that the server has public dataset as the auxiliary data for knowledge distillation.
enable a complete model's personalized neural architecture search for all
architecture to perform local adaptations; In CFL,
To circumvent these limitations, our goal design an archilecture-persnnnleedd FL framework with the following requirements:
R1: Allowing heterogeneous architectures for all clients, which can capture data heterogeneity;
R2: Searching and personalizing the entire architecture space to avoid hewristic huper--poramete search traditionally required to determine the personalized layers and globally shared layers;
R3: Requiring no auzlliary data the client- or server-side (unlike knowledge distillation-baa based PFL appronches);
We now introduce SPIDER which meets the above requirements in a unified framework.
4 Methodology: SPIDER
4.1
Overview
The overall framework of SPIDER is illustrated in Figure 1 In this framework, each client maintains models: homogeneous global model for collaborative training with other clients, and heterogeneous local
5

---

Published in Transactions on Machine Learning Research (11/2023)
model that initially shares the same super architecture space the global model. SPIDER is formulated as rcchiteduure-persnnlliez bi-level optimization problem (Section 4.2) It consists of two main compo- nents: SPIDER Trainer (Section 4.3) and SPIDER-Searcher (Section 4.4). SPIDER Trainer is an architecture- penssonauurr- training framework that enables the collaborative training of hetcrogeneous neural architectures across clients. Tt utilizes alternate bi-level optimization and weight sharing between personal- ized architectures and the global model to facilitate federated training. SPaDER-Scarcher, on the other hand, is designed to dynamicallly adjust the architecture of each client's 's local model. It uses a novel neural architecture search (NAS) method that progressively searches for optimal local subnets by perturbing op- erations at the accuracy level. This allows for personalized architecture search based on each client's 's local data distribution. Each client's 's local model goes through three phases: pre-training to warm up the initial model, progressive neural architecture search, and final training of the searched architeccure-personnliaall model. These phases are depicted in Figure [
SPIDER. can meet the design goals R1-R3 introduced in Section 3 because 1) each client performs inde- pendent architecture personalization using its own private data (R1), 2) the SPIDER's search space is not restricted to portion of the model (R2), and 3) no auxiliary data is used to assist the search/traie process (R3).
4.2 SPIDER Formulation: Architecture-persoonee
Bi-level Optimization
SPIDER aims to personalize (weave) a distinct neural architecture (web) for each client. To accomplish this, SPIDER leverages two models, namely local model As and global model A, at each client & E 21,2.... ..., c).
The two- woo-model design has been shown effective in personalized federated learning (Li et al., 2021 Chen
ct al. 2022). However, the prior works only target predefined architectures. Our work proposes to facilitate personalization at the architectural level by neural architect ure search (NAS). To achieve this, we use DARTS' search space (Liu et al., (219)) for both A and As. Essentially, DARIS' search space consists of repetitions of cell-based microstructures. Each cell can be considered as a directed acyelic graph (DAG) with N nodes and E edges, where each node represents latent feature map, and each edge can be associated with multiple operations O (e.g. "skip connect', 'sep conv 3x3"). For instance, in Figure D during Phase 1, the cell is depicted with 4 nodes and 3 operations per edge for the purpose of illustration. At any specific edge i, each operation j has its corresponding learnable architecture parameter 0aj E {0, 1}. More specifically, the output at the edge is constructed by x = Σj aizo,(x²), where x² is the input at the edge i, (z(x)) is the output of function oy(.) (operation index j in the operation candidate search space O) applied to the input x², ar is the associated architecture parameter and x is the mixed operation output at the edge i. We introduce A and A& to be collection of all aij and 0ijy for Vi,j, respectively. Initially, all diy are fixed to for all E F. and j E O in A and As. This search space searches an architecture among 4 possible architecture configurations for each client. We provide details of the DARTS search space in Appendix B.2
In most PFL works (Li et al. 2021). the global model architecture and the local model architectures are fixed (pre-defined). However, SPIDER facilitates each client, to search for its own local architecture Ag- To achieve this objective, SPIDER formulates a local objective F& (e architecture search for As by learning its architecture parameters Gijy and optimize its correspouding model weights ug. Additionally, it also maintains a global objective to update the global model weights. It connects the local and global objectives with distance-based regularization between the parameters of the global model and the local model. More specifically, the SPIDER formulation follows an architecture- bi-level optimization problem for each client k:
which aims to perform neural
min 2y PRER AACA
Ag: A) = Fu Av) + ReggUE, Re))vx,w")
(2)
s.t. w" arg min G (FF A), .... Fx (x(
(3)
WERd
where global model architecture A is parameterized by weight parameters w with size d, and local model architecture Aş is parameterizod by weight parameters 0% with parameter size dy for client k. Reg(e,, w²) illustrates a distance-based regularization between the global and local model. We present the local objective in Equation 2 and the global objective in Equation 3 Further, we use cross-entropy loss function
6

---

Published in Transactions on Machine Learning Research (11/2023)
our Fs(.) objective function. As shown in Equation 2 the lower-level optimization optimizes the global model parameters w", and the upper-level optimization optimizes the local architecture A and its weight parameters Uy. Here the hyperparameter À controls the interpolation between local model weight parameters VA and global model weight parameters w" models. Since A& evolves as learning progresses, it becomes a subset of the global model A, denoted as A S A, and RRde C Rd
4.3 SPIDER Trainer: Federated Training on Heterogeneous Architectures
SPIDER facilitates each silo to search and train an architecture better suited for its specific data distribution. However as the local architecture Aę evolves due to our proposed formulation, becomes a subset of the global model A, denoted as A S A, and I C R In such a sctting, the challenge of the proposod global
local optimization objective is to devise the regularization between the architecture-honogeeeee model and architecture- architeetur--beteergeeeuuu local models. To solve this challenge, we propose SPIDER trainer, an peersooaurrrpesssnaaiiiddd training framework that can collaboratively train heterogencous local neural architectures across clients.
To clearly show how SPIDER handles the optimization difficulty of Equation 2 we first downgrade the objective to the case that all clients use predefined (fixed) heterogeneous architectures (derived from the Supernet A). More specially, we reduce the aforementioned optimization framework in Equation and Equation to the following:
ha(U Au; A) F. As) + 2 2 211 w whsnare
min uy 2
(4)
s.t. w" Earg min G (F. A). ..., Fx(w,A)),
(5)
weRd
where local model's weights o are regularized towards the global model w via l2 norm regularization,
where w Wepeere are the weight parameters of the operation set space of A overlapping (sharing) with As for client k. Note that, now, only 22 needs to be optimized in Equation [ while A is fixed during the optimization. We summarize the pseudo-code of the optimization procedure as SPIDER-Trainer in Algorithm At each device k, once we receive the aggregated global model w", we calculate w (Line #12 and Line #14). Next, we solve the local sub-problem of G(-) approximately, that is Fa(w,,A (Line #17). Further, each client & solves its local objective given Equation | (Line #18). The updates to w" are computed the same way as performed in the standard federated setting, such as FedAvg (MeMahan et al.
2017 (Line #6).
(1) Enabling regularization between arbitrary personalized architecture and the global model SPIDER- - connects each personalized model with the global model by enabling the regular- ization between two different architectures: an arbitrary personalized architecture for the local model As and the global model with Supernet A. This is done by weight sharing. w is used to regularize a subnet (Ax) model parameters Us towards the global model thared/commnn parameters w w kahare" as shown in Equation
(2) Avoiding hetcrogencous aggregation SPIDER-Traine avoids the aggregation of heterogencous model architectures the server side. As such, no sophisticated and unstable aggregation methods are required (e.g. knowledge distillation (Lin et al. 2020), clustering (Dudziak et al. 2022) etc.), and it is flexible to use other aggregation methods beyond FedAvg (e.g. Karimireddy et al. 2020 Reddi et al. 2021 to update the global model.
4.4
Personalizing Architecture
Although SPIDER Irainer is able to collaboratively train heterogeneous architectures, manual design of the architecture for each client is impractical or suboptimal. As such, we further add a neural architecture search (NAS) component, Searher, in Algorithm 1 (Line #10) to adapt A to its local data distribution in a progressive manner. Furthermore, for the architectural evolution of the local model, please note that ai takes discrete values. Therefore, most NAS works utilize the continuous relaxation of these discrete variables
7

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm SPIDER Trainer
1: Initialization:
initialize e number of clients; AA is the global model with model weight parameters wo; A is the local model with architecture parameters aijx and model weight parameters dy associated with the k-th client; is the total number of rounds; t, is the number of rounds to start the architecture search; T is the recovery periods the units of rounds; (.).) is the cross-entropy loss function calculated at the local data of client k.
2: Server executes:
3: for each round == 0,1.
do 4: for each client kt in parallel do
5: w. (+1 f- CCieent.ocalSeacch/( w',
6:
wht1 Ex=
K
No w+
7:
8:
function ClientLocalSearchhe w', 0): // Run on client
9:
Set w' w
10:
Search Local model: A&tt SPIDERR-SarchherAA..
11:
Raharn =] // empty set
12:
for each cny in A do
13:
if azj is nonzero then
14:
(1 w Kature
A kangre : w(g) // Append only those weight parameters of wt which have an overlap-
ping operation set space (edge and operation j) between A and As architectures for each epoch p do
15:
16:
for minibatch in training data do
17:
Update Global model: w.t wi - The F. Update Local Model: U return w
18:
vi me (V. (uf. Ab+2)
) + A(ol w
to ensure diffcrentiability of the aij parameters (Liu et al. 2019 He ct al. 2020d).. However, SPIDER employs the Pertubation-based SPIDER-Searcher learn Osy a way that only requires evaluation-bassd search rather than training- based search (optimizing agy). As shown by (Wang al. 2021), the evaluation-basee search can avoid suboptimal architecture often caused by differentiable NAS. We now present the details of the SPIDER-Searcher.
Progressive Neural Architecture Search SPIDER-Searcher dynamically changes the architecture of As during the federated training process. This is feasible because the weight sharing- based regularization can handle an arbitrary personalized architecture (introduced in Section 4.3 Due to this characteristic, SPIDER Searcher
search A₃ in progressive manner (shown in Figure 1 Phase 1: At the beginning, Av is same as the Supernet A in architecture design. The intention of SPIDER-Searcher in this phase is to warm up the training of the initial Aş on the client's 's distribution, therefore, the architecture Aş does not change for some initial foderated rounds; Phase 2: After warming up the Ag, SPIDER-SSardeee edgeeb search gradually. In each edge search, only the operation with the highest impact on the accuracy kept.. It also uses a few rounds of training as recovery time before proceeding to the next round of edge search. This process continues until all edges finish searching; Phase 3: After all edges finish searching, SPIDER. Searcher does not change the client's architecture Ak. This serves the searched architechure-perssnalleed model. This three- three-phase procedure is summarized as Algorithm 2 Now, we procced to claborate on how we calculate the impact of an operation on the Supernet.
performs
final training of
Operation-level perturbation- selection In Phase we specify selecting the operation with the highest impact using operation-level perturbation. More specially, instead of optimizing the mixed operation architecture parameters using another bi-level optimization DARTS (also known as gradient- based architecture search) (Liu et al.] 2019) to pick optimal operation according to the magnitude of a parameters (oansssmmmma (magnilude-buse basedd seleclion), we assign a constant value lo a and use the impacl of an operation on the local validation accuracy (perturbation) as a criterion to search on the edge. We use this simplified method as much more efficient given that only requires evalnation-basod search rather than training- based search (optimizing a.y). Further, this method avoids suboptimal architecture (Wang et al. 2021) lead by magd selection in differentiable NAS.
8

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 2 SPIDER-Searcher
1: Search Space: A& the local model with architecture parameters Cijja associated with the k-th client; & is the superset of all edges {l.... }}; is the remaining subset of edges that have been searched yet; and each edge has multiple operations {1,..., o}; ts is the number of rounds to start the architecture search; T is the recovery periods the units of rounds; t the current round number
2: function ProgressiveNASI, (SAL L, т, i)
3: if t 2 L, and (t is multiple of 7) and [E.] then
4: 1 RANDOM (E,) // random selection
5: // searching without training
6:
for all operation j edge do evaluate validation accuracy Au when operation day is set to zero (removed)(ACC for the selected edge i, select operation j for which (ACO)eg)) is highest
7:
9:
update architecture A set ag = and au = 0 Vi (where # (), remove i from E,
10:
else
11:
return A directly
12:
return updated Axt after selection
5
Experiments
This section presents the experimental results of the proposed method. All our experiments are based on non-lID data distribution among clients. To generate this non-llI data distribution across clients, we employed the widely used latent Dirichlet Distribution (LDA) (He al. 2020c Yurochkin et al.) 2019).
5.1 Experimental Setup
Tasks and Datasets We perform an image classification task on three well-known datasets, CIFAR10,
CIFAR100 and CINIC10. CIFAR10 dataset (Krizhevsky et al. 2009) consists of 60000 32x32 color images in 10 classes, with 6000 images per class, and CIFARI00 dataset (Krizhevsky et al. 2009 consists of 60,000 images in 100 classes, with 600 images per class. CIFAR100 has more classes and comparatively fewer data per class, therefore, it considered more challenging than CIFAR10. In addition, CINIC10 consists of 270,000 32x32 color images in 10 classes, with 90,000 images per train, test, and validation subset. CINIC10 dataset (Darlow et al., 2018) is a larger dataset and includes images from ImageNet as well. We generate non-lll) data across clients by exploiting LDA distribution with parameter (a 0.2) for the training data of CIFAR1O, CIFAR100, and CINIC10 datascts. Since need validation data for SPIDER-Scarchor split the total training data samples present each client into training (60%), validation (20%), and testing sets (20%). We use this 60/20/200 20/200 train/valid/ten split during Phase 1 and 2 of the SPIDER Training. Once each client has selected the architecture, they combine the validation data with training data in Phase 3 and use for training. The test set remains the same throughout the training. For all other personalization schemes used for comparison, we split the data samples of each client with 80 training and 20 % test for fair comparison. We provide the LDA data, distribution class-split, Hlpee-Paramctee Scarch Sot, and Architecture Search Space details in Appendix A
Implementation and Deployment We implement the proposed method for distributed computing with nine nodes, each equipped with an NVIDIA RTX 2080Ti GPU card. We this cross-silo FL setting with one node representing the server and cight nodes representing the clients. These client nodes can represent real-world organizations such as hospitals and clinics that aim to collaboratively search for personalized architectures for local benefits such as higher accuracy in a privacy-preservinn focus on the cross-silo sctting where clients have ample computational resources and data heterogeneity is a main concern (Huang al. 2022). However, we provide an analysis of the computational cost of SPIDER. in the Appendix
FL manner. Primarily, we
9

---

Published in Transactions on Machine Learning Research (11/2023)
5.2 Results
Here, we report the comparison of our proposed method SPIDER with the other state-of-the-a art personalized methods; Ditto, perFedAvg, local adaptation, KNN-Per and FedMN. Since these schemes use pre-defined architecture, we use the Resnet18 model because of its comparable model size. Since we are exploiting DaRes-based search space, we also use a DARTS model (Liu et al., 2019) searched on CIFAR10 dataset as our base model for 12 regularization- based personalization method Ditto. Furthermore, we also provide a comparison with Centralized NAS and Global NAS (FodNAS) in Appendix D
Table 1: Average (local) test Accuracy and fairness (standard deviation across clients) comparison of SPIDER with the representative stale-of-the-art personalization techniques CIFAR1O, CIFAR100, and CINIC10 Dolasels
CIFAR10
CIFAR100
CINIC10
Mechud
Average Accuracy
Parameter Size
Standard Devlation
Average Accuracy
Parameter Slze
Standard Deviation
Average
Parameter Size
Standard Deviation
Accuracy
Local Adagtation Re
90.39+0.17
11M
1.9
80.2--0.6
90.96=0.96
Ditto Beaket18
88.91=0.43
11M
2.7
61.06=0.29
11M
2.4
90.42=0.73
11M
5.5
porPedlug Reslfet18
90.9110:43
11M
1.6
56.56:0.23
11M
70.930.28
Dibto DARTS
89.82:0:21
33M
1.9
00.18:0:27
33M
23
83.61=0.28
3.3M
6.4
KXW-PPr BeaNeti8
91.26:0.91
11M
1.6
67.06=0044
11M
25
85.00=0..0
11M
5.8
FodMy: Resllet18
88.48:00.25
11M
2.1
61.02=0044
11M
3.4
81.41=0611
11M
6,7
SPIDER
94.55=0.12
2.4M
1.5
72.36=0007
2.3M
2.4
87.25-t0.65
2.1M
3.5
5.2.1 Average Test Accuracy and Fairness (Standard Deviation across Clients)
As shown in Table [ proposed method, SPIDER, successfully achieves the objective R1 by person alizing archilectures for each client, surpassing representative state-of-the-arl personalization methods such as Ditto, local adaptation, perFedAvg, FedMN, and KNN-Per on three image classification datasets: CT- FARIO, CIFAR100, and CINICIO. Among these methods, KNN-Per obtains the second-highest accuracy for all three datasets. However, SPIDER outperforms KNN-Per by a margin of 3.29%, 5.31%, and 1.59% on CTFAR1O, CIFAR100, and CINIC10, respectively. Moreover, the performance gain of SPIDER over the hypernetwork- based architocture scarch method, FedMN, is quite substantial, reaching 6.10%, 11.31%, and 5.52% on CIFARIO, CIFARI00, and CINIC1O, respectively. We altribute this improvement to SPIDER's ability to adapt to unseen data more effectively by tailoring the entire architecture for each individual client based on their specific data distribution.
In the context of personalization, the standard deviation (std) of the local test accuracy across different. silos is also an important metric, as it reflects the fairness the method across silos. In Table [ we observe that SPIDER achieves similar or lower standard deviation compared to other basclines while providing higher average test accuracy. This demonstrates the effectiveness of SPIDER in achieving both improved average accuracy and lower disparity across silos.
5.2.2 Architecture Heterogeneity and Personalization Gain
SPIDER helps each client search for its own architecture tailored for its own specific data distribution. Due to data heterogeneity across clients, we observe architectures to be heterogeneous across clients as shown in Figure 2 Essentially, we visualize the reduction cells searched for CIFAR100 in Figure P Tt can be seen from Figure 2 that the searched cells are odge-wise and operation- wise heterogencous from client to client. Hence, we achieve the objective R2, the search and training of heterogeneous architectures across clients. The objective R3, dependence auxiliary data, is obtained by the algorithmic framework of SPIDER as it does not rely on any auxiliary data. A byproduct of using DARTS scarch space is that the searched local model size obtained with SPIDER is quite smaller.
To further investigate whether the architecture searched by one client best suited for its own data dis- tribution, we perform experiments where the final architecture of one client is applicd to another client's data, and the architecture fine-tuned on that data. We denote the local accuracy obtained on client i's data using architecture j as pi-- We calculate the personalization gain or drop (if negative) of other clients'
10

---

Published in Transactions on Machine Learning Research (11/2023)
Silo
Silo
dep comneet
(k:1)
sep LO.,0O00,.
Mp
c_16-11
sop_0oeK,
Silo
sep_
Silo 5
(k-2)
ship
Skip orrns
sep.0ee 333
11-7)
Mip ORREAT
skip
Mip contect
sep_sten_
OOREAT
Silo
3d
skip couENt
Ikp connact
Silo 3
Jap con, 363
3e3
(k-2)
1x
skip contect
50p coe: 3k3
RepLeuY 333
Tel
Silo
connect
Silo 7
c. (b-2)
coty Ixc
OVEY Sel
363
(k]
conr 1s3
303
cared
(I-1)
sep 363
spp nan
3,3
Figure Searched Architectures (Reduction Cells) for CIFAR100 Dataset: Each normal cell takes the outputs of previous cells, cell k and cell 1, 1, as its input. Each cell contains seven nodes: two imput nodes, one output node, and four intermediate nodes inside the cell. The output node concatenates all intermediate nodes' output depth-wist wise. It can be observed that the searched cells are edge- wise and operation wist heterogeneous from client to client.
architectures on client i's data using the following formula:
g: ΣS c--1 0 (Pit Pit)
(6)
This calculation is performed across all silos. The quantity 9, represents the personalization gain of architec- ture compared to other clients' architectures on its own dataset. We further calculate the personalization lization
gain of the SPIDER scheme as the mean of all ga, i.e. Σ gi where is the total number of clients. We fino-tune the architecture learned through SPIDER on other clients' data for 30 epochs and report the best accuracy achieved. The values of Pıy and 9, for all i and j are reported in Figure As shown in Figures 3a 35. and 3c we observe that for all three datasets, the P matrix is diagonally dominant, indicating that for the majority of clients, their searched architectures outperform the other architectures. Similarly, the per- sonalization gain 9, of each client is positive, as shown in Figures 3d 3c and 3i Averaged over all clients, the average personalization gains on CIFAR1O, CIFAR100, and CINIC1O datasets are 4.70%, 7.84%, and 4.19%, respectively. This highlights the importance of architecture personalization. which can be potentially more effective than weight personalization lization alone, as demonstrated by our empirical results.
6 Conclusion
We proposed SPIDER, an algorithmic framework that aims to search personalized neural archilectures for each client in Federated Learning. SPIDER uses progressive NAS to search personalized local archi- tectures and weight-sharing- based global regularization to exchange information between the architecture- homogencous global and sraiteetuur-heterogennnt local models. Our experimental results demonstrate promising prediction performance compared with other state-of-the-art personalization methods.
basodd
11

---

Published in Transactions on Machine Learning Research (11/2023)
52
Ed
25
%4
F2
52
F2
22
57
0
50
22
22
D0
26
23
54
22
20
57
25
55
20
23
14
55
65
S4
12
R5
25
22
24
2C
52
25
20
25
0
50
50
60
PM
R2
42
61
4
R2
12
42
74
74
74
32
24
74
60
50
60
60
K5
6
66
ET
cn
83
22
92
80
775
(a) CIFAR10: local accuracy Poj Vi.j
(b) CIFAR100: tocal accuracy Psy Vi.j
(c) CINICIO: locat accuracy Psy Vi.j
92
73
61
05
7.6
D.C
9:7
6.
7.5
6.5
18
0
3.3
4:3
F2
3.2
2.2
4
2.7
2.3
42
64
62
62
5.2
2.2
7.5
4.7
6.2
4
42
2.5
3.2
2.0
2.5
5.1
3.3
54
9
x+
32
2.2
3.5
94
20
21
0.4
9.2
21
20
6.2
4.7
3/2
6.7
4.7
0
2.3
3.2
5.3
49
49
:
4.1
4.3
4.
4.4
4+
4.1
35
66
37
A1
4.2
5.1
5.4
(d) CIEAR1O: (pu P5:) Vj and ga
(c) CIEAR100: (pa Pjs) Vj and y:
( CINICIO: (pu Pjs) and gi
Figure 3: Figures 3a
3b and
3c represent local accuracy values Pij, obtained by finetuning elient j architecture
on client i's data on CIFAR1O, CIPAR100 and CINIC10 datasets, respectively. Figures Sd 8d and 31 represent
the accuracy gain (pu- - Pig) Vj and the resultant personalization gain values (g.) for client obtoined CIFAR1O, CIFAR100 and CINIC1O dalasels, respectively.
Future Works: While the SPIDER. framework exploits NAS as tool to personalize the client architec- tures and architecture weight paramcters in Federated Learning, there are several limitations worth further investigation. Future studies could 1) extend SPIDER to cross-device FL by making it compute-effciint, 2) examine hardware heterogeneity with the data heterogeneity challenge of FT. in the context of architecture personalization, and 3) investigate other modality tasks such as eoxt-modeling.
Acknowledgements
This material is based upon work supported by ONR grant N00014-23-1-2191, ARO grant W911NF-22-1 0165, NSF grant DMS-2134148, and gifts from Intel, Qualcomm, and Konica Minolta.
References
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang, Fedrolex: Model-heterogeneooss federated learning with rolling sub-model extraction. Conference on Neural Information Processing Systems (NeurIPS), 2022.
Huili Chen, Jie Ding, Eric W Tramel, Shuang Wu, Anit Kumar Sahu, Salman Avestimehr, and Tao Zhang, Self-aware personalized federated learning. Advances in Neural Information Processing Systems, 35:20675- 20688, 2022.
G Cheng, K Chadha, and J Duchi. Fino-tuning in federated learning: simple but tough-to-bent baseline. arXiv preprint Xrx::22008. 07313, 2021.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet cifar-10. arviv preprint 1ix:18800.00000 01000,, 2018.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heteroft: Computation and communication efficient federated learning for heterogeneous clients. International Conference on Learning Representalions (TCLR), 2021.
Jie Ding, Vahid Tarokh, and Yuhong Yang. Model selection techniques: An overview. IEEE Signal Processing Magazine, 35(6):16-38 34, 2018.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Lukasz Dudziak, Stefanos Laskaridis, and Javier Fernandez-Marquue, Fedoras: Federated architecture search under system heterogeneity. NeurIPS workshop Federated Learning: Recent Advances and New Chal- lenges, 2022.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. 34th Conference on Newral Information Processing Systems (NeurIPS), 2020.
Anubhav Garg, Amit Kumar Saha, and Debo Dutta, Direct federated neural architecture search. arXiv preprint Xrx::22010.000 06222,,, 2020.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. efficient framework for clustered federated lcarning. 34th Conference on Neural Information Processing Systems (NeurIPS), abs//200.040088, 04088 2020.
Chaoyang Hc, Murali Annavaram, and Salman Avestimchr. Group knowledge transfer: Federated lcarning of large cnns at the edge. 34th Conference on Neural Information Processing Systems (NeurIPS), 2020a.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Towards non-iid and invisible data with fednas: federated deep learning via neural architecture search. CVPR 2020 Workshop on Neural Architecture Search and Beyond for Representation Learning, 2020b.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. NewrIPS Federated Learning workshop, 2020c.
Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020d.
Minh Hoang and Carl Kingsford. Personalizod neural architocture search for federated learning. In 1st NewrIPS Workshop on New Frontiers in Federaled Learning (NFFI, 2021), 2021.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Dsnas: Di- rect neural architecture search without parameter retraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12084-12092, 2020.
Chao Huang, Jianwei Huang, and Xin Liu. Cross-silo federated learning: Challenges and opportunities. arXiv preprint Xiv:22200. 12949, 2022.
Berivan Isik, Francesco Pase, Denix Gunduz, Tsachy Weissman, and Michele Zorzi. Sparse random networks for commuicationn-effcieett federated learning. International Conference on Learning Representations (ICLR), 2023.
Yihan Jiang, Jakub Koneený, Keith Rush, and Srceram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. Ar abs/1909.1248s 2019.
Sai Pranccth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank J. Reddi, Scbastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Youngkee Kim, Won Joon Yun, Youn Kyu Lee, Soyi Jung, and Joongheon Kim. Trends in neural architec- ture search: Towards the acceleration of search. In 2021 International Conference on Information and Communication Technology Convergence (ICTC), pp. 421 424. IEEE, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of the 3rd MLSys Conference (MLSys), 2018.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust. federated learning through personalization. In International Conference Machine Learning, pp. 6357-636773 66676.68. PMLR, 2021.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. NewrIPS Workshop on Federaled Learning, 2019.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust. model fusion in federated learning. 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. International
Conference Learning Representations (ICLR), 2019.
Mi Luo, Fei Chen, Zhenguo Li, and Jiashi Feng. Architecture personalization in resource-constraiee feder- ated learning. 1st NeurlPS Workshop on New Prontiers in Federated Learning (NFFL), 2021.
Abdul Majeed, Xiaohan Zhang, and Scong Oun Hwang. Applications and challenges of federated learning paradigm in the big data era with special emphasis on covid-19. Big Data and Cognitive Computing, (4):127, 2022.
Othmane Marfoq, Giovanni Neglia, Richard Vidal, and Laetitia Kameni. Personalized federated learning through local memorization. In International Conference on Machine Learning, pp. 15070 15092. PMLR, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communictionn-ffiitee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273 1282. PMLR, 2017.
Krishna Pillutla, Kshitiz Malik, Abdel- Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In International Conference on Machine Learning, pp. 17716-17756 PMLR, 2022.
Sashank J. Reddi, Zachary B. Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneený, San- jiv Kumar, and H. Brendan McMahan. Adaptive federated optimization. International Conference on Learning Representations (ICLR), 2021.
Felix Sattler, Klbus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. 1EEE Transactions on Neural Networks and Learning Systems, 32:3710-3722. 2021.
Ishika Singh, Haoyi Zhou, Kunlin Yang, Meng Ding, Bill Lin, and Pengtao Xie. Differentially-priiate federated neural architecture search. Xrxiv preprinl arXiv:2006. 10559, 2020.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S. Talwalkar. Federated multi-task learning. NIPS, 2017.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking archi- tecture selection in differentiable nas. International Conference on Learning Representations (TCI.), 2021.
Tianchun Wang, Wei Cheng, Dongsheng Luo, Wenchao Yu, Jingchao Ni, Liang Tong, Haifeng Chen, and Xiang Zhang. Personalized foderated learning via heterogencous modular networks. In 2022 IEEE Inter- national Conference on Dala Mining (ICDM), pp.
IEEE, 2022.
Mengwei Xu, Yuxin Zhao, Kaigui Bian, Gang Huang, Qiaozhu Mci, and Xuanzhe Liu. Federated neural archilecture search. Xivi preprint "XXi:2002.063552 2020.
Jinliang Yuan, Mengwei Xu, Yuxin Zhao, Kaigui Bian, Gang Huang, Xuanzhe Liu, and Shangguang Wang. Resource-aware federated neural architecture search over heterogeneous mobile devices. IEEE Transactions on Big Data, 2022.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Hangyu Zhu and Yaochu Jin. Real-time federated evolutionary neural architecture search. IEEE Transactions on Evolutionary Computation, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2023)
A Appendix
A Data Heterogeneity
A.1 Data Heterogeneity
Figure and represents data distribution across 8 clients for CIFARIO and CIFAR100 datascts, respec- lively. Sub-figure Ma and 5a represent the label distribution across clients, where a darker color indicates It can be seen from these sub-figures that the data distribution is hetero-
more images of that class/labes gencous label-wise, c.g. Client with ID O has class and 9 in excess, whercas, Client with ID 7 has more samples of class and 7 for the CIFAR10 dataset. We observe similar trend of heterogencity for CIFAR100 well, where have 100 class labels distributed across 8 clients via LDA distribution with parameter 0.2. Furthermore, the sub-figures 4b and 5b represent the total number of data samples preset at each client. These figures illustrate that the number of samples is also varying across silos. However, the variation in the total number of samples by silo is more prominent for the CIFAR10 dataset.
distribution
3951,
0.6
8000
3.5
6000
9.4
5000
0.1
0.2
3000
2000
1000
Clantnumbar
Ciinnt number
(a) Label distribution per client
(b) Image distribution per client
Figure 4: CIFARIO: LDA distribution (o=00.2) across 8 clients (Seed 9). We can see that the data distribution is beterogeneous in both label and image distribution clients.
label distribution
Min 5425, Max 6694
7000
0.08
15
6000
24
0.08
5000
4000
0.64
2000
83
88
2000
76
0.02
0.00
Chuntnumbar
(a) Label distribution per client
(b) Immage distribution per client
Pigure CIFA R100: LDA distrilntion clients (Seed 9). We that the data distribution heteroge- neous in both label and image distribution across clients.

---

Published in Transactions on Machine Learning Research (11/2023)
B Hyper-Parane Param--ar and Architecture Search Space
B.1 Hyper-parametes Search
For empirical results CIFAR1O, CIFAR100, and CINIC10, we use a batch size of 32 for all our experiments. We use LDA distribution with a 0.2 a parameter value. We conduct experiments with two seeds for all methods and report the average values in Table | For SPIDER, we use the first rounds, for SPaDER-Searcher, we use recovery period of 20. Furthermore, we use learning rate in the search range of {0.01, 0.033 for SPIDER. For SPIDER, we used À search from the set of {0.01, 0.1, 1). For the other personalized schemes such as Ditto, perFedAvg, KNN-Per, FedMN, and local adaptation with Resnet we searched learning rate over the set {0.1, 0.3, 0.01, 0.03, 0.001, 0.003}. The reason for having larger set of learning rates for these methods is that we found 0.001 and 0.003 work better for some of these methods. For Ditto, we used À from the set {0.01, 0.1, 0.5, 1, For PerFedAvg, we used the local Ir as a factor of {1,3, 5 of the global Ir. We used 1000 rounds of communication for the reported results and observed that they were sufficient to achieve convergence shown in Figure G For FedMN,
rounds as warmup
used 500 rounds of communication pretraining. After pretraining, the next 500 for federated local training. We report the bost average local test accuracy during the lederated local training phase in Table [ For KNN-Per, we evaluate the global model on the hyper-parameter selected from the set {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} and report the best average local test accuracy in Table 1 We used stochastic gradient descent (SGD) optimizer for all the methods.
Average (Local) Best Accearyy Anspaation
Average ILocal Tost AcOrayy
Bocal) AeetAr
Sotta
TerTeding
3iteo DaTTS
Lecal sctotatiion
Etess Seameld
Partedes
DATTS
Lood. Mastatien Ootte boreet
0.2
(a) Average Validation Accuracy compar-
(b) Average Validation Accuracy compar-
(c) Average Volidation Accuracy com- parison CINIC10 Dataset
ison CIFAR10 Dotasct
CIEAR100 Dataset
mariing Look
230
000
6.0
wmw Mm Am
M
680
(d) Averuge Local Training Loss on CI- (c) Average Local Training Loss on Cl- (f) Average Local Training PAR100 Dataset
Loss on
FAR10 Datasct
CINIC10 Datasct
represent the average validation accuracy comparison between SPIDER and Avr Ditto, and Local Adaptation on CIFAR10, CIFAR100, and CINIC10 datasets, respectively. SPIDER oulperforms the representative baselines on all three datasets. Figures 6d be and 0 illustrate the change in Average local training loss of the local models as SPIDER Search progresses through three phases (Phase (Local Architecture Pre-troining), Phase (Local Archilecture Search), Phase (Local Architecture Training)) on CIFAR10, CIFAR100, and CINIC10 Dalasets, respectively
Pigure Figure 6a
2

---

Published in Transactions on Machine Learning Research (11/2023)
B.2 Architecture Search Space
As mentioned in Section 4.2 we have used DARTS (s2) search space (Wang et al. 2021 in our proposed work. During the search, we are using a total of 8 cells as shown by Figure 7 Each type of cell consists of 14 edges; each edge connects two intermediate representations (node) by a mixture of two operations frequently used in various modern CNNs, sep convolution 3x3 and skip connection, hence the name 's2 DARTS search space' (Wang et al., 2021). In this search space, there are two types of cells: normal and roduction cells. Hence, our supernet A consists of scarch space of normal cells and reduction cells, cach with 14 edges and 2 operations al each edge. As the search progresses via progressive NAS, some operalions and edges are removed based on the perturbation criterion as explained in Algorithm 2 It is important to note that the same cells follow the same construction; therefore, if one operation is selected for one normal cell at a particular edge, the same operation will be selected for the same edge at all normal cells. More specifically, we have 14 learnable edges and 2 operations at each edge a cell, there are a total of 214 possible configurations. Since we have two types of cells, a normal cell, and a reduction cell, there are a total of (2 possible architecture designs. The other DARTS search spaces, s3 and s4 (Wang et al. 2021) have larger search space, e.g., larger operation candidate set, and therefore, require more computational resources.
C Gradient-based NAS as SPaDeR-Searcher
SPIDER is formulated as bilevel optimization problem as shown in Equation and 3 where the lower- level optimization optimizes the global model parameters w", and the upper-level optimization optimizes the local architecture A& and its weight parameters 4y for client k. For the proposed lower-level optimiza- tion of SPIDER, we exploit the perturbation-bssee neural archilecture search method as it simplifies the architecture search process by selecting architectures based on the evaluation criterion only. This searcher picks optimal operations basod on the impact of its absence (a.k.a perturbation) on the local validation accuracy as explained detail in Algorithm 2 On the other hand, differentiable NAS another widely known method of architecture selection which is often formulated in itself as a bilevel optimization problem with architecture parameters as an upper-level variable and w as lower-level variables. As an ablation study, replace perturbation-basee NAS Searcher with MileNAS Searcher (He et al.) 2020d) which is a gradient-based NAS Searcher. We use its first-order approximation which essentially applies soft op- eration to the architecture parameters a for each edge and updates them by the stochastic gradient step Att = -4,-- To (V,, + An (va A()) at each iteration of local epochs. ptr and F are calculated at the training and validation data, respectively. For simplicity, Aş here represents a collection of all architecture parameters in the local model of client k. and na is the learning rate of architecture param- eters. We select The to be 0.01 and the regularization parameter da as 1. We apply MileNAS solver for the first 400 communication rounds and find the local architectures based on the magnitude of the architecture parameters at each edge, and then train them for the next 600 rounds by following SPIDER formulation given in Equation 4 and 5 As shown in Table 2 the performance gap between the two methods is negligible for the CIFAR100 datasel. However, perturbation-basee NAS is a simpler method as it does not require a gradient step for the architecture search/ /update at each step, and therefore, takes less amount of time in comparison to gradient-based NAS.
Table 2: Average (local) test Accuracy comparison of Perturbation-bused NAS with gradient- based NAS in SPIDER on CIFAR100 dataset.
Method
Average Accuracy
Perturhation. haasrd NAS Wang et al. 2021
72.36
MiLeNAS [ee et al 2020d
72. 10
3

---

Published in Transactions on Machine Learning Research (11/2023)
3x3 conv
Normal Cell x N
Reduction Cell
Normal Cell N
Reduction Cell
Normal Cell x N
softmax
Figure 7: Supernet A search space. Note that since we have 8 cell based search space, N = 2 for our construction.
D Federated Neural Architecture Search versus SPIDER
In this section, we compare the proposed Personalized Neural Architecture search method SPIDER with its two base cases, centralized learning where X 0 and architecture personalization base case where which essentially refers to the setting where local model weights are identical to the global model weights. We also compare SPIDER with the federated neural architecture search method, which searches global architecture in a federated manner and then trains it via FedAvg. This investigation can help us appreciate the benefits of architecture personalization and federated learning in a cross-silo setting.
D.1 Federated Neural Architecture Search
We analyze NAS performance to search for global architecture in federated setting proposed by previous works (He et al. 2020b). To keep the algorithmic comparison fair, we have used the NAS setting as same as SPIDER. Essentially, only one Supernet (DARTS) maintained at each silo. Each silo trains the Supernet following the FedAvg algorithm for 30 rounds of warmup. Next, the neural architecture search is performed following perturbation- NAS, the architecture is obtained and trained in federated manner by the FedAvg algorithm. This can be considered slight variation of FedNAS (He et al., (2200)) where replace MiLeNAS He et al. (2020d) with slate-of-the-arl NAS method, Perturbation--asee NAS (Wang el. al.) 2021) to keep the comparison fair. Figure represents the average validation accuracy across each silo. We obtained total of 68.03% average local test accuracy with global NAS. Although this accuracy is still higher than various PFL baselines, it is lower than SPIDER (72.36%). This shows that a global architecture search may not capture the data heterogeneity in FT. with non-I.I.D data.
NAS the server using the server's test/ sett/viidation data. After
Average Lccul Accuracy
Centralized
0.4
0.2
rourd
200
400
SBC
Figure Average Validation Accuracy comparison of SPIDER with the Federated NAS on CIFAR100 datasel. For SPIDER, we show the resulls of two different values of X iapere-araaneer. First sclup of SPIDER is Centrulized NAS where = 0. The second selling of SPIDER 0.01, which oulperforms centralized NAS, special case of SPIDER (A 0) as well as Federated NAS.
D.2 SPIDER: Personalized Federated Neural Architecture Search
We can observe from Figure X that SPIDER with 0.01 hyper-paranen setting outperforms the Feder- ated NAS. However, is important to explore two base cases of SPIDER. The first setting where À 0 and therefore, has learning between the local and global models. It is where each silo performs local
4

---

Published in Transactions on Machine Learning Research (11/2023)
neural architecture search. Each silo uses only one model, Supernet, and deploys the perturbation-base NAS Searcher locally without any collaboration from the other silos. The Local Supernet has the same hyper-parameters of warmup (30 local epochs) and recovery period of 20 local epochs after every pruning step, followed by local training with a total of 1000 epochs. We obtained 72.36% average local test accuracy with SPIDER, and 69.31% average local lest accuracy with Centralized NAS on CIFARI00 dataset. This highlights the significance of the proposed regularization. This also shows that tailoring architectures to the silo's data distributions even for centralized training can be very powerful. However, collaboration with other silos as is performed by SPIDER, can enhance the prediction performance of personalized architectures at individual silos.
E
Computational Cost Analysis of Training
We propose SPIDER as a cross-silo framework where clients have ample computational resources and data heterogencity is a main concern (Huang et al.) (022). As shown in Table 3 we achieve substantial per- formance improvement in terms of average local accuracy over the state-of- ohe-rrrttttrttt personalization methods. For research purposes, it is essential to compare the computational cost. and wall clock time of the proposed method with the other representative personalized federated learning (PFL) methods. Therefore, peak memory costs and total wall clock time for all methods. Note that we obtain the peak memory cost val- ues for the maximum number of compute required for the model training, which includes forward/bar backward pass size, and parameter size of all the models, global and local model, for a batch size of 32 on the NVIDIA RTX 2080Ti GPU card.
report
We observe the peak compute cost per silo be 4.18GB, 2.14GB, 1.45GB, 1,45B, 2.90GB, 1.45GB, and 1.98GB for SPIDER, FedNAS, Local adaptation, KNN-Per, Ditto, perFedAvg, and FodMN, respectively. Though SPIDER requires relatively higher peak memory for maintaining two models, global supernet and local model, at each silo, it outperforms the state-of-the-art personalization methods substantially. For example, SPIDER yiclds a 6. 6.18% performance gain relative to the second-best. method, KNN-Per, on the CIFAR100 dataset which can be a significant gain for organizations that require high accuracy without the constraiit of small computing cost.
Further, we report the end-to-end time of each method for a total of 1,000 rounds on the CIFAR100 dataset. For implementation, we used GPU NVIDIA RTX 2080Ti GPU cards where each GPU represents physical node with an NVIDIA RTX 2080Ti GPU card on the fedml platform implementation for FI. (He et al. 2020c). Please note that to report FedMN and KNN-Per method results, we used their GitHub code directly where they perform fodcrated lcarning on one GPU in a sequential manner. Therefore, the time comparisons of these two methods might not be fair. Overall, for wall-clock time comparisons, we observe that there are two factors contributing to the wall-clock overhead.
Overhead of Neural Architecture Search (NAS): One notable observation is that methods that use predefined architecture tend to have significantly lower wall clock times. However, this comes at the cost of lower prediction performance. It is important to note that these low wall clock times may not hold true in practical scenarios, especially when dealing with the data-invisibility challenge in Federated Learning (FL).
The data invisibility challenge in FL refers to the distributed learning setup where clients' private data remain invisible the server. As a result, from the server's perspective, it bocomes unclcar how to select a pre-defined architecture from pool of all available candidates. For instance, in our work, we have reported results for the Ditto method using two predefined architectures, resnet18 and DARTS. Considering both architectural experiments, the wall clock time has doubled to 8 hours and is even higher if we include the hyper-parameter search time for both architectures, yet the performance remains significantly lower 12% lower than SPIDER).
Given the data hetcrogeneity and data invisibility challenges of FL., mamually scarching for a neural archi- tecture that works optimally for all clients can bocome prohibitively expensive. This challenge has been one of the motivations for developing SPIDER as personalized neural architecture search framework.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Table 8: Accuracy versus Computational Cost Tradeoff on CIFARI00 dataset for SPIDER versus the other represen- tative personalized federated learning techniques such as Local Adaptation, Ditto, KNN-Per, perFedAug. FedMN.
Mathod
Accurscy
Peak Monory Cost
Wall Clock Time
SPIEER
T2.06%
L18GB
15 hour
FoINAS
68.00%
2140BB
hour
Tocal Adsptation Ro
00.92%
145GB
hour
KNN-Pa- RosNet18
67.06%
1.4GBB
hour
ReaNeti8
porTodAng ResNotiS
FalMN ReoNe118
92%
Overhead of Bilevel Optimization: Another factor contributing to overhead is the Bilevel Optimization where the lower-level optimization optimizes the global model parameters and the upper-level optimization optimizes the local architecture and its weights parameters. For example, we observe the wall clock time of SPIDER to be almost double the wall clock time of FedNAS (NAS-based method that does not deploy bilevel optimization for personalization). However, SPIDER demonstrates approximately 5% increase prediction performance. Likewise, we observe the wall clock of Ditto to be double the wall clock time of Local Adaptation.
It is important to highlight that our main focus has been to address the data-heterogennityy challenge, especially in cross-silo FL settings where silos can have higher computational resources and non-llD) data distributions are the main challenge. We show via extensive empirical experiments that the proposed method yields higher predictive performance with lower standard deviation (a proxy metric for fairness) across silos as compared to other state-of-tho-art methods the cost of higher computational power.


Reviews:
[{"review_id": "aOyUQXPgnL", "reviewer": "Reviewer_wBn9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is well motivated. That is, most previous methods on personalized FL only aim to personalize the weights of the neural network. This paper goes one step further and additionally personalize the neural architectures, and hence achieves improvements of these previous methods.\n- The algorithmic designs are all intuitive and reasonable. The proposed method of regularization to enforce knowledge sharing between the global model and local models is particularly interesting.\n- The experiments are nicely done. The results in Table 1 indeed demonstrate improvements over the previous methods. Fig 2 is a nice illustration of the reasonableness of the proposed approach of personalizing the neural architectures for different clients. The design of the personalization gain is also interesting and reasonable.\n\nWeaknesses:\n- I think the computational cost of the proposed method should be evaluated or discussed. If I understand correctly, the proposed method of personalizing the neural architectures is much more complicated and hence potentially more time-consuming than the previous methods which only personalize the model weights (e.g., the baseline methods compared in Table 1). I understand that sometimes speding more computation is necessary to achieve improvement in the performance such as those shown in Table 1, but I think comparisons should be made in terms of the computational cost.\n- Although there has been some previous works on doing NAS in FL (as reviewed in Section 2), however, the proposed method is only compared with one of these methods (Federated NAS) in Appendix C, and the results show that the proposed method isn't really siginificantly better than FedNAS since the proposed  only starts to outperform FedNAS after a large number of iterations. I think more comparisons with the previous federated NAS methods should be performed, or if comparisons with the previous methods mentioned in Section 2 are not feasible, it should be clearly explained. \n- As described in Section 4.4, the local architecture searching phase is performed using a simple perturbation-based heuristic instead of a gradient-based NAS method, even though the proposed method adopts the DARTS search space which was developed for gradient-based NAS. I think it would be nice to see whether the use of gradient-based NAS (such as DARTS) can further improve the local architecture search phase.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for personalizing the neural architectures for differnt clients in federated learning. The proposed method consists of two phases. During the training phase, the heterogeneous architectures of different clients are collaboratively trained by regularizing their weights using a global architecture model. Then during the searching phase, the personalized architectures of every client is further optimized using a simple perturbation-based heuristic method.The experimental comparison shows that the proposed method which personalizes the neural architecture improves over previous methods which only personalize the model weights.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the points I've listed under Weaknesses above for the requested clarifications and changes.", "broader_impact_concerns": "I do not see ethical concerns about this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r48Ou4jSqL", "reviewer": "Reviewer_umJq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's strength lies in its exhibition. The method is well-motivated and explained along with helpful graphics and algorithm descriptions. The problem is highly relevant and I enjoyed reading the paper. \nThe paper's weaknesses are as follows:\n- Adversarial attacks claim. There is no study of the method's robustness to adversarial attacks and the claim is hand-wavy. Given that the method optimizes $w_k$ locally and communicates to the server, there is ample opportunity to adversarially attack $w$ by a malicious client. \n- Missing baseline and opportunity. The authors present their method as a solution to R1 & R2 specifically. It involves not only optimizing $A_k$, but also includes finding localized model weights $v_k$ for those localized architectures. The FL-aspect stems solely from $\\frac{\\lambda}{2}Reg(v_k,w*)$. I would like to see the baseline in which $\\lambda \\rightarrow \\inf$ and the clients' and server weights are kept identical. Instead of personalizing both, weights and architecture, such a method would be simpler and involve only architecture search as a means of personalization. With a search-space of size $2^{14}$, there should be enough opportunity for personalized results. Such a simpler method would also open up the possibility for cross-device FL and one-shot personalization as a client could perform the perturbation based architecture search upon joining the federation. This method would still satisfy R1-R3 as far as I can tell.\n- Missing evaluation opportunity: I would have loved to understand if similarity in data is followed by similarity in the selected model architecture. Do two clients with similar (or even identical) label-distribution 'find' a similar (or identical) architecture? For two clients with identical label distribution, such an outcome would be desirable since it increases the statistical strength of the shared model architecture weights.\n- Datasets and settings: While I believe the results to be conclusive given the presented datasets, it would have been a stronger statement for the paper to present a dataset of different modality, scale or task-structure. At the moment, the experiments capture only image classification on low-resolution natural images. The problem definition of FL-NAS however captures a much wider range. I would encourage the authors to consider e.g. a text-modeling task or an object-localization task and dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for model personalization in cross-silo federated learning by finding a silo-specific NN architecture and corresponding weight vectors. The authors showcase their method with a range of toy-examples and analyze its performance comparing it with other model personalization methods in FL, including an analysis of std as a proxy for fairness in this setup.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In order to recommend acceptance, I would like the authors to discuss the simple baseline of keeping client and server weights identical, similar to the traditional FedAvg setting. Is there a reason why this baseline does not make sense in this setting? \nFurther, please substantiate or remove the claim about adversarial robustness. \n\nBeyond those, I found the following small issues:\n- Figure 3 legend: You reference 2a, 2b and 2c as sub-plots, the numbering is off. \n- Figure 2 legend vs text. In the legend you claim Cifar10, in the text you mention Cifar100.", "broader_impact_concerns": "There are no concerns with ethical implications. I would encourage the authors to consider the case of hardware-heterogeneity and how different resource constraints could impact the NAS procedure as an outlook to potential future work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yJgt1AoqAs", "reviewer": "Reviewer_ZfAU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n- The overall idea is clear and makes sense.\n- The empirical performance shows that compared to other state-of-the-art methods SPIDER obtains better accuracy with a smaller model.\n\n**Weakness**:\n- It is unclear to me how large is the wall clock overhead of SPIDER compared to other methods. \n- The authors did not provide guarantee that such bilevel optimization will converge.\n- How does the stochasticity of NAS influence the result?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of personalization in federated learning. The authors propose a novel neural architecture search based method, termed SPIDER, to address this problem. More precisely, they \napply general federated learning to sync global model and use a NAS method to search for local models and use ditto regularization to penalize the distance between shared weights and local weights. They use DARTS search space and use validation set to pick the best architecture. The empirical performances suggest that SPIDER outperforms other state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses in the previous section.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The computational overhead of the methods was discussed. Although the proposed method incurs more computational time, the large performance advantage of the proposed method seems to justify the additional computations.
Decision: Accept (poster)